diff --git a/.claude/skills/planning-prebid-aws/SKILL.md b/.claude/skills/planning-prebid-aws/SKILL.md new file mode 100644 index 000000000..27134562a --- /dev/null +++ b/.claude/skills/planning-prebid-aws/SKILL.md @@ -0,0 +1,91 @@ +--- +name: planning-prebid-aws +disable-model-invocation: true +description: Plan Prebid Server Go deployments on AWS through an interactive requirements interview. Use when choosing deployment architecture or AWS services, or generating Terraform, runtime configuration, deployment scripts, and runbooks for a new or revised PBS deployment. Stops before cloud changes and live traffic. +--- + +# Planning Prebid Server Go on AWS + +Produce an approved deployment design and locally checked implementation files. Prebid Server Go is fixed; ask for its release and image source, not its implementation language. + +## Authority + +This workflow permits planning and, after design approval, file generation and safe local checks. Cloud provisioning, credential writes, application deployment, production traffic changes, and teardown require a separate execution workflow with explicit authorization. Generated scripts and CI jobs must remain inactive. Ask before authenticated AWS inspection or state access, naming the account, role, regions, and read scope. Keep real credentials out of the conversation and generated files. + +## 1. Establish the baseline + +Inspect repository instructions, Git status, existing infrastructure, runtime files, and supplied documents. Preserve unrelated changes. Identify the target repository and deployment directory before proposing edits. + +For read-only discovery from any existing `trusted-server.toml`, and for designing the operator commands and PBS config/secret delivery, read [configuration and secrets](references/configuration-and-secrets.md). Select the authoritative config source before inferring requirements; examples and disabled integrations are not active deployment inputs. + +Record requirements in one decision record, initially in the conversation, then in the approved deployment plan: + +| Requirement | Value | Status | Evidence or decision owner | Blocks | +| -------------------- | ----------------------- | ---------------------------------- | -------------------------- | ----------------------------------------- | +| One row per decision | Answer or open question | Confirmed, proposed, or unresolved | Source or person | Design, generation, live rollout, or none | + +Treat supplied examples as evidence of intent, not accepted requirements for this deployment. Reconcile conflicting inputs with the user. Inspect existing answers before asking again. + +Done when the current deployment, requested outcome, reusable resources, selected Trusted Server config source or its absence, and unanswered decisions are identified. + +## 2. Interview by decision + +Ask two to four related questions per turn. Start with questions that change the architecture; explain the consequence of unfamiliar choices. Offer a recommendation the user can accept rather than requiring AWS expertise. + +Cover these topics, skipping already confirmed answers: + +- Purpose: disposable test, live pilot, or ongoing production; retained provider and migration scope. +- Availability: acceptable interruption, host/AZ/region failure tolerance, maintenance windows, recovery time, and loss tolerance for required data. +- Workload: absolute peak eligible auction QPS, allocation, regional mix, bidder fan-out, payload sizes, burst duration, and caller latency budget. +- Operations: fixed headroom versus automatic scaling, budget ceiling, owner and backup, existing CI and AWS platform, regions, DNS, network restrictions, and bidder IP allowlists. +- Security: permitted callers/publishers, public or private access, data residency, retention, and the privacy policy owner. +- Auction behavior: caller, bidder set, formats, consent and identity, account settings, stored requests, and cache dependencies. Read [Prebid Go requirements](references/prebid-go.md) before resolving these inputs. +- Outbound bidder connectivity: for each bidder and region, choose public/NAT egress, an internal RTB Fabric link, or an RTB Fabric outbound external link. If RTB Fabric is a candidate, read [RTB Fabric connectivity](references/rtb-fabric.md) before asking the conditional questions. Record partner participation, gateway and link ownership, PBS endpoint mapping, regional support, quotas, timeout, cost, fallback, and monitoring. + +When RTB Fabric is a candidate, ask these questions in related groups: + +1. Which bidders participate in RTB Fabric, in which regions, and which partner provides each responder gateway ID? Who accepts and owns each link? +2. For each participating bidder, should PBS use an internal Fabric link or an outbound external link? What remains on NAT egress, and what is the explicit behavior when a Fabric link is pending, unavailable, or over quota? +3. Which pinned PBS Go release and adapter configuration own the endpoint mapping? Who approves link creation, partner acceptance, configuration rollout, and endpoint changes? +4. What peak transactions per second, payload sizes, bidder deadlines, regional failover load, and monthly volume should size each link and compare Fabric cost with NAT? + +Translate "production ready" into measurable availability, security, capacity, and recovery requirements. Scaling and availability are separate decisions. Offer a measurement plan for unknown traffic rather than inventing capacity. + +Done when every topic is confirmed, explicitly inapplicable, or recorded as an unresolved blocker. For an RTB Fabric branch, every participating bidder and region has a selected path, partner owner, endpoint mapping, quota and timeout check, cost assumption, fallback, and monitoring owner. Continue a provisional design around unknowns, but pause affected file generation until architecture-changing decisions are approved. + +## 3. Recommend and obtain approval + +Read [architecture decisions](references/architecture.md). If the interview selected RTB Fabric, also read [RTB Fabric connectivity](references/rtb-fabric.md). For Terraform state/authentication decisions and HCL generation, read [Terraform guidance](references/terraform.md). For a two-region standalone-host pilot, consult [the worked example](examples/two-region-pilot.md); its values remain conditional. + +Present one recommended design and only alternatives that resolve a real tradeoff. Include: + +- A top-to-bottom Mermaid diagram and the failure boundaries. +- Each selected AWS service, its requirement, and whether to reuse or create it. +- Capacity assumptions, cost drivers and estimate date, accepted limitations, and blockers. +- Runtime, infrastructure, secret, and traffic-control ownership. +- Use the experimental `ts prebid server` commands for supported local checks, secret writes, and EC2 status. Record the selected YAML/secret delivery path and unsupported operations explicitly. Deployment, rollback, and other runtime support need a separately approved implementation; avoid competing wrappers for implemented commands. +- Target files and checks, with cloud-dependent checks separated from local checks. + +Ask the user to approve the architecture, assumptions, target files, and accepted limitations. Approval to generate files is not approval to execute them. Reopen approval if later findings change topology, cost commitments, or ownership. + +Done when the user explicitly approves the design and file scope. If blockers remain, agree on a bounded draft and label it incomplete rather than producing apparently deployable infrastructure. + +## 4. Generate the approved files + +Read [file generation and validation](references/file-generation.md). Follow existing repository conventions and generate only artifacts used by the selected design. Keep the decision record in the deployment plan; reference it from the runbook rather than repeating it. + +Read the [PBS CLI usage and descriptor schema](../../../crates/trusted-server-cli/README.md) before generating inputs consumed by `ts prebid server`. Its current descriptor supports EC2/Compose only. Keep other architecture choices available, but mark their CLI integration deferred rather than generating unsupported fields. + +Verify version-specific PBS fields and adapter bindings against the selected release. For an approved RTB Fabric branch, reread [RTB Fabric connectivity](references/rtb-fabric.md) and keep partner acceptance, link activation, and any unsupported CLI integration visible as separate work. Verify AWS/Terraform behavior and pricing against current primary documentation. Record source links, versions, and verification dates in the deployment plan. Unavailable evidence remains a named blocker; do not invent image digests, configuration keys, prices, or benchmark results. + +Done when every approved artifact exists, has a named owner and check, and every unresolved input is visible and prevents unsafe use where applicable. Every generated operator command must have documented inputs, access requirements, output, failure behavior, and a recovery action; proposing command names alone is not implementation. + +## 5. Validate and hand off + +Run the applicable safe checks in the loaded generation references and inspect the final diff. Report changed paths, exact commands, results, checks not run, and remaining blockers. Separate these states: + +- Draft: unresolved generation inputs or incomplete artifacts. +- Locally checked: applicable local checks passed; cloud and integration behavior remain unverified. +- Ready for deployment review: file scope complete, blockers for generation cleared, local evidence recorded, and cloud plan/live checks listed for the authorized operator. + +This skill never establishes production readiness from generated files alone. End with the next approval or evidence needed, not a provisioning command executed on the user's behalf. diff --git a/.claude/skills/planning-prebid-aws/examples/two-region-pilot.md b/.claude/skills/planning-prebid-aws/examples/two-region-pilot.md new file mode 100644 index 000000000..d942c708e --- /dev/null +++ b/.claude/skills/planning-prebid-aws/examples/two-region-pilot.md @@ -0,0 +1,74 @@ +# Worked example: two-region pilot + +This is a conditional design derived from a supplied pilot plan, not a default architecture or evidence of provisioned capacity. Use it only when the user accepts its limitations. Operational rules live in the skill references. + +## Example requirements + +Assume the user has confirmed: + +- Prebid Server Go on AWS with Terraform. +- One deployment in `us-east-1` and one in `us-west-2`. +- A retained provider and a caller-controlled trial targeting 5% of eligible traffic. +- A fixed approved bidder set, versioned configuration, and credential rotation. +- Host maintenance interruptions, manual host replacement, and no autoscaling for this bounded pilot. + +Still resolve absolute peak traffic, regional distribution, bidder fan-out, caller type, domain ownership, inventory/cache dependencies, secret permissions, budget, recovery expectations, and numeric rollout gates. The 5% value supplies no instance-size evidence. + +## Conditional recommendation + +```mermaid +flowchart TD + A["Existing caller"] --> S["Caller experiment allocation"] + S -->|"95% of eligible traffic"| P["Existing provider"] + S -->|"5% of eligible traffic"| R["Selected regional routing"] + R --> E["East EC2: Caddy and PBS Go"] + R --> W["West EC2: Caddy and PBS Go"] + E --> B["Approved bidder endpoints"] + W --> B +``` + +Per region, propose a public subnet, internet gateway, EC2 host with encrypted storage and Elastic IP, Compose, Caddy, SSM access, regional secret access, release artifacts, and telemetry. Select actual instance sizes and images only after workload and compatibility evidence. + +Prefer existing caller-side regional routing when suitable. Otherwise evaluate shared-hostname Route 53 latency routing with independent regional health checks and a verified DNS-challenge certificate setup. Implement only the selected branch. + +Use one pilot Terraform root with two provider aliases and explicit regional module calls, unless ownership requires separate roots. Keep backend bootstrap independent. Generate Compose/Caddy and host deployment tools only after approving this profile. + +No ALB, NAT gateway, ECS, distributed database, or cache is justified solely by the pilot's two regions. Add or replace components when a confirmed requirement demands it. Exclude inventory with unmet dependencies explicitly rather than presenting a reduced-function trial as equivalent. + +## Approval and evidence gates + +The user must accept one-host-per-region outages and maintenance behavior. If regional failover concentrates the pilot on one node, prove that capacity or agree on reducing allocation. Automatic replacement, uninterrupted deployment, or tighter recovery requirements invalidate this profile and require another architecture decision. + +File generation starts after architecture-changing questions and target paths are approved. Live-traffic gates remain deferred to authorized execution: zero-allocation deployment checks, internal inventory validation, then agreed 1% and 5% observation windows. A tested caller kill switch and measured refresh bound are required before the live pilot. + +## Configuration and operator walkthroughs + +The experimental `ts prebid server` CLI implements local inspection/checks, secret value writes, and EC2 infrastructure status. The release, runtime delivery, and rollback scenarios below remain acceptance criteria for a future approved implementation, not executed deployment evidence. Consult the [current command contract](../references/configuration-and-secrets.md#operator-command-contract) before documenting an invocation. + +| Input or task | Expected behavior | +| ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Several `trusted-server.toml` files, including a disabled example | Ask which source/environment is authoritative; preserve every source file and report uncertainty about remote overrides | +| The config lists server bidders, client-side bidders, and bundle adapters | Classify them separately; verify candidate server adapters and report host-secret needs as required, not needed, or unresolved without activating bidders | +| An operator changes a PBS timeout | Check and render regional YAML, prepare an immutable release, preview and approve deployment; no manual environment-file edits or Terraform apply | +| A binding conflicts with YAML or a required secret key is missing | Reject conflicting local inputs or stop the authorized deployment preflight; preserve working capacity and never print credential values | +| A credential contains quotes, dollars, or newlines | Prompt without echo or accept file/stdin, validate and safely encode it; no secret-value command argument or log output | +| East consumes a rotated credential while West still runs the old version | Report actual regional versions and pending work, verify replication and replacement, then coordinate partner revocation; never report global success from East alone | +| An operator requests rollback after the old credential was revoked | Refuse known-incompatible recovery and explain the credential action needed; a Git release rollback cannot restore bidder validity | +| A command resolves the wrong AWS account or an ambiguous deployment | Stop before mutation and require corrected target selection; local inspect/check stay credential-free | +| ECS is selected instead of Compose | Generate one concrete YAML delivery mechanism and task secret references behind the same operator commands; do not generate unused host loaders | + +## Walkthrough assertions + +Use these contrasts when reviewing the skill. They are expected behavior, not executed deployment tests. Terraform cases exercise the rules in `references/terraform.md`. + +| Input | Expected planning behavior | +| ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| This pilot, with outage acceptance but unknown QPS | Preserve the supplied answers; ask about workload and other blockers; propose this profile provisionally without inventing instance capacity | +| Production requires AZ survival, uninterrupted releases, and automatic scaling | Reopen topology; evaluate multi-AZ ECS/ALB or EC2 Auto Scaling/ALB against the existing platform; define minimum/failover capacity and scaling evidence; generate only the selected runtime's files | +| Pilot adds video with independent regional caches | Resolve cache write/retrieval and failure behavior before generating affected infrastructure, or obtain approval to exclude that inventory | +| User approves file generation but has supplied no AWS execution authorization | Generate inactive tools and run safe local checks; leave cloud actions and traffic changes unexecuted | +| A requirement is unknown or a validation tool is unavailable | Record a blocker or not-run check; preserve the distinction between a draft and locally checked files | +| A proposed local test omits `command` and uses a real AWS provider | Do not run it: the default is apply. Generate an explicitly commanded, isolated mocked test or defer it as an authorized cloud integration test | +| A plan-mode suite mocks East but retains a real West provider alias | Reject the credential-free claim; inspect all provider mappings and setup dependencies, mock the remaining external provider, and select exact reviewed test files | +| The existing S3 backend uses DynamoDB locking | Preserve it while proposing a version/permissions/recovery-aware migration; no backend replacement, lock removal, or state migration during generation | +| An upstream CI example uses stored AWS keys and applies after merge | Generate the approved short-lived identity and protected saved-plan review procedure as inactive tooling; upstream examples grant no execution authority | diff --git a/.claude/skills/planning-prebid-aws/references/architecture.md b/.claude/skills/planning-prebid-aws/references/architecture.md new file mode 100644 index 000000000..df3783e65 --- /dev/null +++ b/.claude/skills/planning-prebid-aws/references/architecture.md @@ -0,0 +1,74 @@ +# Architecture decisions + +Select the smallest design that meets the approved requirements. Reuse an established AWS platform when it meets them; avoid introducing a second operations model just for PBS. + +## Runtime and availability + +| Requirement | Candidate | Decision to resolve | +| ------------------------------------------------------------------ | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------- | +| Bounded pilot accepting host outages and maintenance interruptions | One EC2 host per selected region, Compose, Caddy, SSM | Explicit outage acceptance, replacement procedure, and measured fixed headroom | +| Automatic host replacement with an existing EC2 operating model | Launch template, Auto Scaling group, regional ALB | Multi-AZ placement, minimum capacity, health checks, draining, and instance refresh policy | +| Managed container lifecycle and rolling deployments | ECS service with Fargate or EC2 capacity, regional ALB | Existing platform, resource fit, sustained cost, capacity provider, and rollout headroom | +| Variable demand requiring elastic capacity | Scaling policy on the chosen runtime | Minimum/maximum capacity, signal, target, startup time, cooldown, quotas, and behavior at the cap | + +A region containing one host is still a regional single point of failure. Host replacement, multi-AZ availability, regional failover, and zero-downtime releases each need their own evidence. ECS alone establishes none of these without the corresponding service configuration and capacity. + +Use EKS only when the user's existing platform or an explicit requirement justifies Kubernetes ownership. Do not infer a need for orchestration from the word "production" alone. + +## Network, ingress, and egress + +| Need | Candidate services or mechanism | Required evidence | +| --------------------------------------- | ------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------- | +| Public HTTPS on standalone hosts | Public subnet, internet gateway, Elastic IP, Caddy | Accepted exposure, durable certificate storage, renewal and replacement path | +| Managed HTTPS with multiple backends | ALB and ACM | Certificate ownership/validation, target health, draining, timeout budget, private management endpoints | +| Private compute reaching public bidders | NAT-based egress or an approved existing egress service | AZ failure behavior, routing, hourly/data charges, and outbound address stability | +| Private PBS-to-bidder connectivity | AWS RTB Fabric requester gateway and standard or outbound external links | Partner participation, gateway and link ownership, regional support, quotas, bidder endpoint mapping, timeout, fallback, and cost | +| Bidder source-IP allowlists | Stable egress IPs | Every normal, scaling, and failover path uses partner-approved addresses | +| Existing caller selects regions | Regional hostnames | Caller routing and failure policy, TLS, identity behavior | +| DNS-based regional selection | Route 53 latency records with per-region health checks | Independent health targets, cached-answer behavior, all-unhealthy behavior, and failover capacity | +| Public auctions from untrusted callers | Existing controls or WAF with compatible ingress | Auction payload compatibility, rate limits, false-positive testing, cost, and owner | + +Private subnets do not provide internet egress by themselves. VPC endpoints may serve supported AWS APIs but do not replace bidder internet access. Public addressing on replaceable compute does not by itself provide stable egress for allowlists. + +For a shared hostname on standalone Caddy hosts, specify a workable multi-host certificate issuance method. Route 53 DNS challenges require the matching Caddy module, a pinned custom image, and scoped DNS permissions. Choose one routing/TLS design rather than generating all alternatives. + +Define trusted proxy hops and client-IP handling, ingress ports, metrics isolation, outbound needs, and management access. Specify publisher/account admission and request limits for public auctions; CORS alone is not authorization. For EC2, use SSM without inbound SSH, require IMDSv2, encrypt disks, and separate host AWS access from application access. For ECS, distinguish the execution role from the application task role. + +## Supporting services + +For each item, record reuse, create, or not needed, with the reason: + +- State storage and locking, independently owned from the deployment it records. +- Secrets Manager or an approved existing secret service, with regional access, rotation, and least-privilege permissions. +- Artifact storage, typically versioned S3 releases and an approved image registry such as ECR. Include checksums, retention, and regional startup independence where required. +- Logs, metrics, alarms, notification delivery, and budget alerts through CloudWatch or existing systems. Resolve metrics collection and dashboards, not just a log group. +- Stored-request/account storage and Prebid Cache only where the selected auction workload needs them. A generic Redis or database deployment is not automatically a PBS integration. + +Alert on caller failures and latency, bidder behavior, capacity, resource exhaustion, and cost. Budget alerts are notifications, not a hard spending cap. Define data retention and redaction before recording auction diagnostics. + +## Capacity and cost + +Calculate from measured workload: + +```text +pilot_peak_qps = eligible_peak_qps * allocation_fraction +regional_peak_qps = pilot_peak_qps * measured_regional_share +outbound_qps = pilot_peak_qps * measured_bidder_calls_per_auction +``` + +Use representative payloads and controlled bidder responses to establish throughput, tail latency, resource limits, and slow-bidder behavior. Include deployment overlap and the approved failure scenario in required headroom. If failover concentrates traffic, either prove surviving capacity or define how allocation is reduced. + +Scaling needs a load-tested signal that predicts auction saturation, plus minimum capacity for bursts during startup. Consider CPU, memory, concurrency, connection limits, and outbound bandwidth. Never equate an instance count or CPU percentage with a measured QPS guarantee. + +Estimate compute, storage, public IPv4, load balancing, NAT processing, internet/inter-AZ/inter-region transfer, DNS/health checks, secret retrieval, artifacts, and telemetry. State region, prices checked on, traffic assumptions, and uncertainty. Unknown volume means a conditional estimate, not a quoted total. + +## Sources to verify for the selected branch + +Consult current AWS documentation when recommending concrete settings: + +- [ECS service autoscaling](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html) +- [EC2 Auto Scaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/what-is-amazon-ec2-auto-scaling.html) +- [NAT gateway basics](https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-basics.html) +- [Route 53 latency routing](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-latency.html) +- [Caddy Route 53 module](https://github.com/caddy-dns/route53) +- [AWS Pricing Calculator](https://calculator.aws/) diff --git a/.claude/skills/planning-prebid-aws/references/configuration-and-secrets.md b/.claude/skills/planning-prebid-aws/references/configuration-and-secrets.md new file mode 100644 index 000000000..f0ff876d6 --- /dev/null +++ b/.claude/skills/planning-prebid-aws/references/configuration-and-secrets.md @@ -0,0 +1,110 @@ +# Configuration, secrets, and operator workflow + +Use this reference for Trusted Server configuration discovery, the experimental `ts prebid server` operator commands, and proposed runtime delivery. The current commands are implemented in `crates/trusted-server-cli`; deployment and rollback remain design work. Their existence does not authorize cloud operations. Read the [CLI usage and schema](../../../../crates/trusted-server-cli/README.md) before generating its input files or documenting invocations. + +## Discover requirements from Trusted Server + +Before asking questions the repository can answer: + +1. Locate `trusted-server.toml` in the selected project, including ignored operator-owned files and any path supplied by the user. Limit discovery to that project. When several files or environments could apply, ask which is authoritative; never select one by modification time. If none exists, continue the interview and label example files as examples. +2. Parse the selected file with a TOML parser and inspect only relevant fields. Check their semantics against the repository's Trusted Server schema. Comments, disabled integrations, and browser build inputs do not establish active server-side demand. +3. Ask whether environment overrides, remote configuration, or request-time parameters make the file incomplete or stale. Record the effective source as unresolved when it cannot be established locally. Cloud inspection still requires authorization. +4. Add observations to the existing deployment decision record with source path/key, candidate requirement, confidence/status, and the remaining question. Report credential identifiers or presence only; keep raw TOML, credential values, and commercially sensitive publisher values out of reports and generated examples. +5. Verify candidate adapters against the selected PBS Go release and confirm partner authorization. Record each host-secret requirement as required, not needed, or unresolved. A bidder name alone proves neither a secret requirement nor permission to use the bidder. + +| Trusted Server input | Discovery use | +| ------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- | +| `auction.enabled` | Determine active versus disabled server-auction intent without expanding defaults | +| `auction.providers.` with `profile = "prebid-server"` | Identify each configured PBS provider; withhold endpoint values | +| Provider `timeout_ms` and `profile_config.test_mode` or `debug` | Record explicit timeout and diagnostic intent, not permission to contact bidders | +| Provider `profile_config.bid_param_override_rules` | Count conditional publisher or placement inputs without reporting their values | +| `auction.bidders..provider` | Group candidate server-side adapters under the referenced PBS provider | +| `integrations.prebid.enabled`, `account_id`, `timeout_ms`, `debug` | Record browser integration intent and compatibility questions separately from server demand | +| `integrations.prebid.client_side_bidders` | Browser-side participation; do not automatically enable these adapters in PBS | +| `integrations.prebid.bundle` | Browser bundle and identity capabilities, not proof of server-side use | +| Relevant privacy, format, and stored-request settings | Identify dependencies that need confirmation from the caller or request path | + +Use `ts prebid server inspect --config ` for redacted local discovery. It reports explicit values without expanding defaults or proving host-secret requirements. Resolve schema defaults and adapter metadata separately, recording unsupported details as unresolved rather than fetching credentials or inventing mappings. Read config without invoking commands that publish or rewrite it. Leave `trusted-server.toml` unchanged. Propose caller endpoint/account changes separately after integration approval; discovery must not activate bidders, publish configuration, or change traffic. + +Done when the selected source and its limits are recorded, every observed bidder is classified, and missing inputs remain visible rather than filled from examples. + +## One home for each input + +Use existing repository paths where available. For a new deployment, propose: + +| Input | Owner and destination | +| ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | +| `runtime/pbs.yaml` | Git-owned nonsecret PBS baseline | +| `runtime/regions/` | Optional nonsecret regional overrides, rendered into each region's YAML | +| `runtime/secret-bindings.json` | Git-owned secret identifiers, JSON key names, supported PBS destinations, types, required/optional status, and regional scope; no values | +| `deployment.yaml` | Nonsecret deployment descriptor: environment, AWS account/profile or role, regions, selected runtime, resource identifiers, and artifact locations | +| Credential values | Secrets Manager through an authorized operator or automation workflow | +| Publisher/site/placement parameters | The approved caller configuration or stored-request system, passed with auctions | + +PBS Go supports files and environment variables, with environment values taking precedence. Default to YAML for normal PBS settings and environment bindings for supported host secrets. Keep unrelated process/AWS environment variables separate. Any nonsecret PBS environment override must be explicit, generated, and justified; never forward arbitrary operator `PBS_*` variables into the container. + +Resolve one YAML file per region using a real parser. Define precedence as baseline then regional overrides; merge mapping keys and replace sequences as whole values. Validate the resolved fields and binding types against the pinned PBS release. A secret-bound field has one source: reject conflicting YAML values, duplicate environment destinations, and unknown bindings instead of silently choosing a winner. Keep unresolved required settings blocking release preparation. + +Cover external URL, listener, adapters/endpoints, timeout, privacy/account policy, identity, stored requests/cache, and telemetry. Preserve required upstream static assets. The privacy policy owner approves privacy defaults; a smoke test does not justify weakening them. + +## Release and runtime delivery + +This is the design contract for a separately approved runtime integration, not a capability of the current CLI. If Terraform user data currently owns runtime files, agree on the ownership transition before implementing independent release updates. A Secrets Manager write alone does not install a secret loader or replace consumers. + +```mermaid +flowchart TD + T["Selected Trusted Server config"] --> Q["Read-only discovery and approved requirements"] + Q --> Y["PBS YAML, regional overrides, secret bindings"] + Y --> R["Validated immutable release, no secret values"] + R --> D["Separately authorized deployment"] + S["Secrets Manager versions"] --> D + D --> P["PBS: resolved YAML and supported secret environment"] + P --> V["Health, auction checks, and deployment record"] +``` + +An immutable release records resolved nonsecret configuration checksums, PBS and ancillary image digests, required bindings, and optional stored-request assets. Pin CPU-compatible images. Resolve intended regional secret versions for an authorized deployment and record their identifiers with the release and target. Make containers consume those versions; recording a version while injecting an unconstrained latest value is not reproducible. + +Generate only the approved runtime's mechanism: + +- EC2/Compose: a versioned release supplies `/etc/config/pbs.yaml` through a read-only mount. A host-side role retrieves the selected secret versions and validates required keys before replacement. A parser/encoder writes restricted runtime environment files atomically on ephemeral storage; recreate them on boot. Test Compose quoting, dollars, newlines, and empty values. Never construct environment files through shell evaluation. Keep instance-role credentials inaccessible to the application. +- ECS: prefer an immutable configuration-bearing image containing the resolved nonsecret YAML and required assets. If using artifact retrieval instead, specify the downloader, permissions, checksums, startup ordering, and failure handling; an S3 object alone is not configuration delivery. Use supported task-definition secret references with the selected versions and execution/task-role permissions. Verify replacement tasks can load configuration and credentials before retiring healthy capacity according to the approved availability policy. + +Never package or log secret values. Treat container inspection and debug output as privileged; test redaction with dummy secrets. Follow [Terraform ownership](terraform.md#credentials-and-resource-ownership) for metadata and access policy instead of recreating that responsibility in runtime scripts. + +## Operator command contract + +Use `ts prebid server` instead of generating deployment-local wrappers for these implemented operations. Keep existing `ts config`, `ts deploy`, and `ts prebid client` behavior unchanged. + +| Command | Current result and boundary | +| ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `ts prebid server inspect --config ` | Local discovery with unresolved requirements; no source changes or secret retrieval | +| `ts prebid server check --deployment ` | Local schema, binding, and regional merge checks; no AWS calls, upstream PBS schema validation, or startup proof | +| `ts prebid server secrets set --deployment --region ` | Authorized complete-value write to an existing declared secret; hidden terminal or file/stdin input, account/confirmation checks, and retry UUID; no deployment | +| `ts prebid server status --deployment ` | Authorized EC2 infrastructure reads for declared instances; PBS health, installed release, and consumed secret versions stay unknown | + +There are no `ts prebid server deploy` or `ts prebid server rollback` subcommands. Versioned release delivery, runtime injection, and release-aware status need separate implementation and approval. Do not present the future release contract as a working command. Current descriptors accept only `ec2-compose`; an ECS design requires another approved implementation rather than a fabricated descriptor. + +Read targets from the deployment descriptor. Allow an explicit deployment selection; if selection is ambiguous, stop. Before cloud operations, display environment/account/regions and verify authenticated account identity against the descriptor. Require deliberate regional scope for mutations rather than silently changing every region or defaulting to production. Local commands do not authenticate. Cloud read commands must not retrieve credential values merely to report status. + +Interactive mutations require confirmation of the concrete target and proposed change. Automation uses explicit target, release or secret-write inputs and an external approval gate; missing approval is an error, not a prompt fallback. Release preparation may render/build locally, but publication and cloud changes occur only after approval. If the preview changes, obtain approval again. A skill run that only authorizes file generation must not invoke these cloud commands. + +Secret input supports a hidden interactive prompt in the operator's terminal, not the agent conversation, and mutually exclusive file/stdin inputs. Accept no credential values as ordinary arguments. Validate a complete payload against the binding definition; use restricted temporary storage only when needed and clean up tool-created files on success or failure. Leave operator-supplied input files unchanged. Write the declared primary secret for replicated credentials or the selected independently owned regional secret, not a replica as if it were writable independently. Preserve retry identity for a logical write so retrying it does not create unintended versions. Report changed version identifiers and regions requiring deployment, never the payload. + +A future approved deployment implementation must own the render/package/inject/restart sequence; operators should not manually edit generated environment files or ECS task definitions. Reuse the [release lifecycle](file-generation.md#generated-deployment-tools) for locking, health deadlines, draining, and failure handling. Routine configuration or credential-value changes do not require Terraform apply. New secret metadata, permissions, or infrastructure changes follow the separate reviewed Terraform workflow. None of these commands modifies caller traffic allocation implicitly. + +For supported CLI commands, reference the CLI's input and failure contract in the runbook. For any separately approved deployment tool, document inputs, local versus cloud access, output, nonzero failure behavior, and the recovery action. Distinguish desired configuration from observed running state in status output; provide machine-readable nonsecret output where automation consumes it. + +## Rotation and recovery + +Updating Secrets Manager does not refresh a running container's environment. Coordinate credential issuance and overlap with the bidder, write the new value, verify regional availability, replace consumers in the approved order, and verify each region before retiring the old credential. Replication alone proves neither successful injection nor bidder authorization. Keep partial regional success visible with the release and secret versions actually running. + +On required-secret retrieval or validation failure, preserve working capacity and report the blocker. On failed deployment, use the recorded prior release and compatible credential versions; refuse a known-incompatible rollback. If partner validity is unknown, report that uncertainty before replacement rather than promising recovery. An application rollback cannot restore a credential revoked by the bidder. + +Done when the runbook gives each supported routine task one documented command, expected evidence, and failure/recovery behavior, with unimplemented release tasks clearly deferred. Prove discovery preserves the source file and redacts values; test rendering determinism, override conflicts, missing/optional keys, special-character dummy secrets, wrong-account/ambiguous-target refusal, retries, partial regional rollout, and revoked-credential recovery. Separate local fixture results from deferred live integration evidence. + +## Sources + +- [PBS Go configuration guide](https://github.com/prebid/prebid-server/blob/master/docs/developers/configuration.md), then the configuration definitions and adapter schema at the pinned release +- [ECS Secrets Manager injection](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/secrets-envvar-secrets-manager.html) +- [Secrets Manager regional replication](https://docs.aws.amazon.com/secretsmanager/latest/userguide/replicate-secrets.html) +- [Secrets Manager value updates](https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/put-secret-value.html) diff --git a/.claude/skills/planning-prebid-aws/references/file-generation.md b/.claude/skills/planning-prebid-aws/references/file-generation.md new file mode 100644 index 000000000..fb27e3fbe --- /dev/null +++ b/.claude/skills/planning-prebid-aws/references/file-generation.md @@ -0,0 +1,66 @@ +# File generation and validation + +Generate files only after the design and target paths are approved. Use the repository's existing layout; the paths below are suggestions for a new deployment, not a template to copy wholesale. The authority boundary in `SKILL.md` applies to every check and generated tool. + +## Artifact contract + +| Artifact | Suggested location | Contents and owner | +| -------------------------- | ------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Deployment plan | `docs/pbs-deployment-plan.md` | Requirements record, approved architecture, sources, costs, blockers, file/check inventory; user approves | +| State bootstrap, if needed | `infra/bootstrap/` | Independently managed backend, access and locking setup; infrastructure owner | +| Deployment root | `infra//` | Providers, variables/validation, explicit regional modules, outputs, selected DNS configuration, nonsecret examples, provider lock file; Terraform owns AWS resources | +| Reusable infrastructure | `infra/modules/` | Modules justified by repeated topology, not one module per AWS service | +| Runtime inputs | `runtime/` | Resolved PBS configuration, regional inputs, optional stored requests, image/release manifest; Git owns nonsecret content | +| Standalone-host runtime | `runtime/compose.yaml`, `runtime/Caddyfile` | Compose/Caddy and boot service only for the approved host profile | +| Managed-container runtime | Existing ECS release/task-definition layout | Task resource limits, health, logs, secrets references, service rollout settings; explicit Terraform/deployer ownership | +| Operator interface | Existing `ts prebid server` CLI and approved CI layout | CLI descriptor and documented invocations; any missing release stages need separate implementation approval | +| Runbook | `docs/pbs-runbook.md` | Preconditions, operator commands, rollout/recovery/rotation/teardown procedures and deferred tests | + +Each generated artifact must have a consumer. Add ignore rules for local credentials, runtime secret files, `.terraform`, state files, saved plans, and generated sensitive output. Track nonsecret examples. Use `example.com` hostnames and visibly fictional identifiers in examples. + +For Terraform artifacts, follow [Terraform generation and review](terraform.md), including module tests under `tests/` and the saved-plan procedure in the runbook. + +For the deployment descriptor, resolved PBS YAML, secret bindings, runtime delivery, and command behavior, follow [configuration and secrets](configuration-and-secrets.md). Generate inputs for the existing CLI rather than replacement wrappers. Document unimplemented release stages as deferred unless their implementation is separately approved. + +## Generated deployment tools + +Use the target-selection and approval contract in [operator commands](configuration-and-secrets.md#operator-command-contract), including descriptor-based interactive use and explicit automation inputs. New CI deployment jobs must remain inactive and approval-gated. File generation must not trigger existing auto-apply or deployment jobs; inspect those triggers before editing their watched paths. + +The current `ts prebid server` CLI does not implement deployment or rollback. For a separately approved deployment implementation, implement or explicitly defer each release stage: + +1. Serialize competing deployments to the same target. +2. Retrieve and verify an immutable release, image availability, and required secrets before replacing working capacity. +3. Record previous release and secret version identifiers without values. +4. Deploy with the approved outage/draining policy and bounded health deadlines. +5. Check HTTPS, application health, and an agreed auction fixture. +6. Follow [rotation and recovery](configuration-and-secrets.md#rotation-and-recovery) before restoring the previous release or reporting a recovery blocker. +7. Record target, release, timestamps, and outcome. + +Define retry limits, interrupted-run handling, and behavior when the same release is requested twice. On standalone hosts, provide boot recovery, deployment locking, bounded logs, certificate persistence, and an explicit response to hung-but-running containers. A Compose unhealthy status alone is not a restart policy. On managed runtimes, express equivalent rollout/rollback controls through the platform rather than adding host scripts. + +## Runbook and deferred evidence + +Reference the approved decision record. Include operator prerequisites and explicit account/region targeting, initial deployment, configuration updates, secret rotation, failed-release recovery, host/task replacement, alert response, and teardown with retention rules. Clearly mark these as procedures not yet executed. + +For a pilot, include the caller kill switch, its refresh deadline, staged allocation gates, and verification that traffic drained before teardown. DNS changes and host shutdown are not substitutes for caller rollback. For ongoing production, use the approved traffic and recovery policy rather than inventing an old-provider fallback. + +List deferred checks with expected results and an owner: authenticated Terraform plan review, TLS/DNS, IAM and exposed ports, regional dependency availability, missing secrets, invalid configuration, reboot/replacement, interrupted deployment, credential rotation, representative load, failover, alert delivery, and applicable caller/auction checks. Every approved availability and scaling claim needs a matching test. + +## Safe local checks + +Run applicable checks on the generated paths, recording exact commands and results. Inspect project scripts before executing them. Use isolated dummy credentials, fake bidder endpoints, and fixtures; local startup must not contact production bidders or fetch real AWS secrets. + +| Area | Check | +| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Terraform | Follow [Terraform local checks](terraform.md#tests-and-safe-local-checks), including inspected, explicitly selected mocked tests | +| Runtime structure | Parse YAML/JSON, verify manifest paths/checksums, and validate regional override structure | +| Compose branch | `docker compose --env-file -f config --quiet` and dummy-value round-trip checks | +| Operator tool | Language syntax/lint and [configuration/secrets contract tests](configuration-and-secrets.md#rotation-and-recovery), plus failed-release and interrupted/repeated invocation tests | +| PBS behavior | Approved isolated container startup and smoke fixture against controlled bidder responses, if a suitable local runtime is available | +| Final files | Diff review for scope, secret exposure, unresolved placeholders, inactive deployment triggers, and one writer per mutable resource | + +If a tool, image, network permission, or fixture is unavailable, mark that check not run with the next action. Never substitute a checklist for executed evidence or call a no-bid response proof of bidder success. + +## Sources to verify during generation + +- [SSM Run Command](https://docs.aws.amazon.com/systems-manager/latest/userguide/run-command.html) diff --git a/.claude/skills/planning-prebid-aws/references/prebid-go.md b/.claude/skills/planning-prebid-aws/references/prebid-go.md new file mode 100644 index 000000000..b90892ffb --- /dev/null +++ b/.claude/skills/planning-prebid-aws/references/prebid-go.md @@ -0,0 +1,45 @@ +# Prebid Go requirements + +Resolve these requirements before claiming the infrastructure can serve the selected inventory. Go is fixed; select an approved release, image source, digest, and compatible CPU architecture. + +## Caller and bidders + +Identify the caller as browser Prebid.js, backend, or edge service. Capture a sanitized baseline containing eligible inventory, enabled bidders, account/placement parameters, timeouts, privacy signals, and host-specific endpoints. + +For each bidder, confirm server-side authorization, host credentials, supported formats, approved endpoints, adapter support in the pinned release, and source-IP allowlists. Existing provider credentials and account IDs may not transfer to a self-hosted PBS. + +Use [configuration and secrets](configuration-and-secrets.md) to discover candidate requirements from the selected `trusted-server.toml`, distinguish caller parameters from host credentials, and define supported runtime bindings. Discovery is evidence for the interview, not bidder activation. + +For browsers, resolve CORS/OPTIONS, cookie scope, user-sync and callback URLs, consent-dependent endpoints, and identity behavior across regions. For backend/edge callers, define trusted forwarding hops and preserve device IP, privacy signals, request identifiers, and the caller deadline. Verify outgoing bidder requests use the intended device context rather than the proxy's identity. + +## Stored requests, accounts, and cache + +Ask whether eligible auctions reference stored-request IDs or depend on dynamically updated account settings. For a small fixed set, local versioned definitions may suffice. Otherwise reproduce the update and lookup behavior or explicitly restrict eligibility. Validate every referenced ID used in fixtures. + +Video and AMP can introduce Prebid Cache requirements. Determine the actual request and response behavior rather than assuming all formats share the same dependencies. Specify write location, public retrieval URL, expiration, and failure behavior. + +With independent regional caches, a key written in East must remain retrievable from the URL returned in that auction. A shared latency-routed lookup hostname can send retrieval to the wrong cache. Choose region-specific retrieval URLs with accepted availability limits, or a storage/routing design that guarantees retrieval through the approved failure scenario. Do not substitute a database product for a verified cache integration. + +## Pilot allocation and rollback + +Load this section when comparing with an existing provider or migrating traffic. + +- Define eligibility and the sampling unit, such as session or auction. Preserve cohort membership as allocation increases; zero allocation overrides old assignments for new auctions. +- Select a complete provider configuration together, including auction/sync endpoints, account IDs, and request transformations. Preserve unrelated client-side bidders. +- Keep experiment allocation in the caller or its remotely refreshed configuration. DNS regional routing is not precise per-auction allocation and is not the kill switch. +- Specify configuration refresh, maximum staleness, failure behavior, and the observed time to stop new experimental requests. Keep healthy endpoints available while in-flight work and cached configurations drain. +- Send each live auction through its selected provider. Do not duplicate bidder auctions for comparison or add unconditional sequential fallback after a partial auction or timeout. + +Agree numeric transport-error, timeout, latency, and business thresholds, with baseline, sample size, observation window, and response owner before live rollout. Measure assigned and observed traffic share, caller p50/p95/p99 latency, bidder failures/no-bids, identity match where relevant, and revenue using a consistent denominator. Record region, release, and cohort without full-payload logging by default. + +A legitimate no-bid is not an infrastructure failure. `/status` is one health signal, not proof of bidder permission, valid demand, or business equivalence. Plan auction-level smoke checks and delayed business evaluation separately. + +## Sources + +Use these entry points, then inspect the corresponding tag or commit for the chosen PBS Go release. Upstream `master` is navigation, not a reproducible configuration contract. + +- [Go configuration definitions](https://github.com/prebid/prebid-server/blob/master/config/config.go) +- [Go stored requests](https://docs.prebid.org/prebid-server/features/pbs-storedreqs-go.html) +- [Prebid.js PBS integration](https://docs.prebid.org/dev-docs/modules/prebidServer.html) +- [PBS status endpoint](https://docs.prebid.org/prebid-server/endpoints/pbs-endpoint-status.html) +- [PBS overview and cache dependencies](https://docs.prebid.org/prebid-server/overview/prebid-server-overview.html) diff --git a/.claude/skills/planning-prebid-aws/references/rtb-fabric.md b/.claude/skills/planning-prebid-aws/references/rtb-fabric.md new file mode 100644 index 000000000..c325f1593 --- /dev/null +++ b/.claude/skills/planning-prebid-aws/references/rtb-fabric.md @@ -0,0 +1,69 @@ +# AWS RTB Fabric connectivity + +Load this reference when a bidder partner offers AWS RTB Fabric connectivity or the deployment is considering it as an alternative to public/NAT egress. + +## Decision boundary + +For this PBS workflow, RTB Fabric is an optional outbound path from PBS to selected bidders. PBS is the requester. A partner's bidder is the responder. + +- A requester gateway is colocated with the PBS VPC. +- A standard Fabric link connects that gateway to a partner responder gateway. The partner must provide its gateway ID and accept the request before traffic can use the link. +- An outbound external link targets a partner's public HTTP or HTTPS endpoint. AWS may use its Global Network or public Internet routing, so treat it as a different security, latency, and cost profile from a standard link. +- PBS still fans out auctions to its configured bidders. Fabric does not multiplex traffic to multiple partners. Plan one endpoint path per participating bidder. +- Keep the regional ALB or other approved PBS ingress path separate. Replacing it with Fabric requires a different responder or inbound design and separate approval. + +Route only partner-approved bidder endpoints through Fabric. Keep the approved NAT or external path for bidders without Fabric participation until a reviewed migration removes that dependency. + +## Interview record + +For each bidder and PBS region, record: + +| Decision | Required evidence | +| -------------------- | ----------------------------------------------------------------------------------------------------------------------- | +| Participation | Partner name, responder gateway ID or public endpoint, AWS account owner, and acceptance owner | +| Link type | Standard internal link or outbound external link, with the selected region | +| PBS binding | Pinned PBS Go release, adapter endpoint field, config source, rollout owner, and link URL update procedure | +| Capacity | Peak transactions per second, bidder timeout, payload sizes, burst duration, and regional failover load | +| Fallback | Behavior while a link is requested, inactive, over quota, timing out, or unavailable | +| Operations | Link creation, partner acceptance, status checks, endpoint changes, deletion order, alerts, and recovery owner | +| Cost | Region, monthly sent transactions, payload distribution, no-bid volume, link count, external traffic, and estimate date | +| Privacy and security | OpenRTB fields, TLS mode, source-IP requirements, data residency, logging, and retention | + +Gateway IDs, link state, quota increases, and partner acceptance are external facts. Ask before authenticated AWS inspection, naming the account, role, regions, and read scope. Do not treat a partner's willingness to participate as proof that its gateway, endpoint, adapter, or regional capacity is ready. + +## Constraints to verify + +AWS currently lists RTB Fabric in US East (N. Virginia), US West (Oregon), Singapore, Tokyo, Frankfurt, and Ireland. Verify regional availability again during design. Gateways are regional, so a multi-region PBS deployment needs a regional connectivity decision for every participating bidder. + +The current default quotas include two gateways per account and region, two standard links per gateway, two outbound external links per gateway, one Availability Zone per gateway, 1,000 transactions per second per link, and a 1.5-second HTTP request timeout. Gateway and throughput quotas may be adjustable. The control-plane quota is 10 API requests per second per account and region and is not adjustable. Use the selected bidder deadline and measured traffic, not the defaults, as the capacity requirement. + +RTB Fabric supports OpenRTB versions including 2.6 and does not support OpenRTB 3.0. Verify the selected PBS Go release, bidder adapter, headers, custom extensions, and TLS behavior against the partner contract. + +AWS charges for sent RTB requests and responses according to region, traffic type, volume, and payload. No-bid traffic has its own pricing dimension, received traffic is not charged, and there is no Free Tier. Use the current pricing page or calculator. Do not copy rates from an example or assume that Fabric eliminates all NAT cost. + +CloudWatch exposes RTB Fabric request count, success and failure count, HTTP status, forwarding and total latency, target counts, filtered transactions, and internal or external no-bid metrics. Use Sum for volume metrics and P90, P95, or P99 for latency metrics where the metric supports those statistics. Define alarms and retention with the rest of the PBS observability plan. + +## Terraform and lifecycle + +The AWS Labs Terraform module creates requester and responder gateways, links, and inbound external links through the AWS Cloud Control provider. Pin a reviewed module release. Its documentation currently shows `v0.3.0` as the pinned example. + +Treat a standard partner link as a two-party lifecycle, not a normal single-account Terraform resource. The requester can create the request, but the partner must accept it. Keep acceptance, activation, and endpoint publication as separate operator-owned steps unless a separately approved automation owns both accounts. + +The module documents these limitations: + +- `http_responder_allowed` is immutable and may be absent from Cloud Control responses, which can cause Terraform state drift. The documented recovery is a one-time state cleanup and re-import. Changing the value requires link replacement. +- Link module configuration is currently supported from the requester side, not the responder side. +- Managed responder endpoints use EKS or EC2 Auto Scaling groups. The current Trusted Server descriptor supports `ec2-compose` with explicit instance IDs, so it does not establish RTB Fabric responder support. + +Keep secret values out of link configuration, Terraform variables, state, and reports. Link IDs, gateway IDs, and endpoint URLs are deployment metadata, but still require the selected access controls and change owner. + +## Sources + +- [AWS RTB Fabric FAQ](https://aws.amazon.com/rtb-fabric/faqs/) +- [RTB Fabric concepts](https://docs.aws.amazon.com/rtb-fabric/latest/userguide/what-is-rtb-fabric.html) +- [Create links](https://docs.aws.amazon.com/rtb-fabric/latest/userguide/creating-rtb-links.html) +- [Create outbound external links](https://docs.aws.amazon.com/rtb-fabric/latest/userguide/creating-outbound-external-links.html) +- [RTB Fabric quotas](https://docs.aws.amazon.com/rtb-fabric/latest/userguide/rtb-fabric-quotas.html) +- [RTB Fabric metrics](https://docs.aws.amazon.com/rtb-fabric/latest/userguide/monitoring-cloudwatch-metrics.html) +- [AWS Prebid Server partner onboarding](https://docs.aws.amazon.com/solutions/latest/prebid-server-deployment-on-aws/use-the-solution.html) +- [AWS Labs Terraform module](https://github.com/awslabs/rtb-fabric-terraform-module) diff --git a/.claude/skills/planning-prebid-aws/references/terraform.md b/.claude/skills/planning-prebid-aws/references/terraform.md new file mode 100644 index 000000000..55517f7a0 --- /dev/null +++ b/.claude/skills/planning-prebid-aws/references/terraform.md @@ -0,0 +1,74 @@ +# Terraform generation and review + +Apply this reference when choosing Terraform state/authentication boundaries, generating HCL, or checking generated modules. The planning-only authority in `SKILL.md` governs all upstream guidance and commands below. + +## Upstream guidance + +Consult the matching HashiCorp skill as reference material, using an approved local installation if available or the linked source. Installation is a separate user decision. Record the revision consulted in the deployment plan; these skills were reviewed at `c2d65dfe492f74d360d35b859b88932222470bd8`. + +| Branch | Reference | +| ---------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Writing or reviewing HCL | [terraform-style-guide](https://github.com/hashicorp/agent-skills/blob/c2d65dfe492f74d360d35b859b88932222470bd8/plugins/terraform/skills/terraform-style-guide/SKILL.md) | +| Writing `.tftest.hcl` tests | [terraform-test](https://github.com/hashicorp/agent-skills/blob/c2d65dfe492f74d360d35b859b88932222470bd8/plugins/terraform/skills/terraform-test/SKILL.md) | +| Refactoring existing resource addresses into modules | [refactor-module](https://github.com/hashicorp/agent-skills/blob/c2d65dfe492f74d360d35b859b88932222470bd8/plugins/terraform/skills/refactor-module/SKILL.md) | + +Verify examples against primary documentation and the selected Terraform/provider versions. Upstream examples do not authorize state access, integration tests, apply, or migration. Keep PBS-specific requirements here rather than adopting upstream example credentials, instance sizes, or runtime choices. + +## State and backend + +Identify the existing backend, state owner, Terraform version, and production/nonproduction boundaries before choosing storage. Reuse an approved backend. Keep bootstrap ownership separate from resources whose teardown it records. + +For a newly selected S3 backend, require bucket versioning, encryption, blocked public access, restricted state access, and native locking with `use_lockfile = true` on a compatible Terraform version. Include the required read/write/delete permissions for the `.tflock` object; lock deletion does not require permission to delete the state object. Verify these details against the [S3 backend documentation](https://developer.hashicorp.com/terraform/language/backend/s3). + +DynamoDB-based locking is deprecated. For an existing backend using it, propose a migration that accounts for every Terraform client, lock permissions, state recovery, and the cutover owner. Preserve current locking until the separately authorized migration establishes the replacement. File generation must not migrate state, disable locking, or replace the existing backend automatically. + +Separate production and nonproduction state and access. Split regional state only when ownership, failure isolation, or infrastructure change lifecycles justify it. One pilot root with explicit regional provider mappings remains a valid approved choice; backend selection does not select ECS or impose East/West regions. + +## Versions and module contracts + +- Select a reproducible Terraform CLI version in the repository's toolchain and compatible `required_version` constraints. Verify feature support before selecting a version; sample version numbers are not defaults. +- Declare compatible provider constraints and generate `.terraform.lock.hcl` in each root. Commit the locks and review upgrades deliberately. The lock file records provider selections and checksums, not the CLI or remote module versions. +- Pin external registry modules to explicit versions for reproducible roots; pin Git sources with immutable commit references. A module's `version` argument applies only to registry sources. Local modules share their caller's repository revision. See [dependency locking](https://developer.hashicorp.com/terraform/language/files/dependency-lock) and [module sources](https://developer.hashicorp.com/terraform/language/modules/sources). +- Keep provider configurations in roots; reusable modules declare requirements and receive provider mappings. Test every selected regional mapping, including aliased providers. +- Expose settings expected to vary. Give every input a type and description, and every output a description. Validate real restrictions, resource bounds, and conditional requirements. Required unknowns must fail validation or an explicit preflight instead of silently choosing a deployment target. +- Use stable keys for independently named resource collections. Avoid index-driven address churn when membership changes. Keep module boundaries tied to responsibility and shared lifecycle rather than wrapping each AWS resource. +- Apply common ownership/cost tags through each AWS provider configuration where supported, including aliases. Check resource-specific exceptions and override behavior; `default_tags` is not proof every resource is tagged. + +When refactoring existing infrastructure, obtain authorization for scoped state inspection and map old/new addresses before proposing `moved` blocks. Require a reviewed migration plan showing intended address moves without unintended resource destruction or replacement. Leave state mutation and migration execution to the authorized operator. + +## Credentials and resource ownership + +Prefer short-lived authentication through the approved AWS identity system: [Identity Center/SSO](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html) for operators and [OIDC federation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc.html) for CI where supported. Scope CI trust to the intended repository, branch or protected environment, and audience. Separate plan/apply permissions where practical; the plan role still needs only the state/lock and read permissions its work requires. Keep cloud credentials unavailable to untrusted pull-request code. Record any platform exception for approval instead of generating permanent access-key secrets by default. + +Terraform owns AWS resources, secret metadata, and access policy. An authorized external workflow owns credential values. Keep secret values out of user data, Terraform variables/data sources, plans, and state; `sensitive` masks display but does not exclude ordinary values from state. Output identifiers and endpoints only. Native secret references preserve this boundary for the selected runtime; ephemeral/write-only support is not a reason to route bidder credentials through Terraform. See [sensitive data](https://developer.hashicorp.com/terraform/language/manage-sensitive-data). + +Prefer provider-native resources, image builds, and explicit deployment tools over provisioner side effects. Keep application releases outside `local-exec`, `remote-exec`, and shell-command wrappers such as `null_resource`. For ECS, assign task-definition revisions and the service's selected revision to one writer. If a deployer selects revisions outside Terraform, document and test drift handling so a later apply cannot silently revert a release. + +## Tests and safe local checks + +Generate module tests for input restrictions, conditional resources, capacity bounds, provider mappings, tags/security settings, secret references, and meaningful outputs as applicable to the approved topology. Include negative cases using `expect_failures` for custom validation. Use explicit mock values when assertions depend on computed attributes; plan-time values may otherwise remain unknown. + +Before running tests: + +1. Enumerate the exact test files and their run blocks, setup modules, providers, aliases, and external data sources. Inspect downloaded modules and executable hooks. Terraform can mix real and mocked providers in one suite. +2. Prefer explicit `command = plan` with mocked external providers. Plan mode alone is not credential-free: real providers and data sources can contact AWS. Mock every external provider used by the selected tests, including aliases and setup dependencies; exclude command-executing data sources and other unapproved side effects. +3. Fully mocked apply-mode tests may check computed results after the same inspection. Mocks work with both plan and apply. An omitted command defaults to apply, so require explicit commands; a real-provider apply test is a cloud integration test outside this workflow. +4. Select individual reviewed files with `terraform test -filter=tests/.tftest.hcl`, repeating `-filter` for additional files. Filters take file paths, not run names or naming substrings. `-test-directory` alone is not isolation because Terraform also discovers tests in the root directory. +5. Run without cloud credentials or metadata-credential access and with network restrictions appropriate to the test environment. Confirm the expected tests actually ran; zero tests is not a pass. Record commands, counts, results, and remaining cloud-validation gaps. + +Use the selected CLI's help and [test command documentation](https://developer.hashicorp.com/terraform/cli/commands/test) to verify flags. Consult [test syntax](https://developer.hashicorp.com/terraform/language/tests) and [mocking](https://developer.hashicorp.com/terraform/language/tests/mocking) rather than copying upstream CLI examples blindly. + +Run `terraform fmt -check -recursive `. In each root, inspect dependency sources before `terraform init -backend=false`, then run `terraform validate`. Backend-disabled init still downloads providers/modules; follow dependency-download policy. Once dependencies are installed, keep the test run separate from downloads. Use existing configured lint/security scanners when present, or propose their addition rather than silently installing tools. + +Validation and mocks prove configuration behavior, not IAM permissions, quotas, real bidder connectivity, or deployment capacity. If tooling or safe isolation is unavailable, mark checks not run. Live integration tests need a separately authorized test account, cost limits, and cleanup verification; automatic teardown is not a guarantee that all resources were removed. + +## Saved-plan handoff + +Generate the following procedure in the runbook or inactive, approval-gated CI. Do not execute it as a local validation step: + +1. An authorized operator verifies the account/role, regions, backend/state target, source revision, toolchain, dependency locks, and input set before producing `terraform plan -out=`. +2. Store the plan as a protected, short-lived artifact with its checksum and provenance. Saved plans and JSON renderings can contain cleartext secrets. Keep them out of Git and public PR logs; restrict review access and retention. +3. Review additions, changes, deletions/replacements, IAM/network exposure, cost implications, and target identity. Tie approval to that artifact and source revision, not just a PR's earlier speculative plan. +4. The separately authorized apply stage verifies the artifact identity and uses `terraform apply ` rather than silently generating another plan. Re-plan and re-review after relevant code/input/state changes, an expired approval window, or known drift. A saved plan is not a lock on AWS and does not guarantee success against out-of-band changes. + +Authenticated plan/refresh, backend migration, state mutation, and real-resource apply/destroy remain execution tasks. See [saved-plan behavior and sensitivity](https://developer.hashicorp.com/terraform/cli/commands/plan). diff --git a/.github/workflows/pbs-example.yml b/.github/workflows/pbs-example.yml new file mode 100644 index 000000000..dced0c7b6 --- /dev/null +++ b/.github/workflows/pbs-example.yml @@ -0,0 +1,72 @@ +name: PBS example checks + +on: + push: + branches: [main] + paths: + - 'deploy/pbs-example/**' + - '.tool-versions' + - '.github/workflows/pbs-example.yml' + pull_request: + paths: + - 'deploy/pbs-example/**' + - '.tool-versions' + - '.github/workflows/pbs-example.yml' + workflow_dispatch: + +permissions: + contents: read + +jobs: + local-checks: + name: PBS Terraform and Compose checks + runs-on: ubuntu-latest + timeout-minutes: 20 + env: + TF_IN_AUTOMATION: 'true' + AWS_EC2_METADATA_DISABLED: 'true' + steps: + - uses: actions/checkout@v4 + + - name: Read pinned Terraform version + id: terraform + shell: bash + run: echo "version=$(awk '$1 == "terraform" { print $2 }' .tool-versions)" >> "$GITHUB_OUTPUT" + + - uses: hashicorp/setup-terraform@v3 + with: + terraform_version: ${{ steps.terraform.outputs.version }} + terraform_wrapper: false + + - name: Read pinned Node.js version + id: node + shell: bash + run: echo "version=$(awk '$1 == "nodejs" { print $2 }' .tool-versions)" >> "$GITHUB_OUTPUT" + + - uses: actions/setup-node@v4 + with: + node-version: ${{ steps.node.outputs.version }} + + - name: Check Docker Compose prerequisite + run: docker compose version + + # These suites use mocked providers and plan mode. No AWS credentials, + # Terraform apply, image pulls, or container startup are needed. + - name: Format and test Terraform root + run: | + terraform fmt -check -recursive deploy/pbs-example + terraform -chdir=deploy/pbs-example init -backend=false -input=false -lockfile=readonly + terraform -chdir=deploy/pbs-example validate + terraform -chdir=deploy/pbs-example test -filter=tests/root_unit_test.tftest.hcl + + - name: Validate and test regional module + run: | + terraform -chdir=deploy/pbs-example/modules/regional init -backend=false -input=false -lockfile=readonly + terraform -chdir=deploy/pbs-example/modules/regional validate + terraform -chdir=deploy/pbs-example/modules/regional test -filter=tests/security_unit_test.tftest.hcl + git diff --exit-code -- deploy/pbs-example/.terraform.lock.hcl deploy/pbs-example/modules/regional/.terraform.lock.hcl + + - name: Check runtime inputs and smoke wiring without starting containers + run: | + bash -n deploy/pbs-example/scripts/smoke-runtime.sh + env -u NODE_OPTIONS -u NODE_PATH node deploy/pbs-example/scripts/test-smoke-runtime.mjs diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index a7fc78d08..a122fc83a 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -88,6 +88,15 @@ jobs: target: wasm32-wasip1 cache-shared-key: cargo-${{ runner.os }} + - name: Read pinned Node.js version for CLI fixtures + id: node-version + shell: bash + run: echo "version=$(awk '$1 == "nodejs" { print $2 }' .tool-versions)" >> "$GITHUB_OUTPUT" + + - uses: actions/setup-node@v4 + with: + node-version: ${{ steps.node-version.outputs.version }} + - name: Build Axum adapter run: cargo build -p trusted-server-adapter-axum @@ -226,6 +235,15 @@ jobs: components: "clippy, rustfmt" cache-shared-key: cargo-cli-${{ runner.os }} + - name: Read pinned Node.js version for CLI fixtures + id: node-version + shell: bash + run: echo "version=$(awk '$1 == "nodejs" { print $2 }' .tool-versions)" >> "$GITHUB_OUTPUT" + + - uses: actions/setup-node@v4 + with: + node-version: ${{ steps.node-version.outputs.version }} + # No separate `cargo fmt` here: `trusted-server-cli` is a workspace member, # so the main `cargo fmt --all` job already formats it. Clippy still needs a # native run — the workspace clippy jobs are wasm/adapter-scoped and never diff --git a/.tool-versions b/.tool-versions index 758146800..1b29972c2 100644 --- a/.tool-versions +++ b/.tool-versions @@ -1,5 +1,7 @@ -fastly 15.1.0 -rust 1.95.0 -nodejs 24.12.0 -viceroy 0.17.0 -wasmtime 44.0.1 +fastly 15.1.0 +rust 1.95.0 +nodejs 24.12.0 +viceroy 0.17.0 +wasmtime 44.0.1 +awscli 2.36.45 +terraform 1.16.2 diff --git a/AGENTS.md b/AGENTS.md index 738fcab34..ee2ad3fd0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -39,6 +39,8 @@ Supporting files: `edgezero.toml`, `fastly.toml`, | Fastly CLI | 15.1.0 (from `.tool-versions`) | | Viceroy | 0.17.0 (from `.tool-versions`) | | Wasmtime | 44.0.1 (from `.tool-versions`) | +| AWS CLI | 2.36.45 (from `.tool-versions`) | +| Terraform | 1.16.2 (from `.tool-versions`) | --- diff --git a/CHANGELOG.md b/CHANGELOG.md index 9d410ea4f..101087b8d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- **Breaking:** `ts prebid bundle` is now `ts prebid client`, alongside the new `ts prebid server` namespace. Update scripts and runbooks to use `ts prebid client` with the same arguments. The old `bundle` spelling is no longer accepted and has no compatibility alias. - The S2S `/_ts/api/v1/batch-sync` endpoint now validates the full batch and calls the CAS-protected update path once per distinct normalized EC ID. The last valid UID wins within a group, and infrastructure failures reject the failing and each unprocessed group, so accepted and `kv_unavailable` input indexes may interleave. - **Breaking:** Auction providers and bidder routes now use the configuration-first `[auction.providers.]` and `[auction.bidders.]` maps. The removed `[auction].providers = [...]` list and removed server fields under `[integrations.prebid]` and `[integrations.aps]` are rejected even when those integrations are disabled, and `ts config push` rejects the old shape before publication. Move PBS `server_url` to provider `endpoint`, server timeout to provider `timeout_ms`, request controls and bidder-parameter overrides to the `prebid-server` `profile_config`, notification suppression to `notifications`, and each former server bidder to an `[auction.bidders.]` route. Move APS endpoint, timeout, account, inventory, debug, and creative controls to an `aps` provider and its `profile_config`. Browser Prebid settings remain under `[integrations.prebid]`; values such as timeout and debug that previously affected both browser and server behavior must now be configured for each owner. Provider endpoints must be absolute HTTPS URLs. Only bidder codes present in `[auction.bidders]` are folded into Trusted Server requests; unlisted publisher bids remain native browser demand. Provider response names now use the configured provider ID, such as `pbs-main`, instead of the legacy literal `prebid`; audit consumers that match `AuctionResponse.provider`. This schema has no mixed-version-safe deployment order: old binaries reject the maps and new binaries reject the retired fields, so activate the new binary and config blob together. Rollbacks must restore an old-schema blob together with the old binary. - **Breaking** — Admin Basic-auth coverage now includes `GET /_ts/admin/ec`, `GET /_ts/admin/ec/{id}`, and `GET /_ts/admin/eids`. Existing configurations whose `[[handlers]]` patterns protect only the key-management endpoints now fail startup; broaden coverage before deploying, preferably with a namespace-boundary pattern such as `^/_ts/admin(?:/|$)`. Coverage of the dynamic `/_ts/admin/ec/{id}` route is no longer inferred from ID-shaped samples: the router accepts any segment after `/_ts/admin/ec/` and Basic Auth runs on the raw path before routing, so patterns anchored to the EC ID grammar (for example `^/_ts/admin/ec/[a-f0-9]{64}[.][A-Za-z0-9]{6}$`) are rejected in favor of a prefix-level matcher. Placeholder and well-known weak handler passwords (`changeme`, `password`, `admin`, `replace-with-…`) now fail startup on every handler rather than only on handlers inferred to cover an admin endpoint, because first-match-wins handler selection lets a narrow handler shadow the admin namespace. diff --git a/Cargo.lock b/Cargo.lock index 7a45462b1..679c9dd4f 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -4104,6 +4104,17 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "rpassword" +version = "7.5.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2da316a15f47e3d053de9cb2c439650bd8fa4aaeb9365f2e5f27f492ff73c196" +dependencies = [ + "libc", + "rtoolbox", + "windows-sys 0.61.2", +] + [[package]] name = "rsa" version = "0.9.10" @@ -4124,6 +4135,16 @@ dependencies = [ "zeroize", ] +[[package]] +name = "rtoolbox" +version = "0.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a1efe12a1469752d0e6ff5ebec0b6ef4924cc5c4c71046b0ec730040535819d" +dependencies = [ + "libc", + "windows-sys 0.61.2", +] + [[package]] name = "rusqlite" version = "0.32.1" @@ -4568,6 +4589,19 @@ dependencies = [ "syn 2.0.118", ] +[[package]] +name = "serde_yaml_ng" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b4db627b98b36d4203a7b458cf3573730f2bb591b28871d916dfa9efabfd41f" +dependencies = [ + "indexmap 2.14.0", + "itoa", + "ryu", + "serde", + "unsafe-libyaml", +] + [[package]] name = "servo_arc" version = "0.4.3" @@ -5459,11 +5493,13 @@ dependencies = [ "rcgen", "regex", "reqwest 0.12.28", + "rpassword", "rustls", "rustls-pemfile", "scraper", "serde", "serde_json", + "serde_yaml_ng", "similar", "temp-env", "tempfile", @@ -5475,6 +5511,7 @@ dependencies = [ "tracing", "trusted-server-core", "url", + "uuid", "webpki-roots", "which", "x509-parser", @@ -5672,6 +5709,12 @@ dependencies = [ "subtle", ] +[[package]] +name = "unsafe-libyaml" +version = "0.2.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861" + [[package]] name = "untrusted" version = "0.9.0" diff --git a/Cargo.toml b/Cargo.toml index ac0cac621..51e3038ea 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -91,11 +91,13 @@ rand = "0.8" rcgen = { version = "0.13", features = ["x509-parser"] } regex = "1.12.3" reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] } +rpassword = "7.5" rustls = "0.23" rustls-pemfile = "2" scraper = "0.24.0" serde = { version = "1.0", features = ["derive"] } serde_json = "1.0.149" +serde_yaml_ng = "0.10" sha2 = "0.10.9" similar = "2.7" simple_logger = "5" diff --git a/crates/trusted-server-cli/Cargo.toml b/crates/trusted-server-cli/Cargo.toml index 19fb13366..7de87ae15 100644 --- a/crates/trusted-server-cli/Cargo.toml +++ b/crates/trusted-server-cli/Cargo.toml @@ -18,6 +18,7 @@ workspace = true chromiumoxide = { workspace = true } clap = { workspace = true } derive_more = { workspace = true } +error-stack = { workspace = true } edgezero-cli = { workspace = true } edgezero-core = { workspace = true } futures = { workspace = true } @@ -26,6 +27,7 @@ http = { workspace = true } log = { workspace = true } rand = { workspace = true } regex = { workspace = true } +rpassword = { workspace = true } # Also in the macOS block for the dev proxy; needed on every host so the probe can install # the process-level crypto provider its `-no-provider` reqwest build expects. rustls = { workspace = true } @@ -47,6 +49,7 @@ reqwest = { version = "0.12", default-features = false, features = [ scraper = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } +serde_yaml_ng = { workspace = true } similar = { workspace = true } tempfile = { workspace = true } tokio = { workspace = true } @@ -55,6 +58,7 @@ toml_edit = { workspace = true } tracing = { workspace = true } trusted-server-core = { workspace = true } url = { workspace = true } +uuid = { workspace = true } which = { workspace = true } # `ts dev proxy` is macOS-only — CA trust via the login keychain, Safari @@ -67,7 +71,6 @@ which = { workspace = true } base64 = { workspace = true } bytes = { workspace = true } directories = { workspace = true } -error-stack = { workspace = true } http-body-util = { workspace = true } hyper = { workspace = true, features = ["http1", "server", "client"] } hyper-util = { workspace = true, features = ["tokio"] } diff --git a/crates/trusted-server-cli/README.md b/crates/trusted-server-cli/README.md index 443919b71..e73c64628 100644 --- a/crates/trusted-server-cli/README.md +++ b/crates/trusted-server-cli/README.md @@ -21,3 +21,113 @@ serially. See the [CLI guide](../../docs/guide/cli.md) for the generated two-platform command inventory and the [EdgeZero guide](../../docs/guide/edgezero.md) for configuration lifecycle and store semantics. + +## Experimental PBS commands + +`ts prebid server` manages local configuration inputs and a small set of AWS operations for self-hosted Prebid Server Go. It sits beside `ts prebid client`, which builds browser JavaScript, and remains separate from the existing Trusted Server `ts config` and `ts deploy` commands. + +The namespace is experimental; its interface may change without a deprecation cycle. There is no separate binary or crate. + +### Build and try locally + +Use this branch's executable, not an older installed `ts`: + +```bash +cargo build_cli_linux +cargo run_cli_linux prebid server --help +cargo run_cli_linux prebid server inspect --config trusted-server.example.toml --json +cargo run_cli_linux prebid server check --deployment crates/trusted-server-cli/examples/pbs/deployment.yaml +``` + +On Apple Silicon macOS use `build_cli_macos` and `run_cli_macos`. Other hosts can run `cargo run --package trusted-server-cli --target "$(rustc -vV | awk '/host:/ { print $2 }')" -- prebid server --help`. The examples contain fictional resource identifiers, a fictional image digest, and a fictional adapter binding. They exercise local checks only and must not be used as real deployment settings. + +### Commands and current limits + +| Command | What it does | Access | +| ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------- | +| `ts prebid server inspect --config ` | Reports routed Prebid Server providers and bidders separately from browser Prebid fields; host-secret requirements remain unresolved | Local read-only | +| `ts prebid server check --deployment ` | Validates schema, targets, binding metadata, and regional YAML merging | Local read-only | +| `ts prebid server secrets set --deployment --region ` | Writes a complete JSON value to an existing, declared Secrets Manager secret after identity and confirmation checks | AWS reads and one value write | +| `ts prebid server status --deployment ` | Reports EC2 instance state and infrastructure health for the explicitly listed instances | AWS reads | + +Add `--json` anywhere under `ts prebid server` for a machine-readable report. Errors and operator notices go to stderr; failures exit with code 2. A partial status report still appears on stdout, with `complete: false` and exit code 2. + +Not implemented: container deployment, rollback, runtime secret injection, caller updates, Terraform execution, ECS status, or PBS application health checks. `status` always reports the installed release and consumed secret versions as unknown. EC2 health is not PBS readiness. + +`check` does not validate every PBS configuration field against the upstream Go schema, independently verify adapter documentation, retrieve secrets, pull images, start PBS, or prove a deployment ready. Bindings are operator-supplied metadata, not a bundled bidder catalog. Validate actual adapter mappings and startup against the pinned PBS release before use. + +### Discovering requirements + +`inspect` reads exactly the chosen file and never rewrites or publishes it. Omitting `--config` reads `trusted-server.toml` in the working directory. It discovers server demand from `[auction.providers.*]` entries using the `prebid-server` profile and bidders routed through `[auction.bidders.*]`. The JSON report groups routed bidders under each provider. Browser settings still come from `[integrations.prebid]`, including the runtime's array, indexed-map, and string encodings for `client_side_bidders`. Bundle selections come from `[integrations.prebid.bundle.modules]`: `bidder`, `user_id`, and `analytics` appear as `bundle_adapters`, `identity_modules`, and `analytics_modules` in JSON and as separate lists in human output. The command reports explicitly supplied values only; it does not expand defaults, environment overrides, remote configuration, or request-time inputs. Confirm which source and environment are authoritative before relying on the report. + +Account identifiers, endpoint values, and bid-parameter values are withheld. Parser errors also withhold source snippets. Server-side bidders, client-side bidders, and browser bundle adapters remain separate; listing a bidder does not establish partner authorization or a host-secret requirement. Disabled auctions and integrations remain disabled. + +### Deployment descriptor + +See [deployment.yaml](examples/pbs/deployment.yaml), [PBS YAML](examples/pbs/pbs.yaml), [regional overrides](examples/pbs/east.yaml), and [bindings.json](examples/pbs/bindings.json). + +Version 1 requires: + +- `schema_version: 1`, an explicit environment, and `runtime: ec2-compose`. +- An explicit 12-digit AWS account ID and AWS CLI profile. Environment and bidder identifiers use letters, digits, underscores, and hyphens. Profile names may also use periods. +- A digest-pinned PBS image and a baseline YAML path. +- A nonempty region map with optional override paths and explicit EC2 instance IDs for `status`. +- An optional binding-file path. Omit it when no host secrets are needed. + +Paths resolve relative to the descriptor, not the working directory. No automatic descriptor discovery or production default exists. Unknown descriptor fields, unsupported runtime types, duplicate YAML keys, tags, and implicit YAML merge keys are rejected. + +Mappings merge recursively with regional values taking precedence; sequences and scalars replace whole values. Rendering happens in memory and writes no generated files. The command does not read arbitrary operator `PBS_*` environment overrides. + +#### Binding schema + +The binding file is a JSON or YAML mapping keyed by bidder identifier. Each entry declares: + +- `verified_image`: the exact PBS image string matching the descriptor. +- `source`: an HTTPS reference used by the operator to verify the mapping. +- `secrets`: one complete Secrets Manager ARN for every descriptor region, matching its account and region. +- `keys`: credential key names mapped to `env`, `pbs_path`, and optional `required`, which defaults to true. + +Version 1 supports string-valued credentials and one binding set across all selected regions. Each secret ARN must belong to exactly one bidder binding because updates replace the complete JSON object. A PBS destination cannot overlap another binding or a value already present in the resolved YAML, including a non-mapping parent. Names alone do not prove that PBS supports the mapping; the metadata records an operator decision. + +### Setting a secret + +Prerequisites: AWS CLI v2 on PATH, a trusted local AWS profile using short-lived credentials, and a previously provisioned secret with approved permissions. AWS CLI command history must be disabled; the tool checks both the selected profile and default history settings before submitting a value. It suppresses AWS stderr and disables configured endpoint URL overrides for API calls. + +The command verifies the account through STS, describes the exact declared secret, and refuses replica writes or secrets scheduled for deletion. Interactive use reads a complete JSON object through a hidden terminal prompt, shows the target and retry UUID, and requires typing `yes` before writing. The prompt belongs in the operator's terminal, not an agent conversation. + +For approved automation, supply a file or stdin, `--yes`, and a stable UUID identifying the logical write: + +```bash +ts prebid server secrets set examplebidder \ + --deployment /secure/path/deployment.yaml \ + --region us-east-1 \ + --file /secure/path/credential.json \ + --request-token 11111111-2222-4333-8444-555555555555 \ + --yes --json +``` + +The command above is an interface example, not authorization to run it. Generate a new UUID for each logical update and reuse that UUID with identical values for retries. Reusing it with different values fails in Secrets Manager. `--yes` confirms only this value write; it does not authorize deployment, partner rotation, or future writes. Use protected CI approval gates before invoking it. + +File and stdin inputs are mutually exclusive. `--stdin` requires `--yes` and `--request-token`. Input must contain only declared keys, include all required nonempty string values, and fit the Secrets Manager size limit. Optional keys may be omitted. Duplicate keys and non-string values are rejected. Quotes, dollar signs, newlines, and backslashes are encoded as JSON, never shell expressions. + +Secret values never enter command arguments or reports. The AWS CLI receives JSON through a tool-owned temporary file, owner-only on Unix, which is removed on normal success and error paths. Operator-provided input files are not changed or deleted. Input-file errors print the full escaped path to stderr, including under `--json`; directory layouts and partner names in paths can enter logs even though credential contents are withheld. Run on a trusted host with protected temporary storage; abrupt process termination can leave temporary files requiring cleanup. Windows temporary-file ACL behavior has not been validated. + +A successful write reports its version identifier. It does not create secret metadata, change infrastructure, replace containers, or rotate the bidder's credential. Check regional replication, separately replace consumers, and verify them before revoking old partner credentials. A failed or unverifiable response means the write is not confirmed; the outcome may be uncertain. Retain the displayed request token and reuse it only for the original identical payload. Use a new token only for separately intended changed values. Provider error details are withheld, so the CLI cannot distinguish a rejected write from a lost response. + +### Ownership and sandbox compatibility + +These commands do not adopt the existing sandbox's Terraform state, edit its files, or change its IAM roles. A descriptor must reference resources the operator has explicitly approved. The sandbox currently bootstraps runtime files through Terraform user data and has no runtime secret loader. Writing a secret therefore does not make that sandbox consume it. + +Deployment and rollback require a separately approved move to versioned runtime releases. Until that exists, `ts prebid server` has no deployment or rollback subcommands and the skill must not promise them. + +### Verification + +```bash +./scripts/test-cli.sh +cargo fmt --all -- --check +cargo clippy-cli +``` + +Unit tests cover local discovery, rendering, binding conflicts, account checks, confirmation, payload validation, retries, and status limitations. Unix process-level tests run the actual `ts` binary with a fake `aws` executable and require the Node.js version pinned in `.tool-versions`. The fake uses only built-in modules and needs no npm install. They verify no AWS execution for local commands, private temporary requests, absence of credentials in arguments/output, cleanup, history/account refusal, and partial-report exit codes. They never contact AWS. The unset-history case covers exit 1 with empty stdout, but this fake response does not establish the real AWS CLI contract. An authorized runtime owner must verify real secret-write version IDs and retry behavior against a throwaway secret before operational use. + +Process tests also send deeply nested flow sequences, flow mappings, and block mappings through `check`. The pinned `serde_yaml_ng` parser rejects them with sanitized errors before the recursive configuration walkers run. These tests guard the parser's depth-limit behavior; they are not proof against every possible YAML resource-exhaustion input. diff --git a/crates/trusted-server-cli/examples/pbs/bindings.json b/crates/trusted-server-cli/examples/pbs/bindings.json new file mode 100644 index 000000000..24bd8a03a --- /dev/null +++ b/crates/trusted-server-cli/examples/pbs/bindings.json @@ -0,0 +1,21 @@ +{ + "examplebidder": { + "verified_image": "registry.example.com/pbs@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "source": "https://example.com/adapter-reference", + "secrets": { + "us-east-1": "arn:aws:secretsmanager:us-east-1:123456789012:secret:pbs/example-AbCdEf" + }, + "keys": { + "api_key": { + "env": "PBS_ADAPTERS_EXAMPLEBIDDER_API_KEY", + "pbs_path": ["adapters", "examplebidder", "api_key"], + "required": true + }, + "optional_token": { + "env": "PBS_ADAPTERS_EXAMPLEBIDDER_OPTIONAL_TOKEN", + "pbs_path": ["adapters", "examplebidder", "optional_token"], + "required": false + } + } + } +} diff --git a/crates/trusted-server-cli/examples/pbs/deployment.yaml b/crates/trusted-server-cli/examples/pbs/deployment.yaml new file mode 100644 index 000000000..cf0031c69 --- /dev/null +++ b/crates/trusted-server-cli/examples/pbs/deployment.yaml @@ -0,0 +1,16 @@ +# Fictional targets and image. Local check fixture, not deployable infrastructure. +schema_version: 1 +environment: sandbox +runtime: ec2-compose +aws: + account_id: '123456789012' + profile: pbs-sandbox +pbs: + config: pbs.yaml + image: registry.example.com/pbs@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + bindings: bindings.json +regions: + us-east-1: + instance_ids: + - i-0123456789abcdef0 + overrides: east.yaml diff --git a/crates/trusted-server-cli/examples/pbs/east.yaml b/crates/trusted-server-cli/examples/pbs/east.yaml new file mode 100644 index 000000000..9c7f78ddd --- /dev/null +++ b/crates/trusted-server-cli/examples/pbs/east.yaml @@ -0,0 +1,3 @@ +external_url: https://pbs-east.example.com +auction_timeouts_ms: + default: 900 diff --git a/crates/trusted-server-cli/examples/pbs/pbs.yaml b/crates/trusted-server-cli/examples/pbs/pbs.yaml new file mode 100644 index 000000000..346cf1e9b --- /dev/null +++ b/crates/trusted-server-cli/examples/pbs/pbs.yaml @@ -0,0 +1,6 @@ +# Illustrates local merging only; validate actual PBS fields at the selected release. +port: 8000 +external_url: https://pbs.example.com +auction_timeouts_ms: + default: 1000 + max: 1200 diff --git a/crates/trusted-server-cli/src/commands/mod.rs b/crates/trusted-server-cli/src/commands/mod.rs index 9e456ce87..0f4354f00 100644 --- a/crates/trusted-server-cli/src/commands/mod.rs +++ b/crates/trusted-server-cli/src/commands/mod.rs @@ -1,5 +1,6 @@ pub(crate) mod audit; pub(crate) mod config; +pub(crate) mod pbs; // `dev` is `pub` so the macOS-gated `tests/proxy_e2e.rs` suite can reach // `commands::dev::proxy`, `origin` is `pub` so `tests/origin_probe.rs` can drive the // shareability probe against a local fixture, and `cache` is `pub` so diff --git a/crates/trusted-server-cli/src/commands/pbs/aws.rs b/crates/trusted-server-cli/src/commands/pbs/aws.rs new file mode 100644 index 000000000..70a29e9b8 --- /dev/null +++ b/crates/trusted-server-cli/src/commands/pbs/aws.rs @@ -0,0 +1,193 @@ +use std::io::Write; +use std::process::{Command, Stdio}; + +use error_stack::Report; +use serde_json::{Value, json}; +use tempfile::NamedTempFile; + +use super::config::{AwsTarget, Deployment, invalid}; +use super::{PbsError, Result}; + +/// An AWS call carries data separately from arguments; payloads never enter shell history or argv. +pub(super) trait Aws { + /// Invoke one AWS API with explicit identity and region. + /// + /// # Errors + /// Returns sanitized process/API failures; implementations must not expose raw stderr/payloads. + fn call( + &self, + target: &AwsTarget, + region: &str, + service: &'static str, + operation: &'static str, + input: &Value, + ) -> Result; +} + +pub(super) struct AwsCli; + +/// Provider output is withheld, so a rejected write cannot be distinguished from a lost response. +pub(super) const WRITE_NOT_CONFIRMED: &str = "write not confirmed; outcome may be uncertain; retain the request token and reuse it only for the original identical payload; use a new token for separately intended changed values"; + +impl Aws for AwsCli { + fn call( + &self, + target: &AwsTarget, + region: &str, + service: &'static str, + operation: &'static str, + input: &Value, + ) -> Result { + if operation == "put-secret-value" { + ensure_history_disabled(target)?; + } + // tempfile creates an owner-only file on Unix. No command uses a shell or logs this payload. + let mut payload = NamedTempFile::new() + .map_err(|_| Report::new(PbsError::Io("cannot create private AWS request file")))?; + serde_json::to_writer(payload.as_file_mut(), input) + .map_err(|_| Report::new(PbsError::Io("cannot encode AWS request")))?; + payload + .flush() + .map_err(|_| Report::new(PbsError::Io("cannot flush AWS request")))?; + let output = Command::new("aws") + .args([ + "--profile", + &target.profile, + "--region", + region, + "--output", + "json", + "--no-cli-pager", + "--no-cli-auto-prompt", + "--cli-connect-timeout", + "10", + "--cli-read-timeout", + "20", + service, + operation, + "--cli-input-json", + ]) + .arg(format!("file://{}", payload.path().display())) + .env("AWS_CLI_AUTO_PROMPT", "off") + .env("AWS_PAGER", "") + .env("AWS_MAX_ATTEMPTS", "2") + .env("AWS_IGNORE_CONFIGURED_ENDPOINT_URLS", "true") + .stdin(Stdio::null()) + .output() + .map_err(|_| Report::new(PbsError::Aws("could not execute AWS CLI v2")))?; + // Closing removes the temporary payload, including on every earlier error path via Drop. + drop(payload); + if !output.status.success() { + let message = if operation == "put-secret-value" { + WRITE_NOT_CONFIRMED + } else { + operation + }; + return Err(Report::new(PbsError::Aws(message))); + } + serde_json::from_slice(&output.stdout).map_err(|_| { + let message = if operation == "put-secret-value" { + WRITE_NOT_CONFIRMED + } else { + "invalid JSON response" + }; + Report::new(PbsError::Aws(message)) + }) + } +} + +/// Refuse credential writes when AWS CLI history could retain the request payload. +/// +/// # Errors +/// Rejects enabled/unknown history settings and failures to inspect configuration. +fn ensure_history_disabled(target: &AwsTarget) -> Result<()> { + for key in ["cli_history", "default.cli_history"] { + let output = Command::new("aws") + .args([ + "--profile", + &target.profile, + "--no-cli-pager", + "--no-cli-auto-prompt", + "configure", + "get", + key, + ]) + .env("AWS_PAGER", "") + .env("AWS_CLI_AUTO_PROMPT", "off") + .stdin(Stdio::null()) + .output() + .map_err(|_| Report::new(PbsError::Aws("cannot verify CLI history is disabled")))?; + let value = String::from_utf8_lossy(&output.stdout); + if !matches!(output.status.code(), Some(0 | 1)) || !matches!(value.trim(), "" | "disabled") + { + return Err(Report::new(PbsError::Aws( + "disable AWS CLI history before writing credentials", + ))); + } + } + Ok(()) +} + +/// Verify account identity before any deployment resource lookup or mutation. +/// +/// # Errors +/// Rejects undeclared regions, failed identity reads, and mismatched accounts. +pub(super) fn verify_identity(deployment: &Deployment, region: &str, aws: &dyn Aws) -> Result<()> { + if !deployment.regions.contains_key(region) { + return Err(invalid( + "select a region declared in the deployment descriptor", + )); + } + let identity = aws.call( + &deployment.aws, + region, + "sts", + "get-caller-identity", + &json!({}), + )?; + if identity.get("Account").and_then(Value::as_str) != Some(deployment.aws.account_id.as_str()) { + return Err(Report::new(PbsError::AccountMismatch)); + } + Ok(()) +} + +#[cfg(test)] +pub(super) mod tests { + use std::cell::RefCell; + use std::collections::VecDeque; + + use super::*; + + pub(crate) struct FakeAws { + pub replies: RefCell>, + pub calls: RefCell>, + } + + impl FakeAws { + pub(crate) fn new(replies: Vec) -> Self { + Self { + replies: RefCell::new(replies.into()), + calls: RefCell::new(Vec::new()), + } + } + } + + impl Aws for FakeAws { + fn call( + &self, + _target: &AwsTarget, + region: &str, + _service: &'static str, + operation: &'static str, + input: &Value, + ) -> Result { + self.calls + .borrow_mut() + .push((region.to_owned(), operation.to_owned(), input.clone())); + self.replies + .borrow_mut() + .pop_front() + .ok_or_else(|| Report::new(PbsError::Aws("simulated failure"))) + } + } +} diff --git a/crates/trusted-server-cli/src/commands/pbs/config.rs b/crates/trusted-server-cli/src/commands/pbs/config.rs new file mode 100644 index 000000000..805722c06 --- /dev/null +++ b/crates/trusted-server-cli/src/commands/pbs/config.rs @@ -0,0 +1,568 @@ +use std::collections::{BTreeMap, BTreeSet}; +use std::path::{Path, PathBuf}; + +use error_stack::Report; +use serde::Deserialize; +use serde_json::json; +use serde_yaml_ng::Value; +use url::Url; + +use super::{Output, PbsError, Result, identifier, read_text}; + +const MAX_CONFIG_BYTES: usize = 2 * 1024 * 1024; + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct Descriptor { + schema_version: u32, + environment: String, + runtime: Runtime, + aws: AwsTarget, + pbs: PbsInputs, + regions: BTreeMap, +} + +#[derive(Deserialize)] +#[serde(rename_all = "kebab-case")] +enum Runtime { + Ec2Compose, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct AwsTarget { + pub account_id: String, + pub profile: String, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct PbsInputs { + config: PathBuf, + image: String, + bindings: Option, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Region { + #[serde(default)] + pub instance_ids: Vec, + overrides: Option, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Binding { + pub verified_image: String, + pub source: String, + pub secrets: BTreeMap, + pub keys: BTreeMap, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct KeyBinding { + pub env: String, + pub pbs_path: Vec, + #[serde(default = "required")] + pub required: bool, +} + +fn required() -> bool { + true +} + +pub(super) struct Deployment { + pub environment: String, + pub aws: AwsTarget, + pub regions: BTreeMap, + pub bindings: BTreeMap, + pub rendered: BTreeMap, + image: String, +} + +impl Deployment { + /// Load and validate every local deployment input without calling AWS or writing files. + /// + /// # Errors + /// Rejects malformed schemas, unsupported runtimes, invalid targets, and binding conflicts. + pub fn load(path: &Path) -> Result { + let base = path.parent().unwrap_or_else(|| Path::new(".")); + let descriptor: Descriptor = decode_yaml(path)?; + let Descriptor { + schema_version, + environment, + runtime: Runtime::Ec2Compose, + aws, + pbs, + regions, + } = descriptor; + if !valid_profile(&aws.profile) { + return Err(invalid( + "AWS profile must use 1-128 letters, digits, periods, underscores, or hyphens", + )); + } + if schema_version != 1 + || !identifier(&environment) + || !valid_account(&aws.account_id) + || regions.is_empty() + || !pinned_image(&pbs.image) + { + return Err(invalid( + "descriptor requires version 1, explicit account/profile, regions, and digest-pinned PBS image", + )); + } + let baseline = read_yaml(&resolve(base, &pbs.config)?)?; + if !baseline.is_mapping() { + return Err(invalid("PBS configuration must be a YAML mapping")); + } + validate_yaml(&baseline)?; + let bindings: BTreeMap = match &pbs.bindings { + Some(path) => decode_yaml(&resolve(base, path)?)?, + None => BTreeMap::new(), + }; + let mut rendered = BTreeMap::new(); + for (name, region) in ®ions { + if !valid_region(name) + || region.instance_ids.len() > 100 + || region.instance_ids.iter().any(|id| !valid_instance(id)) + || region.instance_ids.iter().collect::>().len() + != region.instance_ids.len() + { + return Err(invalid("invalid region or EC2 instance identifiers")); + } + let mut config = baseline.clone(); + if let Some(path) = ®ion.overrides { + let overrides = read_yaml(&resolve(base, path)?)?; + if !overrides.is_mapping() { + return Err(invalid("regional overrides must be a YAML mapping")); + } + validate_yaml(&overrides)?; + merge(&mut config, overrides); + } + rendered.insert(name.clone(), config); + } + let deployment = Self { + environment, + aws, + regions, + bindings, + rendered, + image: pbs.image, + }; + deployment.validate_bindings()?; + Ok(deployment) + } + + /// Validate declared binding evidence and reject conflicting destinations in every region. + /// + /// # Errors + /// Rejects incomplete metadata, wrong-account ARNs, duplicate destinations and YAML conflicts. + fn validate_bindings(&self) -> Result<()> { + let mut envs = BTreeSet::new(); + let mut paths = Vec::>::new(); + let mut secret_arns = BTreeSet::new(); + for (bidder, binding) in &self.bindings { + let source = Url::parse(&binding.source) + .map_err(|_| invalid("binding source must be an HTTPS documentation URL"))?; + if !identifier(bidder) + || binding.verified_image != self.image + || binding.keys.is_empty() + || binding.secrets.is_empty() + || source.scheme() != "https" + || source.host_str().is_none() + || !source.username().is_empty() + || source.password().is_some() + { + return Err(invalid( + "binding requires matching image, documentation source, keys, and regional secret ARNs", + )); + } + for (region, arn) in &binding.secrets { + if !secret_arns.insert(arn) { + return Err(invalid( + "each secret ARN must belong to only one bidder binding", + )); + } + if !self.regions.contains_key(region) + || !secret_arn(arn, region, &self.aws.account_id) + { + return Err(invalid( + "secret ARN must match a declared region and AWS account", + )); + } + } + // Version 1 uses one binding set across regions; require explicit coverage. + if binding.secrets.len() != self.regions.len() { + return Err(invalid( + "every binding must declare a secret ARN for every deployment region", + )); + } + for (key, target) in &binding.keys { + if !identifier(key) + || !target.env.starts_with("PBS_") + || target.env.len() > 256 + || !target.env.bytes().all(|byte| { + byte.is_ascii_uppercase() || byte.is_ascii_digit() || byte == b'_' + }) + || target.pbs_path.is_empty() + || target.pbs_path.iter().any(|part| !identifier(part)) + { + return Err(invalid("invalid secret key or PBS binding destination")); + } + if !envs.insert(&target.env) + || paths.iter().any(|path| { + path.starts_with(&target.pbs_path) || target.pbs_path.starts_with(path) + }) + { + return Err(invalid( + "secret bindings must have distinct non-overlapping PBS destinations", + )); + } + paths.push(target.pbs_path.clone()); + for config in self.rendered.values() { + if occupied(config, &target.pbs_path) { + return Err(invalid( + "secret destination conflicts with a YAML value or non-mapping parent", + )); + } + } + } + } + Ok(()) + } +} + +pub(super) fn valid_account(value: &str) -> bool { + value.len() == 12 && value.bytes().all(|byte| byte.is_ascii_digit()) +} + +fn valid_profile(value: &str) -> bool { + !value.is_empty() + && value.len() <= 128 + && value + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'.' | b'_' | b'-')) +} + +fn valid_region(value: &str) -> bool { + let parts: Vec<_> = value.split('-').collect(); + parts.len() >= 3 + && value.len() < 40 + && value + .bytes() + .all(|byte| byte.is_ascii_lowercase() || byte.is_ascii_digit() || byte == b'-') + && parts + .last() + .is_some_and(|part| !part.is_empty() && part.bytes().all(|byte| byte.is_ascii_digit())) +} + +fn valid_instance(value: &str) -> bool { + value.strip_prefix("i-").is_some_and(|id| { + matches!(id.len(), 8 | 17) && id.bytes().all(|byte| byte.is_ascii_hexdigit()) + }) +} + +fn pinned_image(value: &str) -> bool { + value + .rsplit_once("@sha256:") + .is_some_and(|(image, digest)| { + !image.is_empty() + && !image.contains('@') + && !image.chars().any(char::is_whitespace) + && digest.len() == 64 + && digest + .bytes() + .all(|byte| byte.is_ascii_digit() || matches!(byte, b'a'..=b'f')) + }) +} + +fn secret_arn(value: &str, region: &str, account: &str) -> bool { + let parts: Vec<_> = value.splitn(7, ':').collect(); + parts.len() == 7 + && parts[0] == "arn" + && matches!(parts[1], "aws" | "aws-us-gov" | "aws-cn") + && parts[2] == "secretsmanager" + && parts[3] == region + && parts[4] == account + && parts[5] == "secret" + && !parts[6].is_empty() + && parts[6] + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || b"/_+=.@-".contains(&byte)) +} + +/// Resolve descriptor-relative paths, retaining support for explicit operator-owned absolute files. +/// +/// # Errors +/// Rejects empty paths. +fn resolve(base: &Path, path: &Path) -> Result { + if path.as_os_str().is_empty() { + return Err(invalid("configuration path cannot be empty")); + } + Ok(base.join(path)) +} + +/// Parse YAML into a value first to detect duplicate mapping keys before typed conversion. +/// +/// # Errors +/// Rejects invalid YAML without exposing parser snippets. +fn read_yaml(path: &Path) -> Result { + serde_yaml_ng::from_str(&read_text(path, MAX_CONFIG_BYTES)?) + .map_err(|_| invalid("cannot parse YAML/JSON input; source details withheld")) +} + +/// Decode a schema with unknown fields rejected by each descriptor type. +/// +/// # Errors +/// Rejects unsupported schema fields and types without echoing values. +fn decode_yaml(path: &Path) -> Result { + let value = read_yaml(path)?; + validate_yaml(&value)?; + serde_yaml_ng::from_value(value).map_err(|_| { + invalid("unsupported deployment/binding schema; see ts prebid server documentation") + }) +} + +/// Require plain string-keyed data: YAML tags and merge keys obscure the effective configuration. +/// +/// # Errors +/// Rejects tagged values, merge keys, and non-string mapping keys. +fn validate_yaml(value: &Value) -> Result<()> { + match value { + Value::Mapping(map) => { + for (key, value) in map { + if key.as_str().is_none_or(|key| key == "<<") { + return Err(invalid( + "YAML mappings require string keys and explicit overrides, not merge keys", + )); + } + validate_yaml(value)?; + } + } + Value::Sequence(items) => { + for item in items { + validate_yaml(item)?; + } + } + Value::Tagged(_) => return Err(invalid("YAML tags are not supported")), + _ => {} + } + Ok(()) +} + +/// Recursively merge maps; replace arrays and scalar values as whole values. +fn merge(base: &mut Value, overrides: Value) { + match (base, overrides) { + (Value::Mapping(base), Value::Mapping(overrides)) => { + for (key, value) in overrides { + if let Some(previous) = base.get_mut(&key) { + merge(previous, value); + } else { + base.insert(key, value); + } + } + } + (base, overrides) => *base = overrides, + } +} + +fn occupied(value: &Value, path: &[String]) -> bool { + if path.is_empty() { + return true; + } + match value.as_mapping() { + Some(map) => map + .get(Value::String(path[0].clone())) + .is_some_and(|value| occupied(value, &path[1..])), + None => true, + } +} + +pub(super) fn invalid(message: &'static str) -> Report { + Report::new(PbsError::Input(message)) +} + +/// Report structural validation only, withholding resolved YAML values. +/// +/// # Errors +/// Returns an error if resolved YAML cannot be serialized. +pub(super) fn check(deployment: &Deployment) -> Result { + for value in deployment.rendered.values() { + serde_yaml_ng::to_string(value).map_err(|_| invalid("cannot serialize resolved YAML"))?; + } + Ok(Output { + failure: None, + summary: format!("PBS local checks passed for {}", deployment.environment), + details: vec![ + format!("Regions: {}", deployment.regions.keys().cloned().collect::>().join(", ")), + format!("Declared bidder secret bindings: {}", deployment.bindings.len()), + "Structural checks only: PBS startup, adapter mappings, credential availability and AWS behavior remain unverified.".to_owned(), + "No configuration files were written or AWS calls made.".to_owned(), + ], + data: json!({"environment": deployment.environment, "regions": deployment.regions.keys().collect::>(), + "binding_count": deployment.bindings.len(), "local_checks": "passed", "pbs_runtime_validation": "not_run", + "aws_validation": "not_run", "binding_evidence": "operator_declared_not_independently_verified"}), + }) +} + +#[cfg(test)] +pub(super) mod tests { + use std::fs; + + use super::*; + + pub(crate) fn fixture() -> (tempfile::TempDir, PathBuf) { + let dir = tempfile::tempdir().expect("should create temp directory"); + fs::write( + dir.path().join("pbs.yaml"), + "port: 8000\nauction_timeouts_ms:\n default: 1000\n max: 1200\nlist: [a, b]\n", + ) + .expect("should write PBS config"); + fs::write( + dir.path().join("east.yaml"), + "auction_timeouts_ms:\n default: 900\nlist: [c]\n", + ) + .expect("should write overrides"); + let image = format!("registry.example.com/pbs@sha256:{}", "a".repeat(64)); + let path = dir.path().join("deployment.yaml"); + fs::write(&path, format!("schema_version: 1\nenvironment: sandbox\nruntime: ec2-compose\naws:\n account_id: '123456789012'\n profile: pbs-sandbox\npbs:\n config: pbs.yaml\n image: {image}\n bindings: bindings.json\nregions:\n us-east-1:\n instance_ids: [i-0123456789abcdef0]\n overrides: east.yaml\n")).expect("should write descriptor"); + let bindings = json!({"examplebidder": { + "verified_image": image, "source": "https://example.com/adapter-reference", + "secrets": {"us-east-1": "arn:aws:secretsmanager:us-east-1:123456789012:secret:pbs/example-AbCdEf"}, + "keys": {"api_key": {"env": "PBS_ADAPTERS_EXAMPLEBIDDER_API_KEY", "pbs_path": ["adapters", "examplebidder", "api_key"]}} + }}); + fs::write(dir.path().join("bindings.json"), bindings.to_string()) + .expect("should write bindings"); + (dir, path) + } + + #[test] + fn validates_the_committed_pbs_example_descriptor() { + let path = Path::new(env!("CARGO_MANIFEST_DIR")) + .join("../../deploy/pbs-example/deployment.example.yaml"); + let deployment = Deployment::load(&path) + .expect("should load the committed PBS example descriptor and its inputs"); + let output = check(&deployment).expect("should check the committed PBS example"); + assert_eq!(output.data["local_checks"], "passed"); + } + + #[test] + fn pinned_image_requires_lowercase_sha256_hex() { + let image = |digest: &str| format!("registry.example.com/pbs@sha256:{digest}"); + assert!(pinned_image(&image(&"0123456789abcdef".repeat(4)))); + for uppercase in 'A'..='F' { + assert!(!pinned_image(&image(&format!( + "{}{uppercase}", + "a".repeat(63) + )))); + } + for digest in ["a".repeat(63), "a".repeat(65), "g".repeat(64)] { + assert!(!pinned_image(&image(&digest))); + } + } + + #[test] + fn resolves_relative_paths_merges_maps_and_replaces_arrays() { + let (_dir, path) = fixture(); + let deployment = Deployment::load(&path).expect("should load descriptor"); + let east = &deployment.rendered["us-east-1"]; + assert_eq!(east["auction_timeouts_ms"]["default"].as_u64(), Some(900)); + assert_eq!(east["auction_timeouts_ms"]["max"].as_u64(), Some(1200)); + assert_eq!( + east["list"].as_sequence().expect("should have list").len(), + 1 + ); + let again = Deployment::load(&path).expect("should load again"); + assert_eq!(deployment.rendered, again.rendered); + assert_eq!( + check(&deployment).expect("should validate").data["aws_validation"], + "not_run" + ); + } + + #[test] + fn accepts_dotted_aws_profiles_and_reports_invalid_profiles() { + let (_dir, path) = fixture(); + let descriptor = fs::read_to_string(&path).expect("should read descriptor"); + fs::write( + &path, + descriptor.replace("profile: pbs-sandbox", "profile: pbs.sandbox"), + ) + .expect("should write dotted profile"); + Deployment::load(&path).expect("should accept dotted AWS profile"); + + fs::write( + &path, + descriptor.replace("profile: pbs-sandbox", "profile: pbs sandbox"), + ) + .expect("should write invalid profile"); + let error = Deployment::load(&path) + .err() + .expect("should reject invalid AWS profile"); + assert!( + error.to_string().contains("AWS profile"), + "should identify the invalid field: {error}" + ); + } + + #[test] + fn yaml_and_secret_destinations_cannot_conflict() { + let (dir, path) = fixture(); + fs::write( + dir.path().join("east.yaml"), + "adapters:\n examplebidder:\n api_key: NEVER_PRINT_ME\n", + ) + .expect("should write conflicting override"); + let error = Deployment::load(&path) + .err() + .expect("should reject conflicting source"); + assert!(error.to_string().contains("conflicts")); + assert!(!format!("{error:?}").contains("NEVER_PRINT_ME")); + } + + #[test] + fn duplicate_yaml_keys_and_unsupported_runtime_fail_closed() { + let (dir, path) = fixture(); + fs::write(dir.path().join("pbs.yaml"), "port: 8000\nport: 9000\n") + .expect("should write duplicates"); + assert!(Deployment::load(&path).is_err()); + fs::write(dir.path().join("pbs.yaml"), "port: 8000\n").expect("should restore config"); + let text = fs::read_to_string(&path).expect("should read descriptor"); + fs::write(&path, text.replace("ec2-compose", "ecs")).expect("should change runtime"); + assert!(Deployment::load(&path).is_err()); + } + + #[test] + fn bidders_cannot_overwrite_each_others_secret_object() { + let (dir, path) = fixture(); + let file = dir.path().join("bindings.json"); + let mut bindings: serde_json::Value = + serde_json::from_str(&fs::read_to_string(&file).expect("should read bindings")) + .expect("should parse bindings"); + let mut other = bindings["examplebidder"].clone(); + other["keys"]["api_key"]["env"] = json!("PBS_ADAPTERS_OTHERBIDDER_API_KEY"); + other["keys"]["api_key"]["pbs_path"] = json!(["adapters", "otherbidder", "api_key"]); + bindings["otherbidder"] = other; + fs::write(file, bindings.to_string()).expect("should write bindings"); + assert!( + Deployment::load(&path).is_err(), + "complete-value writes require distinct secret ownership per bidder" + ); + } + + #[test] + fn unknown_schema_fields_and_wrong_account_secret_arns_are_rejected() { + let (dir, path) = fixture(); + let original = fs::read_to_string(&path).expect("should read descriptor"); + fs::write(&path, format!("{original}typo: value\n")).expect("should append unknown field"); + assert!(Deployment::load(&path).is_err()); + fs::write(&path, original).expect("should restore descriptor"); + let bindings = dir.path().join("bindings.json"); + let original = fs::read_to_string(&bindings).expect("should read bindings"); + fs::write(bindings, original.replace("123456789012", "999999999999")) + .expect("should change ARN"); + assert!(Deployment::load(&path).is_err()); + } +} diff --git a/crates/trusted-server-cli/src/commands/pbs/inspect.rs b/crates/trusted-server-cli/src/commands/pbs/inspect.rs new file mode 100644 index 000000000..0bfd39df1 --- /dev/null +++ b/crates/trusted-server-cli/src/commands/pbs/inspect.rs @@ -0,0 +1,573 @@ +use std::collections::BTreeMap; +use std::path::Path; + +use error_stack::Report; +use serde::{Deserialize, Deserializer, Serialize}; +use serde_json::{Value, json}; +use trusted_server_core::auction_config_types::{BidderId, ProviderId}; + +use super::{Output, PbsError, Result, identifier, read_text}; + +/// Deliberately partial: inspecting PBS requirements must not require unrelated TS settings. +#[derive(Default, Deserialize)] +struct Source { + auction: Option, + #[serde(default)] + integrations: Integrations, +} + +#[derive(Default, Deserialize)] +struct Auction { + enabled: Option, + #[serde(default)] + providers: BTreeMap, + #[serde(default)] + bidders: BTreeMap, +} + +#[derive(Deserialize)] +struct AuctionProvider { + profile: Option, + endpoint: Option, + timeout_ms: Option, + profile_config: Option, +} + +impl AuctionProvider { + fn profile_bool(&self, key: &str) -> Option { + self.profile_config.as_ref()?.get(key)?.as_bool() + } + + fn override_rule_count(&self) -> usize { + self.profile_config + .as_ref() + .and_then(|config| config.get("bid_param_override_rules")) + .and_then(toml::Value::as_array) + .map_or(0, Vec::len) + } +} + +#[derive(Deserialize)] +struct AuctionBidder { + provider: ProviderId, +} + +#[derive(Serialize)] +struct ServerBidderCandidate { + bidder: String, + source_key: String, + host_secret_requirement: &'static str, + partner_authorization: &'static str, +} + +#[derive(Serialize)] +struct ServerProviderReport { + provider: String, + endpoint_configured: bool, + timeout_ms_explicit: Option, + test_mode_explicit: Option, + debug_explicit: Option, + bid_param_override_rule_count: usize, + server_bidder_candidates: Vec, +} + +#[derive(Default, Deserialize)] +struct Integrations { + prebid: Option, +} + +#[derive(Default, Deserialize)] +struct Prebid { + enabled: Option, + account_id: Option, + timeout_ms: Option, + debug: Option, + #[serde(default, deserialize_with = "bidder_list")] + client_side_bidders: Vec, + #[serde(default)] + bundle: Bundle, +} + +#[derive(Default, Deserialize)] +struct Bundle { + #[serde(default)] + modules: BundleModules, +} + +/// Explicit selections from core's bundle module schema; never expand generator presets. +#[derive(Default, Deserialize)] +struct BundleModules { + #[serde(default)] + bidder: Vec, + #[serde(default)] + user_id: Vec, + #[serde(default)] + analytics: Vec, +} + +/// Accept the same encodings as the private core list deserializer without expanding defaults. +/// Unparseable escapes instead reach identifier validation for a sanitized error. +/// Parity tests compare accepted representations with `PrebidIntegrationConfig`. +/// +/// # Errors +/// Rejects malformed lists, invalid numeric indexes, and non-string items. +fn bidder_list<'de, D: Deserializer<'de>>( + deserializer: D, +) -> std::result::Result, D::Error> { + match Value::deserialize(deserializer)? { + Value::Array(values) => { + serde_json::from_value(Value::Array(values)).map_err(serde::de::Error::custom) + } + Value::Object(values) => { + let mut indexed = Vec::with_capacity(values.len()); + for (index, value) in values { + let index = index.parse::().map_err(serde::de::Error::custom)?; + let value: String = + serde_json::from_value(value).map_err(serde::de::Error::custom)?; + indexed.push((index, value)); + } + indexed.sort_by_key(|(index, _)| *index); + Ok(indexed.into_iter().map(|(_, value)| value).collect()) + } + Value::String(value) => { + let text = value.trim(); + let bracketed = text.starts_with('[') && text.ends_with(']'); + if bracketed && let Ok(values) = serde_json::from_str::>(text) { + return Ok(values); + } + let parts = if bracketed { + text[1..text.len() - 1] + .trim() + .split(',') + .map(str::trim) + .filter(|part| !part.is_empty()) + .collect::>() + } else if text.contains(',') { + text.split(',') + .map(str::trim) + .filter(|part| !part.is_empty()) + .collect() + } else { + vec![text] + }; + Ok(parts + .into_iter() + .map(|part| { + let json = format!("\"{}\"", part.replace('"', "\\\"")); + // Core propagates this error; keep raw text here so identifier validation + // rejects it without a serde snippet that could echo source contents. + serde_json::from_str(&json).unwrap_or_else(|_| part.to_owned()) + }) + .collect()) + } + _ => Err(serde::de::Error::custom( + "expected bidder list, indexed map, or list string", + )), + } +} + +/// Inspect selected local fields, preserving the source and redacting account/parameter values. +/// +/// # Errors +/// Rejects invalid TOML or malformed identifiers with sanitized messages. +pub(super) fn inspect(path: &Path) -> Result { + let text = read_text(path, 2 * 1024 * 1024)?; + let source: Source = toml::from_str(&text).map_err(|_| { + Report::new(PbsError::Input( + "cannot parse Trusted Server TOML; source details withheld", + )) + })?; + let auction_present = source.auction.is_some(); + let auction = source.auction.unwrap_or_default(); + let prebid_present = source.integrations.prebid.is_some(); + let prebid = source.integrations.prebid.unwrap_or_default(); + for name in prebid + .client_side_bidders + .iter() + .chain(&prebid.bundle.modules.bidder) + .chain(&prebid.bundle.modules.user_id) + .chain(&prebid.bundle.modules.analytics) + { + if !identifier(name) { + return Err(Report::new(PbsError::Input( + "invalid bidder or bundle-module identifier", + ))); + } + } + let server_providers: Vec<_> = auction + .providers + .iter() + .filter(|(_, provider)| provider.profile.as_deref() == Some("prebid-server")) + .map(|(provider_id, provider)| { + let requirements: Vec<_> = auction + .bidders + .iter() + .filter(|(_, bidder)| &bidder.provider == provider_id) + .map(|(bidder_id, _)| ServerBidderCandidate { + bidder: bidder_id.as_str().to_owned(), + source_key: format!("auction.bidders.{}.provider", bidder_id.as_str()), + host_secret_requirement: "unresolved", + partner_authorization: "unresolved", + }) + .collect(); + ServerProviderReport { + provider: provider_id.as_str().to_owned(), + endpoint_configured: provider.endpoint.is_some(), + timeout_ms_explicit: provider.timeout_ms, + test_mode_explicit: provider.profile_bool("test_mode"), + debug_explicit: provider.profile_bool("debug"), + bid_param_override_rule_count: provider.override_rule_count(), + server_bidder_candidates: requirements, + } + }) + .collect(); + let warnings = [ + "Local file only: confirm environment, remote configuration, and request-time overrides.", + "Omitted fields/defaults are not expanded; empty candidate lists are not proof of no demand.", + "Disabled auctions, providers, and browser bundle adapters do not authorize PBS activation.", + "Host secret requirements need adapter metadata verified against the selected PBS release.", + ]; + let mut details = vec![ + format!( + "Auction section present: {auction_present}; enabled explicitly: {:?}", + auction.enabled + ), + format!( + "Prebid browser section present: {prebid_present}; enabled explicitly: {:?}", + prebid.enabled + ), + format!("Prebid Server providers: {}", server_providers.len()), + ]; + details.extend(server_providers.iter().map(|provider| { + let bidders = provider + .server_bidder_candidates + .iter() + .map(|candidate| candidate.bidder.as_str()) + .collect::>() + .join(", "); + format!( + "Server provider {}: bidders: {}; endpoint configured: {}; timeout explicit: {:?}; test mode explicit: {:?}; debug explicit: {:?}; bid-parameter rules: {}; values withheld", + provider.provider, + bidders, + provider.endpoint_configured, + provider.timeout_ms_explicit, + provider.test_mode_explicit, + provider.debug_explicit, + provider.bid_param_override_rule_count + ) + })); + details.extend([ + format!( + "Client-side bidders: {}", + prebid.client_side_bidders.join(", ") + ), + format!( + "Browser bundle adapters: {}", + prebid.bundle.modules.bidder.join(", ") + ), + format!( + "Browser identity modules: {}", + prebid.bundle.modules.user_id.join(", ") + ), + format!( + "Browser analytics modules: {}", + prebid.bundle.modules.analytics.join(", ") + ), + ]); + details.extend(warnings.iter().map(|warning| (*warning).to_owned())); + Ok(Output { + failure: None, + summary: "Local PBS requirements discovery; no files changed or AWS calls made".to_owned(), + details, + data: json!({ + "source": path, + "source_sections": { + "server": ["auction.providers", "auction.bidders"], + "browser": "integrations.prebid" + }, + "auction_section_present": auction_present, + "auction_enabled_explicit": auction.enabled, + "server_providers": server_providers, + "section_present": prebid_present, + "enabled_explicit": prebid.enabled, + "account_id_configured": prebid.account_id.is_some(), + "timeout_ms_explicit": prebid.timeout_ms, + "debug_explicit": prebid.debug, + "client_side_bidders": prebid.client_side_bidders, + "bundle_adapters": prebid.bundle.modules.bidder, + "identity_modules": prebid.bundle.modules.user_id, + "analytics_modules": prebid.bundle.modules.analytics, + "warnings": warnings + }), + }) +} + +#[cfg(test)] +mod tests { + use std::fs; + + use super::*; + + #[test] + fn discovery_classifies_bidders_redacts_values_and_preserves_source() { + let dir = tempfile::tempdir().expect("should create temp directory"); + let path = dir.path().join("trusted-server.toml"); + let source = r#" +[auction] +enabled = true + +[auction.providers.pbs-main] +protocol = "openrtb-2.6" +profile = "prebid-server" +endpoint = "https://user:NEVER_PRINT_ME@pbs.example.com/path?token=NEVER_PRINT_ME" +timeout_ms = 900 +routing = "explicit" + +[auction.providers.pbs-main.profile_config] +debug = false +test_mode = true +bid_param_override_rules = [{ when = { bidder = "serverbidder" }, set = { placementId = "NEVER_PRINT_ME" } }] + +[auction.bidders.serverbidder] +provider = "pbs-main" + +[auction.providers.pbs-secondary] +protocol = "openrtb-2.6" +profile = "prebid-server" +endpoint = "https://NEVER_PRINT_ME@secondary.example.com/openrtb2/auction" +routing = "explicit" + +[auction.bidders.otherbidder] +provider = "pbs-secondary" + +[integrations.prebid] +enabled = false +account_id = "NEVER_PRINT_ME" +client_side_bidders = ["browserbidder"] + +[integrations.prebid.bundle.modules] +bidder = ["exampleBidAdapter"] +user_id = ["sharedIdSystem"] +analytics = ["exampleAnalyticsAdapter"] +"#; + fs::write(&path, source).expect("should write fixture"); + let report = inspect(&path).expect("should inspect config"); + assert_eq!(report.data["auction_section_present"], true); + assert_eq!(report.data["auction_enabled_explicit"], true); + assert_eq!(report.data["enabled_explicit"], false); + assert_eq!( + report.data["server_providers"].as_array().map(Vec::len), + Some(2) + ); + assert_eq!(report.data["server_providers"][0]["provider"], "pbs-main"); + assert_eq!( + report.data["server_providers"][0]["server_bidder_candidates"][0]["bidder"], + "serverbidder" + ); + assert_eq!( + report.data["server_providers"][1]["provider"], + "pbs-secondary" + ); + assert_eq!( + report.data["server_providers"][1]["server_bidder_candidates"][0]["bidder"], + "otherbidder" + ); + assert_eq!( + report.data["server_providers"][0]["server_bidder_candidates"][0]["source_key"], + "auction.bidders.serverbidder.provider" + ); + assert_eq!( + report.data["server_providers"][0]["endpoint_configured"], + true + ); + assert_eq!( + report.data["server_providers"][0]["timeout_ms_explicit"], + 900 + ); + assert_eq!( + report.data["server_providers"][0]["test_mode_explicit"], + true + ); + assert_eq!(report.data["server_providers"][0]["debug_explicit"], false); + assert_eq!( + report.data["server_providers"][0]["bid_param_override_rule_count"], + 1 + ); + assert_eq!(report.data["client_side_bidders"][0], "browserbidder"); + assert_eq!(report.data["bundle_adapters"], json!(["exampleBidAdapter"])); + assert_eq!(report.data["identity_modules"], json!(["sharedIdSystem"])); + assert_eq!( + report.data["analytics_modules"], + json!(["exampleAnalyticsAdapter"]) + ); + assert_eq!( + report.data["server_providers"][0]["server_bidder_candidates"][0]["host_secret_requirement"], + "unresolved" + ); + let mut human = Vec::new(); + report + .write(false, &mut human) + .expect("should render human report"); + let human = String::from_utf8(human).expect("should emit UTF-8"); + assert!(human.contains("pbs-main")); + assert!(human.contains("serverbidder")); + assert!(human.contains("Browser bundle adapters: exampleBidAdapter")); + assert!(human.contains("Browser identity modules: sharedIdSystem")); + assert!(human.contains("Browser analytics modules: exampleAnalyticsAdapter")); + assert!(!human.contains("NEVER_PRINT_ME")); + let mut json = Vec::new(); + report + .write(true, &mut json) + .expect("should render JSON report"); + assert!(!String::from_utf8_lossy(&json).contains("NEVER_PRINT_ME")); + assert_eq!( + fs::read_to_string(path).expect("should read fixture"), + source + ); + } + + #[test] + fn invalid_toml_never_exposes_parser_snippets() { + let dir = tempfile::tempdir().expect("should create temp directory"); + let path = dir.path().join("invalid.toml"); + fs::write(&path, "secret = NEVER_PRINT_ME").expect("should write fixture"); + let error = inspect(&path).err().expect("should reject invalid TOML"); + assert!(!format!("{error:?}").contains("NEVER_PRINT_ME")); + } + + #[test] + fn invalid_list_identifier_reports_identifier_error() { + let file = tempfile::NamedTempFile::new().expect("should create config"); + fs::write( + file.path(), + r#" +[integrations.prebid] +client_side_bidders = 'examplebidder\' +"#, + ) + .expect("should write config"); + + let error = inspect(file.path()) + .err() + .expect("should reject invalid identifier"); + + assert!( + error + .to_string() + .contains("invalid bidder or bundle-module identifier"), + "should report identifier validation: {error}" + ); + } + + #[test] + fn accepts_the_runtime_browser_bidder_list_encodings() { + for input in [ + "['examplebidder', 'otherbidder']", + "'examplebidder,otherbidder'", + "'[examplebidder, otherbidder]'", + "'[\"examplebidder\", \"otherbidder\"]'", + "'example\\u0062idder,otherbidder'", + "{ '10' = 'otherbidder', '2' = 'examplebidder' }", + ] { + let text = format!("[integrations.prebid]\nclient_side_bidders={input}\n"); + let file = tempfile::NamedTempFile::new().expect("should create config"); + fs::write(file.path(), &text).expect("should write config"); + let runtime: trusted_server_core::integrations::prebid::PrebidIntegrationConfig = + toml::from_str(&format!("client_side_bidders={input}")) + .expect("runtime should accept encoding"); + let output = inspect(file.path()).expect("inspect should accept runtime encoding"); + assert_eq!( + output.data["client_side_bidders"], + json!(runtime.client_side_bidders) + ); + } + } + + #[test] + fn bundle_selections_match_core_without_expanding_defaults() { + for input in [ + "", + "[bundle]", + "[bundle.modules]", + "[bundle.modules]\nbidder = []\nuser_id = []\nanalytics = []", + "[bundle.modules]\nbidder = ['exampleBidAdapter', 'otherBidAdapter']\nuser_id = ['sharedIdSystem']\nanalytics = ['exampleAnalyticsAdapter']", + "[bundle.modules]\nuser_id = ['sharedIdSystem']", + "[bundle.modules]\nanalytics = ['exampleAnalyticsAdapter']", + ] { + let runtime: trusted_server_core::integrations::prebid::PrebidIntegrationConfig = + toml::from_str(input).expect("should parse current core bundle schema"); + let text = format!( + "[integrations.prebid]\n{}", + input.replace("[bundle", "[integrations.prebid.bundle") + ); + let file = tempfile::NamedTempFile::new().expect("should create config"); + fs::write(file.path(), &text).expect("should write config"); + let output = inspect(file.path()).expect("should inspect current core bundle schema"); + assert_eq!( + output.data["bundle_adapters"], + json!(runtime.bundle.modules.bidder) + ); + assert_eq!( + output.data["identity_modules"], + json!(runtime.bundle.modules.user_id.unwrap_or_default()) + ); + assert_eq!( + output.data["analytics_modules"], + json!(runtime.bundle.modules.analytics.unwrap_or_default()) + ); + } + } + + #[test] + fn retired_bundle_fields_are_not_reported_as_current_selections() { + let input = + "[bundle]\nadapters = ['exampleBidAdapter']\nuser_id_modules = ['sharedIdSystem']"; + assert!( + toml::from_str::( + input + ) + .is_err(), + "core should reject the retired bundle schema" + ); + let file = tempfile::NamedTempFile::new().expect("should create config"); + fs::write( + file.path(), + input.replace("[bundle]", "[integrations.prebid.bundle]"), + ) + .expect("should write config"); + // Inspection is deliberately partial, not full runtime validation. + let output = + inspect(file.path()).expect("should ignore fields outside the inspected schema"); + for field in ["bundle_adapters", "identity_modules", "analytics_modules"] { + assert_eq!(output.data[field], json!([]), "should not infer {field}"); + } + } + + #[test] + fn invalid_bundle_selections_never_echo_source_values() { + for field in ["bidder", "user_id", "analytics"] { + for value in ["['NEVER_PRINT_ME invalid']", "'NEVER_PRINT_ME'"] { + let file = tempfile::NamedTempFile::new().expect("should create config"); + fs::write( + file.path(), + format!("[integrations.prebid.bundle.modules]\n{field} = {value}\n"), + ) + .expect("should write config"); + let error = inspect(file.path()) + .err() + .expect("should reject invalid module selection"); + assert!(!format!("{error:?}").contains("NEVER_PRINT_ME")); + } + } + } + + #[test] + fn missing_section_stays_unresolved_instead_of_enabling_defaults() { + let dir = tempfile::tempdir().expect("should create temp directory"); + let path = dir.path().join("empty.toml"); + fs::write(&path, "").expect("should write fixture"); + let output = inspect(&path).expect("should inspect empty file"); + assert_eq!(output.data["section_present"], false); + assert!(output.data["enabled_explicit"].is_null()); + } +} diff --git a/crates/trusted-server-cli/src/commands/pbs/mod.rs b/crates/trusted-server-cli/src/commands/pbs/mod.rs new file mode 100644 index 000000000..521f7c6ed --- /dev/null +++ b/crates/trusted-server-cli/src/commands/pbs/mod.rs @@ -0,0 +1,275 @@ +//! Experimental PBS operator commands, independent of Trusted Server deployment. + +mod aws; +mod config; +mod inspect; +mod secrets; +mod status; + +use std::fs::File; +use std::io::{self, BufRead, IsTerminal, Read, Write}; +use std::path::{Path, PathBuf}; + +use clap::{Args, Subcommand}; +use derive_more::Display; +use error_stack::Report; +use serde_json::Value; + +use aws::AwsCli; +use config::Deployment; + +/// Arguments for the experimental `ts prebid server` namespace. +#[derive(Debug, Args)] +pub(crate) struct PbsArgs { + /// Emit a nonsecret JSON report instead of a human summary. + #[arg(long, global = true)] + json: bool, + #[command(subcommand)] + command: PbsCommand, +} + +#[derive(Debug, Subcommand)] +enum PbsCommand { + /// Inspect local Trusted Server configuration without modifying it or contacting AWS. + Inspect { + /// Explicit source file; omitted fields/defaults and remote overrides remain unresolved. + #[arg(long, default_value = "trusted-server.toml")] + config: PathBuf, + }, + /// Validate declared PBS inputs and regional YAML merges locally, without AWS access. + Check(TargetArgs), + /// Read EC2 infrastructure status, not PBS health or the installed release. + Status(TargetArgs), + /// Manage values for existing, explicitly declared Secrets Manager secrets. + #[command(subcommand)] + Secrets(SecretCommand), +} + +#[derive(Debug, Args)] +struct TargetArgs { + /// Deployment descriptor; paths inside it are relative to this file. + #[arg(long)] + deployment: PathBuf, +} + +#[derive(Debug, Subcommand)] +enum SecretCommand { + /// Write a complete JSON credential payload; does not deploy or rotate partner credentials. + Set(secrets::SetArgs), +} + +/// Sanitized errors: source parser output and AWS stderr can contain credential values. +#[derive(Debug, Display)] +pub(crate) enum PbsError { + #[display("invalid PBS input: {_0}")] + Input(&'static str), + #[display("PBS I/O failed: {_0}")] + Io(&'static str), + #[display("PBS I/O failed: cannot open input file {path:?}")] + InputFile { path: PathBuf }, + #[display("AWS operation failed: {_0}; provider output withheld")] + Aws(&'static str), + #[display("AWS account does not match the deployment descriptor; no further calls made")] + AccountMismatch, + #[display("operation was not confirmed; no secret value was written")] + NotConfirmed, +} + +impl std::error::Error for PbsError {} + +pub(super) type Result = std::result::Result>; + +pub(super) struct Output { + summary: String, + details: Vec, + data: Value, + failure: Option, +} + +impl Output { + /// Write only explicitly constructed nonsecret report fields. + /// + /// # Errors + /// Returns a sanitized error if output cannot be written or serialized. + fn write(&self, json: bool, out: &mut dyn Write) -> Result<()> { + if json { + serde_json::to_writer_pretty(&mut *out, &self.data) + .map_err(|_| Report::new(PbsError::Io("cannot write JSON report")))?; + writeln!(out).map_err(|_| Report::new(PbsError::Io("cannot write report")))?; + } else { + writeln!(out, "{}", self.summary) + .map_err(|_| Report::new(PbsError::Io("cannot write report")))?; + for detail in &self.details { + writeln!(out, " {detail}") + .map_err(|_| Report::new(PbsError::Io("cannot write report")))?; + } + } + Ok(()) + } +} + +/// Execute PBS commands. Only explicitly selected cloud commands construct an AWS client. +/// +/// # Errors +/// Returns sanitized validation, I/O, identity, confirmation, or AWS errors. +pub(crate) fn run(args: &PbsArgs) -> Result<()> { + let output = match &args.command { + PbsCommand::Inspect { config } => inspect::inspect(config)?, + PbsCommand::Check(target) => config::check(&Deployment::load(&target.deployment)?)?, + PbsCommand::Status(target) => { + let deployment = Deployment::load(&target.deployment)?; + Terminal.notice(&format!( + "Read EC2 status: environment={}; account={}; profile={}; regions={}", + deployment.environment, + deployment.aws.account_id, + deployment.aws.profile, + deployment + .regions + .keys() + .cloned() + .collect::>() + .join(", ") + ))?; + status::status(&deployment, &AwsCli)? + } + PbsCommand::Secrets(SecretCommand::Set(set)) => { + let deployment = Deployment::load(&set.deployment)?; + secrets::set(set, &deployment, &AwsCli, &mut Terminal)? + } + }; + output.write(args.json, &mut io::stdout().lock())?; + match output.failure { + Some(error) => Err(Report::new(error)), + None => Ok(()), + } +} + +/// Read bounded local input without including file contents in error reports. +/// +/// # Errors +/// Returns an I/O error or rejects oversized/non-UTF-8 input. +pub(super) fn read_text(path: &Path, limit: usize) -> Result { + let file = File::open(path).map_err(|_| { + Report::new(PbsError::InputFile { + path: path.to_path_buf(), + }) + })?; + read_bounded(file, limit) +} + +/// Read bounded input, including piped credential JSON. +/// +/// # Errors +/// Rejects oversized/non-UTF-8 input and read failures without disclosing contents. +pub(super) fn read_bounded(input: impl Read, limit: usize) -> Result { + let mut text = String::new(); + input + .take(limit as u64 + 1) + .read_to_string(&mut text) + .map_err(|_| Report::new(PbsError::Io("cannot read UTF-8 input")))?; + if text.len() > limit { + return Err(Report::new(PbsError::Input("input exceeds size limit"))); + } + Ok(text) +} + +/// Keep report identifiers bounded and free of terminal control sequences. +pub(super) fn identifier(value: &str) -> bool { + !value.is_empty() + && value.len() <= 128 + && value + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'_' | b'-')) +} + +fn is_confirmation(input: &str) -> bool { + input.len() <= 16 && input.trim() == "yes" +} + +/// Operator interaction is injectable so tests cannot accidentally read a terminal. +pub(super) trait Interaction { + /// Read a complete secret JSON object without echoing it. + /// + /// # Errors + /// Returns an error when an interactive terminal is unavailable. + fn secret(&mut self) -> Result; + /// Request approval of a nonsecret target description. + /// + /// # Errors + /// Returns an error on unavailable terminal or failed I/O. + fn confirm(&mut self, target: &str) -> Result; + /// Display nonsecret target and retry information separately from JSON stdout. + /// + /// # Errors + /// Returns an error if the notice cannot be written. + fn notice(&mut self, message: &str) -> Result<()>; +} + +struct Terminal; + +impl Interaction for Terminal { + fn secret(&mut self) -> Result { + if !io::stdin().is_terminal() { + return Err(Report::new(PbsError::Input( + "use --file or --stdin for noninteractive secret input", + ))); + } + rpassword::prompt_password("Secret JSON (hidden): ") + .map_err(|_| Report::new(PbsError::Io("cannot read hidden secret input"))) + } + + fn confirm(&mut self, target: &str) -> Result { + if !io::stdin().is_terminal() { + return Err(Report::new(PbsError::Input( + "automation requires --yes and --request-token", + ))); + } + self.notice(target)?; + self.notice("Write this secret version? Type yes to confirm:")?; + let line = io::stdin() + .lock() + .lines() + .next() + .transpose() + .map_err(|_| Report::new(PbsError::Io("cannot read confirmation")))? + .unwrap_or_default(); + Ok(is_confirmation(&line)) + } + + fn notice(&mut self, message: &str) -> Result<()> { + writeln!(io::stderr().lock(), "{message}") + .map_err(|_| Report::new(PbsError::Io("cannot write operator notice"))) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn read_bounded_preserves_size_utf8_and_error_redaction_guards() { + assert_eq!( + read_bounded(&b"abcd"[..], 4).expect("should accept limit"), + "abcd" + ); + for (input, limit) in [(&b"DUMMY_SECRET"[..], 4), (&b"\xffDUMMY_SECRET"[..], 64)] { + let error = read_bounded(input, limit).expect_err("should reject invalid input"); + assert!(!format!("{error:?}").contains("DUMMY_SECRET")); + } + struct FailedRead; + impl Read for FailedRead { + fn read(&mut self, _buffer: &mut [u8]) -> io::Result { + Err(io::Error::other("DUMMY_SECRET")) + } + } + let error = read_bounded(FailedRead, 4).expect_err("should reject read failure"); + assert!(!format!("{error:?}").contains("DUMMY_SECRET")); + } + + #[test] + fn confirmation_accepts_only_bounded_yes() { + assert!(is_confirmation("yes")); + assert!(!is_confirmation("no")); + assert!(!is_confirmation("yes-but-with-more-than-sixteen-bytes")); + } +} diff --git a/crates/trusted-server-cli/src/commands/pbs/secrets.rs b/crates/trusted-server-cli/src/commands/pbs/secrets.rs new file mode 100644 index 000000000..3aa885187 --- /dev/null +++ b/crates/trusted-server-cli/src/commands/pbs/secrets.rs @@ -0,0 +1,392 @@ +use std::collections::BTreeMap; +use std::io; +use std::path::PathBuf; + +use clap::Args; +use error_stack::Report; +use serde::de::{MapAccess, Visitor}; +use serde::{Deserialize, Deserializer}; +use serde_json::{Value, json}; +use uuid::Uuid; + +use super::aws::{Aws, WRITE_NOT_CONFIRMED, verify_identity}; +use super::config::{Binding, Deployment, invalid}; +use super::{Interaction, Output, PbsError, Result, read_bounded, read_text}; + +const MAX_SECRET_BYTES: usize = 65_536; + +/// No argument accepts a secret value. File/stdin and terminal input carry the payload. +#[derive(Debug, Args)] +pub(super) struct SetArgs { + /// Bidder identifier declared in the binding file. + bidder: String, + /// Deployment descriptor; paths inside it are relative to this file. + #[arg(long)] + pub(super) deployment: PathBuf, + /// Exactly one declared region; replicas cannot be written independently. + #[arg(long)] + region: String, + /// Read a complete JSON string-valued object from this file; never changes the file. + #[arg(long, conflicts_with = "stdin")] + file: Option, + /// Read the JSON object from stdin; requires --yes and --request-token. + #[arg(long)] + stdin: bool, + /// Approve the declared target noninteractively, for an externally approved automation job. + #[arg(long, requires = "request_token")] + yes: bool, + /// Stable UUID for retries of this logical write. Reuse with identical values only. + #[arg(long)] + request_token: Option, +} + +struct Payload(BTreeMap); + +impl<'de> Deserialize<'de> for Payload { + fn deserialize>(deserializer: D) -> std::result::Result { + struct UniqueKeys; + impl<'de> Visitor<'de> for UniqueKeys { + type Value = Payload; + fn expecting(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter.write_str("a JSON object with unique keys and string values") + } + fn visit_map>( + self, + mut map: M, + ) -> std::result::Result { + let mut values = BTreeMap::new(); + while let Some((key, value)) = map.next_entry::()? { + if values.insert(key, value).is_some() { + return Err(serde::de::Error::custom("duplicate credential key")); + } + } + Ok(Payload(values)) + } + } + deserializer.deserialize_map(UniqueKeys) + } +} + +/// Validate a complete payload without returning secret-bearing parser diagnostics. +/// +/// # Errors +/// Rejects duplicate/unknown/missing keys, non-string values, empty required values and oversized JSON. +fn validate_payload(text: &str, binding: &Binding) -> Result { + if text.len() > MAX_SECRET_BYTES { + return Err(invalid("secret JSON exceeds Secrets Manager size limit")); + } + let payload: Payload = serde_json::from_str(text) + .map_err(|_| invalid("secret input must be a JSON object with unique keys and string values; contents withheld"))?; + if payload.0.keys().any(|key| !binding.keys.contains_key(key)) { + return Err(invalid("secret JSON includes undeclared keys")); + } + for (key, rule) in &binding.keys { + if rule.required && payload.0.get(key).is_none_or(String::is_empty) { + return Err(invalid("secret JSON is missing a required nonempty string")); + } + } + if payload.0.is_empty() { + return Err(invalid( + "secret JSON must contain at least one declared key", + )); + } + let encoded = + serde_json::to_string(&payload.0).map_err(|_| invalid("cannot encode credential JSON"))?; + if encoded.len() > MAX_SECRET_BYTES { + return Err(invalid( + "encoded credential JSON exceeds Secrets Manager size limit", + )); + } + Ok(encoded) +} + +/// Set a declared credential version after identity, metadata, input and approval checks. +/// +/// # Errors +/// Rejects unapproved input, account/secret mismatches, replicas, deletion, malformed values and AWS errors. +pub(super) fn set( + args: &SetArgs, + deployment: &Deployment, + aws: &dyn Aws, + interaction: &mut dyn Interaction, +) -> Result { + if (args.yes && args.request_token.is_none()) || (args.stdin && !args.yes) { + return Err(invalid( + "noninteractive writes require --yes and --request-token", + )); + } + let binding = deployment + .bindings + .get(&args.bidder) + .ok_or_else(|| invalid("bidder has no declared secret binding"))?; + let arn = binding + .secrets + .get(&args.region) + .ok_or_else(|| invalid("bidder has no secret in the selected region"))?; + let target = format!( + "Environment: {}; account: {}; profile: {}; region: {}; bidder: {}; secret: {arn}", + deployment.environment, + deployment.aws.account_id, + deployment.aws.profile, + args.region, + args.bidder + ); + interaction.notice(&target)?; + verify_identity(deployment, &args.region, aws)?; + let metadata = aws.call( + &deployment.aws, + &args.region, + "secretsmanager", + "describe-secret", + &json!({"SecretId": arn}), + )?; + if metadata.get("ARN").and_then(Value::as_str) != Some(arn.as_str()) + || metadata + .get("DeletedDate") + .is_some_and(|value| !value.is_null()) + { + return Err(invalid( + "declared secret identity differs or is scheduled for deletion", + )); + } + if metadata + .get("PrimaryRegion") + .and_then(Value::as_str) + .is_some_and(|region| region != args.region) + { + return Err(invalid( + "selected secret is a replica; write to its approved primary region instead", + )); + } + let text = if let Some(path) = &args.file { + read_text(path, MAX_SECRET_BYTES)? + } else if args.stdin { + read_bounded(io::stdin().lock(), MAX_SECRET_BYTES)? + } else { + interaction.secret()? + }; + let payload = validate_payload(&text, binding)?; + let token = args.request_token.unwrap_or_else(Uuid::new_v4).to_string(); + interaction.notice(&format!("Request token for identical retries: {token}"))?; + if !args.yes && !interaction.confirm(&target)? { + return Err(Report::new(PbsError::NotConfirmed)); + } + let response = aws.call( + &deployment.aws, + &args.region, + "secretsmanager", + "put-secret-value", + &json!({"SecretId": arn, "ClientRequestToken": token, "SecretString": payload}), + )?; + if response.get("ARN").and_then(Value::as_str) != Some(arn.as_str()) + || response.get("VersionId").and_then(Value::as_str) != Some(token.as_str()) + { + return Err(Report::new(PbsError::Aws(WRITE_NOT_CONFIRMED))); + } + Ok(Output { + failure: None, + summary: format!("Stored credential version for {} in {}; no containers changed", args.bidder, args.region), + details: vec![format!("Version / retry token: {token}"), + "Confirm replica readiness and separately deploy consuming containers before revoking old partner credentials.".to_owned()], + data: json!({"environment": deployment.environment, "account_id": deployment.aws.account_id, + "region": args.region, "bidder": args.bidder, "secret_arn": arn, "version_id": token, + "deployed": false, "partner_credential_rotated": false, + "regions_requiring_deployment_review": binding.secrets.keys().collect::>()}), + }) +} + +#[cfg(test)] +mod tests { + use std::fs; + + use super::*; + use crate::commands::pbs::aws::tests::FakeAws; + use crate::commands::pbs::config::tests::fixture; + + const ARN: &str = "arn:aws:secretsmanager:us-east-1:123456789012:secret:pbs/example-AbCdEf"; + const TOKEN: &str = "11111111-2222-4333-8444-555555555555"; + + struct Prompt { + approve: bool, + notices: Vec, + } + + impl Interaction for Prompt { + fn secret(&mut self) -> Result { + Ok("{\"api_key\":\"dummy\"}".to_owned()) + } + fn confirm(&mut self, _target: &str) -> Result { + Ok(self.approve) + } + fn notice(&mut self, text: &str) -> Result<()> { + self.notices.push(text.to_owned()); + Ok(()) + } + } + + fn args(path: &std::path::Path) -> SetArgs { + SetArgs { + bidder: "examplebidder".to_owned(), + deployment: path.to_path_buf(), + region: "us-east-1".to_owned(), + file: None, + stdin: false, + yes: false, + request_token: Some(Uuid::parse_str(TOKEN).expect("should parse token")), + } + } + + fn replies() -> Vec { + vec![ + json!({"Account": "123456789012"}), + json!({"ARN": ARN}), + json!({"ARN": ARN, "VersionId": TOKEN}), + ] + } + + #[test] + fn wrong_account_and_declined_confirmation_never_write() { + let (_dir, path) = fixture(); + let deployment = Deployment::load(&path).expect("should load deployment"); + let aws = FakeAws::new(vec![json!({"Account": "999999999999"})]); + let mut prompt = Prompt { + approve: true, + notices: vec![], + }; + assert!(set(&args(&path), &deployment, &aws, &mut prompt).is_err()); + assert_eq!(aws.calls.borrow().len(), 1); + let aws = FakeAws::new(replies()); + prompt.approve = false; + assert!(set(&args(&path), &deployment, &aws, &mut prompt).is_err()); + assert_eq!(aws.calls.borrow().len(), 2); + } + + #[test] + fn special_characters_round_trip_without_output_or_source_file_changes() { + let (dir, path) = fixture(); + let secret = "dummy-$\"\nline\\end"; + let original = json!({"api_key": secret}).to_string(); + let input = dir.path().join("secret.json"); + fs::write(&input, &original).expect("should write secret fixture"); + let deployment = Deployment::load(&path).expect("should load deployment"); + let mut arguments = args(&path); + arguments.file = Some(input.clone()); + arguments.yes = true; + let aws = FakeAws::new(replies()); + let mut prompt = Prompt { + approve: false, + notices: vec![], + }; + let report = + set(&arguments, &deployment, &aws, &mut prompt).expect("should write approved secret"); + let calls = aws.calls.borrow(); + assert_eq!(calls[2].1, "put-secret-value"); + let stored: Value = serde_json::from_str( + calls[2].2["SecretString"] + .as_str() + .expect("should have encoded JSON"), + ) + .expect("should parse payload"); + assert_eq!(stored["api_key"], secret); + assert_eq!(calls[2].2["ClientRequestToken"], TOKEN); + for json in [false, true] { + let mut output = Vec::new(); + report + .write(json, &mut output) + .expect("should render report"); + assert!(!String::from_utf8_lossy(&output).contains("dummy-")); + } + assert!(!prompt.notices.join("\n").contains("dummy-")); + assert_eq!( + fs::read_to_string(input).expect("should read input"), + original + ); + assert_eq!(report.data["deployed"], false); + } + + #[test] + fn replicas_and_missing_confirmation_inputs_fail_closed() { + let (_dir, path) = fixture(); + let deployment = Deployment::load(&path).expect("should load deployment"); + let aws = FakeAws::new(vec![ + json!({"Account": "123456789012"}), + json!({"ARN": ARN, "PrimaryRegion": "us-west-2"}), + ]); + let mut prompt = Prompt { + approve: true, + notices: vec![], + }; + assert!(set(&args(&path), &deployment, &aws, &mut prompt).is_err()); + assert_eq!(aws.calls.borrow().len(), 2); + let mut arguments = args(&path); + arguments.yes = true; + arguments.request_token = None; + let aws = FakeAws::new(vec![]); + assert!(set(&arguments, &deployment, &aws, &mut prompt).is_err()); + assert!(aws.calls.borrow().is_empty()); + } + + #[test] + fn undeclared_region_and_mismatched_secret_metadata_never_write() { + let (_dir, path) = fixture(); + let deployment = Deployment::load(&path).expect("should load deployment"); + let mut arguments = args(&path); + arguments.region = "us-west-2".to_owned(); + let aws = FakeAws::new(vec![]); + let mut prompt = Prompt { + approve: true, + notices: vec![], + }; + assert!(set(&arguments, &deployment, &aws, &mut prompt).is_err()); + assert!(aws.calls.borrow().is_empty()); + for metadata in [ + json!({"ARN": "arn:aws:secretsmanager:us-east-1:123456789012:secret:other-AbCdEf"}), + json!({"ARN": ARN, "DeletedDate": 1234567890}), + ] { + let aws = FakeAws::new(vec![json!({"Account": "123456789012"}), metadata]); + assert!(set(&args(&path), &deployment, &aws, &mut prompt).is_err()); + assert_eq!(aws.calls.borrow().len(), 2); + } + } + + #[test] + fn payload_rejects_missing_extra_duplicate_and_nonstring_keys_without_leaks() { + let (_dir, path) = fixture(); + let deployment = Deployment::load(&path).expect("should load deployment"); + let binding = &deployment.bindings["examplebidder"]; + for payload in [ + "{}", + "{\"api_key\":null}", + "{\"api_key\":\"\"}", + "{\"api_key\":\"dummy\",\"extra\":\"NEVER_PRINT_ME\"}", + "{\"api_key\":\"NEVER_PRINT_ME\",\"api_key\":\"second\"}", + "NEVER_PRINT_ME", + ] { + let error = validate_payload(payload, binding) + .expect_err("should reject invalid credential payload"); + assert!(!format!("{error:?}").contains("NEVER_PRINT_ME")); + } + } + + #[test] + fn retries_reuse_the_same_token_and_payload_and_failures_are_reported() { + let (_dir, path) = fixture(); + let deployment = Deployment::load(&path).expect("should load deployment"); + let mut all = replies(); + all.extend(replies()); + let aws = FakeAws::new(all); + let mut prompt = Prompt { + approve: true, + notices: vec![], + }; + for _ in 0..2 { + set(&args(&path), &deployment, &aws, &mut prompt).expect("should submit retry"); + } + assert_eq!(aws.calls.borrow()[2].2, aws.calls.borrow()[5].2); + let aws = FakeAws::new(vec![ + json!({"Account": "123456789012"}), + json!({"ARN": ARN}), + ]); + assert!(set(&args(&path), &deployment, &aws, &mut prompt).is_err()); + } +} diff --git a/crates/trusted-server-cli/src/commands/pbs/status.rs b/crates/trusted-server-cli/src/commands/pbs/status.rs new file mode 100644 index 000000000..8656f2fb9 --- /dev/null +++ b/crates/trusted-server-cli/src/commands/pbs/status.rs @@ -0,0 +1,184 @@ +use serde_json::{Value, json}; + +use super::aws::{Aws, verify_identity}; +use super::config::{Deployment, invalid}; +use super::{Output, PbsError, Result}; + +/// Read only the explicitly declared EC2 instances. Never infer PBS health from EC2 health. +/// +/// # Errors +/// Rejects missing instance targets, wrong accounts, and failed identity verification. +/// Identity failures abort the entire report. Resource-query failures are included as unknown in a +/// partial report with a failing exit status. +pub(super) fn status(deployment: &Deployment, aws: &dyn Aws) -> Result { + if deployment + .regions + .values() + .any(|region| region.instance_ids.is_empty()) + { + return Err(invalid( + "status requires explicit instance_ids in every selected descriptor region", + )); + } + let mut regions = Vec::new(); + let mut details = Vec::new(); + let mut failed = false; + for (region, target) in &deployment.regions { + verify_identity(deployment, region, aws)?; + let response = aws.call( + &deployment.aws, + region, + "ec2", + "describe-instance-status", + &json!({"InstanceIds": target.instance_ids, "IncludeAllInstances": true}), + ); + let statuses = response + .as_ref() + .ok() + .and_then(|value| value.get("InstanceStatuses")) + .and_then(Value::as_array); + let mut instances = Vec::new(); + for id in &target.instance_ids { + let matching: Vec<_> = statuses + .into_iter() + .flatten() + .filter(|value| { + value.get("InstanceId").and_then(Value::as_str) == Some(id.as_str()) + }) + .collect(); + let state = if matching.len() == 1 { + Some(matching[0]) + } else { + None + }; + let instance_state = allowed( + state, + "/InstanceState/Name", + &[ + "pending", + "running", + "shutting-down", + "terminated", + "stopping", + "stopped", + ], + ); + let instance_health = allowed( + state, + "/InstanceStatus/Status", + &[ + "ok", + "impaired", + "initializing", + "insufficient-data", + "not-applicable", + ], + ); + let system_health = allowed( + state, + "/SystemStatus/Status", + &[ + "ok", + "impaired", + "initializing", + "insufficient-data", + "not-applicable", + ], + ); + failed |= state.is_none() + || instance_state == "unknown" + || instance_health == "unknown" + || system_health == "unknown"; + details.push(format!("{region} {id}: state={instance_state}, EC2 instance={instance_health}, system={system_health}; PBS health/release unknown")); + instances.push(json!({"instance_id": id, "state": instance_state, "instance_health": instance_health, + "system_health": system_health, "pbs_health": "unknown", "release_id": null, "secret_versions": null})); + } + regions.push(json!({"region": region, "query": if statuses.is_some() { "completed" } else { "failed_or_invalid" }, "instances": instances})); + } + Ok(Output { + summary: format!( + "EC2 infrastructure status for {}; PBS runtime status is not implemented", + deployment.environment + ), + details, + data: json!({"environment": deployment.environment, "account_id": deployment.aws.account_id, + "complete": !failed, "regions": regions}), + failure: failed.then_some(PbsError::Aws("status report is incomplete")), + }) +} + +fn allowed(record: Option<&Value>, pointer: &str, choices: &[&'static str]) -> &'static str { + let value = record + .and_then(|record| record.pointer(pointer)) + .and_then(Value::as_str); + choices + .iter() + .copied() + .find(|choice| Some(*choice) == value) + .unwrap_or("unknown") +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::commands::pbs::aws::tests::FakeAws; + use crate::commands::pbs::config::tests::fixture; + + #[test] + fn queries_only_declared_instances_and_does_not_claim_pbs_health() { + let (_dir, path) = fixture(); + let deployment = Deployment::load(&path).expect("should load deployment"); + let aws = FakeAws::new(vec![ + json!({"Account": "123456789012"}), + json!({"InstanceStatuses": [{ + "InstanceId": "i-0123456789abcdef0", "InstanceState": {"Name": "running"}, + "InstanceStatus": {"Status": "ok"}, "SystemStatus": {"Status": "ok"} + }]}), + ]); + let output = status(&deployment, &aws).expect("should read status"); + assert!(output.failure.is_none()); + assert_eq!( + output.data["regions"][0]["instances"][0]["pbs_health"], + "unknown" + ); + assert!(output.data["regions"][0]["instances"][0]["release_id"].is_null()); + assert_eq!( + aws.calls.borrow()[1].2["InstanceIds"], + json!(["i-0123456789abcdef0"]) + ); + } + + #[test] + fn absent_and_failed_queries_produce_incomplete_unknown_reports() { + let (_dir, path) = fixture(); + let deployment = Deployment::load(&path).expect("should load deployment"); + for replies in [ + vec![json!({"Account": "123456789012"})], + vec![ + json!({"Account": "123456789012"}), + json!({"InstanceStatuses": []}), + ], + ] { + let aws = FakeAws::new(replies); + let output = status(&deployment, &aws).expect("should preserve partial report"); + assert!(output.failure.is_some()); + assert_eq!(output.data["complete"], false); + assert_eq!( + output.data["regions"][0]["instances"][0]["state"], + "unknown" + ); + } + } + + #[test] + fn wrong_account_prevents_resource_queries() { + let (_dir, path) = fixture(); + let deployment = Deployment::load(&path).expect("should load deployment"); + let aws = FakeAws::new(vec![json!({"Account": "999999999999"})]); + let error = status(&deployment, &aws) + .err() + .expect("should reject account mismatch"); + assert!(matches!(error.current_context(), PbsError::AccountMismatch)); + assert_eq!(aws.calls.borrow().len(), 1); + } +} diff --git a/crates/trusted-server-cli/src/prebid_bundle.rs b/crates/trusted-server-cli/src/prebid_bundle.rs index 9d00251a4..609c95053 100644 --- a/crates/trusted-server-cli/src/prebid_bundle.rs +++ b/crates/trusted-server-cli/src/prebid_bundle.rs @@ -10,7 +10,7 @@ use toml_edit::{DocumentMut, Item, table, value}; pub(crate) type CliResult = Result; -const NODE_MODULES_MISSING_HELP: &str = "Prebid bundling dependencies are missing. Run `cd crates/trusted-server-js/lib && npm ci`, then retry `ts prebid bundle`."; +const NODE_MODULES_MISSING_HELP: &str = "Prebid bundling dependencies are missing. Run `cd crates/trusted-server-js/lib && npm ci`, then retry `ts prebid client`."; const USER_ID_REGISTRY_RELATIVE_PATH: &str = "src/integrations/prebid/user_id_modules.json"; #[derive(Debug, clap::Args)] @@ -326,7 +326,7 @@ fn validate_managed_user_id_modules( .any(|module| module == &requirement.module_name) { return cli_error(format!( - "{} configures managed User ID {:?}, which requires Prebid module {:?}, but the generated manifest omits it; add {:?} to integrations.prebid.bundle.modules.user_id and rerun `ts prebid bundle`", + "{} configures managed User ID {:?}, which requires Prebid module {:?}, but the generated manifest omits it; add {:?} to integrations.prebid.bundle.modules.user_id and rerun `ts prebid client`", config_path.display(), requirement.config_name, requirement.module_name, @@ -641,7 +641,7 @@ fn find_js_lib_dir(start: &Path) -> CliResult { } cli_error( - "failed to locate crates/trusted-server-js/lib; run `ts prebid bundle` from the Trusted Server repository", + "failed to locate crates/trusted-server-js/lib; run `ts prebid client` from the Trusted Server repository", ) } diff --git a/crates/trusted-server-cli/src/run.rs b/crates/trusted-server-cli/src/run.rs index 61e8139d2..17fff5559 100644 --- a/crates/trusted-server-cli/src/run.rs +++ b/crates/trusted-server-cli/src/run.rs @@ -10,6 +10,7 @@ use trusted_server_core::config::TrustedServerAppConfig; use crate::commands::audit::{AuditArgs, run_audit}; use crate::commands::config::ad_templates::{AdTemplatesCommand, run_ad_templates}; use crate::commands::config::init::{ConfigInitArgs, run_config_init}; +use crate::commands::pbs::{self, PbsArgs}; use crate::prebid_bundle::{NpmPrebidBundleGenerator, PrebidBundleArgs, run_bundle}; #[derive(Debug, Parser)] @@ -80,8 +81,10 @@ struct PrebidArgs { #[derive(Debug, Subcommand)] enum PrebidCommand { - /// Generate a local external Prebid bundle and update config metadata. - Bundle(PrebidBundleArgs), + /// Generate a local external Prebid client bundle and update config metadata. + Client(PrebidBundleArgs), + /// Configure and operate a self-hosted Prebid Server deployment. + Server(PbsArgs), } /// Process-level outcome for commands that distinguish drift from tool errors. @@ -149,17 +152,18 @@ fn dispatch(args: Args) -> Result { Command::Healthcheck(args) => { edgezero_cli::run_healthcheck(&args).map(|()| RunOutcome::Success) } - Command::Prebid(prebid) => { - let mut generator = NpmPrebidBundleGenerator; - let mut stdout = std::io::stdout(); - let mut stderr = std::io::stderr(); - match prebid.command { - PrebidCommand::Bundle(args) => { - run_bundle(&args, &mut generator, &mut stdout, &mut stderr) - .map(|()| RunOutcome::Success) - } + Command::Prebid(prebid) => match prebid.command { + PrebidCommand::Client(args) => { + let mut generator = NpmPrebidBundleGenerator; + let mut stdout = std::io::stdout(); + let mut stderr = std::io::stderr(); + run_bundle(&args, &mut generator, &mut stdout, &mut stderr) + .map(|()| RunOutcome::Success) } - } + PrebidCommand::Server(args) => pbs::run(&args) + .map(|()| RunOutcome::Success) + .map_err(|error| error.current_context().to_string()), + }, Command::Provision(args) => { edgezero_cli::run_provision(&args).map(|()| RunOutcome::Success) } @@ -203,6 +207,100 @@ mod tests { ); } + #[test] + fn prebid_rejects_retired_bundle_command() { + let error = Args::try_parse_from(["ts", "prebid", "bundle"]) + .expect_err("should reject intentionally retired bundle spelling"); + assert_eq!(error.kind(), clap::error::ErrorKind::InvalidSubcommand); + } + + #[test] + fn prebid_server_secret_help_explains_descriptor_relative_paths() { + let error = Args::try_parse_from(["ts", "prebid", "server", "secrets", "set", "--help"]) + .expect_err("should display secret help"); + assert_eq!(error.kind(), clap::error::ErrorKind::DisplayHelp); + assert!( + error + .to_string() + .contains("Deployment descriptor; paths inside it are relative to this file") + ); + } + + #[test] + fn parses_prebid_server_inspect() { + assert!( + Args::try_parse_from([ + "ts", + "prebid", + "server", + "inspect", + "--config", + "trusted-server.toml", + "--json", + ]) + .is_ok(), + "should accept PBS commands under the Prebid namespace" + ); + } + + #[test] + fn prebid_server_rejects_ambiguous_secret_inputs_and_unimplemented_commands() { + for arguments in [ + vec!["ts", "prebid", "server", "deploy"], + vec!["ts", "prebid", "server", "rollback", "--release", "example"], + vec!["ts", "prebid", "server", "check"], + vec![ + "ts", + "prebid", + "server", + "secrets", + "set", + "examplebidder", + "--deployment", + "deployment.yaml", + "--region", + "us-east-1", + "--yes", + ], + vec![ + "ts", + "prebid", + "server", + "secrets", + "set", + "examplebidder", + "--deployment", + "deployment.yaml", + "--region", + "us-east-1", + "--file", + "secret.json", + "--stdin", + ], + ] { + assert!( + Args::try_parse_from(arguments).is_err(), + "should reject unsafe or unsupported command shape" + ); + } + assert!( + Args::try_parse_from([ + "ts", + "prebid", + "server", + "check", + "--deployment", + "deployment.yaml", + "--json", + ]) + .is_ok() + ); + assert!( + Args::try_parse_from(["ts", "prebid", "client", "--config", "trusted-server.toml"]) + .is_ok() + ); + } + #[test] fn parses_active_version() { let args = parse(&[ @@ -1050,22 +1148,24 @@ mod tests { } #[test] - fn prebid_bundle_defaults_match_spec() { - let args = parse(&["ts", "prebid", "bundle"]); + fn prebid_client_defaults_match_spec() { + let args = parse(&["ts", "prebid", "client"]); let Command::Prebid(prebid) = args.command else { panic!("expected prebid command"); }; - let PrebidCommand::Bundle(bundle) = prebid.command; - assert_eq!(bundle.config, PathBuf::from("trusted-server.toml")); - assert_eq!(bundle.out, PathBuf::from("dist/prebid")); + let PrebidCommand::Client(client) = prebid.command else { + panic!("expected prebid client command"); + }; + assert_eq!(client.config, PathBuf::from("trusted-server.toml")); + assert_eq!(client.out, PathBuf::from("dist/prebid")); } #[test] - fn prebid_bundle_accepts_custom_paths() { + fn prebid_client_accepts_custom_paths() { let args = parse(&[ "ts", "prebid", - "bundle", + "client", "--config", "publisher.toml", "--out", @@ -1074,14 +1174,16 @@ mod tests { let Command::Prebid(prebid) = args.command else { panic!("expected prebid command"); }; - let PrebidCommand::Bundle(bundle) = prebid.command; - assert_eq!(bundle.config, PathBuf::from("publisher.toml")); - assert_eq!(bundle.out, PathBuf::from("build/prebid")); + let PrebidCommand::Client(client) = prebid.command else { + panic!("expected prebid client command"); + }; + assert_eq!(client.config, PathBuf::from("publisher.toml")); + assert_eq!(client.out, PathBuf::from("build/prebid")); } #[test] - fn prebid_bundle_does_not_accept_adapter_option() { - let error = Args::try_parse_from(["ts", "prebid", "bundle", "--adapter", "fastly"]) + fn prebid_client_does_not_accept_adapter_option() { + let error = Args::try_parse_from(["ts", "prebid", "client", "--adapter", "fastly"]) .expect_err("should reject prebid adapter option"); assert!( error.to_string().contains("unexpected argument") diff --git a/crates/trusted-server-cli/tests/fixtures/pbs-aws.cjs b/crates/trusted-server-cli/tests/fixtures/pbs-aws.cjs new file mode 100644 index 000000000..24a52f294 --- /dev/null +++ b/crates/trusted-server-cli/tests/fixtures/pbs-aws.cjs @@ -0,0 +1,72 @@ +#!/usr/bin/env node +// Fake AWS process for PBS CLI tests. Never contacts AWS. +const assert = require('node:assert/strict') +const { + appendFileSync, + readFileSync, + statSync, + writeFileSync, +} = require('node:fs') +const { join } = require('node:path') + +const args = process.argv.slice(2) +const root = process.env.PBS_FAKE_ROOT +appendFileSync(join(root, 'calls'), `${JSON.stringify(args)}\n`) +assert(args.includes('--profile')) +assert.equal(args[args.indexOf('--profile') + 1], 'pbs-sandbox') +if (args.includes('configure')) { + if (process.env.PBS_FAKE_HISTORY === 'unset') process.exit(1) + console.log( + process.env.PBS_FAKE_HISTORY === 'enabled' ? 'enabled' : 'disabled' + ) + process.exit(0) +} +assert(args.includes('--region')) +assert.equal(args[args.indexOf('--region') + 1], 'us-east-1') +assert.equal(process.env.AWS_IGNORE_CONFIGURED_ENDPOINT_URLS, 'true') +assert(args.includes('--cli-input-json')) +const input = args[args.indexOf('--cli-input-json') + 1] +assert(input.startsWith('file://')) +const path = input.slice('file://'.length) +assert.equal(statSync(path).mode & 0o7777, 0o600) +appendFileSync(join(root, 'payload_paths'), `${path}\n`) +const request = JSON.parse(readFileSync(path, 'utf8')) +const arn = + 'arn:aws:secretsmanager:us-east-1:123456789012:secret:pbs/example-AbCdEf' +if (args.includes('get-caller-identity')) { + console.log( + JSON.stringify({ Account: process.env.PBS_FAKE_ACCOUNT ?? '123456789012' }) + ) +} else if (args.includes('describe-secret')) { + console.log(JSON.stringify({ ARN: arn })) +} else if (args.includes('put-secret-value')) { + const failure = process.env.PBS_FAKE_FAILURE + if (failure === 'collision' || failure === 'transport') { + const error = + failure === 'collision' + ? 'ResourceExistsException' + : 'Connection reset by peer' + console.error(`${error}: ${request.SecretString}`) + process.exit(1) + } + writeFileSync(join(root, 'captured_request.json'), JSON.stringify(request)) + if (failure === 'invalid-json') { + console.log(`invalid response ${request.SecretString}`) + } else if (failure === 'unverified') { + console.log( + JSON.stringify({ + ARN: arn, + VersionId: 'unexpected', + SecretString: request.SecretString, + }) + ) + } else { + console.log( + JSON.stringify({ ARN: arn, VersionId: request.ClientRequestToken }) + ) + } +} else if (args.includes('describe-instance-status')) { + console.log(JSON.stringify({ InstanceStatuses: [] })) +} else { + process.exit(2) +} diff --git a/crates/trusted-server-cli/tests/pbs_cli.rs b/crates/trusted-server-cli/tests/pbs_cli.rs new file mode 100644 index 000000000..07536965a --- /dev/null +++ b/crates/trusted-server-cli/tests/pbs_cli.rs @@ -0,0 +1,432 @@ +//! Exercise the actual CLI and process adapter against a local fake AWS executable. +#![cfg(unix)] + +use std::fs; +use std::os::unix::fs::PermissionsExt; +use std::path::Path; +use std::process::{Command, Output}; + +use serde_json::{Value, json}; +use tempfile::TempDir; + +const TOKEN: &str = "11111111-2222-4333-8444-555555555555"; + +fn fixture() -> TempDir { + assert!( + which::which("node").is_ok(), + "pbs_cli tests need the pinned Node.js on PATH for the fake AWS executable" + ); + let dir = tempfile::tempdir().expect("should create fixture directory"); + let source = Path::new(env!("CARGO_MANIFEST_DIR")).join("examples/pbs"); + for name in ["deployment.yaml", "pbs.yaml", "east.yaml", "bindings.json"] { + fs::copy(source.join(name), dir.path().join(name)).expect("should copy fixture"); + } + let fake = dir.path().join("aws"); + fs::write(&fake, include_str!("fixtures/pbs-aws.cjs")) + .expect("should write fake AWS executable"); + fs::set_permissions(fake, fs::Permissions::from_mode(0o700)) + .expect("should set executable mode"); + dir +} + +fn command(dir: &Path) -> Command { + let mut command = Command::new(env!("CARGO_BIN_EXE_ts")); + command + .current_dir(dir) + .env("PBS_FAKE_ROOT", dir) + .env( + "PATH", + format!( + "{}:{}", + dir.display(), + std::env::var("PATH").unwrap_or_default() + ), + ) + .env_remove("AWS_ACCESS_KEY_ID") + .env_remove("AWS_SECRET_ACCESS_KEY") + .env_remove("AWS_SESSION_TOKEN") + .env_remove("NODE_OPTIONS") + .env_remove("NODE_PATH") + .env("AWS_EC2_METADATA_DISABLED", "true"); + command +} + +fn secret_command(dir: &Path) -> Command { + let mut command = command(dir); + command.args([ + "prebid", + "server", + "secrets", + "set", + "examplebidder", + "--deployment", + "deployment.yaml", + "--region", + "us-east-1", + "--file", + "secret.json", + "--yes", + "--request-token", + TOKEN, + "--json", + ]); + command +} + +fn assert_no_secret(output: &Output) { + assert!(!String::from_utf8_lossy(&output.stdout).contains("DUMMY_SECRET")); + assert!(!String::from_utf8_lossy(&output.stderr).contains("DUMMY_SECRET")); +} + +fn assert_payload_cleanup(dir: &Path) { + let paths = + fs::read_to_string(dir.join("payload_paths")).expect("should capture payload paths"); + for path in paths.lines() { + assert!( + !Path::new(path).exists(), + "should remove tool-created request payloads" + ); + } +} + +#[test] +fn local_commands_never_execute_aws_and_preserve_the_source() { + let dir = fixture(); + let toml = "[auction]\nenabled=true\n[auction.providers.pbs-main]\nprofile='prebid-server'\nendpoint='https://DUMMY_SECRET@pbs.example.com/openrtb2/auction'\n[auction.bidders.examplebidder]\nprovider='pbs-main'\n[integrations.prebid]\nenabled=false\naccount_id='DUMMY_SECRET'\n[integrations.prebid.bundle.modules]\nbidder=['exampleBidAdapter']\nuser_id=['sharedIdSystem']\nanalytics=['exampleAnalyticsAdapter']\n"; + fs::write(dir.path().join("trusted-server.toml"), toml).expect("should write TOML"); + let output = command(dir.path()) + .args(["prebid", "server", "inspect", "--json"]) + .output() + .expect("should run CLI"); + assert!(output.status.success()); + assert_no_secret(&output); + let report: Value = serde_json::from_slice(&output.stdout).expect("should emit JSON"); + assert_eq!(report["enabled_explicit"], false); + assert_eq!(report["bundle_adapters"], json!(["exampleBidAdapter"])); + assert_eq!(report["identity_modules"], json!(["sharedIdSystem"])); + assert_eq!( + report["analytics_modules"], + json!(["exampleAnalyticsAdapter"]) + ); + assert_eq!(report["server_providers"][0]["provider"], "pbs-main"); + assert_eq!( + report["server_providers"][0]["server_bidder_candidates"][0]["bidder"], + "examplebidder" + ); + assert_eq!( + fs::read_to_string(dir.path().join("trusted-server.toml")).expect("should read TOML"), + toml + ); + let output = command(dir.path()) + .args([ + "prebid", + "server", + "check", + "--deployment", + "deployment.yaml", + "--json", + ]) + .output() + .expect("should run checks"); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + assert!( + !dir.path().join("calls").exists(), + "local commands must not invoke AWS" + ); +} + +#[test] +fn deeply_nested_yaml_is_rejected_without_aborting_or_echoing_input() { + let flow_sequence = format!( + "nested: {}NEVER_PRINT_ME{}", + "[".repeat(1000), + "]".repeat(1000) + ); + let flow_mapping = format!( + "{}\"NEVER_PRINT_ME\"{}", + "{\"nested\":".repeat(1000), + "}".repeat(1000) + ); + let mut block_mapping = (0..256) + .map(|depth| format!("{}nested:\n", " ".repeat(depth))) + .collect::(); + block_mapping.push_str(&format!("{}value: NEVER_PRINT_ME\n", " ".repeat(256))); + for yaml in [flow_sequence, flow_mapping, block_mapping] { + let dir = fixture(); + fs::write(dir.path().join("pbs.yaml"), yaml).expect("should write nested input"); + let output = command(dir.path()) + .args([ + "prebid", + "server", + "check", + "--deployment", + "deployment.yaml", + "--json", + ]) + .output() + .expect("should run CLI"); + assert_eq!( + output.status.code(), + Some(2), + "should reject input without a signal" + ); + assert!(output.stdout.is_empty()); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!(stderr.contains("cannot parse YAML/JSON input; source details withheld")); + assert!(!stderr.contains("NEVER_PRINT_ME")); + assert!(!dir.path().join("calls").exists(), "should not call AWS"); + } +} + +#[test] +fn missing_descriptor_inputs_report_escaped_paths() { + for missing in ["deployment.yaml", "pbs.yaml", "bindings.json", "east.yaml"] { + let dir = fixture(); + let inputs = dir.path().join("inputs\n\t\u{1b}[31m\"\\"); + fs::create_dir(&inputs).expect("should create nested input directory"); + for name in ["deployment.yaml", "pbs.yaml", "bindings.json", "east.yaml"] { + if name != missing { + fs::copy(dir.path().join(name), inputs.join(name)).expect("should copy input"); + } + } + let path = inputs.join(missing); + let output = command(dir.path()) + .args(["prebid", "server", "check", "--deployment"]) + .arg(inputs.join("deployment.yaml")) + .output() + .expect("should run CLI"); + assert_missing_path(&output, &path); + assert!(!dir.path().join("calls").exists(), "should not call AWS"); + } +} + +#[test] +fn missing_secret_input_reports_escaped_path_without_writing() { + let dir = fixture(); + let path = dir.path().join("missing\n\t\u{1b}[31m\"\\secret.json"); + let output = command(dir.path()) + .args([ + "prebid", + "server", + "secrets", + "set", + "examplebidder", + "--deployment", + "deployment.yaml", + "--region", + "us-east-1", + "--yes", + "--request-token", + TOKEN, + "--file", + ]) + .arg(&path) + .output() + .expect("should run CLI"); + assert_missing_path(&output, &path); + let calls = fs::read_to_string(dir.path().join("calls")).expect("should read calls"); + assert!(!calls.contains("put-secret-value")); + assert_payload_cleanup(dir.path()); +} + +fn assert_missing_path(output: &Output, path: &Path) { + assert_eq!(output.status.code(), Some(2)); + assert!(output.stdout.is_empty()); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + stderr.contains(&format!("cannot open input file {path:?}")), + "{stderr}" + ); + assert!( + !stderr.contains(&*path.to_string_lossy()), + "should escape control characters" + ); + assert!(!stderr.contains("os error")); + assert!(!stderr.contains('\u{1b}')); + assert!(!stderr.contains('\t')); + assert_no_secret(output); +} + +#[test] +fn secret_payload_is_private_not_in_argv_and_deleted_after_use() { + let dir = fixture(); + let input = json!({"api_key": "DUMMY_SECRET-$\"\nvalue"}).to_string(); + fs::write(dir.path().join("secret.json"), &input).expect("should write dummy credential"); + let output = secret_command(dir.path()) + .output() + .expect("should execute CLI"); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + assert_no_secret(&output); + let calls = + fs::read_to_string(dir.path().join("calls")).expect("should read captured arguments"); + assert!( + !calls.contains("DUMMY_SECRET"), + "secret values must not appear in arguments" + ); + let request: Value = serde_json::from_str( + &fs::read_to_string(dir.path().join("captured_request.json")) + .expect("should read dummy request"), + ) + .expect("should parse request"); + assert_eq!(request["ClientRequestToken"], TOKEN); + let secret: Value = serde_json::from_str( + request["SecretString"] + .as_str() + .expect("should have string payload"), + ) + .expect("should parse payload"); + assert_eq!( + secret, + serde_json::from_str::(&input).expect("should parse fixture") + ); + assert_eq!( + fs::read_to_string(dir.path().join("secret.json")).expect("should preserve input"), + input + ); + assert_payload_cleanup(dir.path()); +} + +#[test] +fn unset_cli_history_allows_write_after_both_history_probes() { + let dir = fixture(); + fs::write( + dir.path().join("secret.json"), + "{\"api_key\":\"DUMMY_SECRET\"}", + ) + .expect("should write dummy credential"); + let output = secret_command(dir.path()) + .env("PBS_FAKE_HISTORY", "unset") + .output() + .expect("should run CLI"); + assert!( + output.status.success(), + "{}", + String::from_utf8_lossy(&output.stderr) + ); + assert_no_secret(&output); + let calls = fs::read_to_string(dir.path().join("calls")).expect("should read calls"); + let calls = calls + .lines() + .map(|line| serde_json::from_str::>(line).expect("should parse arguments")) + .collect::>(); + let probes = calls + .iter() + .filter(|args| args.iter().any(|arg| arg == "configure")) + .map(|args| { + args.last() + .expect("should have a configuration key") + .as_str() + }) + .collect::>(); + assert_eq!(probes, ["cli_history", "default.cli_history"]); + assert_eq!( + calls + .iter() + .filter(|args| args.iter().any(|arg| arg == "put-secret-value")) + .count(), + 1 + ); + assert_payload_cleanup(dir.path()); +} + +#[test] +fn aws_errors_never_forward_provider_stderr_and_cleanup_payloads() { + for failure in ["collision", "transport", "invalid-json", "unverified"] { + let dir = fixture(); + let input = "{\"api_key\":\"DUMMY_SECRET\"}"; + fs::write(dir.path().join("secret.json"), input).expect("should write fixture"); + let output = secret_command(dir.path()) + .env("PBS_FAKE_FAILURE", failure) + .output() + .expect("should run CLI"); + assert_eq!(output.status.code(), Some(2)); + assert_no_secret(&output); + let stderr = String::from_utf8_lossy(&output.stderr); + for guidance in [ + "write not confirmed", + "uncertain", + "retain the request token", + "reuse it only for the original identical payload", + "new token for separately intended changed values", + "provider output withheld", + TOKEN, + ] { + assert!( + stderr.contains(guidance), + "{failure}: missing {guidance}: {stderr}" + ); + } + assert!(!stderr.contains("ResourceExistsException")); + assert!(!stderr.contains("Connection reset by peer")); + assert!(!stderr.contains("unexpected")); + assert!(output.stdout.is_empty()); + let calls = fs::read_to_string(dir.path().join("calls")).expect("should read calls"); + assert_eq!( + calls.matches("put-secret-value").count(), + 1, + "should not retry" + ); + assert!(!calls.contains("DUMMY_SECRET")); + assert_eq!( + fs::read_to_string(dir.path().join("secret.json")).expect("should read input"), + input + ); + assert_payload_cleanup(dir.path()); + } +} + +#[test] +fn enabled_cli_history_and_wrong_accounts_block_writes() { + for (setting, value) in [ + ("PBS_FAKE_HISTORY", "enabled"), + ("PBS_FAKE_ACCOUNT", "999999999999"), + ] { + let dir = fixture(); + fs::write( + dir.path().join("secret.json"), + "{\"api_key\":\"DUMMY_SECRET\"}", + ) + .expect("should write fixture"); + let output = secret_command(dir.path()) + .env(setting, value) + .output() + .expect("should run CLI"); + assert_eq!(output.status.code(), Some(2)); + assert_no_secret(&output); + let calls = fs::read_to_string(dir.path().join("calls")).expect("should read calls"); + assert!(!calls.contains("put-secret-value")); + assert!(!dir.path().join("captured_request.json").exists()); + assert_payload_cleanup(dir.path()); + } +} + +#[test] +fn incomplete_status_returns_json_and_nonzero_exit() { + let dir = fixture(); + let output = command(dir.path()) + .args([ + "prebid", + "server", + "status", + "--deployment", + "deployment.yaml", + "--json", + ]) + .output() + .expect("should run status"); + assert_eq!(output.status.code(), Some(2)); + let report: Value = serde_json::from_slice(&output.stdout).expect("should emit partial JSON"); + assert_eq!(report["complete"], false); + assert_eq!( + report["regions"][0]["instances"][0]["pbs_health"], + "unknown" + ); +} diff --git a/crates/trusted-server-core/src/integrations/prebid.rs b/crates/trusted-server-core/src/integrations/prebid.rs index a031c2554..f1d1d0b50 100644 --- a/crates/trusted-server-core/src/integrations/prebid.rs +++ b/crates/trusted-server-core/src/integrations/prebid.rs @@ -495,12 +495,12 @@ impl IntegrationConfig for LegacyPrebidServerConfig { #[derive(Debug, Clone, Default, Deserialize, Serialize)] #[serde(deny_unknown_fields)] pub struct PrebidBundleBuildConfig { - /// Typed Prebid.js module selections consumed by `ts prebid bundle`. + /// Typed Prebid.js module selections consumed by `ts prebid client`. #[serde(default)] pub modules: PrebidBundleModulesConfig, } -/// Exact Prebid.js module stems selected by `ts prebid bundle`. +/// Exact Prebid.js module stems selected by `ts prebid client`. /// /// The CLI validates these values. The runtime only parses them so app config /// carrying build inputs remains loadable. diff --git a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts index 2ca2fe6b1..32fabe43d 100644 --- a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts @@ -321,7 +321,7 @@ function managedUserIdEntry(managed: InjectedManagedUserId): PrebidUserIdConfigE * Drops managed User ID entries that address a submodule an earlier entry * already claimed. * - * `ts prebid bundle` rejects such a pair, but an operator running a prebuilt + * `ts prebid client` rejects such a pair, but an operator running a prebuilt * external bundle never invokes it, and core's duplicate check compares names * rather than the submodules they resolve to. Prebid registers one submodule * for a module's name and each of its aliases and reads only the first matching @@ -2610,7 +2610,7 @@ export function installPrebidNpm(config?: Partial): typeof pbjs `[tsjs-prebid] client-side bidder "${bidder}" has no adapter in the external ` + 'Prebid bundle. Add its exact Prebid module stem to ' + '[integrations.prebid.bundle.modules].bidder in trusted-server.toml and ' + - 'rebuild it with `ts prebid bundle`.' + 'rebuild it with `ts prebid client`.' ); } } diff --git a/deploy/pbs-example/.gitignore b/deploy/pbs-example/.gitignore new file mode 100644 index 000000000..bf80e0040 --- /dev/null +++ b/deploy/pbs-example/.gitignore @@ -0,0 +1,18 @@ +# Terraform working data and local state +.terraform/ +*.tfstate +*.tfstate.* +.terraform.tfstate.lock.info + +# Saved plans can contain provider-returned metadata +*.tfplan + +# Generated operator inputs +/deployment.generated.yaml +/runtime/secret-bindings.generated.json + +# Local inputs and crash output +*.auto.tfvars +*.tfvars +crash.log +crash.*.log diff --git a/deploy/pbs-example/.terraform.lock.hcl b/deploy/pbs-example/.terraform.lock.hcl new file mode 100644 index 000000000..4f85daae2 --- /dev/null +++ b/deploy/pbs-example/.terraform.lock.hcl @@ -0,0 +1,29 @@ +# This file is maintained automatically by "terraform init". +# Manual edits may be lost in future updates. + +provider "registry.terraform.io/hashicorp/aws" { + version = "6.64.0" + constraints = "6.64.0" + hashes = [ + "h1:2fTLxzUDmp/KVIHbIeLTB4bIzWHx8E6Dw+1ALLUi+Yw=", + "h1:4siTahLyzGh4BoMQcL7VXeL/mn8iR/zKv93NyhQob+0=", + "h1:EEWCXlg69fty/Qi+kehrREnVEaWdKnLlVon542b6nxQ=", + "h1:wXARLY+IeQ7ufYxCLTPCwToWGMRvOpiOTfJS97iwUzI=", + "zh:07172315d67bc9781240272759cdfc7bd32b7e72384a56862c2c1da3cca99a81", + "zh:154ce7d2659de9a59ddfe96d7cab41a9ddc2cb267a7d4bcdf4e737ff2ffdec06", + "zh:17324d4335a7a7ac01cc23eded530775606680ff53b47cb74a3cb95d1121f836", + "zh:307ab92324ec5a61b124881ab8cac1d9e316f4527dfd0e1b59794c229407eb4e", + "zh:31e25f1903661332e36a95283042dd3ec50b47c186db00663fbd976a11e6a6b2", + "zh:3311d9f3bd12a24886027dbe73859dcd1e67bd0e3046227a338cf2c7ca04d18e", + "zh:37916156a3aac3b29be3acebd15d53145ea4ab5d4aaa825eaebe75481fa00500", + "zh:4158cb8c38b3ac6aa98eb15935ec6bd7c30838d85d2b00acc9812df8382ae908", + "zh:5bfb9499c66d9db5b34dc5c60f426a1ab1baa5457ce2aefebca826a9c3f92fb0", + "zh:6eb29ead5a4aca3b1f35812e7e8c75419180e1928e479b458f206861277736db", + "zh:7a82b6dd0c0cdef8045a4adfbddd36acb86b6b23fcbed8e189c2d71f7dc4a502", + "zh:9556bd792032c3f7e73ea4dd08cec88dc1327f5a4a57d79c30ba844ae2b9a3c0", + "zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425", + "zh:c5234180464cb800c83a41f57462742b802c150ad7d4417626fcd9cb511c01d2", + "zh:cd776b83b1f7b36635957350afe7ce28ba4e4ea3a5e2deb00d13dbd3b35d9d40", + "zh:fb583a7b791c6f915b86573d04f05ddbf7f1a5e4120c5d8a7450a3086c1225c4", + ] +} diff --git a/deploy/pbs-example/DEPLOYMENT_PLAN.md b/deploy/pbs-example/DEPLOYMENT_PLAN.md new file mode 100644 index 000000000..839b5836e --- /dev/null +++ b/deploy/pbs-example/DEPLOYMENT_PLAN.md @@ -0,0 +1,134 @@ +# Prebid Server AWS example deployment plan + +Status: draft generated files, locally checked after validation. This directory is an example only. It has not been planned against a live AWS account, applied, deployed, or load tested. + +## Decision record + +| Requirement | Value | Status | Evidence or decision owner | Blocks | +| --------------------- | -------------------------------------------------------------------------------------------- | --------------------------- | --------------------------------------------------------- | ------------------------------------------------------------------ | +| Purpose | Production-shaped architecture example, not a live deployment | Confirmed | User approval | None | +| Caller | Trusted Server | Confirmed | User | Caller egress ranges remain required | +| Regions | `us-east-1` and `us-west-2` | Confirmed | User | Account-specific AZ selection remains unresolved | +| Regional topology | Two AZs, one PBS EC2 host per AZ | Confirmed | User | AMIs and subnets remain required | +| Total PBS hosts | Four | Confirmed | Derived from topology | Capacity remains unmeasured | +| Runtime | EC2 with Docker Compose | Confirmed | User | Release and secret injection are deferred implementation work | +| Regional ingress | Public HTTPS ALB | Confirmed | User | ACM certificate ARNs and caller CIDRs required | +| Global routing | Route 53 latency aliases with ALB health evaluation | Proposed | Architecture decision | Existing hosted-zone ID required | +| WAF | Not included in this demo | Confirmed | User | Abuse controls remain outside this example | +| Terraform state | Local backend | Confirmed | User | No team locking or remote recovery | +| PBS release | Go v4.7.0, digest pinned | Proposed and verified | PBS release and Docker metadata checked during generation | Recheck before any future use | +| Workload | 200 global peak auctions/s, 4 bidders, 1.5 s caller timeout, 1 s PBS timeout | Proposed example assumption | Planning assumption | No capacity claim until load tested | +| Secrets | Independent regional Secrets Manager metadata and EC2 read policy; values written separately | Confirmed | User and repository CLI contract | Real bidder mapping, authorization, and runtime injection required | +| Trusted Server config | No authoritative `trusted-server.toml` exists in this checkout | Confirmed | Repository inspection | Caller behavior remains an external input | +| Deployment directory | `deploy/pbs-example/` | Confirmed | User | None | + +## Architecture + +```mermaid +flowchart TD + T["Trusted Server"] --> R["Route 53 latency aliases"] + R --> E["us-east-1 ALB"] + R --> W["us-west-2 ALB"] + E --> E1["East private PBS host, AZ 1"] + E --> E2["East private PBS host, AZ 2"] + W --> W1["West private PBS host, AZ 1"] + W --> W2["West private PBS host, AZ 2"] + E1 --> B["Bidder endpoints"] + E2 --> B + W1 --> B + W2 --> B +``` + +Each region has a public ALB in two public subnets and one private PBS host in each AZ. Each private subnet has its own NAT gateway so normal outbound traffic remains in the AZ and can use a stable Elastic IP for bidder allowlists. Route 53 returns the lowest-latency healthy regional ALB. DNS caching means failover is not immediate. + +The EC2 AMIs are inputs rather than built by Terraform. They must contain the approved OS, SSM agent, Docker, and Compose. The generated Terraform does not install packages or deploy PBS through user data. Runtime release delivery, configuration rendering, secret injection, restart, and rollback need a separately approved implementation. + +## Services and ownership + +| Service or artifact | Action | Owner | Purpose | +| ----------------------------------------------- | ------------------------------ | --------------------------------- | -------------------------------------------------------------------- | +| VPC, subnets, routes, IGW, NAT, security groups | Create | Terraform | Regional network and egress | +| ALB, target group, HTTPS listener | Create | Terraform | Regional HTTPS ingress and health routing | +| Route 53 records | Reuse zone, create records | Terraform | Latency-based regional selection | +| EC2 instances and IAM profiles | Create | Terraform | Compose hosts and SSM access | +| Secrets Manager secret metadata | Create | Terraform | One independently encrypted secret per region for the example bidder | +| Secret values | External write | Authorized operator or automation | Credential lifecycle; never Terraform | +| PBS YAML and Compose definition | Git-owned | Runtime owner | Nonsecret runtime contract | +| Regional YAML merge and rendered file | Deferred runtime release owner | Deployment implementation | One resolved config per region | +| PBS process lifecycle | Deferred runtime release owner | Deployment implementation | Start, replace, health, rollback | +| Terraform state | Local operator | Terraform | Demo-only state; no shared locking | + +## Capacity assumptions + +The example uses these fictional planning values: + +- 200 peak eligible auctions per second globally. +- 100 QPS per region during normal routing. +- Either region must be able to receive 200 QPS during regional failover. +- Four bidder requests per auction, or 800 outbound requests per second globally. +- 16 KiB average request and 32 KiB average response. +- Trusted Server timeout of 1.5 seconds and PBS auction timeout of 1 second. +- `c7i.large` is an initial instance-type example, not a throughput guarantee. + +A controlled load test must measure CPU, memory, connection reuse, outbound bandwidth, bidder tail latency, ALB target health, and PBS queueing before changing these values. The four-host count does not prove capacity or availability. + +## Failure boundaries and limitations + +- Loss of one host leaves the other regional target serving traffic. The ALB health check must drain the failed target. +- Loss of one AZ removes that region's one target and leaves the region degraded. A later scale-out would be required for full AZ redundancy. +- Loss of a whole region depends on Route 53 resolver behavior and surviving-region capacity. +- NAT, ALB, and EC2 resources have separate regional failure and cost boundaries. +- Local state has no remote locking or shared recovery. Only one operator may use it. +- No WAF or rate-based abuse control is included. +- The example uses a fictional bidder binding. The adapter name, credential keys, source authorization, and PBS mapping must be replaced and verified against v4.7.0 before use. +- A Secrets Manager write does not refresh a running Compose container. The deferred runtime implementation must retrieve a selected version, render an environment file atomically, replace the consumer, and verify health before retiring the old version. +- Customer-managed KMS keys are regional inputs. East and West accept separate ARNs; null uses each region's Secrets Manager service key. +- The regional module fixes the PBS host port at `8000`, matching `runtime/compose.yaml` and `runtime/pbs.yaml`. A port change requires coordinated Terraform and runtime edits; it is not a deployment input. +- The committed descriptor and binding files are fictional local fixtures. After an authorized apply, Terraform renders ignored operator files containing the actual instance IDs and secret ARNs. + +## Deferred security and observability decisions + +ALB access logging is deliberately deferred. This example has no request-level ALB forensic record. Before production use, the platform owner must propose the log bucket, delivery permissions, encryption, access controls, and deletion lifecycle. The privacy owner must approve which request metadata may be retained, its retention period, and who may query it. The runtime owner must define incident retrieval and verify delivery. No log bucket or access-log policy is provisioned here. + +AWS API traffic, including SSM and Secrets Manager requests and any direct KMS requests, uses public service endpoints through each AZ's NAT gateway over TLS. Private hosts do not imply a private AWS API path. The platform and security owners must decide whether regional interface VPC endpoints are required, including endpoint policies, security groups, and private DNS. Compare endpoint hourly and data-processing charges against NAT charges and the required security boundary. Endpoints for AWS APIs would not replace public bidder egress; this example adds none. + +The AWS-managed `AmazonSSMManagedInstanceCore` attachment is the deliberate broad IAM exception. Its SSM agent permissions include `Resource: "*"`, unlike the inline policy scoped to declared bidder secret ARNs and the optional regional KMS key. The security owner must review the managed policy and its future updates before deployment. Do not describe the entire instance role as resource-scoped. + +## Cost drivers + +No price estimate is claimed. The main drivers are four EC2 instances, four NAT gateways and their Elastic IPs, two ALBs, public IPv4 addresses, cross-AZ traffic if routing changes, CloudWatch alarms, Secrets Manager, Route 53 records, and bidder internet traffic. A current estimate requires selected regions, traffic volume, and current AWS pricing verification. + +## Generated files and checks + +The `PBS example checks` workflow runs Terraform format/validate and both mocked suites, JSON parsing, shell syntax, and Compose wiring checks on changes to this directory, `.tool-versions`, or the workflow. Both provider locks are checked without updates. CI uses the Node.js version in `.tool-versions` and Docker Compose, but no npm dependencies, AWS credentials, or container startup. The example maintainer owns these checks. The CLI test suite separately validates the committed deployment descriptor and its referenced inputs on every PR. The runtime owner must still run the CLI descriptor check when adapting inputs and obtain separate approval for the real startup smoke and cloud verification. A green mock suite is not deployment evidence. + +| Path | Consumer | Local check | +| ----------------------------------------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `README.md` | Example user | Safe walkthrough and stop boundary review | +| `terraform.tf`, `providers.tf`, `variables.tf`, `dns.tf`, `locals.tf`, `outputs.tf` | Terraform | Format and validate; review generated descriptor and binding outputs | +| `main.tf`, `modules/regional/` | Terraform | Mocked plan tests for topology, provider mappings, IMDSv2, encrypted root volumes, private subnet/AZ placement, TLS policy, scoped ALB ingress/egress, and secret access | +| `tests/root_unit_test.tftest.hcl` | Example maintainer | CI mocked root plan tests for regional wiring, generated CLI inputs, and invalid deployment inputs | +| `terraform.tfvars.example` | Authorized operator | Local mocked plan tests; review fictional inputs before adapting | +| `.gitignore` | Example maintainer | Verify state, saved plans, local inputs, and generated operator files remain ignored | +| `runtime/pbs.yaml`, `runtime/regions/` | PBS release process and CLI check | YAML parse and `ts prebid server check` | +| `runtime/compose.yaml` | Deferred EC2 runtime owner | Compose rendering and fake-command smoke wiring; separately approved pinned-image startup smoke | +| `scripts/smoke-runtime.sh` | Runtime owner | CI shell syntax and fake-command wiring; real startup smoke requires separate authorization | +| `scripts/test-smoke-runtime.mjs` | Example maintainer | Pinned Node.js, JSON parsing, and Docker Compose rendering; no image pull or containers | +| Root and regional `.terraform.lock.hcl` | Example maintainer | Readonly initialization and tests; regenerate both locks for the four supported platforms on provider upgrades | +| `runtime/secret-bindings.example.json` | Existing PBS CLI | JSON parse and fictional deployment check | +| `deployment.example.yaml` | Existing PBS CLI | CLI regression against committed inputs and local `ts prebid server check` | +| Terraform-rendered generated descriptor and bindings | Authorized operator | Review actual IDs, then CLI check and status | +| `DEPLOYMENT_PLAN.md` | Reviewers | Diff and decision review | +| `RUNBOOK.md` | Authorized operator | Procedure review; no cloud execution | + +## Sources and verification + +- PBS Go v4.7.0 release: https://github.com/prebid/prebid-server/releases +- PBS v4.7.0 configuration guide: https://github.com/prebid/prebid-server/blob/v4.7.0/docs/developers/configuration.md +- PBS v4.7.0 configuration definitions: https://github.com/prebid/prebid-server/blob/v4.7.0/config/config.go +- PBS v4.7.0 Docker metadata: https://hub.docker.com/v2/repositories/prebid/prebid-server/tags/v4.7.0 +- ALB requirements: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/application-load-balancers.html +- Route 53 latency routing: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-latency.html +- Secrets Manager value updates: https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/put-secret-value.html +- Terraform S3 backend guidance, not selected for this demo: https://developer.hashicorp.com/terraform/language/backend/s3 +- HashiCorp Terraform style guide revision consulted: `c2d65dfe492f74d360d35b859b88932222470bd8` diff --git a/deploy/pbs-example/README.md b/deploy/pbs-example/README.md new file mode 100644 index 000000000..1fe071669 --- /dev/null +++ b/deploy/pbs-example/README.md @@ -0,0 +1,83 @@ +# PBS AWS example + +This directory is the locally checked output of the Prebid Server AWS planning workflow. It is a reference for preparing infrastructure and operator inputs. It does not deploy PBS, inject runtime secrets, or change traffic. + +## What it models + +```mermaid +flowchart TD + T["Trusted Server"] --> R["Route 53 latency aliases"] + R --> E["us-east-1 ALB"] + R --> W["us-west-2 ALB"] + E --> E1["Private PBS host, AZ 1"] + E --> E2["Private PBS host, AZ 2"] + W --> W1["Private PBS host, AZ 1"] + W --> W2["Private PBS host, AZ 2"] + E1 --> B["Public bidder endpoints"] + E2 --> B + W1 --> B + W2 --> B +``` + +Each region has two AZs, one EC2 host per AZ, a public ALB, and one NAT gateway per AZ. Terraform creates regional Secrets Manager metadata but never stores credential values. + +## Safe local walkthrough + +These commands do not contact AWS. They require the pinned Terraform, Rust, and Node.js toolchains and Docker with the Compose plugin. The Node.js check uses only built-in modules; no npm install is needed. On Apple Silicon macOS, replace `cargo run_cli_linux` with `cargo run_cli_macos`. Other hosts can use `cargo run --package trusted-server-cli --target "$(rustc -vV | awk '/host:/ { print $2 }')" --`. + +```bash +terraform fmt -check -recursive deploy/pbs-example +terraform -chdir=deploy/pbs-example init -backend=false -input=false -lockfile=readonly +terraform -chdir=deploy/pbs-example validate +terraform -chdir=deploy/pbs-example test -filter=tests/root_unit_test.tftest.hcl +terraform -chdir=deploy/pbs-example/modules/regional init -backend=false -input=false -lockfile=readonly +terraform -chdir=deploy/pbs-example/modules/regional validate +terraform -chdir=deploy/pbs-example/modules/regional test -filter=tests/security_unit_test.tftest.hcl +cargo run_cli_linux prebid server check \ + --deployment deploy/pbs-example/deployment.example.yaml \ + --json +docker compose \ + --env-file deploy/pbs-example/runtime/examples/compose.env \ + -f deploy/pbs-example/runtime/compose.yaml \ + config --quiet +bash -n deploy/pbs-example/scripts/smoke-runtime.sh +env -u NODE_OPTIONS -u NODE_PATH node deploy/pbs-example/scripts/test-smoke-runtime.mjs +``` + +The `PBS example checks` GitHub Actions workflow runs Terraform formatting, both readonly-lock initializations, validation, and both mocked test suites. It also checks JSON, shell syntax, and Compose wiring. Changes to this example, `.tool-versions`, or the workflow trigger it. CI reads the Node.js version from `.tool-versions` and checks Docker Compose availability. The wiring command clears inherited Node.js preload options and module paths, and the script passes only `PATH` and `HOME` plus its explicit test inputs to subprocesses. It also parses `runtime/secret-bindings.example.json`. The CLI test suite separately loads and checks the committed deployment descriptor and its referenced inputs on every PR. The example owner must still run the descriptor command above when adapting inputs. + +The Terraform tests use mocked AWS providers and explicit plan mode. The deployment descriptor and binding file contain fictional account, profile, secret, instance, and adapter identifiers for local validation only. Their pinned image digest identifies the upstream PBS Go v4.7.0 release, not a fictional image; preserve it unless deliberately selecting and verifying another release, and keep both files' image references identical. The wiring test renders Compose JSON and exercises the smoke script with fake lifecycle and health commands. It verifies the deployment all-interface binding, smoke-only loopback binding, and forced dummy input selectors without starting containers. + +A separately approved `deploy/pbs-example/scripts/smoke-runtime.sh` run pulls and starts the pinned image with dummy values on loopback, checks `/status` and startup-log redaction, and sends no auction request. Static rendering and fake-command checks are not runtime startup evidence. + +## Adapting the example + +1. Copy this directory into an environment-specific deployment root. +2. Replace the fictional values from `terraform.tfvars.example` in a protected, ignored `.tfvars` file. +3. Replace `examplebidder` with adapters and credential mappings verified against the pinned PBS release and approved by each bidder. +4. Supply approved regional AMIs. Each AMI must contain the chosen OS, SSM agent, Docker, and Compose. This repository does not build the AMIs. +5. Run the local checks above. +6. Obtain separate authorization for AWS identity checks, Terraform plan, apply, and secret writes. +7. Follow `RUNBOOK.md` for the reviewed plan and apply process. + +## Generating real CLI inputs after apply + +A successful apply produces actual instance IDs and regional secret ARNs. Render ignored operator files from those outputs instead of editing the fictional fixtures: + +```bash +terraform output -raw deployment_descriptor_yaml > deployment.generated.yaml +terraform output -raw secret_bindings_json > runtime/secret-bindings.generated.json + +ts prebid server check --deployment deployment.generated.yaml +ts prebid server status --deployment deployment.generated.yaml +``` + +Run these commands from this directory. `status` performs authenticated EC2 reads and requires separate authorization. Review both generated files before using them. They contain resource identifiers but no credential values. + +An authorized operator can then write a complete regional secret value with `ts prebid server secrets set`, as documented in `RUNBOOK.md`. + +## Stop boundary + +The current workflow stops after infrastructure preparation and secret-value writes. It does not install a secret loader, resolve regional PBS YAML onto hosts, start or replace Compose, verify application health, or roll back a release. Those operations require a separately approved runtime implementation. A secret write alone does not update a running container. + +See `DEPLOYMENT_PLAN.md` for decisions and limitations, and `RUNBOOK.md` for operator contracts. diff --git a/deploy/pbs-example/RUNBOOK.md b/deploy/pbs-example/RUNBOOK.md new file mode 100644 index 000000000..b759f4c45 --- /dev/null +++ b/deploy/pbs-example/RUNBOOK.md @@ -0,0 +1,146 @@ +# PBS example runbook + +This runbook describes local checks and the separately authorized operations that would be needed for a real deployment. It does not authorize AWS access, Terraform apply, secret writes, traffic changes, or runtime deployment. + +## Preconditions + +- Work only in `deploy/pbs-example`. +- Use Terraform `1.16.2` and the committed AWS provider lock files in both the root and regional module. Both lock AWS `6.64.0` for Linux and macOS on AMD64 and ARM64. +- Local wiring checks require the Node.js version in `.tool-versions` and Docker with the Compose plugin. The check uses only built-in Node.js modules; no npm install is needed. +- Use fictional values from `terraform.tfvars.example`, `deployment.example.yaml`, and `runtime/secret-bindings.example.json` only for local validation. +- Replace the fictional AWS account, profile, certificate, hosted-zone, AMI, and CIDR inputs before any authorized cloud plan. Terraform renders actual instance and secret identifiers after apply. +- Confirm the approved Trusted Server egress CIDRs. Do not use the documentation CIDR as a real allowlist. +- Confirm the PBS v4.7.0 image digest and every adapter binding again before release preparation. +- Keep state, saved plans, credentials, and AWS provider output out of Git and public logs. Input-file errors echo the full escaped path to stderr, including under `--json`; directory layouts and partner names in paths can therefore enter logs even though credential contents are withheld. + +## Local validation + +From the repository root: + +```bash +terraform fmt -check -recursive deploy/pbs-example +terraform -chdir=deploy/pbs-example init -backend=false -input=false -lockfile=readonly +terraform -chdir=deploy/pbs-example validate +terraform -chdir=deploy/pbs-example test \ + -filter=tests/root_unit_test.tftest.hcl +terraform -chdir=deploy/pbs-example/modules/regional init \ + -backend=false -input=false -lockfile=readonly +terraform -chdir=deploy/pbs-example/modules/regional validate +terraform -chdir=deploy/pbs-example/modules/regional test \ + -filter=tests/security_unit_test.tftest.hcl +``` + +`init -backend=false` downloads the locked provider but does not access the configured local state or AWS. Both selected test files use mocked AWS providers and explicit plan commands. Confirm that the root file runs five tests and the module file runs eight tests. Root assertions require distinct instance IDs in each regional descriptor. Module tests also cover resource-name boundaries and partition-specific SSM policy ARNs. Regional assertions cover IMDSv2, encrypted root volumes, private subnet/AZ placement, the TLS policy, scoped ALB ingress/egress, alarms, and secret access. `validate` and mocked tests do not prove AWS permissions, quotas, AMI existence, certificates, subnet availability, or capacity. + +The `PBS example checks` workflow runs these Terraform checks and the JSON, shell, and Compose wiring checks below. The CLI test suite separately validates the committed deployment descriptor and its referenced inputs on every PR. The example owner remains responsible for local validation of adapted inputs. Provider upgrades must refresh both locks with `terraform providers lock -platform=darwin_arm64 -platform=darwin_amd64 -platform=linux_amd64 -platform=linux_arm64` from each directory and rerun both suites. + +Check the PBS descriptor without AWS access. On Apple Silicon macOS, use `cargo run_cli_macos` instead of `cargo run_cli_linux`. On another host, use `cargo run --package trusted-server-cli --target "$(rustc -vV | awk '/host:/ { print $2 }')" --`: + +```bash +cargo run_cli_linux prebid server check \ + --deployment deploy/pbs-example/deployment.example.yaml \ + --json +``` + +Expected result: the descriptor, image, bindings, regions, and regional YAML merge pass local validation. This command does not retrieve secrets, contact AWS, pull the PBS image, or start PBS. + +Check Compose with dummy values: + +```bash +docker compose \ + --env-file deploy/pbs-example/runtime/examples/compose.env \ + -f deploy/pbs-example/runtime/compose.yaml \ + config --quiet +``` + +Expected result: Compose renders successfully without pulling or starting the image. The dummy environment file is not a credential. + +Validate the JSON input and smoke command wiring without starting containers: + +```bash +bash -n deploy/pbs-example/scripts/smoke-runtime.sh +env -u NODE_OPTIONS -u NODE_PATH node deploy/pbs-example/scripts/test-smoke-runtime.mjs +``` + +The wiring command clears inherited Node.js preload options and module paths. The script passes only `PATH` and `HOME` plus explicit test inputs to subprocesses. It parses the binding JSON, renders production and smoke configurations, uses fake Docker lifecycle and curl commands, and checks that inherited selectors cannot replace the dummy inputs. It does not pull images, start PBS, or prove runtime health. + +### Separately approved local runtime smoke + +```bash +deploy/pbs-example/scripts/smoke-runtime.sh +``` + +The smoke script pulls the pinned image if needed, starts it with dummy values bound only to `127.0.0.1:18080`, requires `/status` to return `ok`, rejects either dummy credential appearing in startup logs, and removes its container and network on exit. It sends no auction request and contacts no bidder. Set `PBS_SMOKE_PORT` only when port `18080` is unavailable. The script forces the checked-in baseline and dummy secret file regardless of inherited `PBS_CONFIG_FILE` or `PBS_SECRET_ENV_FILE` values. Shared deployment Compose still binds all host interfaces so the ALB can reach PBS; the PBS security group restricts ingress. + +## Authorized Terraform workflow + +These steps remain deferred. They require an approved AWS account, role, region scope, cost limit, and operator authorization. + +1. Replace fictional variables with approved values in a protected local tfvars file. +2. Recheck the AWS identity and account against the variables. +3. Run `terraform init` with the local backend and review any lock-file change. +4. Run `terraform plan -out=pbs-example.tfplan`. +5. Review ALB exposure, CIDR rules, NAT gateways, IAM secret access, Route 53 records, instance placement, and expected cost. +6. Store the saved plan privately with its checksum. Do not place it in Git or a public CI log. +7. Apply only the reviewed saved plan after a separate approval. + +Failure behavior: stop on wrong-account identity, an unexpected resource action, a certificate or zone mismatch, broad ingress, missing egress approval, or a secret policy broader than the declared bidder bindings. Reconcile the inputs and create a new plan. Never reuse a stale saved plan after source, state, credentials, or assumptions change. + +## Generate operator inputs + +After an authorized apply, run these commands from `deploy/pbs-example`: + +```bash +terraform output -raw deployment_descriptor_yaml > deployment.generated.yaml +terraform output -raw secret_bindings_json > runtime/secret-bindings.generated.json + +ts prebid server check --deployment deployment.generated.yaml --json +``` + +Inputs: the applied local state and unchanged Terraform source. Access: local state reads only; the CLI check makes no AWS call. Outputs: ignored, nonsecret files containing actual instance IDs and secret ARNs. + +Failure behavior: stop if either output is empty, references the wrong account or region, or fails the CLI check. Do not repair generated identifiers manually. Reconcile Terraform source and state, then render both files again. + +`ts prebid server status --deployment deployment.generated.yaml` is a separate authenticated EC2 read. Run it only after checking the account, role, and regions and obtaining cloud-read authorization. + +## Secret value workflow + +Terraform creates regional secret metadata and grants the EC2 role read access. It never writes credential values. + +After an authorized operator has created the infrastructure and verified the declared secret ARNs, write a complete value through the existing CLI: + +```bash +ts prebid server secrets set examplebidder \ + --deployment deployment.generated.yaml \ + --region us-east-1 \ + --file /secure/path/examplebidder.json \ + --request-token 11111111-2222-4333-8444-555555555555 \ + --yes --json +``` + +Inputs: an approved profile, the exact descriptor, one complete JSON payload, a fresh retry UUID, and operator approval. The payload must contain only the declared keys and must not appear in command arguments or logs. + +Output: a nonsecret Secrets Manager version identifier and the target region. The command does not deploy PBS, refresh Compose, or prove bidder authorization. + +Failure behavior: preserve the retry UUID and input file. A failed or unverifiable response means the write is not confirmed and its outcome may be uncertain. Reuse the token only for the original identical payload; use a new token only for separately intended changed values. Withheld provider errors prevent distinguishing a rejected write from a lost response. On partial regional success, record each region separately. + +## Runtime release and rollback + +The current CLI has no deploy or rollback command. A future approved runtime implementation must: + +1. Resolve baseline plus regional overrides into one YAML file per region. +2. Retrieve and validate the selected immutable secret version. +3. Render `/run/pbs/secrets/examplebidder.env` with restrictive permissions, without shell evaluation. +4. Start or replace Compose with the pinned PBS image and read-only YAML. +5. Check ALB target health, `/status`, logs, and an isolated auction fixture. +6. Drain the old consumer before retiring it. +7. Record the release digest, resolved-config checksum, secret version, target, and result. +8. Use the recorded prior release and compatible secret version for rollback. + +Do not manually edit generated runtime environment files. A secret update alone does not change a running container. + +## Deferred evidence + +Before relying on secret writes, the runtime owner must arrange an authorized sandbox check of the real AWS CLI and Secrets Manager contract. Use a throwaway secret and dummy values to confirm that `VersionId` equals `ClientRequestToken`, an identical retry is idempotent, and changed values with the same token are rejected. The pinned AWS CLI `2.36.45` was checked with isolated local configuration files: `aws configure get cli_history` and `default.cli_history` both returned exit 1 with empty stdout when unset, without API calls. Repeat that local check on CLI upgrades. Process tests also cover this shape with a fake AWS executable. Neither check establishes real Secrets Manager behavior; no live verification is implied by the local checks. + +An authorized operator still needs to verify TLS and DNS, IAM permissions, public ingress, NAT and bidder egress, real adapter credentials, AMI contents, reboot and host replacement, interrupted runtime replacement, secret rotation, representative load, regional failover, alarm delivery, and the Trusted Server auction path. diff --git a/deploy/pbs-example/deployment.example.yaml b/deploy/pbs-example/deployment.example.yaml new file mode 100644 index 000000000..8df93a10e --- /dev/null +++ b/deploy/pbs-example/deployment.example.yaml @@ -0,0 +1,22 @@ +# Fictional local-check descriptor. Replace all account, profile, secret, and instance IDs before authorized use. +schema_version: 1 +environment: example +runtime: ec2-compose +aws: + account_id: '123456789012' + profile: pbs-example +pbs: + config: runtime/pbs.yaml + image: prebid/prebid-server@sha256:f0fee9caab93e14e9988b376c2c5371412628b7de1eca7a663bf203c4dd530a7 + bindings: runtime/secret-bindings.example.json +regions: + us-east-1: + instance_ids: + - i-0123456789abcde01 + - i-0123456789abcde02 + overrides: runtime/regions/east.yaml + us-west-2: + instance_ids: + - i-0fedcba9876543210 + - i-0fedcba9876543211 + overrides: runtime/regions/west.yaml diff --git a/deploy/pbs-example/dns.tf b/deploy/pbs-example/dns.tf new file mode 100644 index 000000000..902c22a18 --- /dev/null +++ b/deploy/pbs-example/dns.tf @@ -0,0 +1,35 @@ +resource "aws_route53_record" "east" { + zone_id = var.route53_zone_id + name = var.pbs_hostname + type = "A" + + set_identifier = "us-east-1" + + alias { + name = module.east.alb_dns_name + zone_id = module.east.alb_zone_id + evaluate_target_health = true + } + + latency_routing_policy { + region = "us-east-1" + } +} + +resource "aws_route53_record" "west" { + zone_id = var.route53_zone_id + name = var.pbs_hostname + type = "A" + + set_identifier = "us-west-2" + + alias { + name = module.west.alb_dns_name + zone_id = module.west.alb_zone_id + evaluate_target_health = true + } + + latency_routing_policy { + region = "us-west-2" + } +} diff --git a/deploy/pbs-example/locals.tf b/deploy/pbs-example/locals.tf new file mode 100644 index 000000000..db3d60304 --- /dev/null +++ b/deploy/pbs-example/locals.tf @@ -0,0 +1,58 @@ +locals { + common_tags = merge({ + Environment = "example" + ManagedBy = "Terraform" + Project = "prebid-server-example" + Owner = "example-team" + }, var.tags) + + pbs_image = "prebid/prebid-server@sha256:f0fee9caab93e14e9988b376c2c5371412628b7de1eca7a663bf203c4dd530a7" + + deployment_descriptor = { + schema_version = 1 + environment = "example" + runtime = "ec2-compose" + aws = { + account_id = var.aws_account_id + profile = var.aws_profile + } + pbs = { + config = "runtime/pbs.yaml" + image = local.pbs_image + bindings = "runtime/secret-bindings.generated.json" + } + regions = { + us-east-1 = { + instance_ids = values(module.east.instance_ids) + overrides = "runtime/regions/east.yaml" + } + us-west-2 = { + instance_ids = values(module.west.instance_ids) + overrides = "runtime/regions/west.yaml" + } + } + } + + secret_bindings = { + examplebidder = { + verified_image = local.pbs_image + source = "https://example.com/pbs-adapter-reference" + secrets = { + us-east-1 = module.east.secret_arns["examplebidder"] + us-west-2 = module.west.secret_arns["examplebidder"] + } + keys = { + api_key = { + env = "PBS_ADAPTERS_EXAMPLEBIDDER_API_KEY" + pbs_path = ["adapters", "examplebidder", "api_key"] + required = true + } + optional_token = { + env = "PBS_ADAPTERS_EXAMPLEBIDDER_OPTIONAL_TOKEN" + pbs_path = ["adapters", "examplebidder", "optional_token"] + required = false + } + } + } + } +} diff --git a/deploy/pbs-example/main.tf b/deploy/pbs-example/main.tf new file mode 100644 index 000000000..35c901fab --- /dev/null +++ b/deploy/pbs-example/main.tf @@ -0,0 +1,37 @@ +module "east" { + source = "./modules/regional" + + providers = { + aws = aws + } + + alarm_actions = var.alarm_actions + availability_zones = var.east_availability_zones + ami_id = var.east_ami_id + certificate_arn = var.east_certificate_arn + instance_type = var.instance_type + name = "pbs-example-us-east-1" + secrets_kms_key_arn = var.east_secrets_kms_key_arn + tags = local.common_tags + trusted_server_cidr_blocks = var.trusted_server_cidr_blocks + vpc_cidr = var.east_vpc_cidr +} + +module "west" { + source = "./modules/regional" + + providers = { + aws = aws.west + } + + alarm_actions = var.alarm_actions + availability_zones = var.west_availability_zones + ami_id = var.west_ami_id + certificate_arn = var.west_certificate_arn + instance_type = var.instance_type + name = "pbs-example-us-west-2" + secrets_kms_key_arn = var.west_secrets_kms_key_arn + tags = local.common_tags + trusted_server_cidr_blocks = var.trusted_server_cidr_blocks + vpc_cidr = var.west_vpc_cidr +} diff --git a/deploy/pbs-example/modules/regional/.terraform.lock.hcl b/deploy/pbs-example/modules/regional/.terraform.lock.hcl new file mode 100644 index 000000000..4f85daae2 --- /dev/null +++ b/deploy/pbs-example/modules/regional/.terraform.lock.hcl @@ -0,0 +1,29 @@ +# This file is maintained automatically by "terraform init". +# Manual edits may be lost in future updates. + +provider "registry.terraform.io/hashicorp/aws" { + version = "6.64.0" + constraints = "6.64.0" + hashes = [ + "h1:2fTLxzUDmp/KVIHbIeLTB4bIzWHx8E6Dw+1ALLUi+Yw=", + "h1:4siTahLyzGh4BoMQcL7VXeL/mn8iR/zKv93NyhQob+0=", + "h1:EEWCXlg69fty/Qi+kehrREnVEaWdKnLlVon542b6nxQ=", + "h1:wXARLY+IeQ7ufYxCLTPCwToWGMRvOpiOTfJS97iwUzI=", + "zh:07172315d67bc9781240272759cdfc7bd32b7e72384a56862c2c1da3cca99a81", + "zh:154ce7d2659de9a59ddfe96d7cab41a9ddc2cb267a7d4bcdf4e737ff2ffdec06", + "zh:17324d4335a7a7ac01cc23eded530775606680ff53b47cb74a3cb95d1121f836", + "zh:307ab92324ec5a61b124881ab8cac1d9e316f4527dfd0e1b59794c229407eb4e", + "zh:31e25f1903661332e36a95283042dd3ec50b47c186db00663fbd976a11e6a6b2", + "zh:3311d9f3bd12a24886027dbe73859dcd1e67bd0e3046227a338cf2c7ca04d18e", + "zh:37916156a3aac3b29be3acebd15d53145ea4ab5d4aaa825eaebe75481fa00500", + "zh:4158cb8c38b3ac6aa98eb15935ec6bd7c30838d85d2b00acc9812df8382ae908", + "zh:5bfb9499c66d9db5b34dc5c60f426a1ab1baa5457ce2aefebca826a9c3f92fb0", + "zh:6eb29ead5a4aca3b1f35812e7e8c75419180e1928e479b458f206861277736db", + "zh:7a82b6dd0c0cdef8045a4adfbddd36acb86b6b23fcbed8e189c2d71f7dc4a502", + "zh:9556bd792032c3f7e73ea4dd08cec88dc1327f5a4a57d79c30ba844ae2b9a3c0", + "zh:9b12af85486a96aedd8d7984b0ff811a4b42e3d88dad1a3fb4c0b580d04fa425", + "zh:c5234180464cb800c83a41f57462742b802c150ad7d4417626fcd9cb511c01d2", + "zh:cd776b83b1f7b36635957350afe7ce28ba4e4ea3a5e2deb00d13dbd3b35d9d40", + "zh:fb583a7b791c6f915b86573d04f05ddbf7f1a5e4120c5d8a7450a3086c1225c4", + ] +} diff --git a/deploy/pbs-example/modules/regional/README.md b/deploy/pbs-example/modules/regional/README.md new file mode 100644 index 000000000..b1f3a2868 --- /dev/null +++ b/deploy/pbs-example/modules/regional/README.md @@ -0,0 +1,5 @@ +# Regional PBS infrastructure module + +This local module creates one regional ALB, two public and two private subnets, one NAT gateway per AZ, one private EC2 Compose host per AZ, Secrets Manager metadata, IAM access, and CloudWatch alarms. + +It does not install software, write secret values, deploy PBS, or perform runtime replacement. diff --git a/deploy/pbs-example/modules/regional/compute.tf b/deploy/pbs-example/modules/regional/compute.tf new file mode 100644 index 000000000..d6772b89d --- /dev/null +++ b/deploy/pbs-example/modules/regional/compute.tf @@ -0,0 +1,35 @@ +# No user_data by design. Host bootstrap, PBS YAML rendering, secret-file +# materialization, and Compose startup require a separately approved runtime +# loader. Applied hosts stay ALB-unhealthy until it runs. See ../../README.md, +# "Stop boundary". +resource "aws_instance" "pbs" { + for_each = local.availability_zone_index + + ami = var.ami_id + instance_type = var.instance_type + iam_instance_profile = aws_iam_instance_profile.pbs.name + subnet_id = aws_subnet.private[each.key].id + vpc_security_group_ids = [aws_security_group.pbs.id] + associate_public_ip_address = false + + metadata_options { + http_endpoint = "enabled" + http_protocol_ipv6 = "disabled" + http_put_response_hop_limit = 1 + http_tokens = "required" + instance_metadata_tags = "disabled" + } + + root_block_device { + delete_on_termination = true + encrypted = true + volume_size = 30 + volume_type = "gp3" + } + + tags = merge(var.tags, { + Name = "${var.name}-${each.key}-pbs" + }) + + depends_on = [aws_iam_role_policy.runtime] +} diff --git a/deploy/pbs-example/modules/regional/load_balancing.tf b/deploy/pbs-example/modules/regional/load_balancing.tf new file mode 100644 index 000000000..481b1998d --- /dev/null +++ b/deploy/pbs-example/modules/regional/load_balancing.tf @@ -0,0 +1,59 @@ +resource "aws_lb" "main" { + name = "${var.name}-alb" + internal = false + load_balancer_type = "application" + security_groups = [aws_security_group.alb.id] + subnets = [for availability_zone in var.availability_zones : aws_subnet.public[availability_zone].id] + drop_invalid_header_fields = true + idle_timeout = 30 + + tags = merge(var.tags, { + Name = "${var.name}-alb" + }) +} + +resource "aws_lb_target_group" "pbs" { + name = "${var.name}-pbs" + port = local.pbs_port + protocol = "HTTP" + target_type = "instance" + vpc_id = aws_vpc.main.id + deregistration_delay = 30 + + health_check { + enabled = true + path = "/status" + port = "traffic-port" + protocol = "HTTP" + matcher = "200-399" + interval = 15 + timeout = 5 + healthy_threshold = 2 + unhealthy_threshold = 3 + } + + tags = merge(var.tags, { + Name = "${var.name}-pbs" + }) +} + +resource "aws_lb_target_group_attachment" "pbs" { + for_each = aws_instance.pbs + + target_group_arn = aws_lb_target_group.pbs.arn + target_id = each.value.id + port = local.pbs_port +} + +resource "aws_lb_listener" "https" { + load_balancer_arn = aws_lb.main.arn + port = 443 + protocol = "HTTPS" + ssl_policy = "ELBSecurityPolicy-TLS13-1-2-2021-06" + certificate_arn = var.certificate_arn + + default_action { + type = "forward" + target_group_arn = aws_lb_target_group.pbs.arn + } +} diff --git a/deploy/pbs-example/modules/regional/locals.tf b/deploy/pbs-example/modules/regional/locals.tf new file mode 100644 index 000000000..57c182868 --- /dev/null +++ b/deploy/pbs-example/modules/regional/locals.tf @@ -0,0 +1,13 @@ +locals { + # Coupled to runtime/compose.yaml's host/container ports and runtime/pbs.yaml's + # port. Changing this requires matching runtime edits and wiring checks. + pbs_port = 8000 + + availability_zone_index = { + for index, availability_zone in var.availability_zones : availability_zone => index + } + + bidder_secret_names = { + examplebidder = "${var.name}/examplebidder" + } +} diff --git a/deploy/pbs-example/modules/regional/monitoring.tf b/deploy/pbs-example/modules/regional/monitoring.tf new file mode 100644 index 000000000..8c5a56766 --- /dev/null +++ b/deploy/pbs-example/modules/regional/monitoring.tf @@ -0,0 +1,65 @@ +resource "aws_cloudwatch_metric_alarm" "instance_cpu" { + for_each = aws_instance.pbs + + alarm_name = "${var.name}-${each.key}-high-cpu" + comparison_operator = "GreaterThanThreshold" + evaluation_periods = 3 + metric_name = "CPUUtilization" + namespace = "AWS/EC2" + period = 60 + statistic = "Average" + threshold = 80 + alarm_actions = var.alarm_actions + treat_missing_data = "missing" + + dimensions = { + InstanceId = each.value.id + } + + tags = merge(var.tags, { + Name = "${var.name}-${each.key}-high-cpu" + }) +} + +resource "aws_cloudwatch_metric_alarm" "unhealthy_hosts" { + alarm_name = "${var.name}-unhealthy-hosts" + comparison_operator = "GreaterThanThreshold" + evaluation_periods = 2 + metric_name = "UnHealthyHostCount" + namespace = "AWS/ApplicationELB" + period = 60 + statistic = "Maximum" + threshold = 0 + alarm_actions = var.alarm_actions + treat_missing_data = "breaching" + + dimensions = { + LoadBalancer = aws_lb.main.arn_suffix + TargetGroup = aws_lb_target_group.pbs.arn_suffix + } + + tags = merge(var.tags, { + Name = "${var.name}-unhealthy-hosts" + }) +} + +resource "aws_cloudwatch_metric_alarm" "elb_5xx" { + alarm_name = "${var.name}-elb-5xx" + comparison_operator = "GreaterThanThreshold" + evaluation_periods = 3 + metric_name = "HTTPCode_ELB_5XX_Count" + namespace = "AWS/ApplicationELB" + period = 60 + statistic = "Sum" + threshold = 1 + alarm_actions = var.alarm_actions + treat_missing_data = "notBreaching" + + dimensions = { + LoadBalancer = aws_lb.main.arn_suffix + } + + tags = merge(var.tags, { + Name = "${var.name}-elb-5xx" + }) +} diff --git a/deploy/pbs-example/modules/regional/network.tf b/deploy/pbs-example/modules/regional/network.tf new file mode 100644 index 000000000..a1f251012 --- /dev/null +++ b/deploy/pbs-example/modules/regional/network.tf @@ -0,0 +1,109 @@ +resource "aws_vpc" "main" { + cidr_block = var.vpc_cidr + enable_dns_hostnames = true + enable_dns_support = true + + tags = merge(var.tags, { + Name = "${var.name}-vpc" + }) +} + +resource "aws_internet_gateway" "main" { + vpc_id = aws_vpc.main.id + + tags = merge(var.tags, { + Name = "${var.name}-igw" + }) +} + +resource "aws_subnet" "public" { + for_each = local.availability_zone_index + + vpc_id = aws_vpc.main.id + availability_zone = each.key + cidr_block = cidrsubnet(var.vpc_cidr, 4, each.value) + map_public_ip_on_launch = false + + tags = merge(var.tags, { + Name = "${var.name}-${each.key}-public" + Tier = "public" + }) +} + +resource "aws_subnet" "private" { + for_each = local.availability_zone_index + + vpc_id = aws_vpc.main.id + availability_zone = each.key + cidr_block = cidrsubnet(var.vpc_cidr, 4, each.value + 8) + + tags = merge(var.tags, { + Name = "${var.name}-${each.key}-private" + Tier = "private" + }) +} + +resource "aws_route_table" "public" { + vpc_id = aws_vpc.main.id + + route { + cidr_block = "0.0.0.0/0" + gateway_id = aws_internet_gateway.main.id + } + + tags = merge(var.tags, { + Name = "${var.name}-public" + }) +} + +resource "aws_route_table_association" "public" { + for_each = local.availability_zone_index + + route_table_id = aws_route_table.public.id + subnet_id = aws_subnet.public[each.key].id +} + +resource "aws_eip" "nat" { + for_each = local.availability_zone_index + + domain = "vpc" + + tags = merge(var.tags, { + Name = "${var.name}-${each.key}-nat" + }) +} + +resource "aws_nat_gateway" "main" { + for_each = local.availability_zone_index + + allocation_id = aws_eip.nat[each.key].id + subnet_id = aws_subnet.public[each.key].id + + depends_on = [aws_internet_gateway.main] + + tags = merge(var.tags, { + Name = "${var.name}-${each.key}-nat" + }) +} + +resource "aws_route_table" "private" { + for_each = local.availability_zone_index + + vpc_id = aws_vpc.main.id + + route { + cidr_block = "0.0.0.0/0" + nat_gateway_id = aws_nat_gateway.main[each.key].id + } + + tags = merge(var.tags, { + Name = "${var.name}-${each.key}-private" + }) +} + +resource "aws_route_table_association" "private" { + for_each = local.availability_zone_index + + route_table_id = aws_route_table.private[each.key].id + subnet_id = aws_subnet.private[each.key].id +} diff --git a/deploy/pbs-example/modules/regional/outputs.tf b/deploy/pbs-example/modules/regional/outputs.tf new file mode 100644 index 000000000..2d5268e28 --- /dev/null +++ b/deploy/pbs-example/modules/regional/outputs.tf @@ -0,0 +1,34 @@ +output "alb_dns_name" { + description = "Regional ALB DNS name." + value = aws_lb.main.dns_name +} + +output "alb_zone_id" { + description = "Route 53 hosted-zone ID for the regional ALB alias." + value = aws_lb.main.zone_id +} + +output "instance_ids" { + description = "PBS EC2 instance IDs keyed by Availability Zone." + value = { for availability_zone, instance in aws_instance.pbs : availability_zone => instance.id } +} + +output "nat_eip_addresses" { + description = "Stable NAT gateway egress addresses keyed by Availability Zone for bidder allowlists." + value = { for availability_zone, eip in aws_eip.nat : availability_zone => eip.public_ip } +} + +output "private_subnet_ids" { + description = "Private subnet IDs keyed by Availability Zone." + value = { for availability_zone, subnet in aws_subnet.private : availability_zone => subnet.id } +} + +output "secret_arns" { + description = "Bidder Secrets Manager ARNs keyed by bidder identifier." + value = { for bidder, secret in aws_secretsmanager_secret.bidder : bidder => secret.arn } +} + +output "vpc_id" { + description = "Regional VPC ID." + value = aws_vpc.main.id +} diff --git a/deploy/pbs-example/modules/regional/secrets.tf b/deploy/pbs-example/modules/regional/secrets.tf new file mode 100644 index 000000000..360b41709 --- /dev/null +++ b/deploy/pbs-example/modules/regional/secrets.tf @@ -0,0 +1,72 @@ +resource "aws_secretsmanager_secret" "bidder" { + for_each = local.bidder_secret_names + + description = "Credential payload for ${each.key}; values are written outside Terraform." + name = each.value + kms_key_id = var.secrets_kms_key_arn + recovery_window_in_days = 7 + + tags = merge(var.tags, { + Name = each.value + Secret = each.key + }) +} + +resource "aws_iam_role" "pbs" { + name = "${var.name}-runtime" + assume_role_policy = jsonencode({ + Version = "2012-10-17" + Statement = [{ + Action = "sts:AssumeRole" + Effect = "Allow" + Principal = { + Service = "ec2.amazonaws.com" + } + }] + }) + + tags = merge(var.tags, { + Name = "${var.name}-runtime" + }) +} + +resource "aws_iam_instance_profile" "pbs" { + name = "${var.name}-runtime" + role = aws_iam_role.pbs.name + + tags = merge(var.tags, { + Name = "${var.name}-runtime" + }) +} + +data "aws_partition" "current" {} + +resource "aws_iam_role_policy_attachment" "ssm" { + role = aws_iam_role.pbs.name + policy_arn = "arn:${data.aws_partition.current.partition}:iam::aws:policy/AmazonSSMManagedInstanceCore" +} + +resource "aws_iam_role_policy" "runtime" { + name = "runtime" + role = aws_iam_role.pbs.name + + policy = jsonencode({ + Version = "2012-10-17" + Statement = concat([ + { + Sid = "ReadBidderSecrets" + Effect = "Allow" + Action = [ + "secretsmanager:DescribeSecret", + "secretsmanager:GetSecretValue", + ] + Resource = [for secret in aws_secretsmanager_secret.bidder : secret.arn] + }, + ], var.secrets_kms_key_arn == null ? [] : [{ + Sid = "DecryptBidderSecrets" + Effect = "Allow" + Action = ["kms:Decrypt"] + Resource = var.secrets_kms_key_arn + }]) + }) +} diff --git a/deploy/pbs-example/modules/regional/security.tf b/deploy/pbs-example/modules/regional/security.tf new file mode 100644 index 000000000..9e85e62d0 --- /dev/null +++ b/deploy/pbs-example/modules/regional/security.tf @@ -0,0 +1,55 @@ +resource "aws_security_group" "alb" { + name = "${var.name}-alb" + description = "HTTPS ingress for the regional PBS ALB" + vpc_id = aws_vpc.main.id + + tags = merge(var.tags, { + Name = "${var.name}-alb" + }) +} + +resource "aws_vpc_security_group_ingress_rule" "trusted_server_https" { + for_each = var.trusted_server_cidr_blocks + + description = "Trusted Server HTTPS" + security_group_id = aws_security_group.alb.id + cidr_ipv4 = each.value + from_port = 443 + to_port = 443 + ip_protocol = "tcp" +} + +resource "aws_vpc_security_group_egress_rule" "alb_to_pbs" { + description = "Forward requests to private PBS hosts" + security_group_id = aws_security_group.alb.id + referenced_security_group_id = aws_security_group.pbs.id + from_port = local.pbs_port + to_port = local.pbs_port + ip_protocol = "tcp" +} + +resource "aws_security_group" "pbs" { + name = "${var.name}-pbs" + description = "PBS host traffic from the regional ALB and outbound bidder access" + vpc_id = aws_vpc.main.id + + ingress { + description = "PBS HTTP from the regional ALB" + from_port = local.pbs_port + to_port = local.pbs_port + protocol = "tcp" + security_groups = [aws_security_group.alb.id] + } + + egress { + description = "Bidder and AWS API access through the per-AZ NAT gateway" + from_port = 0 + to_port = 0 + protocol = "-1" + cidr_blocks = ["0.0.0.0/0"] + } + + tags = merge(var.tags, { + Name = "${var.name}-pbs" + }) +} diff --git a/deploy/pbs-example/modules/regional/terraform.tf b/deploy/pbs-example/modules/regional/terraform.tf new file mode 100644 index 000000000..1e8efd5f5 --- /dev/null +++ b/deploy/pbs-example/modules/regional/terraform.tf @@ -0,0 +1,8 @@ +terraform { + required_providers { + aws = { + source = "hashicorp/aws" + version = "= 6.64.0" + } + } +} diff --git a/deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl b/deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl new file mode 100644 index 000000000..b15b4bab4 --- /dev/null +++ b/deploy/pbs-example/modules/regional/tests/security_unit_test.tftest.hcl @@ -0,0 +1,274 @@ +mock_provider "aws" { + mock_data "aws_partition" { + defaults = { partition = "aws" } + } + + mock_resource "aws_instance" { + defaults = { + id = "i-0123456789abcdef0" + } + } + + mock_resource "aws_eip" { + defaults = { + public_ip = "192.0.2.10" + } + } + + mock_resource "aws_secretsmanager_secret" { + override_during = plan + + defaults = { + arn = "arn:aws:secretsmanager:us-east-1:123456789012:secret:example-AbCdEf" + } + } +} + +override_resource { + target = aws_subnet.public["us-east-1a"] + override_during = plan + values = { + id = "subnet-00000000000000001" + } +} + +override_resource { + target = aws_subnet.public["us-east-1b"] + override_during = plan + values = { + id = "subnet-00000000000000002" + } +} + +override_resource { + target = aws_subnet.private["us-east-1a"] + override_during = plan + values = { + id = "subnet-00000000000000003" + } +} + +override_resource { + target = aws_subnet.private["us-east-1b"] + override_during = plan + values = { + id = "subnet-00000000000000004" + } +} + +override_resource { + target = aws_security_group.alb + override_during = plan + values = { + id = "sg-00000000000000001" + } +} + +override_resource { + target = aws_security_group.pbs + override_during = plan + values = { + id = "sg-00000000000000002" + } +} + +variables { + alarm_actions = [] + ami_id = "ami-0123456789abcdef0" + availability_zones = ["us-east-1a", "us-east-1b"] + certificate_arn = "arn:aws:acm:us-east-1:123456789012:certificate/11111111-1111-4111-8111-111111111111" + instance_type = "c7i.large" + name = "pbs-example-us-east-1" + tags = { ManagedBy = "Terraform" } + trusted_server_cidr_blocks = ["192.0.2.0/24", "198.51.100.0/24"] + vpc_cidr = "10.80.0.0/16" +} + +run "plans_private_hosts_and_scoped_ingress" { + command = plan + + assert { + condition = length(aws_instance.pbs) == 2 + error_message = "The regional module should plan one PBS host in each of two AZs." + } + + assert { + condition = alltrue([for instance in aws_instance.pbs : instance.associate_public_ip_address == false]) + error_message = "PBS hosts should not receive public IP addresses." + } + + # Explicit instance paths keep Terraform 1.16.2 failure diagnostics from serializing + # the whole instance map, which contains provider-marked sensitive attributes. + assert { + condition = alltrue([ + aws_instance.pbs["us-east-1a"].metadata_options[0].http_tokens == "required", + aws_instance.pbs["us-east-1b"].metadata_options[0].http_tokens == "required", + ]) + error_message = "PBS hosts should require IMDSv2 session tokens." + } + + assert { + condition = alltrue([ + aws_instance.pbs["us-east-1a"].root_block_device[0].encrypted, + aws_instance.pbs["us-east-1b"].root_block_device[0].encrypted, + ]) + error_message = "PBS host root volumes should be encrypted." + } + + assert { + condition = alltrue([ + aws_instance.pbs["us-east-1a"].subnet_id == aws_subnet.private["us-east-1a"].id, + aws_instance.pbs["us-east-1b"].subnet_id == aws_subnet.private["us-east-1b"].id, + ]) + error_message = "PBS hosts should launch in the private subnet of their own AZ." + } + + assert { + condition = aws_lb_listener.https.ssl_policy == "ELBSecurityPolicy-TLS13-1-2-2021-06" + error_message = "The ALB listener should pin the TLS 1.3/1.2 policy." + } + + assert { + condition = ( + aws_vpc_security_group_egress_rule.alb_to_pbs.security_group_id == aws_security_group.alb.id && + aws_vpc_security_group_egress_rule.alb_to_pbs.referenced_security_group_id == aws_security_group.pbs.id && + aws_vpc_security_group_egress_rule.alb_to_pbs.ip_protocol == "tcp" && + aws_vpc_security_group_egress_rule.alb_to_pbs.from_port == 8000 && + aws_vpc_security_group_egress_rule.alb_to_pbs.to_port == 8000 && + aws_vpc_security_group_egress_rule.alb_to_pbs.cidr_ipv4 == null && + aws_vpc_security_group_egress_rule.alb_to_pbs.cidr_ipv6 == null && + aws_vpc_security_group_egress_rule.alb_to_pbs.prefix_list_id == null + ) + error_message = "ALB outbound traffic should reach only the PBS security group on the PBS TCP port." + } + + assert { + condition = length(aws_vpc_security_group_ingress_rule.trusted_server_https) == length(var.trusted_server_cidr_blocks) + error_message = "The ALB should have one ingress rule for each approved caller CIDR." + } + + assert { + condition = alltrue([ + for cidr, rule in aws_vpc_security_group_ingress_rule.trusted_server_https : + rule.security_group_id == aws_security_group.alb.id && + rule.cidr_ipv4 == cidr && contains(var.trusted_server_cidr_blocks, cidr) && + rule.ip_protocol == "tcp" && rule.from_port == 443 && rule.to_port == 443 + ]) + error_message = "Every ALB ingress rule should permit only HTTPS from its approved caller CIDR." + } + + assert { + condition = ( + aws_lb_target_group.pbs.port == 8000 && + alltrue([for attachment in aws_lb_target_group_attachment.pbs : attachment.port == 8000]) && + alltrue([for rule in aws_security_group.pbs.ingress : rule.from_port == 8000 && rule.to_port == 8000]) + ) + error_message = "ALB targets and PBS ingress must match the runtime's fixed port 8000." + } + + assert { + condition = aws_iam_role_policy_attachment.ssm.policy_arn == "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore" + error_message = "Commercial-region hosts should attach the commercial SSM managed policy." + } + + assert { + condition = length(aws_secretsmanager_secret.bidder) == 1 + error_message = "The example should create metadata for only the declared example bidder secret." + } + + assert { + condition = alltrue([ + for alarm in aws_cloudwatch_metric_alarm.instance_cpu : + alarm.treat_missing_data == "missing" + ]) + error_message = "Missing CPU samples should not trigger high-CPU alarms." + } + + assert { + condition = length(jsondecode(aws_iam_role_policy.runtime.policy).Statement[0].Resource) == length(aws_secretsmanager_secret.bidder) + error_message = "The runtime role should read only the declared bidder secrets." + } +} + +run "rejects_overlong_resource_names" { + command = plan + + variables { + name = "abcdefghijklmnopqrstuvwxyzabc" + } + + expect_failures = [var.name] +} + +run "rejects_invalid_resource_names" { + command = plan + + variables { + name = "-pbs-example" + } + + expect_failures = [var.name] +} + +run "rejects_reserved_alb_prefix" { + command = plan + + variables { + name = "internal-pbs" + } + + expect_failures = [var.name] +} + +run "accepts_maximum_resource_name" { + command = plan + + variables { + name = "abcdefghijklmnopqrstuvwxyzab" + } + + assert { + condition = aws_lb.main.name == "${var.name}-alb" && aws_lb_target_group.pbs.name == "${var.name}-pbs" + error_message = "Names should preserve the full prefix and resource suffix." + } +} + +run "accepts_single_character_resource_name" { + command = plan + + variables { + name = "a" + } + + assert { + condition = aws_lb.main.name == "a-alb" && aws_lb_target_group.pbs.name == "a-pbs" + error_message = "A single alphanumeric character is a valid name prefix." + } +} + +run "uses_current_partition_for_ssm_policy" { + command = plan + + override_data { + target = data.aws_partition.current + values = { partition = "aws-us-gov" } + } + + assert { + condition = aws_iam_role_policy_attachment.ssm.policy_arn == "arn:aws-us-gov:iam::aws:policy/AmazonSSMManagedInstanceCore" + error_message = "The SSM managed policy ARN should follow the provider's AWS partition." + } +} + +run "scopes_optional_kms_decryption" { + command = plan + + variables { + secrets_kms_key_arn = "arn:aws:kms:us-east-1:123456789012:key/11111111-1111-4111-8111-111111111111" + } + + assert { + condition = jsondecode(aws_iam_role_policy.runtime.policy).Statement[1].Resource == var.secrets_kms_key_arn + error_message = "KMS decryption should be scoped to the selected regional key." + } +} diff --git a/deploy/pbs-example/modules/regional/variables.tf b/deploy/pbs-example/modules/regional/variables.tf new file mode 100644 index 000000000..46fe9383b --- /dev/null +++ b/deploy/pbs-example/modules/regional/variables.tf @@ -0,0 +1,60 @@ +variable "alarm_actions" { + description = "CloudWatch alarm action ARNs." + type = set(string) +} + +variable "ami_id" { + description = "Pre-baked regional AMI containing the approved host runtime." + type = string +} + +variable "availability_zones" { + description = "Exactly two Availability Zones for the regional ALB and PBS hosts." + type = list(string) + + validation { + condition = length(var.availability_zones) == 2 && length(distinct(var.availability_zones)) == 2 + error_message = "availability_zones must contain exactly two distinct Availability Zones." + } +} + +variable "certificate_arn" { + description = "Existing ACM certificate ARN for the regional ALB." + type = string +} + +variable "instance_type" { + description = "EC2 instance type for the regional PBS hosts." + type = string +} + +variable "name" { + description = "Stable name prefix for regional resources, leaving room for the ALB and target-group suffixes." + type = string + + validation { + condition = can(regex("^[a-zA-Z0-9]([a-zA-Z0-9-]{0,26}[a-zA-Z0-9])?$", var.name)) && !startswith(var.name, "internal-") + error_message = "name must be 1-28 alphanumeric/hyphen characters, must not start or end with a hyphen, and must not use the reserved ALB prefix internal-." + } +} + +variable "secrets_kms_key_arn" { + description = "Optional customer-managed KMS key ARN for regional bidder secrets." + type = string + default = null +} + +variable "tags" { + description = "Common resource tags." + type = map(string) +} + +variable "trusted_server_cidr_blocks" { + description = "CIDR blocks permitted to reach the regional ALB." + type = set(string) +} + +variable "vpc_cidr" { + description = "Regional VPC CIDR block." + type = string +} diff --git a/deploy/pbs-example/outputs.tf b/deploy/pbs-example/outputs.tf new file mode 100644 index 000000000..117489ace --- /dev/null +++ b/deploy/pbs-example/outputs.tf @@ -0,0 +1,49 @@ +output "east_alb_dns_name" { + description = "DNS name of the us-east-1 regional ALB." + value = module.east.alb_dns_name +} + +output "east_instance_ids" { + description = "PBS EC2 instance IDs in us-east-1." + value = module.east.instance_ids +} + +output "east_nat_eip_addresses" { + description = "Stable us-east-1 NAT egress addresses for bidder allowlists." + value = module.east.nat_eip_addresses +} + +output "east_secret_arns" { + description = "Bidder Secrets Manager ARNs in us-east-1. Values are never managed by Terraform." + value = module.east.secret_arns +} + +output "deployment_descriptor_yaml" { + description = "Rendered nonsecret CLI descriptor. Write it to the ignored deployment.generated.yaml after apply." + value = yamlencode(local.deployment_descriptor) +} + +output "secret_bindings_json" { + description = "Rendered nonsecret binding metadata. Write it to the ignored runtime/secret-bindings.generated.json after apply." + value = jsonencode(local.secret_bindings) +} + +output "west_alb_dns_name" { + description = "DNS name of the us-west-2 regional ALB." + value = module.west.alb_dns_name +} + +output "west_instance_ids" { + description = "PBS EC2 instance IDs in us-west-2." + value = module.west.instance_ids +} + +output "west_nat_eip_addresses" { + description = "Stable us-west-2 NAT egress addresses for bidder allowlists." + value = module.west.nat_eip_addresses +} + +output "west_secret_arns" { + description = "Bidder Secrets Manager ARNs in us-west-2. Values are never managed by Terraform." + value = module.west.secret_arns +} diff --git a/deploy/pbs-example/providers.tf b/deploy/pbs-example/providers.tf new file mode 100644 index 000000000..875201637 --- /dev/null +++ b/deploy/pbs-example/providers.tf @@ -0,0 +1,20 @@ +provider "aws" { + profile = var.aws_profile + region = "us-east-1" + allowed_account_ids = [var.aws_account_id] + + default_tags { + tags = local.common_tags + } +} + +provider "aws" { + alias = "west" + profile = var.aws_profile + region = "us-west-2" + allowed_account_ids = [var.aws_account_id] + + default_tags { + tags = local.common_tags + } +} diff --git a/deploy/pbs-example/runtime/README.md b/deploy/pbs-example/runtime/README.md new file mode 100644 index 000000000..9c795880a --- /dev/null +++ b/deploy/pbs-example/runtime/README.md @@ -0,0 +1,9 @@ +# PBS runtime example + +This directory owns nonsecret PBS configuration and the Compose shape. The image is pinned to Prebid Server Go v4.7.0 by digest. Compose uses bounded local logs rather than implying that a CloudWatch log shipper is installed. + +The `/run/pbs/secrets/examplebidder.env` file is a runtime contract, not a checked-in credential file. A separately approved host-side loader must retrieve the selected regional Secrets Manager version, validate the complete JSON payload, render this restricted environment file atomically, and start or replace Compose. The current `ts prebid server` CLI does not implement that release or injection workflow. + +`secret-bindings.example.json` and `examples/pbs-secrets.env` contain fictional values for local checks only. An authorized Terraform apply can render the ignored `secret-bindings.generated.json` with actual regional secret ARNs, but never secret values. + +Deployment Compose binds `0.0.0.0:8000` by default so the ALB can reach PBS through the host security group. `scripts/smoke-runtime.sh` forces `PBS_BIND_ADDRESS=127.0.0.1` and checked-in dummy inputs for local startup. `scripts/test-smoke-runtime.mjs` verifies both bindings and the smoke selectors with Compose rendering and fake commands only. Use the Node.js wiring command in the parent runbook. Run the shell smoke from the parent example directory or use its repository-relative path. diff --git a/deploy/pbs-example/runtime/compose.yaml b/deploy/pbs-example/runtime/compose.yaml new file mode 100644 index 000000000..2269a765a --- /dev/null +++ b/deploy/pbs-example/runtime/compose.yaml @@ -0,0 +1,20 @@ +services: + pbs: + image: prebid/prebid-server@sha256:f0fee9caab93e14e9988b376c2c5371412628b7de1eca7a663bf203c4dd530a7 + restart: unless-stopped + ports: + - "${PBS_BIND_ADDRESS:-0.0.0.0}:${PBS_HOST_PORT:-8000}:8000" + env_file: + - ${PBS_SECRET_ENV_FILE:-/run/pbs/secrets/examplebidder.env} + volumes: + - ${PBS_CONFIG_FILE:-./pbs.yaml}:/etc/config/pbs.yaml:ro + read_only: true + security_opt: + - no-new-privileges:true + cap_drop: + - ALL + logging: + driver: local + options: + max-file: "3" + max-size: 10m diff --git a/deploy/pbs-example/runtime/examples/README.md b/deploy/pbs-example/runtime/examples/README.md new file mode 100644 index 000000000..a8e9ca3e4 --- /dev/null +++ b/deploy/pbs-example/runtime/examples/README.md @@ -0,0 +1 @@ +These files contain fictional values for local structure checks only. They are not bidder credentials and must never be copied into an AWS secret. diff --git a/deploy/pbs-example/runtime/examples/compose.env b/deploy/pbs-example/runtime/examples/compose.env new file mode 100644 index 000000000..a1727c63e --- /dev/null +++ b/deploy/pbs-example/runtime/examples/compose.env @@ -0,0 +1,3 @@ +# Local-only Compose validation selectors. +PBS_CONFIG_FILE=./pbs.yaml +PBS_SECRET_ENV_FILE=./examples/pbs-secrets.env diff --git a/deploy/pbs-example/runtime/examples/pbs-secrets.env b/deploy/pbs-example/runtime/examples/pbs-secrets.env new file mode 100644 index 000000000..ff5644bab --- /dev/null +++ b/deploy/pbs-example/runtime/examples/pbs-secrets.env @@ -0,0 +1,3 @@ +# Fictional dummy values for local Compose parsing only. +PBS_ADAPTERS_EXAMPLEBIDDER_API_KEY=example-only-api-key +PBS_ADAPTERS_EXAMPLEBIDDER_OPTIONAL_TOKEN=example-only-optional-token diff --git a/deploy/pbs-example/runtime/pbs.yaml b/deploy/pbs-example/runtime/pbs.yaml new file mode 100644 index 000000000..858d694e5 --- /dev/null +++ b/deploy/pbs-example/runtime/pbs.yaml @@ -0,0 +1,11 @@ +# PBS Go v4.7.0 baseline. Verify every field against the pinned release before deployment. +external_url: https://pbs.example.com +host: 0.0.0.0 +port: 8000 +status_response: ok +auction_timeouts_ms: + default: 1000 + max: 1200 +stored_requests_timeout_ms: 100 +gdpr: + default_value: "1" diff --git a/deploy/pbs-example/runtime/regions/east.yaml b/deploy/pbs-example/runtime/regions/east.yaml new file mode 100644 index 000000000..58cb7ae81 --- /dev/null +++ b/deploy/pbs-example/runtime/regions/east.yaml @@ -0,0 +1,2 @@ +# Regional nonsecret override. The shared external URL remains the Route 53 hostname. +datacenter: us-east-1 diff --git a/deploy/pbs-example/runtime/regions/west.yaml b/deploy/pbs-example/runtime/regions/west.yaml new file mode 100644 index 000000000..84227de00 --- /dev/null +++ b/deploy/pbs-example/runtime/regions/west.yaml @@ -0,0 +1,2 @@ +# Regional nonsecret override. The shared external URL remains the Route 53 hostname. +datacenter: us-west-2 diff --git a/deploy/pbs-example/runtime/secret-bindings.example.json b/deploy/pbs-example/runtime/secret-bindings.example.json new file mode 100644 index 000000000..dc6340844 --- /dev/null +++ b/deploy/pbs-example/runtime/secret-bindings.example.json @@ -0,0 +1,22 @@ +{ + "examplebidder": { + "verified_image": "prebid/prebid-server@sha256:f0fee9caab93e14e9988b376c2c5371412628b7de1eca7a663bf203c4dd530a7", + "source": "https://example.com/pbs-adapter-reference", + "secrets": { + "us-east-1": "arn:aws:secretsmanager:us-east-1:123456789012:secret:pbs-example-us-east-1/examplebidder-AbCdEf", + "us-west-2": "arn:aws:secretsmanager:us-west-2:123456789012:secret:pbs-example-us-west-2/examplebidder-GhIjKl" + }, + "keys": { + "api_key": { + "env": "PBS_ADAPTERS_EXAMPLEBIDDER_API_KEY", + "pbs_path": ["adapters", "examplebidder", "api_key"], + "required": true + }, + "optional_token": { + "env": "PBS_ADAPTERS_EXAMPLEBIDDER_OPTIONAL_TOKEN", + "pbs_path": ["adapters", "examplebidder", "optional_token"], + "required": false + } + } + } +} diff --git a/deploy/pbs-example/scripts/smoke-runtime.sh b/deploy/pbs-example/scripts/smoke-runtime.sh new file mode 100755 index 000000000..7a5d89a9b --- /dev/null +++ b/deploy/pbs-example/scripts/smoke-runtime.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +set -euo pipefail + +example_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +project_name="pbs-example-smoke" +host_port="${PBS_SMOKE_PORT:-18080}" +compose=( + docker compose + --project-name "$project_name" + --env-file "$example_dir/runtime/examples/compose.env" + -f "$example_dir/runtime/compose.yaml" +) + +cleanup() { + "${compose[@]}" down --volumes --remove-orphans >/dev/null 2>&1 || true +} +trap cleanup EXIT + +# Smoke always uses loopback and checked-in dummy inputs, never inherited selectors. +export PBS_BIND_ADDRESS="127.0.0.1" +export PBS_HOST_PORT="$host_port" +export PBS_CONFIG_FILE="$example_dir/runtime/pbs.yaml" +export PBS_SECRET_ENV_FILE="$example_dir/runtime/examples/pbs-secrets.env" +"${compose[@]}" up --detach + +for _ in $(seq 1 30); do + if response="$(curl --fail --silent --show-error "http://127.0.0.1:${host_port}/status" 2>/dev/null)"; then + if [[ "$response" != "ok" ]]; then + printf 'Unexpected PBS status response: %s\n' "$response" >&2 + exit 1 + fi + + logs="$("${compose[@]}" logs --no-color)" + if grep -Fq "example-only-api-key" <<<"$logs" || grep -Fq "example-only-optional-token" <<<"$logs"; then + printf 'PBS startup logs exposed a dummy secret.\n' >&2 + exit 1 + fi + + printf 'PBS runtime and startup-log redaction smoke passed on port %s.\n' "$host_port" + exit 0 + fi + sleep 1 +done + +"${compose[@]}" logs --no-color >&2 +printf 'PBS did not become healthy within 30 seconds.\n' >&2 +exit 1 diff --git a/deploy/pbs-example/scripts/test-smoke-runtime.mjs b/deploy/pbs-example/scripts/test-smoke-runtime.mjs new file mode 100644 index 000000000..17aec86bd --- /dev/null +++ b/deploy/pbs-example/scripts/test-smoke-runtime.mjs @@ -0,0 +1,125 @@ +#!/usr/bin/env node +// Render Compose and exercise smoke command wiring without starting containers. +import assert from 'node:assert/strict' +import { execFileSync } from 'node:child_process' +import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { delimiter, join } from 'node:path' +import { fileURLToPath } from 'node:url' + +const example = fileURLToPath(new URL('../', import.meta.url)) +const env = Object.fromEntries( + ['PATH', 'HOME'] + .filter((key) => process.env[key] !== undefined) + .map((key) => [key, process.env[key]]) +) + +// Inherit stderr so a failed Compose command explains why it failed. +function run(command, args, environment = env) { + return execFileSync(command, args, { + env: environment, + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'inherit'], + }) +} + +function renderProduction() { + const compose = join(example, 'runtime/compose.yaml') + // Older Compose releases stat env_file even with --no-env-resolution. + // Check the deployment default separately and render with dummy values. + assert( + readFileSync(compose, 'utf8').includes( + '${PBS_SECRET_ENV_FILE:-/run/pbs/secrets/examplebidder.env}' + ) + ) + const service = JSON.parse( + run('docker', ['compose', '-f', compose, 'config', '--format', 'json'], { + ...env, + PBS_SECRET_ENV_FILE: join(example, 'runtime/examples/pbs-secrets.env'), + }) + ).services.pbs + const port = service.ports[0] + assert.equal(port.host_ip, '0.0.0.0') + assert.equal(port.published, '8000') + assert.equal(port.target, 8000) +} + +function checkSmoke() { + const directory = mkdtempSync(join(tmpdir(), 'pbs-smoke-wiring-')) + try { + writeFileSync( + join(directory, 'docker'), + `#!/usr/bin/env node +const assert = require('node:assert/strict') +const { execFileSync } = require('node:child_process') +const { appendFileSync, writeFileSync } = require('node:fs') +const args = process.argv.slice(2) +assert.equal(args[0], 'compose') +const operation = args[7] +assert(['up', 'logs', 'down'].includes(operation)) +appendFileSync(process.env.CALLS, operation + '\\n') +if (operation === 'up') { + const rendered = execFileSync('docker', [...args.slice(0, 7), 'config', '--format', 'json'], { + env: { ...process.env, PATH: process.env.REAL_PATH }, + encoding: 'utf8', + stdio: ['ignore', 'pipe', 'inherit'], + }) + writeFileSync(process.env.RENDERED, rendered) +} +`, + { mode: 0o700 } + ) + writeFileSync( + join(directory, 'curl'), + `#!/usr/bin/env node +const assert = require('node:assert/strict') +assert.equal(process.argv.at(-1), 'http://127.0.0.1:18081/status') +console.log('ok') +`, + { mode: 0o700 } + ) + run(join(example, 'scripts/smoke-runtime.sh'), [], { + ...env, + PATH: `${directory}${delimiter}${env.PATH}`, + REAL_PATH: env.PATH, + CALLS: join(directory, 'calls'), + RENDERED: join(directory, 'rendered.json'), + PBS_SMOKE_PORT: '18081', + PBS_HOST_PORT: '19000', + PBS_BIND_ADDRESS: '0.0.0.0', + PBS_CONFIG_FILE: '/nonexistent/inherited-pbs.yaml', + PBS_SECRET_ENV_FILE: '/nonexistent/inherited-secrets.env', + }) + assert.deepEqual( + readFileSync(join(directory, 'calls'), 'utf8').trim().split('\n'), + ['up', 'logs', 'down'] + ) + const service = JSON.parse( + readFileSync(join(directory, 'rendered.json'), 'utf8') + ).services.pbs + const port = service.ports[0] + assert.equal(port.host_ip, '127.0.0.1') + assert.equal(port.published, '18081') + assert.equal(port.target, 8000) + assert.equal(service.volumes[0].source, join(example, 'runtime/pbs.yaml')) + assert.equal( + service.environment.PBS_ADAPTERS_EXAMPLEBIDDER_API_KEY, + 'example-only-api-key' + ) + assert.equal( + service.environment.PBS_ADAPTERS_EXAMPLEBIDDER_OPTIONAL_TOKEN, + 'example-only-optional-token' + ) + } finally { + rmSync(directory, { recursive: true, force: true }) + } +} + +JSON.parse( + readFileSync(join(example, 'runtime/secret-bindings.example.json'), 'utf8') +) +renderProduction() +checkSmoke() +console.log( + 'Runtime JSON, Compose production/smoke bindings and dummy selectors passed; no containers started.' +) diff --git a/deploy/pbs-example/terraform.tf b/deploy/pbs-example/terraform.tf new file mode 100644 index 000000000..b857d386f --- /dev/null +++ b/deploy/pbs-example/terraform.tf @@ -0,0 +1,14 @@ +terraform { + required_version = ">= 1.16.2, < 1.17.0" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "= 6.64.0" + } + } + + backend "local" { + path = "terraform.tfstate" + } +} diff --git a/deploy/pbs-example/terraform.tfvars.example b/deploy/pbs-example/terraform.tfvars.example new file mode 100644 index 000000000..4c544dbbf --- /dev/null +++ b/deploy/pbs-example/terraform.tfvars.example @@ -0,0 +1,19 @@ +# Fictional local example values. Do not use these identifiers for AWS access. +aws_account_id = "123456789012" +aws_profile = "pbs-example" + +east_ami_id = "ami-0123456789abcdef0" +east_certificate_arn = "arn:aws:acm:us-east-1:123456789012:certificate/11111111-1111-4111-8111-111111111111" +west_ami_id = "ami-0fedcba9876543210" +west_certificate_arn = "arn:aws:acm:us-west-2:123456789012:certificate/22222222-2222-4222-8222-222222222222" + +# Existing hosted-zone ID in the fictional example account. +route53_zone_id = "Z00000000000000000000" + +# Optional regional customer-managed keys. Keep null to use each region's +# Secrets Manager service key. +# east_secrets_kms_key_arn = "arn:aws:kms:us-east-1:123456789012:key/11111111-1111-4111-8111-111111111111" +# west_secrets_kms_key_arn = "arn:aws:kms:us-west-2:123456789012:key/22222222-2222-4222-8222-222222222222" + +# Replace with the actual egress ranges approved for Trusted Server. +trusted_server_cidr_blocks = ["203.0.113.0/24"] diff --git a/deploy/pbs-example/tests/root_unit_test.tftest.hcl b/deploy/pbs-example/tests/root_unit_test.tftest.hcl new file mode 100644 index 000000000..459352eb6 --- /dev/null +++ b/deploy/pbs-example/tests/root_unit_test.tftest.hcl @@ -0,0 +1,166 @@ +mock_provider "aws" { + mock_data "aws_partition" { + defaults = { partition = "aws" } + } + + mock_resource "aws_eip" { + defaults = { + public_ip = "192.0.2.10" + } + } + + mock_resource "aws_lb" { + defaults = { + dns_name = "east-alb.example.com" + zone_id = "Z00000000000000000001" + } + } + + mock_resource "aws_secretsmanager_secret" { + override_during = plan + + defaults = { + arn = "arn:aws:secretsmanager:us-east-1:123456789012:secret:example-AbCdEf" + } + } +} + +mock_provider "aws" { + alias = "west" + + mock_data "aws_partition" { + defaults = { partition = "aws" } + } + + mock_resource "aws_eip" { + defaults = { + public_ip = "192.0.2.20" + } + } + + mock_resource "aws_lb" { + defaults = { + dns_name = "west-alb.example.com" + zone_id = "Z00000000000000000002" + } + } + + mock_resource "aws_secretsmanager_secret" { + override_during = plan + + defaults = { + arn = "arn:aws:secretsmanager:us-west-2:123456789012:secret:example-GhIjKl" + } + } +} + +override_resource { + target = module.east.aws_instance.pbs["us-east-1a"] + override_during = plan + values = { id = "i-00000000000000001" } +} + +override_resource { + target = module.east.aws_instance.pbs["us-east-1b"] + override_during = plan + values = { id = "i-00000000000000002" } +} + +override_resource { + target = module.west.aws_instance.pbs["us-west-2a"] + override_during = plan + values = { id = "i-00000000000000003" } +} + +override_resource { + target = module.west.aws_instance.pbs["us-west-2b"] + override_during = plan + values = { id = "i-00000000000000004" } +} + +variables { + aws_account_id = "123456789012" + aws_profile = "pbs-example" + east_ami_id = "ami-0123456789abcdef0" + east_certificate_arn = "arn:aws:acm:us-east-1:123456789012:certificate/11111111-1111-4111-8111-111111111111" + route53_zone_id = "Z00000000000000000000" + west_ami_id = "ami-0fedcba9876543210" + west_certificate_arn = "arn:aws:acm:us-west-2:123456789012:certificate/22222222-2222-4222-8222-222222222222" +} + +run "plans_both_regional_provider_mappings" { + command = plan + + assert { + condition = length(distinct(values(module.east.instance_ids))) == 2 + error_message = "The east provider mapping should plan two PBS instances." + } + + assert { + condition = length(distinct(values(module.west.instance_ids))) == 2 + error_message = "The west provider mapping should plan two PBS instances." + } + + assert { + condition = ( + length(local.deployment_descriptor.regions) == 2 && + alltrue([for region in local.deployment_descriptor.regions : length(distinct(region.instance_ids)) == 2]) + ) + error_message = "The generated deployment descriptor should include two distinct instance IDs in each region." + } + + assert { + condition = toset(keys(local.secret_bindings.examplebidder.secrets)) == toset(["us-east-1", "us-west-2"]) + error_message = "The generated binding should name an independent secret in each region." + } + + assert { + condition = yamldecode(output.deployment_descriptor_yaml).pbs.bindings == "runtime/secret-bindings.generated.json" + error_message = "The generated descriptor should reference the ignored generated binding file." + } + + assert { + condition = jsondecode(output.secret_bindings_json).examplebidder.secrets.us-east-1 != jsondecode(output.secret_bindings_json).examplebidder.secrets.us-west-2 + error_message = "The rendered binding should contain distinct regional secret ARNs." + } +} + +run "rejects_duplicate_east_availability_zones" { + command = plan + + variables { + east_availability_zones = ["us-east-1a", "us-east-1a"] + } + + expect_failures = [var.east_availability_zones] +} + +run "rejects_empty_caller_allowlist" { + command = plan + + variables { + trusted_server_cidr_blocks = [] + } + + expect_failures = [var.trusted_server_cidr_blocks] +} + +run "rejects_east_key_from_wrong_region" { + command = plan + + variables { + east_secrets_kms_key_arn = "arn:aws:kms:us-west-2:123456789012:key/11111111-1111-4111-8111-111111111111" + } + + expect_failures = [var.east_secrets_kms_key_arn] +} + +run "rejects_west_key_from_wrong_account" { + command = plan + + variables { + west_secrets_kms_key_arn = "arn:aws:kms:us-west-2:999999999999:key/22222222-2222-4222-8222-222222222222" + } + + expect_failures = [var.west_secrets_kms_key_arn] +} diff --git a/deploy/pbs-example/variables.tf b/deploy/pbs-example/variables.tf new file mode 100644 index 000000000..2b3775cc6 --- /dev/null +++ b/deploy/pbs-example/variables.tf @@ -0,0 +1,155 @@ +variable "alarm_actions" { + description = "SNS topic ARNs that should receive CloudWatch alarm notifications. Leave empty for this local example." + type = set(string) + default = [] +} + +variable "aws_account_id" { + description = "The 12-digit AWS account expected by Terraform. Replace the fictional example value before any authorized plan." + type = string + + validation { + condition = can(regex("^[0-9]{12}$", var.aws_account_id)) + error_message = "aws_account_id must be a 12-digit AWS account ID." + } +} + +variable "aws_profile" { + description = "The short-lived AWS CLI profile used by an authorized operator." + type = string +} + +variable "east_ami_id" { + description = "Pre-baked us-east-1 AMI containing the approved host runtime, SSM agent, Docker, and Compose." + type = string + + validation { + condition = can(regex("^ami-[0-9a-f]+$", var.east_ami_id)) + error_message = "east_ami_id must be an AMI ID." + } +} + +variable "east_certificate_arn" { + description = "Existing ACM certificate ARN for the us-east-1 regional ALB." + type = string + + validation { + condition = can(regex("^arn:aws[a-z-]*:acm:us-east-1:[0-9]{12}:certificate/.+$", var.east_certificate_arn)) + error_message = "east_certificate_arn must be an ACM certificate ARN in us-east-1." + } +} + +variable "east_secrets_kms_key_arn" { + description = "Optional us-east-1 customer-managed KMS key ARN for bidder secrets. Null uses the regional Secrets Manager service key." + type = string + default = null + + validation { + condition = var.east_secrets_kms_key_arn == null || can(regex("^arn:aws[a-z-]*:kms:us-east-1:${var.aws_account_id}:key/.+$", var.east_secrets_kms_key_arn)) + error_message = "east_secrets_kms_key_arn must be a KMS key ARN in us-east-1 and the declared AWS account." + } +} + +variable "east_vpc_cidr" { + description = "CIDR block for the us-east-1 VPC." + type = string + default = "10.80.0.0/16" +} + +variable "east_availability_zones" { + description = "Two us-east-1 Availability Zones used by the ALB and PBS host." + type = list(string) + default = ["us-east-1a", "us-east-1b"] + + validation { + condition = length(var.east_availability_zones) == 2 && length(distinct(var.east_availability_zones)) == 2 + error_message = "east_availability_zones must contain two distinct Availability Zones." + } +} + +variable "instance_type" { + description = "EC2 instance type for each PBS host. This is a starting assumption, not a capacity guarantee." + type = string + default = "c7i.large" +} + +variable "pbs_hostname" { + description = "Shared HTTPS hostname returned by Route 53 latency records." + type = string + default = "pbs.example.com" + + validation { + condition = !can(regex("[/\\s]", var.pbs_hostname)) + error_message = "pbs_hostname must be a hostname without a path or whitespace." + } +} + +variable "route53_zone_id" { + description = "Existing Route 53 public hosted zone ID that owns pbs_hostname." + type = string +} + +variable "tags" { + description = "Additional tags applied to supported AWS resources." + type = map(string) + default = {} +} + +variable "trusted_server_cidr_blocks" { + description = "Approved Trusted Server egress CIDR blocks allowed to reach the public ALBs. The default is documentation-only." + type = set(string) + default = ["203.0.113.0/24"] + + validation { + condition = length(var.trusted_server_cidr_blocks) > 0 && alltrue([for cidr in var.trusted_server_cidr_blocks : can(cidrhost(cidr, 0))]) + error_message = "trusted_server_cidr_blocks must contain at least one valid CIDR block." + } +} + +variable "west_ami_id" { + description = "Pre-baked us-west-2 AMI containing the approved host runtime, SSM agent, Docker, and Compose." + type = string + + validation { + condition = can(regex("^ami-[0-9a-f]+$", var.west_ami_id)) + error_message = "west_ami_id must be an AMI ID." + } +} + +variable "west_certificate_arn" { + description = "Existing ACM certificate ARN for the us-west-2 regional ALB." + type = string + + validation { + condition = can(regex("^arn:aws[a-z-]*:acm:us-west-2:[0-9]{12}:certificate/.+$", var.west_certificate_arn)) + error_message = "west_certificate_arn must be an ACM certificate ARN in us-west-2." + } +} + +variable "west_secrets_kms_key_arn" { + description = "Optional us-west-2 customer-managed KMS key ARN for bidder secrets. Null uses the regional Secrets Manager service key." + type = string + default = null + + validation { + condition = var.west_secrets_kms_key_arn == null || can(regex("^arn:aws[a-z-]*:kms:us-west-2:${var.aws_account_id}:key/.+$", var.west_secrets_kms_key_arn)) + error_message = "west_secrets_kms_key_arn must be a KMS key ARN in us-west-2 and the declared AWS account." + } +} + +variable "west_vpc_cidr" { + description = "CIDR block for the us-west-2 VPC." + type = string + default = "10.81.0.0/16" +} + +variable "west_availability_zones" { + description = "Two us-west-2 Availability Zones used by the ALB and PBS host." + type = list(string) + default = ["us-west-2a", "us-west-2b"] + + validation { + condition = length(var.west_availability_zones) == 2 && length(distinct(var.west_availability_zones)) == 2 + error_message = "west_availability_zones must contain two distinct Availability Zones." + } +} diff --git a/docs/guide/cli.md b/docs/guide/cli.md index ab013051b..8b851bd82 100644 --- a/docs/guide/cli.md +++ b/docs/guide/cli.md @@ -44,7 +44,13 @@ commands are shown explicitly. | `ts dev proxy ca uninstall` | macOS only | Remove the CA from the OS trust store | `ts dev proxy ca uninstall` | | `ts healthcheck` | Linux + macOS | Probe a deployed version until it reports healthy | `ts healthcheck [OPTIONS] --adapter --domain --service-id --version ` | | `ts prebid` | Linux + macOS | Trusted Server Prebid commands | `ts prebid ` | -| `ts prebid bundle` | Linux + macOS | Generate a local external Prebid bundle and update config metadata | `ts prebid bundle [OPTIONS]` | +| `ts prebid client` | Linux + macOS | Generate a local external Prebid client bundle and update config metadata | `ts prebid client [OPTIONS]` | +| `ts prebid server` | Linux + macOS | Configure and operate a self-hosted Prebid Server deployment | `ts prebid server [OPTIONS] ` | +| `ts prebid server inspect` | Linux + macOS | Inspect local Trusted Server configuration without modifying it or contacting AWS | `ts prebid server inspect [OPTIONS]` | +| `ts prebid server check` | Linux + macOS | Validate declared PBS inputs and regional YAML merges locally, without AWS access | `ts prebid server check [OPTIONS] --deployment ` | +| `ts prebid server status` | Linux + macOS | Read EC2 infrastructure status, not PBS health or the installed release | `ts prebid server status [OPTIONS] --deployment ` | +| `ts prebid server secrets` | Linux + macOS | Manage values for existing, explicitly declared Secrets Manager secrets | `ts prebid server secrets [OPTIONS] ` | +| `ts prebid server secrets set` | Linux + macOS | Write a complete JSON credential payload; does not deploy or rotate partner credentials | `ts prebid server secrets set [OPTIONS] --deployment --region ` | | `ts provision` | Linux + macOS | Provision platform resources through a target adapter | `ts provision [OPTIONS] --adapter ` | | `ts rollback` | Linux + macOS | Roll a service back to a previously active deployment version | `ts rollback [OPTIONS] --adapter --service-id --version ` | | `ts serve` | Linux + macOS | Serve the project locally through a target adapter | `ts serve --adapter ` | @@ -661,9 +667,13 @@ APIs. ## Generate an external Prebid bundle -`ts prebid bundle` builds the local external Prebid browser bundle configured in +`ts prebid client` builds the local external Prebid browser bundle configured in `trusted-server.toml`. +> This command was previously `ts prebid bundle`. Update scripts and runbooks to +> use `ts prebid client` with the same arguments. The old spelling is retired and +> is not accepted as an alias. + ```toml [integrations.prebid.bundle.modules] bidder = ["rubiconBidAdapter", "kargoBidAdapter"] @@ -681,7 +691,7 @@ Run the command after installing JS dependencies: ```bash cd crates/trusted-server-js/lib && npm ci cd ../../.. -ts prebid bundle +ts prebid client ``` By default, generated artifacts are written to `dist/prebid/`. The versioned @@ -697,8 +707,122 @@ HTTPS asset URL, and include that host plus any redirect targets in Use custom paths when needed: ```bash -ts prebid bundle --config publisher-a.toml --out build/prebid +ts prebid client --config publisher-a.toml --out build/prebid ``` -`ts prebid bundle` is local-only. It has no `--adapter` option and does not +`ts prebid client` is local-only. It has no `--adapter` option and does not upload, provision, deploy, or push config. + +## Operate a self-hosted Prebid Server + +The experimental `ts prebid server` namespace supports four bounded operations: + +- `inspect` reads selected local Trusted Server configuration. +- `check` validates a deployment descriptor and regional configuration locally. +- `secrets set` writes one approved value to an existing AWS Secrets Manager secret. +- `status` reads declared EC2 infrastructure state, not PBS health or readiness. + +AWS operations require AWS CLI v2 on `PATH`; `inspect` and `check` do not contact +AWS. Add `--json` anywhere under `ts prebid server` for machine-readable output. +Failures use exit code 2, including incomplete `status` reports that still write +partial JSON to stdout. + +### Deployment descriptor and bindings + +Pass an explicit `--deployment `; there is no automatic discovery or +production default. This fictional descriptor is suitable only for local checks: + +```yaml +schema_version: 1 +environment: sandbox +runtime: ec2-compose +aws: + account_id: '123456789012' + profile: pbs-sandbox +pbs: + config: pbs.yaml + image: registry.example.com/pbs@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa + bindings: bindings.json +regions: + us-east-1: + instance_ids: + - i-0123456789abcdef0 + overrides: east.yaml +``` + +Version 1 requires an explicit environment, `ec2-compose` runtime, 12-digit AWS +account ID, AWS CLI profile, digest-pinned image, baseline YAML path, and nonempty +region map. Environment and bidder identifiers use letters, digits, underscores, +and hyphens; profile names may also contain periods. Instance IDs are optional, +but `status` needs explicitly listed instances to report infrastructure health. +Regional override paths and `pbs.bindings` are optional. Omit bindings when no +host secrets are needed. + +Paths resolve relative to the descriptor. Unknown schema fields, duplicate YAML +keys, tags, and implicit YAML merge keys are rejected. Regional mappings merge +recursively over the baseline; sequences and scalars replace whole values. The +CLI validates in memory and writes no rendered files. + +The binding file is a JSON or YAML mapping keyed by bidder identifier. Each +entry requires: + +- `verified_image`, exactly matching the descriptor's image. +- `source`, an HTTPS reference used by the operator to verify the mapping. +- `secrets`, a complete Secrets Manager ARN for each descriptor region, matching + its account and region. Each secret must belong to only one bidder binding. +- `keys`, mapping credential names to an environment variable `env`, a YAML path + array `pbs_path`, and optional `required`, which defaults to true. + +For example, a key mapping can be: + +```json +{ + "api_key": { + "env": "PBS_ADAPTERS_EXAMPLEBIDDER_API_KEY", + "pbs_path": ["adapters", "examplebidder", "api_key"], + "required": true + } +} +``` + +Bindings support string-valued credentials. Environment variables and YAML +destinations must not conflict across bindings or with values already present in +the resolved YAML. These are operator-supplied mappings, not a verified adapter +catalog. `check` does not validate the complete upstream PBS schema, verify +bidder authorization, retrieve secrets, pull images, or prove application health. + +### Secret-write safeguards + +`secrets set` replaces the complete JSON value of an existing declared secret. +It does not create secret metadata, deploy PBS, refresh a container, or rotate a +bidder's credential. Use a trusted host and short-lived AWS credentials. Obtain +separate approval for each target and value write. + +Before writing, the CLI verifies the account through STS, describes the declared +secret, rejects replicas and secrets scheduled for deletion, and checks both the +selected profile's and default AWS CLI history settings. History must be disabled +or unset. Provider stderr is withheld and configured API endpoint overrides are +disabled. + +Interactive use reads JSON in a hidden terminal prompt and requires typing +`yes`. For approved automation, use `--file ` or `--stdin` with `--yes` and +an explicit `--request-token `. The payload must contain only declared keys, +include required nonempty string values, and fit the Secrets Manager size limit. +Duplicate keys and non-string values are rejected. + +Secret contents do not enter arguments or reports. The AWS request goes through a +tool-owned temporary file, owner-only on Unix, removed on normal success and +error paths. Abrupt termination can leave that file behind; Windows temporary-file +ACLs have not been verified. Operator-provided input files are never deleted. +Input-file errors print the full escaped path to stderr, including under `--json`. +Keep sensitive directory and partner names out of public logs. + +A successful write reports its version identifier, not runtime readiness. A failed +or unverifiable response means the outcome may be uncertain. Preserve the request +UUID and input. Reuse that UUID only with the original identical payload; use a +new UUID only for a separately intended update. Do not retry changed values under +the same UUID. `--yes` authorizes only this write, not deployment or future writes. + +See the +[experimental PBS command reference](https://github.com/IABTechLab/trusted-server/blob/main/crates/trusted-server-cli/README.md) +for complete binding fixtures, command examples, and remaining runtime limits. diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 09a59a01f..f7220d58c 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -1662,9 +1662,9 @@ overrides, and notification suppression belong under `[auction]`. | `script_patterns` | Array[String] | `["/prebid.js", "/prebid.min.js", "/prebidjs.js", "/prebidjs.min.js"]` | Publisher Prebid script paths intercepted by Trusted Server | | `external_bundle_url` | String | Required when enabled | HTTPS publisher-specific Prebid.js bundle URL | | `external_bundle_sha256` / `*_sri` | String | `None` | Optional bundle integrity and cache metadata | -| `bundle.modules.bidder` | Array[String] | Required and non-empty | Exact bidder module stems used by `ts prebid bundle` | -| `bundle.modules.user_id` | Array[String] | Curated preset when omitted | Exact User ID module stems used by `ts prebid bundle` | -| `bundle.modules.analytics` | Array[String] | `[]` | Exact analytics module stems used by `ts prebid bundle` | +| `bundle.modules.bidder` | Array[String] | Required and non-empty | Exact bidder module stems used by `ts prebid client` | +| `bundle.modules.user_id` | Array[String] | Curated preset when omitted | Exact User ID module stems used by `ts prebid client` | +| `bundle.modules.analytics` | Array[String] | `[]` | Exact analytics module stems used by `ts prebid client` | | `managed_user_ids` | Array[Table] | `[]` | Prebid User ID modules Trusted Server installs and keeps installed (see below) | Server-side bidder codes are derived from validated `[auction.bidders.*]` @@ -1758,7 +1758,7 @@ module uses the same vendor-neutral surface. The managed `name` must match a The module must be present in the built bundle. Name it under `[integrations.prebid.bundle].user_id_modules`, or omit that list to take the -generator's default preset. `ts prebid bundle` resolves each managed `name` +generator's default preset. `ts prebid client` resolves each managed `name` through the checked-in `user_id_modules.json` registry, rejects unknown names, ambiguous names, and two names that resolve to the same module, and confirms the required modules in the newly generated diff --git a/docs/guide/integrations/prebid.md b/docs/guide/integrations/prebid.md index 3610cbde7..8b9394ffb 100644 --- a/docs/guide/integrations/prebid.md +++ b/docs/guide/integrations/prebid.md @@ -45,7 +45,7 @@ name = "idl_env" expires = 15 refresh_in_seconds = 1800 -# External bundle generation inputs used by `ts prebid bundle`. +# External bundle generation inputs used by `ts prebid client`. # Values are exact Prebid module stems without `.js`. [integrations.prebid.bundle.modules] bidder = ["rubiconBidAdapter"] @@ -198,11 +198,11 @@ versioned contract change. There is no time-based expiry in this fix. ## External Bundle Generation -Use `ts prebid bundle` to build the publisher-specific browser bundle from +Use `ts prebid client` to build the publisher-specific browser bundle from `[integrations.prebid.bundle.modules]` selections: ```bash -ts prebid bundle +ts prebid client ``` The command writes generated artifacts to `dist/prebid/` by default and updates @@ -248,7 +248,7 @@ upstream stems. For example, `adapters = ["rubicon"]` becomes `bidder = ["rubiconBidAdapter"]`; `client_side_bidders` continues to use the runtime code `rubicon`. -`ts prebid bundle` rejects the removed `adapters`, `user_id_modules`, and +`ts prebid client` rejects the removed `adapters`, `user_id_modules`, and `analytics_adapters` fields with the replacement path. Runtime config validation, `ts config push`, and server startup also reject the old bundle fields. @@ -645,7 +645,7 @@ bidder = ["rubiconBidAdapter", "appnexusBidAdapter", "openxBidAdapter"] user_id = ["sharedIdSystem", "uid2IdSystem"] ``` -Run `ts prebid bundle` after changing the module list. The generator resolves +Run `ts prebid client` after changing the module list. The generator resolves `prebid.js/modules/.js` through the pinned package and records both stems and registered bidder codes in `manifest.json`. At runtime, TSJS checks each `client_side_bidders` runtime code against that manifest. @@ -749,7 +749,7 @@ The module must be present in the built bundle. Name it under `bundle.user_id_modules`, or omit that list to take the generator's default preset, which covers the commonly used modules. -`ts prebid bundle` resolves every managed `name` through the checked-in +`ts prebid client` resolves every managed `name` through the checked-in `user_id_modules.json` registry. An unknown name, a name that maps to more than one module, or two managed names that resolve to the same module — `sharedId` and `pubCommonId` both select `sharedIdSystem`, for example — fail before bundle @@ -780,7 +780,7 @@ expires = 15 refresh_in_seconds = 1800 ``` -Run `ts prebid bundle`, upload the generated content-addressed bundle, copy its +Run `ts prebid client`, upload the generated content-addressed bundle, copy its hash metadata into `[integrations.prebid]`, and validate the configuration before rollout. diff --git a/docs/superpowers/plans/2026-07-24-prebid-refresh-gam-path-opt-out.md b/docs/superpowers/plans/2026-07-24-prebid-refresh-gam-path-opt-out.md index edb0e27db..46f12cf79 100644 --- a/docs/superpowers/plans/2026-07-24-prebid-refresh-gam-path-opt-out.md +++ b/docs/superpowers/plans/2026-07-24-prebid-refresh-gam-path-opt-out.md @@ -2,6 +2,9 @@ **Design:** `docs/superpowers/specs/2026-07-24-prebid-refresh-gam-path-opt-out-design.md` +> **Update (2026-09-16):** The `ts prebid bundle` command referenced in this +> plan was later renamed to `ts prebid client`. + **Goal:** Let operators exclude selected GAM ad-unit-path suffixes from Trusted Server's Prebid refresh auctions without suppressing the corresponding GAM refresh. diff --git a/docs/superpowers/specs/2026-06-17-prebid-bundle-cli-design.md b/docs/superpowers/specs/2026-06-17-prebid-bundle-cli-design.md index 69a8f35de..8e8701579 100644 --- a/docs/superpowers/specs/2026-06-17-prebid-bundle-cli-design.md +++ b/docs/superpowers/specs/2026-06-17-prebid-bundle-cli-design.md @@ -3,6 +3,10 @@ **Date:** 2026-06-17 **Status:** Implemented **Scope:** `ts prebid bundle` local external Prebid bundle generation + +> **Update (2026-09-16):** The command described here was later renamed to +> `ts prebid client` when the `ts prebid server` namespace was introduced. + **Related context:** - `docs/superpowers/specs/2026-05-28-external-prebid-first-party-proxy-design.md` diff --git a/trusted-server.example.toml b/trusted-server.example.toml index f552c989e..e841515ae 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -491,11 +491,11 @@ client_side_bidders = [] # bidders running via native Prebid.js adapter # Keep selected GAM inventory out of Trusted Server's Prebid refresh auctions. # Matching slots still refresh through GAM. # excluded_gam_ad_unit_path_suffixes = ["/trackingonly"] -# Runtime bundle metadata — set after running `ts prebid bundle` and uploading: +# Runtime bundle metadata — set after running `ts prebid client` and uploading: # external_bundle_url = "https://assets.example.com/prebid/trusted-prebid-.js" # external_bundle_sha256 = "" # external_bundle_sri = "" -# Bundle build inputs consumed by `ts prebid bundle`, not by the edge runtime. +# Bundle build inputs consumed by `ts prebid client`, not by the edge runtime. # Values are exact upstream module stems without `.js`. # [integrations.prebid.bundle.modules] # bidder = ["rubiconBidAdapter"] @@ -512,8 +512,8 @@ client_side_bidders = [] # bidders running via native Prebid.js adapter # the checked-in User ID registry. Each `name` must appear only once. # # The module must be present in the built bundle: name it under -# [integrations.prebid.bundle].user_id_modules, or omit that list to take the -# generator's default preset. `ts prebid bundle` resolves every managed name +# [integrations.prebid.bundle.modules].user_id, or omit that list to take the +# generator's default preset. `ts prebid client` resolves every managed name # through the checked-in User ID registry and fails if the generated manifest # omits its required module without updating the configured hash or SRI. #