diff --git a/docs/plans/2026-07-13-sdk-evolution-demo-reliability.md b/docs/plans/2026-07-13-sdk-evolution-demo-reliability.md new file mode 100644 index 0000000..d0104b0 --- /dev/null +++ b/docs/plans/2026-07-13-sdk-evolution-demo-reliability.md @@ -0,0 +1,444 @@ +# SDK Evolution Demo Reliability Repair Plan + +Status: implementation-ready +Written: 2026-07-13 +Baseline: `origin/main` at `2e7e4c3` (`v0.5.0`) + +## Outcome + +Make the SDK evolution demo produce truthful, actionable evidence from the +documented local workflow, including when only the selected runtime extra is +installed. A successful decision run must compare an observed locked SDK +baseline with an observed resolver-selected candidate, report incomplete +evidence as incomplete, and preserve the existing fail-closed implementation +gates. + +This plan is intentionally narrower than the historical overhaul in draft PR +#40. It fixes the defects reproduced on current `main` without importing the +older branch's unrelated transaction, resolver, scheduling, or implementation +changes. + +## Reproduced behavior + +The following behavior was reproduced from a fresh `origin/main` worktree: + +1. The focused SDK evolution suite passes (`50 passed`). +2. The full suite passes (`418 passed, 3 skipped`), as do Ruff, mypy, and + `uv lock --check`. +3. A normal Codex-backed report runs all three structured AI stages and reuses + one Codex SDK process, but its behavior headline says `pass` while the raw + Claude and Antigravity baseline probes contain import failures. +4. Adding `--inspect-candidates` installs and probes the candidates, but the + absent locked SDKs are still probed through the ambient environment. The + report therefore compares `fail` to `pass`, emits no candidate API diff, and + is rejected by the deterministic API-diff gate. +5. Installing every vendor extra before the same command produces two valid API + diffs, passing before/after behavior probes, `safe_to_implement=true`, and a + passing reviewer decision. +6. The `openai-codex-cli-bin` distribution imports as `codex_cli_bin`; the + snapshot map currently asks for `openai_codex_cli_bin` and records a false + import error. +7. The public examples use bare `python -m`. In the fresh worktree that selected + Python 3.9 despite `.python-version` declaring 3.10, and failed before CLI + parsing because `dataclasses.field(kw_only=...)` requires Python 3.10. + +The Agent SDK execution layer is therefore healthy. The broken boundaries are +locked-baseline evidence collection, behavior aggregation, distribution +introspection, and the operator command contract. + +## First-principles model + +### Actors and boundaries + +- `uv.lock` defines the current dependency baseline. +- The active environment may contain only core plus the selected runtime extra. +- `collect_evidence` records locked, installed, latest, and resolver-selected + versions. +- API snapshots and behavior probes observe baseline and candidate packages. +- Diff and summary layers convert observations into evidence. +- Deterministic guards decide whether implementation is allowed. +- AI stages interpret the evidence but cannot override deterministic guards. +- Documentation and repo-local runbooks define the supported operator command. + +### Invariants + +1. A version comparison is valid only when both versions were actually + observed. +2. The current side of an SDK update comparison is the locked version, not an + arbitrary ambient import. +3. Missing or skipped evidence is `incomplete`; it is never equivalent to + `pass` or to a behavioral change. +4. Candidate and isolated-baseline installation remains explicit and runs only + behind `--inspect-candidates` in a credential-scrubbed environment. +5. A required failed or incomplete behavior probe blocks implementation. +6. The documented command supplies the selected runtime dependency and the + explicit candidate-inspection consent required to satisfy the gates. +7. The outer `uv run` is locked and cannot rewrite dependency state while + preparing the environment; the agent's explicit refresh preview remains the + only freshness path. +8. Vendor-specific surfaces remain explicit; this repair does not flatten + provider capabilities or change adapter contracts. + +### Failure dynamics + +The failure is environment-dependent and deterministic. It occurs when a +locked optional SDK is absent or differs from the active environment. It is +masked when all vendor extras happen to be installed, which is why unit tests +and developer environments can appear healthy. + +## Five whys and fix layer + +1. **Why is the report false-green or non-actionable?** The locked Claude and + Antigravity baselines are not observed in the normal Codex-only environment. +2. **Why are they not observed?** Isolated baseline probing requires + `installed_version` to be present and different from `locked_version`; an + absent installation takes the ambient path. +3. **Why does bad evidence reach the report?** The ambient failure is stamped + with the locked version, failed snapshots are omitted from diffs, and the + behavior summary consumes diffs rather than raw probe results. +4. **Why does this happen in the supported workflow?** The runbooks install only + enough for auth and omit `--inspect-candidates` from the decision and + implementation commands; public examples also bypass the repository's + declared Python and lockfile through bare `python -m`. +5. **Why did tests not catch it?** They cover installed-version drift and + opt-in candidate safety, but not a missing optional baseline, raw failure + aggregation, or command/document conformance. + +The proximal symptom is a contradictory report or a deterministic rejection. +The mechanism is invalid provenance followed by loss of missing/failure state. +The trigger is an absent optional SDK in a runtime-specific environment. The +root cause is the evidence contract across collection, aggregation, and +operator invocation. The fix belongs in those layers because they own what was +observed, how it is classified, and how operators request actionable evidence. + +## Stacked delivery model + +Every PR is based on the branch immediately above it. Review each PR against its +declared base so only that phase's delta is visible. + +| PR | Branch | Base | Scope | +| --- | --- | --- | --- | +| 1 | `agent/sdk-evolution-demo-reliability-plan` | `main` | This plan only | +| 2 | `agent/sdk-evolution-locked-baselines` | PR 1 branch | Observe the locked baseline correctly | +| 3 | `agent/sdk-evolution-evidence-integrity` | PR 2 branch | Make summaries and gates fail closed | +| 4 | `agent/sdk-evolution-cli-bin-snapshots` | PR 3 branch | Fix CLI binary introspection | +| 5 | `agent/sdk-evolution-operator-contract` | PR 4 branch | Make supported commands actionable | + +All PRs remain draft until their phase checks and ordinary correctness review +pass. Reviews must not run security-diff scans, per the delivery instruction. + +## PR 2: Observe locked baselines correctly + +### Production changes + +- In `collect_behavior_evidence`, treat `installed_version is None` the same as + installed-version drift when `locked_version` exists. +- With `--inspect-candidates`, probe that locked version in the existing + credential-scrubbed isolated environment and label it `current-baseline`. +- Without the opt-in, do not install anything and do not relabel an ambient + import as the locked baseline. Record only the actual ambient installed + version (or `None` when absent); PR 3 classifies that unavailable locked + comparison as incomplete. +- Apply the same missing-or-drifted decision to `_collect_snapshots`: with the + opt-in, snapshot the locked version in an isolated environment; without it, + retain only an explicitly unusable current-environment snapshot whose + version is the observed installed version, not the requested lock. +- Preserve the fast ambient path when installed and locked versions match. +- Convert isolated snapshot venv creation, package installation, execution, + timeout, and malformed-output failures into bounded structured snapshot + evidence rather than aborting the report. Preserve package, requested version, + and the scrubbed-environment boundary; do not copy unbounded subprocess output + into artifacts. Behavior subprocess error handling lands atomically with its + summary and guard in PR 3. + +### Tests + +- Missing installed SDK plus opt-in probes the locked version as + `current-baseline` before the candidate. +- Missing installed SDK without opt-in performs no isolated install and never + stamps the locked version onto the ambient result. +- Snapshot collection isolates a missing locked baseline when opted in. +- Snapshot collection performs no isolated install without the flag. +- A matching installed/locked version remains eligible for the ambient path. + Drifted or absent baselines use `current-baseline` when opted in and remain + explicitly unusable when the flag is off. +- Assert that isolated baseline and candidate subprocesses retain the scrubbed + environment contract. +- Snapshot venv creation, install, execution, timeout, and malformed-output + failures produce an explicit failed snapshot artifact and a completed report. + +### Files + +- `examples/sdk_evolution_agent/behavior.py` +- `examples/sdk_evolution_agent/snapshots.py` +- `examples/sdk_evolution_agent/cli.py` +- `tests/test_sdk_evolution_agent.py` + +### Acceptance + +- A core/Codex-only environment with `--inspect-candidates` produces valid + locked-to-candidate inputs without requiring `[all]`. +- No freshly downloaded package is installed or imported when the flag is off. +- No `current-environment` failure is presented as an observed locked baseline. + +## PR 3: Make evidence summaries and gates truthful + +### Production changes + +- Change behavior summarization to consume raw probe results as well as diffs. +- Introduce one shared resolver-transition parser returning exact + `(package, from_version, to_version)` tuples and use it for snapshot/behavior + collection plus deterministic gates. Cover prerelease and multiple-update + output so duplicated weaker parsers cannot drift. +- Define four report states: + - `pass`: required observations exist and no contract change is detected. + - `changed`: required observations exist and only non-breaking change exists. + - `incomplete`: a required observation was skipped, missing, malformed, or + failed to execute/import/install. + - `fail`: a probe completed and proved a required contract failure, or a + breaking diff exists. +- Use deterministic precedence `fail > incomplete > changed > pass`. A failure + with `details.missing` is a proven contract failure; `details.error`, skips, + and missing comparisons are incomplete evidence. +- Preserve existing summary keys and add contract-failure, probe-error, skipped, + and missing-comparison counts plus concise reasons. +- Persist the additive summary as `behavior_summary.json` and render the same + counts/reasons in the Markdown report. +- Add `behavior_summary.json` to `current_state.json` artifact references so its + path and hash participate in the durable baseline manifest. +- Keep one-sided skips out of behavior diffs; absence is not a change. +- Compute missing comparisons only for resolver-selected transitions. A package + with a passing baseline and no selected update has no required candidate and + remains eligible for `pass`. +- Convert behavior venv creation, install, probe execution, timeout, and + malformed-output failures into bounded structured probe evidence. Land this + with classification and gating so a formerly loud failure can never become an + intermediate false-green. +- Use one deterministic assessment helper for collection, reporting, and + gating; never trust a contradictory cached summary over raw results/diffs. +- Extend `with_behavior_probe_guard` so `fail` and `incomplete` both set + `safe_to_implement=false`, set `manual_design_required=true`, and preserve a + concrete reason in findings/uncertainty. +- Fail closed when the behavior summary is missing, malformed, or has an unknown + status. +- Require API-diff evidence to match the exact resolver-selected package, + locked `from_version`, and candidate `to_version`; package membership alone + is insufficient. +- Surface API snapshot import/execution errors in `report.md` so a zero diff + count cannot imply successful inspection. +- Keep `pass` and non-breaking `changed` eligible for later deterministic gates. + +### Tests + +- A baseline execution/import error with no candidate diff reports `incomplete`, + not `pass`. +- A completed baseline contract failure reports `fail`. +- Skipped baseline/candidate evidence reports `incomplete`, not `pass`. +- Candidate install/probe execution failure reports `incomplete` and blocks + implementation; a successfully executed probe with missing required contract + fields reports `fail`. +- A complete unchanged pair reports `pass`. +- A complete non-breaking changed pair reports `changed` without being treated + as breaking. +- Status precedence covers fail plus incomplete and changed plus incomplete. +- A passing baseline with no selected update has + `missing_comparison_count=0`; an expected update with a missing candidate is + incomplete. +- The Markdown report renders status, failure/skip/error counts, reasons, and + snapshot errors; `behavior_summary.json` matches it. +- `current_state.json` records the summary artifact's path and hash. +- The behavior guard blocks `fail` and `incomplete`, while allowing `pass` and + non-breaking `changed` to continue to other gates. +- Missing, non-mapping, unknown-status, and contradictory `pass` summaries with + raw fail/skip evidence all block deterministically. +- The API guard rejects a diff for the correct package with the wrong from/to + versions and accepts an exact empty transition diff. +- Shared transition parsing covers multiple packages and prerelease versions. + +### Files + +- `examples/sdk_evolution_agent/behavior.py` +- `examples/sdk_evolution_agent/collectors.py` +- `examples/sdk_evolution_agent/stages.py` +- `examples/sdk_evolution_agent/report.py` +- `examples/sdk_evolution_agent/current_state.py` +- `docs/sdk-evolution-agent.md` +- `tests/test_sdk_evolution_agent.py` + +### Acceptance + +- The headline can be derived from the raw artifact without contradiction. +- No empty diff set can hide failed or skipped required observations. +- Deterministic gates, not an AI interpretation, own the fail-closed decision. + +## PR 4: Fix Codex CLI binary introspection + +### Production changes + +- Map `openai-codex-cli-bin` to its real import module, `codex_cli_bin`. +- Keep the distribution name unchanged for package metadata and resolver work. +- Strengthen the binary behavior probe to import `codex_cli_bin`, call + `bundled_codex_path()`, and prove the bundled executable exists instead of + treating distribution metadata alone as success. +- Preserve the `binary-distribution` probe name when the isolated embedded probe + raises; it currently hardcodes `adapter-contract`, which prevents correct + before/after pairing. +- Do not introduce a special lowest-common-denominator adapter abstraction; the + snapshot remains package-specific evidence. + +### Tests + +- Current-environment snapshot imports `codex_cli_bin` for the distribution. +- The isolated snapshot script receives the same module name. +- The real installed locked package produces a snapshot without import error + when the Codex extra is available. +- The binary behavior probe passes only when the module and bundled path are + usable, and failures remain labeled `binary-distribution`. +- Existing package mappings remain unchanged. + +### Files + +- `examples/sdk_evolution_agent/snapshots.py` +- `examples/sdk_evolution_agent/behavior.py` +- `tests/test_sdk_evolution_agent.py` + +### Acceptance + +- Current reports no longer contain a synthetic CLI-bin import failure. +- A future resolver-selected CLI-bin update can produce a before/after API diff + instead of being permanently blocked by the wrong module name. + +## PR 5: Make the operator workflow actionable + +### Documentation and runbook changes + +- Update `.codex/skills/agent-runtime-kit-upgrade/SKILL.md` so Codex-backed runs + use `uv run --locked --extra codex` for auth, report, and implementation + passes. +- Update `.claude/commands/agent-runtime-kit/upgrade.md` so Claude-backed runs + use `uv run --locked --extra claude`; document the corresponding Codex and + Antigravity extras when another runtime is selected. +- Couple runtime and extra selection explicitly everywhere: + `claude-agent-sdk` uses `--extra claude`, `codex-agent-sdk` uses + `--extra codex`, and `antigravity-agent-sdk` uses `--extra antigravity`. + Remove instructions that say to replace only `--runtime`. +- Replace bare public `python -m` commands with `uv run --locked` so commands + honor the supported Python version and do not rewrite `uv.lock` while setting + up the outer environment. +- Add `--inspect-candidates` to the actionable report-only decision pass and to + the implementation pass. +- Update `docs/sdk-evolution-agent.md` and its design companion with complete + runtime-specific commands, the `incomplete` status contract, and the new + `behavior_summary.json` artifact. +- Add `behavior_summary.json` to the inspection lists in both repo-owned + runbooks. +- State explicitly that `--inspect-candidates` installs both a missing/drifted + locked baseline and resolver-selected candidates in scrubbed environments. +- Preserve the credential-free fake runtime as a pipeline-shape check, not an + upgrade decision. + +### Tests + +- Add a small documentation-conformance test that asserts each repo-owned + runbook's default real-runtime command includes its required extra, + `uv run --locked`, `--refresh-preview`, and `--inspect-candidates`. +- Assert all three runtime-to-extra mappings. +- Assert implementation examples retain the same inspection flag. +- Reject bare `python -m examples.sdk_evolution_agent` in executable operator + command blocks while allowing explanatory prose. +- Assert the docs still describe candidate inspection as explicit opt-in and + never imply that the fake reviewer proves upgrade safety. + +### Files + +- `.codex/skills/agent-runtime-kit-upgrade/SKILL.md` +- `.claude/commands/agent-runtime-kit/upgrade.md` +- `docs/sdk-evolution-agent.md` +- `docs/sdk-evolution-agent-design.md` +- `tests/test_sdk_evolution_docs.py` + +### Acceptance + +- Copying the documented Claude, Codex, or Antigravity command into a fresh + worktree supplies the selected runtime and can satisfy candidate evidence + gates. +- The command does not require installing every vendor extra. +- Runbooks and public docs agree on opt-in execution, authentication, evidence + states, and stop gates. + +## Cross-stack verification + +Run the smallest relevant test set during development, then all of these from +the top branch: + +```bash +env -u UV_EXCLUDE_NEWER -u UV_EXCLUDE_NEWER_PACKAGE uv lock --check +env -u UV_EXCLUDE_NEWER -u UV_EXCLUDE_NEWER_PACKAGE uv run --locked ruff check . +env -u UV_EXCLUDE_NEWER -u UV_EXCLUDE_NEWER_PACKAGE uv run --locked mypy +env -u UV_EXCLUDE_NEWER -u UV_EXCLUDE_NEWER_PACKAGE uv run --locked pytest -q +``` + +Also rehearse these exact report-only paths in a fresh worktree: + +```bash +# Explicitly incomplete discovery pass: candidates exist but inspection is off. +env -u UV_EXCLUDE_NEWER -u UV_EXCLUDE_NEWER_PACKAGE \ + uv run --locked python -m examples.sdk_evolution_agent \ + --runtime fake \ + --refresh-preview + +# Complete deterministic evidence pass from a core-only environment. +env -u UV_EXCLUDE_NEWER -u UV_EXCLUDE_NEWER_PACKAGE \ + uv run --locked python -m examples.sdk_evolution_agent \ + --runtime fake \ + --refresh-preview \ + --inspect-candidates + +# Complete real-runtime pass with only the selected runtime extra. +env -u UV_EXCLUDE_NEWER -u UV_EXCLUDE_NEWER_PACKAGE \ + uv run --locked --extra codex python -m examples.sdk_evolution_agent \ + --runtime codex-agent-sdk \ + --refresh-preview \ + --inspect-candidates +``` + +The first command must report incomplete candidate evidence. The second must +produce valid locked/candidate comparisons. The third must complete all +structured stages, reuse the Codex process, and give the reviewer valid +evidence. A deterministic no-update regression test must separately prove that +a baseline-only package does not invent a missing candidate or downgrade. + +The live rehearsal remains report-only. It must not enable implementation, +create a branch, push, or open an autonomous upgrade PR. + +## Review protocol + +For each phase: + +1. Review only `base...head` for that PR. +2. Check phase acceptance criteria and tests. +3. Run an ordinary correctness, compatibility, and missing-test review. +4. Do not run a security diff or security scan. +5. Fix material findings on the same phase branch and repeat the targeted + review before publishing the next layer. +6. Verify live GitHub base/head topology after publication. + +## Compatibility and risk controls + +- The public `agent_runtime_kit` API is unchanged. +- Candidate inspection stays opt-in and credential-scrubbed. +- The new `incomplete` value appears only in generated example artifacts; any + downstream parser that assumed three status values must be updated in PR 3. +- No vendor dependency ranges or lockfile versions change in this repair stack. +- No scheduled automation is added. +- No direct vendor model API calls are added. +- Existing open PRs #19 and #40 are not used as stack bases because both carry + stale ancestry and broader changes. Their eventual disposition is separate + from this repair stack. + +## Completion definition + +The stack is complete when all five PRs are published with correct bases, each +phase has passed its targeted tests and non-security review, the top branch +passes the full verification matrix, and a Codex-backed report-only rehearsal +from a runtime-specific environment produces truthful actionable evidence. diff --git a/examples/sdk_evolution_agent/behavior.py b/examples/sdk_evolution_agent/behavior.py index 96317b4..b45a264 100644 --- a/examples/sdk_evolution_agent/behavior.py +++ b/examples/sdk_evolution_agent/behavior.py @@ -41,16 +41,14 @@ def collect_behavior_evidence( continue locked_version = _string_or_none(package.get("locked_version")) installed_version = _string_or_none(package.get("installed_version")) - current_version = locked_version or installed_version if ( inspect_candidates and locked_version - and installed_version and locked_version != installed_version ): results.extend(probe_candidate_in_venv(name, locked_version, scope="current-baseline")) else: - results.extend(probe_current_package(name, version=current_version)) + results.extend(probe_current_package(name, version=installed_version)) candidate = update_versions.get(name) if candidate: if inspect_candidates: diff --git a/examples/sdk_evolution_agent/cli.py b/examples/sdk_evolution_agent/cli.py index b776339..87461df 100644 --- a/examples/sdk_evolution_agent/cli.py +++ b/examples/sdk_evolution_agent/cli.py @@ -526,10 +526,10 @@ def _collect_snapshots(evidence: dict[str, Any], *, inspect_candidates: bool = F locked = package.get("locked_version") installed = package.get("installed_version") baseline = locked or installed - if inspect_candidates and locked and installed and locked != installed: + if inspect_candidates and locked and locked != installed: snapshots.append(snapshot_candidate_in_venv(name, str(locked))) else: - snapshots.append(snapshot_current_api(name, version=baseline)) + snapshots.append(snapshot_current_api(name, version=installed)) if not inspect_candidates: continue candidate = update_versions.get(name) diff --git a/examples/sdk_evolution_agent/snapshots.py b/examples/sdk_evolution_agent/snapshots.py index f198536..31f80e8 100644 --- a/examples/sdk_evolution_agent/snapshots.py +++ b/examples/sdk_evolution_agent/snapshots.py @@ -128,51 +128,111 @@ def snapshot_candidate_in_venv( """Inspect a candidate version in an isolated temporary virtualenv.""" module_name = DEFAULT_MODULES.get(package, package.replace("-", "_")) - with tempfile.TemporaryDirectory(prefix="ark-sdk-snapshot-") as directory: - venv = Path(directory) / ".venv" - # Scrub the environment for every subprocess that touches freshly downloaded - # upstream code: give it a throwaway HOME and only PATH, so a malicious or - # buggy candidate package cannot read the caller's credentials/config. - env = isolated_env(Path(directory)) - subprocess.run( - (python, "-m", "venv", str(venv)), check=True, timeout=timeout, env=env + step = "virtual environment creation" + try: + with tempfile.TemporaryDirectory(prefix="ark-sdk-snapshot-") as directory: + venv = Path(directory) / ".venv" + # Scrub the environment for every subprocess that touches freshly downloaded + # upstream code: give it a throwaway HOME and only PATH, so a malicious or + # buggy candidate package cannot read the caller's credentials/config. + env = isolated_env(Path(directory)) + subprocess.run( + (python, "-m", "venv", str(venv)), + check=True, + timeout=timeout, + env=env, + ) + bin_dir = "Scripts" if sys.platform == "win32" else "bin" + venv_python = venv / bin_dir / "python" + step = "package installation" + subprocess.run( + (str(venv_python), "-m", "pip", "install", f"{package}=={version}"), + check=True, + text=True, + capture_output=True, + timeout=timeout, + env=env, + ) + step = "snapshot execution" + completed = subprocess.run( + ( + str(venv_python), + "-c", + _SNAPSHOT_SCRIPT, + package, + version, + module_name, + ), + check=True, + text=True, + capture_output=True, + timeout=timeout, + env=env, + ) + except subprocess.TimeoutExpired as exc: + return _failed_isolated_snapshot( + package, + version, + module_name, + f"{step} timed out after {exc.timeout}s", ) - bin_dir = "Scripts" if sys.platform == "win32" else "bin" - venv_python = venv / bin_dir / "python" - subprocess.run( - (str(venv_python), "-m", "pip", "install", f"{package}=={version}"), - check=True, - text=True, - capture_output=True, - timeout=timeout, - env=env, + except subprocess.CalledProcessError as exc: + detail = _bounded_failure_detail(exc.stderr or exc.stdout or str(exc)) + return _failed_isolated_snapshot( + package, + version, + module_name, + f"{step} failed: {detail}", ) - completed = subprocess.run( - ( - str(venv_python), - "-c", - _SNAPSHOT_SCRIPT, - package, - version, - module_name, - ), - check=True, - text=True, - capture_output=True, - timeout=timeout, - env=env, + except OSError as exc: + return _failed_isolated_snapshot( + package, + version, + module_name, + f"{step} failed: {_bounded_failure_detail(exc)}", ) - raw = json.loads(completed.stdout) + + try: + raw = json.loads(completed.stdout) + return ApiSnapshot( + package=raw["package"], + version=raw["version"], + module=raw["module"], + members=tuple(ApiMember(**item) for item in raw.get("members", ())), + import_error=raw.get("import_error"), + source="isolated-venv", + ) + except (json.JSONDecodeError, KeyError, TypeError, ValueError) as exc: + output = _bounded_failure_detail(completed.stdout) + detail = f"malformed snapshot output: {exc}" + if output: + detail += f"; stdout={output}" + return _failed_isolated_snapshot(package, version, module_name, detail) + + +def _failed_isolated_snapshot( + package: str, + version: str, + module_name: str, + error: str, +) -> ApiSnapshot: return ApiSnapshot( - package=raw["package"], - version=raw["version"], - module=raw["module"], - members=tuple(ApiMember(**item) for item in raw.get("members", ())), - import_error=raw.get("import_error"), + package=package, + version=version, + module=module_name, + import_error=_bounded_failure_detail(error, limit=560), source="isolated-venv", ) +def _bounded_failure_detail(value: object, *, limit: int = 480) -> str: + if isinstance(value, bytes): + text = value.decode(errors="replace") + else: + text = str(value) + return " ".join(text.split())[:limit] + + def _member_kind(value: Any) -> str: if inspect.isclass(value): return "class" diff --git a/tests/test_sdk_evolution_agent.py b/tests/test_sdk_evolution_agent.py index 81b8817..b3dfec2 100644 --- a/tests/test_sdk_evolution_agent.py +++ b/tests/test_sdk_evolution_agent.py @@ -2,6 +2,7 @@ import os import stat +import subprocess import sys import types from pathlib import Path @@ -53,7 +54,11 @@ SchemaValidationError, validate_mapping, ) -from examples.sdk_evolution_agent.snapshots import diff_snapshots, snapshot_current_api +from examples.sdk_evolution_agent.snapshots import ( + diff_snapshots, + snapshot_candidate_in_venv, + snapshot_current_api, +) from examples.sdk_evolution_agent.stages import ( SDK_EVOLUTION_CODEX_HOME, SDK_EVOLUTION_CODEX_MODEL, @@ -492,6 +497,156 @@ def isolated(package: str, version: str, *, scope: str = "candidate"): assert behavior["diffs"][0].severity == "changed" +def test_behavior_evidence_uses_locked_baseline_when_sdk_not_installed( + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[tuple[str, str, str]] = [] + + def isolated(package: str, version: str, *, scope: str = "candidate"): + calls.append((package, version, scope)) + return (_probe(package, version, scope, "pass", {"scope": scope}),) + + def current(package: str, *, version: str | None = None): + raise AssertionError(f"ambient probe must not represent {package} {version}") + + monkeypatch.setattr( + "examples.sdk_evolution_agent.behavior.probe_candidate_in_venv", + isolated, + ) + monkeypatch.setattr( + "examples.sdk_evolution_agent.behavior.probe_current_package", + current, + ) + + collect_behavior_evidence( + [ + { + "name": "claude-agent-sdk", + "locked_version": "0.2.106", + "installed_version": None, + } + ], + {"claude-agent-sdk": "0.2.110"}, + inspect_candidates=True, + ) + + assert calls == [ + ("claude-agent-sdk", "0.2.106", "current-baseline"), + ("claude-agent-sdk", "0.2.110", "candidate"), + ] + + +def test_behavior_evidence_without_opt_in_reports_actual_ambient_version( + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[tuple[str, str | None]] = [] + + def current(package: str, *, version: str | None = None): + calls.append((package, version)) + return (_probe(package, version, "current-environment", "fail", {}),) + + def isolated(package: str, version: str, *, scope: str = "candidate"): + raise AssertionError(f"isolated probe must not run for {package} {version} {scope}") + + monkeypatch.setattr( + "examples.sdk_evolution_agent.behavior.probe_current_package", + current, + ) + monkeypatch.setattr( + "examples.sdk_evolution_agent.behavior.probe_candidate_in_venv", + isolated, + ) + + behavior = collect_behavior_evidence( + [ + { + "name": "claude-agent-sdk", + "locked_version": "0.2.106", + "installed_version": None, + } + ], + {"claude-agent-sdk": "0.2.110"}, + ) + + assert calls == [("claude-agent-sdk", None)] + assert [result.status for result in behavior["results"]] == ["fail", "skip"] + + +def test_behavior_evidence_without_opt_in_uses_drifted_installed_version( + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[tuple[str, str | None]] = [] + + def current(package: str, *, version: str | None = None): + calls.append((package, version)) + return (_probe(package, version, "current-environment", "pass", {}),) + + def isolated(package: str, version: str, *, scope: str = "candidate"): + raise AssertionError(f"isolated probe must not run for {package} {version} {scope}") + + monkeypatch.setattr( + "examples.sdk_evolution_agent.behavior.probe_current_package", + current, + ) + monkeypatch.setattr( + "examples.sdk_evolution_agent.behavior.probe_candidate_in_venv", + isolated, + ) + + collect_behavior_evidence( + [ + { + "name": "claude-agent-sdk", + "locked_version": "0.2.96", + "installed_version": "0.2.106", + } + ], + {"claude-agent-sdk": "0.2.110"}, + ) + + assert calls == [("claude-agent-sdk", "0.2.106")] + + +def test_behavior_evidence_reuses_matching_ambient_baseline( + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[tuple[str, str, str | None]] = [] + + def current(package: str, *, version: str | None = None): + calls.append(("current", package, version)) + return (_probe(package, version, "current-environment", "pass", {}),) + + def isolated(package: str, version: str, *, scope: str = "candidate"): + calls.append((scope, package, version)) + return (_probe(package, version, scope, "pass", {}),) + + monkeypatch.setattr( + "examples.sdk_evolution_agent.behavior.probe_current_package", + current, + ) + monkeypatch.setattr( + "examples.sdk_evolution_agent.behavior.probe_candidate_in_venv", + isolated, + ) + + collect_behavior_evidence( + [ + { + "name": "claude-agent-sdk", + "locked_version": "0.2.106", + "installed_version": "0.2.106", + } + ], + {"claude-agent-sdk": "0.2.110"}, + inspect_candidates=True, + ) + + assert calls == [ + ("current", "claude-agent-sdk", "0.2.106"), + ("candidate", "claude-agent-sdk", "0.2.110"), + ] + + def test_behavior_candidate_probes_are_opt_in(monkeypatch: pytest.MonkeyPatch) -> None: # Probing a candidate pip-installs and imports freshly downloaded upstream # code; without --inspect-candidates that must never happen, and the gap @@ -646,6 +801,111 @@ def run_new(value: str, *, verbose: bool = False) -> str: assert diff.changed == ("run",) +@pytest.mark.parametrize( + ("failure_call", "detail"), + [ + (1, "venv creation failed"), + (2, "package install failed"), + (3, "snapshot execution failed"), + ], +) +def test_candidate_snapshot_records_subprocess_failures( + monkeypatch: pytest.MonkeyPatch, + failure_call: int, + detail: str, +) -> None: + calls = 0 + + def fake_run(args: Any, **kwargs: Any) -> Any: + nonlocal calls + calls += 1 + if calls == failure_call: + raise subprocess.CalledProcessError( + returncode=1, + cmd=args, + stderr=(detail + " ") * 100, + ) + return types.SimpleNamespace(stdout="{}", stderr="", returncode=0) + + monkeypatch.setattr("examples.sdk_evolution_agent.snapshots.subprocess.run", fake_run) + + snapshot = snapshot_candidate_in_venv("claude-agent-sdk", "0.2.110") + + assert snapshot.package == "claude-agent-sdk" + assert snapshot.version == "0.2.110" + assert snapshot.source == "isolated-venv" + assert snapshot.import_error is not None + assert detail in snapshot.import_error + assert len(snapshot.import_error) <= 600 + + +def test_candidate_snapshot_records_timeout(monkeypatch: pytest.MonkeyPatch) -> None: + def fake_run(args: Any, **kwargs: Any) -> Any: + raise subprocess.TimeoutExpired(cmd=args, timeout=kwargs["timeout"]) + + monkeypatch.setattr("examples.sdk_evolution_agent.snapshots.subprocess.run", fake_run) + + snapshot = snapshot_candidate_in_venv("claude-agent-sdk", "0.2.110", timeout=7) + + assert snapshot.import_error is not None + assert "timed out after 7s" in snapshot.import_error + assert snapshot.source == "isolated-venv" + + +def test_candidate_snapshot_records_malformed_output(monkeypatch: pytest.MonkeyPatch) -> None: + calls = 0 + + def fake_run(args: Any, **kwargs: Any) -> Any: + nonlocal calls + calls += 1 + return types.SimpleNamespace( + stdout="not-json" if calls == 3 else "", + stderr="", + returncode=0, + ) + + monkeypatch.setattr("examples.sdk_evolution_agent.snapshots.subprocess.run", fake_run) + + snapshot = snapshot_candidate_in_venv("claude-agent-sdk", "0.2.110") + + assert snapshot.import_error is not None + assert "malformed snapshot output" in snapshot.import_error + assert snapshot.source == "isolated-venv" + + +def test_candidate_snapshot_scrubs_subprocess_env( + monkeypatch: pytest.MonkeyPatch, +) -> None: + captured_envs: list[dict[str, str] | None] = [] + calls = 0 + + def fake_run(args: Any, **kwargs: Any) -> Any: + nonlocal calls + calls += 1 + captured_envs.append(kwargs.get("env")) + payload = ( + '{"package":"claude-agent-sdk","version":"0.2.110",' + '"module":"claude_agent_sdk","members":[],"import_error":null}' + ) + return types.SimpleNamespace( + stdout=payload if calls == 3 else "", + stderr="", + returncode=0, + ) + + monkeypatch.setenv("ARK_FAKE_SECRET", "not-for-snapshots") + monkeypatch.setattr("examples.sdk_evolution_agent.snapshots.subprocess.run", fake_run) + + snapshot = snapshot_candidate_in_venv("claude-agent-sdk", "0.2.110") + + assert snapshot.import_error is None + assert len(captured_envs) == 3 + for env in captured_envs: + assert env is not None + assert "ARK_FAKE_SECRET" not in env + assert env.get("HOME") != os.environ.get("HOME") + + def test_parse_args_candidate_inspection_is_opt_in() -> None: # Candidate inspection pip-installs+imports upstream code, so it is off by # default and only enabled with the explicit flag. @@ -803,6 +1063,56 @@ def isolated_snapshot(package: str, version: str) -> ApiSnapshot: ] +def test_collect_snapshots_uses_locked_baseline_when_sdk_not_installed( + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[tuple[str, str, str | None]] = [] + + def current_snapshot(package: str, *, version: str | None = None) -> ApiSnapshot: + raise AssertionError(f"ambient snapshot must not represent {package} {version}") + + def isolated_snapshot(package: str, version: str) -> ApiSnapshot: + calls.append(("isolated", package, version)) + return ApiSnapshot( + package=package, + version=version, + module=package.replace("-", "_"), + source="isolated-venv", + ) + + monkeypatch.setattr( + "examples.sdk_evolution_agent.cli.snapshot_current_api", + current_snapshot, + ) + monkeypatch.setattr( + "examples.sdk_evolution_agent.cli.snapshot_candidate_in_venv", + isolated_snapshot, + ) + + _collect_snapshots( + { + "packages": [ + { + "name": "claude-agent-sdk", + "locked_version": "0.2.106", + "installed_version": None, + "latest_version": "0.2.110", + }, + ], + "refresh_preview": { + "stdout": "", + "stderr": "Update claude-agent-sdk v0.2.106 -> v0.2.110\n", + }, + }, + inspect_candidates=True, + ) + + assert calls == [ + ("isolated", "claude-agent-sdk", "0.2.106"), + ("isolated", "claude-agent-sdk", "0.2.110"), + ] + + def test_collect_snapshots_without_opt_in_never_installs_even_when_lock_drifted( monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -846,7 +1156,55 @@ def isolated_snapshot(package: str, version: str) -> ApiSnapshot: inspect_candidates=False, ) - assert calls == [("current", "claude-agent-sdk", "0.2.96")] + assert calls == [("current", "claude-agent-sdk", "0.2.106")] + + +def test_collect_snapshots_without_opt_in_reports_missing_ambient_sdk( + monkeypatch: pytest.MonkeyPatch, +) -> None: + calls: list[tuple[str, str, str | None]] = [] + + def current_snapshot(package: str, *, version: str | None = None) -> ApiSnapshot: + calls.append(("current", package, version)) + return ApiSnapshot( + package=package, + version=version, + module=package.replace("-", "_"), + import_error="not installed", + ) + + def isolated_snapshot(package: str, version: str) -> ApiSnapshot: + raise AssertionError(f"isolated snapshot must not run for {package} {version}") + + monkeypatch.setattr( + "examples.sdk_evolution_agent.cli.snapshot_current_api", + current_snapshot, + ) + monkeypatch.setattr( + "examples.sdk_evolution_agent.cli.snapshot_candidate_in_venv", + isolated_snapshot, + ) + + snapshots = _collect_snapshots( + { + "packages": [ + { + "name": "claude-agent-sdk", + "locked_version": "0.2.106", + "installed_version": None, + "latest_version": "0.2.110", + } + ], + "refresh_preview": { + "stdout": "", + "stderr": "Update claude-agent-sdk v0.2.106 -> v0.2.110\n", + }, + }, + inspect_candidates=False, + ) + + assert calls == [("current", "claude-agent-sdk", None)] + assert snapshots[0].import_error == "not installed" def test_candidate_api_diff_guard_blocks_missing_update_diff() -> None: