Skip to content

fix(scaffold-verifier): refuse runnable check on observed partial read with collateral miss (#2142) - #2167

Open
yunusarda-sudo wants to merge 2 commits into
OpenAdaptAI:mainfrom
yunusarda-sudo:fix/scaffold-verifier-refuse-observed-partial-read
Open

yunusarda-sudo wants to merge 2 commits into
OpenAdaptAI:mainfrom
yunusarda-sudo:fix/scaffold-verifier-refuse-observed-partial-read

Conversation

@yunusarda-sudo

Copy link
Copy Markdown
Contributor

Summary

Closes #2142.

Issue #2129 refused a runnable verifier when a probed read cannot see the whole save (an open collateral_unaudited miss or uncovered write). However, an observed read in the recording's own traffic with the same miss still produced a runnable draft check. While carrying needs_operator_confirmation, emitting a runnable record_check.proposed.yaml in this case was inconsistent with the probed-read rule because a write outside the read would pass.

This PR applies the same refusal policy to observed reads: no runnable record_check.proposed.yaml is emitted while a write outside the read would pass. The TODO draft (effect_contract.yaml) is kept, and the refusal reason is recorded in the proof (record_check.proof.json).

Changes

  1. openadapt_flow/record_check_synthesis.py:
    • Extended Proposal with refusal_reason: Optional[str] = None.
    • Updated Proposal.refused to return True when refusal_reason is set (or read_discovery is refused).
    • Included refusal_reason in Proposal.proof() when refused.
    • Replaced _refuse_partial_probed_read with generalized _refuse_partial_read: if proposal.search.surfaces has observed reads but unaudited misses (collateral_unaudited or uncovered writes) remain, set proposal.refusal_reason, mark the proposal as refused, and append the refusal reason to notes. Kept _refuse_partial_probed_read = _refuse_partial_read as an alias.
  2. openadapt_flow/scaffold_verifier.py:
    • In build_record_check: when proposal.refused, write rcs.PROOF_NAME (record_check.proof.json) with status "refused" and return None (omitting record_check.proposed.yaml).
    • Guarded read_discovery.json writing to execute only when proposal.search.read_discovery is not None.
  3. openadapt_flow/__main__.py:
    • Added CLI reporting for recorded read refusal when status in record_check.proof.json is "refused".
  4. tests/test_record_check_synthesis.py:
    • Added unit test test_cli_refuses_an_observed_partial_read_with_an_open_collateral_miss asserting that observed partial reads with open collateral misses refuse runnable proposed check emission, write refused proof, and keep the TODO draft.
    • Updated test_cli_exits_nonzero_for_a_draft to use --complete-read so the check tests live failure rather than refusal.

Verification

  • uv run pytest tests/test_record_check_synthesis.py (42/42 passed).
  • uv run pytest tests/test_zero_config_read_discovery.py (26/26 passed).
  • uv run pytest tests/test_scaffold_verifier.py (17/17 passed).
  • uv run pytest tests/test_record_check_proof_injection.py (22/22 passed).
  • uv run pytest tests/test_fault_gate.py (11/11 passed).
  • uv run python scripts/check_engine_generality.py (passes: 301 baseline hits left).
  • uv run mypy openadapt_flow/record_check_synthesis.py openadapt_flow/scaffold_verifier.py openadapt_flow/__main__.py (Success: no issues found in 3 source files).
  • uv run ruff check and uv run ruff format --check (clean).

…d with collateral miss (OpenAdaptAI#2142)

Issue OpenAdaptAI#2129 refused a runnable verifier when a probed read cannot see the whole save (an open collateral_unaudited miss or uncovered write). However, an observed read in the recording's own traffic with the same miss still produced a runnable draft check.

Apply the same refusal policy to observed reads:
1. In openadapt_flow/record_check_synthesis.py:
   - Extend Proposal with refusal_reason: Optional[str] = None.
   - Update Proposal.refused to return True when refusal_reason is set.
   - Include refusal_reason in Proposal.proof() when refused.
   - In _refuse_partial_read: if proposal.search.surfaces has observed reads but unaudited collateral misses remain, refuse the proposal and record the refusal reason.
2. In openadapt_flow/scaffold_verifier.py:
   - When proposal.refused, write record_check.proof.json with status "refused" and omit record_check.proposed.yaml.
   - Guard read_discovery.json write only when read_discovery is present.
3. In openadapt_flow/__main__.py:
   - Report recorded read refusal clearly in CLI when status is "refused".
4. In tests/test_record_check_synthesis.py:
   - Add regression test test_cli_refuses_an_observed_partial_read_with_an_open_collateral_miss.
   - Update test_cli_exits_nonzero_for_a_draft to use --complete-read.

Closes OpenAdaptAI#2142.

Signed-off-by: Yunus Ardaguney <yunusarda-sudo@users.noreply.github.com>
@yunusarda-sudo
yunusarda-sudo force-pushed the fix/scaffold-verifier-refuse-observed-partial-read branch from 0de52ae to d8a57ae Compare November 10, 2026 09:09

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

scaffold-verifier: refuse a runnable check for an observed partial read with an open collateral miss

1 participant