Skip to content

test(codex): add automatic recall regression benchmark - #3880

Open
AirHua-byte wants to merge 1 commit into
volcengine:mainfrom
AirHua-byte:codex/codex-recall-benchmark
Open

test(codex): add automatic recall regression benchmark#3880
AirHua-byte wants to merge 1 commit into
volcengine:mainfrom
AirHua-byte:codex/codex-recall-benchmark

Conversation

@AirHua-byte

Copy link
Copy Markdown

Description

Adds a privacy-safe benchmark for the real Codex automatic-recall hook. It records latency, false-injection rate, positive-recall rate, and injected-token estimates, and compares reports produced from the same fixture without publishing query text, URIs, or recalled content.

Human Involvement

  • A human participated in the implementation or review loop
  • This PR was generated entirely by AI agents without human participation in the loop

Related Issue

Part of #3686

Type of Change

  • Documentation update
  • Test update

Changes Made

  • Exercise the real Codex auto-recall hook against JSONL fixtures.
  • Emit aggregate latency, recall, false-injection, and token-budget metrics only.
  • Add same-fixture report comparison and a synthetic public negative fixture.

Testing

  • Added focused tests
  • Targeted tests pass locally
  • Tested on macOS

PYTHONPATH=. <validated-python> -m pytest -o addopts='' --confcutdir=tests -q tests/benchmark/test_codex_recall_benchmark.py — 5 passed; 4 existing dependency deprecation warnings.

Checklist

  • Code follows the project's style
  • Self-review completed
  • Documentation updated
  • The tracked fixture contains synthetic negative prompts only

Additional Notes

Deployment-specific positive fixtures stay local and untracked. Reports contain no query text, URI, or recalled memory content.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

1 participant