Skip to content

feat: add trace-derived validation gate#56

Open
SecretSettler wants to merge 1 commit into
feat/skill-prompt-dedup-canaryfrom
feat/trace-validation-set
Open

feat: add trace-derived validation gate#56
SecretSettler wants to merge 1 commit into
feat/skill-prompt-dedup-canaryfrom
feat/trace-validation-set

Conversation

@SecretSettler

Copy link
Copy Markdown
Member

Summary

  • add a local-only trace validation-set builder for Hermes sessions with privacy-safe manifests
  • add a runner that compares baseline/off vs candidate modes and gates payload-changing optimizations on preservation invariants
  • document the workflow and add synthetic fixtures/tests (no real trace data committed)

Validation

  • python3 -m pytest tests/test_trace_validation_builder.py tests/test_trace_validation_runner.py tests/test_prompt_dedup_canary.py -q
  • python3 -m pytest -q
  • python3 scripts/build_trace_validation_set.py --since-hours 72 --limit 25 --min-input-tokens 1000 --out /root/contextpilot/validation_sets --date 2026-06-14
  • python3 scripts/run_trace_validation.py /root/contextpilot/validation_sets/validation_set_2026-06-14.jsonl --candidate-mode canary --format markdown

Privacy

  • generated real trace corpora stay local/gitignored and are not committed
  • reports/manifests contain salted ids, counters, enums, pass/fail only
  • actual token savings remain unavailable unless an exact tokenizer is configured

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant