Skip to content

Add prompt dedup A/B simulation harness#54

Open
SecretSettler wants to merge 1 commit into
feat/actual-token-prompt-shadowfrom
feat/prompt-dedup-ab-harness
Open

Add prompt dedup A/B simulation harness#54
SecretSettler wants to merge 1 commit into
feat/actual-token-prompt-shadowfrom
feat/prompt-dedup-ab-harness

Conversation

@SecretSettler

Copy link
Copy Markdown
Member

Summary

  • Add offline-only system/skill prompt dedup A/B simulation to the Hermes opportunity analyzer
  • Split candidate classes: same-type skill prompt, same-type system prompt, and cross-type system/skill
  • Report simulated before/after char deltas and opt-in tokenizer deltas when an explicit backend is configured
  • Keep all figures advisory/simulation-only; no runtime replacement, canonicalization, summarization, or prompt mutation
  • Add CLI flags: --disable-prompt-dedup-ab and --prompt-dedup-tokenizer
  • Document this as the evidence gate before canary replacement

Verification

  • /usr/local/lib/hermes-agent/venv/bin/python -m pytest tests/test_hermes_context_opportunity_analyzer.py -q
  • /usr/local/lib/hermes-agent/venv/bin/python -m pytest tests/test_hermes_plugin_patch.py tests/test_hermes_context_opportunity_analyzer.py tests/test_contextpilot_savings.py tests/test_contextpilot_savings_skill.py tests/test_hermes_contextpilot_monitor.py -q
  • /usr/local/lib/hermes-agent/venv/bin/python -m pytest -q
  • /usr/local/lib/hermes-agent/venv/bin/python scripts/analyze_hermes_context_opportunities.py --since-hours 24 --out-dir /tmp/contextpilot_prompt_ab_check --date prompt_ab_check --top-n 5
  • Claude Code read-only review: Approve, no blockers

Stacked on PR #53, which is stacked on PR #52.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant