Add scenario validation pipeline and make systemPrompts reviewable#393
Open
kcarnold wants to merge 1 commit into
Open
Add scenario validation pipeline and make systemPrompts reviewable#393kcarnold wants to merge 1 commit into
kcarnold wants to merge 1 commit into
Conversation
Problem: System prompts for colleague chat scenarios were stored as single
long strings with \n escapes, making them hard to review and diff. More
importantly, there was no systematic way to validate that a scenario's
system prompt actually produces the intended colleague behavior across
different participant types (e.g., does it resist drafting requests? stay
in character with a confused newbie? avoid info-dumping on a passive ack?).
Changes:
1. systemPrompt → systemPromptLines (scenarios.json)
Store prompts as arrays of strings for readability. studyConfig.ts
joins them at load time so the runtime ScenarioConfig type and all
consumers (chat route, evalColleague) are unchanged.
2. Scenario design pipeline (scripts/scenario_design/)
Four manually-sequenced scripts for validating scenario prompts:
- generate.ts: situation description + criteria → scenario JSON entry
- simulate.ts: runs 4 participant archetypes (eager, lazy, confused,
boundary-pusher) through multi-turn conversations with the colleague
- judge.ts: evaluates conversations against 8 criteria using
generateObject + Zod for structured verdicts
- fix.ts: analyzes failures and proposes minimal prompt edits
criteria.md defines scenario-agnostic behavioral rules (information
gating, refusal to draft, fact consistency, tone, etc.) intended to
double as paper appendix material.
All scripts use the existing ai SDK + openai provider — no new deps.
Run with: npx tsx scripts/scenario_design/<script>.ts
3. Cross-pollinated the two scenario systemPrompts to align on:
- UNPOLISHED tone for both colleagues
- No promises / "I'll get back to you" for both
- Patience with new users for both
- User role descriptions matching taskInstructions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem: System prompts for colleague chat scenarios were stored as single long strings with \n escapes, making them hard to review and diff. More importantly, there was no systematic way to validate that a scenario's system prompt actually produces the intended colleague behavior across different participant types (e.g., does it resist drafting requests? stay in character with a confused newbie? avoid info-dumping on a passive ack?).
Changes:
systemPrompt → systemPromptLines (scenarios.json) Store prompts as arrays of strings for readability. studyConfig.ts joins them at load time so the runtime ScenarioConfig type and all consumers (chat route, evalColleague) are unchanged.
Scenario design pipeline (scripts/scenario_design/) Four manually-sequenced scripts for validating scenario prompts:
criteria.md defines scenario-agnostic behavioral rules (information
gating, refusal to draft, fact consistency, tone, etc.) intended to
double as paper appendix material.
All scripts use the existing ai SDK + openai provider — no new deps.
Run with: npx tsx scripts/scenario_design/<script>.ts
Cross-pollinated the two scenario systemPrompts to align on: