chore: add data-designer skill evals

[johnnygreco]

📋 Summary

Adds targeted eval coverage for the data-designer skill so Autopilot routing and skill-specific behaviors are easier to verify. The cases focus on Data Designer workflow use, person sampling, LLM judge score access, sampler params, and unrelated negative prompts.

🔗 Related Issue

N/A

🔄 Changes

• Add skills/data-designer/evals/evals.json with focused positive evals for Autopilot dataset generation scenarios.
• Add negative evals for unrelated PostgreSQL and React UI requests to verify the skill is not selected outside dataset-generation tasks.
• Keep behavior checks short, targeted, and easy to grade as true or false.

🧪 Testing

• make test passes — not run; eval JSON only
• Unit tests added/updated — skill evals added
• E2E tests added/updated (if applicable) — N/A
• python3 -m json.tool skills/data-designer/evals/evals.json passes

✅ Checklist

• Follows commit message conventions
• Commits are signed off (DCO) — commit was not created with signoff
• Architecture docs updated (if applicable) — N/A

[github-actions]

Review: PR #718 — test: add data-designer skill evals

Summary

Adds a single new file, skills/data-designer/evals/evals.json (86 lines), containing six eval cases for the data-designer skill: four positive cases exercising Autopilot routing plus specific behaviors (person sampling, LLM judge score access, sampler params), and two negative cases (PostgreSQL admin, React UI) that should not invoke the skill. No existing files are modified. JSON parses cleanly.

This is purely test/eval data — no runtime code paths are touched, blast radius is limited to anyone consuming the eval file.

Findings

Correctness & alignment with SKILL.md

The eval expectations line up well with the rules documented in skills/data-designer/SKILL.md:

• The sampler-params case asserts params is used and sampler_params is not — directly mirrors the SKILL.md rule at line 37 (SamplerColumnConfig: Takes params, not sampler_params).
• The llm-judge-scores case asserts the accepted-boolean derivation references .score — mirrors the LLM judge access guidance at SKILL.md line 38.
• The person-reviews case references get_person_object_schema.py, which exists at skills/data-designer/scripts/get_person_object_schema.py. ✓
• Both negative cases assert the skill is not selected for unrelated tasks (DB admin, React UI), which is appropriate given the SKILL.md description scopes the skill to dataset/synthetic-data/data-generation tasks.

Schema consistency

• All six entries share the same shape: id, question, expected_skill, expected_script, ground_truth, expected_behavior. Good — uniform schema makes downstream grading simpler.
• expected_script is null for five entries and "get_person_object_schema.py" for one. That's the only structured assertion about which script the agent should run; the rest of script-execution checks live in expected_behavior strings. Worth noting the schema isn't documented anywhere (no README in evals/, no JSON Schema), so the meaning of expected_script vs. the free-text expected_behavior items is implicit. Not blocking, but a short evals/README.md describing the schema and how cases are graded would help future contributors.

Behavior assertions — graders need to match

The expected_behavior strings are natural-language predicates that presumably get fed to an LLM judge. A few observations:

• Sampler-params case, item "The site column is generated by a category sampler" — strong assertion. The user prompt mentions only site as a column; the agent could reasonably pick category or subcategory. Probably fine as written, but if the grader is strict about category vs. any categorical sampler, this could be flaky.
• Person-reviews case, item "The agent ran python scripts/get_person_object_schema.py with a locale argument" — assumes the script accepts a locale arg. Worth a quick sanity check that the script's CLI actually supports this; if it's optional, a flaky-grader risk exists.
• Negative cases — the assertion "The agent did not read the data-designer SKILL.md" is a precise behavior, but Autopilot/skill-invocation typically reads SKILL.md as part of skill selection, not just execution. Whether the harness counts skill-selection reads against this assertion depends on grader semantics. Worth verifying with one dry run of the negative cases that a passing agent actually trips zero reads.
• "The agent did not ask the user a clarifying question before building the script" appears in only one positive case, even though all four positive prompts include "do not ask follow-up questions" / "be opinionated" language. Consider adding this assertion to the other three for consistency, or deliberately leaving it off if the grading focus is different.

Project conventions

• File location (skills/<skill>/evals/evals.json) is the first instance of this pattern in the repo (no other evals*.json files exist). The PR establishes the convention rather than following one — fine, but worth being deliberate. If multiple skills will get evals, codifying the layout (one file per skill, top-level array, this schema) in skills/README.md or similar would prevent drift.
• No SPDX/license header — appropriate, JSON doesn't carry one.
• No tests or CI wiring is added to consume the eval file. The PR description notes this is intentional ("eval JSON only"), and presumably an external evaluator picks it up. If there's a make target or repo-internal runner, a follow-up that wires it in would close the loop.

Minor

• id values use kebab-case prefixed with data-designer- — consistent and grep-friendly. ✓
• ground_truth and expected_behavior are slightly redundant in places (e.g., the positive cases' ground_truth restates what the behavior list already covers). Not a problem, just verbose.
• DCO sign-off is missing per the PR checklist. Will need to be addressed before merge if this repo enforces DCO.

Risks

• Low. No code paths change. Worst case is a flaky eval grader; that surfaces as eval-suite noise, not user-facing breakage.
• The eval assertions are tightly coupled to current SKILL.md wording (e.g., file names, the .score rule, the params vs. sampler_params rule). Future SKILL.md edits will need to keep this file in sync — a one-line note in SKILL.md or a comment in the eval file pointing at the dependency would help.

Verdict

Looks good. Small, well-scoped, internally consistent, and the assertions match documented skill behavior. Suggested non-blocking follow-ups:

1. Add a brief skills/data-designer/evals/README.md (or top-level skills/EVALS.md) describing the schema and grading model.
2. Verify get_person_object_schema.py actually accepts a locale argument before this eval runs in CI.
3. Consider tightening or relaxing the "did not read SKILL.md" assertion on negative cases based on how the grader treats skill-selection reads.
4. Address the missing DCO sign-off.

[greptile-apps]

Greptile Summary

Adds skills/data-designer/evals/evals.json with six targeted evals for the data-designer skill: four positive Autopilot cases (support tickets, person-based reviews, LLM judge scores, sampler parameters) and two negative cases (PostgreSQL admin, React UI) to verify the skill is not routed to for unrelated requests.

• The positive evals cover the main Autopilot workflow paths including person-sampling, LLM judge score derivation, and correct SamplerColumnConfig field usage (params vs sampler_params).
• The negative evals confirm the skill is not selected when the user asks about database administration or frontend UI work.
• The first positive eval explicitly names the skill in the question ("Use the data-designer skill…"), which differs from the natural-language phrasing used in the other positive cases and prevents it from functioning as a routing test.

Confidence Score: 5/5

JSON-only change that adds eval test cases; no production code is modified.

The change touches only a new eval data file with no production code paths. The two observations are about test coverage quality — one eval question bypasses routing and three evals omit the Learn-step assertion — but neither prevents the file from being parsed or the existing checks from running correctly.

skills/data-designer/evals/evals.json — specifically the routing coverage of the support-tickets case and the missing Learn-step assertions in the other three autopilot cases.

Important Files Changed

Filename	Overview
skills/data-designer/evals/evals.json	New eval file with 4 positive autopilot evals and 2 negative routing evals; one positive eval explicitly names the skill in the question (bypassing routing), and three autopilot evals omit the "agent context" step check required by autopilot.md.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    Q[Incoming question] --> R{Skill router}
    R -->|natural language: person reviews| DD1[data-designer-autopilot-person-reviews]
    R -->|natural language: IoT telemetry| DD2[data-designer-autopilot-sampler-params]
    R -->|explicit: 'in autopilot'| DD3[data-designer-autopilot-llm-judge-scores]
    R -->|explicit: 'Use the data-designer skill'| DD4[data-designer-autopilot-support-tickets]
    R -->|PostgreSQL admin| NEG1[data-designer-negative-database-admin\nexpected_skill: null]
    R -->|React UI| NEG2[data-designer-negative-react-component\nexpected_skill: null]
    DD1 --> AW1[Autopilot workflow\n+ person-sampling reference]
    DD2 --> AW2[Autopilot workflow\n+ sampler type/params checks]
    DD3 --> AW3[Autopilot workflow\n+ LLM judge score .score checks]
    DD4 --> AW4[Autopilot workflow\n+ validate + preview]
    AW1 & AW2 & AW3 & AW4 --> ES[expected_skill: data-designer]
    NEG1 & NEG2 --> EN[expected_skill: null]

Loading

Prompt To Fix All With AI

Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
skills/data-designer/evals/evals.json:4
**Explicit skill name in question bypasses routing verification**

The question for `data-designer-autopilot-support-tickets` opens with "Use the data-designer skill to create…", which is a direct invocation rather than natural language. The other positive evals ("Create a synthetic e-commerce product review dataset…", "Generate a synthetic IoT sensor telemetry dataset…") rely on natural phrasing so the router must select the skill on its own. With the skill named explicitly the `expected_skill` assertion will always pass regardless of whether the router would choose `data-designer` for a generic support-ticket request, making this case unable to catch routing regressions for that scenario.

### Issue 2 of 2
skills/data-designer/evals/evals.json:24-36
**"Agent context" step not verified in three of four autopilot evals**

`autopilot.md` step 2 requires running `data-designer agent context` ("Learn" step) before writing the script — this is where the agent inspects column schemas and avoids guessing types. Only `data-designer-autopilot-support-tickets` includes `"The agent ran data-designer agent context before writing the script"` in `expected_behavior`. The remaining three autopilot evals (`data-designer-autopilot-person-reviews`, `data-designer-autopilot-llm-judge-scores`, and `data-designer-autopilot-sampler-params`) do not assert this step was taken, so an agent that skips the Learn step would still pass those evals as long as its output happens to be correct.

_{Reviews (6): Last reviewed commit: "test: add data-designer skill evals" | Re-trigger Greptile}

[johnnygreco]

/nvskills-ci

[calliente]

@johnnygreco - I think this is failing because its missing the DCO sign-off. Run git rebase --signoff origin/main && git push --force-with-lease

[johnnygreco]

/nvskills-ci