Test code quality rubric with static datasets (Meta-Evals) by reidbaker-agent · Pull Request #194 · flutter/agent-plugins

reidbaker-agent · 2026-07-23T17:56:11Z

Adds a static dataset of good and bad code and a meta-eval dummy skill to test that the code quality rubric successfully catches anti-patterns and permits clean code.

Addresses PR #189 feedback by moving to a static fixture approach instead of having the LLM generate bad code on the fly.

gemini-code-assist

Code Review

This pull request introduces a comprehensive framework for evaluating AI agent skills, including a new run-evals skill, a test-rubric dummy skill for meta-evaluations, universal code quality rubrics, and a test suite to ensure structural consistency of evaluation files. Feedback focuses on making the newly added tests in skills_evals_test.dart more robust by dynamically resolving the package directory rather than relying directly on Directory.current, which can cause failures when tests are executed from the repository root.

gemini-code-assist · 2026-07-23T17:57:32Z

+void main() {
+  group('Evals structure consistency', () {
+    // Ensures all evals.json files dynamically share the exact same JSON schema.
+    // Keys are not hardcoded to ensure enforcement remains schema-agnostic and flexible.
+    test('all evals.json files across skills share consistent structure and keys', () {
+      final List<File> evalsFiles = [
+        ..._findEvalsFiles(Directory(p.join(Directory.current.path, 'skills'))),
+        ..._findEvalsFiles(Directory(p.join(Directory.current.path, '.agents', 'skills'))),
+      ]..sort((a, b) => a.path.compareTo(b.path));


Using Directory.current directly can make tests fragile and fail when executed from the repository root (e.g., during CI/CD or monorepo tasks) rather than the package root (tool/dart_skills_lint).

We can resolve the package directory dynamically by checking if Directory.current points to the package root, and if not, appending the relative path.

Suggested change

void main() {

group('Evals structure consistency', () {

// Ensures all evals.json files dynamically share the exact same JSON schema.

// Keys are not hardcoded to ensure enforcement remains schema-agnostic and flexible.

test('all evals.json files across skills share consistent structure and keys', () {

final List<File> evalsFiles = [

..._findEvalsFiles(Directory(p.join(Directory.current.path, 'skills'))),

..._findEvalsFiles(Directory(p.join(Directory.current.path, '.agents', 'skills'))),

]..sort((a, b) => a.path.compareTo(b.path));

void main() {

final packageDir = Directory.current.path.endsWith('tool/dart_skills_lint')

? Directory.current

: Directory(p.join(Directory.current.path, 'tool', 'dart_skills_lint'));

group('Evals structure consistency', () {

// Ensures all evals.json files dynamically share the exact same JSON schema.

// Keys are not hardcoded to ensure enforcement remains schema-agnostic and flexible.

test('all evals.json files across skills share consistent structure and keys', () {

final List<File> evalsFiles = [

..._findEvalsFiles(Directory(p.join(packageDir.path, 'skills'))),

..._findEvalsFiles(Directory(p.join(packageDir.path, '.agents', 'skills'))),

]..sort((a, b) => a.path.compareTo(b.path));

gemini-code-assist · 2026-07-23T17:57:32Z

+    // Note: We intentionally only require an evals.json file for published skills.
+    // Contributor skills in .agents/skills/ are not currently required to have one.
+    test('all published skills have an evals.json file', () {
+      final skillsDir = Directory(p.join(Directory.current.path, 'skills'));


Use the dynamically resolved packageDir instead of Directory.current to ensure the test runs correctly from any working directory.

Suggested change

final skillsDir = Directory(p.join(Directory.current.path, 'skills'));

final skillsDir = Directory(p.join(packageDir.path, 'skills'));

gemini-code-assist · 2026-07-23T17:57:32Z

+    });
+
+    test('all rubric JSON files in evals/ share consistent structure and keys', () {
+      final rubricsDir = Directory(p.join(Directory.current.path, 'evals'));


Use the dynamically resolved packageDir instead of Directory.current to ensure the test runs correctly from any working directory.

Suggested change

final rubricsDir = Directory(p.join(Directory.current.path, 'evals'));

final rubricsDir = Directory(p.join(packageDir.path, 'evals'));

…name

…ng Isolate.resolvePackageUri

reidbaker-agent · 2026-07-23T20:00:08Z

Meta-Eval Run Results

I've successfully run the test-rubric evaluation against our static fixtures to prove the rubric works! The agent successfully graded both the positive and negative test cases.

🔴 Eval 1: Negative Test Case (`bad_script.dart`)

Input: A script intentionally violating our hygiene and idiom rules.
Agent Output: The agent correctly flagged the failures according to the rubric criteria:

effective_dart_and_idioms: Flagged the use of raw string concatenation (r'Hello ' + user) instead of string interpolation ('Hello $user').
cross_platform_compatibility: Flagged the hardcoded Windows path (C:\my\path\data.txt).
directory_and_placement_hygiene: Flagged that the script was improperly located in a scratch tmp/ directory instead of the standard bin/ directory.

The agent then successfully fixed the script, moved it to bin/, and ensured it passed dart analyze and tests.

🟢 Eval 2: Positive Test Case (`good_script.dart`)

Input: A completely clean, idiomatic Dart script in a compliant directory.
Agent Output: The agent correctly analyzed the file and reported that it fully conforms to the rubric criteria with 0 violations found.

The code_quality_rubric.json is robust and agents can successfully parse and apply its criteria!

…evals.json

reidbaker-agent · 2026-07-23T22:20:03Z

Updated Meta-Eval Run Results

After updating the run-evals prompts to ensure agents do not fix static test fixtures during grading tasks, I ran the meta-evaluations against the test data again. The agents correctly graded the code without modifying the files!

🔴 Eval 1: Negative Test Case (`bad_script.dart`)

Input: A script intentionally violating our hygiene and idiom rules.
Agent Output: The agent correctly flagged the failures according to the rubric criteria without touching the code:

effective_dart_and_idioms: Fail - Flagged the use of raw strings alongside string concatenation.
cross_platform_compatibility: Fail - Flagged the hardcoded Windows path ('C:\my\path\data.txt').
directory_and_placement_hygiene: Fail - Flagged that the script was improperly located in tmp/ instead of a standard directory.
(The agent correctly returned an empty git diff!)

🟢 Eval 2: Positive Test Case (`good_script.dart`)

Input: A clean, idiomatic Dart script.
Agent Output: The agent correctly evaluated the file without making modifications:

effective_dart_and_idioms: Pass
cross_platform_compatibility: Pass
directory_and_placement_hygiene: Pass
(It did note a compilation failure strictly due to the test fixture missing a pubspec.yaml for the path dependency, but otherwise gave it a clean bill of health).

Both evaluations passed successfully according to the updated prompt instructions!

reidbaker · 2026-07-23T22:21:44Z

/gemini-review

reidbaker · 2026-07-24T03:20:10Z

@@ -0,0 +1,15 @@
+// ignore_for_file: unused_local_variable, prefer_final_locals, avoid_print, use_raw_strings


This file needs a copyright statement.

reidbaker · 2026-07-24T03:20:17Z

@@ -0,0 +1,15 @@
+import 'package:path/path.dart' as p;


This file needs a copyright statement.

gemini-code-assist Bot reviewed Jul 23, 2026

View reviewed changes

reidbaker-agent mentioned this pull request Jul 23, 2026

Add run-evals orchestration skill and evals for published skills #189

Merged

reidbaker force-pushed the feature/test-code-quality-rubric branch from aa8c477 to de6c96b Compare July 23, 2026 18:33

reidbaker added 4 commits July 23, 2026 15:01

Add static dataset and meta-eval for code quality rubric

3c92a1e

Add positive test case and README docs for meta-evals

5e71e4f

Update evals.json schema: move repo_criteria to top and remove skill_…

139b5a1

…name

Address remaining PR 194 feedback: fix test-rubric lints

3104642

reidbaker force-pushed the feature/test-code-quality-rubric branch from 352023e to 3104642 Compare July 23, 2026 19:01

reidbaker reviewed Jul 23, 2026

View reviewed changes

Comment thread tool/dart_skills_lint/dart_skills_lint.yaml

reidbaker added 2 commits July 23, 2026 15:08

Address PR review: resolve package directory dynamically in tests usi…

cc0793a

…ng Isolate.resolvePackageUri

Add contributor-pr-description to prevent-skills-sh-publishing config

aafba2e

reidbaker added 2 commits July 23, 2026 17:52

Refactor test-rubric dummy skill into standalone code_quality_rubric_…

a3616a8

…evals.json

Update baseline execution prompt to avoid fixing code on grading tasks

8cb2f8f

reidbaker marked this pull request as ready for review July 23, 2026 22:21

reidbaker self-requested a review July 23, 2026 22:21

reidbaker approved these changes Jul 24, 2026

View reviewed changes

reidbaker reviewed Jul 24, 2026

View reviewed changes

reidbaker approved these changes Jul 24, 2026

View reviewed changes

reidbaker added 2 commits July 23, 2026 23:21

Add copyright headers to test fixtures

bc82b1f

Update with_skill execution prompt to avoid fixing code on grading tasks

95b7efc

reidbaker approved these changes Jul 24, 2026

View reviewed changes

reidbaker merged commit c5cb871 into flutter:main Jul 24, 2026
13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Test code quality rubric with static datasets (Meta-Evals)#194

Test code quality rubric with static datasets (Meta-Evals)#194
reidbaker merged 10 commits into
flutter:mainfrom
reidbaker:feature/test-code-quality-rubric

reidbaker-agent commented Jul 23, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Uh oh!

Uh oh!

reidbaker-agent commented Jul 23, 2026

Uh oh!

reidbaker-agent commented Jul 23, 2026

Uh oh!

reidbaker commented Jul 23, 2026

Uh oh!

reidbaker Jul 24, 2026

Uh oh!

reidbaker Jul 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	final skillsDir = Directory(p.join(Directory.current.path, 'skills'));
	final skillsDir = Directory(p.join(packageDir.path, 'skills'));

	final rubricsDir = Directory(p.join(Directory.current.path, 'evals'));
	final rubricsDir = Directory(p.join(packageDir.path, 'evals'));

		@@ -0,0 +1,15 @@
		// ignore_for_file: unused_local_variable, prefer_final_locals, avoid_print, use_raw_strings

Uh oh!

Conversation

reidbaker-agent commented Jul 23, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

reidbaker-agent commented Jul 23, 2026

Meta-Eval Run Results

🔴 Eval 1: Negative Test Case (bad_script.dart)

🟢 Eval 2: Positive Test Case (good_script.dart)

Uh oh!

reidbaker-agent commented Jul 23, 2026

Updated Meta-Eval Run Results

🔴 Eval 1: Negative Test Case (bad_script.dart)

🟢 Eval 2: Positive Test Case (good_script.dart)

Uh oh!

reidbaker commented Jul 23, 2026

Uh oh!

reidbaker Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

reidbaker Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

🔴 Eval 1: Negative Test Case (`bad_script.dart`)

🟢 Eval 2: Positive Test Case (`good_script.dart`)

🔴 Eval 1: Negative Test Case (`bad_script.dart`)

🟢 Eval 2: Positive Test Case (`good_script.dart`)