C++ Grader

An LLM grades short C++ submissions against a rubric (Correctness, Efficiency, Code Style). This repository compares three prompting methods (zero-shot, few-shot, chain-of-thought) and two models (gpt-oss-20b, gpt-oss-120b, via Groq) with one person's scores for 21 submissions, adds a compile-and-run baseline for Correctness, and includes a small web app that grades one submission at a time.

Read this first: the 21 submissions are synthetic (written by the author to cover typical error types, not real student work) and all scores come from a single annotator. The results are a small case study. They do not show how the models would grade real students, or how another grader would score the same code.

Research questions

1. How close are LLM grades to a human's grades, for the final score and for each criterion?
2. Do few-shot examples or chain-of-thought instructions change that compared with zero-shot prompting?
3. Does the larger model (gpt-oss-120b) agree better with the human than the smaller one (gpt-oss-20b)?
4. How does a test-based Correctness score (compile the code, run it on test cases) compare with the human Correctness label and with the LLM's?

Web app

backend/app_groq.py is a small Flask app. It serves a page where you pick one of the benchmark examples (or paste your own problem, rubric, teacher solution and submission), grades it with the few-shot method, and shows the score per criterion with the model's explanation. For an unedited example it also shows the human label next to the model's score.

It is meant for local use: it listens on 127.0.0.1 only, runs Flask's development server with the debugger off, validates and size-limits its input, does not send CORS headers, and returns generic error messages (details go to the server log). It has no login and is not hardened for deployment.

Key results

Final score (0 to 10) against the human score, 21 submissions, 3 runs per setup, 189 gradings per model, no failed gradings. Cells show the estimate with a 95% bootstrap interval over the submissions. MAE is the mean absolute error in points (lower is better); exact match and QWK (quadratic weighted kappa) are higher-is-better.

Model	Method	MAE (points)	Exact match	QWK
gpt-oss-20b	zero-shot	1.30 [0.78, 1.86]	0.43 [0.25, 0.60]	0.83 [0.68, 0.94]
gpt-oss-20b	few-shot	1.27 [0.57, 2.17]	0.56 [0.37, 0.75]	0.84 [0.62, 0.96]
gpt-oss-20b	chain-of-thought	1.35 [0.75, 2.05]	0.48 [0.30, 0.65]	0.82 [0.64, 0.93]
gpt-oss-120b	zero-shot	0.94 [0.59, 1.32]	0.46 [0.27, 0.65]	0.93 [0.84, 0.96]
gpt-oss-120b	few-shot	0.70 [0.35, 1.10]	0.60 [0.40, 0.79]	0.95 [0.88, 0.98]
gpt-oss-120b	chain-of-thought	1.08 [0.56, 1.67]	0.51 [0.30, 0.70]	0.87 [0.72, 0.96]

Dots are the mean over 3 runs; lines are the 95% bootstrap intervals. The full tables (per criterion, per problem, Spearman correlation, run-to-run standard deviations, paired differences) are in results/analysis/summary.md; the other plots are in results/analysis/.

What the results show, and what they do not:

1. The better setups are off by about one point on average. The best setup, gpt-oss-120b with few-shot, has an MAE of 0.70 points, an exact match with the human score for 60% of submissions and a QWK of 0.95. The weakest, gpt-oss-20b with chain-of-thought, has an MAE of 1.35. Even the best setup misses the exact human score for 40% of submissions.
2. Few-shot had the lowest MAE for both models, but the evidence is weak. On gpt-oss-120b, few-shot vs zero-shot is −0.24 MAE (95% interval [−0.46, −0.05]). It is the only one of the 14 paired comparisons whose interval excludes 0, and it is indicative only: n = 21, 14 comparisons, no correction for multiple comparisons. In the sensitivity analysis below, the same interval is [−0.44, 0.00] and no longer excludes 0. On gpt-oss-20b the three methods are within 0.08 MAE of each other (1.27 to 1.35).
3. Chain-of-thought did not improve on zero-shot for either model (MAE +0.05 on 20b and +0.14 on 120b, both intervals include 0). On gpt-oss-120b it had the highest MAE of the three methods.
4. The larger model has the lower MAE and higher QWK for every method, but no difference is clearly separated from 0. The MAE difference is −0.27 to −0.57 points and the QWK difference +0.05 to +0.10, all with intervals that include 0. Most of the gap is on the hardest problem (q5): the 20b models have a q5 MAE of 2.9 to 3.5 points against 0.7 to 1.7 for the 120b models (4 submissions, descriptive only). Without q5 students 2, 3 and 4, the MAE gap shrinks to −0.17, 0.00 and −0.04 (sensitivity analysis).
5. Agreement is highest for Correctness and lowest for Code Style. For Correctness, exact match is 0.62 to 0.81 and Spearman 0.92 to 0.97 across the six setups. For Code Style, QWK is 0.42 to 0.66. The best method also differs by criterion: gpt-oss-120b few-shot has the lowest Correctness MAE of all six (0.37) but one of the lowest Code Style QWK values (0.43).
6. Run-to-run variation is of the same size as many differences between methods. The standard deviation of MAE over the 3 runs is 0.15 to 0.29 points (temperature 0.3), so a single run would not be enough to rank these methods.

Sensitivity analysis (secondary)

This is a secondary analysis. It is reported next to the main results above and does not replace them. The final-score metrics and the paired differences were repeated without q5 students 2, 3 and 4 (n = 18).

Why these three: students 2 and 4 get full Correctness (4/4) although they have no mutex and the problem requires a thread-safe cache; student 3 behaves as FIFO, not LRU, and its label (Correctness 0, Code Style 2) breaks the grading policy that Correctness 0 means 0 for every criterion (see Limitations).

The list was fixed before this analysis was run. It is not fully independent of model output: q5 student 2 was first noticed during the small parsing test (6 calls with gpt-oss-20b), and students 3 and 4 came from the label audit, which was done without model output.

Model	Method	MAE, all 21	MAE, n = 18 [95% interval]	QWK, all 21	QWK, n = 18 [95% interval]
gpt-oss-20b	zero-shot	1.30	1.07 [0.54, 1.67]	0.83	0.86 [0.67, 0.97]
gpt-oss-20b	few-shot	1.27	0.70 [0.33, 1.13]	0.84	0.95 [0.88, 0.98]
gpt-oss-20b	chain-of-thought	1.35	0.94 [0.48, 1.48]	0.82	0.90 [0.79, 0.96]
gpt-oss-120b	zero-shot	0.94	0.91 [0.54, 1.30]	0.93	0.93 [0.84, 0.97]
gpt-oss-120b	few-shot	0.70	0.70 [0.33, 1.13]	0.95	0.95 [0.86, 0.98]
gpt-oss-120b	chain-of-thought	1.08	0.91 [0.43, 1.50]	0.87	0.92 [0.80, 0.98]

In the sensitivity analysis:

• The gpt-oss-20b setups improve the most (few-shot MAE 1.27 to 0.70), the gpt-oss-120b setups much less, so most of the gap between the models in the main results comes from these three submissions.
• Few-shot still has the lowest MAE for both models (0.70 each).
• None of the 14 paired differences has an interval that excludes 0.

Compile-and-run baseline

To get a Correctness score that does not depend on the annotator or an LLM, every submission was compiled with g++ and run on test cases written from the problem statement only (52 tests over 5 problems; expected values are checked against Python models of the statements). The tests were not adjusted to match the labels or the submissions. Correctness points are split over the tests (for q5, LRU behaviour counts 3 points and thread safety 1 point), computed exactly and rounded half up. Details, decisions and known issues are in compile_tests/README.md.

Against the human Correctness labels, the test-based score matches exactly for 13 of 21 submissions (MAE 0.71 [0.29, 1.19], QWK 0.79, Spearman 0.92). The 8 disagreements:

Submission	Tests passed	Test-based Correctness	Human Correctness	What happens
q1 student 4	1 / 9	1	0	partial credit from the tests, label 0
q2 student 3	3 / 10	2 (1.5 rounded up)	0	same
q4 student 4	7 / 12	3 (2.92)	0	same
q5 student 3	5 / 9	3 (2.5 rounded up)	0	FIFO instead of LRU: fails 4 of the 8 LRU tests, passes the thread test
q3 student 3	7 / 12	4 (3.5 rounded up)	2	fails 5 of 12 tests, including the empty vector
q3 student 4	10 / 12	5	3	fails the all-negative and the INT_MIN tests
q5 student 2	8 / 9	3	4	no mutex; fails the thread-safety stress test
q5 student 4	8 / 9	3	4	same

What this shows:

• The tests and the labels rank the submissions almost the same way (Spearman 0.92) but not on the same scale. In 4 of the 8 disagreements the tests give partial credit where the human gave 0.
• In both q5 cases without a mutex, the stress test fails while the label gives full Correctness. This agrees with the label-audit finding below.
• For Correctness, the LLM setups have an MAE of 0.37 to 0.59 against the labels, and the tests 0.71. The intervals overlap widely, so there is no evidence here that either is closer to the labels. The baseline is a second opinion on Correctness, not ground truth, and only Correctness is measured.
• The thread-safety test can give false passes (a cache without a lock can pass by luck), and the teacher solution fails the q3 INT_MIN test (see Limitations).

Method

Dataset (dataset1/). Five problems of increasing difficulty: q1 sum of two integers, q2 reverse a string, q3 second largest element, q4 longest common subsequence, q5 thread-safe LRU cache. Each has a problem statement, a teacher solution, a rubric, 4 or 5 student submissions (21 in total) and the human scores. The submissions are synthetic and the human scores come from one person, the author. dataset1/ is the single source of truth; static/examples.json for the web app is generated from it.

Rubric. Correctness, Efficiency and Code Style, 10 points in total: q1, q2 and q4 use 5/3/2 points, q3 uses 6/3/1, q5 uses 4/4/2. The rubric files list only criteria and maximum points, with no level descriptions. Human scores are integers per criterion.

Grading policy. The prompts tell the model: "If Correctness is 0, give 0 for every criterion." This rule is not in the rubric files. It was added because the author's labels follow it (4 of the 5 labels with Correctness 0 are 0 for every criterion; q5 student 3 is the exception). The validator does not enforce it.

Methods. All three share a system prompt (grader role and criterion definitions), the task layout (problem, teacher solution, rubric, submission) and the JSON output format whose keys are the rubric's criteria. They differ only in what each adds:

• Zero-shot: nothing else.
• Few-shot: six worked grading examples on other problems (factorial, count of distinct values, primality, average, vowel count, word count), each with its rubric and its grade. One of them is a Correctness-0 case graded 0 for every criterion.
• Chain-of-thought: the model must first write out the requirements, three traced test cases, the complexity and a style assessment in a reasoning field, then score.

Prompt provenance. The prompts contain no benchmark problems, submissions or labels; a one-off overlap check on all 63 prompts (21 submissions × 3 methods; the check script is not in the repository) found no benchmark content. They were rewritten to remove the benchmark content of the first version's prompts and smoke-tested only on a made-up problem, so that the benchmark stayed unseen until the final runs. There were two earlier contacts with model output on benchmark submissions: the archived results of the first version (a different model and different prompts), and a 6-call parsing test (q5 students 1 and 2, three methods, gpt-oss-20b) run with the earlier prompts to check that outputs parse and fit the token budget. The parsing-test outputs are not used in any result. The one label-informed element of the prompts is the Correctness-0 rule and its example (see Grading policy). The last prompt change was committed 19 seconds before the final runs started, and no prompt file has changed since.

Models and settings. openai/gpt-oss-20b and openai/gpt-oss-120b on Groq. Temperature 0.3, at most 4,096 completion tokens, reasoning effort medium, reasoning text not returned. Each model graded every submission 3 times with each method: 189 gradings per model, none failed and none truncated. All methods use one shared validator that maps the model's keys to the rubric's criteria, converts to integers, clamps each score to its range and recomputes the total as the sum; it adds no grading policy of its own.

Metrics. For the final score and each criterion: mean absolute error (MAE), exact-match rate, quadratic weighted kappa (QWK, categories 0 to the maximum) and Spearman correlation. Each is computed per run and averaged over the 3 runs, and the standard deviation over runs is reported. The 95% intervals come from a percentile bootstrap over submissions (10,000 resamples, fixed seed); all runs of a resampled submission stay together. Paired comparisons use the same resampled submissions on both sides. With 21 submissions and 14 comparisons, an interval that excludes 0 is a lead, not a confirmed difference.

Limitations

Data

• The 21 submissions are synthetic: the author wrote them to cover typical error types. They are not real student work.
• All labels come from a single annotator (the author), so there is no measure of how much human graders would differ from each other. With n = 21, differences between methods or models of a few tenths of a point are within noise.

Label audit

The 21 human labels were checked against the problem statements, the rubrics and the code, without using any model output. Behaviour claims were checked with Python ports of the student logic (no C++ compiler was available at the time; the compile-and-run baseline was added later). No label was changed; the findings are listed so that readers can judge the reference scores.

• Mechanical checks pass: every breakdown sums to its final score, every score is within the rubric maximum, and every label uses exactly the rubric's criteria.
• q5 student 2: no mutex, although the problem requires a thread-safe cache, yet full Correctness (4/4). The missing requirement is penalised under Code Style (0/2) instead.
• q5 student 4: also no mutex, also full Correctness (4/4). Its non-O(1) operations (map, O(n) eviction scan) are reflected in Efficiency 0.
• q5 student 3: behaves as FIFO, not LRU (verified). Labelled 0/0/2. It is the one label that breaks the grading policy "if Correctness is 0, give 0 for every criterion".
• q2 student 2 vs q2 student 4: the same algorithm (append characters from the end into a new string, += vs push_back), both O(n) time and space, but Efficiency 1 vs 2.
• The Efficiency rubric has no level descriptions, and complexity does not map to points consistently:
  • q3 student 3 (O(n)) gets 0 and q3 student 4 (O(n), two passes) gets 1, although both match the reference's complexity;
  • q3 student 2 (O(n log n), sort) gets 0, the same as q4 student 3 (exponential).
• q4 student 3: Code Style 0 for short, readable code; the rubric does not explain the deduction.
• q1 student 3: returns double where the problem only says "returns their sum" (no type given). The full Code Style deduction (0/2) is a judgement call.

Reference solutions: edge cases

The reference solutions were not changed. Two of them mishandle an edge case, and the student submissions identical to them get full marks:

• q3 (second largest): the teacher solution, and the identical q3 student 1, return -1 when the true second-largest value is INT_MIN. The compile tests include this case and the teacher solution fails it.
• q5 (LRU cache): the teacher solution, and the identical q5 student 1, call cache.back() on an empty list when the capacity is 0 (undefined behaviour). The compile tests leave capacity 0 out because the statement does not define it.

Grading policy for Correctness 0

• The rubric files contain no such rule. The prompts add "If Correctness is 0, give 0 for every criterion" because 4 of the 5 labels with Correctness 0 follow it (q1 students 4 and 5, q2 student 3, q4 student 4). q5 student 3 (0/0/2) is the exception. So one element of the prompts was informed by the labels.
• The policy is stated in the prompts only; the validator does not enforce it.

Models and method

• The original model, llama-3.1-8b-instant, was decommissioned by Groq on 2026-08-16, so the original results cannot be reproduced. They are archived in results/archive/2026-01-09_original/ and are not reported here: their prompts contained benchmark submissions and benchmark-specific rules, so they do not measure grading ability.
• GPT-OSS models reason internally before answering, so explicit chain-of-thought prompting may add little. In this setup chain-of-thought differs from zero-shot mainly by requiring a written, structured analysis in the answer. This is a possible explanation for finding 3; it was not tested.
• The runs used a free-tier Groq account and took roughly a day per model, mostly waiting for the token quota, with some network outages. progress.jsonl in each results folder keeps every attempt, including the failed attempts that were retried; the final reports contain no failed gradings.
• The validator was changed after the runs finished: infinite scores are now treated as invalid and halves round up (matching the compile tests). The validator had adjusted 0 of the 378 gradings, and re-validating every saved model output with the new version gives identical scores, so the results are unaffected.
• Temperature 0.3 makes runs differ; exact numbers will vary when the benchmark is repeated.

Compile-and-run baseline

• It measures only Correctness, and the scale differs from the labels: the tests give partial credit on trivial inputs where the labels give 0.
• Thread safety is tested with a stress test (8 threads, 20,000 operations each, run 20 times in separate processes). A data race is nondeterministic, so a cache without a lock can still pass by luck. ThreadSanitizer would be reliable but is not available with MinGW.
• Expected values for get on a missing key (-1) are a convention, not a requirement of the statement.

Web app

• It is a local demo on Flask's development server, without authentication, and it grades with a single method (few-shot).

How to run

All commands are run from the project root.

1. Environment. Developed and run on Python 3.14; the unit tests also run on Python 3.12 (CI).

python -m venv .venv
# Windows (PowerShell):  .venv\Scripts\Activate.ps1
# macOS/Linux:           source .venv/bin/activate
pip install -r requirements.txt

2. API key (needed for the web app and the benchmark; the key is only read from the environment). Get a key at https://console.groq.com/keys.

# Windows (PowerShell):  $env:GROQ_API_KEY = "your-key"
# macOS/Linux:           export GROQ_API_KEY="your-key"

3. Web app. Open http://127.0.0.1:5001 in a browser. Set GRADER_MODEL to use another Groq model (default openai/gpt-oss-20b).

python -m backend.app_groq

4. Benchmark. One model at a time; 189 gradings per model. Progress is saved after every grading. If the run stops (Groq quota pause longer than --max-quota-wait, or a lost connection), run the same command again to resume. Results go to results/<model>/.

python -m backend.sdk.benchmark_runner_all_methods --model openai/gpt-oss-20b --runs 3
python -m backend.sdk.benchmark_runner_all_methods --model openai/gpt-oss-120b --runs 3

5. Compile-and-run baseline. Needs g++ on the PATH (see compile_tests/README.md for installing it on Windows). It makes no API calls.

python compile_tests/check_cases.py        # checks the test cases against models of the statements
python compile_tests/run_tests.py          # writes results/compile_tests/results.json

6. Analysis. Reads the committed (or your own) results and needs no API key. It takes a few minutes with 10,000 bootstrap resamples; add --bootstrap 1000 for a quick draft. It refuses runs that are incomplete. Output goes to results/analysis/.

python -m analysis.analyze_results

7. Tests. No API key or network access needed.

pip install -r requirements-dev.txt
python -m pytest

If you change dataset1/, regenerate the examples for the web page with python tools/generate_examples_for_html.py, which also checks that the dataset is consistent.

Project structure

backend/
  app_groq.py                       Flask app (web page and /grade endpoint)
  sdk/
    llm_common.py                   Groq client, retries, JSON extraction, score validator
    prompt_common.py                prompt text shared by all methods
    grading_agent_zeroshot_sdk.py   zero-shot prompt
    grading_agent_fewshot_sdk.py    few-shot prompt (six worked examples)
    grading_agent_cot_sdk.py        chain-of-thought prompt
    grading_methods.py              registry of the three methods
    benchmark_runner_all_methods.py resumable benchmark runner
frontend/                           web page (index.html, script.js, styles.css)
static/examples.json                benchmark examples for the page (generated)
dataset1/                           q1 ... q5: problem, teacher solution, rubric, submissions, human scores
compile_tests/                      compile-and-run baseline (cases, harness, README)
analysis/                           metrics, bootstrap analysis, tables and plots
tools/generate_examples_for_html.py validates dataset1/ and writes static/examples.json
tests/                              pytest tests (validator, JSON extraction, web app, analysis)
results/
  openai_gpt-oss-20b/               reports per method and progress.jsonl
  openai_gpt-oss-120b/              reports per method and progress.jsonl
  compile_tests/results.json        compile-and-run results
  analysis/                         summary.md, summary.json and plots
  archive/2026-01-09_original/      first version's results, kept for transparency, not valid
.github/workflows/tests.yml         runs the tests on Python 3.12 and 3.14

License

MIT, see LICENSE.