MultiPL-E pass@k averages every *.results.json in the shared system temp dir, so leftovers from earlier runs or the previous task are scored too

[shaurya416]

bigcode_eval/tasks/multiple.py:155 and :182-195

        # a common temp dir for all the problems
        temp_dir = tempfile.gettempdir()
        ...
        for file in tqdm(list_files):
            evaluate_problem(temp_dir, file, max_workers)

        # compute pass@k scores
        result_array = np.array(
            [for_file(p) for p in Path(temp_dir).glob("*.results.json")]
        )
        result = result_array.mean(axis=0)

evaluate_problem writes <temp_dir>/<problem name>.results.json and nothing ever deletes those files. The pass@k line then globs the whole of tempfile.gettempdir() (/tmp on Linux) rather than the list_files it just wrote, so any *.results.json already there is scored too. Problem names are identical across MultiPL-E languages (HumanEval_0_has_close_elements, ...), so a run only overwrites the files whose names it shares; every other file survives and is averaged in.

Two ordinary ways to hit it:

1. --tasks multiple-js,multiple-d (or any pair) in one invocation. main.py:252 runs the tasks in one process. humaneval-js has 161 problems, humaneval-d has 156 (rs 156, sh 158, jl 159), so five js results files are still in /tmp when the d score is computed and go into d's mean.
2. Any run with --limit N (or a re-run of a different model) on a filesystem that has done a full run before: the 161-N leftover files dominate the mean. A fresh docker run of the evaluation-harness-multiple image starts with an empty /tmp, so this path needs a kept container or a run outside one; case 1 needs nothing unusual.

Measured

estimator/for_file (single_experiment_pass_k.py:6-22) and lines 182-195 copied verbatim into a standalone script; temp_dir pointed at a fresh directory per case and pre-populated with results files. Confirmed afterwards on the real bigcode_eval/tasks/multiple.py at 8fc5bae in a fresh TMPDIR: a second task with one failing problem reported pass@1 = 0.5.

case	temp dir contents	reported	correct
control, must fail	1 problem, 0/1 OK	pass@1 = 0.0	0.0
control, must pass	1 problem, 1/1 OK	pass@1 = 1.0	1.0
control, intended	2 problems, 0/1 and 1/1	pass@1 = 0.5	0.5
stale file	this run 0/1 OK + one leftover with 20/20 OK	pass@1 = 0.5	0.0
--limit 1 after a full run	this run 0/1 + 163 leftovers all pass	pass@1 = 0.9939	0.0
inverse	this run 1/1 + 163 leftovers all fail	pass@1 = 0.0061	1.0
multiple-js,multiple-d	156 d files all fail + 5 js leftovers all pass	d pass@1 = 0.0311	0.0
inverse	156 d all pass + 5 js leftovers all fail	d pass@1 = 0.9689	1.0
stale n=1 file, this run n=20	both 0 passes	pass@10 = 0.5	0.0
no files at all	empty dir	RuntimeWarning: Mean of empty slice then TypeError	clear error

The last row shows the failure is loud only when the directory is empty; with leftovers present the number is simply wrong and nothing is printed.

The stale-n=1 row is a second effect: estimator(n, c, k) returns 1.0 whenever n - c < k, including c == 0, so a leftover file with fewer samples than this run's k contributes a full pass to pass@10/pass@100. code_eval.py:169 guards this with (total >= k).all(); for_file has no such guard.

Consequence

multiple-<lang> pass@k is not a function of the generations being evaluated. Whether a result is right depends on what ran before on the same machine or earlier in the same command line. The two multi-task rows above need no unusual flags at all. No test covers multiple.py, so nothing catches it.

Suggested fix

Do what the harness already does elsewhere: execute.py:111-116 (create_tempdir), beyond_eval.py:83-88, eval_scala.py:11 and eval_java.py:24 all wrap their scratch space in with tempfile.TemporaryDirectory() as temp_dir:. Replacing temp_dir = tempfile.gettempdir() with that context manager scopes the files to the run and deletes them afterwards.

Independently of that, the glob is unnecessary: list_files is the exact set written this run, so

result_array = np.array(
    [for_file(get_test_results_json_path(temp_dir, f, None)) for f in list_files]
)

(using the helper already in multiple_metrics/evaluation.py:52) scores only this run's problems even if the directory is shared. Doing both closes the multi-task case and the leftover case; a len(list_files) == len(result_array) assertion would turn the empty-dir TypeError into a readable message. A guard mirroring code_eval.py:169 (if n >= k) in for_file would stop under-sampled files from reporting pass@k = 1.0.

Happy to open the PR.