Score MultiPL-E pass@k from a per-run temp dir

[shaurya416]

Fixes #327

Cause

GeneralMultiPLE.process_results writes each problem and its <name>.results.json into tempfile.gettempdir() (/tmp on Linux) and then computes pass@k over Path(temp_dir).glob("*.results.json"). Nothing removes those files, and problem names are the same in every MultiPL-E language (HumanEval_0_has_close_elements, ...). So any results file already in the system temp dir is averaged into the score: files from an earlier run, or from the previous task when several multiple-<lang> tasks run in one invocation. For example, with --tasks multiple-js,multiple-d, the js results for problems that d does not have are still in the temp dir when d is scored.

Change

temp_dir = tempfile.gettempdir() becomes temp_dir = tempfile.mkdtemp(), so each process_results call writes into, and globs, a directory of its own. Nothing else changes: the files are still kept after the run and the existing "Saved N problems in ..." line prints the new directory.

tests/test_multiple.py adds tests for process_results. They build the task without loading the dataset, point tempfile.tempdir at pytest's tmp_path (standing in for the shared system temp dir), and replace evaluate_problem with a function that writes a results file in the same format without running any code.

Validation

The new test file was run twice from a scratch directory outside the repository, against a small package holding the verbatim source of GeneralMultiPLE and the top-level imports of multiple.py (taken with ast.get_source_segment, once from the file before this change and once after), plus the verbatim get_test_results_json_path, evaluate_problem, estimator and for_file. Python 3.12.4, pytest 9.1.1.

test	kind	before	after
test_single_run_mixed (1 of 2 problems pass, clean dir)	control	pass	pass
test_single_run_all_pass_and_all_fail (same names run twice)	control	pass	pass
test_several_samples_per_problem (n=10, expects pass@1 0.15, pass@10 0.5)	control	pass	pass
test_leftover_results_file_is_left_alone (another run's file is not deleted)	control	pass	pass
test_leftover_results_file_is_not_scored (1 failing problem + 1 leftover file)	bug	fail: {'pass@1': 0.5} == {'pass@1': 0.0}	pass
test_previous_task_in_same_process_is_not_scored (js 3 problems, then d 2 problems)	bug	fail: {'pass@1': 0.3333333333333333} == {'pass@1': 0.0}	pass

Totals: before 2 failed, 4 passed; after 6 passed.

Stand-ins used in that scratch package (everything not listed was verbatim repository text):

• numpy: a small list-backed array/ndarray (mean(axis=0), k / arr, 1.0 - arr, iteration), arange, prod, plus isscalar and bool_, which pytest.approx looks up whenever a module named numpy is loaded.
• tqdm.tqdm: returns its iterable.
• datasets.load_dataset: raises if called; the tests never call it.
• bigcode_eval.base.Task: an empty class.
• The package __init__.py files are empty (the real bigcode_eval/tasks/__init__.py imports every task), and from .containerized_eval import eval_string_script was left out of evaluation.py; evaluate_problem is replaced by the test anyway.

flake8 7.4.1 (unpinned, as in CI, where the lint step is commented out), default settings: tests/test_multiple.py is clean; bigcode_eval/tasks/multiple.py reports the same 19 existing warnings before and after.

Not run: the new test inside the real package with the real dependencies (numpy, datasets, transformers), the existing suite in tests/, any real MultiPL-E evaluation or code execution, the Docker image, and any formatter (the repository pins none and has no formatter config).

Notes

• Files are still kept after the run, as today. Switching to tempfile.TemporaryDirectory() would also delete them, but that would remove the per-problem results people can read after a run, so I left it out.
• The under-sampled leftover effect from the issue (estimator returns 1.0 when n - c < k, so a stale file with fewer samples than k counts as a pass) can no longer come from leftovers, because the glob only sees this run's files. for_file itself is unchanged.