replace_evaluation_results() deletes registry lines it could not read

[feiiiiii5]

Describe the bug

replace_evaluation_results() rebuilds a scorer-metrics registry from _load_jsonl() (pyrit/score/scorer_evaluation/scorer_metrics_io.py:366-378), but that helper is a lookup: it skips lines it cannot parse, and if the read itself fails it logs Failed to load registry ... and returns whatever it managed to read (:304-311). Used as the source of truth for a truncating rewrite, both behaviours destroy data:

• an entry whose line is not valid JSON — a write killed mid-line, a hand edit — is deleted from the rewritten file for good;
• one undecodable byte anywhere in the file stops the read early, and the rewrite then contains only the new entry: every other pre-computed metric in that file is gone, and the call returns without an error.

These files ship with the package (pyrit/datasets/scorer_evals/, e.g. refusal_metrics.jsonl at 191 KB) and hold scores produced by hours of model calls, so this is not a rebuildable cache. The docstring states that only the entry with the matching eval_hash is removed and that the write is atomic; open(file_path, "w") over a short list does neither.

Steps/Code to Reproduce

import json
from pathlib import Path

from pyrit.models import ComponentIdentifier
from pyrit.score.scorer_evaluation.scorer_metrics import ObjectiveScorerMetrics
from pyrit.score.scorer_evaluation.scorer_metrics_io import (
    _load_jsonl,
    add_evaluation_results,
    replace_evaluation_results,
)

path = Path("/tmp/pyrit-issue-repro/registry.jsonl")
path.parent.mkdir(parents=True, exist_ok=True)
path.unlink(missing_ok=True)
ident = ComponentIdentifier(class_name="A", class_module="pyrit.score.test", params={"model_name": "m"})
metrics = ObjectiveScorerMetrics(
    num_responses=100, num_human_raters=3, accuracy=0.9,
    accuracy_standard_error=0.02, f1_score=0.91, precision=0.93, recall=0.90,
)

add_evaluation_results(file_path=path, scorer_identifier=ident, eval_hash="hash_a", metrics=metrics)
path.write_text(path.read_text() + '{"hash_b": "b", "metrics": {"acc\n', encoding="utf-8")  # torn line

replace_evaluation_results(file_path=path, scorer_identifier=ident, eval_hash="hash_new", metrics=metrics)
print("torn line survived the rewrite:", '{"hash_b"' in path.read_text(encoding="utf-8"))

path.write_bytes(path.read_bytes() + b'{"eval_hash": "bad", "metrics": \xff\xfe}\n')  # undecodable byte
replace_evaluation_results(file_path=path, scorer_identifier=ident, eval_hash="hash_new2", metrics=metrics)
print("entries left:", [e["eval_hash"] for e in _load_jsonl(path)])

No network, no API key, no GPU.

Expected Results

The rewrite removes only the entry with the matching eval_hash. An unparseable line is left as it was, and a file that cannot be read is not rewritten at all.

Actual Results

On fc692226 (current main):

Invalid JSON at line 2 in /tmp/pyrit-issue-repro/registry.jsonl: Unterminated string starting at: line 1 column 29 (char 28)
Failed to load registry from /tmp/pyrit-issue-repro/registry.jsonl: 'utf-8' codec can't decode byte 0xff in position 882: invalid start byte
torn line survived the rewrite: False
entries left: ['hash_new2']

The torn line was deleted, and after the failed read hash_a was deleted too — with no exception reaching the caller.

Screenshots

Not applicable.

Versions

• OS: macOS 27.2 (arm64)
• Python version: 3.11.15
• PyRIT version: 1.2.0.dev0, run from source at fc692226
• pyrit.show_version() was not run; the repro above only touches pyrit.score and pyrit.models.