Skip to content

Parse hellaswag_arabic endings with ast.literal_eval - #1348

Open
Nakul-Sinha wants to merge 1 commit into
huggingface:mainfrom
Nakul-Sinha:fix/arabic-hellaswag-literal-eval
Open

Parse hellaswag_arabic endings with ast.literal_eval#1348
Nakul-Sinha wants to merge 1 commit into
huggingface:mainfrom
Nakul-Sinha:fix/arabic-hellaswag-literal-eval

Conversation

@Nakul-Sinha

Copy link
Copy Markdown

Fixes #1293

Problem

hellaswag_arabic_pfn parses the endings field with the builtin eval():

endings = [
    re.sub(r"\[.*?\]", "", e) for e in eval(line["endings"])
]  # endings is a string representation of a list

The field holds a string representation of a list, but eval() executes arbitrary Python rather than just parsing a literal. The value comes from OALL/AlGhafa-Arabic-LLM-Benchmark-Translated, a community-hosted dataset on the Hub, so a crafted endings value in that dataset or any later revision of it would run code on whichever machine is running the evaluation.

Change

Use ast.literal_eval, which accepts the same literal syntax but evaluates only literals, and add the ast import.

Testing

python -m py_compile src/lighteval/tasks/multilingual/tasks/arabic.py passes.

Checked that the parsing behaviour is unchanged for real inputs and that the injection no longer executes:

legit    -> ['ending ', 'ending two', 'ending three']
arabic   -> ['النهاية', 'الثانية']
dq quote -> ['a', 'b']
malicious ("__import__('os').system(...)") -> blocked: ValueError
malicious ("eval('1+1')")                  -> blocked: ValueError

Single and double quoted lists, and non-ASCII contents, all parse identically to before. main_inspect.py:568 also calls eval(...), but that is inspect_ai's own eval function rather than the builtin, so it is unrelated and left alone.

AI assistance was used for this change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hellaswag_arabic_pfn uses eval() instead of ast.literal_eval() on a dataset field, arbitrary code execution from an untrusted HF dataset

1 participant