Add comprehensive cost reporting system for benchmark evaluations

[juanmichelini]

Summary

This PR introduces cost reporting for benchmark evaluations, providing both standalone script functionality and integrated cost reporting within the evaluation pipeline.

Key Features

1. Standalone Cost Reporting Script (benchmarks/utils/report_costs.py)

• Comprehensive cost calculation: Extracts and sums accumulated_cost from all lines in JSONL files (not just the last line)
• Multi-file support: Handles both main output.jsonl and critic attempt files (output.critic_attempt_*.jsonl)
• Flexible input: Accepts either directory path or specific output.jsonl file path
• Dual output formats:
  • Console output with formatted cost breakdown
  • JSON output saved as cost_report.jsonl in the same directory
• Error handling: Robust error handling with detailed logging
• Executable script: Can be run directly from command line

2. Integrated Cost Reporting in Evaluation Pipeline

• Automatic integration: Cost reporting now runs as the final step in eval_infer.py
• Shared functionality: Uses the same generate_cost_report function from the utils module
• Reusable across benchmarks: The generate_cost_report function is available for other benchmark scripts

3. Accurate Cost Calculations

• Fixed critical bug: Previously only captured the last line's cost; now correctly sums all accumulated costs
• Proper field mapping:
  • only_main_output_cost: Cost from main output.jsonl file
  • sum_critic_files: Total cost from all critic attempt files
  • total_cost: Set to sum_critic_files (since main output is a subset of critic outputs)

Testing Results

Extensive testing on SWE-bench evaluation outputs:

• Main output.jsonl: 500 lines, $1,139.35
• Critic files: 3 files totaling 515 lines, $1,157.72
• Total cost: $1,157.72 (correctly excludes double-counting)

Usage Examples

Standalone Script

# Run on directory containing output.jsonl files
python benchmarks/utils/report_costs.py /path/to/eval/directory

# Run on specific output.jsonl file
python benchmarks/utils/report_costs.py /path/to/output.jsonl

Integrated in Evaluation

Cost reporting now runs automatically as the final step when running evaluations.

Files Changed

• benchmarks/utils/report_costs.py: New comprehensive cost reporting script
• benchmarks/swe_bench/eval_infer.py: Integrated cost reporting as final step

Testing

• ✅ Standalone script functionality tested with real evaluation data
• ✅ Integration testing with eval_infer.py pipeline
• ✅ JSON output format validation
• ✅ Error handling for edge cases (missing files, malformed data)
• ✅ Cross-platform compatibility (Linux/macOS)
• ✅ Pre-commit hooks passing (ruff, pyright, pycodestyle)

[simonrosenberg]

why use epilog instead of adding this epilog to the description?

[juanmichelini]

it's so the examples go after the params not before. Like here https://www.blog.pythonlibrary.org/2015/10/08/a-intro-to-argparse/#:~:text=Getting%20Started,the%20command%2Dline%20after%20all. In this case both would work though