Skip to content

docs: document post-processing patterns for eval results (markdown summary, CI integration) #700

@christso

Description

@christso

Objective

Document how to build post-processing scripts/skills that parse AgentV output artifacts (results.jsonl, results.json, JUnit XML) into presentation formats like markdown summaries, PR comments, and CI job summaries.

Design Latitude

  • Add an Astro docs page covering post-processing patterns
  • Include example scripts for common use cases:
    • Parse results.jsonl → markdown summary table
    • Parse results.json → GitHub Actions job summary ($GITHUB_STEP_SUMMARY)
    • Parse results.jsonl::error:: / ::warning:: annotations
  • Optionally create a reusable skill for markdown summary generation
  • Keep this outside of AgentV core — presentation is a downstream concern

Acceptance Signals

  • Astro docs page with at least 2 example scripts (markdown summary, GitHub job summary)
  • Examples show how to read AgentV output formats and produce formatted output
  • Skill definition (SKILL.md or similar) for reusable summary generation if applicable

Non-Goals

  • No changes to AgentV core output pipeline
  • No new built-in output writer
  • Not a GitHub Actions reusable workflow (separate scope)

Context

Originally proposed as a built-in Markdown output writer. Reconsidered — markdown summary is a presentation concern better handled downstream. AgentV already generates structured output (JSONL, JSON, JUnit XML). This matches how microsoft/skills does it: their workflow reads results.json and formats the GitHub job summary inline.

See microsoft/skills harness research.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions