Skip to content

fix: repo review cleanup — CI, eval harness, manifests, and docs - #35

Open
samhita-alla wants to merge 1 commit into
flyteorg:mainfrom
samhita-alla:review-fixes
Open

fix: repo review cleanup — CI, eval harness, manifests, and docs#35
samhita-alla wants to merge 1 commit into
flyteorg:mainfrom
samhita-alla:review-fixes

Conversation

@samhita-alla

Copy link
Copy Markdown
Collaborator

CI (.github/workflows/skill-evals.yml):

  • quote the flyte pin: unquoted 'flyte>=2.5.0' was a shell redirect that installed unconstrained flyte and wrote pip output to a file named '=2.5.0'
  • add a harness-tests job so the evals unit tests actually run in CI
  • trigger the workflow on changes to itself

Eval harness:

  • select.py: map evals/scenarios//** changes to that skill so scenario-only PRs run evals instead of skipping them
  • evaluate.py: fix the expect_status tautology (X and (Y or X)) that made real_run.expect_status never actually assert
  • runners/base.py: exclude .hermes and opencode.json from workspace snapshots (.hermes leaked the installed SKILL.md into judge prompts as agent output, inflating treatment scores; opencode.json leaked the GLM API key)
  • eval_wf.py: surface failed flyte.map elements as errored scorecard rows via a new errored_scenario() helper instead of silently dropping them
  • test_report.py: rewrite stale pass/fail-era tests against the rating-based report, covering the skipped/errored/regression paths

Config:

  • commit .agents/plugins/marketplace.json — the canonical Codex marketplace path per the OpenAI plugin spec — with a description on the plugin entry
  • align the plugin descriptions across all four manifests
  • marketplace description: 'Claude Code skills' -> 'Agent skills' (the repo also targets Codex, Hermes, opencode, and pi)
  • bump package.json to 0.3.0 to match the plugin manifests
  • gitignore .venv/, *.egg-info/, *.zip, __MACOSX/

Docs:

  • READMEs: lead with the plugin being for Claude Code and OpenAI Codex (full plugin incl. MCP servers) plus any Agent Skills harness, and give Codex its own install section — it gets the MCP servers, not just skills
  • root README: deploy-flyte-kind supports DigitalOcean only (EC2/GCP belong to deploy-flyte-kind-vm); standardize 'Flyte v2' -> 'Flyte 2'
  • flyte-sdk-types: FlyteFile (a Flyte 1 type) -> flyte.io.File
  • deploy-flyte-kind description: mention the OIDC auth the skill covers

CI (.github/workflows/skill-evals.yml):
- quote the flyte pin: unquoted 'flyte>=2.5.0' was a shell redirect that
  installed unconstrained flyte and wrote pip output to a file named '=2.5.0'
- add a harness-tests job so the evals unit tests actually run in CI
- trigger the workflow on changes to itself

Eval harness:
- select.py: map evals/scenarios/<skill>/** changes to that skill so
  scenario-only PRs run evals instead of skipping them
- evaluate.py: fix the expect_status tautology (X and (Y or X)) that made
  real_run.expect_status never actually assert
- runners/base.py: exclude .hermes and opencode.json from workspace snapshots
  (.hermes leaked the installed SKILL.md into judge prompts as agent output,
  inflating treatment scores; opencode.json leaked the GLM API key)
- eval_wf.py: surface failed flyte.map elements as errored scorecard rows via a
  new errored_scenario() helper instead of silently dropping them
- test_report.py: rewrite stale pass/fail-era tests against the rating-based
  report, covering the skipped/errored/regression paths

Config:
- commit .agents/plugins/marketplace.json — the canonical Codex marketplace
  path per the OpenAI plugin spec — with a description on the plugin entry
- align the plugin descriptions across all four manifests
- marketplace description: 'Claude Code skills' -> 'Agent skills' (the repo
  also targets Codex, Hermes, opencode, and pi)
- bump package.json to 0.3.0 to match the plugin manifests
- gitignore .venv/, *.egg-info/, *.zip, __MACOSX/

Docs:
- READMEs: lead with the plugin being for Claude Code and OpenAI Codex (full
  plugin incl. MCP servers) plus any Agent Skills harness, and give Codex its
  own install section — it gets the MCP servers, not just skills
- root README: deploy-flyte-kind supports DigitalOcean only (EC2/GCP belong to
  deploy-flyte-kind-vm); standardize 'Flyte v2' -> 'Flyte 2'
- flyte-sdk-types: FlyteFile (a Flyte 1 type) -> flyte.io.File
- deploy-flyte-kind description: mention the OIDC auth the skill covers

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant