fix: repo review cleanup — CI, eval harness, manifests, and docs - #35
Open
samhita-alla wants to merge 1 commit into
Open
fix: repo review cleanup — CI, eval harness, manifests, and docs#35samhita-alla wants to merge 1 commit into
samhita-alla wants to merge 1 commit into
Conversation
CI (.github/workflows/skill-evals.yml): - quote the flyte pin: unquoted 'flyte>=2.5.0' was a shell redirect that installed unconstrained flyte and wrote pip output to a file named '=2.5.0' - add a harness-tests job so the evals unit tests actually run in CI - trigger the workflow on changes to itself Eval harness: - select.py: map evals/scenarios/<skill>/** changes to that skill so scenario-only PRs run evals instead of skipping them - evaluate.py: fix the expect_status tautology (X and (Y or X)) that made real_run.expect_status never actually assert - runners/base.py: exclude .hermes and opencode.json from workspace snapshots (.hermes leaked the installed SKILL.md into judge prompts as agent output, inflating treatment scores; opencode.json leaked the GLM API key) - eval_wf.py: surface failed flyte.map elements as errored scorecard rows via a new errored_scenario() helper instead of silently dropping them - test_report.py: rewrite stale pass/fail-era tests against the rating-based report, covering the skipped/errored/regression paths Config: - commit .agents/plugins/marketplace.json — the canonical Codex marketplace path per the OpenAI plugin spec — with a description on the plugin entry - align the plugin descriptions across all four manifests - marketplace description: 'Claude Code skills' -> 'Agent skills' (the repo also targets Codex, Hermes, opencode, and pi) - bump package.json to 0.3.0 to match the plugin manifests - gitignore .venv/, *.egg-info/, *.zip, __MACOSX/ Docs: - READMEs: lead with the plugin being for Claude Code and OpenAI Codex (full plugin incl. MCP servers) plus any Agent Skills harness, and give Codex its own install section — it gets the MCP servers, not just skills - root README: deploy-flyte-kind supports DigitalOcean only (EC2/GCP belong to deploy-flyte-kind-vm); standardize 'Flyte v2' -> 'Flyte 2' - flyte-sdk-types: FlyteFile (a Flyte 1 type) -> flyte.io.File - deploy-flyte-kind description: mention the OIDC auth the skill covers Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Samhita Alla <aallasamhita@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI (.github/workflows/skill-evals.yml):
Eval harness:
Config:
Docs: