Repository navigation
CI: add e2e runs from benchx_examples - #102
Merged
Merged
Conversation
…orkflow
prototype/examples/{demo-suite,demo,archery-local} and prototype/demo.sh now
live in benchx-dev/benchx_examples as examples/arrow/{demo-suite,demo,archery-local}
(the demo script is examples/arrow/demo/run.sh).
- Remove them from this repository, and the archery-local step from tests.yml.
- e2e.yml gets a second job, arrow, that runs demo/run.sh and
archery-local/run.sh from benchx_examples and uploads their output. The
header now describes both jobs.
- prototype/README.md and docs/design/prototype-design.md point to
benchx_examples where they described these examples.
The arrow job needs examples/arrow on the benchx_examples ref it clones
(main by default), so it passes once that move is merged there.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…kflow prototype/examples/pyperf-bench now lives in benchx-dev/benchx_examples as examples/pyperf-suite. - Remove it from this repository, and its step from tests.yml. - e2e.yml gets a third job, pyperf-suite, that runs run.sh from benchx_examples and uploads its out/ directory. The header lists all three jobs. - prototype/README.md points to benchx_examples where it described the example. The job needs examples/pyperf-suite on the benchx_examples ref it clones (main by default). benchx_examples PR #3 adds the same files unchanged, which do not run from there; the port with adapted instructions is needed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A fourth job, matmul-params, runs the example from benchx_examples: it installs benchx, numpy and the CPU build of torch, and runs the quick version from the example's README (--processes 2 --values 2 --min-time 0.01). It uploads out/, which holds the result files and the store. The job's run steps were replayed locally against benchx_examples main (3a6b668): six cases, twelve results, all successful. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The example lives in benchx-dev/benchx_examples now, and the e2e workflow runs it from there. The files in this directory were the same, except for the README's install lines. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A fifth job, numpy-venvs, runs the example from benchx_examples: it installs benchx in editable mode, from which run.sh installs benchx in the two venvs it makes (numpy 1.26.4 and 2.0.2, three rounds), then it runs run.sh. It uploads out/. The job's run steps were replayed locally against the example as it is on the numpy-venvs-example branch of benchx_examples: all steps passed. That branch is not on main there yet, and the job clones main, so the job fails until it is; a run on demand can name the branch (examples_ref). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The example lives in benchx-dev/benchx_examples now, and the e2e workflow runs it from there, so the tests workflow no longer runs it, and the README and prototype-design.md point at the new place. The files there are the ones removed here, except that run.sh no longer assumes the benchx package is two levels up, the README says how to install benchx and links to UC-04 in this repository, and .gitignore also ignores __pycache__/. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Since the CI is green, and the coverage loss is fairly trivial and easy to fix (end-to-end examples only run on Linux, with a single python version), I'm going to self-merge it now, before #104 goes in. Post-merge reviews are still most appreciated! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(Resend gh-98 , which got stuck in GH timeouts.)
This PR moves all end-to-end demos from prototype/examples to https://github.com/benchx-dev/benchx_examples, as discussed in the call last week.
Additional machinery:
benchx/prototype/examplesdirectory, since all its contents is now inbenchx_examples, verbatim;tests.ymljob only contains the tests for the prototype itself;e2e.yml, in separate jobs for numpy-spin, arrow and bare pyperf demos.The CI times are: under 3 mins for the unit tests, ~6 mins for the arrow/archery demos, and ~6 mins for numpy-spin.
The only coverage that's lost is that end-to-end examples only run on Linux (were previously run on MacOS, too). Not a big deal I'd think, but am happy to make them CI matrices, too.