Artifacts and how to get them #5585

ShadowCurse · 2025-12-16T16:28:20Z

Changes

make devtool pull latest artifacts from s3
add ability to download multiple custom artifacts
add ability to specify which artifacts to use in tests
add ability to specify artifacts for A/B tests

Reason

Artifacts are rebuilt periodically now, so update our tools to be able to pull latest ones.
Additionally, while we are at it, add ability to download and switch between multiple
artifacts. This also opened a path for A/B tests of artifacts as well.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache 2.0 license. For more information on following Developer
Certificate of Origin and signing off your commits, please check
CONTRIBUTING.md.

PR Checklist

This functionality cannot be added in rust-vmm.

codecov · 2025-12-16T16:35:09Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.23%. Comparing base (a61c603) to head (5e9524c).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5585      +/-   ##
==========================================
- Coverage   83.23%   83.23%   -0.01%     
==========================================
  Files         277      277              
  Lines       29262    29262              
==========================================
- Hits        24357    24356       -1     
- Misses       4905     4906       +1

Flag	Coverage Δ
5.10-m5n.metal	`83.57% <ø> (ø)`
5.10-m6a.metal	`82.91% <ø> (ø)`
5.10-m6g.metal	`80.19% <ø> (ø)`
5.10-m6i.metal	`83.57% <ø> (ø)`
5.10-m7a.metal-48xl	`82.90% <ø> (+<0.01%)`	⬆️
5.10-m7g.metal	`80.19% <ø> (ø)`
5.10-m7i.metal-24xl	`83.54% <ø> (ø)`
5.10-m7i.metal-48xl	`83.54% <ø> (ø)`
5.10-m8g.metal-24xl	`80.18% <ø> (ø)`
5.10-m8g.metal-48xl	`80.18% <ø> (ø)`
6.1-m5n.metal	`83.60% <ø> (+<0.01%)`	⬆️
6.1-m6a.metal	`82.93% <ø> (-0.01%)`	⬇️
6.1-m6g.metal	`80.18% <ø> (-0.01%)`	⬇️
6.1-m6i.metal	`83.60% <ø> (-0.01%)`	⬇️
6.1-m7a.metal-48xl	`82.93% <ø> (ø)`
6.1-m7g.metal	`80.19% <ø> (ø)`
6.1-m7i.metal-24xl	`83.61% <ø> (ø)`
6.1-m7i.metal-48xl	`83.61% <ø> (+<0.01%)`	⬆️
6.1-m8g.metal-24xl	`80.18% <ø> (-0.01%)`	⬇️
6.1-m8g.metal-48xl	`80.18% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

tests/framework/defs.py

tests/conftest.py

tools/devtool

Manciukic · 2025-12-18T11:03:51Z

tools/devtool

 ensure_ci_artifacts() {
-    if ! command -v aws >/dev/null; then
-      die "AWS CLI not installed, which is required for downloading artifacts for integration tests."
+    local artifacts=$1


nit: please use singular and plural consistently. Also, it's not clear whether this is an S3 key, a URI, or something else

tools/devtool

.buildkite/pipeline_perf.py

tools/devtool

I selected approve by mistakr

JackThomson2 · 2026-01-06T15:21:09Z

tools/ab_test.py

+    run_parser.add_argument(
+        "--artifacts-a",
+        help="Name of the artifacts directory in the build/artifacts to use for revision A test. If the directory does not exist, the name will be treated as S3 path and artifacts will be downloaded from there.",
+        # Type is string since it can be an s3 paht which if passed to `Path` constructor


JackThomson2 · 2026-01-06T15:37:12Z

tools/devtool


-ensure_ci_artifacts() {
-    if ! command -v aws >/dev/null; then
-      die "AWS CLI not installed, which is required for downloading artifacts for integration tests."


Did we mean to drop this? Seems like it still would be useful

I think ok_or_die "Failed to download artifacts using awscli!" line serves same function here.

JackThomson2 · 2026-01-06T15:42:41Z

tools/devtool

+cmd_set_current_artifacts() {
+    local artifacts=$1
+    if [ -z $artifacts ]; then
+      say "No artifacts were specified"


Should the function bail out here?

JackThomson2 · 2026-01-06T15:49:04Z

Side note it would be nice if we could fetch the artifacts inside docker, so we wouldn't need AWS cli on the host

JackThomson2 · 2026-01-06T16:04:25Z

tests/README.md

 them to the common s3 bucket.*\
-`A4:` Add your custom images to the `build/img` subdirectory in the Firecracker
-source tree. This directory is bind-mounted in the container and used as a local
-image cache.


Could we get a new section to show how to test with different artifacts

Manciukic

Just a few more bits for small fixes and clarity.

Manciukic · 2026-01-07T13:49:41Z

tests/framework/defs.py

 SUPPORTED_HOST_KERNELS = ["5.10", "6.1"]

-IMG_DIR = Path(DEFAULT_TEST_SESSION_ROOT_PATH) / "img"
+ARTIFACT_DIR = Path(DEFAULT_TEST_SESSION_ROOT_PATH) / "current_artifacts"


don't we need to read the current_artifacts file to set the ARTIFACT_DIR correctly?

So this code is supposed to execute in the devctr, where artifacts are copied to /src/current_artifacts. I guess using same name for the directory with actual artifacts in the devctr and for link file outside devctr is confusing. I can change python code to use artifacts_dir for the name if you want.

Manciukic · 2026-01-07T13:51:39Z

tests/framework/defs.py

+        open(Path(LOCAL_BUILD_PATH) / "current_artifacts", "r", encoding="utf-8")
+        .read()


nit: a more compact way is to use read_text('utf-8') directly on the Path object. It also prevents the fd leak we have here.

Manciukic · 2026-01-07T13:59:01Z

tools/devtool

+
+    if [ ! -z $FORCE_ARTIFACT_DOWNLOAD ]; then
+      say "Removing " $local_artifacts_path
+      rm -r $local_artifacts_path


should this also be -f (force)?

Manciukic · 2026-01-07T14:02:59Z

tools/test.sh

+  cp -ruvfL $(cat build/current_artifacts) /srv/current_artifacts
+else
+  # The directory must exist for pytest to function
+  mkdir -p /srv/current_artifacts


why are we creating it as a dir if it's otherwise a file ?

it is a file outside a devctr and a directory inside

Manciukic · 2026-01-07T14:09:21Z

tools/devtool

 # File with a single line specifing the name of the
 # currently used artifacts
-LOCAL_ARTIFACTS_CURRENT_DIR_FILE="build/current_artifacts"
+LOCAL_CURRENT_ARTIFACTS_LINK_PATH="build/current_artifacts"


nit: I wouldn't call it a link if it's not actually one

Manciukic · 2026-01-07T14:12:55Z

tools/devtool

+    echo "        like 'download_ci_artifacts'. Alternatively it is possible to manually write local "
+    echo "        path to artifacts directory into $LOCAL_CURRENT_ARTIFACTS_LINK_PATH file"
+    echo ""
+    echo "    ensure_current_artifacts_are_set_up s3_uri/directory name"


The current name is inconsistent with the actual command cmd_ensure_current_artifacts_are_set.
Maybe this could have an easier-to-use name, like ensure_artifacts.

Btw, what's the process to update the local artifacts to the latest ones? It seems like it's currently a manual combination of commands. The quickest would be to delete the current_artifacts file and call ensure_artifacts.

Last nit, isn't the parameter optional?

renamed to cmd_ensure_current_artifacts. Yes, updating is just removing either a current_artifacts or deleting whole artifacts dir would work too.

The snapshot version check was iterating over last several FC versions pulled from S3, but it was unnecessary since there were no cross FC checks. Remove checks of old FC versions and only test the current version. Also remove fixup state from `setup-ci-artifacts.sh` since it is redundant now. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Instead of using fixed artifact path for a current FC version pull newest artifacts from S3. This way we can update them independently while keeping old versions around. Unfortunately there is no simple way to query newest added directory in S3, so do the next best thing of looking through all files in all directories and determining the `newest` directory be the `LastModified` status of files inside. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

The old setup used `build/img` as the base directory for artifacts pulled from S3. In S3 the artifacts directory contains 1 subdirectory per architecture (x86_64 and aarch64). This was causing final local artifacts path to be `build/img/x86_64` or `build/img/aarch64`. This commit changes the structure a to have a separate `build/artifacts` directory with subdirectories containing different versions of artifacts. The path to currently used artifacts will be placed in `build/current_artifacts` file. This make is easy to switch between multiple versions of artifacts without a need to delete/download them. This also opens a door for A/B testing of artifacts. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Downloading artifacts is a separate step that should not mess with currently selected artifacts. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Modify `download_ci_artifacts` to accept artifacts paths as input and download by passing these paths to `ensure_ci_artifacts`. `ensure_ci_artifacts` will default to latest s3 artifacts if no args were provided. `--force` still works and only deletes specified artifacts. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add arguments names for binary paths for ab_test.py Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Update `ab_test.py` with ability to accept custom artifacts for A and B runs. Additionally update `pipeline_perf.py` as well. Now REVISION_A_ARTIFACTS and REVISION_B_ARTIFACTS environment variables specify custom artifacts which will be used in A/B test Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Some tests like style checks do not require artifacts to be present. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

There is no reason for style check to download artifacts Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add missing descriptions for additional arguments Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

The command is used internally in the `devtool` to make sure there are at lest some current artifacts set up. This behaviour is also required in the popular docker test, so convert it to a callable command. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

Add `--artifacts` option to `devtool test` to simplify changing between different sets of artifacts. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

- rename ensure_current_artifacts_are_set_up to ensure_current_artifacts to shorten a name a bit - rename LOCAL_ARTIFACTS_CURRENT_DIR_FILE to LOCAL_CURRENT_ARTIFACTS_FILE to make name more descriptive - Simplify logs regarding artifacts status - remove ` symbols from cmd_help as bash was interpreting them as actual commands to execute - other minor renamings Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

ShadowCurse requested review from Manciukic, kalyazin, pb8o and xmarcalx as code owners December 16, 2025 16:28

ShadowCurse force-pushed the pull_latest_s3_artifacts branch from 7682ec8 to 24326d6 Compare December 16, 2025 16:31

ShadowCurse force-pushed the pull_latest_s3_artifacts branch 7 times, most recently from 1de4267 to 59fb38f Compare December 17, 2025 17:10

ShadowCurse self-assigned this Dec 17, 2025

ShadowCurse added Status: Awaiting review Indicates that a pull request is ready to be reviewed Type: Enhancement Indicates new feature requests labels Dec 17, 2025

Manciukic previously approved these changes Dec 18, 2025

View reviewed changes

ShadowCurse force-pushed the pull_latest_s3_artifacts branch 7 times, most recently from 075b628 to 2b3c8b9 Compare December 19, 2025 15:50

ShadowCurse force-pushed the pull_latest_s3_artifacts branch 2 times, most recently from 89f232b to ccd1bd1 Compare January 5, 2026 13:29

JackThomson2 reviewed Jan 6, 2026

View reviewed changes

ShadowCurse force-pushed the pull_latest_s3_artifacts branch 2 times, most recently from 49138bf to eb6bbc3 Compare January 7, 2026 12:51

Manciukic reviewed Jan 7, 2026

View reviewed changes

ShadowCurse force-pushed the pull_latest_s3_artifacts branch 4 times, most recently from 3965541 to e446012 Compare January 8, 2026 15:34

ShadowCurse added 13 commits January 8, 2026 15:53

refactor: split downloading and setting current artifacts

ecc91b0

Downloading artifacts is a separate step that should not mess with currently selected artifacts. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

refactor: specify explicit arguments for ab_test.py

9122ccb

Add arguments names for binary paths for ab_test.py Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

chore(test.sh): skip aritfact setup if no current is specified

7a99dde

Some tests like style checks do not require artifacts to be present. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

chore(devtool): remove artifacts check from checkstyle command

51a701a

There is no reason for style check to download artifacts Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

chore(devtool): update cmd_test help

87c6ebe

Add missing descriptions for additional arguments Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

feat(devtool): specify wanted artifacts for test command

d5815d5

Add `--artifacts` option to `devtool test` to simplify changing between different sets of artifacts. Signed-off-by: Egor Lazarchuk <yegorlz@amazon.co.uk>

ShadowCurse force-pushed the pull_latest_s3_artifacts branch from e446012 to 5e9524c Compare January 8, 2026 15:55

		open(Path(LOCAL_BUILD_PATH) / "current_artifacts", "r", encoding="utf-8")
		.read()

Artifacts and how to get them #5585

Are you sure you want to change the base?

Artifacts and how to get them #5585

Conversation

ShadowCurse commented Dec 16, 2025

Changes

Reason

License Acceptance

PR Checklist

Uh oh!

codecov bot commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JackThomson2 commented Jan 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Manciukic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Dec 16, 2025 •

edited

Loading