docs(snapshots): Document SnapshotPreviews selective testing#18277
docs(snapshots): Document SnapshotPreviews selective testing#18277cameroncooke wants to merge 1 commit into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
Document an advanced SnapshotPreviews workflow that renders only selected preview groups in CI while keeping Sentry aware of the full tracked image set. The guide separates name generation from image rendering so skipped snapshots are distinguished from removed ones on selective uploads. Link to the new page from the snapshots index and clarify how SnapshotPreviews emits logical, unprefixed image names.
cameroncooke
force-pushed
the
cameroncooke/EME-1182-selective_snapshots
branch
from
260fb88 to
92cefab
Compare
cameroncooke
changed the title
| Selective testing is an advanced SnapshotPreviews workflow for large suites. Use it when rendering every preview on every pull request is too expensive, but you still want Sentry to tell apart: | ||
|
|
||
| - snapshots rendered in this run; | ||
| - snapshots skipped because their group wasn't selected; |
There was a problem hiding this comment.
nit: remove the semi colons on these two lines
|
|
||
| <Include name="feature-available-for-user-group-early-adopter" /> | ||
|
|
||
| Selective testing is an advanced SnapshotPreviews workflow for large suites. Use it when rendering every preview on every pull request is too expensive, but you still want Sentry to tell apart: |
There was a problem hiding this comment.
nit: I'd put the word "subset" or something somewhere in these two sentences. Just make it dumb obvious that only a subset of the snapshots are being generated
| - snapshots skipped because their group wasn't selected; | ||
| - snapshots intentionally removed from the tracked set. | ||
|
|
||
| The idea is to separate **name generation** from **image rendering**. First, write the complete list of image names for the full tracked suite. Then render only the groups selected for this CI run and upload those images with the complete image-name file. |
There was a problem hiding this comment.
Important point to make: You can totally use selective testing just by uploading a subset of images without the list of image names.
We just won't be able to flag any removals or renames. Including the --all-image-file-names field is an optional add-on to get visibility of the removals and renames.
I think the current wording presents the image file names as a requirement
| ["ModuleB", "ModuleC"] | ||
| } | ||
| } | ||
| ``` |
|
|
||
| ## Step 1: Upload a complete baseline build | ||
|
|
||
| On your base branch, render and upload the full snapshot suite. This is a normal, complete upload. Do not pass an image-name file on the baseline upload. |
There was a problem hiding this comment.
Do not pass an image-name file on the baseline upload.
I feel like this is another example of why we should make it clear near the top that the image name generation stuff is optional ONLY for the selective testing case.
If so, I feel like it might be implicit here that it shouldn't be included for full (non selective) uploads 🤔
idk, curious your thoughts
|
|
||
| ```bash | ||
| TEST_RUNNER_SNAPSHOTS_EXPORT_DIR="/private/tmp/base_snapshots/iphone" \ | ||
| xcodebuild test \ |
There was a problem hiding this comment.
should this also be xcodebuild build-for-testing?
| /private/tmp/snapshotpreviews_root/DerivedData/Build/Products/<generated>.xctestrun | ||
| ``` | ||
|
|
||
| ## Step 3: Generate image names for every selectable group |
There was a problem hiding this comment.
again, would flag this as an optional section/step
also perhaps this step could be after the selected snapshot generation step?
|
|
||
| ## Step 5: Upload the selective build | ||
|
|
||
| Upload the rendered image directory with the complete image-name file. Passing `--all-image-file-names-file` tells Sentry this is a selective upload, so listed images that were not uploaded are treated as skipped rather than removed. |
There was a problem hiding this comment.
Passing --all-image-file-names-file tells Sentry this is a selective upload
--selective tells sentry it's a selective build, but i then made it so that including --all-image-file-names-file implicitly implies that it's selective so that you can just do that rather than --selective --all-image-file-names-file
might be worth making sure this is clear here 🤷
There was a problem hiding this comment.
nit:
"so listed images that were not uploaded are treated as skipped rather than removed."
->
"so listed images that were not uploaded are correctly identified as skipped rather than removed."
2 participants
Add an Apple Snapshots guide for using SnapshotPreviews with Sentry selective testing.
What Changed
This adds a dedicated selective testing page under the Apple Snapshots docs and links to it from the Apple Snapshots setup guide.
The new page explains how to:
SnapshotTestgroups;--all-image-file-names-file.Why This Exists
Selective Sentry uploads need to distinguish images that were skipped in the current run from images that were removed from the snapshot suite. SnapshotPreviews provides the image-name manifest; Sentry uses that manifest to treat listed-but-missing images as skipped rather than removed.
Workflow Constraints
The page documents the constraints users need to account for:
SnapshotTestgroup;Examples
The examples use placeholder project, scheme, and test class names so readers can adapt the workflow to their own app. The page links to the SnapshotPreviews MultiModuleDemo workflow as a complete reference implementation.
Refs EME-1182