Build/Test Tools: Run the Add New Media E2E spec in parallel.

tests/e2e/README.md

@@ -1 +1 @@
-# E2E TestsEnd-To-End (E2E) tests for WordPress.## Running the testsThe e2e tests require a production-like environment to run. By default, they will assume an environment is available at `http://localhost:8889`, with username `admin` and password `password`.If you don't already have an environment ready, you can set one up by following [these instructions](https://github.com/WordPress/wordpress-develop/blob/master/README.md).Then you can launch the tests by running:```npm run test:e2e```which will run the test suite using a headless browser.If your environment has a different url, username or password to the default, you can provide the base URL, username and password like this:```WP_BASE_URL=http://mycustomurl WP_USERNAME=username WP_PASSWORD=password npm run test:e2e```**DO NOT run these tests in an actual production environment, as they will delete all your content.**For debugging purposes, you might want to follow the test visually. You can do so by running the tests in an interactive mode:```npm run test:e2e -- --ui```[UI Mode](https://playwright.dev/docs/test-ui-mode) let's you explore, run and debug tests with a time travel experience complete with watch mode.All test files are loaded into the testing sidebar where you can expand each file and describe block to individually run, view, watch and debug each test.You can also run a single test file separately:```npm run test:e2e tests/e2e/specs/hello.test.js```## ProjectsThe suite has three Playwright projects, defined in `playwright.config.js`:* `all`: every spec, one at a time on one worker. `npm run test:e2e` runs this project.* `parallel`: specs that can share the site with other tests. Spec files run at the same time on several workers, one for every two CPU cores. Tests inside one file still run in order. Run it with `npm run test:e2e -- --project=parallel`.* `serial`: specs that break the site for other tests. They run one at a time on one worker. Run it with `npm run test:e2e -- --project=serial`.CI runs `parallel` and `serial` in separate jobs.New specs go in `parallel` by default. Add a spec to `SERIAL_SPECS` if it makes the site unusable for other tests. If it changes content or settings that other tests use, for example with `requestUtils.deleteAllPosts()`, follow [Parallel safety](#parallel-safety).Before you open a pull request with a new spec, run its project a few times with `--shuffle`. Playwright then runs the spec files in a random order, so a spec that needs another spec to run first, or that breaks a spec that runs after it, fails:```npm run test:e2e -- --project=parallel --shuffle```Playwright prints the seed of each run, for example `shuffle seed 317753389`. To run the same order again, give the seed: `--shuffle 317753389`.## Parallel safetySpecs in `parallel` share one site. A test that changes shared data can break a test in another file that runs at the same time. To make a spec safe, use the first of these that works:1. Scope the data. Let the test create its own data, for example a post with a unique title, and read only that data.2. If the test must change shared data, such as a site setting, give it a lock. Do this only if every spec that writes the same data holds the same lock.3. If you cannot lock every spec that writes the data, add the spec to `SERIAL_SPECS`.A lock is a name. Playwright never runs two tests that hold the same lock name at the same time. A lock only schedules tests. It does not isolate data: a test without the lock can still change the data while a locked test runs.The lock names are in `utils/locks.js`. Import the constant, so that all specs use the same name. Playwright gives no error for a lock name that no other test uses, or for an import name that is not in `utils/locks.js`:```import { GENERAL_SETTINGS } from '../utils/locks';test.describe( 'Settings -> General', { lock: GENERAL_SETTINGS }, () => {} );```A form save writes every field on the page, not only the field that the test changed. A spec that saves the General Settings form (`options-general.php`) must hold `GENERAL_SETTINGS`. A spec in `parallel` that changes a General Settings option in a different way, for example with `siteSettings` or `siteOptions`, must also hold it.## Logged-out pagesMost specs run logged in as the admin user. There are two ways to run a test logged out:* To make the `page` of a whole `describe` group logged out, add `test.use( { storageState: { cookies: [], origins: [] } } )` to the group. See `specs/dashboard.test.js`.* To use a logged-out page next to the logged-in `page` in one test, import `test` and `expect` from `../fixtures` and use the `loggedOutPage` fixture. Its browser context has no cookies, and it closes when the test ends. Unlike `page`, it resolves a URL that starts with `/` against the host only, not the full `WP_BASE_URL`, so it does not work with a site in a subdirectory. See `specs/cache-control-headers-directives.test.js`.## FixturesTo create posts, pages or users, or to change a setting, an option, or a must-use plugin, import `test` and `expect` from `../fixtures`, and use one of the fixtures below. Each fixture cleans up when the test ends, also if the test fails. If the cleanup fails, the test fails.* `siteSettings`: reads and changes the settings of `/wp/v2/settings` through REST, with `get()`, `remember( keys )` and `set( patch )`. It restores only the keys passed to `remember` or `set`. See `specs/privacy.test.js`.* `siteOptions`: reads and changes options through WP-CLI, with `get( names )`, `remember( names )` and `set( values )`. It reads and restores the stored `gmt_offset`, not the value calculated for a named timezone. It restores only the keys passed to `remember` or `set`. See `specs/general-settings-invalid-timezone.test.js`.* `muPlugin`: writes a must-use plugin with `write( fileName, php )`, and deletes the file when the test ends. Call `write` before the test visits a page. See `specs/fatal-error-handler.test.js`.* `postFactory`: creates posts with `create( data )` and pages with `createPage( data )`, and force-deletes them when the test ends. `uniqueTitle( prefix )` adds a UUID to the prefix. See `specs/edit-posts.test.js`.* `userFactory`: creates users with `create( { roles, ...data } )`, which returns `{ user, username, password }`. For a user that the test creates through a form, call `adopt( username )` before the form is submitted. When the test ends, it deletes each user and the user's auto-drafts. To log in as the user, import `signIn( page, username, password )` from `../fixtures/user`. See `specs/users.test.js`.If the test changes a setting or an option through a form, first call `remember` with an array of the keys that the form changes. The fixture then reads the current value, without a write, and restores it when the test ends. `remember` returns the current values.## Documentation* Block Editor Handbook end to end testing overview: https://developer.wordpress.org/block-editor/contributors/code/testing-overview/#end-to-end-testing* Gutenberg e2e-test-utils-playwright package API docs: https://github.com/WordPress/gutenberg/tree/trunk/packages/e2e-test-utils-playwright* Playwright API docs: https://playwright.dev/docs (the version we are using is indicated in the `@wordpress/scripts` package: https://github.com/WordPress/gutenberg/blob/trunk/packages/scripts/package.json)
+# E2E TestsEnd-To-End (E2E) tests for WordPress.## Running the testsThe e2e tests require a production-like environment to run. By default, they will assume an environment is available at `http://localhost:8889`, with username `admin` and password `password`.If you don't already have an environment ready, you can set one up by following [these instructions](https://github.com/WordPress/wordpress-develop/blob/master/README.md).Then you can launch the tests by running:```npm run test:e2e```which will run the test suite using a headless browser.If your environment has a different url, username or password to the default, you can provide the base URL, username and password like this:```WP_BASE_URL=http://mycustomurl WP_USERNAME=username WP_PASSWORD=password npm run test:e2e```**DO NOT run these tests in an actual production environment, as they will delete all your content.**For debugging purposes, you might want to follow the test visually. You can do so by running the tests in an interactive mode:```npm run test:e2e -- --ui```[UI Mode](https://playwright.dev/docs/test-ui-mode) let's you explore, run and debug tests with a time travel experience complete with watch mode.All test files are loaded into the testing sidebar where you can expand each file and describe block to individually run, view, watch and debug each test.You can also run a single test file separately:```npm run test:e2e tests/e2e/specs/hello.test.js```## ProjectsThe suite has three Playwright projects, defined in `playwright.config.js`:* `all`: every spec, one at a time on one worker. `npm run test:e2e` runs this project.* `parallel`: specs that can share the site with other tests. Spec files run at the same time on several workers, one for every two CPU cores. Tests inside one file still run in order. Run it with `npm run test:e2e -- --project=parallel`.* `serial`: specs that break the site for other tests. They run one at a time on one worker. Run it with `npm run test:e2e -- --project=serial`.CI runs `parallel` and `serial` in separate jobs.New specs go in `parallel` by default. Add a spec to `SERIAL_SPECS` if it makes the site unusable for other tests. If it changes content or settings that other tests use, for example with `requestUtils.deleteAllPosts()`, follow [Parallel safety](#parallel-safety).Before you open a pull request with a new spec, run its project a few times with `--shuffle`. Playwright then runs the spec files in a random order, so a spec that needs another spec to run first, or that breaks a spec that runs after it, fails:```npm run test:e2e -- --project=parallel --shuffle```Playwright prints the seed of each run, for example `shuffle seed 317753389`. To run the same order again, give the seed: `--shuffle 317753389`.## Parallel safetySpecs in `parallel` share one site. A test that changes shared data can break a test in another file that runs at the same time. To make a spec safe, use the first of these that works:1. Scope the data. Let the test create its own data, for example a post with a unique title, and read only that data.2. If the test must change shared data, such as a site setting, give it a lock. Do this only if every spec that writes the same data holds the same lock.3. If you cannot lock every spec that writes the data, add the spec to `SERIAL_SPECS`.A lock is a name. Playwright never runs two tests that hold the same lock name at the same time. A lock only schedules tests. It does not isolate data: a test without the lock can still change the data while a locked test runs.The lock names are in `utils/locks.js`. Import the constant, so that all specs use the same name. Playwright gives no error for a lock name that no other test uses, or for an import name that is not in `utils/locks.js`:```import { GENERAL_SETTINGS } from '../utils/locks';test.describe( 'Settings -> General', { lock: GENERAL_SETTINGS }, () => {} );```A form save writes every field on the page, not only the field that the test changed. A spec that saves the General Settings form (`options-general.php`) must hold `GENERAL_SETTINGS`. A spec in `parallel` that changes a General Settings option in a different way, for example with `siteSettings` or `siteOptions`, must also hold it.## Logged-out pagesMost specs run logged in as the admin user. There are two ways to run a test logged out:* To make the `page` of a whole `describe` group logged out, add `test.use( { storageState: { cookies: [], origins: [] } } )` to the group. See `specs/dashboard.test.js`.* To use a logged-out page next to the logged-in `page` in one test, import `test` and `expect` from `../fixtures` and use the `loggedOutPage` fixture. Its browser context has no cookies, and it closes when the test ends. Unlike `page`, it resolves a URL that starts with `/` against the host only, not the full `WP_BASE_URL`, so it does not work with a site in a subdirectory. See `specs/cache-control-headers-directives.test.js`.## FixturesTo create posts, pages or users, to upload attachments, or to change a setting, an option, or a must-use plugin, import `test` and `expect` from `../fixtures`, and use one of the fixtures below. Each fixture cleans up when the test ends, also if the test fails. If the cleanup fails, the test fails.* `siteSettings`: reads and changes the settings of `/wp/v2/settings` through REST, with `get()`, `remember( keys )` and `set( patch )`. It restores only the keys passed to `remember` or `set`. See `specs/privacy.test.js`.* `siteOptions`: reads and changes options through WP-CLI, with `get( names )`, `remember( names )` and `set( values )`. It reads and restores the stored `gmt_offset`, not the value calculated for a named timezone. It restores only the keys passed to `remember` or `set`. See `specs/general-settings-invalid-timezone.test.js`.* `muPlugin`: writes a must-use plugin with `write( fileName, php )`, and deletes the file when the test ends. Call `write` before the test visits a page. See `specs/fatal-error-handler.test.js`.* `postFactory`: creates posts with `create( data )` and pages with `createPage( data )`, and force-deletes them when the test ends. `uniqueTitle( prefix )` adds a UUID to the prefix. See `specs/edit-posts.test.js`.* `attachments`: uploads a file with `upload( absolutePath )`, records an attachment that the browser uploaded with `track( id )`, and deletes them when the test ends. See `specs/media-new-client-side-upload.test.js`.* `userFactory`: creates users with `create( { roles, ...data } )`, which returns `{ user, username, password }`. For a user that the test creates through a form, call `adopt( username )` before the form is submitted. When the test ends, it deletes each user and the user's auto-drafts. To log in as the user, import `signIn( page, username, password )` from `../fixtures/user`. See `specs/users.test.js`.If the test changes a setting or an option through a form, first call `remember` with an array of the keys that the form changes. The fixture then reads the current value, without a write, and restores it when the test ends. `remember` returns the current values.A fixture goes in `fixtures/` only when 2 or more spec files use it. A fixture that only one spec uses stays in that spec.## Documentation* Block Editor Handbook end to end testing overview: https://developer.wordpress.org/block-editor/contributors/code/testing-overview/#end-to-end-testing* Gutenberg e2e-test-utils-playwright package API docs: https://github.com/WordPress/gutenberg/tree/trunk/packages/e2e-test-utils-playwright* Playwright API docs: https://playwright.dev/docs (the version we are using is indicated in the `@wordpress/scripts` package: https://github.com/WordPress/gutenberg/blob/trunk/packages/scripts/package.json)

tests/e2e/fixtures/attachment.js

@@ -0,0 +1,70 @@
+/**
+ * External dependencies
+ */
+import { createReadStream } from 'node:fs';
+
+/**
+ * Uploads attachments, records the ones the browser uploads, then deletes
+ * them in teardown.
+ *
+ * @param {Object}                                                      fixtures              The fixtures this one depends on.
+ * @param {import('@wordpress/e2e-test-utils-playwright').RequestUtils} fixtures.requestUtils The request utils fixture.
+ * @param {Function}                                                    use                   Hands the fixture to the test.
+ * @return {Promise<void>}
+ */
+export const attachments = async ( { requestUtils }, use ) => {
+	const ids = new Set();
+
+	await use( {
+		async upload( absolutePath ) {
+			// Not requestUtils.uploadMedia(): it throws away the response on an
+			// error, but the server can save the attachment and then fail while
+			// it makes the sub-sizes. Core sends the id in a header before that.
+			if ( ! requestUtils.storageState?.nonce ) {
+				await requestUtils.setupRest();
+			}
+			const { rootURL, nonce } = requestUtils.storageState;
+			const response = await requestUtils.request.post(
+				`${ rootURL }wp/v2/media`,
+				{
+					headers: { 'X-WP-Nonce': nonce },
+					multipart: { file: createReadStream( absolutePath ) },
+					failOnStatusCode: false,
+				}
+			);
+			const id = Number(
+				response.headers()[ 'x-wp-upload-attachment-id' ]
+			);
+			if ( id > 0 ) {
+				ids.add( id );
+			}
+			if ( ! response.ok() ) {
+				throw new Error(
+					`Upload failed with HTTP ${ response.status() }: ${ await response.text() }`
+				);
+			}
+			return response.json();
+		},
+		track( id ) {
+			ids.add( id );
+		},
+	} );
+
+	const failures = [];
+	for ( const id of ids ) {
+		try {
+			await requestUtils.deleteMedia( id );
+		} catch ( error ) {
+			// A test can delete its own attachment through the UI.
+			if ( error.data?.status !== 404 ) {
+				failures.push( `${ id }: ${ error.message }` );
+			}
+		}
+	}
+
+	if ( failures.length ) {
+		throw new Error(
+			`attachments could not delete:\n${ failures.join( '\n' ) }`
+		);
+	}
+};

tests/e2e/fixtures/index.js

@@ -6,6 +6,7 @@ import { test as base, expect } from '@wordpress/e2e-test-utils-playwright';
 /**
  * Internal dependencies
  */
+import { attachments } from './attachment';
 import { postFactory } from './post';
 import { muPlugin, siteOptions, siteSettings } from './site';
 import { userFactory } from './user';
@@ -18,6 +19,7 @@ export const test = base.extend( {
 		await use( await context.newPage() );
 		await context.close();
 	},
+	attachments,
 	muPlugin,
 	postFactory,
 	siteOptions,

tests/e2e/playwright.config.js

@@ -30,7 +30,6 @@ const SERIAL_SPECS = [
 	'empty-trash-restore-trashed-posts.test.js',
 	'media-library-client-side-upload.test.js',
 	'media-modal-client-side-upload.test.js',
-	'media-new-client-side-upload.test.js',
 ];
 
 const config = defineConfig( {