Skip to content

docs: document exclude-optimization (2/2) #1729

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
marcossevilla wants to merge 1 commit into from
+112 −43
Open

docs: document exclude-optimization (2/2) #1729

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 27 additions & 1 deletion site/docs/commands/test.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,9 @@ very_good test [arguments]
-r, --recursive Run tests recursively for all nested packages.
--[no-]optimization Whether to apply optimizations for test performance.
Automatically disabled when --platform is specified.
Add the `skip_very_good_optimization` tag to specific test files to disable them individually.
Add the `skip_very_good_optimization` tag to specific test files to disable them individually, or use --exclude-optimization to exclude them by path.
(defaults to on)
--exclude-optimization=<glob> A glob which will be used to exclude matching test files from the optimized bundle (e.g. 'test/integration'). Excluded files still run, as their own test suites. Can be passed multiple times.
-j, --concurrency The number of concurrent test suites run.
(defaults to "4")
-t, --tags Run only tests associated with the specified tags.
Expand Down Expand Up @@ -109,6 +110,28 @@ By default, all tests run with optimizations enabled; use the `--no-optimization
import 'package:test/test.dart';
```

To exclude tests by path instead of tagging each file, pass one or more globs to `--exclude-optimization`, or list them under `optimization.exclude` in [`very_good.yaml`](../configuration.md):

```sh
very_good test --exclude-optimization test/integration
```

```yaml
# very_good.yaml
test:
optimization:
exclude:
- test/integration
```

Globs are matched against each test file's path relative to the package root, and they match everything nested underneath what they name, so `test/integration` excludes every test in that directory. See [`optimization`](../configuration.md#optimization) for the glob syntax details.

Excluded files are left out of the optimized bundle but still run, each as its own test suite. Because they run as their own suites, their file-level `@Tags` are honored again, so combining `--exclude-optimization test/integration` with `--exclude-tags integration` skips a suite annotated with `@Tags(['integration'])`.

Prefer narrow globs. Every excluded file becomes its own suite with its own VM startup, so excluding hundreds of files gives up most of what the optimization buys you.

Exclusions only matter while optimization is on. `--no-optimization`, `--platform`, `--update-goldens`, and targeting specific test files all disable optimization entirely, which makes the globs a no-op.

### Configuring defaults with `very_good.yaml`

To avoid repeating flags every time you run `very_good test` locally or on CI, you may create a `very_good.yaml` file at the root of your project. The `test` section accepts the same names as the CLI flags in snake_case (e.g. `--min-coverage` becomes `min_coverage`). Values from `very_good.yaml` are used as defaults; anything you pass on the command line takes precedence.
Expand All @@ -123,6 +146,9 @@ test:
dart_define:
- FLAVOR=development
file_reporter: json:reports/tests.json
optimization:
exclude:
- test/integration
```

With the file above, running `very_good test` behaves the same as running `very_good test --min-coverage 100 --exclude-coverage '**/*.g.dart' --report-on lib/ --dart-define=FLAVOR=development`. You can still override any of these values on the command line, for example `very_good test --min-coverage 90` to lower the coverage threshold for a single run.
Expand Down
127 changes: 85 additions & 42 deletions site/docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,9 @@ the command line without editing the file.

Each top-level key in `very_good.yaml` maps to a Very Good CLI command. Every
field mirrors a CLI flag using `snake_case` (e.g. `--min-coverage` becomes
`min_coverage`). Unrecognized keys cause the CLI to exit with a configuration
error.
`min_coverage`); `test.optimization` is the one field that also accepts a map,
grouping `--optimization` and `--exclude-optimization` together. Unrecognized
keys cause the CLI to exit with a configuration error.

```yaml
# very_good.yaml
Expand All @@ -58,7 +59,10 @@ Defaults for [`very_good test`](commands/test.md).
```yaml
test:
coverage: true
optimization: false
optimization:
enabled: true
exclude:
- test/integration
concurrency: 8
tags: my-tag
exclude_coverage: '**/*.g.dart'
Expand All @@ -82,27 +86,63 @@ test:
file_reporter: json:reports/tests.json
```

| Field | Type | Notes |
| ----------------------- | ------------------ | ----------------------------------------------------------------------------- |
| `coverage` | `bool` | Whether to collect coverage information. |
| `optimization` | `bool` | Whether to apply optimizations for test performance. |
| `concurrency` | `int` | Positive integer. The number of concurrent test suites run. |
| `tags` | `string` | Run only tests associated with the specified tags. |
| `exclude_coverage` | `string` | A glob that excludes matching files from coverage. |
| `exclude_tags` | `string` | Run only tests that do not have the specified tags. |
| `min_coverage` | `number` | Between `0` and `100`. Enforces a minimum coverage percentage. |
| `show_uncovered` | `bool` | Whether to show uncovered lines when coverage is below 100%. |
| `collect_coverage_from` | `imports` \| `all` | Whether to collect coverage from imported files only or all files. |
| `update_goldens` | `bool` | Whether `matchesGoldenFile()` calls should update the golden files. |
| `fail_fast` | `bool` | Whether to stop running tests after the first failure. |
| `dart_define` | `string` \| `list` | Additional `--dart-define` values. |
| `dart_define_from_file` | `string` \| `list` | Paths of `.json` or `.env` files with `--dart-define-from-file` values. |
| `platform` | `string` | The platform to run tests on (`chrome`, `vm`, `android`, `ios`). |
| `report_on` | `string` \| `list` | File paths to report coverage information to. |
| `run_skipped` | `bool` | Whether to run skipped tests instead of skipping them. |
| `flavor` | `string` | The flavor to build for testing. |
| `timeout` | `int` | Positive integer (seconds). Maximum time tests may run before being killed. |
| `file_reporter` | `string` | Additional file reporter as `<name>:<path>` (e.g. `json:reports/tests.json`). |
| Field | Type | Notes |
| ----------------------- | ------------------ | ----------------------------------------------------------------------------------------- |
| `coverage` | `bool` | Whether to collect coverage information. |
| `optimization` | `bool` \| `map` | Whether to apply optimizations for test performance. See [`optimization`](#optimization). |
| `concurrency` | `int` | Positive integer. The number of concurrent test suites run. |
| `tags` | `string` | Run only tests associated with the specified tags. |
| `exclude_coverage` | `string` | A glob that excludes matching files from coverage. |
| `exclude_tags` | `string` | Run only tests that do not have the specified tags. |
| `min_coverage` | `number` | Between `0` and `100`. Enforces a minimum coverage percentage. |
| `show_uncovered` | `bool` | Whether to show uncovered lines when coverage is below 100%. |
| `collect_coverage_from` | `imports` \| `all` | Whether to collect coverage from imported files only or all files. |
| `update_goldens` | `bool` | Whether `matchesGoldenFile()` calls should update the golden files. |
| `fail_fast` | `bool` | Whether to stop running tests after the first failure. |
| `dart_define` | `string` \| `list` | Additional `--dart-define` values. |
| `dart_define_from_file` | `string` \| `list` | Paths of `.json` or `.env` files with `--dart-define-from-file` values. |
| `platform` | `string` | The platform to run tests on (`chrome`, `vm`, `android`, `ios`). |
| `report_on` | `string` \| `list` | File paths to report coverage information to. |
| `run_skipped` | `bool` | Whether to run skipped tests instead of skipping them. |
| `flavor` | `string` | The flavor to build for testing. |
| `timeout` | `int` | Positive integer (seconds). Maximum time tests may run before being killed. |
| `file_reporter` | `string` | Additional file reporter as `<name>:<path>` (e.g. `json:reports/tests.json`). |

#### `optimization`

`optimization` accepts either a boolean or a map. A boolean is shorthand for
`enabled`, so `optimization: false` and `optimization: {enabled: false}` are
equivalent.

```yaml
test:
optimization:
enabled: true
exclude:
- test/integration
- test/**/serial_*_test.dart
```

| Field | Type | Notes |
| --------- | ------------------ | ------------------------------------------------------------------------------------------ |
| `enabled` | `bool` | Whether to apply optimizations for test performance. Mirrors `--optimization`. |
| `exclude` | `string` \| `list` | Globs of test files to keep out of the optimized bundle. Mirrors `--exclude-optimization`. |

Globs are matched against each test file's path relative to the package root
and match everything nested underneath what they name, so `test/integration`
excludes every test in that directory. `**` matches zero or more directories,
so `test/**/serial_*_test.dart` matches both `test/serial_a_test.dart` and
`test/nested/serial_a_test.dart`. Matching is case-sensitive on every
platform.

Excluded files still run, each as its own test suite, which also means their
file-level `@Tags` are honored again. See
[Skip optimization for specific tests](commands/test.md#skip-optimization-for-specific-tests).

With `--recursive`, the closest `very_good.yaml` applies to every package in the
run and each glob is matched against the package currently being tested, so
`test/integration` excludes that directory in every package. Per-package
exclusions are not expressible.

### `create`

Expand Down Expand Up @@ -135,7 +175,10 @@ Defaults for [`very_good dart test`](commands/test.md). The fields mirror
dart:
test:
coverage: true
optimization: false
optimization:
enabled: true
exclude:
- test/integration
concurrency: 8
tags: my-tag
exclude_coverage: '**/*.g.dart'
Expand All @@ -153,23 +196,23 @@ dart:
file_reporter: json:reports/tests.json
```

| Field | Type | Notes |
| ----------------------- | ------------------ | ----------------------------------------------------------------------------- |
| `coverage` | `bool` | Whether to collect coverage information. |
| `optimization` | `bool` | Whether to apply optimizations for test performance. |
| `concurrency` | `int` | Positive integer. The number of concurrent test suites run. |
| `tags` | `string` | Run only tests associated with the specified tags. |
| `exclude_coverage` | `string` | A glob that excludes matching files from coverage. |
| `exclude_tags` | `string` | Run only tests that do not have the specified tags. |
| `min_coverage` | `number` | Between `0` and `100`. Enforces a minimum coverage percentage. |
| `show_uncovered` | `bool` | Whether to show uncovered lines when coverage is below 100%. |
| `collect_coverage_from` | `imports` \| `all` | Whether to collect coverage from imported files only or all files. |
| `fail_fast` | `bool` | Whether to stop running tests after the first failure. |
| `platform` | `string` | The platform to run tests on (`chrome`, `vm`). |
| `report_on` | `string` \| `list` | File paths to report coverage information to. |
| `run_skipped` | `bool` | Whether to run skipped tests instead of skipping them. |
| `check_ignore` | `bool` | Whether to respect coverage ignore comments (e.g. `// coverage:ignore-line`). |
| `file_reporter` | `string` | Additional file reporter as `<name>:<path>` (e.g. `json:reports/tests.json`). |
| Field | Type | Notes |
| ----------------------- | ------------------ | ----------------------------------------------------------------------------------------- |
| `coverage` | `bool` | Whether to collect coverage information. |
| `optimization` | `bool` \| `map` | Whether to apply optimizations for test performance. See [`optimization`](#optimization). |
| `concurrency` | `int` | Positive integer. The number of concurrent test suites run. |
| `tags` | `string` | Run only tests associated with the specified tags. |
| `exclude_coverage` | `string` | A glob that excludes matching files from coverage. |
| `exclude_tags` | `string` | Run only tests that do not have the specified tags. |
| `min_coverage` | `number` | Between `0` and `100`. Enforces a minimum coverage percentage. |
| `show_uncovered` | `bool` | Whether to show uncovered lines when coverage is below 100%. |
| `collect_coverage_from` | `imports` \| `all` | Whether to collect coverage from imported files only or all files. |
| `fail_fast` | `bool` | Whether to stop running tests after the first failure. |
| `platform` | `string` | The platform to run tests on (`chrome`, `vm`). |
| `report_on` | `string` \| `list` | File paths to report coverage information to. |
| `run_skipped` | `bool` | Whether to run skipped tests instead of skipping them. |
| `check_ignore` | `bool` | Whether to respect coverage ignore comments (e.g. `// coverage:ignore-line`). |
| `file_reporter` | `string` | Additional file reporter as `<name>:<path>` (e.g. `json:reports/tests.json`). |

### `packages.get`

Expand Down