diff --git a/develop-docs/backend/testing/index.mdx b/develop-docs/backend/testing/index.mdx new file mode 100644 index 00000000000000..9d1f633ea9c096 --- /dev/null +++ b/develop-docs/backend/testing/index.mdx @@ -0,0 +1,6 @@ +--- +title: Testing +sidebar_order: 70 +--- + +Guides for testing backend code at Sentry. diff --git a/develop-docs/backend/testing/selective-testing.mdx b/develop-docs/backend/testing/selective-testing.mdx new file mode 100644 index 00000000000000..8b867a76d61beb --- /dev/null +++ b/develop-docs/backend/testing/selective-testing.mdx @@ -0,0 +1,56 @@ +--- +title: Selective Testing +--- + +Sentry's backend has a large test suite that takes a very long time to run. To mitigate long runtimes as part of a PR workflow, we employ a selective testing strategy. + +## How it works + +Selective testing uses code coverage data and direct import analysis to choose what tests to run based on what files changed in a PR. + +- Code coverage shows which tests execute specific source code. +- Direct import analysis catches tests that import changed files, covering gaps in code coverage. + +# Using selective tests + +## Locally (Sentry employees only) + +Both `sentry` and `getsentry` have a `make test-selective` make command that will leverage the same mechanisms to run only the tests you need locally. Try it out! + +You'll need the GCloud CLI authenticated with your `@sentry.io` Google workspace account to access the coverage data. + +## In CI + +Selective testing will automatically run for any PR created in the `sentry`/`getsentry` repos. + +`master` will always run the full testing suite. + +### Opting-out in CI + +Have a particularly sensitive PR you want to run all tests on? Add the “Trigger: Override Selective Testing” label and your PR will run all tests. + +Selective testing override + +There are a few key exceptions that will always run a full testing suite: + +- DB Migrations +- Changes to specific options and configuration areas of `sentry` +- Notable “core” files + +## Limitations + +### Code coverage holes + +The main limitation of selective testing is code coverage gaps. Code coverage is not perfect and can miss some spots. One example when a method in a base class is used, the implementing class might not be marked as executed. Take this example: + +```bash +# my-model.py +class MyModel + +# Usage in another file +MyModel.dict() +``` + +While you might think `MyModel` would be marked as executed by the `MyModel.dict()` call, `dict()` is a function on `BaseModel` in pydantic, so code coverage sees this as `BaseModel` being used, not `MyModel`. This means if I changed `MyModel`, selective testing might miss usages of `MyModel.dict()` and therefore could miss breaking tests. + +The workaround here is simple - also run tests that directly import any changed file. \ No newline at end of file diff --git a/public/selective-testing/selective-testing-override-label.png b/public/selective-testing/selective-testing-override-label.png new file mode 100644 index 00000000000000..a40dd7c4235496 Binary files /dev/null and b/public/selective-testing/selective-testing-override-label.png differ diff --git a/tsconfig.json b/tsconfig.json index 29aa1222c67e15..dccdafc56a0a74 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -81,4 +81,4 @@ "next-env.d.ts", ".next/types/**/*.ts" ] -} +} \ No newline at end of file