Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions develop-docs/backend/testing/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
title: Testing
sidebar_order: 70
---

Guides for testing backend code at Sentry.
56 changes: 56 additions & 0 deletions develop-docs/backend/testing/selective-testing.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
---
title: Selective Testing
---

Sentry's backend has a large test suite that takes a very long time to run. To mitigate long runtimes as part of a PR workflow, we employ a selective testing strategy.

## How it works

Selective testing uses code coverage data and direct import analysis to choose what tests to run based on what files changed in a PR.

- Code coverage shows which tests execute specific source code.
- Direct import analysis catches tests that import changed files, covering gaps in code coverage.

# Using selective tests

## Locally (Sentry employees only)

Both `sentry` and `getsentry` have a `make test-selective` make command that will leverage the same mechanisms to run only the tests you need locally. Try it out!

You'll need the GCloud CLI authenticated with your `@sentry.io` Google workspace account to access the coverage data.

## In CI

Selective testing will automatically run for any PR created in the `sentry`/`getsentry` repos.

`master` will always run the full testing suite.

### Opting-out in CI

Have a particularly sensitive PR you want to run all tests on? Add the “Trigger: Override Selective Testing” label and your PR will run all tests.

<img src="/selective-testing/selective-testing-override-label.png" alt="Selective testing override" style={{maxWidth: 300}} />

There are a few key exceptions that will always run a full testing suite:

- DB Migrations
- Changes to specific options and configuration areas of `sentry`
- Notable “core” files

## Limitations

### Code coverage holes

The main limitation of selective testing is code coverage gaps. Code coverage is not perfect and can miss some spots. One example when a method in a base class is used, the implementing class might not be marked as executed. Take this example:

```bash
# my-model.py
class MyModel

# Usage in another file
MyModel.dict()
```

While you might think `MyModel` would be marked as executed by the `MyModel.dict()` call, `dict()` is a function on `BaseModel` in pydantic, so code coverage sees this as `BaseModel` being used, not `MyModel`. This means if I changed `MyModel`, selective testing might miss usages of `MyModel.dict()` and therefore could miss breaking tests.

The workaround here is simple - also run tests that directly import any changed file.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion tsconfig.json
Original file line number Diff line number Diff line change
Expand Up @@ -81,4 +81,4 @@
"next-env.d.ts",
".next/types/**/*.ts"
]
}
}
Loading