Skip to content

chore: Refactor integration tests #2941

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

chore: Refactor integration tests #2941

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 54 additions & 33 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,11 @@ Before submitting an issue or a pull request, please search the repository for e

## Submitting a pull request

0. Fork and clone the repository.
0. Create a new branch: `git switch -c my-branch-name`.
0. Make your change, add tests, and make sure the tests still pass.
0. Push to your fork and submit a pull request.
0. Pat yourself on the back and wait for your pull request to be reviewed and merged.
1. Fork and clone the repository.
2. Create a new branch: `git switch -c my-branch-name`.
3. Make your change, add tests, and make sure the tests still pass.
4. Push to your fork and submit a pull request.
5. Pat yourself on the back and wait for your pull request to be reviewed and merged.

Here are a few things you can do that will increase the likelihood of your pull request being accepted:

Expand All @@ -38,21 +38,28 @@ Once you have the repository cloned, there's a couple of additional steps you'll
- You _must_ make sure that the "Template Repository" item in Settings is checked for this repo.
- If you haven't already, generate a Personal Access Token (PAT) for authenticating your test runs.
- Export the necessary configuration for authenticating your provider with GitHub

```sh
export GITHUB_TOKEN=<token of a user with an organization account>
export GITHUB_ORGANIZATION=<name of an organization>
export GITHUB_TEST_AUTH_MODE="organization"
export GITHUB_OWNER="<name of an organization>"
export GITHUB_USERNAME="<username of the user who created the token>"
export GITHUB_TOKEN="<token of a user with an organization account>"
```

- Build the project with `make build`
- Try an example test run from the default (`main`) branch, like `TF_LOG=DEBUG TF_ACC=1 go test -v ./... -run ^TestAccGithubRepositories`. All those tests should pass.

### Local Development Iteration

1. Write a test describing what you will fix. See [`github_label`](./github/resource_github_issue_label_test.go) for an example format.
1. Run your test and observe it fail. Enabling debug output allows for observing the underlying requests and responses made as well as viewing state (search `STATE:`) generated during the acceptance test run.

```sh
TF_LOG=DEBUG TF_ACC=1 go test -v ./... -run ^TestAccGithubIssueLabel
```

1. Align the resource's implementation to your test case and observe it pass:

```sh
TF_ACC=1 go test -v ./... -run ^TestAccGithubIssueLabel
```
Expand All @@ -67,7 +74,8 @@ Println debugging can easily be used to obtain information about how code change

If a full debugger is desired, VSCode may be used. In order to do so,

0. Create a launch.json file with this configuration:
1. Create a launch.json file with this configuration:

```json
{
"name": "Attach to Process",
Expand All @@ -77,18 +85,18 @@ If a full debugger is desired, VSCode may be used. In order to do so,
"processId": 0,
}
```
Setting a `processId` of 0 allows a dropdown to select the process of the provider.

0. Add a sleep call (e.g. `time.Sleep(10 * time.Second)`) in the [`func providerConfigure(p *schema.Provider) schema.ConfigureFunc`](https://github.com/integrations/terraform-provider-github/blob/cec7e175c50bb091feecdc96ba117067c35ee351/github/provider.go#L274C1-L274C64) before the immediate `return` call. This will allow time to connect the debugger while the provider is initializing, before any critical logic happens.
Setting a `processId` of 0 allows a dropdown to select the process of the provider.

0. Build the terraform provider with debug flags enabled and copy it to the appropriate bin folder with a command like `go build -gcflags="all=-N -l" -o ~/go/bin/`.
1. Add a sleep call (e.g. `time.Sleep(10 * time.Second)`) in the [`func providerConfigure(p *schema.Provider) schema.ConfigureFunc`](https://github.com/integrations/terraform-provider-github/blob/cec7e175c50bb091feecdc96ba117067c35ee351/github/provider.go#L274C1-L274C64) before the immediate `return` call. This will allow time to connect the debugger while the provider is initializing, before any critical logic happens.

0. Create or edit a `dev.tfrc` that points toward the newly-built binary, and export the `TF_CLI_CONFIG_FILE` variable to point to it. Further instructions on this process may be found in the [Building the provider](#using-a-local-version-of-the-provider) section.
2. Build the terraform provider with debug flags enabled and copy it to the appropriate bin folder with a command like `go build -gcflags="all=-N -l" -o ~/go/bin/`.

0. Run a terraform command (e.g. `terraform apply`). While the provider pauses on initialization, go to VSCode and click "Attach to Process". In the search box that appears, type `terraform-provi` and select the terraform provider process.
3. Create or edit a `dev.tfrc` that points toward the newly-built binary, and export the `TF_CLI_CONFIG_FILE` variable to point to it. Further instructions on this process may be found in the [Building the provider](#using-a-local-version-of-the-provider) section.

0. The debugger is now connected! During a typical terraform command, the plugin will be invoked multiple times. If the debugger disconnects and the plugin is invoked again later in the run, the developer will have to re-attach each time as the process ID changes.
4. Run a terraform command (e.g. `terraform apply`). While the provider pauses on initialization, go to VSCode and click "Attach to Process". In the search box that appears, type `terraform-provi` and select the terraform provider process.

5. The debugger is now connected! During a typical terraform command, the plugin will be invoked multiple times. If the debugger disconnects and the plugin is invoked again later in the run, the developer will have to re-attach each time as the process ID changes.

## Manual Testing

Expand All @@ -99,7 +107,7 @@ Manual testing should be performed on each PR opened in order to validate the pr
Build the provider and specify the output directory:

```sh
$ go build -gcflags="all=-N -l" -o ~/go/bin/
go build -gcflags="all=-N -l" -o ~/go/bin/
```

This enables verifying your locally built provider using examples available in the `examples/` directory.
Expand All @@ -121,7 +129,7 @@ provider_installation {
}
```

See https://www.terraform.io/docs/cli/config/config-file.html for more details.
See <https://www.terraform.io/docs/cli/config/config-file.html> for more details.

When running examples, you should spot the following warning to confirm you are using a local build:

Expand All @@ -137,37 +145,50 @@ The following provider development overrides are set in the CLI configuration:
Commonly required environment variables are listed below:

```sh
# enable debug logging
# Enable debug logging
export TF_LOG=DEBUG

# enable testing of organization scenarios instead of individual or anonymous
export GITHUB_ORGANIZATION=

# enable testing of individual scenarios instead of organization or anonymous
export GITHUB_OWNER=
# Enables acceptance tests
export TF_ACC="1"

# enable testing of enterprise appliances
# Configure the URL override for GHES.
export GITHUB_BASE_URL=

# enable testing of GitHub Paid features, these normally also require an organization e.g. repository push rulesets
export GITHUB_PAID_FEATURES=true

# leverage helper accounts for tests requiring them
# examples include:
# - https://github.com/github-terraform-test-user
# - https://github.com/terraformtesting
export GITHUB_TEST_OWNER=
export GITHUB_TEST_ORGANIZATION=
export GITHUB_TEST_USER_TOKEN=
```
# Configure acceptance testing mode; one of anonymous, individual, organization, team or enterprise. If not set will default to anonymous
export GITHUB_TEST_AUTH_MODE=

See [this project](https://github.com/terraformtesting/acceptance-tests) for more information on our old system for automated testing.
# Configure authentication for testing
export GITHUB_OWNER=
export GITHUB_USERNAME=
export GITHUB_TOKEN=

# Configure user level values
export GITHUB_TEST_USER_REPOSITORY=

# Configure for the org under test
export GITHUB_TEST_ORG_USER=
export GITHUB_TEST_ORG_SECRET_NAME=
export GITHUB_TEST_ORG_REPOSITORY=
export GITHUB_TEST_ORG_TEMPLATE_REPOSITORY=
export GITHUB_TEST_ORG_APP_INSTALLATION_ID=

# Configure external (non-org) users
export GITHUB_TEST_EXTERNAL_USER=
export GITHUB_TEST_EXTERNAL_USER_TOKEN=
export GITHUB_TEST_EXTERNAL_USER2=

# Configure test options
export GITHUB_TEST_ADVANCED_SECURITY=
```

There are also a small amount of unit tests in the provider. Due to the nature of the provider, such tests are currently only recommended for exercising functionality completely internal to the provider. These may be executed by running `make test`.

### GitHub Organization

If you do not have an organization already that you are comfortable running tests against, you will need to [create one](https://help.github.com/en/articles/creating-a-new-organization-from-scratch). The free "Team for Open Source" org type is fine for these tests. The name of the organization must then be exported in your environment as `GITHUB_ORGANIZATION`.
If you do not have an organization already that you are comfortable running tests against, you will need to [create one](https://help.github.com/en/articles/creating-a-new-organization-from-scratch). The free "Team for Open Source" org type is fine for these tests. The name of the organization must then be exported in your environment as `GITHUB_OWNER`.

Make sure that your organization has a `terraform-template-module` repository ([terraformtesting/terraform-template-module](https://github.com/terraformtesting/terraform-template-module) is an example you can clone) and that its "Template repository" item in Settings is checked.

Expand Down
10 changes: 6 additions & 4 deletions GNUmakefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@ TEST?=$$(go list ./... |grep -v 'vendor')
WEBSITE_REPO=github.com/hashicorp/terraform-website
PKG_NAME=github

export TESTARGS=-race -coverprofile=coverage.txt -covermode=atomic

default: build

tools:
Expand All @@ -16,18 +18,18 @@ fmt:
golangci-lint fmt ./...

fmtcheck:
@sh -c "'$(CURDIR)/scripts/gofmtcheck.sh'"
@sh -c "'$(CURDIR)/scripts/fmtcheck.sh'"

lint:
@echo "==> Checking source code against linters..."
golangci-lint run ./...
golangci-lint run --fix ./...

test:
CGO_ENABLED=0 go test ./...
# commenting this out for release tooling, please run testacc instead

testacc: fmtcheck
TF_ACC=1 CGO_ENABLED=0 go test $(TEST) -v $(TESTARGS) -timeout 120m
testacc:
TF_ACC=1 CGO_ENABLED=0 go test -run "^TestAcc*" $(TEST) -v $(TESTARGS) -timeout 120m -count=1

test-compile:
@if [ "$(TEST)" = "./..." ]; then \
Expand Down
Loading