An open-source, extensible test suite that validates Posit Team deployments are installed correctly and functioning properly.
VIP uses BDD-style tests (pytest-bdd + Playwright) to verify Connect, Workbench, and Package Manager. Results are compiled into a Quarto report that can be published to a Connect server.
# Clone the repo
git clone https://github.com/posit-dev/vip.git
cd vip
# Set up with uv (recommended)
uv sync # install all dependencies
uv run playwright install chromium
# Or set up with pip
pip install -e ".[dev]"
playwright install chromium
# Configure
cp vip.toml.example vip.toml
# Edit vip.toml with your deployment details
# Set secrets via environment variables
export VIP_CONNECT_API_KEY="your-api-key"
export VIP_TEST_USERNAME="test-user"
export VIP_TEST_PASSWORD="test-password"
# Run all tests
uv run pytest # with uv
pytest # with pip
# Run tests for a single product
uv run pytest -m connect
uv run pytest -m workbench
uv run pytest -m package_manager
# Generate the Quarto report
uv run pytest --vip-report=report/results.json
cd report && quarto renderCopy vip.toml.example to vip.toml and edit it for your deployment. Each
product section can be disabled individually by setting enabled = false.
Secrets (API keys, passwords) should be set via environment variables rather than stored in the configuration file:
| Variable | Purpose |
|---|---|
VIP_CONNECT_API_KEY |
Connect admin API key |
VIP_WORKBENCH_API_KEY |
Workbench admin API key |
VIP_PM_TOKEN |
Package Manager token |
VIP_TEST_USERNAME |
Test user login name |
VIP_TEST_PASSWORD |
Test user login password |
VIP_CLUSTER_PROVIDER |
Cloud provider (aws or azure) |
VIP_CLUSTER_NAME |
EKS/AKS cluster name |
VIP_CLUSTER_REGION |
Cloud region |
VIP_AWS_PROFILE |
AWS profile name |
VIP_AWS_ROLE_ARN |
IAM role for cross-account access |
You can also point to the config file explicitly:
pytest --vip-config=/path/to/vip.tomlVIP tests that verify login flows and authenticated functionality need user credentials. How you provide them depends on the deployment's identity provider.
Set credentials via environment variables and run normally:
export VIP_TEST_USERNAME="test-user"
export VIP_TEST_PASSWORD="test-password"
uv run pytestFor PTD deployments with Keycloak, ptd verify handles this automatically —
it provisions a test user and passes credentials to VIP.
External identity providers require a real browser login. Use
--interactive-auth to launch a visible browser, authenticate through the
IdP, and then run the remaining tests headlessly with the captured session:
uv run pytest --interactive-authThis will:
- Open a Chromium window and navigate to the Connect login page
- Wait for you to complete the OIDC login flow (Okta, Azure AD, etc.)
- Navigate the Connect UI to mint a temporary API key (
_vip_interactive) - Capture the browser session state (cookies, localStorage)
- Close the browser and run all tests headlessly
- Delete the API key when the session finishes
Both Playwright browser tests (using the saved session state) and httpx API tests (using the minted key) work with a single interactive login.
Note:
--interactive-authis not available in container/CI environments. For automated runs against OIDC deployments, pre-provision credentials and set the environment variables above.
When using ptd verify, auth mode is selected automatically based on the
Site CR:
| Deployment auth | ptd verify mode | What happens |
|---|---|---|
| Keycloak | ptd verify <target> (K8s Job) |
Test user auto-provisioned |
| Okta / OIDC | ptd verify <target> --local --interactive-auth |
Browser popup for login |
| Any | ptd verify <target> --local |
Uses pre-existing credentials from Secret or env vars |
VIP can connect to Kubernetes clusters to run tests or manage credentials:
# Connect to a cluster (standalone, for debugging)
vip cluster connect my-targetThe cluster configuration comes from the [cluster] section in vip.toml or
can be overridden via CLI flags. See the Configuration section below for
details.
VIP can verify Posit Team deployments running in Kubernetes. The vip verify
command connects to a cluster, reads the Site custom resource, generates a
configuration, provisions credentials, and runs the test suite.
# Connect to cluster and run all tests as a K8s Job
vip verify ganso01-staging
# Use interactive auth for OIDC deployments
vip verify ganso01-staging --interactive-auth
# Run locally instead of K8s Job
vip verify ganso01-staging --local
# Just generate and print the vip.toml config
vip verify ganso01-staging --config-only
# Run specific test categories
vip verify ganso01-staging --categories prerequisites
# Clean up test credentials
vip verify cleanup ganso01-stagingTo use vip verify, add a [cluster] section to vip.toml:
[cluster]
provider = "aws" # "aws" or "azure"
name = "my-cluster-20260101" # EKS/AKS cluster name
region = "us-east-1" # Cloud region
profile = "ptd-staging" # AWS: profile name
role_arn = "arn:aws:iam::123:role/admin" # AWS: cross-account role (optional)AWS EKS:
- Requires
profile,region, and clustername - Optional
role_arnfor cross-account access - Uses boto3 to generate a kubeconfig with EKS token authentication
Azure AKS:
- Requires
subscription_id,resource_group, and clustername - Uses Azure SDK to retrieve kubeconfig with managed identity auth
Network access:
VIP assumes the Kubernetes API is reachable (via Tailscale, VPN, or direct
access). If the [cluster] section is omitted, VIP uses the current
KUBECONFIG.
How credentials are provisioned depends on the deployment's auth provider:
| Deployment auth | Command | What happens |
|---|---|---|
| Keycloak | vip verify <target> |
Test user auto-provisioned |
| Okta/OIDC | vip verify <target> --interactive-auth |
Browser login + token minting |
| Pre-existing | vip verify <target> |
Uses credentials from Secret or env vars |
Interactive auth requires the VIP CLI to be available in the Job container. For Keycloak deployments, a test user is created automatically with a cryptographically random password.
| Category | Marker | Description |
|---|---|---|
| Prerequisites | prerequisites |
Components installed, auth configured, admins onboarded |
| Package Manager | package_manager |
CRAN/PyPI mirrors, repos, private packages |
| Connect | connect |
Login, content deploy, data sources, packages, email, runtimes |
| Workbench | workbench |
Login, IDE launch, sessions, packages, data sources, runtimes |
| Cross-product | cross_product |
SSL, monitoring, system resources |
| Performance | performance |
Load times, package install speed, concurrency, resource usage |
| Security | security |
HTTPS enforcement, auth policy, secrets storage |
Run a specific category:
pytest -m connect
pytest -m "performance and connect"Tests can be pinned to product versions with the min_version marker:
@pytest.mark.min_version(product="connect", version="2024.05.0")
def test_new_api_feature():
...Tests that target a version newer than the deployment under test are automatically skipped.
Customers can add site-specific tests without modifying the VIP source tree.
- Create a directory with
.featureand.pyfiles following the same conventions as the built-in tests. - Point VIP at it via configuration or the CLI:
# vip.toml
[general]
extension_dirs = ["/opt/vip-custom-tests"]pytest --vip-extensions=/opt/vip-custom-testsThe custom tests directory is added to pytest's collection path at runtime.
See examples/custom_tests/ for a working example.
After running the tests, generate a report:
pytest --vip-report=report/results.json
cd report
quarto renderThe rendered report can be published to Connect:
quarto publish connect --server https://connect.example.com- Non-destructive - tests create, verify, and clean up their own content. They never modify or delete existing customer content.
- Diagnostic - tests are sequenced so failures localize problems. Prerequisites run first; product tests follow.
- Loosely coupled - the suite avoids tight coupling to product client libraries. API calls use plain HTTP where practical.
- Duplication over coupling - code duplication with product-internal test suites is acceptable if it keeps VIP independent and version-flexible.
# Install uv (if you don't have it)
curl -LsSf https://astral.sh/uv/install.sh | sh
# Install all dependencies (including dev tools like ruff)
uv sync
# Or with pip
pip install -e ".[dev]"VIP uses ruff for both linting and code formatting. The easiest way to run checks is with just:
just check # run both lint and format checks
just fix # auto-fix lint issues and reformat
# Or individually
just lint # ruff check
just format-check # ruff format --check
just lint-fix # ruff check --fix
just format # ruff formatWithout just, run ruff directly:
uv run ruff check src/ tests/ # lint
uv run ruff format --check src/ tests/ # format check
uv run ruff check --fix src/ tests/ # auto-fix lint
uv run ruff format src/ tests/ # reformatuv run mypy src/MIT - see LICENSE.