From 2024c0993491ed7667c8080c1bd2ea7f531184c2 Mon Sep 17 00:00:00 2001 From: zivxx Date: Mon, 19 Oct 2026 14:56:12 +0300 Subject: [PATCH 1/4] feat(fleet-ops-demo): manage the schema as Terraform, with CI for a policy repository gitops/ is laid out like the repository Permit's GitOps reads: the schema in terraform/ (the same objects internal/schema writes through the API), a workflow that plans each change on its pull request and applies it on push (main to Development, release/* to Production), and ci/permit_snapshot.py, which saves the role assignments and tuples a plan deletes and puts them back when the change is reverted. The workflows are templates: GitHub only runs the ones at the repository root. The example's own CI checks the Terraform's format and validates it without credentials. Co-Authored-By: Claude Opus 5.5 --- .github/dependabot.yml | 6 + .github/workflows/fleet-ops-demo.yaml | 27 +++ fleet-ops-demo/.pre-commit-config.yaml | 6 + fleet-ops-demo/README.md | 6 + .../.github/workflows/policy-tests.yaml | 36 ++++ .../gitops/.github/workflows/terraform.yaml | 157 ++++++++++++++++++ fleet-ops-demo/gitops/README.md | 67 ++++++++ fleet-ops-demo/gitops/ci/permit_snapshot.py | 131 +++++++++++++++ fleet-ops-demo/gitops/terraform/.gitignore | 7 + .../gitops/terraform/.terraform.lock.hcl | 24 +++ .../gitops/terraform/attribute_groups.tf | 43 +++++ fleet-ops-demo/gitops/terraform/cloud.tf | 6 + .../gitops/terraform/derivations.tf | 34 ++++ fleet-ops-demo/gitops/terraform/main.tf | 31 ++++ fleet-ops-demo/gitops/terraform/resources.tf | 112 +++++++++++++ fleet-ops-demo/gitops/terraform/roles.tf | 107 ++++++++++++ 16 files changed, 800 insertions(+) create mode 100644 fleet-ops-demo/gitops/.github/workflows/policy-tests.yaml create mode 100644 fleet-ops-demo/gitops/.github/workflows/terraform.yaml create mode 100644 fleet-ops-demo/gitops/README.md create mode 100644 fleet-ops-demo/gitops/ci/permit_snapshot.py create mode 100644 fleet-ops-demo/gitops/terraform/.gitignore create mode 100644 fleet-ops-demo/gitops/terraform/.terraform.lock.hcl create mode 100644 fleet-ops-demo/gitops/terraform/attribute_groups.tf create mode 100644 fleet-ops-demo/gitops/terraform/cloud.tf create mode 100644 fleet-ops-demo/gitops/terraform/derivations.tf create mode 100644 fleet-ops-demo/gitops/terraform/main.tf create mode 100644 fleet-ops-demo/gitops/terraform/resources.tf create mode 100644 fleet-ops-demo/gitops/terraform/roles.tf diff --git a/.github/dependabot.yml b/.github/dependabot.yml index fe71ced..906d0ff 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -47,3 +47,9 @@ updates: update-types: [minor, patch] patterns: - '*' + - package-ecosystem: terraform + directory: /fleet-ops-demo/gitops/terraform + schedule: + interval: weekly + cooldown: + default-days: 7 diff --git a/.github/workflows/fleet-ops-demo.yaml b/.github/workflows/fleet-ops-demo.yaml index bde577a..2baa397 100644 --- a/.github/workflows/fleet-ops-demo.yaml +++ b/.github/workflows/fleet-ops-demo.yaml @@ -98,3 +98,30 @@ jobs: run: | "$RUNNER_TEMP/opa" check --v0-compatible policy/ "$RUNNER_TEMP/opa" test --v0-compatible policy/ + + terraform: + name: Terraform format and validate + runs-on: ubuntu-24.04 + timeout-minutes: 10 + defaults: + run: + working-directory: fleet-ops-demo/gitops/terraform + steps: + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Set up Terraform + uses: hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1 + with: + terraform_version: 1.13.5 + terraform_wrapper: false + - name: Format + run: terraform fmt -check -diff -recursive + # No credentials here: -backend=false skips HCP Terraform, and validate never calls Permit. + - name: Init + run: terraform init -input=false -backend=false + - name: Validate + run: terraform validate + - name: Compile the snapshot script + run: python3 -I -m py_compile ../ci/permit_snapshot.py diff --git a/fleet-ops-demo/.pre-commit-config.yaml b/fleet-ops-demo/.pre-commit-config.yaml index c622582..c061670 100644 --- a/fleet-ops-demo/.pre-commit-config.yaml +++ b/fleet-ops-demo/.pre-commit-config.yaml @@ -27,3 +27,9 @@ repos: language: system pass_filenames: false files: ^policy/.*\.rego$ + - id: terraform-fmt + name: terraform fmt + entry: terraform fmt -check -diff -recursive gitops/terraform + language: system + pass_filenames: false + files: ^gitops/terraform/.*\.tf$ diff --git a/fleet-ops-demo/README.md b/fleet-ops-demo/README.md index e13fda0..10b50f2 100644 --- a/fleet-ops-demo/README.md +++ b/fleet-ops-demo/README.md @@ -36,6 +36,11 @@ conditions with the values evaluated, the tenant-isolation verdict and the custo repository (`policy/custom.rego`); every allow must also satisfy them. They are configuration, not out-of-the-box behavior. +**Policy as code.** [`gitops/`](gitops/) holds the same schema as Terraform, with the CI that plans each +change on its pull request and applies it on push: `main` to Development, release branches to Production. +A destructive change and its `git revert` are covered too, data included. See +[gitops/README.md](gitops/README.md). + ## Run it You need Docker, Go 1.26 or later, a Permit project with an empty environment, and, for the custom @@ -198,6 +203,7 @@ filtered by user. The decision record carries `debug.acting_org` (`same_org` or | `internal/checks`, `cmd/verify` | The 53 pre-flight checks behind `make verify` | | `cmd/apply-policy` | Installs the custom rules in the GitOps branch, restarts the PDP and waits for the new version | | `internal/permitapi` | A timeout, retries and paging around every API call | +| `gitops/` | The schema as Terraform, and the CI for a policy repository: plan on pull requests, apply on push, restore after a revert | ### Where the SDK is used diff --git a/fleet-ops-demo/gitops/.github/workflows/policy-tests.yaml b/fleet-ops-demo/gitops/.github/workflows/policy-tests.yaml new file mode 100644 index 0000000..b9e05c7 --- /dev/null +++ b/fleet-ops-demo/gitops/.github/workflows/policy-tests.yaml @@ -0,0 +1,36 @@ +name: Policy tests + +# Every pull request, and every push to main or a release branch, runs the custom rules' unit tests +# before anything reaches an environment. +on: + pull_request: + push: + branches: [main, 'release/**'] + +permissions: + contents: read + +jobs: + rego: + name: Custom Rego checks and tests + runs-on: ubuntu-24.04 + timeout-minutes: 10 + steps: + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Install verified opa binary + run: | + curl --fail --silent --show-error --location --max-time 60 \ + https://github.com/open-policy-agent/opa/releases/download/v1.21.1/opa_linux_amd64_static \ + --output "$RUNNER_TEMP/opa" + printf '%s %s\n' \ + '668506eb17a2eaa1fce6cc0d1f42ef85125d4ac5bda5fc74d1152d0c77145031' \ + "$RUNNER_TEMP/opa" | sha256sum --check + chmod +x "$RUNNER_TEMP/opa" + # Permit's generated policy and the custom rules use Rego v0 syntax. + - name: Check and test the custom rules + run: | + "$RUNNER_TEMP/opa" check --v0-compatible custom/ + "$RUNNER_TEMP/opa" test --v0-compatible --verbose custom/ diff --git a/fleet-ops-demo/gitops/.github/workflows/terraform.yaml b/fleet-ops-demo/gitops/.github/workflows/terraform.yaml new file mode 100644 index 0000000..d64b8e3 --- /dev/null +++ b/fleet-ops-demo/gitops/.github/workflows/terraform.yaml @@ -0,0 +1,157 @@ +name: Terraform + +# The schema is Terraform in terraform/. A pull request shows its plan for the environment it targets. +# A push applies it: main to Development, release/* to Production. Before every apply, CI saves what the +# plan deletes (role assignments, relationships), because Permit can't undo a delete; when a push is a +# `git revert`, CI puts back what the reverted commit's run saved (ci/permit_snapshot.py). +# +# Needs, in the repository's settings: +# secrets PERMIT_API_KEY_DEV, PERMIT_API_KEY_PROD environment-level API keys +# TF_API_TOKEN an HCP Terraform team or user token +# variables PERMIT_PROJECT the Permit project key +# TF_CLOUD_ORGANIZATION the HCP Terraform organization, with workspaces +# fleet-ops-dev and fleet-ops-prod (local execution) +# PERMIT_API_URL (optional) defaults to https://api.permit.io +on: + pull_request: + branches: [main, 'release/**'] + paths: [terraform/**, ci/**, .github/workflows/terraform.yaml] + push: + branches: [main, 'release/**'] + paths: [terraform/**, ci/**, .github/workflows/terraform.yaml] + +permissions: + contents: read + +# One run at a time per environment; never cancel an apply half way. +concurrency: + group: terraform-${{ github.event_name }}-${{ github.base_ref || github.ref_name }} + cancel-in-progress: false + +env: + TF_IN_AUTOMATION: "1" + TF_CLOUD_ORGANIZATION: ${{ vars.TF_CLOUD_ORGANIZATION }} + TF_TOKEN_app_terraform_io: ${{ secrets.TF_API_TOKEN }} + PERMIT_API_URL: ${{ vars.PERMIT_API_URL || 'https://api.permit.io' }} + TF_VAR_permit_api_url: ${{ vars.PERMIT_API_URL || 'https://api.permit.io' }} + PERMIT_PROJECT: ${{ vars.PERMIT_PROJECT }} + +jobs: + plan: + name: Terraform plan + if: github.event_name == 'pull_request' + runs-on: ubuntu-24.04 + timeout-minutes: 10 + permissions: + contents: read + pull-requests: write + env: + TF_WORKSPACE: ${{ github.base_ref == 'main' && 'fleet-ops-dev' || 'fleet-ops-prod' }} + TF_VAR_permit_api_key: ${{ github.base_ref == 'main' && secrets.PERMIT_API_KEY_DEV || secrets.PERMIT_API_KEY_PROD }} + defaults: + run: + working-directory: terraform + steps: + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Set up Terraform + uses: hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1 + with: + terraform_version: 1.13.5 + terraform_wrapper: false + - name: Format + run: terraform fmt -check -diff -recursive + - name: Init + run: terraform init -input=false + - name: Validate + run: terraform validate + - name: Plan + run: terraform plan -input=false -no-color -out=tfplan | tee plan.txt + - name: Post the plan on the pull request + env: + GH_TOKEN: ${{ github.token }} + PR: ${{ github.event.pull_request.number }} + TARGET: ${{ github.base_ref == 'main' && 'Development' || 'Production' }} + run: | + { + echo "#### Terraform plan for **${TARGET}**" + echo + grep -E '^Plan:|^No changes' plan.txt || true + echo + echo '```diff' + sed -n '/Terraform will perform the following actions/,/^Plan:/p' plan.txt \ + | sed -e 's/^\(\s*\)~/\1!/' | grep -v '^Plan:' || true + echo '```' + } > comment.md + cat comment.md >> "$GITHUB_STEP_SUMMARY" + gh pr comment "$PR" --repo "$GITHUB_REPOSITORY" --body-file comment.md + + apply: + name: Terraform apply + if: github.event_name == 'push' + runs-on: ubuntu-24.04 + timeout-minutes: 15 + permissions: + contents: read + actions: read + env: + TF_WORKSPACE: ${{ github.ref_name == 'main' && 'fleet-ops-dev' || 'fleet-ops-prod' }} + PERMIT_ENV: ${{ github.ref_name == 'main' && 'dev' || 'production' }} + PERMIT_API_KEY: ${{ github.ref_name == 'main' && secrets.PERMIT_API_KEY_DEV || secrets.PERMIT_API_KEY_PROD }} + TF_VAR_permit_api_key: ${{ github.ref_name == 'main' && secrets.PERMIT_API_KEY_DEV || secrets.PERMIT_API_KEY_PROD }} + defaults: + run: + working-directory: terraform + steps: + - name: Checkout code + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + - name: Set up Terraform + uses: hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1 + with: + terraform_version: 1.13.5 + terraform_wrapper: false + - name: Init + run: terraform init -input=false + - name: Plan + run: | + terraform plan -input=false -no-color -out=tfplan | tee plan.txt + terraform show -json tfplan > plan.json + - name: Save what this change deletes + run: python3 ../ci/permit_snapshot.py save plan.json "../snapshot-${PERMIT_ENV}-${GITHUB_SHA}.json" + - name: Keep the snapshot with this run + if: hashFiles(format('snapshot-{0}-{1}.json', env.PERMIT_ENV, github.sha)) != '' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: snapshot-${{ env.PERMIT_ENV }}-${{ github.sha }} + path: snapshot-${{ env.PERMIT_ENV }}-${{ github.sha }}.json + retention-days: 30 + - name: Apply + run: terraform apply -input=false -no-color tfplan + - name: Restore what a reverted commit deleted + env: + GH_TOKEN: ${{ github.token }} + run: | + # Every commit in this push that says "This reverts commit ": restore that commit's snapshot. + python3 - "$GITHUB_EVENT_PATH" > ../reverted.txt <<'PY' + import json, re, sys + event = json.load(open(sys.argv[1])) + for c in event.get("commits", []): + for sha in re.findall(r"This reverts commit ([0-9a-f]{40})", c["message"]): + print(sha) + PY + while read -r sha; do + name="snapshot-${PERMIT_ENV}-${sha}" + id=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${name}" --jq '.artifacts[0].id // empty') + if [ -z "$id" ]; then + echo "Reverts ${sha:0:7}: that change deleted nothing, so there's nothing to restore." + continue + fi + gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${id}/zip" > "../${name}.zip" + unzip -o -q "../${name}.zip" -d .. + echo "Reverts ${sha:0:7}: restoring what it deleted." + python3 ../ci/permit_snapshot.py restore "../${name}.json" + done < ../reverted.txt diff --git a/fleet-ops-demo/gitops/README.md b/fleet-ops-demo/gitops/README.md new file mode 100644 index 0000000..8ac241c --- /dev/null +++ b/fleet-ops-demo/gitops/README.md @@ -0,0 +1,67 @@ +# Policy as code: the Fleet Ops schema in Terraform + +This folder is laid out like the Git repository that Permit's +[GitOps](https://docs.permit.io/integrations/gitops/github) reads. Copy its contents to the root of that +repository to manage the whole policy in Git: + +- **the schema** (resource types, roles and what they may do, relations, derivations, attribute rules) as + Terraform, with Permit's [Terraform provider](https://registry.terraform.io/providers/permitio/permit-io); +- **the custom Rego rules**, which `make apply-policy` installs as `custom/root.rego`, tested on every pull + request; +- **the CI** that plans every change on its pull request and applies it when it's pushed. + +Each Permit environment follows one branch: **Development** follows `main` and **Production** follows a +release branch (`release/*`). Moving a change between environments is ordinary Git: merge, cherry-pick, +revert. + +| Path | What it is | +|---|---| +| `terraform/` | The schema: the same objects `make setup` writes through the API (`internal/schema/schema.go`) | +| `terraform/cloud.tf` | Keeps the state in HCP Terraform, one workspace per environment. Delete it to keep the state in a local file | +| `.github/workflows/terraform.yaml` | On a pull request: format check, validate, plan, and the plan posted as a comment. On a push: save what the plan deletes, apply, and restore after a revert | +| `.github/workflows/policy-tests.yaml` | `opa check` and `opa test` on the custom rules, on every pull request and push | +| `ci/permit_snapshot.py` | Saves the data a plan deletes, and puts it back after a revert (below) | + +## What a destructive change costs, and the snapshot step + +Git and Terraform version the schema: every change is a commit, and rolling back is `git revert` followed +by an apply. They don't version Permit's data. Deleting a role in Permit also deletes every assignment of +it, and deleting a relation deletes its relationship tuples. Reverting the commit brings the role back, +empty: nobody holds it anymore. + +`ci/permit_snapshot.py` covers that gap. It is part of this example, not a feature of Permit or of +Terraform: + +1. Before each apply, `save` reads the plan (`terraform show -json`), and for every role or relation the + plan destroys, lists its role assignments or tuples through Permit's API and keeps them with the run. +2. When a push contains a commit that says "This reverts commit X", `restore` fetches the file X's run + kept and recreates exactly those assignments and tuples, skipping people and instances that no longer + exist. + +If you'd rather a role were never deleted by accident, Terraform's `lifecycle { prevent_destroy = true }` +on it makes any plan that would delete it fail. + +## Setting it up + +1. **A Permit project with two environments**, Development and Production, both empty, and a + repository connected through GitOps. Point each environment at its branch with the environment's + `custom_branch_name` setting (environments API): `main` for Development, your release branch for + Production. +2. **HCP Terraform**: an organization with two workspaces, `fleet-ops-dev` and `fleet-ops-prod`, set to + local execution (the runner is GitHub Actions; HCP Terraform keeps the state and its history). +3. **Repository settings**, listed at the top of `terraform.yaml`: the two environments' API keys and an + HCP Terraform token as secrets, the Permit project key and the HCP Terraform organization as + variables. +4. **The custom rules.** `make apply-policy` installs `policy/custom.rego` as `custom/root.rego`. Copy + `policy/custom_test.rego` to `custom/root_test.rego` so `policy-tests.yaml` runs it; the PDP loads only + the files `.manifest` lists, so the tests never reach it. +5. **The first apply** creates the schema. Then load the seed data with `make reset`, not `make setup`: + `reset` changes only data, while `setup` would also write the schema Terraform now manages. + +To try the schema from your machine first: delete `cloud.tf`, then + +```bash +cd terraform +terraform init +TF_VAR_permit_api_key= terraform plan +``` diff --git a/fleet-ops-demo/gitops/ci/permit_snapshot.py b/fleet-ops-demo/gitops/ci/permit_snapshot.py new file mode 100644 index 0000000..353df51 --- /dev/null +++ b/fleet-ops-demo/gitops/ci/permit_snapshot.py @@ -0,0 +1,131 @@ +"""Save what a Terraform change is about to delete in Permit, and put it back after a rollback. + +Permit has no undo: deleting a role deletes every assignment of it, and deleting a relation deletes its +relationship tuples. `git revert` brings the schema back, not that data. So before each apply, CI runs + + permit_snapshot.py save + +which reads the plan (`terraform show -json`) and saves the role assignments of every role the plan +destroys and the tuples of every relation it destroys. After a revert is applied, CI runs + + permit_snapshot.py restore + +which puts back exactly what that snapshot saved, skipping people or instances that no longer exist and +anything already there. Reads PERMIT_API_URL, PERMIT_PROJECT, PERMIT_ENV and PERMIT_API_KEY.""" +import json +import os +import sys +import time +import urllib.error +import urllib.parse +import urllib.request + +API = (os.environ.get("PERMIT_API_URL") or "https://api.permit.io").rstrip("/") +PROJECT, ENV, KEY = os.environ["PERMIT_PROJECT"], os.environ["PERMIT_ENV"], os.environ["PERMIT_API_KEY"] +FACTS = f"{API}/v2/facts/{PROJECT}/{ENV}" + + +def call(method, url, body=None, missing_ok=False): + data = json.dumps(body).encode() if body is not None else None + for attempt in range(6): + req = urllib.request.Request(url, data=data, method=method, headers={ + "Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}) + try: + with urllib.request.urlopen(req, timeout=60) as res: + raw = res.read() + return json.loads(raw) if raw else None + except urllib.error.HTTPError as e: + if e.code == 404 and missing_ok: + return None + if e.code == 429 and attempt < 5: + time.sleep(2 ** attempt) + continue + raise SystemExit(f"{method} {url}: HTTP {e.code} {e.read().decode()[:300]}") + raise SystemExit(f"{method} {url}: gave up after retries") + + +def list_all(path, **params): + rows, page = [], 1 + while True: + query = urllib.parse.urlencode({**params, "page": page, "per_page": 100}) + batch = call("GET", f"{FACTS}/{path}?{query}") or [] + rows += batch + if len(batch) < 100: + return rows + page += 1 + + +def destroyed(plan, kind): + """The `before` state of every resource of this type that the plan deletes (or replaces).""" + return [c["change"]["before"] for c in plan.get("resource_changes", []) + if c["type"] == kind and "delete" in c["change"]["actions"]] + + +def save(plan_path, out_path): + plan = json.load(open(plan_path)) + assignments, tuples = [], [] + for role in destroyed(plan, "permitio_role"): + resource = role.get("resource") + for ra in list_all("role_assignments", role=role["key"]): + instance = ra.get("resource_instance") or "" + if (resource and instance.startswith(f"{resource}:")) or (not resource and not instance): + assignments.append({k: ra.get(k) for k in ("user", "role", "tenant", "resource_instance") if ra.get(k)}) + for rel in destroyed(plan, "permitio_relation"): + for t in list_all("relationship_tuples", relation=rel["key"], object_type=rel["object_resource"]): + if t["subject"].startswith(f"{rel['subject_resource']}:"): + tuples.append({k: t[k] for k in ("subject", "relation", "object", "tenant")}) + if not assignments and not tuples: + print("Nothing to save: the plan deletes no role assignments or relationships.") + return + snapshot = {"environment": ENV, "commit": os.environ.get("GITHUB_SHA", ""), "role_assignments": assignments, + "tuples": tuples} + json.dump(snapshot, open(out_path, "w"), indent=1) + print(f"Saved what this change deletes in {ENV}: {len(assignments)} role assignments, {len(tuples)} relationships.") + for ra in assignments: + print(f" {ra['user']} is {ra['role']} on {ra.get('resource_instance') or ra['tenant']}") + + +def restore(path): + snap = json.load(open(path)) + users, instances, restored, skipped = {}, {}, [], 0 + + def exists(cache, url): + if url not in cache: + cache[url] = call("GET", url, missing_ok=True) is not None + return cache[url] + + have = {(a["user"], a["role"], a["tenant"], a.get("resource_instance") or "") + for role in {ra["role"] for ra in snap["role_assignments"]} for a in list_all("role_assignments", role=role)} + todo = [] + for ra in snap["role_assignments"]: + key = (ra["user"], ra["role"], ra["tenant"], ra.get("resource_instance") or "") + instance = ra.get("resource_instance") + if key in have: + continue + if not exists(users, f"{FACTS}/users/{urllib.parse.quote(ra['user'])}") or ( + instance and not exists(instances, f"{FACTS}/resource_instances/{urllib.parse.quote(instance)}")): + skipped += 1 + continue + todo.append(ra) + if todo: + call("POST", f"{FACTS}/role_assignments/bulk", todo) + restored += [f"{ra['user']} is {ra['role']} on {ra.get('resource_instance') or ra['tenant']}" for ra in todo] + tuples = [t for t in snap.get("tuples", []) + if exists(instances, f"{FACTS}/resource_instances/{urllib.parse.quote(t['subject'])}") + and exists(instances, f"{FACTS}/resource_instances/{urllib.parse.quote(t['object'])}")] + skipped += len(snap.get("tuples", [])) - len(tuples) + if tuples: + call("POST", f"{FACTS}/relationship_tuples/bulk", {"operations": tuples}) + print(f"Restored in {ENV}: {len(todo)} role assignments, {len(tuples)} relationships; " + f"{skipped} skipped because their person or instance no longer exists.") + for line in restored: + print(f" {line}") + + +if __name__ == "__main__": + if sys.argv[1:2] == ["save"] and len(sys.argv) == 4: + save(sys.argv[2], sys.argv[3]) + elif sys.argv[1:2] == ["restore"] and len(sys.argv) == 3: + restore(sys.argv[2]) + else: + raise SystemExit(__doc__) diff --git a/fleet-ops-demo/gitops/terraform/.gitignore b/fleet-ops-demo/gitops/terraform/.gitignore new file mode 100644 index 0000000..dca0f92 --- /dev/null +++ b/fleet-ops-demo/gitops/terraform/.gitignore @@ -0,0 +1,7 @@ +.terraform/ +*.tfstate +*.tfstate.* +tfplan +plan.txt +plan.json +comment.md diff --git a/fleet-ops-demo/gitops/terraform/.terraform.lock.hcl b/fleet-ops-demo/gitops/terraform/.terraform.lock.hcl new file mode 100644 index 0000000..20c6f77 --- /dev/null +++ b/fleet-ops-demo/gitops/terraform/.terraform.lock.hcl @@ -0,0 +1,24 @@ +# This file is maintained automatically by "terraform init". +# Manual edits may be lost in future updates. + +provider "registry.terraform.io/permitio/permit-io" { + version = "0.0.26" + constraints = "0.0.26" + hashes = [ + "h1:VJ6xTPLST/DIRVW3+LoMXRPyQv4Hl5KHdVictA3v//A=", + "zh:14ba51a201f1c0d9a194e544f38303d9f32518652b9606c72726bdc51a98aa28", + "zh:2177a660a7baede51103da7ab6550ef8e7f2629c4eb91b5876404511ed78f695", + "zh:2c85546652abf3aa249f8bd60e6f856a77f19e41eff20ecf1846fa0b90424472", + "zh:482a09e8cd8f57c41b7910e967859c7672d4e4234f8ac7c4ea5df18e5886e533", + "zh:6665f5cf2b50454bc86b1cd8b12ab63366450c5329f107ec6d13e2bbb79d2c5f", + "zh:7703de2c27c6eb5b33fb6b14e1c0ddacdbcf588028bd0d3a6ec46b8ee22b67d2", + "zh:7acd2eb6a326c5dbb54004a05abf52e5502d1f912226378c7f23ebaa5ec04ecb", + "zh:8756e10f84787547259dcf625c26a4d807770a2c565cbd094d1ce34e2686d960", + "zh:890df766e9b839623b1f0437355032a3c006226a6c200cd911e15ee1a9014e9f", + "zh:a85461c873328c82eccf3f73560e4cd71eb5851d6fc7b3ccfd16d863742eaca9", + "zh:b12c834ac3474d2430bce173b05680fc4e90e7fcc8301b7ee600f5238a0fee73", + "zh:c3df54767718ea0ae4a728d91bd2c7760b2d036d45287bbec3afc74de363c822", + "zh:cae117151386ca374e2ee412f6f1aeae1aea804f5f7b3d4b48d7732a4aa6681b", + "zh:f710a87d9b055d8559424ae69ebc2ccb71e08acc85a7d3dcdc9e97042ded8a60", + ] +} diff --git a/fleet-ops-demo/gitops/terraform/attribute_groups.tf b/fleet-ops-demo/gitops/terraform/attribute_groups.tf new file mode 100644 index 0000000..41a2280 --- /dev/null +++ b/fleet-ops-demo/gitops/terraform/attribute_groups.tf @@ -0,0 +1,43 @@ +# Groups of trucks by attribute (scenario 11), and the org-level roles they're granted to. + +resource "permitio_resource_set" "south_assets" { + key = "south_assets" + name = "Assets where region = South" + resource = permitio_resource.asset.key + conditions = jsonencode({ + allOf = [{ "resource.region" = { equals = "South" } }] + }) +} + +resource "permitio_resource_set" "acct1_assets" { + key = "acct1_assets" + name = "Assets where served_account = Acct1" + resource = permitio_resource.asset.key + conditions = jsonencode({ + allOf = [{ "resource.served_account" = { equals = "Acct1" } }] + }) +} + +# Permit makes a user set for each org-level role (`__autogen_`): everyone holding it in the acting +# organization. + +resource "permitio_condition_set_rule" "operator_views_south" { + user_set = "__autogen_operator" + permission = "asset:view" + resource_set = permitio_resource_set.south_assets.key + depends_on = [permitio_role.operator] +} + +resource "permitio_condition_set_rule" "operator_operates_south" { + user_set = "__autogen_operator" + permission = "asset:operate" + resource_set = permitio_resource_set.south_assets.key + depends_on = [permitio_role.operator] +} + +resource "permitio_condition_set_rule" "viewer_views_acct1" { + user_set = "__autogen_viewer" + permission = "asset:view" + resource_set = permitio_resource_set.acct1_assets.key + depends_on = [permitio_role.viewer] +} diff --git a/fleet-ops-demo/gitops/terraform/cloud.tf b/fleet-ops-demo/gitops/terraform/cloud.tf new file mode 100644 index 0000000..fce3fcd --- /dev/null +++ b/fleet-ops-demo/gitops/terraform/cloud.tf @@ -0,0 +1,6 @@ +# State in HCP Terraform, one workspace per environment. The organization and the workspace come from +# TF_CLOUD_ORGANIZATION and TF_WORKSPACE, which the workflow sets. Delete this file to keep the state in a +# local file instead, for example to try the schema against one environment from your machine. +terraform { + cloud {} +} diff --git a/fleet-ops-demo/gitops/terraform/derivations.tf b/fleet-ops-demo/gitops/terraform/derivations.tf new file mode 100644 index 0000000..72d7c31 --- /dev/null +++ b/fleet-ops-demo/gitops/terraform/derivations.tf @@ -0,0 +1,34 @@ +# How roles travel down the `parent` links: one role assignment high in the tree reaches everything below it. + +locals { + derivations = { + # An org admin is also org admin of every sub-organization, and region manager of its regions. + org_admin_suborg = { resource = "org", to_role = "org_admin", on_resource = "org", role = "org_admin" } + region_manager = { resource = "region", to_role = "region_manager", on_resource = "org", role = "org_admin" } + # A region manager is site manager of the region's sites. + site_manager = { resource = "site", to_role = "site_manager", on_resource = "region", role = "region_manager" } + # Each site role carries down to the site's trucks. + asset_site_manager = { resource = "asset", to_role = "site_manager", on_resource = "site", role = "site_manager" } + asset_operator = { resource = "asset", to_role = "operator", on_resource = "site", role = "operator" } + asset_technician = { resource = "asset", to_role = "technician", on_resource = "site", role = "technician" } + asset_viewer = { resource = "asset", to_role = "viewer", on_resource = "site", role = "viewer" } + } +} + +resource "permitio_role_derivation" "derived" { + for_each = local.derivations + resource = each.value.resource + to_role = each.value.to_role + on_resource = each.value.on_resource + role = each.value.role + linked_by = "parent" + + depends_on = [ + permitio_role.org_admin, permitio_role.region_manager, permitio_role.site_manager, + permitio_role.site_operator, permitio_role.site_technician, permitio_role.site_viewer, + permitio_role.asset_site_manager, permitio_role.asset_operator, permitio_role.asset_technician, + permitio_role.asset_viewer, + permitio_relation.org_parent, permitio_relation.region_parent, permitio_relation.site_parent, + permitio_relation.asset_parent, + ] +} diff --git a/fleet-ops-demo/gitops/terraform/main.tf b/fleet-ops-demo/gitops/terraform/main.tf new file mode 100644 index 0000000..fa82622 --- /dev/null +++ b/fleet-ops-demo/gitops/terraform/main.tf @@ -0,0 +1,31 @@ +# The Fleet Ops schema as Terraform: the same resource types, roles, relations, derivations and +# attribute rules that `make setup` writes through the API (internal/schema/schema.go). State is local +# unless cloud.tf is present; in CI it lives in HCP Terraform, one workspace per environment. + +terraform { + required_version = ">= 1.5" + + required_providers { + permitio = { + source = "permitio/permit-io" + version = "0.0.26" + } + } +} + +variable "permit_api_key" { + description = "The API key of the environment to apply to." + type = string + sensitive = true +} + +variable "permit_api_url" { + description = "Permit's API." + type = string + default = "https://api.permit.io" +} + +provider "permitio" { + api_url = var.permit_api_url + api_key = var.permit_api_key +} diff --git a/fleet-ops-demo/gitops/terraform/resources.tf b/fleet-ops-demo/gitops/terraform/resources.tf new file mode 100644 index 0000000..3d114a3 --- /dev/null +++ b/fleet-ops-demo/gitops/terraform/resources.tf @@ -0,0 +1,112 @@ +# Resource types: the organization hierarchy and the trucks. + +resource "permitio_resource" "org" { + key = "org" + name = "Organization" + actions = { + "manage_users" = { name = "Manage users" } + "assign_roles" = { name = "Assign roles" } + } +} + +resource "permitio_resource" "region" { + key = "region" + name = "Region" + actions = { + "view" = { name = "View" } + "assign_roles" = { name = "Assign roles" } + } +} + +resource "permitio_resource" "site" { + key = "site" + name = "Site" + actions = { + "view" = { name = "View" } + "assign_roles" = { name = "Assign roles" } + } +} + +resource "permitio_resource" "asset" { + key = "asset" + name = "Asset (truck)" + actions = { + "view" = { name = "View" } + "operate" = { name = "Operate" } + "service_basic" = { name = "Service basic" } + "service_advanced" = { name = "Service advanced" } + "decommission" = { name = "Decommission" } + } + attributes = { + "region" = { type = "string", description = "Operating region (South, West, East)" } + "served_account" = { type = "string", description = "Shipper account the truck serves" } + "lifecycle_state" = { type = "string", description = "unassigned, active, transferring, decommissioned" } + "owner_org" = { type = "string", description = "Organization that owns the truck" } + "operator_org" = { type = "string", description = "Organization that operates the truck" } + } +} + +resource "permitio_resource" "team" { + key = "team" + name = "Team" + actions = { + "view_activity" = { name = "View activity" } + } +} + +# Each type hangs under its parent through a `parent` relation; roles derive along it. + +resource "permitio_relation" "org_parent" { + key = "parent" + name = "Parent" + subject_resource = permitio_resource.org.key + object_resource = permitio_resource.org.key +} + +resource "permitio_relation" "region_parent" { + key = "parent" + name = "Parent" + subject_resource = permitio_resource.org.key + object_resource = permitio_resource.region.key +} + +resource "permitio_relation" "site_parent" { + key = "parent" + name = "Parent" + subject_resource = permitio_resource.region.key + object_resource = permitio_resource.site.key +} + +resource "permitio_relation" "asset_parent" { + key = "parent" + name = "Parent" + subject_resource = permitio_resource.site.key + object_resource = permitio_resource.asset.key +} + +resource "permitio_relation" "team_parent" { + key = "parent" + name = "Parent" + subject_resource = permitio_resource.site.key + object_resource = permitio_resource.team.key +} + +# Who the person is: certifications, employer, and time-boxed break-glass grants. + +resource "permitio_user_attribute" "certifications" { + key = "certifications" + type = "array" + description = "Service certifications held: tier1, tier2, tier3" +} + +resource "permitio_user_attribute" "home_tenant" { + key = "home_tenant" + type = "string" + description = "Organization that employs the user" +} + +resource "permitio_user_attribute" "breakglass" { + key = "breakglass" + type = "json" + description = "Time-boxed grants: tenant, incident_id, expires_at" +} diff --git a/fleet-ops-demo/gitops/terraform/roles.tf b/fleet-ops-demo/gitops/terraform/roles.tf new file mode 100644 index 0000000..c215f7c --- /dev/null +++ b/fleet-ops-demo/gitops/terraform/roles.tf @@ -0,0 +1,107 @@ +# Roles and what each one may do. A role on a site also exists on the site's trucks, derived from it +# (derivations.tf), so each truck role lists the actions it gives on the truck. + +resource "permitio_role" "org_admin" { + key = "org_admin" + name = "Org admin" + resource = permitio_resource.org.key + permissions = ["manage_users", "assign_roles"] +} + +resource "permitio_role" "region_manager" { + key = "region_manager" + name = "Region manager" + resource = permitio_resource.region.key + permissions = ["view", "assign_roles"] +} + +resource "permitio_role" "site_manager" { + key = "site_manager" + name = "Site manager" + resource = permitio_resource.site.key + permissions = ["view", "assign_roles"] +} + +resource "permitio_role" "site_operator" { + key = "operator" + name = "Operator" + resource = permitio_resource.site.key + permissions = ["view"] +} + +resource "permitio_role" "site_technician" { + key = "technician" + name = "Technician" + resource = permitio_resource.site.key + permissions = ["view"] +} + +resource "permitio_role" "site_viewer" { + key = "viewer" + name = "Viewer" + resource = permitio_resource.site.key + permissions = ["view"] +} + +# Trucks + +resource "permitio_role" "asset_site_manager" { + key = "site_manager" + name = "Site manager" + resource = permitio_resource.asset.key + permissions = ["view", "decommission"] +} + +resource "permitio_role" "asset_operator" { + key = "operator" + name = "Operator" + resource = permitio_resource.asset.key + permissions = ["view", "operate"] +} + +resource "permitio_role" "asset_technician" { + key = "technician" + name = "Technician" + resource = permitio_resource.asset.key + permissions = [ + "view", + "service_basic", + "service_advanced", + ] +} + +resource "permitio_role" "asset_viewer" { + key = "viewer" + name = "Viewer" + resource = permitio_resource.asset.key + permissions = ["view"] +} + +# A technician from another company, given on named trucks only: never derived. +resource "permitio_role" "asset_partner_technician" { + key = "partner_technician" + name = "Partner technician" + resource = permitio_resource.asset.key + permissions = ["view", "service_basic", "service_advanced"] +} + +resource "permitio_role" "team_supervisor" { + key = "team_supervisor" + name = "Team supervisor" + resource = permitio_resource.team.key + permissions = ["view_activity"] +} + +# Org-level roles: no permissions of their own. The rules in attribute_groups.tf give them sets of trucks. + +resource "permitio_role" "operator" { + key = "operator" + name = "Operator (org-level)" + description = "Views and operates the organization's assets in region South" +} + +resource "permitio_role" "viewer" { + key = "viewer" + name = "Viewer (org-level)" + description = "Views the organization's assets that serve account Acct1" +} From 824b6753a9eaa3f4543a5458ac667fc430108168 Mon Sep 17 00:00:00 2001 From: zivxx Date: Mon, 19 Oct 2026 16:00:03 +0300 Subject: [PATCH 2/4] fix(fleet-ops-demo): make the snapshot script retry, chunk, and log counts instead of people permit_snapshot.py, from the two verifier reports on 2024c09: - Every call retries 429, 502, 503, 504, timeouts and refused connections with backoff, honouring Retry-After, and every failure ends as one line naming the call; a non-UTF-8 error body no longer raises. The retry count is one constant. - restore posts at most 1000 operations per bulk call, the limit Permit's bulk routes enforce. When Permit rejects a call, it posts that call's items one at a time, so one assignment in a deleted tenant no longer blocks the rest. It reports the refusals; a refusal other than 404 or 409 fails it. - restore checks which tuples already exist, as it already did for role assignments, so its counts are what it created. These listings run first and take a 404 as an error, so a wrong project or environment no longer reads as "everyone is gone". It refuses a snapshot saved in another environment. - save stops the apply when the plan deletes or replaces a resource type that still has instances: Permit deletes the instances and everything on them, and no snapshot brings them back. - save always writes a snapshot, an empty one when nothing is deleted, so a revert can tell "nothing to restore" from "snapshot missing". - The logs show counts per role and per kind of relationship, never user keys; a refusal's text has the user key taken out. - Configuration is read in main(), with an empty variable treated as unset, so the script imports cleanly and a missing setting names itself. PERMIT_API_URL must be http(s). Files are read and written through pathlib with an explicit encoding. Co-Authored-By: Claude Opus 5.5 --- fleet-ops-demo/gitops/ci/permit_snapshot.py | 369 +++++++++++++++----- 1 file changed, 277 insertions(+), 92 deletions(-) diff --git a/fleet-ops-demo/gitops/ci/permit_snapshot.py b/fleet-ops-demo/gitops/ci/permit_snapshot.py index 353df51..47a5c83 100644 --- a/fleet-ops-demo/gitops/ci/permit_snapshot.py +++ b/fleet-ops-demo/gitops/ci/permit_snapshot.py @@ -6,12 +6,18 @@ permit_snapshot.py save which reads the plan (`terraform show -json`) and saves the role assignments of every role the plan -destroys and the tuples of every relation it destroys. After a revert is applied, CI runs +destroys and the tuples of every relation it destroys. If the plan destroys a resource type that still +has instances, it stops the apply instead: Permit would delete those instances too, and no snapshot can +bring them back. After a revert is applied, CI runs permit_snapshot.py restore -which puts back exactly what that snapshot saved, skipping people or instances that no longer exist and -anything already there. Reads PERMIT_API_URL, PERMIT_PROJECT, PERMIT_ENV and PERMIT_API_KEY.""" +which puts back what the snapshot saved. It leaves out what is already there and what names a person or +instance that no longer exists, and lists what Permit refuses (say, an assignment in a deleted tenant). + +Reads PERMIT_PROJECT, PERMIT_ENV, PERMIT_API_KEY and, if set, PERMIT_API_URL. It prints counts, never who +holds what: the snapshot file has that, and more people can usually read a CI log than the file.""" +import http.client import json import os import sys @@ -19,113 +25,292 @@ import urllib.error import urllib.parse import urllib.request +from collections import Counter +from pathlib import Path +from typing import Any + +Row = dict[str, Any] + +RETRIES = 5 # after the first try, waiting 1, 2, 4, 8 and 16 s, or as long as a Retry-After header asks +RETRY_STATUSES = {429, 502, 503, 504} +PAGE_SIZE = 100 # the most rows Permit's list routes return per page to an API key +BULK_SIZE = 1000 # the most operations Permit's bulk routes take in one call + + +class PermitError(Exception): + """A call that Permit answered with an error, or never answered, after any retries.""" + + def __init__(self, message: str, status: int = 0, detail: str = "") -> None: + super().__init__(message) + self.status = status # 0 when no HTTP answer came back at all + self.detail = detail -API = (os.environ.get("PERMIT_API_URL") or "https://api.permit.io").rstrip("/") -PROJECT, ENV, KEY = os.environ["PERMIT_PROJECT"], os.environ["PERMIT_ENV"], os.environ["PERMIT_API_KEY"] -FACTS = f"{API}/v2/facts/{PROJECT}/{ENV}" +class Facts: + """Permit's facts API (users, instances, role assignments, relationships) for one environment.""" -def call(method, url, body=None, missing_ok=False): - data = json.dumps(body).encode() if body is not None else None - for attempt in range(6): - req = urllib.request.Request(url, data=data, method=method, headers={ - "Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}) - try: - with urllib.request.urlopen(req, timeout=60) as res: - raw = res.read() + def __init__(self, api_url: str, project: str, env: str, api_key: str) -> None: + self.env = env + self.base = f"{api_url}/v2/facts/{segment(project)}/{segment(env)}" + self._api_key = api_key + self._exists: dict[str, bool] = {} + + def call(self, method: str, path: str, *, params: dict[str, Any] | None = None, body: Any = None, + missing_ok: bool = False) -> Any: + """One API call, retried while Permit is busy or unreachable. With missing_ok, a 404 is None. + + Retrying a POST is safe here: both bulk routes this script posts to skip rows that already exist.""" + url = f"{self.base}/{path}" + (f"?{urllib.parse.urlencode(params)}" if params else "") + data = None if body is None else json.dumps(body).encode() + attempt = 0 + while True: + request = urllib.request.Request(url, data=data, method=method, headers={ + "Authorization": f"Bearer {self._api_key}", "Content-Type": "application/json"}) + try: + with urllib.request.urlopen(request, timeout=60) as response: + raw = response.read() return json.loads(raw) if raw else None - except urllib.error.HTTPError as e: - if e.code == 404 and missing_ok: - return None - if e.code == 429 and attempt < 5: - time.sleep(2 ** attempt) + except urllib.error.HTTPError as e: # before OSError, which HTTPError is a kind of + if e.code == 404 and missing_ok: + return None + if e.code not in RETRY_STATUSES or attempt == RETRIES: + detail = e.read().decode(errors="replace")[:300] + tries = f" (tried {attempt + 1} times)" if attempt else "" + message = f"{method} {url}: HTTP {e.code} {detail}{tries}" + raise PermitError(message, e.code, detail) from None + wait = retry_after(e) or 2**attempt + except (OSError, http.client.HTTPException) as e: # no answer: refused, reset, timed out + if attempt == RETRIES: + raise PermitError(f"{method} {url}: {e} (tried {attempt + 1} times)") from None + wait = 2**attempt + time.sleep(wait) + attempt += 1 + + def list_all(self, path: str, **filters: str) -> list[Row]: + rows: list[Row] = [] + page = 1 + while True: + batch = self.call("GET", path, params={**filters, "page": page, "per_page": PAGE_SIZE}) or [] + rows += batch + if len(batch) < PAGE_SIZE: + return rows + page += 1 + + def exists(self, path: str) -> bool: + if path not in self._exists: + self._exists[path] = self.call("GET", path, missing_ok=True) is not None + return self._exists[path] + + def user_exists(self, key: str) -> bool: + return self.exists(f"users/{segment(key)}") + + def instance_exists(self, key: str) -> bool: + return self.exists(f"resource_instances/{segment(key)}") + + def post_in_bulk(self, path: str, items: list[Row], field: str | None = None + ) -> tuple[list[Row], list[tuple[Row, PermitError]]]: + """POST items to a bulk route, BULK_SIZE at a time, as a list or as {field: list}. + + Permit rejects a whole call when one item in it is wrong (its tenant or role is gone, say). When + it does, this posts that call's items one by one, so the rest still go in. Returns the items + Permit took, and the ones it refused with its answer.""" + taken: list[Row] = [] + refused: list[tuple[Row, PermitError]] = [] + for start in range(0, len(items), BULK_SIZE): + chunk = items[start:start + BULK_SIZE] + try: + self.call("POST", path, body=chunk if field is None else {field: chunk}) + taken += chunk continue - raise SystemExit(f"{method} {url}: HTTP {e.code} {e.read().decode()[:300]}") - raise SystemExit(f"{method} {url}: gave up after retries") + except PermitError as e: + if not is_refusal(e): + raise + for item in chunk: + try: + self.call("POST", path, body=[item] if field is None else {field: [item]}) + taken.append(item) + except PermitError as e: + if not is_refusal(e): + raise + refused.append((item, e)) + return taken, refused + + +def segment(key: str) -> str: + """A key as one URL path segment.""" + return urllib.parse.quote(key, safe="") + + +def retry_after(error: urllib.error.HTTPError) -> int | None: + """How many seconds a 429 or 503 asks to wait, if it says (at most a minute).""" + value = error.headers.get("Retry-After") or "" + return min(int(value), 60) if value.isdigit() else None + + +def is_refusal(error: PermitError) -> bool: + """Permit answered, and said no to what was sent (not to the key, and not "slow down").""" + return 400 <= error.status < 500 and error.status not in (401, 429) + + +def assignment_key(a: Row) -> tuple[str, str, str, str]: + return a["user"], a["role"], a["tenant"], a.get("resource_instance") or "" + + +def tuple_key(t: Row) -> tuple[str, str, str]: + return t["subject"], t["relation"], t["object"] -def list_all(path, **params): - rows, page = [], 1 - while True: - query = urllib.parse.urlencode({**params, "page": page, "per_page": 100}) - batch = call("GET", f"{FACTS}/{path}?{query}") or [] - rows += batch - if len(batch) < 100: - return rows - page += 1 +def tuple_types(t: Row) -> tuple[str, str, str]: + """(relation, object type, subject type): the filters that list a relation's tuples.""" + return t["relation"], t["object"].split(":", 1)[0], t["subject"].split(":", 1)[0] -def destroyed(plan, kind): +def describe_assignment(a: Row) -> str: + """What an assignment grants, without saying who holds it.""" + instance = a.get("resource_instance") + return f"role {a['role']} on {instance.split(':', 1)[0]}" if instance else f"tenant role {a['role']}" + + +def describe_tuple(t: Row) -> str: + relation, object_type, subject_type = tuple_types(t) + return f"relationship: {subject_type} is {relation} of {object_type}" + + +def print_counts(assignments: list[Row], tuples: list[Row]) -> None: + counts = Counter(map(describe_assignment, assignments)) + Counter(map(describe_tuple, tuples)) + for what, n in sorted(counts.items()): + print(f" {n:6} {what}") + + +def destroyed(plan: Row, kind: str) -> list[Row]: """The `before` state of every resource of this type that the plan deletes (or replaces).""" return [c["change"]["before"] for c in plan.get("resource_changes", []) if c["type"] == kind and "delete" in c["change"]["actions"]] -def save(plan_path, out_path): - plan = json.load(open(plan_path)) - assignments, tuples = [], [] +def save(facts: Facts, plan_path: str, snapshot_path: str) -> None: + plan = json.loads(Path(plan_path).read_text(encoding="utf-8")) + + # Deleting a resource type deletes its instances, every role assignment on them and every relationship + # they are in. None of that shows in the plan, and a snapshot can't recreate the instances. + for resource_type in destroyed(plan, "permitio_resource"): + key = resource_type["key"] + if facts.call("GET", "resource_instances", params={"resource": key, "per_page": 1}): + sys.exit(f"This plan deletes the resource type {key!r}, which still has instances. " + "Permit would delete them with it, along with every role assignment on them and every " + "relationship they are in, and this script can't bring those back. If that is what you " + "mean, delete the instances first, then run this job again.") + + assignments: list[Row] = [] for role in destroyed(plan, "permitio_role"): - resource = role.get("resource") - for ra in list_all("role_assignments", role=role["key"]): - instance = ra.get("resource_instance") or "" - if (resource and instance.startswith(f"{resource}:")) or (not resource and not instance): - assignments.append({k: ra.get(k) for k in ("user", "role", "tenant", "resource_instance") if ra.get(k)}) - for rel in destroyed(plan, "permitio_relation"): - for t in list_all("relationship_tuples", relation=rel["key"], object_type=rel["object_resource"]): - if t["subject"].startswith(f"{rel['subject_resource']}:"): - tuples.append({k: t[k] for k in ("subject", "relation", "object", "tenant")}) + resource = role.get("resource") # None for a tenant-level role + for a in facts.list_all("role_assignments", role=role["key"]): + # The role filter matches this key on every resource type and at tenant level: keep this role's. + instance = a.get("resource_instance") or "" + if instance.startswith(f"{resource}:") if resource else not instance: + assignments.append({k: a[k] for k in ("user", "role", "tenant", "resource_instance") + if a.get(k)}) + + tuples: list[Row] = [] + for relation in destroyed(plan, "permitio_relation"): + for t in facts.list_all("relationship_tuples", relation=relation["key"], + object_type=relation["object_resource"], + subject_type=relation["subject_resource"]): + tuples.append({k: t[k] for k in ("subject", "relation", "object", "tenant")}) + + # Written even when empty: then a later revert reads "nothing to restore" from the file, and a + # missing file always means the snapshot is missing. + snapshot = {"environment": facts.env, "commit": os.environ.get("GITHUB_SHA", ""), + "role_assignments": assignments, "tuples": tuples} + Path(snapshot_path).write_text(json.dumps(snapshot, indent=1), encoding="utf-8") if not assignments and not tuples: print("Nothing to save: the plan deletes no role assignments or relationships.") return - snapshot = {"environment": ENV, "commit": os.environ.get("GITHUB_SHA", ""), "role_assignments": assignments, - "tuples": tuples} - json.dump(snapshot, open(out_path, "w"), indent=1) - print(f"Saved what this change deletes in {ENV}: {len(assignments)} role assignments, {len(tuples)} relationships.") - for ra in assignments: - print(f" {ra['user']} is {ra['role']} on {ra.get('resource_instance') or ra['tenant']}") - - -def restore(path): - snap = json.load(open(path)) - users, instances, restored, skipped = {}, {}, [], 0 - - def exists(cache, url): - if url not in cache: - cache[url] = call("GET", url, missing_ok=True) is not None - return cache[url] - - have = {(a["user"], a["role"], a["tenant"], a.get("resource_instance") or "") - for role in {ra["role"] for ra in snap["role_assignments"]} for a in list_all("role_assignments", role=role)} - todo = [] - for ra in snap["role_assignments"]: - key = (ra["user"], ra["role"], ra["tenant"], ra.get("resource_instance") or "") - instance = ra.get("resource_instance") - if key in have: - continue - if not exists(users, f"{FACTS}/users/{urllib.parse.quote(ra['user'])}") or ( - instance and not exists(instances, f"{FACTS}/resource_instances/{urllib.parse.quote(instance)}")): - skipped += 1 - continue - todo.append(ra) - if todo: - call("POST", f"{FACTS}/role_assignments/bulk", todo) - restored += [f"{ra['user']} is {ra['role']} on {ra.get('resource_instance') or ra['tenant']}" for ra in todo] - tuples = [t for t in snap.get("tuples", []) - if exists(instances, f"{FACTS}/resource_instances/{urllib.parse.quote(t['subject'])}") - and exists(instances, f"{FACTS}/resource_instances/{urllib.parse.quote(t['object'])}")] - skipped += len(snap.get("tuples", [])) - len(tuples) - if tuples: - call("POST", f"{FACTS}/relationship_tuples/bulk", {"operations": tuples}) - print(f"Restored in {ENV}: {len(todo)} role assignments, {len(tuples)} relationships; " - f"{skipped} skipped because their person or instance no longer exists.") - for line in restored: - print(f" {line}") + print(f"Saved what this change deletes in {facts.env}: {len(assignments)} role assignments, " + f"{len(tuples)} relationships.") + print_counts(assignments, tuples) + + +def restore(facts: Facts, snapshot_path: str) -> None: + snapshot = json.loads(Path(snapshot_path).read_text(encoding="utf-8")) + if snapshot.get("environment") != facts.env: + sys.exit(f"{snapshot_path} was saved in {snapshot.get('environment')!r}, not in {facts.env!r}.") + assignments: list[Row] = snapshot.get("role_assignments", []) + tuples: list[Row] = snapshot.get("tuples", []) + if not assignments and not tuples: + print("Nothing to restore: that change deleted no role assignments or relationships.") + return + + # What is already there. These calls come first and take a 404 as an error, so a wrong project or + # environment stops the restore here, instead of every later lookup reading it as "that person is gone". + have: set[tuple[str, ...]] = set() + for role in sorted({a["role"] for a in assignments}): + have.update(assignment_key(a) for a in facts.list_all("role_assignments", role=role)) + for relation, object_type, subject_type in sorted({tuple_types(t) for t in tuples}): + found = facts.list_all("relationship_tuples", relation=relation, object_type=object_type, + subject_type=subject_type) + have.update(tuple_key(t) for t in found) + new_assignments = [a for a in assignments if assignment_key(a) not in have] + new_tuples = [t for t in tuples if tuple_key(t) not in have] + + # Leave out what names a person or instance that is gone. + todo_assignments = [a for a in new_assignments if facts.user_exists(a["user"]) + and (not a.get("resource_instance") or facts.instance_exists(a["resource_instance"]))] + todo_tuples = [t for t in new_tuples + if facts.instance_exists(t["subject"]) and facts.instance_exists(t["object"])] + + assignments_in, assignments_refused = facts.post_in_bulk("role_assignments/bulk", todo_assignments) + tuples_in, tuples_refused = facts.post_in_bulk("relationship_tuples/bulk", todo_tuples, + field="operations") + + print(f"Restored in {facts.env}: {len(assignments_in)} role assignments, {len(tuples_in)} relationships.") + print_counts(assignments_in, tuples_in) + already = len(assignments) - len(new_assignments) + len(tuples) - len(new_tuples) + gone = len(new_assignments) - len(todo_assignments) + len(new_tuples) - len(todo_tuples) + if already: + print(f"Already there: {already}.") + if gone: + print(f"Left out because their person or instance no longer exists: {gone}.") + + # Permit's answer can name the person, so it is printed with their key taken out. + refusals = Counter( + [(f"{describe_assignment(a)} in tenant {a['tenant']}", e.status, + e.detail.replace(a["user"], "")) for a, e in assignments_refused] + + [(describe_tuple(t), e.status, e.detail) for t, e in tuples_refused]) + if refusals: + print(f"Refused by Permit: {len(assignments_refused) + len(tuples_refused)}.") + for (what, status, detail), n in sorted(refusals.items()): + print(f" {n:6} {what}: HTTP {status} {detail}") + # A 404 or 409 means something the item names is gone or has moved, like a person who left: expected. + if any(status not in (404, 409) for _, status, _ in refusals): + sys.exit("Permit refused part of the snapshot for some other reason (listed above). " + "Everything else in it was restored.") + + +def setting(name: str) -> str: + """A required environment variable. Empty counts as unset: an unset repository variable reaches a + workflow as an empty string.""" + value = os.environ.get(name, "") + if not value: + sys.exit(f"{name} is not set.") + return value + + +def main(args: list[str]) -> None: + if not ((args[:1] == ["save"] and len(args) == 3) or (args[:1] == ["restore"] and len(args) == 2)): + sys.exit(__doc__) + api_url = (os.environ.get("PERMIT_API_URL") or "https://api.permit.io").rstrip("/") + if not api_url.startswith(("https://", "http://")): + sys.exit(f"PERMIT_API_URL must be an http(s) URL, not {api_url!r}.") + facts = Facts(api_url, setting("PERMIT_PROJECT"), setting("PERMIT_ENV"), setting("PERMIT_API_KEY")) + try: + if args[0] == "save": + save(facts, args[1], args[2]) + else: + restore(facts, args[1]) + except PermitError as e: + sys.exit(str(e)) if __name__ == "__main__": - if sys.argv[1:2] == ["save"] and len(sys.argv) == 4: - save(sys.argv[2], sys.argv[3]) - elif sys.argv[1:2] == ["restore"] and len(sys.argv) == 3: - restore(sys.argv[2]) - else: - raise SystemExit(__doc__) + main(sys.argv[1:]) From dcfc793decdf56ad1a52affa3f65a4d95b53e7a2 Mon Sep 17 00:00:00 2001 From: zivxx Date: Mon, 19 Oct 2026 16:00:03 +0300 Subject: [PATCH 3/4] fix(fleet-ops-demo): find a revert's snapshot through the run that applied it terraform.yaml and gitops/README.md, from the two verifier reports on 2024c09: - The restore step no longer looks artifacts up by name across the repository, where a pull request from a fork can upload one under any name, and no longer unzips one over the checkout before running the checkout's script. It lists this workflow's push runs on this branch, takes the run that ended on the reverted commit, and downloads that run's snapshot into the runner's temporary folder, where only its JSON is read. - Reverts are found with git log since the branch's last run that wasn't cancelled, not in the push event's commits, so a revert whose run GitHub cancelled while it waited is still restored. Only commits that changed terraform/ count. - A revert whose snapshot can't be found, has expired, or was never kept fails the job with an error pointing at the README, instead of printing "that change deleted nothing". - Artifact names carry the run attempt, so a re-run keeps its own snapshot. - Both jobs use bash with pipefail, so a failed `terraform plan | tee` fails the step. - The concurrency group is the target environment, so two release branches never apply to Production at once. - The heredoc Python is gone: no Python left outside the script that the example's CI compiles. - The README says what the step can't do and how to restore by hand: revert the commit a push ended on, keep git's revert line, 30 days of snapshots, failed runs, restored access and tenant membership, user keys in artifacts, one release branch at a time. Co-Authored-By: Claude Opus 5.5 --- .../gitops/.github/workflows/terraform.yaml | 103 +++++++++++++----- fleet-ops-demo/gitops/README.md | 64 ++++++++++- 2 files changed, 137 insertions(+), 30 deletions(-) diff --git a/fleet-ops-demo/gitops/.github/workflows/terraform.yaml b/fleet-ops-demo/gitops/.github/workflows/terraform.yaml index d64b8e3..b310d61 100644 --- a/fleet-ops-demo/gitops/.github/workflows/terraform.yaml +++ b/fleet-ops-demo/gitops/.github/workflows/terraform.yaml @@ -2,8 +2,9 @@ name: Terraform # The schema is Terraform in terraform/. A pull request shows its plan for the environment it targets. # A push applies it: main to Development, release/* to Production. Before every apply, CI saves what the -# plan deletes (role assignments, relationships), because Permit can't undo a delete; when a push is a -# `git revert`, CI puts back what the reverted commit's run saved (ci/permit_snapshot.py). +# plan deletes (role assignments, relationships), because Permit can't undo a delete; when an apply brings +# a `git revert`, CI puts back what the reverted commit's run saved (ci/permit_snapshot.py, and "What a +# destructive change costs" in README.md for what it can't do). # # Needs, in the repository's settings: # secrets PERMIT_API_KEY_DEV, PERMIT_API_KEY_PROD environment-level API keys @@ -23,9 +24,12 @@ on: permissions: contents: read -# One run at a time per environment; never cancel an apply half way. +# One run at a time per environment, whichever branch it comes from (every release/* branch applies to +# Production), and never cancel an apply half way. GitHub keeps one run waiting per group and cancels an +# older waiting run when a newer one queues; the newer run applies both pushes, and its restore step +# looks for reverts in both. concurrency: - group: terraform-${{ github.event_name }}-${{ github.base_ref || github.ref_name }} + group: terraform-${{ github.event_name }}-${{ (github.base_ref || github.ref_name) == 'main' && 'dev' || 'production' }} cancel-in-progress: false env: @@ -50,6 +54,8 @@ jobs: TF_VAR_permit_api_key: ${{ github.base_ref == 'main' && secrets.PERMIT_API_KEY_DEV || secrets.PERMIT_API_KEY_PROD }} defaults: run: + # pipefail: a failed `terraform plan | tee` fails the step instead of posting an empty plan. + shell: bash --noprofile --norc -euo pipefail {0} working-directory: terraform steps: - name: Checkout code @@ -101,14 +107,19 @@ jobs: PERMIT_ENV: ${{ github.ref_name == 'main' && 'dev' || 'production' }} PERMIT_API_KEY: ${{ github.ref_name == 'main' && secrets.PERMIT_API_KEY_DEV || secrets.PERMIT_API_KEY_PROD }} TF_VAR_permit_api_key: ${{ github.ref_name == 'main' && secrets.PERMIT_API_KEY_DEV || secrets.PERMIT_API_KEY_PROD }} + # The file, and the artifact, for what this run's plan deletes. The attempt number keeps a re-run + # from colliding with what an earlier attempt kept. + SNAPSHOT: snapshot-${{ github.ref_name == 'main' && 'dev' || 'production' }}-${{ github.sha }}-${{ github.run_attempt }} defaults: run: + shell: bash --noprofile --norc -euo pipefail {0} working-directory: terraform steps: - name: Checkout code uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false + fetch-depth: 0 # the restore step reads the commits since the last apply - name: Set up Terraform uses: hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1 with: @@ -121,37 +132,77 @@ jobs: terraform plan -input=false -no-color -out=tfplan | tee plan.txt terraform show -json tfplan > plan.json - name: Save what this change deletes - run: python3 ../ci/permit_snapshot.py save plan.json "../snapshot-${PERMIT_ENV}-${GITHUB_SHA}.json" + run: python3 ../ci/permit_snapshot.py save plan.json "../${SNAPSHOT}.json" + # Kept even when empty, so a revert can tell "deleted nothing" from "snapshot missing". - name: Keep the snapshot with this run - if: hashFiles(format('snapshot-{0}-{1}.json', env.PERMIT_ENV, github.sha)) != '' uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: - name: snapshot-${{ env.PERMIT_ENV }}-${{ github.sha }} - path: snapshot-${{ env.PERMIT_ENV }}-${{ github.sha }}.json + name: ${{ env.SNAPSHOT }} + path: ${{ env.SNAPSHOT }}.json retention-days: 30 - name: Apply run: terraform apply -input=false -no-color tfplan - name: Restore what a reverted commit deleted env: GH_TOKEN: ${{ github.token }} + GH_REPO: ${{ github.repository }} run: | - # Every commit in this push that says "This reverts commit ": restore that commit's snapshot. - python3 - "$GITHUB_EVENT_PATH" > ../reverted.txt <<'PY' - import json, re, sys - event = json.load(open(sys.argv[1])) - for c in event.get("commits", []): - for sha in re.findall(r"This reverts commit ([0-9a-f]{40})", c["message"]): - print(sha) - PY - while read -r sha; do - name="snapshot-${PERMIT_ENV}-${sha}" - id=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts?name=${name}" --jq '.artifacts[0].id // empty') - if [ -z "$id" ]; then - echo "Reverts ${sha:0:7}: that change deleted nothing, so there's nothing to restore." + # This branch's earlier runs of this workflow, newest first, as " ". Only runs + # on a push to this branch count: a pull request from a fork can upload an artifact under any + # name, so a snapshot is never looked up by its name alone. Cancelled runs never ran. + workflow=$(basename "${GITHUB_WORKFLOW_REF%@*}") + gh api -X GET "repos/${GITHUB_REPOSITORY}/actions/workflows/${workflow}/runs" \ + -f event=push -f branch="$GITHUB_REF_NAME" -f status=completed -f per_page=100 \ + --jq '.workflow_runs[] | select(.conclusion != "cancelled") | "\(.id) \(.head_sha)"' \ + > "$RUNNER_TEMP/runs.txt" + + # The schema reverts since the last of those runs, which can be more than this push: see + # `concurrency` above. + since=$(head -n 1 "$RUNNER_TEMP/runs.txt" | cut -d ' ' -f 2) + if [ -z "$since" ]; then + echo "This is the first apply from ${GITHUB_REF_NAME}: no earlier run here to restore from." + exit 0 + fi + if git merge-base --is-ancestor "$since" HEAD 2>/dev/null; then + range="$since..HEAD" + else + range="HEAD^!" # the branch's history was rewritten since: this commit only + fi + reverted=$(git log --format=%B "$range" -- . \ + | { grep -oE 'This reverts commit [0-9a-f]{40}' || true; } | cut -d ' ' -f 4 | sort -u) + + failed=0 + for sha in $reverted; do + # The run that applied a commit is the one that ended on it. + runs=$(awk -v sha="$sha" '$2 == sha { print $1 }' "$RUNNER_TEMP/runs.txt") + if [ -z "$runs" ]; then + echo "::error::Reverts ${sha:0:7}, but no run of this workflow on ${GITHUB_REF_NAME} ended on that commit, so CI can't tell what it deleted. See \"When CI can't restore a revert\" in the README." + failed=1 continue fi - gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${id}/zip" > "../${name}.zip" - unzip -o -q "../${name}.zip" -d .. - echo "Reverts ${sha:0:7}: restoring what it deleted." - python3 ../ci/permit_snapshot.py restore "../${name}.json" - done < ../reverted.txt + kept=0 + for run in $runs; do + # ":" for each snapshot that run kept (an artifact name can't contain ":"). + snapshots=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${run}/artifacts" \ + --jq '.artifacts[] | select(.name | startswith("snapshot-\(env.PERMIT_ENV)-")) | "\(.name):\(.expired)"') + for snapshot in $snapshots; do + kept=1 + name=${snapshot%:*} + if [ "${snapshot##*:}" = true ]; then + echo "::error::Reverts ${sha:0:7}, but its snapshot ${name} has expired. See \"When CI can't restore a revert\" in the README." + failed=1 + continue + fi + # Into the runner's temporary folder, never the checkout: only the JSON is read, as data. + gh run download "$run" --name "$name" --dir "$RUNNER_TEMP/$name" + echo "Reverts ${sha:0:7}: restoring from ${name}." + python3 ../ci/permit_snapshot.py restore "$RUNNER_TEMP/$name/$name.json" || failed=1 + done + done + if [ "$kept" = 0 ]; then + # Every run that gets past its save step keeps one, empty or not. + echo "::error::Reverts ${sha:0:7}, but the run that applied it kept no snapshot: it stopped before saving one, or the snapshot has expired and been removed. See \"When CI can't restore a revert\" in the README." + failed=1 + fi + done + exit "$failed" diff --git a/fleet-ops-demo/gitops/README.md b/fleet-ops-demo/gitops/README.md index 8ac241c..afade6c 100644 --- a/fleet-ops-demo/gitops/README.md +++ b/fleet-ops-demo/gitops/README.md @@ -33,14 +33,70 @@ empty: nobody holds it anymore. Terraform: 1. Before each apply, `save` reads the plan (`terraform show -json`), and for every role or relation the - plan destroys, lists its role assignments or tuples through Permit's API and keeps them with the run. -2. When a push contains a commit that says "This reverts commit X", `restore` fetches the file X's run - kept and recreates exactly those assignments and tuples, skipping people and instances that no longer - exist. + plan destroys, lists its role assignments or tuples through Permit's API and keeps them with the run + as an artifact, for 30 days. A run whose plan deletes nothing keeps an empty one, so a revert of it + can tell "nothing to restore" from "snapshot missing". If the plan destroys a resource type that still + has instances, `save` stops the apply instead: Permit would delete the instances too, with every + assignment on them and every relationship they're in, and no snapshot can bring them back. If that's + what you mean, delete the instances yourself first, then run the job again. +2. After each apply, CI looks through the commits since the branch's last apply for ones that changed + `terraform/` and say "This reverts commit X". For each, it finds the run that applied X, fetches the + snapshot that run kept, and `restore` recreates those assignments and tuples. It leaves out what's + already there and what names a person or instance that no longer exists, and lists anything Permit + refuses, such as an assignment in a tenant deleted since. + +The log shows how many assignments of each role, and relationships of each kind, were saved or restored, +never who holds them. If you'd rather a role were never deleted by accident, Terraform's `lifecycle { prevent_destroy = true }` on it makes any plan that would delete it fail. +### What to know before you rely on it + +- **Revert the commit a push ended on.** CI knows a run by the last commit of its push, so that's the + commit whose snapshot it can find. With squash merges every pull request lands as one commit, and this + always holds. Revert a commit from the middle of a push (a rebase merge, say) and the apply fails and + says so; see [below](#when-ci-cant-restore-a-revert). CI looks back over the branch's last 100 runs, on + that branch only. +- **Keep the line `git revert` writes.** CI finds reverts by the `This reverts commit ` line in the + commit message. A squash merge set to use only the pull request's title drops it, and one set to use + the title and description keeps it only if the description has it. +- **Snapshots last 30 days** (`retention-days` in `terraform.yaml`, at most 90 unless the repository's + settings allow more). Revert an older change and the apply fails, reporting the snapshot expired. +- **A failed run doesn't pass its revert on.** If the run carrying a revert fails before its restore + step, and nothing has been applied from that branch since, fix the cause and re-run it. Once a newer + push has been applied, don't: re-running an older commit's apply applies its older schema. Restore by + hand instead. (A run that GitHub cancelled while it waited is different: the next run applies that + push too, and restores its reverts.) +- **Restore gives back access as it was when the snapshot was taken.** Permit's bulk assignment also + puts the person back in the tenant if they had left it. Someone whose access was removed on purpose + since then gets it back. For Production, consider a GitHub environment with required reviewers on the + `apply` job, so a person approves each apply, revert or not. +- **The snapshot says who had which role.** The log shows only counts, but the artifact holds user keys + (often emails), and anyone with read access to the repository can download it while it lasts. Keep the + policy repository private. +- **Use one release branch at a time.** Every `release/*` branch applies to Production. Runs take turns + per environment, so two never apply at once. But a push to an older release branch applies that branch's + older schema, deleting what newer releases added, and that isn't a revert, so CI saves it but doesn't + put it back. + +### When CI can't restore a revert + +The apply job fails and says why: no run on the branch ended on the reverted commit, or that run's +snapshot has expired or was never kept (the run stopped before saving it). The schema change is applied either way: only the data is missing. To +restore by hand, download the snapshot from the run that applied the reverted change (that run's page in +the Actions tab, under Artifacts, or `gh run download --name `), check it's the one you +mean, and run it with that environment's API key: + +```bash +PERMIT_PROJECT= PERMIT_ENV= PERMIT_API_KEY= \ + python3 ci/permit_snapshot.py restore snapshot---.json +``` + +If the reverted commit came in a push with other commits, that run's snapshot holds what the whole push +deleted, and `restore` puts back whatever of it can exist again. `restore` refuses a snapshot saved in +another environment, and running it twice changes nothing the second time. + ## Setting it up 1. **A Permit project with two environments**, Development and Production, both empty, and a From fa878a1438e632d8c64efdbbd3fafad71045675d Mon Sep 17 00:00:00 2001 From: zivxx Date: Mon, 19 Oct 2026 16:04:30 +0300 Subject: [PATCH 4/4] fix(fleet-ops-demo): look for reverts from the last successful apply, not the last run A run that failed before applying (an HCP Terraform outage, say) moved the starting point past its own revert, so the next push never restored it. Starting from the last successful run looks at a failed run's reverts again; restoring twice is safe, since restore skips what's already there. Co-Authored-By: Claude Opus 5.5 --- .../gitops/.github/workflows/terraform.yaml | 17 ++++++----- fleet-ops-demo/gitops/README.md | 30 ++++++++++--------- 2 files changed, 25 insertions(+), 22 deletions(-) diff --git a/fleet-ops-demo/gitops/.github/workflows/terraform.yaml b/fleet-ops-demo/gitops/.github/workflows/terraform.yaml index b310d61..e81a10e 100644 --- a/fleet-ops-demo/gitops/.github/workflows/terraform.yaml +++ b/fleet-ops-demo/gitops/.github/workflows/terraform.yaml @@ -147,20 +147,21 @@ jobs: GH_TOKEN: ${{ github.token }} GH_REPO: ${{ github.repository }} run: | - # This branch's earlier runs of this workflow, newest first, as " ". Only runs - # on a push to this branch count: a pull request from a fork can upload an artifact under any - # name, so a snapshot is never looked up by its name alone. Cancelled runs never ran. + # This branch's earlier runs of this workflow, newest first, as " ". + # Only runs on a push to this branch count: a pull request from a fork can upload an artifact + # under any name, so a snapshot is never looked up by its name alone. Cancelled runs never ran. workflow=$(basename "${GITHUB_WORKFLOW_REF%@*}") gh api -X GET "repos/${GITHUB_REPOSITORY}/actions/workflows/${workflow}/runs" \ -f event=push -f branch="$GITHUB_REF_NAME" -f status=completed -f per_page=100 \ - --jq '.workflow_runs[] | select(.conclusion != "cancelled") | "\(.id) \(.head_sha)"' \ + --jq '.workflow_runs[] | select(.conclusion != "cancelled") | "\(.id) \(.head_sha) \(.conclusion)"' \ > "$RUNNER_TEMP/runs.txt" - # The schema reverts since the last of those runs, which can be more than this push: see - # `concurrency` above. - since=$(head -n 1 "$RUNNER_TEMP/runs.txt" | cut -d ' ' -f 2) + # The schema reverts since the last run that succeeded, which can be more than this push (see + # `concurrency` above). A failed run may have stopped before applying or restoring, so its + # reverts are looked at again; restoring twice is safe, since restore skips what's already there. + since=$(awk '$3 == "success" { print $2; exit }' "$RUNNER_TEMP/runs.txt") if [ -z "$since" ]; then - echo "This is the first apply from ${GITHUB_REF_NAME}: no earlier run here to restore from." + echo "No earlier successful apply from ${GITHUB_REF_NAME}: no run here to restore from." exit 0 fi if git merge-base --is-ancestor "$since" HEAD 2>/dev/null; then diff --git a/fleet-ops-demo/gitops/README.md b/fleet-ops-demo/gitops/README.md index afade6c..bb2cb75 100644 --- a/fleet-ops-demo/gitops/README.md +++ b/fleet-ops-demo/gitops/README.md @@ -39,11 +39,11 @@ Terraform: has instances, `save` stops the apply instead: Permit would delete the instances too, with every assignment on them and every relationship they're in, and no snapshot can bring them back. If that's what you mean, delete the instances yourself first, then run the job again. -2. After each apply, CI looks through the commits since the branch's last apply for ones that changed - `terraform/` and say "This reverts commit X". For each, it finds the run that applied X, fetches the - snapshot that run kept, and `restore` recreates those assignments and tuples. It leaves out what's - already there and what names a person or instance that no longer exists, and lists anything Permit - refuses, such as an assignment in a tenant deleted since. +2. After each apply, CI looks through the commits since the branch's last successful apply for ones + that changed `terraform/` and say "This reverts commit X". For each, it finds the run that applied X, + fetches the snapshot that run kept, and `restore` recreates those assignments and tuples. It leaves + out what's already there and what names a person or instance that no longer exists, and lists + anything Permit refuses, such as an assignment in a tenant deleted since. The log shows how many assignments of each role, and relationships of each kind, were saved or restored, never who holds them. @@ -63,11 +63,12 @@ on it makes any plan that would delete it fail. the title and description keeps it only if the description has it. - **Snapshots last 30 days** (`retention-days` in `terraform.yaml`, at most 90 unless the repository's settings allow more). Revert an older change and the apply fails, reporting the snapshot expired. -- **A failed run doesn't pass its revert on.** If the run carrying a revert fails before its restore - step, and nothing has been applied from that branch since, fix the cause and re-run it. Once a newer - push has been applied, don't: re-running an older commit's apply applies its older schema. Restore by - hand instead. (A run that GitHub cancelled while it waited is different: the next run applies that - push too, and restores its reverts.) +- **A failed run passes its reverts on.** If the run carrying a revert fails before or during its + restore, the next run on that branch looks at the same reverts again, and restoring twice changes + nothing. To restore sooner, re-run the failed run, but only while nothing newer has been applied from + that branch: re-running an older commit's apply applies its older schema. (A run that GitHub cancelled + while it waited is passed on the same way: the next run applies that push too, and restores its + reverts.) - **Restore gives back access as it was when the snapshot was taken.** Permit's bulk assignment also puts the person back in the tenant if they had left it. Someone whose access was removed on purpose since then gets it back. For Production, consider a GitHub environment with required reviewers on the @@ -83,10 +84,11 @@ on it makes any plan that would delete it fail. ### When CI can't restore a revert The apply job fails and says why: no run on the branch ended on the reverted commit, or that run's -snapshot has expired or was never kept (the run stopped before saving it). The schema change is applied either way: only the data is missing. To -restore by hand, download the snapshot from the run that applied the reverted change (that run's page in -the Actions tab, under Artifacts, or `gh run download --name `), check it's the one you -mean, and run it with that environment's API key: +snapshot has expired or was never kept (the run stopped before saving it). The schema change is applied +either way: only the data is missing. To restore by hand, download the snapshot from the run that +applied the reverted change (that run's page in the Actions tab, under Artifacts, or +`gh run download --name `), check it's the one you mean, and run it with that +environment's API key: ```bash PERMIT_PROJECT= PERMIT_ENV= PERMIT_API_KEY= \