chore/deployment ingester docs

[alanpeixinho]

What it is

• Document schema-change coordination before staging/production deploys (KernelCI dashboard Discord; new tables → notify Denys Fedoryshchenko for manual DB grants).
• Document production ingester deploy via kcidb-ng Build and Deploy, not dashboard GitHub Actions.
• Add DEPLOYMENT.md sections and [!IMPORTANT] ingester callouts on main/staging deploy docs; link from README.md, backend/README.md, and docs/dev-environment.md.

[felipebergamin]

I think that could be nice to have a "Quick reference", step by step for deployments.

later we could also create a skill for that on another PR. Adding gates to the migrations and how they should be handled.

This was written by cursor, it might need some improvements:

---

Prod does not deploy on merge. Push to main only runs CI, staging (local build + e2e), and a first GHCR publish. Production is always manual.

Preconditions

• Deploy the tip of origin/main (or a SHA already on main).
• CI / staging / e2e green on that commit.
• Migrations adding tables: notify Denys and wait for his ack before deploying. Staging and prod share one PostgreSQL; migrations run on backend container start. After a migration deploy he has to apply manual database permission fixes. Do not ship a release that includes new Django migrations without his approval. Announce on Discord as well.

Steps

git fetch origin main --tags
git checkout main && git pull --ff-only origin main
git log -1 --oneline
git tag -l "release/$(date +%Y%m%d).*"

Check for new migrations vs the previous tag:

git diff --name-only release/<previous>..HEAD -- '**/migrations/*.py'

If that list is non-empty, stop until Denys has approved.

1. Tag release/YYYYMMDD.N (N = 0, 1, … that calendar day) on the commit you are shipping, then push the tag.

git tag release/YYYYMMDD.N <sha>
git push origin release/YYYYMMDD.N

2. Publish GHCR Images (Actions → Run workflow) on that same main commit. Wait until backend, frontend, and proxy succeed.
   The merge-time publish happened before the tag, so :latest still has the old version string until this republish.

3. Deploy production Dashboard (Actions → Run workflow). Input tag = release/YYYYMMDD.N. Run it from main while main is still that tagged commit.

4. Confirm the version in the prod side menu matches the tag: https://dashboard.kernelci.org
   If this release had migrations, ping Denys so he can run the post-deploy permission fixes.

CLI equivalent:

gh workflow run "Publish GHCR Images" --repo kernelci/dashboard --ref main
gh run watch <run-id> --repo kernelci/dashboard

gh workflow run "Deploy production Dashboard" --repo kernelci/dashboard --ref main -f tag=release/YYYYMMDD.N
gh run watch <run-id> --repo kernelci/dashboard

Gotchas

• The tag input is Discord-only. Checkout uses GITHUB_SHA of the dispatch; the host clone is --depth 1 --branch main. Wrong ref → wrong (or broken) deploy.
• Prod pulls :latest (docker-compose-next.yml). Do not start step 3 until step 2 has finished.
• Ingester and pending_aggregations_processor are not updated (profile with_commands is not passed). --remove-orphans without that profile can even stop them if they were already running.
• staging-db.yaml is a different host. Ignore it for this path.

[tales-aparecida]

FWIW
The TSC talked about detaching migrate from the deployment. I'm unsure that would work well due to how the deployed code would expect to find fields that don't exist yet. Unless we further detach the migration from the usage of the new fields, i.e. separate MRs reshaping data structures from MRs adding new functionality.

[alanpeixinho]

We could maybe go to the path of making migrations a manual github action, and just implementing a guard on the dashboard deploy to check if all migrations required for that version have already been applied. And, if not, we can just fail the deploy.
PS: I believe we should also be able to make migrations apply table permissions without intervention, but I understand this would be undesirable.

[tales-aparecida]

I would personally desire it, from what I understood, it's seen as impossible

[alanpeixinho]

Included some changes we argue here, @felipebergamin.
The broken build on invalid GITHUB_SHA should not happen on the #2131.

[felipebergamin]

@alanpeixinho do you plan to add the quickreference?

I think would be nice to have something to follow when deploying. A checklist like

- Pre-deploy
  - Create tag
  - Push it 
  - .........
  - .........

- Post-deploy
  - Check on dozzle the containers health and init logs
  - Write a changelog and send to kernelci mailing list 
  - .......
  - .......

When onboarding someone they should read the complete document, but after that this checklist can be used to make sure nothing will be forgotten.