From 5eb50a57ef05e564c2834f84f7d92757e4f320bc Mon Sep 17 00:00:00 2001 From: Ayush Raj Date: Sun, 1 Nov 2026 12:06:45 +0530 Subject: [PATCH 1/4] docs: add detailed AWS IAM OIDC and SSM production setup guide --- docs/AWS_IAM_OIDC_SSM_PRODUCTION_SETUP.md | 631 ++++++++++++++++++++++ 1 file changed, 631 insertions(+) create mode 100644 docs/AWS_IAM_OIDC_SSM_PRODUCTION_SETUP.md diff --git a/docs/AWS_IAM_OIDC_SSM_PRODUCTION_SETUP.md b/docs/AWS_IAM_OIDC_SSM_PRODUCTION_SETUP.md new file mode 100644 index 0000000..78cb3a7 --- /dev/null +++ b/docs/AWS_IAM_OIDC_SSM_PRODUCTION_SETUP.md @@ -0,0 +1,631 @@ +# CloudMonitor — AWS IAM, GitHub OIDC, SSM and Production Setup + +**Verified:** 1 November 2026 +**Repository:** ayushcmd/cloud_monitor +**AWS region:** ap-southeast-2 (Sydney) +**GitHub Environment:** production +**Production host directory:** /home/ubuntu/cloud_monitor +**Status:** First SSM production deployment verified + +This is the button-by-button record of the AWS and GitHub configuration used to move CloudMonitor production delivery from runner-originated SSH to GitHub OIDC plus AWS Systems Manager Run Command. + +> **Secrets and identifiers:** Values below use placeholders where appropriate. Never commit production .env contents, JWTs, database passwords, agent keys, AWS access keys, private keys or secret values. Do not paste them into issue comments or workflow logs. + +--- + +## 1. Architecture and roles + +~~~text +GitHub Actions (production Environment) + | + | OIDC token: issuer token.actions.githubusercontent.com + | audience sts.amazonaws.com + v +IAM role: CloudMonitorGitHubDeployRole + | + | short-lived STS credentials + | ssm:SendCommand + v +AWS Systems Manager Run Command + | + v +CloudMonitor EC2 instance / SSM Agent + | + | runs AWS-RunShellScript + v +/home/ubuntu/cloud_monitor + | + | fetch origin/main, checkout, Docker Compose + v +frontend | backend | postgres | prometheus + | node-exporter | alertmanager +~~~ + +There are two distinct roles: + +| Role | Trusted entity | Purpose | +|---|---|---| +| CloudMonitorEC2SSMRole | AWS service: EC2 | Lets the EC2 instance's SSM Agent register with and communicate with Systems Manager | +| CloudMonitorGitHubDeployRole | GitHub OIDC web identity | Lets the approved GitHub Actions job assume a short-lived AWS role and call the SSM deployment path | + +The EC2 role is attached to the instance as an instance profile. The GitHub role is not attached to EC2. + +**Current workflow behavior:** the CD workflow is manual-only through workflow_dispatch. A push or merge to main runs CI but does not automatically deploy production. Deploy and rollback require an explicit confirmation input. Do not describe this as push-triggered automatic CD. + +GitHub-hosted runners do not need inbound SSH access to deploy through SSM. Keep restricted administrator SSH as a break-glass path until an alternative human access method, such as Session Manager, is separately tested. Do not open port 22 to the internet. + +--- + +## 2. Prepare the values + +Collect these from your own account: + +| Value | Source | +|---|---| +| AWS account ID | AWS account menu or STS caller identity | +| Region | ap-southeast-2 | +| EC2 instance ID | EC2 → Instances → CloudMonitor production host | +| GitHub repository | ayushcmd/cloud_monitor | +| GitHub Environment | production | +| Exact OIDC subject | Inspect OIDC claims step of an oidc-check workflow run | +| Deployment role ARN | IAM → Roles → CloudMonitorGitHubDeployRole | + +Do not select another project's instance. CloudMonitor and other project instances may appear in the same EC2 or Fleet Manager list. + +--- + +## 3. EC2 instance role and SSM Agent + +### 3.1 Create or verify the EC2 role + +1. Open AWS Console → IAM. +2. Select **Roles** in the left navigation. +3. Select **Create role**. +4. Under **Trusted entity type**, choose **AWS service**. +5. Choose **EC2** as the use case. +6. Select **Next** to add permissions. +7. Search for and select **AmazonSSMManagedInstanceCore**. +8. Select **Next**. +9. Set role name to **CloudMonitorEC2SSMRole**. +10. Add a description explaining that the role allows the CloudMonitor EC2 host to communicate with Systems Manager. +11. Review that the trusted entity is EC2 and the policy is AmazonSSMManagedInstanceCore. +12. Select **Create role**. + +If the role already exists with this trust and policy, reuse it. Do not create a duplicate. + +### 3.2 Attach the role to CloudMonitor EC2 + +1. Open **EC2** → **Instances**. +2. Select only the CloudMonitor production instance. +3. Choose **Actions** → **Security** → **Modify IAM role**. +4. Select **CloudMonitorEC2SSMRole** from the IAM role dropdown. +5. Select **Update IAM role**. +6. Return to instance details and confirm the role is attached. + +### 3.3 Verify the SSM Agent + +On the authorized Ubuntu host, run: + +~~~bash +sudo snap list amazon-ssm-agent +sudo snap services amazon-ssm-agent +sudo ssm-cli get-diagnostics --output table +~~~ + +Check that the agent is enabled/running, EC2 IMDS is accessible, AWS credentials resolve to the EC2 instance role, and regional SSM endpoints are reachable. + +The EC2 host needs outbound HTTPS (TCP 443) to required AWS endpoints. SSM does not require an inbound listener or inbound security-group port. + +### 3.4 Confirm Fleet Manager registration + +1. Open **Systems Manager**. +2. Open **Fleet Manager** → **Managed nodes**. +3. Refresh the list. +4. Find the CloudMonitor instance by its instance ID/name. +5. Confirm its node state is Running and ping status is **Online**. + +If it is not Online, check the EC2 role, SSM Agent, IMDS and outbound endpoint connectivity before continuing. + +--- + +## 4. GitHub OIDC identity provider + +### 4.1 Check for an existing provider first + +1. Open **IAM** → **Identity providers**. +2. Find **token.actions.githubusercontent.com**. +3. Open the provider. +4. Confirm type is **OpenID Connect**. +5. Confirm audience is **sts.amazonaws.com**. + +The CloudMonitor AWS account already had this provider. Reuse it. Do not create duplicates or delete a provider used by other roles. + +### 4.2 Create it only if absent + +1. Select **Add provider**. +2. Choose **OpenID Connect**. +3. Provider URL: **https://token.actions.githubusercontent.com** +4. Audience: **sts.amazonaws.com** +5. Select **Add provider**. +6. Reopen the provider and verify the URL and audience. + +--- + +## 5. GitHub deployment role + +### 5.1 Create the role + +1. Open **IAM** → **Roles** → **Create role**. +2. For **Trusted entity type**, choose **Web identity**. +3. Choose identity provider **token.actions.githubusercontent.com**. +4. Choose audience **sts.amazonaws.com**. +5. Enter GitHub organization/owner **ayushcmd** and repository **cloud_monitor** if prompted. +6. Continue to **Add permissions**. + +The simplified console form may initially generate broader trust conditions. After role creation, edit the Trust relationship JSON so the subject is restricted to the exact production Environment subject shown by the workflow's OIDC check. + +### 5.2 Add the inline SSM policy + +1. On **Add permissions**, select **Create inline policy**. +2. Open the **JSON** editor. +3. Replace the editor contents with the policy below, inserting your AWS account ID and exact EC2 instance ID. +4. Continue and name the inline policy **CloudMonitorSSMDeployPolicy**. + +~~~json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "DescribeManagedInstances", + "Effect": "Allow", + "Action": "ssm:DescribeInstanceInformation", + "Resource": "*" + }, + { + "Sid": "AllowApprovedRunShellScriptDocument", + "Effect": "Allow", + "Action": "ssm:SendCommand", + "Resource": "arn:aws:ssm:ap-southeast-2::document/AWS-RunShellScript" + }, + { + "Sid": "AllowCommandOnlyOnCloudMonitorInstance", + "Effect": "Allow", + "Action": "ssm:SendCommand", + "Resource": "arn:aws:ec2:ap-southeast-2::instance/" + }, + { + "Sid": "ReadCommandInvocationResults", + "Effect": "Allow", + "Action": "ssm:GetCommandInvocation", + "Resource": "*" + } + ] +} +~~~ + +The two ssm:SendCommand resource entries are intentional: one is the approved SSM document and one is the exact EC2 target. Keep the target resource restricted to the CloudMonitor instance. Do not substitute a wildcard EC2 resource or attach AdministratorAccess. + +The read actions above use Resource "*". Review AWS service authorization/resource-level support before trying to narrow those permissions. + +### 5.3 Name and create the role + +1. On **Name, review, and create**, enter role name **CloudMonitorGitHubDeployRole**. +2. Suggested description: Allows the CloudMonitor GitHub Actions production workflow to authenticate through OIDC and deploy to EC2 with Systems Manager Run Command. +3. Confirm the trusted entity is GitHub OIDC. +4. Confirm inline policy CloudMonitorSSMDeployPolicy is attached. +5. Select **Create role**. +6. Open the created role and copy its ARN for the GitHub Environment variable. + +### 5.4 Edit the trust relationship + +Open the role → **Trust relationships** → **Edit trust policy**. + +Use this pattern and replace placeholders with the actual AWS account ID and exact OIDC subject: + +~~~json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Federated": "arn:aws:iam:::oidc-provider/token.actions.githubusercontent.com" + }, + "Action": "sts:AssumeRoleWithWebIdentity", + "Condition": { + "StringEquals": { + "token.actions.githubusercontent.com:aud": "sts.amazonaws.com", + "token.actions.githubusercontent.com:sub": "" + } + } + } + ] +} +~~~ + +The immutable subject observed for this CloudMonitor setup was: + +~~~text +repo:ayushcmd@252997256/cloud_monitor@1419373799:environment:production +~~~ + +GitHub subject formats can depend on repository OIDC subject-claim settings. The subject printed in the current workflow run is authoritative. It must match the trust policy exactly. Do not replace it with repo-wide or branch-wide wildcards. + +1. Select **Update policy**. +2. Reopen **Trust relationships** and verify the saved JSON. + +--- + +## 6. GitHub production Environment + +### 6.1 Open the existing Environment + +1. Open GitHub repository **ayushcmd/cloud_monitor**. +2. Select **Settings**. +3. Open **Environments**. +4. Open the existing **production** Environment. +5. Do not delete/recreate it; it already holds the workflow configuration. + +### 6.2 Restrict deployment branches + +1. In **Deployment branches and tags**, choose **Selected branches and tags**. +2. Select **Add deployment branch or tag rule**. +3. Add branch **main**. +4. Confirm that one branch, main, is allowed. + +### 6.3 Review Environment protection + +The verified setup left **Required reviewers** and **Wait timer** unchecked and allowed administrators to bypass configured rules. + +For stronger governance, consider adding a required reviewer and disabling administrator bypass if the repository plan/settings permit. These were not mandatory approval gates during the verified deployment. + +### 6.4 Add the three Environment variables + +Scroll to **Environment variables** → **Add environment variable**. Create: + +| Variable name | Value | +|---|---| +| AWS_ACCOUNT_ID | Your 12-digit AWS account ID | +| AWS_DEPLOY_ROLE_ARN | ARN of CloudMonitorGitHubDeployRole | +| EC2_INSTANCE_ID | CloudMonitor production EC2 instance ID | + +The workflow sets AWS_REGION to ap-southeast-2 in YAML. These are GitHub Environment variables, not repository secrets. + +### 6.5 Legacy SSH secrets + +The Environment previously contained EC2_HOST, EC2_USER, EC2_SSH_PRIVATE_KEY and EC2_SSH_KNOWN_HOSTS secrets. The current SSM workflow does not consume them. They were left in place during migration. Search all workflows before removing them. Never reveal private-key values. + +--- + +## 7. GitHub Actions operations + +Workflow file: **.github/workflows/cd.yml** +Workflow name: **CloudMonitor CD — Deploy to AWS EC2 via SSM** + +The workflow job has contents:read and id-token:write. The id-token permission permits requesting the OIDC token; AWS permissions are granted by the assumed IAM role, not by the GitHub permission alone. + +### Available operations + +| Operation | Behavior | Requires production confirmation | +|---|---|---| +| oidc-check | Inspects issuer/audience/subject, assumes the AWS role and verifies caller identity | No | +| ssm-preflight | Confirms EC2 Online in SSM and runs read-only Docker/Compose checks | No | +| deploy | Fetches main, builds/updates production services and runs health checks | Yes | +| rollback | Restores the previously recorded Git revision and rebuilds application services | Yes | + +The OIDC token itself is never printed. The workflow prints only the claims needed to compare with IAM trust. + +### Run oidc-check + +1. Open repository **Actions**. +2. Select **CloudMonitor CD — Deploy to AWS EC2 via SSM**. +3. Select **Run workflow**. +4. Under **Use workflow from**, choose main. +5. Choose operation **oidc-check**. +6. Leave production confirmation unchecked. +7. Select **Run workflow**. +8. Expand OIDC inspection, AWS credential configuration and role-assumption steps. +9. Confirm issuer, audience, subject and STS caller identity are correct. + +### Run ssm-preflight + +1. Open the same workflow and select **Run workflow**. +2. Choose main. +3. Choose operation **ssm-preflight**. +4. Leave production confirmation unchecked. +5. Select **Run workflow**. +6. Confirm the EC2 node is Online and preflight output reports the expected Compose services. + +This operation is read-only. It does not deploy or restart the application. + +### Run deploy + +Only after CI is green, OIDC check and SSM preflight pass, production configuration is validated and a fresh database backup is verified: + +1. Open **Actions** → **CloudMonitor CD — Deploy to AWS EC2 via SSM**. +2. Select **Run workflow**. +3. Choose main. +4. Choose operation **deploy**. +5. Check the confirmation input **Required for production deploy and rollback**. +6. Select **Run workflow** and complete any configured Environment approval. +7. Expand **Run SSM preflight, deployment, or rollback**. +8. Confirm the deployment SHA and health checks succeeded. + +Do not use deploy just to test connectivity; use oidc-check and ssm-preflight for that. + +### What deploy does + +The current script: +1. Checks the deployment directory and production .env. +2. Refuses to deploy if .env is Git-tracked. +3. Records the current Git SHA under /var/lib/cloudmonitor/previous-deploy-sha. +4. Fetches origin/main and checks out the remote main revision. +5. Validates production Compose configuration. +6. Pulls Prometheus, Node Exporter and Alertmanager images. +7. Builds backend and frontend images on EC2. +8. Runs docker compose up -d --remove-orphans. +9. Waits for frontend HTTP 200. +10. Waits for backend /health to report status=healthy. +11. Waits for the required Prometheus scrape pools to be up. +12. Records /var/lib/cloudmonitor/last-successful-deploy-sha. + +It does not use docker compose down -v. Never add that command to production deployment or recovery. + +--- + +## 8. One-time Git checkout bootstrap on the existing EC2 host + +### Why it was necessary + +The existing production directory had been prepared from an archive and did not contain a .git directory. The SSM workflow expects Git metadata to fetch and check out origin/main. + +This was a one-time repair for the existing host. A new server should be provisioned from a deliberate Git clone/deployment process instead of blindly repeating this procedure. + +### Safe sequence used + +1. Inspect the current production directory, .env, archives, SQL backup and Compose files. +2. Confirm .env exists and has restrictive permissions; mode 600 was verified. +3. Create and verify a PostgreSQL backup before deployment. +4. Initialize Git in the existing directory as the ubuntu user, not root. +5. Configure origin to the CloudMonitor GitHub repository. +6. Add local-only exclusions for backup artifacts where needed using .git/info/exclude. Keep the committed .gitignore as the normal project exclusion source. +7. Verify .env, .env.save, archives, SQL dumps and local database files are ignored and not staged. +8. Inspect staged filenames before creating a local baseline commit. +9. Create a local baseline commit/tag representing the existing deployed source. +10. Fetch origin/main without checking it out; compare baseline and candidate main. +11. Allow the guarded deployment workflow to perform checkout only after checks pass. + +The EC2 baseline/tag is a local recovery reference, not a GitHub backup or substitute for a SQL dump. Never push production secrets, backup files or the baseline archive to GitHub. + +The one-time bootstrap used a 69-file baseline after filename checks. That number is specific to the original host state, not a requirement for future releases. + +--- + +## 9. Production secrets and configuration + +Production environment file: **/home/ubuntu/cloud_monitor/.env** + +Rules: +- Keep it outside Git tracking. +- Keep permissions restrictive; mode 600 was verified. +- Never print the file or pass secret values into logs. +- Never overwrite it with .env.example or .env.production.example. +- Keep PostgreSQL connection settings consistent. +- Use a high-entropy random JWT signing secret. +- Validate candidate configuration before restarting services. + +During the first security-enabled deployment, the JWT secret was rotated to a cryptographically random 256-bit value without printing it. Existing JWTs signed with the previous secret became invalid and users had to authenticate again. Never record the new value in this guide. + +The candidate production config was validated using the running backend container's environment before the deployment. The validator accepted the configured JWT and PostgreSQL settings. + +--- + +## 10. Database backup, persistence and rollback + +### PostgreSQL volume + +The verified production named volume is **cloudmonitor-prod_postgres_data**. The first SSM deployment preserved this volume. + +Never run: +- docker compose down -v +- docker volume rm cloudmonitor-prod_postgres_data +- commands that recreate or delete the production data volume + +### Backup + +Before the first SSM deployment, a PostgreSQL custom-format dump was created on the EC2 host, protected with restrictive permissions and validated using pg_restore --list. + +A same-host backup is not an off-host disaster-recovery copy. Maintain a separate protected backup and periodically test restores. + +### Schema compatibility + +Before deployment, the database was checked: +- Application tables were present. +- Agent fields os_type and agent_version were present. +- Alembic revision was 003_os_and_version. + +The deployment workflow's code rollback is not a database rollback. It does not restore a SQL dump or reverse Alembic migrations. Keep schema changes backward-compatible where possible and take a separate backup before a schema-changing release. + +### Run rollback + +1. Open **Actions** → **CloudMonitor CD — Deploy to AWS EC2 via SSM** → **Run workflow**. +2. Choose main. +3. Choose operation **rollback**. +4. Check production confirmation. +5. Select **Run workflow**. +6. Inspect SSM output and verify frontend/backend health. + +The rollback script reads the previous SHA from /var/lib/cloudmonitor/previous-deploy-sha, checks that the commit exists, checks it out, rebuilds backend/frontend and starts Compose. It records the former current SHA for the next rollback. + +Rollback restores application source/build state only. It does not restore a database backup or undo schema/data changes. If schema incompatibility is suspected, stop and use a database-specific recovery plan. + +--- + +## 11. Completed production verification — 1 November 2026 + +The first controlled SSM deployment completed successfully. + +| Verification | Observed result | +|---|---| +| GitHub Actions SSM deployment | Successful | +| Deployed Git commit | f536dad97518a761c12b83ff578ff3194e9ed04f | +| Last-successful-deploy SHA | Matched deployed commit | +| Frontend | HTTP 200 | +| Backend /health | status=healthy | +| PostgreSQL | Healthy | +| Prometheus | Healthy | +| Alertmanager | Running | +| Node Exporter | Running | +| Prometheus required targets | alertmanager, cloudmonitor-backend, node-exporter, prometheus all up | +| PostgreSQL volume | cloudmonitor-prod_postgres_data, preserved | +| Alembic revision | 003_os_and_version | + +These checks verify the SSM delivery path and the application state observed immediately after deployment. They do not imply every broader production-security task is complete. + +### Post-deployment check + +Run via Systems Manager Run Command (AWS-RunShellScript), targeting only the CloudMonitor instance: + +~~~bash +set -eu +cd /home/ubuntu/cloud_monitor + +echo "=== DEPLOYED COMMIT ===" +sudo -u ubuntu -- git rev-parse HEAD + +echo "=== LAST SUCCESSFUL DEPLOY ===" +sudo cat /var/lib/cloudmonitor/last-successful-deploy-sha + +echo "=== COMPOSE SERVICES ===" +sudo docker compose -f docker-compose.prod.yml ps + +echo "=== FRONTEND ===" +curl -sS -o /dev/null -w "HTTP %{http_code}\\n" http://127.0.0.1/ + +echo "=== BACKEND ===" +curl -fsS http://127.0.0.1/health + +echo "=== PROMETHEUS TARGETS ===" +curl -fsS http://127.0.0.1/prom/api/v1/targets +~~~ + +Redact private identifiers from output. Never print .env. + +--- + +## 12. Troubleshooting + +### EC2 not Online in Fleet Manager + +Check: +1. CloudMonitorEC2SSMRole is attached as the EC2 instance profile. +2. The role includes AmazonSSMManagedInstanceCore. +3. SSM Agent is installed, enabled and running. +4. IMDS is reachable. +5. Outbound HTTPS to regional SSM endpoints works. +6. Re-run sudo ssm-cli get-diagnostics --output table. + +### OIDC role assumption fails + +Check: +1. Workflow job has permissions id-token: write. +2. Job uses environment production. +3. OIDC provider audience is sts.amazonaws.com. +4. The subject printed by Inspect OIDC claims exactly matches token.actions.githubusercontent.com:sub in the IAM trust policy. +5. The role ARN is in the correct Environment variable. +6. The assumed role belongs to the intended AWS account. + +Do not solve a subject mismatch by changing the trust subject to a wildcard. + +### SSM command fails + +Open the GitHub run and expand **Run SSM preflight, deployment, or rollback**. Capture the SSM command ID, then open Systems Manager → **Run Command** → command ID → instance invocation → **Output** and **Error**. + +Typical causes: +- **not a git repository:** the original deployment directory came from an archive; check Git initialization, checkout and remote. +- **set: Illegal option -o pipefail:** POSIX /bin/sh was used for a Bash script. The current workflow explicitly wraps the remote script with bash -lc. +- **confirm_production=true:** deploy/rollback were requested without the confirmation input. +- **EC2 not Online:** repair instance role, SSM Agent or outbound connectivity. +- **Permission denied:** inspect ownership/permissions; do not recursively change ownership of production data. + +### Production config validation fails + +Do not print secrets. Check that JWT_SECRET_KEY and DATABASE_URL exist, use valid production values and match the configured PostgreSQL user/password/database. The production validator rejects known development/placeholder values and insufficient JWT-secret length. + +### Deployment health check fails + +Inspect service state and logs without exposing credentials: + +~~~bash +sudo docker compose -f /home/ubuntu/cloud_monitor/docker-compose.prod.yml ps +sudo docker logs --tail 100 cloudmonitor-backend +sudo docker logs --tail 100 cloudmonitor-frontend +sudo docker logs --tail 100 cloudmonitor-prometheus +~~~ + +Check frontend HTTP status, backend /health, Prometheus target status, free disk and memory. If deployment failed after changing code, inspect whether automatic code rollback completed before initiating another rollback. + +--- + +## 13. Security work that remains outside Issue #9 + +The SSM migration is complete, but broader production security is not. + +| Item | Status after SSM migration | +|---|---| +| GitHub runner needs inbound SSH to deploy | No | +| EC2 SSM managed-node connectivity | Verified Online | +| GitHub OIDC role assumption | Verified | +| SSM deployment and application health checks | Verified | +| PostgreSQL volume preservation | Verified | +| HTTP-to-HTTPS/TLS | Outstanding; production was verified over HTTP | +| Unauthenticated Prometheus read access | Lifecycle actions were blocked; read access remains a separate hardening task | +| Agent Docker socket privilege hardening | Separate security task | +| SSM Session Manager as human break-glass access | Not established by deployment verification | +| Automatic push-to-main production deployment | Not enabled; CD remains manual-only | + +Track these items in the broader production-security issue and relevant follow-up issues. Do not describe CloudMonitor as fully security-hardened solely because SSM deployment works. + +--- + +## 14. Console navigation quick reference + +| Task | Console navigation | +|---|---| +| Create/inspect EC2 role | AWS Console → IAM → Roles | +| Attach EC2 role | EC2 → Instances → select CloudMonitor → Actions → Security → Modify IAM role | +| Check OIDC provider | IAM → Identity providers | +| Create GitHub deploy role | IAM → Roles → Create role → Web identity | +| Edit OIDC trust policy | IAM → Roles → CloudMonitorGitHubDeployRole → Trust relationships → Edit trust policy | +| Edit inline policy | IAM → Roles → CloudMonitorGitHubDeployRole → Permissions → CloudMonitorSSMDeployPolicy | +| Verify managed node | Systems Manager → Fleet Manager → Managed nodes | +| Run host-side command | Systems Manager → Run Command → AWS-RunShellScript | +| Configure GitHub variables | GitHub repository → Settings → Environments → production → Environment variables | +| Restrict deploy branch | GitHub repository → Settings → Environments → production → Deployment branches and tags | +| Run deployment | GitHub repository → Actions → CloudMonitor CD — Deploy to AWS EC2 via SSM → Run workflow | + +--- + +## 15. Related resources + +- [CloudMonitor repository](https://github.com/ayushcmd/cloud_monitor) +- [CI workflow](../.github/workflows/ci.yml) +- [SSM CD workflow](../.github/workflows/cd.yml) +- [Production Compose](../docker-compose.prod.yml) +- [Issue #9 — completed SSH-to-SSM migration](https://github.com/ayushcmd/cloud_monitor/issues/9) +- [Issue #12 — broader production security hardening](https://github.com/ayushcmd/cloud_monitor/issues/12) +- [AWS Systems Manager documentation](https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html) +- [AmazonSSMManagedInstanceCore policy](https://docs.aws.amazon.com/aws-managed-policy/latest/reference/AmazonSSMManagedInstanceCore.html) +- [GitHub Actions OIDC in AWS](https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services) +- [AWS Systems Manager service authorization reference](https://docs.aws.amazon.com/service-authorization/latest/reference/list_ssm.html) + +--- + +## 16. Completion record + +**Issue #9:** Closed as completed on 1 November 2026 after successful GitHub OIDC role assumption, SSM preflight, SSM production deployment, application health verification and PostgreSQL volume verification. + +**Verified deployment date:** 1 November 2026 +**Verified deployed commit:** f536dad97518a761c12b83ff578ff3194e9ed04f + +Review the live AWS Console, GitHub Environment, IAM trust policy and workflow file if they change. This guide documents the setup and observed state; it does not replace live configuration verification. From 0c561035f4882b8a03dbe0275bf93878ad9bd678 Mon Sep 17 00:00:00 2001 From: Ayush Raj Date: Sun, 1 Nov 2026 12:07:07 +0530 Subject: [PATCH 2/4] docs: mark SSH-to-SSM migration complete --- docs/SSM_DEPLOYMENT_MIGRATION.md | 263 ++++++++++++++----------------- 1 file changed, 115 insertions(+), 148 deletions(-) diff --git a/docs/SSM_DEPLOYMENT_MIGRATION.md b/docs/SSM_DEPLOYMENT_MIGRATION.md index c3e32ce..a95b74e 100644 --- a/docs/SSM_DEPLOYMENT_MIGRATION.md +++ b/docs/SSM_DEPLOYMENT_MIGRATION.md @@ -1,156 +1,123 @@ # Issue #9 — Migrate EC2 deployment from SSH to AWS Systems Manager -**Status:** SSM workflow prepared on a feature branch. The existing EC2 instance role is attached, SSM reports the instance Online, the GitHub OIDC provider and deployment role exist, and the repository production environment has the three AWS variables plus a main-only deployment rule. GitHub OIDC role assumption and Run Command execution still require workflow verification. +**Status:** Completed and verified on 1 November 2026 +**Repository:** ayushcmd/cloud_monitor +**Production region:** ap-southeast-2 (Sydney) +**Production deployment workflow:** CloudMonitor CD — Deploy to AWS EC2 via SSM -## Why the SSH deployment is failing +## Outcome -The current CD run fails at the SSH connectivity step with a TCP port 22 connection timeout. This is a network reachability failure before the deployment command starts; it is not evidence of a bad SSH key. +CloudMonitor's GitHub Actions deployment path has been migrated from runner-originated SSH to GitHub OIDC plus AWS Systems Manager Run Command. -The replacement workflow uses GitHub Actions OIDC to assume a narrowly scoped AWS role and Systems Manager Run Command to contact one EC2 instance. The workflow wraps each remote script with `bash -lc` so Bash-specific constructs are not accidentally interpreted by the default POSIX shell. The EC2 SSM Agent initiates outbound HTTPS connections. GitHub-hosted runners do not need inbound SSH access to the instance. AWS documents outbound HTTPS on port 443 to regional SSM endpoints; no inbound SSM port is required: https://docs.aws.amazon.com/systems-manager/latest/userguide/troubleshooting-ssm-agent.html +The first controlled production deployment completed successfully. GitHub Actions assumed the dedicated AWS deployment role, verified that the EC2 instance was Online in Systems Manager, invoked AWS-RunShellScript, deployed GitHub main and passed its frontend, backend and Prometheus health checks. -## Important rollout guard +### Verified production state -The updated `.github/workflows/cd.yml` is intentionally **workflow_dispatch-only** during migration. It does not deploy on a push or PR merge. This prevents merging the workflow from unexpectedly initiating production deployment before AWS prerequisites are in place. - -Available operations: - -1. `oidc-check` — requests a GitHub OIDC token, prints only issuer/audience/subject claims (never the token), assumes the configured AWS role with STS, and verifies the AWS account/caller identity. It does not send an SSM command or change EC2. -2. `ssm-preflight` — authenticates to AWS, verifies the configured EC2 instance is Online in SSM, and sends a read-only preflight command. -3. `deploy` — requires `confirm_production=true`; deploys `origin/main` through SSM and runs local frontend/backend/Prometheus health checks on EC2. -4. `rollback` — requires `confirm_production=true`; restores the previous Git revision recorded on EC2. - -After one successful controlled SSM deployment and operational review, a follow-up change can re-enable push-to-main for automatic CD. Do not add the push trigger before SSM is verified. - -## AWS setup checklist - -### 1. Confirm the existing EC2 SSM registration - -The CloudMonitor EC2 instance already has CloudMonitorEC2SSMRole with AmazonSSMManagedInstanceCore, and Fleet Manager reports the instance as Online. The GitHub role policy is restricted to the exact EC2 instance ARN, so an EC2 tag is not required for this implementation. The workflow polls for up to 30 minutes and sets a 35-minute GitHub job timeout to allow image build and health-check time. - -### 2. EC2 instance profile — completed - -The existing CloudMonitor EC2 instance has CloudMonitorEC2SSMRole attached with the AWS-managed AmazonSSMManagedInstanceCore policy. SSM Agent diagnostics passed, and the instance is listed as Online in Fleet Manager. No further EC2 role change is needed. - -### 3. SSM Agent and network connectivity — completed - -The Ubuntu 24.04 host has SSM Agent 3.3.4793.0 running and enabled. ssm-cli get-diagnostics --output table reported success for IMDS, SSM, EC2 Messages, SSM Messages, S3, KMS, Logs, Monitoring, AWS role credentials, and agent service checks. Fleet Manager subsequently showed the CloudMonitor instance as Online. No inbound SSH change is required for GitHub-hosted runners to use SSM. Keep administrator SSH access as a break-glass path until the new deployment route is fully tested. - -### 4. GitHub OIDC identity provider — already exists - -The AWS account already has the OpenID Connect provider token.actions.githubusercontent.com with audience sts.amazonaws.com. Reuse it; do not create a duplicate or delete the existing provider. - -### 5. GitHub deployment role trust policy — already created - -The role CloudMonitorGitHubDeployRole exists and trusts the GitHub OIDC provider for the immutable CloudMonitor repository identity and GitHub Environment production. The expected subject is repo:ayushcmd@252997256/cloud_monitor@1419373799:environment:production. The workflow's oidc-check now verifies the emitted subject and performs an actual AWS STS role assumption. If the STS step fails, compare the emitted subject with the role trust policy before changing either. - -Depending on the repository's OIDC subject-claim setting, the subject may use one of these forms: - -- Legacy: `repo:ayushcmd/cloud_monitor:environment:production` -- Immutable: `repo:ayushcmd@252997256/cloud_monitor@1419373799:environment:production` - -The environment's deployment branch rule allows main only. Do not allow repository-wide or branch-wide wildcards in the IAM trust policy. - -Example trust policy (replace the account ID and subject with the actual values): - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Federated": "arn:aws:iam:::oidc-provider/token.actions.githubusercontent.com" - }, - "Action": "sts:AssumeRoleWithWebIdentity", - "Condition": { - "StringEquals": { - "token.actions.githubusercontent.com:aud": "sts.amazonaws.com", - "token.actions.githubusercontent.com:sub": "" - } - } - } - ] -} -``` - -### 6. Attach the deployment role permissions - -The GitHub role is limited to AWS-RunShellScript on the exact CloudMonitor EC2 instance, plus the read operations needed to verify the command. The policy attached in AWS uses the actual account and instance identifiers. - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "DescribeManagedInstances", - "Effect": "Allow", - "Action": "ssm:DescribeInstanceInformation", - "Resource": "*" - }, - { - "Sid": "SendOnlyAwsRunShellScriptDocument", - "Effect": "Allow", - "Action": "ssm:SendCommand", - "Resource": "arn:aws:ssm:ap-southeast-2::document/AWS-RunShellScript" - }, - { - "Sid": "SendOnlyToCloudMonitorInstance", - "Effect": "Allow", - "Action": "ssm:SendCommand", - "Resource": "arn:aws:ec2:ap-southeast-2::instance/" - }, - { - "Sid": "ReadCommandInvocationResult", - "Effect": "Allow", - "Action": "ssm:GetCommandInvocation", - "Resource": "*" - } - ] -} -``` - -The read actions above use `Resource: "*"`; verify resource-level support and condition keys against the current Systems Manager service authorization reference: https://docs.aws.amazon.com/service-authorization/latest/reference/list_ssm.html - -### 7. Add GitHub Environment variables - -In GitHub → Settings → Environments → `production` → Environment variables, configure: - -| Variable | Value | -|---|---| -| `AWS_ACCOUNT_ID` | Your 12-digit AWS account ID | -| `AWS_DEPLOY_ROLE_ARN` | ARN of the GitHub OIDC deployment role | -| `EC2_INSTANCE_ID` | Existing CloudMonitor EC2 instance ID | - -The region is set in the workflow to `ap-southeast-2`. Consider adding required reviewers to the `production` GitHub Environment so deploy and rollback operations receive an approval gate. - -## Runbook after this PR is merged - -1. Open Actions → **CloudMonitor CD — Deploy to AWS EC2 via SSM** → Run workflow. -2. Select `oidc-check`. Confirm the emitted OIDC subject matches the IAM trust policy and that STS returns the expected AWS account and assumed role. -3. Run `ssm-preflight`. It must confirm the EC2 managed node is Online and complete the harmless read-only command. -5. Run `deploy` with `confirm_production=true` and complete any GitHub Environment approval. Verify the public application from outside EC2 and verify PostgreSQL data/persistence using a known non-sensitive application-level check. The workflow's health probes run against localhost on EC2. -6. Use `rollback` with explicit confirmation if rollback is required. -7. Only after the first successful SSM deployment and rollback procedure are reviewed should a follow-up PR restore automatic deployment on pushes to `main`. - -## Data and access safety - -- No SSH key or known-host secret is used by the SSM workflow. -- The workflow does not require inbound SSH from GitHub-hosted runners. -- It checks that `/home/ubuntu/cloud_monitor/.env` exists and is not Git-tracked; it never prints the file contents. -- It uses `docker compose config --quiet`, `build`, and `up -d --remove-orphans`. -- It does **not** use `docker compose down -v`, delete volumes, or recreate PostgreSQL storage. -- The deploy operation records the previous Git revision under `/var/lib/cloudmonitor/` and attempts a source/build rollback if deployment or local health checks fail. This is a **code/config rollback only**: it does not restore PostgreSQL data or reverse database schema migrations. Keep schema changes backward-compatible across rollback, and take/verify a separate database backup before any release that changes schema or data. -- Keep SSH access available for trusted administration until SSM is Online, the read-only preflight succeeds, and the required break-glass access method has been tested. -- No AWS resources are changed by merging this workflow/documentation PR. Actual AWS configuration and production deployment remain separate, deliberate steps. - -## Current acceptance status - -| Acceptance criterion | Status | +| Acceptance criterion | Result | |---|---| -| EC2 registered and Online in SSM | Complete — Fleet Manager reports Online | -| GitHub OIDC provider, role, and trust policy | Configured — live STS assumption pending `oidc-check` | -| GitHub invokes the approved SSM command | Pending `ssm-preflight` | -| Runner no longer requires inbound SSH | Implemented in the prepared workflow; pending live preflight | -| Production deployment and health checks succeed | Pending controlled deployment | -| PostgreSQL persistent storage is preserved | Workflow avoids volume removal/recreation; live verification pending | -| Least-privilege IAM and rollback procedure | Policy and runbook prepared; AWS review/verification pending | +| EC2 registered and Online in Systems Manager | Complete | +| GitHub OIDC provider and deployment role | Complete; live role assumption verified | +| GitHub Actions can invoke the approved SSM command | Complete | +| GitHub-hosted runner needs inbound SSH to deploy | No | +| Production deployment and health checks succeed | Complete | +| PostgreSQL data and named volume preserved | Complete | +| IAM trust and deployment permissions documented | Complete | +| Rollback and troubleshooting documented | Complete | +| HTTP application access | Verified | +| HTTPS/TLS | Tracked separately under broader production security work | + +Production verification recorded: +- Deployed commit: f536dad97518a761c12b83ff578ff3194e9ed04f +- Frontend: HTTP 200 +- Backend /health: status=healthy +- PostgreSQL: healthy +- Prometheus required targets: alertmanager, cloudmonitor-backend, node-exporter and prometheus all up +- PostgreSQL volume: cloudmonitor-prod_postgres_data, preserved +- Alembic revision checked before deployment: 003_os_and_version + +The successful deployment does not mean all production-security work is complete. HTTP/TLS hardening, unauthenticated Prometheus read access, agent Docker socket privilege hardening and human SSM Session Manager access remain separate work. + +## Architecture + +~~~text +GitHub Actions (production Environment) + | + | GitHub OIDC token + v +IAM role: CloudMonitorGitHubDeployRole + | + | short-lived AWS credentials + | ssm:SendCommand + v +AWS Systems Manager Run Command + | + v +CloudMonitor EC2 / SSM Agent + | + v +/home/ubuntu/cloud_monitor + | + v +Docker Compose production services +~~~ + +Two IAM roles are used for different purposes: + +- CloudMonitorEC2SSMRole — trusted by the EC2 service and attached to the instance profile. It includes AmazonSSMManagedInstanceCore. +- CloudMonitorGitHubDeployRole — trusts the GitHub OIDC provider and permits the production GitHub Actions job to invoke the approved SSM command on the specific CloudMonitor instance. + +The OIDC provider is token.actions.githubusercontent.com with audience sts.amazonaws.com. The trust policy is scoped to the CloudMonitor repository's production Environment subject. The exact subject must match the claims printed by the workflow's OIDC inspection step. + +## GitHub workflow behavior + +The workflow is .github/workflows/cd.yml. + +It is currently manual-only through workflow_dispatch. A push or merge to main runs CI but does not automatically deploy production. This is intentional: production deploy/rollback require an explicit operation selection and confirmation. + +| Operation | Purpose | Confirmation | +|---|---|---| +| oidc-check | Inspect OIDC claims and verify AWS STS role assumption | Not required | +| ssm-preflight | Verify the EC2 managed node is Online and run read-only preflight checks | Not required | +| deploy | Fetch main, build/update production services and run health checks | Required | +| rollback | Restore the previous recorded Git revision and rebuild application services | Required | + +The deployment script records the previous revision under /var/lib/cloudmonitor/previous-deploy-sha and the successful revision under /var/lib/cloudmonitor/last-successful-deploy-sha. It validates Compose configuration, pulls monitoring images, builds backend/frontend, runs docker compose up -d --remove-orphans and checks frontend, backend and required Prometheus targets. + +The workflow does not use docker compose down -v and does not delete/recreate the PostgreSQL volume. + +## Important first-deployment bootstrap note + +The existing production directory initially came from a deployment archive and did not contain Git metadata. The SSM workflow expects a Git checkout. + +For this one-time migration, the existing directory was inspected, sensitive files and backups were protected/excluded, a local baseline commit/tag was created, and origin/main was fetched and compared before the guarded deployment workflow checked out main. + +This local EC2 baseline is a recovery reference, not a GitHub backup. Production .env and SQL/archive backups must never be pushed to GitHub. + +## Operational safety + +- Keep production .env at /home/ubuntu/cloud_monitor/.env, outside Git tracking and with restrictive permissions. +- Never print or commit production secrets. +- Keep administrator SSH restricted as a break-glass path until a separate human access method is tested. GitHub-hosted runners no longer need inbound SSH to deploy. +- Take and verify a PostgreSQL backup before risky or schema-changing releases. +- The rollback operation is a source/build rollback, not a database rollback. It does not restore PostgreSQL data or reverse schema migrations. +- Never run docker compose down -v or remove cloudmonitor-prod_postgres_data. +- After deployment, verify the frontend, backend health, Prometheus targets, PostgreSQL health and existing data. + +## Detailed setup and click-by-click runbook + +See [AWS IAM, GitHub OIDC, SSM and Production Setup](AWS_IAM_OIDC_SSM_PRODUCTION_SETUP.md). It documents: +- EC2 instance role creation and attachment +- SSM Agent diagnostics and Fleet Manager registration +- GitHub OIDC provider verification +- GitHub deployment IAM role, trust relationship and inline policy +- GitHub production Environment variables and main-only branch restriction +- OIDC check, SSM preflight, deploy and rollback operations +- First-time Git checkout bootstrap, backup, production verification and troubleshooting + +## References + +- [GitHub Actions SSM workflow](../.github/workflows/cd.yml) +- [Production Compose](../docker-compose.prod.yml) +- [AWS Systems Manager documentation](https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html) +- [GitHub Actions OIDC in AWS](https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services) +- [AWS Systems Manager service authorization reference](https://docs.aws.amazon.com/service-authorization/latest/reference/list_ssm.html) From 8f433477d3b174c79f8d43e0bf30b2c7b7484b61 Mon Sep 17 00:00:00 2001 From: Ayush Raj Date: Sun, 1 Nov 2026 12:07:39 +0530 Subject: [PATCH 3/4] docs: update CI CD guide for SSM OIDC deployment --- docs/CICD_SETUP_GUIDE.md | 352 +++++++++++++++++++++------------------ 1 file changed, 188 insertions(+), 164 deletions(-) diff --git a/docs/CICD_SETUP_GUIDE.md b/docs/CICD_SETUP_GUIDE.md index ca949a0..8ce78d3 100644 --- a/docs/CICD_SETUP_GUIDE.md +++ b/docs/CICD_SETUP_GUIDE.md @@ -1,170 +1,194 @@ -# CloudMonitor — CI/CD Pipeline Setup & Operations Guide +# CloudMonitor — CI/CD Setup and Operations -This guide describes the automated continuous integration (CI) and continuous deployment (CD) pipeline configured for the **CloudMonitor** observability platform. +**Last verified:** 1 November 2026 +**Repository:** ayushcmd/cloud_monitor +**Production deployment:** GitHub Actions + AWS OIDC + Systems Manager Run Command +**AWS Region:** ap-southeast-2 ---- +This guide describes the CI and the current production CD workflow. The older SSH-based runner-to-EC2 deployment is not the active delivery path. -## 1. Pipeline Architecture & Workflow Files +## 1. Workflow files -The pipeline consists of two GitHub Actions workflow files located in `.github/workflows/`: - -``` +~~~text .github/ └── workflows/ - ├── ci.yml # Continuous Integration (Lint, Test, Build, Syntax Validation) - └── cd.yml # Continuous Deployment (Automated deployment to AWS EC2) -``` - -### A. CI Workflow (`ci.yml`) -- **Trigger**: Every `push` and `pull_request` targeting `main`, `master`, or `cloudmonitor-integration`, plus manual execution via `workflow_dispatch`. -- **Jobs**: - 1. `backend-test`: - - Sets up Python 3.12 with pip caching. - - Installs dependencies from `backend/requirements.txt` and `agent/requirements.txt`. - - Executes the Python test suite across both application endpoints and agent collector modules (`pytest tests/ agent/tests/`). - 2. `frontend-build`: - - Sets up Node.js 20 with npm caching. - - Runs `npm ci` for reproducible dependency installations. - - Builds the production React 18 / Vite bundle and validates that `dist/index.html` was generated. - 3. `docker-compose-validate`: - - Validates the syntax and structure of both `docker-compose.yml` (development) and `docker-compose.prod.yml` (production) via `docker compose config`. - -### B. CD Workflow (`cd.yml`) -- **Trigger**: Automatically on `push` to the `main` branch after CI tests pass, or manually via `workflow_dispatch`. -- **Concurrency**: Guarded with `group: production-deployment, cancel-in-progress: false` to prevent overlapping deployments. -- **Safety**: - - Never drops database volumes or runs `docker compose down -v`. - - Never touches or prints private keys or sensitive credentials. - - Deploys idempotently using `docker-compose.prod.yml up -d --remove-orphans`. - - Performs an automated health check against `/health` and `/prom/api/v1/targets` to verify service availability before completing. - ---- - -## 2. Required GitHub Repository Secrets - -To enable automated deployment to your AWS EC2 instance, configure the following secrets in GitHub: - -### A. Secret Configuration Scope -You can configure these secrets under **Repository secrets** (recommended for simplicity) or under a GitHub **Environment** named `production`: - -1. **Option 1: Repository Secrets (Recommended)**: - - Navigate to: **GitHub Repository → Settings → Secrets and variables → Actions → New repository secret** -2. **Option 2: Environment Secrets**: - - Navigate to: **GitHub Repository → Settings → Environments → New environment (Name: `production`) → Add Environment secret** - -> [!NOTE] -> The CD workflow defines `environment: production`. GitHub Actions automatically checks environment-level secrets first; if not present, it seamlessly falls back to repository-level secrets. - -### B. Required Secret Definitions - -| Secret Name | Description | Example / Value | -| :--- | :--- | :--- | -| `EC2_HOST` | Public IPv4 address or DNS of your EC2 instance | `3.25.150.231` | -| `EC2_USER` | SSH username for the Ubuntu instance | `ubuntu` | -| `EC2_SSH_PRIVATE_KEY` | Entire content of your private SSH key (`cloudmonitor-key.pem`) | `-----BEGIN RSA PRIVATE KEY----- ... -----END RSA PRIVATE KEY-----` | -| `EC2_SSH_KNOWN_HOSTS` | Verified public host key line for the EC2 server (enables strict SSH host verification) | `3.25.150.231 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEBsqrR6gAY2i/qk4kcm1Nycvq7I34j3nyUDEbM4uJpR` | - -> [!IMPORTANT] -> - **EC2_SSH_PRIVATE_KEY**: Paste the private key in full, including the `-----BEGIN ...-----` and `-----END ...-----` headers. The pipeline automatically strips any Windows CRLF line endings. -> - **EC2_SSH_KNOWN_HOSTS**: Unlike insecure configurations that set `StrictHostKeyChecking no`, our pipeline strictly validates the host key against `EC2_SSH_KNOWN_HOSTS` before sending credentials. This protects against Man-in-the-Middle (MITM) attacks. - -### How to Verify the EC2 Host Key Fingerprint -You can cryptographically verify that the host key matches your AWS EC2 instance before adding it: -1. Open the **AWS EC2 Management Console**. -2. Go to **Instances**, select instance `3.25.150.231`. -3. Click **Actions → Monitor and troubleshoot → Get system log**. -4. Scroll to the SSH key generation output generated at first boot: - ```text - ec2: -----BEGIN SSH HOST KEY FINGERPRINTS----- - ec2: 256 SHA256:YnxBgREbLPFkl1UYCgSj+AdjO2VDvgEC+HE5SYyutm8 root@ip-172-31-0-221 (ED25519) - ec2: 256 SHA256:eX9Qj+sNVfqjyE6p5cvICC3g1UjN8zHQ6NOsf6+ko7g root@ip-172-31-0-221 (ECDSA) - ec2: -----END SSH HOST KEY FINGERPRINTS----- - ``` -5. Confirm that the ED25519 fingerprint matches `SHA256:YnxBgREbLPFkl1UYCgSj+AdjO2VDvgEC+HE5SYyutm8`. - -### AWS Security Group Requirements -- For GitHub-hosted runners to execute deployments, the EC2 instance's Security Group must allow inbound TCP traffic on **Port 22 (SSH)**. -- Inbound source should be set to `0.0.0.0/0` (or GitHub Actions IP ranges from `https://api.github.com/meta`). - -### Production `.env` File Guard -- The CD pipeline requires that `/home/ubuntu/cloud_monitor/.env` already exists on the EC2 host. -- It will **never** automatically copy `.env.production.example` over `.env`, and will halt deployment immediately if `.env` is absent. This protects production secrets, database credentials, and avoids accidental data corruption. - ---- - -## 3. Preparation Checklist for First Automated Deployment - -Follow these sequential steps to ensure a smooth first deployment: - -1. [ ] **Verify Server Inbound Rules**: - Ensure AWS Security Group for `3.25.150.231` allows Port 22 (SSH) and Port 80 (HTTP). -2. [ ] **Verify Server Host Configuration**: - Ensure `/home/ubuntu/cloud_monitor/.env` exists on the EC2 instance with valid production secrets (`POSTGRES_PASSWORD`, `JWT_SECRET_KEY`, `DATABASE_URL`). -3. [ ] **Add Required Secrets in GitHub**: - In GitHub repository settings (`Settings -> Secrets and variables -> Actions`), add `EC2_HOST`, `EC2_USER`, `EC2_SSH_PRIVATE_KEY`, and `EC2_SSH_KNOWN_HOSTS`. -4. [ ] **Merge Fix Pull Request**: - Merge the audited remediation PR into `main`. -5. [ ] **Monitor Deployment Run**: - In GitHub Actions, monitor the CD run. It will validate secrets, establish an SSH session with strict verification, pull/build images, restart containers idempotently, and execute bounded health checks on Frontend, Backend, and Prometheus targets. - ---- - -## 4. How Changes Flow to Production - -```mermaid -flowchart LR - Dev["Local Branch (cloudmonitor-integration)"] -->|git push| GH["GitHub: cloudmonitor-integration"] - GH -->|Triggers CI| CI["GitHub Actions: ci.yml (Tests & Builds)"] - CI -->|Create PR| PR["Pull Request: cloudmonitor-integration -> main"] - PR -->|Merge to main| CD["GitHub Actions: cd.yml (Deploy to EC2)"] - CD -->|SSH & Compose Up| EC2["AWS EC2 Host: http://3.25.150.231"] -``` - -1. **Develop and test locally** on the integration branch (`cloudmonitor-integration`). -2. **Push to GitHub**: `git push origin cloudmonitor-integration`. -3. **CI verifies the commit**: Checks that all 24 backend/agent tests pass, the frontend builds cleanly, and Compose specifications are valid. -4. **Open a Pull Request**: Merge `cloudmonitor-integration` into `main`. -5. **CD deploys automatically**: GitHub Actions connects securely to EC2, updates the application services, reloads Nginx, and validates the health check endpoints. - ---- - -## 5. Operational Runbook - -### Inspecting Workflow Runs & Logs -1. Go to the **Actions** tab on your GitHub repository: `https://github.com/ayushcmd/cloud_monitor/actions`. -2. Select any workflow run to inspect step-by-step logs and execution times. -3. If a step fails, the log output displays the exact error message. - -### Diagnosing Common Pipeline Failures -- **Backend Test Failure**: - - Run tests locally or inside the backend container: `PYTHONPATH=. pytest tests/ agent/tests/`. - - Verify that mock objects or required test environment variables are properly defined. -- **Frontend Build Failure**: - - Run `npm run build` inside `frontend/`. - - Check for JSX syntax errors, unresolved imports, or missing dependencies in `package.json`. -- **SSH Connectivity Failure (`cd.yml`)**: - - Ensure the EC2 instance is running and has public IPv4 `3.25.150.231`. - - Check that the AWS Security Group allows inbound SSH (port 22) from GitHub Actions or the relevant CIDR. - - Verify that `EC2_SSH_PRIVATE_KEY` matches the authorized public key on the EC2 server (`~/.ssh/authorized_keys`). -- **Health Check Failure**: - - Inspect running container health on EC2: `ssh -i ubuntu@3.25.150.231 "sudo docker ps"`. - - Check backend logs: `sudo docker logs --tail 50 cloudmonitor-backend`. - -### Safe Rollback Procedure -If a bad commit was deployed and needs to be reverted: -1. **Revert the commit on GitHub**: - ```bash - git revert - git push origin main - ``` -2. **Manual Emergency Rollback on EC2**: - If GitHub Actions is unreachable, roll back directly on the host without data loss: - ```bash - ssh -i cloudmonitor-key.pem ubuntu@3.25.150.231 - cd /home/ubuntu/cloud_monitor - git checkout - sudo docker compose -f docker-compose.prod.yml build backend frontend - sudo docker compose -f docker-compose.prod.yml up -d - ``` - > [!CAUTION] - > Never run `docker compose down -v` on EC2. The `-v` flag deletes the named volume `cloudmonitor-prod_postgres_data`, which contains user accounts, enrolled agents, and telemetry records. + ├── ci.yml # Test, build and Compose validation + └── cd.yml # Manual SSM production operations +~~~ + +### CI workflow + +The CI workflow runs on pushes and pull requests for the configured branches, and can also be run manually. + +Jobs: +1. **Backend & Agent Test Suite** — Python 3.12, installs backend/agent requirements and runs pytest for tests/ and agent/tests/. +2. **Frontend Build & Asset Verification** — Node.js 20, npm ci, npm run build and verifies frontend/dist/index.html. +3. **Docker Compose Specification Validation** — validates docker-compose.yml and docker-compose.prod.yml. + +A green CI run verifies the tested commit against those jobs. Review workflow annotations as well as the final status; warnings may remain even when checks pass. + +### CD workflow + +The active workflow is **CloudMonitor CD — Deploy to AWS EC2 via SSM**. It is currently manual-only through workflow_dispatch. CI runs on push/PR, but merging or pushing to main does not itself deploy production. + +The production job uses: +- GitHub Environment: production +- GitHub job permissions: contents: read and id-token: write +- GitHub OIDC audience: sts.amazonaws.com +- AWS role: CloudMonitorGitHubDeployRole +- AWS Region: ap-southeast-2 +- SSM document: AWS-RunShellScript +- EC2 role: CloudMonitorEC2SSMRole +- Target: the configured CloudMonitor production EC2 instance + +No long-lived AWS access key is required by this OIDC flow. The current SSM workflow does not use SSH private-key or known-host secrets. + +## 2. GitHub Environment configuration + +Open repository **Settings** → **Environments** → **production**. + +### Environment variables + +| Name | Purpose | +|---|---| +| AWS_ACCOUNT_ID | Expected 12-digit AWS account | +| AWS_DEPLOY_ROLE_ARN | ARN of CloudMonitorGitHubDeployRole | +| EC2_INSTANCE_ID | CloudMonitor production EC2 instance ID | + +The region is configured in cd.yml as ap-southeast-2. + +### Deployment branch rule + +In **Deployment branches and tags**, use **Selected branches and tags** and allow main only. The IAM trust subject is scoped to the production Environment and must match the subject printed by the workflow's OIDC inspection step. + +### Protection rules + +Required reviewers and wait timers were not enabled in the verified setup; administrator bypass was allowed. For stronger release governance, consider adding independent required reviewers and disabling administrator bypass if supported by the repository plan/settings. + +Legacy SSH secrets may remain in the Environment from the previous deployment. The current SSM workflow does not reference them. Search every workflow and script before removing any legacy value. Never reveal private key contents. + +## 3. AWS IAM components + +Two IAM roles have different responsibilities: + +| Role | Trusted entity | Purpose | +|---|---|---| +| CloudMonitorEC2SSMRole | EC2 service | Attached to the production instance; includes AmazonSSMManagedInstanceCore | +| CloudMonitorGitHubDeployRole | GitHub OIDC provider | Assumed by GitHub Actions; permits the approved SSM operation on the CloudMonitor EC2 target | + +The OIDC provider is token.actions.githubusercontent.com with audience sts.amazonaws.com. The GitHub deployment role trust policy must match the exact production Environment subject from the workflow's OIDC claim inspection. Do not use a broad repository or organization wildcard. + +The inline policy CloudMonitorSSMDeployPolicy permits the required SSM describe, approved RunShellScript send-command and command-invocation read operations. The send-command resource scope is restricted to the AWS-RunShellScript document and the specific CloudMonitor instance. Do not attach AdministratorAccess. + +For the full click-by-click AWS Console procedure, example trust policy, policy JSON and instance-role attachment, see: + +**[AWS IAM, GitHub OIDC, SSM and Production Setup](AWS_IAM_OIDC_SSM_PRODUCTION_SETUP.md)** + +## 4. Run the SSM workflow + +Open repository **Actions** → **CloudMonitor CD — Deploy to AWS EC2 via SSM** → **Run workflow**. Select workflow source main. + +Available operations: + +| Operation | Use | Production confirmation | +|---|---|---| +| oidc-check | Validate OIDC claims and AWS STS role assumption | No | +| ssm-preflight | Confirm EC2 Online in SSM and inspect Docker/Compose state without deployment | No | +| deploy | Fetch main, build/update production services and run health checks | Required | +| rollback | Restore the previous recorded Git revision and rebuild application services | Required | + +### Recommended production release sequence + +1. Confirm the intended code is merged to main and CI is green. +2. Run oidc-check and confirm the issuer, audience, subject and assumed role. +3. Run ssm-preflight and confirm the CloudMonitor EC2 managed node is Online. +4. Verify production .env exists on EC2, is not Git-tracked and passes configuration validation. +5. Verify a recent PostgreSQL backup exists and can be read. +6. Check EC2 disk/memory headroom and production Compose configuration. +7. Run deploy and check the production confirmation input. +8. Watch the SSM command output inside the workflow. +9. Verify frontend, backend, Prometheus targets and PostgreSQL persistence independently. + +Do not run deploy to test connectivity. Use oidc-check and ssm-preflight for non-deploying checks. + +## 5. What deployment changes on EC2 + +The current deployment script: +1. Checks the production directory and .env file. +2. Refuses to continue if .env is tracked by Git. +3. Records the previous commit in /var/lib/cloudmonitor/previous-deploy-sha. +4. Fetches origin/main and checks out origin/main. +5. Validates production Compose configuration. +6. Pulls Prometheus, Node Exporter and Alertmanager images. +7. Builds backend and frontend images on the EC2 host. +8. Runs docker compose up -d --remove-orphans. +9. Waits for frontend HTTP 200. +10. Waits for backend /health to report status=healthy. +11. Waits for the required Prometheus targets to report up. +12. Records the successful revision in /var/lib/cloudmonitor/last-successful-deploy-sha. + +The script does not run docker compose down -v. It does not intentionally delete or recreate the PostgreSQL volume. + +## 6. Rollback + +Rollback is a deliberate production operation: +1. Open the SSM CD workflow and select **Run workflow**. +2. Choose main and operation rollback. +3. Check the production confirmation input. +4. Start the workflow and inspect SSM output. +5. Verify application health after rollback. + +The workflow restores the previous Git revision, rebuilds backend/frontend and runs Compose. This is a code/build rollback only. It does not restore PostgreSQL data, reverse Alembic migrations or revert arbitrary data changes. Take a database backup before schema-changing releases and keep schema changes backward-compatible where possible. + +## 7. Production safeguards + +- Keep /home/ubuntu/cloud_monitor/.env outside Git and with restrictive permissions. +- Never print or commit secrets. +- Do not replace production .env with an example file. +- Preserve named PostgreSQL volume cloudmonitor-prod_postgres_data. +- Never run docker compose down -v on production. +- Keep administrator access restricted; SSM deployment removes the runner's need for inbound SSH but does not itself establish human Session Manager access. +- Do not claim HTTPS is active solely because security group port 443 is open. TLS remains a separate security task. +- Production Prometheus read access remains a separate hardening task. + +## 8. Verified production deployment + +The first controlled SSM production deployment completed successfully on 1 November 2026. + +| Check | Verified result | +|---|---| +| Deployed commit | f536dad97518a761c12b83ff578ff3194e9ed04f | +| Last-successful-deploy SHA | Matched deployed commit | +| Frontend | HTTP 200 | +| Backend | Healthy | +| PostgreSQL | Healthy | +| Prometheus | Healthy | +| Alertmanager | Running | +| Node Exporter | Running | +| Prometheus target pools | alertmanager, cloudmonitor-backend, node-exporter and prometheus all up | +| PostgreSQL volume | cloudmonitor-prod_postgres_data preserved | + +The migration is tracked as complete in [Issue #9](https://github.com/ayushcmd/cloud_monitor/issues/9). Broader production-security work remains in [Issue #12](https://github.com/ayushcmd/cloud_monitor/issues/12). + +## 9. Troubleshooting quick reference + +| Symptom | Check | +|---|---| +| EC2 not Online in SSM | Attached EC2 instance profile, AmazonSSMManagedInstanceCore, SSM Agent, IMDS and outbound HTTPS | +| OIDC AccessDenied | Exact issuer/audience/subject, IAM trust JSON, production Environment and role ARN | +| SSM command fails | Open SSM command ID and inspect instance invocation Output and Error | +| not a git repository | Confirm /home/ubuntu/cloud_monitor contains .git and origin is correct | +| set: Illegal option -o pipefail | Run Bash scripts through bash -lc, not the default POSIX /bin/sh | +| confirm_production error | Select deploy/rollback and check the confirmation input | +| backend unhealthy | Inspect backend logs, configuration validation and database connectivity | +| Prometheus target down | Inspect target status, Docker service names, network and scrape configuration | +| database/rollback concern | Stop; preserve volume and backup. Code rollback does not restore or reverse database changes | + +## 10. Related documentation + +- [Detailed AWS IAM/OIDC/SSM setup](AWS_IAM_OIDC_SSM_PRODUCTION_SETUP.md) +- [SSM deployment migration record](SSM_DEPLOYMENT_MIGRATION.md) +- [Production AWS operations](AWS_DEPLOYMENT_AND_OPERATIONS.md) +- [GitHub Actions SSM workflow](../.github/workflows/cd.yml) +- [CI workflow](../.github/workflows/ci.yml) +- [Production Compose](../docker-compose.prod.yml) From f42d7aa21fc6f30234a1205fc5d0911bd9f32b97 Mon Sep 17 00:00:00 2001 From: Ayush Raj Date: Sun, 1 Nov 2026 12:07:56 +0530 Subject: [PATCH 4/4] docs: refresh AWS operations guide for SSM delivery --- docs/AWS_DEPLOYMENT_AND_OPERATIONS.md | 78 ++++++++++++++++++--------- 1 file changed, 52 insertions(+), 26 deletions(-) diff --git a/docs/AWS_DEPLOYMENT_AND_OPERATIONS.md b/docs/AWS_DEPLOYMENT_AND_OPERATIONS.md index 893ec03..5e85a9e 100644 --- a/docs/AWS_DEPLOYMENT_AND_OPERATIONS.md +++ b/docs/AWS_DEPLOYMENT_AND_OPERATIONS.md @@ -40,9 +40,9 @@ The current design runs Docker Compose on one EC2 host. PostgreSQL runs as a con | Optional TLS proxy | Caddy behind the tls Compose profile | | Database | PostgreSQL 16 Alpine | | Named database volume | cloudmonitor-prod_postgres_data | -| Deployment mechanism in workflow | GitHub Actions with SSH | -| Known pipeline issue | Recent GitHub-hosted runner-to-EC2 SSH connectivity checks failed | -| Planned alternative | AWS Systems Manager migration, tracked in [issue #9](https://github.com/ayushcmd/cloud_monitor/issues/9) | +| Deployment mechanism in workflow | GitHub Actions OIDC + AWS Systems Manager Run Command | +| Previous pipeline issue | GitHub-hosted runner-to-EC2 SSH connectivity timed out; replaced for deployment by SSM | +| Current deployment path | SSM migration completed and verified; see [Issue #9](https://github.com/ayushcmd/cloud_monitor/issues/9) | The public IP and region are project details supplied during deployment troubleshooting. Before acting, confirm the EC2 instance ID, current state, attached storage, security groups, DNS and active containers. Do not assume the current IP is an Elastic IP or that optional AWS components are active without checking the account. @@ -215,40 +215,66 @@ Node Exporter exposes host OS metrics over the internal Docker network. cAdvisor Grafana is configured in the local Compose stack, not in the current production Compose file. Alertmanager's routes and receiver names are defined, but webhook lists are empty. External email, chat, or webhook delivery should be considered unconfigured until a receiver is added and a test notification is observed. -## 9. CI/CD status and planned Systems Manager migration +## 9. CI/CD status — GitHub Actions with AWS Systems Manager ### CI workflow -The repository's .github/workflows/ci.yml includes: -- Python 3.12 setup and dependency installation -- pytest for backend and agent tests -- Node.js 20 and npm ci -- frontend production build and output verification -- validation of both local and production Compose configurations +The repository's .github/workflows/ci.yml runs on configured branch pushes and pull requests, with manual workflow_dispatch support. It has three jobs: -Check the specific GitHub Actions run before reporting that a commit passed CI. +- Backend & Agent Test Suite: Python 3.12, installs backend/agent dependencies and runs pytest for tests/ and agent/tests/. +- Frontend Build & Asset Verification: Node.js 20, npm ci, npm run build and verifies frontend/dist/index.html. +- Docker Compose Specification Validation: validates development and production Compose files. -### Current SSH-based CD +Review the specific workflow run and annotations before reporting a commit as verified. -The .github/workflows/cd.yml workflow defines SSH deployment from a GitHub-hosted runner to EC2, followed by Compose deployment and health/telemetry checks. It requires configured GitHub secrets and an existing production .env on EC2. +### Active production CD workflow -Recent debugging found TCP port 22 from GitHub-hosted runners was unreachable because the active runner egress address was not allowed by the EC2 security group. That makes the current SSH-based deployment an unreliable, unresolved workflow. CI/CD migration is currently deferred. +The active deployment path is **CloudMonitor CD — Deploy to AWS EC2 via SSM** in .github/workflows/cd.yml. -### Planned migration to AWS Systems Manager +It uses GitHub Actions OIDC to assume CloudMonitorGitHubDeployRole and AWS Systems Manager Run Command to reach the CloudMonitor EC2 instance. The EC2 instance has CloudMonitorEC2SSMRole with AmazonSSMManagedInstanceCore. GitHub-hosted runners do not need inbound SSH access to deploy. -GitHub issue #9 tracks replacing runner-originated SSH with Systems Manager Run Command. Recommended staged approach: +The workflow is manual-only via workflow_dispatch. Pushes/merges to main run CI but do not automatically deploy production. Production deploy/rollback require an explicit operation and confirmation. -1. Read the EC2 instance ID, IAM instance profile, and SSM Agent status. -2. Confirm EC2 is online as a Systems Manager managed node. -3. Configure a least-privilege instance role and required outbound access to SSM endpoints. -4. Test an innocuous read-only SSM command. -5. Configure GitHub-to-AWS authentication with OIDC and narrowly scoped permissions. -6. Update CD to invoke only the approved deployment operation through SSM. -7. Test in a controlled run and verify the web app, backend health, Prometheus targets and PostgreSQL persistence. -8. Keep existing administrator access until SSM deployment is confirmed. -9. Only then remove the deployment's dependency on runner-originated inbound SSH. +Operations: +1. oidc-check — inspect OIDC issuer/audience/subject, assume the AWS role and verify AWS caller identity. +2. ssm-preflight — verify EC2 is Online in SSM and run read-only Docker/Compose checks. +3. deploy — fetch origin/main, build backend/frontend, update Compose services and verify frontend/backend/Prometheus health. +4. rollback — restore the previous recorded Git revision and rebuild backend/frontend. -Do not change the security group or IAM permissions until the staged design has been reviewed. +### AWS and GitHub configuration + +The OIDC provider is token.actions.githubusercontent.com with audience sts.amazonaws.com. The deployment role trust policy is scoped to the CloudMonitor repository's production Environment subject. The role's inline policy is scoped to the approved AWS-RunShellScript document and the CloudMonitor EC2 target, plus the command/instance read permissions needed by the workflow. + +The GitHub production Environment has these variables: +- AWS_ACCOUNT_ID +- AWS_DEPLOY_ROLE_ARN +- EC2_INSTANCE_ID + +The production Environment deployment branch rule allows main. Legacy SSH secrets may still exist in the Environment; the current SSM workflow does not use them. Search all workflows/scripts before removing them. + +### First deployment and current state + +The original production directory was archive-based and did not contain .git metadata. A one-time, locally protected Git baseline was created on EC2 after sensitive files/backups were excluded and checked. The SSM workflow then fetched origin/main and deployed the repository revision. + +The first controlled SSM deployment succeeded on 1 November 2026. Verified commit: f536dad97518a761c12b83ff578ff3194e9ed04f. Frontend returned HTTP 200, backend health reported healthy, PostgreSQL remained healthy, Prometheus required targets were up, and the named database volume cloudmonitor-prod_postgres_data was preserved. + +### Production safety + +- Keep /home/ubuntu/cloud_monitor/.env outside Git and with restrictive permissions. +- Never print secrets or replace the production .env with an example file. +- The workflow does not use docker compose down -v. +- The deploy step records previous and successful Git SHAs under /var/lib/cloudmonitor/. +- The rollback path is a source/build rollback only. It does not restore PostgreSQL data or reverse schema migrations. +- Keep administrator SSH restricted as a break-glass path until a separate human SSM access method is tested. SSM deployment only removes the GitHub runner's need for inbound SSH. +- Do not claim HTTPS is active solely because port 443 is allowed. TLS remains a separate security task. + +### Detailed setup + +See [AWS IAM, GitHub OIDC, SSM and Production Setup](AWS_IAM_OIDC_SSM_PRODUCTION_SETUP.md) for the button-by-button IAM, OIDC provider, role trust, inline policy, GitHub Environment, workflow operation, first-host bootstrap, verification, rollback and troubleshooting procedure. + +See [SSM Deployment Migration](SSM_DEPLOYMENT_MIGRATION.md) for the completion record for Issue #9. + +M permissions until the staged design has been reviewed. ## 10. Safe deployment checklist