Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 92 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ on:
branches: [main, master]
pull_request:

# A newer push to the same branch cancels the in-flight run.
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true

jobs:
build-test:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -34,8 +39,28 @@ jobs:

# Redis is provided above; tests that need Postgres skip automatically
# unless FORGEQUEUE_PG_DSN is set.
- name: Test
run: go test ./... -count=1
- name: Test (race)
run: go test ./... -race -count=1

- name: Vulnerability scan
run: |
go install golang.org/x/vuln/cmd/govulncheck@latest
govulncheck ./...

lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version-file: go.mod
cache: true
# only-new-issues keeps the gate useful without failing on pre-existing
# style debt; tighten by removing it once the tree is clean.
- uses: golangci/golangci-lint-action@v6
with:
version: latest
only-new-issues: true

images:
runs-on: ubuntu-latest
Expand All @@ -62,5 +87,70 @@ jobs:
context: .
file: ${{ matrix.dockerfile }}
push: false
load: true
tags: forgequeue/${{ matrix.name }}:ci
cache-from: type=gha
cache-to: type=gha,mode=max

- name: Scan ${{ matrix.name }} image
uses: aquasecurity/trivy-action@0.28.0
with:
image-ref: forgequeue/${{ matrix.name }}:ci
format: table
severity: HIGH,CRITICAL
ignore-unfixed: true
exit-code: "1"

e2e:
runs-on: ubuntu-latest
needs: images
steps:
- uses: actions/checkout@v4

- uses: actions/setup-go@v5
with:
go-version-file: go.mod
cache: true

- name: Build images
run: |
for svc in apiserver gateway worker web; do
docker build -f "dockerfiles/${svc}.Dockerfile" -t "forgequeue/${svc}:latest" .
done

- name: Create kind cluster
uses: helm/kind-action@v1
with:
cluster_name: forgequeue
config: deploy/k8s/kind-config.yaml

- name: Load images
run: |
kind load docker-image --name forgequeue \
forgequeue/apiserver:latest forgequeue/gateway:latest \
forgequeue/worker:latest forgequeue/web:latest

- name: Deploy
run: kubectl apply -k deploy/k8s/overlays/kind

- name: Wait for rollouts
run: |
kubectl -n forgequeue rollout status statefulset/redis --timeout=180s
kubectl -n forgequeue rollout status statefulset/postgres --timeout=180s
kubectl -n forgequeue rollout status deployment/gateway --timeout=180s
kubectl -n forgequeue rollout status deployment/apiserver --timeout=180s
kubectl -n forgequeue rollout status deployment/worker --timeout=180s
kubectl -n forgequeue rollout status deployment/web --timeout=180s

- name: Smoke test the API
run: |
kubectl -n forgequeue port-forward svc/apiserver 8080:8080 &
PF_PID=$!
trap 'kill $PF_PID' EXIT
for i in $(seq 1 30); do
if curl -fsS http://localhost:8080/healthz; then exit 0; fi
sleep 2
done
echo "apiserver never became healthy" >&2
kubectl -n forgequeue get pods
exit 1
90 changes: 90 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
name: release

# Tag a commit (e.g. v1.0.0) to cut a release. Builds the four images, pushes
# them to GHCR, attaches SBOM + provenance attestations, and signs each image
# with cosign (keyless, via the workflow's OIDC identity).
on:
push:
tags:
- "v*"

env:
REGISTRY: ghcr.io
IMAGE_PREFIX: ${{ github.repository_owner }}/forgequeue

jobs:
publish:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
id-token: write # cosign keyless signing
strategy:
fail-fast: false
matrix:
include:
- name: apiserver
dockerfile: dockerfiles/apiserver.Dockerfile
- name: gateway
dockerfile: dockerfiles/gateway.Dockerfile
- name: worker
dockerfile: dockerfiles/worker.Dockerfile
- name: web
dockerfile: dockerfiles/web.Dockerfile
steps:
- uses: actions/checkout@v4

- uses: docker/setup-buildx-action@v3

- name: Log in to GHCR
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Docker metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}-${{ matrix.name }}
tags: |
type=ref,event=tag
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}

- name: Build and push
id: build
uses: docker/build-push-action@v6
with:
context: .
file: ${{ matrix.dockerfile }}
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
build-args: |
VERSION=${{ github.ref_name }}
COMMIT=${{ github.sha }}
BUILD_DATE=${{ github.event.head_commit.timestamp }}
VITE_APP_VERSION=${{ github.ref_name }}
# SLSA provenance + an SPDX SBOM, stored as in-registry attestations.
provenance: true
sbom: true
cache-from: type=gha
cache-to: type=gha,mode=max

- name: Install cosign
uses: sigstore/cosign-installer@v3

- name: Sign the image (keyless)
env:
DIGEST: ${{ steps.build.outputs.digest }}
IMAGE: ${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}-${{ matrix.name }}
run: cosign sign --yes "${IMAGE}@${DIGEST}"

- name: Record digest
run: |
echo "### ${{ matrix.name }}" >> "$GITHUB_STEP_SUMMARY"
echo '```' >> "$GITHUB_STEP_SUMMARY"
echo "${{ env.REGISTRY }}/${{ env.IMAGE_PREFIX }}-${{ matrix.name }}@${{ steps.build.outputs.digest }}" >> "$GITHUB_STEP_SUMMARY"
echo '```' >> "$GITHUB_STEP_SUMMARY"
93 changes: 84 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,6 @@ ForgeQueue is a Redis-backed task queue written in Go. It implements durable,
at-least-once task execution with leases, heartbeats, crash recovery, and
ownership fencing from first principles.

> **What this is.** A learning / portfolio project that implements a task queue
> the way a real one works: atomic Lua state transitions, lease ownership,
> heartbeats, recovery, retries and a dead-letter queue. It is deliberately
> small and readable rather than production-hardened. It is **not** a drop-in
> replacement for Redis-backed job systems like Asynq, Celery, BullMQ or
> Sidekiq — see [Limitations](#limitations) for the honest list of what it does
> not do.

Key features:
- At-least-once execution with lease-based ownership and fencing
- Automatic retries with exponential backoff
Expand Down Expand Up @@ -137,6 +129,27 @@ be the accurate signal. Metrics are scraped via the `ServiceMonitor` in
kube-prometheus-stack (the `monitoring.coreos.com` CRDs must exist). The Ingress
targets the `forgequeue.local` host and needs an ingress-nginx controller.

Launch hardening baked into the manifests:

- **TLS at the edge.** The Ingress terminates TLS (cert-manager
`letsencrypt-prod` by default) and redirects HTTP to HTTPS; the console and
`/api` share one origin, so the API needs no CORS grant in production.
- **Redis AUTH.** The Redis StatefulSet runs with `requirepass` sourced from
`forgequeue-secrets` (`FORGEQUEUE_REDIS_PASSWORD`); every service reads it
through `redisutil`. `maxmemory-policy noeviction` bounds memory without
silently dropping queued tasks.
- **Alerting.** `observability/prometheusrule.yaml` ships alerts for backlog
growth, dead-lettering, lease losses, a down scrape target, failed backups
and filling persistent volumes.
- **Queue-depth autoscaling.** `observability/keda-scaledobject.yaml` scales
workers on the ready-list length (KEDA); it supersedes the CPU HPA.
- **Off-cluster backups.** `base/backup.yaml` adds a daily `pg_dump` to object
storage in addition to the on-PVC copy, and no-ops when unconfigured.

Images are built and signed (SBOM + provenance + cosign) by
`.github/workflows/release.yml` on a `v*` tag. Pin the printed digests in
`overlays/prod` before deploying.

## Architecture

ForgeQueue consists of two main components that communicate through Redis:
Expand Down Expand Up @@ -279,6 +292,16 @@ The batch's heartbeats are only stopped **after** the ACK/requeue is committed.
| Setting | Location | Default |
| --- | --- | --- |
| Redis address | `FORGEQUEUE_REDIS_ADDR` | `localhost:6379` |
| Redis password (AUTH) | `FORGEQUEUE_REDIS_PASSWORD` | none |
| Redis database index | `FORGEQUEUE_REDIS_DB` | `0` |
| Redis TLS | `FORGEQUEUE_REDIS_TLS` | off |
| Redis TLS verify skip (dev) | `FORGEQUEUE_REDIS_TLS_INSECURE` | off |
| Redis TLS CA bundle | `FORGEQUEUE_REDIS_TLS_CA` | system roots |
| gRPC TLS cert / key | `FORGEQUEUE_GRPC_TLS_CERT` / `_KEY` | off (plaintext) |
| gRPC TLS client CA (mTLS) | `FORGEQUEUE_GRPC_TLS_CLIENT_CA` | off |
| API CORS origins (comma list) | `FORGEQUEUE_CORS_ORIGINS` | same-origin only |
| API max request body | `FORGEQUEUE_MAX_BODY_BYTES` | 1 MiB |
| Postgres sslmode (DSN) | `FORGEQUEUE_PG_SSLMODE` | `disable` |
| Gateway gRPC listen | `gateway.Start()` (hardcoded) | `:50051` |
| Gateway metrics address | `FORGEQUEUE_METRICS_ADDR` | disabled |
| HTTP API listen | `FORGEQUEUE_API_ADDR` | `:8080` |
Expand Down Expand Up @@ -310,6 +333,52 @@ The batch's heartbeats are only stopped **after** the ACK/requeue is committed.
| Backoff base | `wp.backoffBase` (unexported) | 1s |
| Backoff cap | `wp.backoffCap` (unexported) | 60s |

## Observability

The gateway and worker each expose Prometheus metrics at `/metrics` on
`FORGEQUEUE_METRICS_ADDR` (disabled by default). They are scraped as two
separate jobs so "accepted" (gateway) and "executed" (worker) stay
distinguishable:

- `forgequeue_tasks_enqueued_total` — tasks the gateway accepted
- `forgequeue_tasks_processed_total` — tasks the worker ACKed
- `forgequeue_tasks_retried_total` / `forgequeue_tasks_dead_lettered_total`
- `forgequeue_queue_ready / inflight / scheduled / dead` — queue depth by state, sampled from Redis every 5s (O(1) per scrape)
- `forgequeue_task_duration_seconds` — handler duration histogram (p50/p95/p99)
- `forgequeue_redis_claim/ack/heartbeat_seconds` — Redis round-trip histograms
- `forgequeue_lease_losses_total` — fenced lease losses

Run the whole telemetry stack locally with `deploy/observability.sh` (needs
`prometheus` on PATH; Grafana is optional and only started if installed):

```bash
./deploy/observability.sh
# in another terminal, generate some load:
go run ./cmd/demoenqueue
```

That prints the URLs: gateway `:9090/metrics`, worker `:9091/metrics`,
Prometheus UI on `:9095`, and Grafana on `:3000` (no login) when available.
Logs go to `/tmp/fq-*.log`.

### Kubernetes

Under `deploy/k8s/observability` (applied separately after kube-prometheus-stack, since its CRDs must exist):

- `servicemonitor.yaml` — scrapes the `worker` and `gateway` `metrics` ports every 15s
- `prometheusrule.yaml` — alerts for backlog growth, dead-lettering, lease losses, a down scrape target, failed backups and filling persistent volumes
- `keda-scaledobject.yaml` — scales workers on ready-list length; supersedes the CPU HPA in `base/`

### Grafana

Dashboards live under `deploy/grafana`:

- `dashboards/forgequeue.json` — "ForgeQueue Overview": enqueued/processed/retried/dead-lettered stats, queue depth by state, in-flight vs ready, handler duration p50/p95/p99, Redis round trips p95, lease losses, and enqueued vs processed throughput
- `provisioning/datasources/prometheus.yml` — provisions the Prometheus datasource (`uid: forgequeue`, `http://localhost:9095`) so the dashboard wires itself up on import
- `provisioning/dashboards/forgequeue.yml` — file provider that loads every dashboard JSON in `dashboards/`

The dashboard assumes the `forgequeue` Prometheus datasource UID matches what the provisioning file registers.

## Running the whole stack

```bash
Expand Down Expand Up @@ -426,7 +495,13 @@ Real handlers dominate throughput in practice, so treat these as an upper bound.

- **No exactly-once.** Delivery is at-least-once: a lease can expire (worker crash, Redis blip, GC pause) and a task can run twice. Handlers must be idempotent.
- **Single Redis.** No replication, sharding, or failover logic — durability is whatever the Redis deployment provides. The bundled Kubernetes manifest runs one AOF-backed Redis on a PersistentVolumeClaim, but that is still a single point of failure.
- **Partial auth.** The HTTP API has JWT + RBAC, with optional OIDC/SSO login. The **gRPC gateway and Redis have no authentication or TLS**, and the HTTP API itself is plain HTTP. Do not expose any of it to a public network.
- **Partial auth.** The HTTP API and the gRPC gateway authenticate with JWT
bearer tokens (RBAC on the API, per-method permissions on gRPC), with
optional OIDC/SSO login. Transport TLS is available but **off by default**:
enable it on gRPC with the `FORGEQUEUE_GRPC_TLS_*` vars and terminate HTTP
TLS at the ingress (see the Kubernetes section). Redis supports
`requirepass` and TLS through its env vars. On a trusted pod network the
defaults are plaintext; do not expose any port publicly without enabling TLS.
- **No multi-tenancy / queue isolation.** "Queues" are a single logical namespace, not separate Redis keyspaces or broker connections.
- **Completed bodies are deleted**, so a finished job cannot be looked up by ID afterwards (see the HTTP API note above).
- **Idle polling** uses a short sleep interval per worker; blocking variants (e.g. `BLMOVE`) are not implemented.
Expand Down
1 change: 0 additions & 1 deletion cmd/gateway/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,5 @@ import (
)

func main() {
// Call the execution function inside your internal package
gateway.Start()
}
Loading
Loading