webapp: add per scale set job queue view

[benoit-nexthop]

What this does

When all runners of a scale set are busy, it is hard to tell where a given workflow job sits in the queue, or why capacity is not being used. This adds a Job Queue view to the web UI that groups queued and running jobs per scale set (and per pool, matched by labels), ordered by request time, with links to the GitHub run/job pages and to the scale set/pool detail pages. The view updates live via the existing job/instance websocket events.

Supporting changes

• Jobs recorded by scale set listeners now carry the garm scale set ID (new WorkflowJob.ScaleSetFkID column, auto-migrated; exposed as scale_set_id on the Job API model). Previously the scale set ID was deliberately dropped when recording jobs, so queued jobs could not be attributed to a scale set. Existing rows are backfilled on the next message for that job.
• The full RunnerScaleSetStatistic from each session message is persisted on the scale set and exposed as statistics on the API (previously only TotalAssignedJobs survived, as desired_runner_count). The view shows GitHub's numbers (assigned jobs, busy/idle runners) next to GARM's instance counts. This makes two things visible that were previously invisible: jobs GitHub holds while a scale set is saturated (GitHub only delivers individual job messages on assignment), and runners GitHub considers offline while GARM thinks they are idle.
• consolidateRunnerState syncs GitHub's per-runner view (online idle/busy, offline) onto instances that finished installing. Without this, a runner whose agent died after setup stayed "idle" in GARM forever, while GitHub considered it offline and never assigned it jobs — and it counted toward the runner count, starving scale-up. We caught a real production incident (a fleet of registered-but-offline runners with a growing job backlog) within minutes thanks to this divergence being visible in the UI.

Screenshot

Testing

• go test -tags testing ./database/... ./workers/... passes.
• New vitest integration tests for the queue page (grouping, queue ordering, pool label matching, GitHub links, statistics badges).
• Running in our production deployment (org-wide scale sets on a private CloudStack cloud plus a Kubernetes provider).

[gabriel-samfira]

This looks like an amazing UX improvement!

I will allocate some time to properly review, hopefully next week. Apologies for the delay!

[gabriel-samfira]

Just a few comments.

[gabriel-samfira]

I think this needs a:

constraint:OnDelete:SET NULL

Otherwise we'd get a foreign key constraint error when trying to remove a scaleset if a job is associated with it. We have guards to not allow a scaleset to be removed if it has a runner, but a runner may not be spun up instantly as a job is recorded.

[gabriel-samfira]

hmm. Did you use:

make generate

to update this file, or was it manually edited? This file is meant to be generated from the swagger definitions.

Have a look here:

https://openapi-generator.tech/docs/installation/

and here:

https://github.com/cloudbase/garm/blob/main/webapp/DEV_SETUP.md

[gabriel-samfira]

This will not trigger on currentTime. Bellow in this file you set up a ticker in the onMount() function. That ticker updates currentTime every 5 seconds, but in the template bellow, waitingFor() only accepts a job. So unless a job is refreshed, waitingFor() is never fired.

Is that intended? If so, do we still need a ticker for currentTime?

If it's not intended, the waitingFor() function needs to accept a now: number parameter, and we can then call it like:

waitingFor(job, currentTime)

[gabriel-samfira]

If we want this to trigger every 5 seconds, we need to change this to:

{waitingFor(job, currentTime)}

See above comments.

[gabriel-samfira]

I may be missing it, but is inProgress ever rendered?

[gabriel-samfira]

If we ever run into a corrupted stat, listing scalesets will fail. This should probably be logged as an error, but not bail here. See:

garm/database/sql/util.go

Lines 81 to 88 in 2edcc2f

	if len(instance.Capabilities) > 0 {
	var caps params.AgentCapabilities
	if err := json.Unmarshal(instance.Capabilities, &caps); err == nil {
	ret.Capabilities = caps
	} else {
	slog.ErrorContext(s.ctx, "failed to unmarshal capabilities", "instance_name", instance.Name, "error", err)
	}
	}

as a reference.

[gabriel-samfira]

This might have the potential to overload the DB. If the jobs table stays small, this should be fine, but if we ever end up with many jobs, it might impact performance. Worth keeping track of.

[gabriel-samfira]

The valid instance states are:

https://github.com/cloudbase/garm-provider-common/blob/4f5d9bf8094976b96c9e8db1e4f4155afd846ba9/params/params.go#L43-L54

The state pending is not a valid state.

[gabriel-samfira]

The model changes will require a migration to be added. We switched to gormigrate to do away with the custom code in sql.go (and the convoluted conditionals we used to have).

A file in database/sql/migrations/0007_scaleset_job_queue.go with something like:

package migrations

import (
	"github.com/go-gormigrate/gormigrate/v2"
	"gorm.io/datatypes"
	"gorm.io/gorm"
)

// scaleSet0007 adds the cached RunnerScaleSetStatistic received from
// GitHub on the message session.
// This is just a stub, AutoMigrate ignores everything else.
type scaleSet0007 struct {
	RunnerStatistics datatypes.JSON
}

func (scaleSet0007) TableName() string {
	return "scale_sets"
}

// workflowJob0007 adds the scale set attribution for jobs recorded by
// scale set listeners.
type workflowJob0007 struct {
	ScaleSetFkID *uint `gorm:"index"`
}

func (workflowJob0007) TableName() string { return "workflow_jobs" }

func init() {
	Register(&gormigrate.Migration{
		ID: "0007_scaleset_job_queue",
		Migrate: func(tx *gorm.DB) error {
			if err := tx.AutoMigrate(&scaleSet0007{}); err != nil {
				return err
			}
			return tx.AutoMigrate(&workflowJob0007{})
		},
	})
}

At the end, your migrations table should look like this:

sqlite> select * from migrations;
SCHEMA_INIT
0001_baseline
0002_case_insensitive_indexes
0002_lower_indexes
0003_forge_instances
0004_garm_agent_version
0005_allow_insecure_garm_agent
0006_proxies
0007_scaleset_job_queue
sqlite>