Skip to content

feat(wrt): auto-inject KEDA Temporal trigger metadata for per-version ScaledObjects#351

Open
nikoul wants to merge 4 commits into
temporalio:mainfrom
nikoul:nicolas/wrt-keda-temporal-trigger-injection
Open

feat(wrt): auto-inject KEDA Temporal trigger metadata for per-version ScaledObjects#351
nikoul wants to merge 4 commits into
temporalio:mainfrom
nikoul:nicolas/wrt-keda-temporal-trigger-injection

Conversation

@nikoul
Copy link
Copy Markdown

@nikoul nikoul commented May 26, 2026

Summary

Extends WorkerResourceTemplate auto-injection to populate the workerDeploymentName and workerDeploymentBuildId fields in KEDA ScaledObject triggers[*].metadata when the trigger type is "temporal". This unblocks using KEDA's Temporal scaler (kedacore/keda#7672) as the templated resource for per-version backlog scaling, as requested in #286.

Closes (or contributes towards) #286.

Motivation

For users who already run KEDA cluster-wide for non-Temporal workloads, KEDA owns the external.metrics.k8s.io APIService — only one external metrics provider can be installed per cluster. That makes the documented WRT-HPA + prometheus-adapter path unavailable: prometheus-adapter cannot co-exist with KEDA on the same APIService.

KEDA's recent Worker Deployment Versioning support (kedacore/keda#7672) makes it possible for a KEDA ScaledObject to scale a single worker version by querying Temporal Cloud directly with workerDeploymentName + workerDeploymentBuildId. WRT already supports templating arbitrary kinds (verified: spec.template is unstructured, SSA-applied, with a kind allow-list in the chart) and scaleTargetRef: {} auto-fill works recursively. The only missing piece was injecting the per-version identifiers into the KEDA trigger metadata — same source values the controller already injects into HPA External metric selectors, different JSON path.

Behaviour

For each templated resource that contains:

spec:
  triggers:
    - type: temporal
      metadata:
        workerDeploymentName: ""        # placeholder, any string value
        workerDeploymentBuildId: ""     # placeholder, any string value
        # ...other user-provided KEDA Temporal scaler metadata

the controller sets:

  • workerDeploymentNamewrt.Spec.EffectiveWorkerDeploymentName()
  • workerDeploymentBuildId ← the per-version build ID

Injection is opt-in: the keys are only touched if already present in the template's metadata map (mirroring the existing metrics[*].external.metric.selector.matchLabels merge pattern). Non-temporal triggers in the same ScaledObject are left untouched.

Example

apiVersion: temporal.io/v1alpha1
kind: WorkerResourceTemplate
metadata:
  name: my-worker-scaler
  namespace: default
spec:
  workerDeploymentRef:
    name: my-worker
  template:
    apiVersion: keda.sh/v1alpha1
    kind: ScaledObject
    spec:
      scaleTargetRef: {}                     # auto-injected (versioned Deployment)
      minReplicaCount: 1
      maxReplicaCount: 10
      pollingInterval: 30
      triggers:
        - type: temporal
          metadata:
            endpoint: us-east-1.aws.api.temporal.io:7233
            namespace: my-temporal-ns
            taskQueue: my-task-queue
            workerDeploymentName: ""         # auto-injected
            workerDeploymentBuildId: ""      # auto-injected
            targetQueueSize: "10"
          authenticationRef:
            kind: ClusterTriggerAuthentication
            name: temporal-cloud-auth

The controller will stamp one ScaledObject per active worker version, each with the correct workerDeploymentBuildId, and delete them on sunset via the existing DeleteWorkerResources planner (kind-agnostic).

Implementation

  • appendTemporalTriggerMetadata(spec, twdName, buildID) — new helper that walks spec.triggers[*], filters on type == "temporal", and sets the two keys if already present.
  • autoInjectFields signature now takes twdName, buildID so it can pass them through. Behaviour for HPA / matchLabels / scaleTargetRef is unchanged.

Helm-side: adding ScaledObject to workerResourceTemplate.allowedResources + the controller's ClusterRole is left as a follow-up (or a separate commit in this PR if maintainers prefer). For users who want this today, they can add the entry to their chart values without further controller changes.

Tests

New TestAutoInjectFields_TemporalTriggerMetadata covers:

  • injects when both keys present (empty string placeholders)
  • overwrites stale values
  • no-op when keys absent (opt-in semantics)
  • non-temporal triggers are untouched (heterogeneous trigger array)
  • scaleTargetRef: {} auto-fill still works on ScaledObject
  • no panic when spec.triggers is absent

All existing tests still pass (go test ./...).

@nikoul nikoul requested review from a team and jlegrone as code owners May 26, 2026 08:48
@CLAassistant
Copy link
Copy Markdown

CLAassistant commented May 26, 2026

CLA assistant check
All committers have signed the CLA.

@nikoul nikoul force-pushed the nicolas/wrt-keda-temporal-trigger-injection branch 2 times, most recently from 537199f to 1b0f764 Compare May 26, 2026 11:32
Comment thread internal/k8s/workerresourcetemplates.go
assert.Equal(t, "my-tq", md["taskQueue"])
})

t.Run("overwrites pre-existing values when keys are present", func(t *testing.T) {
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is valid, but just an idea, in preparation for the auto-injection of spec.metrics[*].external.metric.selector.matchLabels, the WorkerResourceTemplate validating webhook rejects objects that have the fields set, to avoid confusion and be very explicit where the fields are coming from. I think it would be consistent to do KEDA templating similarly. See https://github.com/temporalio/temporal-worker-controller/blob/main/docs/worker-resource-templates.md#auto-injection for existing semantics.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, it would be most consistent if the controller templates the namespace field in the Temporal trigger too. The idea being, we know the Temporal namespace that the Worker Deployment is in, so no need for the user to mis-configure it.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I think I've implemented something that answers both those suggestions now.

nikoul added a commit to nikoul/temporal-worker-controller that referenced this pull request May 28, 2026
…mentBuildId

Mirror the scaleTargetRef opt-in pattern for KEDA Temporal trigger metadata:
present + empty string is the opt-in sentinel; present + non-empty value is
rejected by the validating webhook with a clear error message. Absent keys
are unchanged (no injection).

Runtime injection in appendTemporalTriggerMetadata becomes strict: only
overwrites when the value is the empty-string sentinel. This is defence
in depth — if a non-empty value somehow reaches runtime (webhook bypassed,
existing resources), the user-provided value is preserved rather than
silently overwritten, matching the scaleTargetRef isEmptyMap pattern.

Existing tests updated to assert the new strict semantics. New
TestWorkerResourceTemplate_ValidateCreate_TemporalTriggerMetadata covers
opt-in, rejection, and non-temporal-trigger pass-through. A small KEDA-aware
test validator (newValidatorNoAPIWithKEDA) is added so KEDA-specific cases
can reach the new validation without being rejected by the allow-list.

Addresses review comment on PR temporalio#351.
nikoul added a commit to nikoul/temporal-worker-controller that referenced this pull request May 28, 2026
… triggers

The Temporal Cloud namespace is known to the controller via the
TemporalConnection, so users should not need to repeat it on every
WorkerResourceTemplate. Extend the trigger-metadata injection to also
fill in namespace when the user opts in with the empty-string sentinel,
and reject non-empty values in the webhook (mirroring the existing
workerDeploymentName / workerDeploymentBuildId pattern).

Threads temporalNamespace through autoInjectFields and
appendTemporalTriggerMetadata. New test asserts injection happens for the
empty-string sentinel and that an absent key remains absent (opt-in
semantics). New webhook test case asserts rejection of a hardcoded value.

Addresses review comment on PR temporalio#351.
@nikoul nikoul force-pushed the nicolas/wrt-keda-temporal-trigger-injection branch from d875de6 to 2a34ef4 Compare May 28, 2026 08:30
nikoul and others added 4 commits May 28, 2026 12:54
… ScaledObjects

Extend WorkerResourceTemplate auto-injection to populate workerDeploymentName
and workerDeploymentBuildId in KEDA ScaledObject triggers[*].metadata when
the trigger type is "temporal". Unblocks using KEDA's Temporal scaler
(kedacore/keda#7672) as the templated resource for per-version backlog
scaling.

Injection is opt-in: keys are only touched when already present in the
template's metadata map, mirroring the existing metrics matchLabels merge
pattern. Non-temporal triggers in the same ScaledObject are left untouched.

Refs temporalio#286.
…mentBuildId

Mirror the scaleTargetRef opt-in pattern for KEDA Temporal trigger metadata:
present + empty string is the opt-in sentinel; present + non-empty value is
rejected by the validating webhook with a clear error message. Absent keys
are unchanged (no injection).

Runtime injection in appendTemporalTriggerMetadata becomes strict: only
overwrites when the value is the empty-string sentinel. This is defence
in depth — if a non-empty value somehow reaches runtime (webhook bypassed,
existing resources), the user-provided value is preserved rather than
silently overwritten, matching the scaleTargetRef isEmptyMap pattern.

Existing tests updated to assert the new strict semantics. New
TestWorkerResourceTemplate_ValidateCreate_TemporalTriggerMetadata covers
opt-in, rejection, and non-temporal-trigger pass-through. A small KEDA-aware
test validator (newValidatorNoAPIWithKEDA) is added so KEDA-specific cases
can reach the new validation without being rejected by the allow-list.

Addresses review comment on PR temporalio#351.
… triggers

The Temporal Cloud namespace is known to the controller via the
TemporalConnection, so users should not need to repeat it on every
WorkerResourceTemplate. Extend the trigger-metadata injection to also
fill in namespace when the user opts in with the empty-string sentinel,
and reject non-empty values in the webhook (mirroring the existing
workerDeploymentName / workerDeploymentBuildId pattern).

Threads temporalNamespace through autoInjectFields and
appendTemporalTriggerMetadata. New test asserts injection happens for the
empty-string sentinel and that an absent key remains absent (opt-in
semantics). New webhook test case asserts rejection of a hardcoded value.

Addresses review comment on PR temporalio#351.
@nikoul nikoul force-pushed the nicolas/wrt-keda-temporal-trigger-injection branch from 2a34ef4 to 5e96a4e Compare May 28, 2026 10:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants