From 5459dd66646440050698dec724fe8be0b6ed1192 Mon Sep 17 00:00:00 2001 From: Jay Tervala Date: Tue, 25 Aug 2026 13:27:33 -0400 Subject: [PATCH 1/3] chart: allow a gateway-only resources override (MLI-8534) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The top-level `resources` block is rendered into all three model-engine deployments — gateway, cacher and endpoint-builder. Their memory profiles are not comparable: the cacher and endpoint-builder run a handful of pods each and sit near 1GiB, while the gateway runs the whole fleet and holds async-task results in memory while proxying them inline. With one shared block the only options are under-declaring the gateway or over-reserving for the other two. On ml-serving-new the gateway currently declares no memory request at all, so the scheduler treats those nodes as memory-empty while pods sit at a ~29GiB median. Adds an optional `gateway.resources`, alongside the existing gateway.* tuning keys. When unset the rendered output is unchanged, so this is a no-op for every existing consumer. No values are changed here; the sizing lands separately in the deploy values. Co-Authored-By: Claude Opus 5 (1M context) --- charts/model-engine/Chart.yaml | 2 +- charts/model-engine/templates/gateway_deployment.yaml | 7 +++++++ charts/model-engine/values.yaml | 7 +++++++ 3 files changed, 15 insertions(+), 1 deletion(-) diff --git a/charts/model-engine/Chart.yaml b/charts/model-engine/Chart.yaml index 26a32221..ed2f7a2a 100644 --- a/charts/model-engine/Chart.yaml +++ b/charts/model-engine/Chart.yaml @@ -15,7 +15,7 @@ type: application # This is the chart version. This version number should be incremented each time you make changes # to the chart and its templates, including the app version. # Versions are expected to follow Semantic Versioning (https://semver.org/) -version: 0.2.9 +version: 0.2.10 # This is the version number of the application being deployed. This version number should be # incremented each time you make changes to the application. Versions are not expected to diff --git a/charts/model-engine/templates/gateway_deployment.yaml b/charts/model-engine/templates/gateway_deployment.yaml index fd2c8627..0ef784f4 100644 --- a/charts/model-engine/templates/gateway_deployment.yaml +++ b/charts/model-engine/templates/gateway_deployment.yaml @@ -77,7 +77,14 @@ spec: - -m - model_engine_server.entrypoints.start_fastapi_server resources: + {{- /* The gateway proxies async-task results inline, so its memory profile is + unrelated to the cacher and endpoint-builder that share .Values.resources. + Falls back to the shared block when gateway.resources is unset. */}} + {{- if .Values.gateway.resources }} + {{- toYaml .Values.gateway.resources | nindent 12 }} + {{- else }} {{- toYaml .Values.resources | nindent 12 }} + {{- end }} {{- include "modelEngine.gatewayEnv" . | indent 10 }} {{- include "modelEngine.volumeMounts" . | indent 10 }} automountServiceAccountToken: {{ .Values.automountServiceAccountToken }} diff --git a/charts/model-engine/values.yaml b/charts/model-engine/values.yaml index 664a9766..4149ca6b 100644 --- a/charts/model-engine/values.yaml +++ b/charts/model-engine/values.yaml @@ -48,6 +48,13 @@ gateway: attempts: 3 retryOn: connect-failure,unavailable,502,504 perTryTimeout: null + # resources overrides the top-level `resources` for the gateway container only. + # That top-level block is shared with the cacher and endpoint-builder, which run a + # handful of pods each and sit near 1GiB; the gateway runs the whole fleet and holds + # task results in memory while it proxies them. Sizing all three off one block means + # either under-declaring the gateway or over-reserving for the other two. + # Leave empty to inherit `resources` unchanged. + resources: {} # rateLimits [optional] per-pod Envoy local_ratelimit token buckets on the gateway # sidecars (inbound). Overflow returns 429 at the proxy without reaching a gateway From 08e71f67d5422ce4a2c5853970384050c7767617 Mon Sep 17 00:00:00 2001 From: Jay Tervala Date: Tue, 25 Aug 2026 16:39:03 -0400 Subject: [PATCH 2/3] chart: deep-merge gateway.resources, make gateway VPA controlledResources overridable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review feedback on two real defects. 1. gateway.resources replaced the shared block wholesale, so a memory-only override silently dropped the shared cpu and ephemeral-storage requests. Now deep-merges over .Values.resources: keys set win, keys absent fall through. deepCopy is required because mergeOverwrite mutates its first argument and .Values.resources is also rendered into the cacher and endpoint-builder — without it, setting gateway.resources would corrupt their output. Verified: with a memory-only override the gateway renders cpu + ephemeral-storage + memory, and both other deployments are unchanged. 2. The gateway VPA hardcoded controlledResources: ["cpu", "memory"], and its minAllowed/maxAllowed come from the shared autoscaling.vertical.* values. In updateMode: Initial the VPA rewrites requests at pod creation and clamps to maxAllowed.memory, so it would silently override any memory pinned via gateway.resources. Adds gateway.vpaControlledResources, defaulting to ["cpu", "memory"] so behaviour is unchanged; set to ["cpu"] alongside a static memory request. Defaults still render byte-identical across all three deployments. Co-Authored-By: Claude Opus 5 (1M context) --- .../model-engine/templates/gateway_deployment.yaml | 13 +++++++------ charts/model-engine/templates/gateway_vpa.yaml | 2 +- charts/model-engine/values.yaml | 12 ++++++++++-- 3 files changed, 18 insertions(+), 9 deletions(-) diff --git a/charts/model-engine/templates/gateway_deployment.yaml b/charts/model-engine/templates/gateway_deployment.yaml index b4969acd..4e76477e 100644 --- a/charts/model-engine/templates/gateway_deployment.yaml +++ b/charts/model-engine/templates/gateway_deployment.yaml @@ -84,12 +84,13 @@ spec: resources: {{- /* The gateway proxies async-task results inline, so its memory profile is unrelated to the cacher and endpoint-builder that share .Values.resources. - Falls back to the shared block when gateway.resources is unset. */}} - {{- if .Values.gateway.resources }} - {{- toYaml .Values.gateway.resources | nindent 12 }} - {{- else }} - {{- toYaml .Values.resources | nindent 12 }} - {{- end }} + gateway.resources deep-merges OVER that shared block, so a memory-only + override keeps the shared cpu / ephemeral-storage requests. + deepCopy is required: mergeOverwrite mutates its first argument, and + .Values.resources is also rendered into the cacher and endpoint-builder. */}} + {{- $base := .Values.resources | default dict }} + {{- $gatewayResources := (.Values.gateway | default dict).resources | default dict }} + {{- toYaml (mergeOverwrite (deepCopy $base) $gatewayResources) | nindent 12 }} {{- include "modelEngine.gatewayEnv" . | indent 10 }} {{- include "modelEngine.volumeMounts" . | indent 10 }} automountServiceAccountToken: {{ .Values.automountServiceAccountToken }} diff --git a/charts/model-engine/templates/gateway_vpa.yaml b/charts/model-engine/templates/gateway_vpa.yaml index 061ed8cf..c5b8639a 100644 --- a/charts/model-engine/templates/gateway_vpa.yaml +++ b/charts/model-engine/templates/gateway_vpa.yaml @@ -23,5 +23,5 @@ spec: maxAllowed: cpu: {{ .Values.autoscaling.vertical.maxAllowed.cpu }} memory: {{ .Values.autoscaling.vertical.maxAllowed.memory }} - controlledResources: ["cpu", "memory"] + controlledResources: {{ (.Values.gateway | default dict).vpaControlledResources | default (list "cpu" "memory") | toJson }} {{- end }} diff --git a/charts/model-engine/values.yaml b/charts/model-engine/values.yaml index 4149ca6b..ed4fa459 100644 --- a/charts/model-engine/values.yaml +++ b/charts/model-engine/values.yaml @@ -48,13 +48,21 @@ gateway: attempts: 3 retryOn: connect-failure,unavailable,502,504 perTryTimeout: null - # resources overrides the top-level `resources` for the gateway container only. + # resources deep-merges over the top-level `resources` for the gateway container only. # That top-level block is shared with the cacher and endpoint-builder, which run a # handful of pods each and sit near 1GiB; the gateway runs the whole fleet and holds # task results in memory while it proxies them. Sizing all three off one block means # either under-declaring the gateway or over-reserving for the other two. - # Leave empty to inherit `resources` unchanged. + # Keys set here win; keys absent here fall through to `resources`, so a memory-only + # override keeps the shared cpu and ephemeral-storage requests. Leave empty to inherit + # `resources` unchanged. resources: {} + # vpaControlledResources sets the gateway VPA's controlledResources. The VPA runs in + # updateMode: Initial, so it rewrites requests at pod creation and clamps them to + # autoscaling.vertical.maxAllowed — which would silently override any memory request + # pinned in `gateway.resources` above. Set to ["cpu"] when pinning memory statically, + # so the VPA keeps managing CPU and stops managing memory. + vpaControlledResources: ["cpu", "memory"] # rateLimits [optional] per-pod Envoy local_ratelimit token buckets on the gateway # sidecars (inbound). Overflow returns 429 at the proxy without reaching a gateway From b8d9a9f0d4d95a2feb163779ae5131f7a78f9293 Mon Sep 17 00:00:00 2001 From: Jay Tervala Date: Tue, 25 Aug 2026 19:01:19 -0400 Subject: [PATCH 3/3] chore: empty commit to retrigger CI Co-Authored-By: Claude Opus 5 (1M context)