[rfc]: Dynamic Upstream Routing#32
Conversation
Document the proposal for upstream routing and configuration schema to support multiple upstream clusters based on namespace.
|
|
||
| The proxy currently supports a single static upstream defined in config. However, we want a single proxy (tenant) to serve many namespaces, each potentially terminating at a different address/upstream. | ||
|
|
||
| This can be for a number of reasons, but notably, during migrations, some teams will want to move over namespace-by-namespace rather than cluster-by-cluster. They’d point all namespaces to the original upstream (typically the OSS frontend) and switch to another one as needed based on some routing rules. |
There was a problem hiding this comment.
Practically speaking, I would likely set the proxy to route a fixed/known set of namespaces to OSS and everything else to their upstream. Then, as you work your way down the list, you end up with a single "forward all namespaces" rule, rather than a bunch of namespace-specific rules.
There was a problem hiding this comment.
I think something important about proxy config - especially when migrations are involved.
The final state configuration shouldn't have residual artefacts that a migration occurred.
|
|
||
| The proxy currently supports a single static upstream defined in config. However, we want a single proxy (tenant) to serve many namespaces, each potentially terminating at a different address/upstream. | ||
|
|
||
| This can be for a number of reasons, but notably, during migrations, some teams will want to move over namespace-by-namespace rather than cluster-by-cluster. They’d point all namespaces to the original upstream (typically the OSS frontend) and switch to another one as needed based on some routing rules. |
There was a problem hiding this comment.
Nice. Yes this will keep granularity in sync with how migrations via s2s proxy currently work: a cluster is connected, but namespaces are replicated per-namespace.
This will allow us to maintain this granularity, and moves the worker pointing logic into a centralised place.
Ie:
- connect clusters (via s2s)
- set namespace to passive on destination cluster (via s2s)
- force replicate (via s2s)
- set destination to active (via s2s)
- point workers to destination cluster (now active) (via worker proxy)
- remove source namespace as a replication target (via s2s)
I wonder if we can add management apis on the worker proxy (this proxy) to facilitate pointing workers from source cluster to destination cluster.
This would enable us to fully script the e2e namespace migration process from source to destination cluster. (current gap is requiring workload owner (source owner) to update worker config and restart worker pods. Could we do this with less downtime, and via API.
|
|
||
| The proxy currently supports a single static upstream defined in config. However, we want a single proxy (tenant) to serve many namespaces, each potentially terminating at a different address/upstream. | ||
|
|
||
| This can be for a number of reasons, but notably, during migrations, some teams will want to move over namespace-by-namespace rather than cluster-by-cluster. They’d point all namespaces to the original upstream (typically the OSS frontend) and switch to another one as needed based on some routing rules. |
There was a problem hiding this comment.
I think something important about proxy config - especially when migrations are involved.
The final state configuration shouldn't have residual artefacts that a migration occurred.
|
|
||
| #### Namespace Rules | ||
|
|
||
| Since namespace translation is defined per upstream and we don’t yet know which upstream we’re using, the local namespace (before translation) is used to perform the match. When a namespace is not defined on the rule, no namespace-based matching will be performed. |
There was a problem hiding this comment.
How will inbound routing work for glob-based rules (where inbound is a one to many relationship)
My understanding: rules are evaluated in order at evaluation-time (not pre compiled as a bi-directional map would be).
So in the case of:
rules:
- match:
namespace: "prod-*"
upstream: prod
- match:
namespace: "*-admin"
upstream: admin
A namespace prod-admin would map to prod-namespace as each rule evaluates in order.
On outbound: prod-admin is evaluated to match prod-* and prod is returned.
On inbound: prod is evaluated to match rule prod-*.
So we know the rule - how is the one-to-many translation identified? This might be later in the file, or I might have missed it.
There was a problem hiding this comment.
This is a good point and a potential source of confusion if not documented well. These rules are for routing to upstreams and are evaluated in the order they are defined in the config. There's no namespace translation here. That would be defined per upstream (not covered in this RFC).
There was a problem hiding this comment.
understood! The proxy is only unidirectionally routing - so there is no inbound translation required. Thanks.
liam-lowe
left a comment
liam-lowe
left a comment
There was a problem hiding this comment.
not sure if you wanted approval on this, or if it was just for gathering feedback.
If gathering feedback - i'm sure you want more than one perspective - but approving to mark my alignment.
| - match: | ||
| namespace: "prod-*" | ||
| headers: | ||
| x-tier: "gold" | ||
| upstream: cloud-namespace |
There was a problem hiding this comment.
Hmm...I had been operating under the assumption that it was, but I don't think it needs to be since the rules are evaluated in order. Just curious, or did you have a use case in mind which would be headers only? Or something else?
There was a problem hiding this comment.
Namespace naming conventions are (were?) loosely enforced, so I think we'd probably want to be able to operate off of headers.
Looking way forward: I want to be able to route Nexus operations dynamically to either TCloud or a local/on-prem cluster. We would be selecting this based on request-boundary headers (e.g. "all Nexus Operations for ServiceA in NamespaceB originating from ApplicationC are routed to on-prem cluster"). I would envision a similar behavior possible for normal namespace traffic as well.
I suspect these more fancy traffic routing capabilities may also just be something we write our own interceptors for.
Routing rules (RFC #32) match namespaces and metadata values using simple globs: exact, prefix (foo*), suffix (*foo), and contains (*foo*). This adds a small, dependency-free pkg/match that compiles a pattern once and matches against it. The implementation deliberately supports only those four forms and rejects an interior "*" (e.g. "a*b") at compile time, so a malformed pattern fails fast at config-load rather than silently matching the wrong thing. A general globber (e.g. path.Match, regexp) was avoided on purpose: those assign meaning to "?", "[]", "\", and treat "/" as a separator, which would change behavior for operator-authored patterns and arbitrary metadata values.
The proxy held a single Config.Upstream, which blocks the dynamic routing work (RFC #32): routing rules need to refer to upstreams by name, and a deployment serves more than one. This replaces it with Config.Upstreams, a named list, as the foundation for the routing block, engine, and connection registry that will follow. Upstream targets may be templated (e.g. "{{ .RemoteNamespace }}.acme:7233") once rendering lands, so Upstream.Validate skips the literal IsHostPort check when the value contains a template action; a static hostPort is still validated. The inbound ListenConfig.Validate stays strict, since a listener address is never templated. Config.PrimaryUpstream returns the first entry as an explicit, documented bridge rather than scatter an Upstreams[0] assumption across the fx modules; it will be removed when routing selects the upstream per request.
Upstream definitions (RFC #32) template their hostPort and serverName with request-derived values so one upstream can serve many namespaces, e.g. "{{ .RemoteNamespace }}.acme-cloud.tmprl.cloud:7233". This adds internal/template, a thin wrapper over text/template that compiles a template once and renders it against a per-request context. Templates are bound to the context they render against. ParseRouting renders against a RoutingContext (local namespace and metadata), used while routing rules pick an upstream; ParseUpstream adds RemoteNamespace, which is only known after an upstream is selected since translation is per-upstream. Binding the type means a routing template that references RemoteNamespace fails at parse time rather than silently rendering empty. Parsing validates by rendering against a probe context, so an unknown field (a typo) is caught at config-load. Absent metadata keys render empty rather than erroring, since the available keys are only known at request time. Must wraps a Parse call for package-level vars and tests.
Routing rules (RFC #32) match namespaces and metadata values using simple globs: exact, prefix (foo*), suffix (*foo), and contains (*foo*). This adds a small, dependency-free pkg/match that compiles a pattern once and matches against it. The implementation deliberately supports only those four forms and rejects an interior "*" (e.g. "a*b") at compile time, so a malformed pattern fails fast at config-load rather than silently matching the wrong thing. A general globber (e.g. path.Match, regexp) was avoided on purpose: those assign meaning to "?", "[]", "\", and treat "/" as a separator, which would change behavior for operator-authored patterns and arbitrary metadata values.
Upstream definitions (RFC #32) template their hostPort and serverName with request-derived values so one upstream can serve many namespaces, e.g. "{{ .RemoteNamespace }}.acme-cloud.tmprl.cloud:7233". This adds internal/template, a thin wrapper over text/template that compiles a template once and renders it against a per-request context. Templates are bound to the context they render against. ParseRouting renders against a RoutingContext (local namespace and metadata), used while routing rules pick an upstream; ParseUpstream adds RemoteNamespace, which is only known after an upstream is selected since translation is per-upstream. Binding the type means a routing template that references RemoteNamespace fails at parse time rather than silently rendering empty. Parsing validates by rendering against a probe context, so an unknown field (a typo) is caught at config-load. Absent metadata keys render empty rather than erroring, since the available keys are only known at request time. Must wraps a Parse call for package-level vars and tests.
Upstream definitions (RFC #32) template their hostPort and serverName with request-derived values so one upstream can serve many namespaces, e.g. "{{ .RemoteNamespace }}.acme-cloud.tmprl.cloud:7233". This adds internal/template, a thin wrapper over text/template that compiles a template once and renders it against a per-request context. Templates are bound to the context they render against. ParseRouting renders against a RoutingContext (local namespace and metadata), used while routing rules pick an upstream; ParseUpstream adds RemoteNamespace, which is only known after an upstream is selected since translation is per-upstream. Binding the type means a routing template that references RemoteNamespace fails at parse time rather than silently rendering empty. Parsing validates by rendering against a probe context, so an unknown field (a typo) is caught at config-load. Absent metadata keys render empty rather than erroring, since the available keys are only known at request time. Must wraps a Parse call for package-level vars and tests.
The proxy held a single Config.Upstream, which blocks the dynamic routing work (RFC #32): routing rules need to refer to upstreams by name, and a deployment serves more than one. This replaces it with Config.Upstreams, a named list, as the foundation for the routing block, engine, and connection registry that will follow. Upstream targets may be templated (e.g. "{{ .RemoteNamespace }}.acme:7233") once rendering lands, so Upstream.Validate skips the literal IsHostPort check when the value contains a template action; a static hostPort is still validated. The inbound ListenConfig.Validate stays strict, since a listener address is never templated. Config.PrimaryUpstream returns the first entry as an explicit, documented bridge rather than scatter an Upstreams[0] assumption across the fx modules; it will be removed when routing selects the upstream per request.
The proxy held a single Config.Upstream, which blocks the dynamic routing work (RFC #32): routing rules need to refer to upstreams by name, and a deployment serves more than one. This replaces it with Config.Upstreams, a named list, as the foundation for the routing block, engine, and connection registry that will follow. Upstream targets may be templated (e.g. "{{ .RemoteNamespace }}.acme:7233") once rendering lands, so Upstream.Validate skips the literal IsHostPort check when the value contains a template action; a static hostPort is still validated. The inbound ListenConfig.Validate stays strict, since a listener address is never templated. Config.PrimaryUpstream returns the first entry as an explicit, documented bridge rather than scatter an Upstreams[0] assumption across the fx modules; it will be removed when routing selects the upstream per request.
3 participants
Document the proposal for upstream routing and configuration schema to support multiple upstream clusters based on namespace.
https://github.com/temporalio/temporal-proxy/blob/rfc-upstream-routing/rfc/routing.md