[helm] Add support advertised listeners for out of cluster clients

[charlesdong1991]

Purpose

Linked issue: close #4092

There is currently no way to run a Fluss client from outside the k8s cluster. Not sure if it is something we want, but I got this issue when trying to set up in our env...

I can close if it is not something we need.

[charlesdong1991]

can you take a look if this is something we want? @affo thanks

[affo]

Thanks @charlesdong1991 — yes, this is something we want. The problem is real (the classic Kafka advertised-listeners issue), the default-off scoping is right, and deliberately not managing per-pod Services keeps this a proper chart-level stopgap until the k8s operator (FIP-41) owns external access. Also +1 on hard-failing when configurationOverrides sets bind.listeners/advertised.listeners — the chart and the override would silently fight otherwise.

I have three design-level asks on the values API before we lock it in (it becomes public API of the chart), plus some smaller points.

1. Per-pod uniqueness: ordinal arrays instead of shell formulas

advertisedPort: "$((9126 + ${POD_NAME##*-}))" works, but it makes "the startup command is a shell script" part of the chart's public contract, and it's not schema-checkable — a typo fails only at client-redirect time. Prefer the Bitnami approach (cf. externalAccess.*.service.nodePorts, "length must be the same as replicaCount"): per-ordinal arrays.

Accepted trade-off: static arrays can't express ${NODE_IP}. Users creating per-pod NodePorts themselves would provide statically known IPs/hostnames (Bitnami externalIPs style). If you think the NODE_IP case is important enough, we could keep a whitelisted ${NODE_IP} substitution in advertisedHost only — open to discussing.

2. Drop listeners.client.advertisedHost/advertisedPort

CLIENT (like INTERNAL) addressing should stay chart-owned: a bad override bricks in-cluster connectivity, and the failure only shows at produce/fetch time. The two use cases it covers are better served otherwise:

• Non-cluster.local clusters (real gap — the FQDN is hardcoded today): add a top-level clusterDomain: cluster.local value, as Bitnami does.
• Split-horizon DNS (the one case the docs cite): covered by EXTERNAL with the split name as advertisedHost; only the single-bootstrap-address aesthetic is lost.

Adding the override back later is a compatible change; removing ig — so let's start narrow. This also deletes the CLIENT branch ofthe replica warning and shrinks the helpers.

3. EXTERNAL auth: make it explicit, not inherited

Right now EXTERNAL silently mirrors the CLIENT protocol and JAAS users. Was that a deliberate choice or a shortcut? Fluss natively supports per-listener auth
(security.protocol.map + per-listener JAAS contexts, <listenertext), and every comparable system (Kafka broker per-listenerconfigs, Bitnami listeners.external.protocol, Strimzi per-listener tls/auth) makes external security a first-class, per-listener setting. Proposal: a
security.external.sasl block symmetric with security.client.sa plain | client — where client is the explicit opt-in totoday's mirroring behavior. Two safety nets regardless of shape:

• VALUES WARNING when external.enabled and the external listener resolves to PLAINTEXT — an unauthenticated listener exposed outside the cluster should
  never be silent.
• A docs warning that SASL/PLAIN without TLS sends credentials in cleartext; until TLS lands (FIP-29), external exposure should be limited to trusted
  networks.

Resulting values.yaml

clusterDomain: cluster.local

listeners:
  internal:
    port: 9123
  client:
    port: 9124            # advertised address stays chart-owned (pod FQDN)
  external:
    enabled: false
    port: 9125
    advertisedHost: ""    # shared default host, e.g. "fluss.example.com"
    advertisedPort: ""    # shared default port (needs per-pod hosts)

coordinator:
  listeners:
    external:
      advertisedHost: ""
      advertisedPort: ""

tablet:
  listeners:
    external:
      advertisedHost: ""   # shared host for all tablets, or:
      advertisedHosts: []  # per-ordinal; length == tablet.numberOfReplicas
      advertisedPorts: []  # per-ordinal; length == tablet.numberOfReplicas

security:
  external:
    sasl:
      mechanism: ""        # "" (PLAINTEXT + warning) | plain | c
      plain:
        users: []          # same shape as security.client.sasl.p

The port-forward example from the docs then becomes:

listeners:
  external:
    enabled: true
coordinator:
  listeners:
    external:
      advertisedHost: "127.0.0.1"
      advertisedPort: "9125"
tablet:
  listeners:
    external:
      advertisedHost: "127.0.0.1"
      advertisedPorts: [9126, 9127, 9128]

Smaller points

• Drop listeners.external.name and hardcode EXTERNAL: nothing validates the name survives server.yaml/JAAS parsing, the k8s port names stay hardcoxternalanyway, and it can collide withconfigurationOverride]`.
• The external-port collision check should also cover metrics.port (9249/9250), otherwise you get a duplicate container port. The per-component override keys are typo-sensitive with no `val array design above a schema becomes easy to add — fine as afollow-up.
  Happy to help iterate — the helpers/tests/docs structure is solid, this is only about the values surface.

[charlesdong1991]

Hi @affo thanks a lot for your reviews!

Per-pod uniqueness: ordinal arrays instead of shell formulas

i switched to per-ordinal arrays. only kept ${NODE_IP} as whitelisted token in advertisedHost since arrays can't cover that case

Drop listeners.client.advertisedHost/advertisedPort

agree, drop both, CLIENT stays chart-owned and added clsuterDomain for non cluster.local case

EXTERNAL auth: make it explicit, not inherited

Honestly, it was a shortcut, EXTERNAL had no credentials of its own so I reused the client ones.. now client explicitly reuses CLIENT ones, and PLAINTEXT warns when the listener is enabled, also updated docs

Let me know how you think! 🙏 thanks again

[charlesdong1991]

regarding smaller points

Smaller points: Drop listeners.external.name and hardcode EXTERNAL

dropped and hardcoded

The external-port collision check should also cover metrics.port (9249/9250), otherwise you get a duplicate container port.

added, check covers metrics ports too

and i could do values schema json as follow-up!

[affo]

Thanks @charlesdong1991 🎉.

Everything I asked for is there: per-ordinal arrays with real validation, clusterDomain, the CLIENT overrides gone, EXTERNAL hardcoded, metrics ports in the collision check, and the PLAINTEXT warning.

One correction, and it's on me.

Please drop security.external.sasl.mechanism: client

I'm sorry about this one — that part of my review was AI output that I didn't catch before posting. You implemented exactly what I asked for; the ask was wrong. Let me explain why so the replacement is obvious.

client is a value of the mechanism enum that isn't a mechanism. It does two unrelated things — resolves EXTERNAL's mechanism from CLIENT's, and sources the JAAS users from CLIENT — and it makes a sibling key conditionally illegal, since security.external.sasl.plain.users becomes a hard validation failure when it's set. That's two axes in one field.

Inheritance itself is fine and has plenty of precedent. Encoding it as an enum value next to mechanism names does not:

• Fluss itself already has the fallback: JaasContext.loadServerContext resolves dynamic config → <listener>.FlussServer → global FlussServer. Per-listener with a fallback to a global default.
• Kafka brokers do the same: listener.name.<l>.<mech>.sasl.jaas.config → <l>.KafkaServer → KafkaServer.
• The Bitnami Kafka chart takes the other approach — credentials live in a global pool (sasl.client.users), and each listener declares only its protocol.
• Strimzi puts authentication.type on the listener and keeps credentials in cluster-wide KafkaUser resources.

None of them lets one listener name another listener as the value of its auth field.

The practical cost is the enum. Today "" | plain | client looks harmless because plain is the only mechanism we have. As soon as a second one lands, client and scram compete for the same slot and there's no way to say "SCRAM, credentials from CLIENT" — at which point we add the credential-source key we should have added now, and client becomes an alias we can't remove.

So: security.external.sasl.mechanism: "" | plain, symmetric with every other listener.

Sharing credentials between CLIENT and EXTERNAL doesn't need new machinery — the user entries already accept existingSecret, so both listeners can point at the same Secret:

security:
  client:
    sasl:
      mechanism: plain
      plain:
        users:
          - existingSecret: { name: fluss-sasl-alice }
  external:
    sasl:
      mechanism: plain
      plain:
        users:
          - existingSecret: { name: fluss-sasl-alice }

Explicit at the point of use, and no hidden coupling between two listeners' config.

If you think the shorthand is worth keeping, the shape I'd accept is security.external.sasl.plain.usersFrom: client — a key that selects a credential source and says so, leaving mechanism as a mechanism. But my preference is to just drop it: the same argument I used for narrowing the CLIENT overrides applies here, and more strongly. Adding it back later is compatible; removing it later isn't, and nothing is public API yet.

This deletes usesClientUsers, the resolvedMechanism indirection, the users-must-be-empty validation, the two client test cases, and the docs line abouts a docs/code mismatch on the way out (thevalues table says security.external.sasl.plain.users is "ignored when mechanism is client", but the chart hard-fails; I hit it).

Smaller points

• tablet.listeners.external and coordinator.listeners.external only exist as comments in values.yaml, so helm show values doesn't reveal them while the docs table advertises [] / "" defaults for them. That's exactly what values.schema.json would pin down — happy for it to stay a follow-up, but worth doing together.
• The values table still documents configurationOverrides.zookeeper.address ending in a literal cluster.local; it's {{ .Values.clusterDomain }} now.
• NODE_IP is injected unconditionally on the coordinator but only when referenced on tablets. It's documented, but I don't think the asymmetry buys anything — the tablet's conditional looks right for both.
• Branch needs a rebase

Everything else is good to go from my side.

[affo]

I mostly have nits for you, but I unblock this PR.
The feature is quite important!

I tag @swuferhong for a final review on the PR

[affo]

this paragraph makes sense, but feels disconnected from the rest 🤔

[affo]

I would probably move to # out of cluster clients? does it make sense to you?

[affo]

A bit better to rephrase and make more digestable :)

[charlesdong1991]

moved and rephrased!

[affo]

Can we separate here the various blocks to make it more readable?
Maybe adding a comment on top:

{{/* ... */}}

[affo]

Also splitting to multiple functions, as you prefer 🤝

[charlesdong1991]

updated!

[affo]

I think it would be better to see this at a glance directly in STS.
Splitting the bootstrap script into multiple places makes it hard to understand the final output and difficult to follow.

I do understand that you do this so that you can reuse across tablet and coordinator though 🤝

Most probably the solution is to make the entire command be generated in .tpl and just template it in the sts, so that we don't end up with part of the command somewhere and part of the command somewhere else.

This is just a need, I don't want to trigger a refactor here, better in a follow up 🤝

[charlesdong1991]

Leaving the helper split for a follow-up

[charlesdong1991]

thanks a lot @affo i updated most of comments you suggest, and leave one as follow-up. 🙏 thanks again!

[affo]

@charlesdong1991 another follow up would be to actually change the schema to support multiple coordinators as well, so that tablet schema is the same.
1.0 supports coordinator HA, so actually you have multiple replicas for the coordinators.
Anyhow, I did not want to stress on this PR and it is fine to have a follow up 🤝

Going to review this again within the day 🤝