CNS-136: expose statement logging sample rate in the operator chart

[jubrad]

https://linear.app/materializeinc/issue/CNS-136/helm-chart-stop-hard-disabling-statement-logging-expose-sample-rate-as

Problem

The operator chart passed --disable-statement-logging unconditionally, so orchestratord always emitted --system-parameter-default=statement_logging_max_sample_rate=0 and query history was permanently empty in self-managed installs. There was no way to turn it back on short of forking the chart.

Solution

Replace the boolean orchestratord flag with --statement-logging-max-sample-rate=<f64> (Option<f64>, unset means no override), surfaced as the operator.args.statementLoggingMaxSampleRate chart value, defaulting to 0.1. 0 still fully disables statement logging, and null inherits environmentd's own default of 0.99. The flag range-checks [0, 1] at parse time, since environmentd rejects a violating value by refusing to open its catalog.

The sample rate bounds the fraction of statements recorded, not the volume written: above ~2 KB/s of row bytes the statement_logging_target_data_rate throttle is the binding limit. So operator.args.statementLoggingTargetDataRate is exposed alongside it, defaulting to unset, as the lever that actually caps how fast query history grows.

Testing

helm unittest misc/helm-charts/operator (42 tests) covers the default, a custom rate, 0, and unset for both values. cargo check/clippy/test -p mz-orchestratord, bin/fmt, and ci/test/lint-main/checks/check-helm-docs.sh pass. Range validation verified against the built binary. No cluster needed.

For the reviewer

• The chart's enableInternalStatementLogging: true default was inert while the max rate was pinned at 0, and now becomes live. Internal-user statements (console polling, mz_system) share the same effective rate. Left alone deliberately to keep this PR from changing a second default, but worth a decision on whether it should flip.
• Query history is never truncated for these five collections (database-issues#7666), so recorded history accumulates for the lifetime of the environment. This is pre-existing and applies to Cloud identically at 0.99; self-managed at 0.1 is strictly cheaper. Bounding cumulative size needs truncation, which is out of scope here.
• The new arguments change the environmentd StatefulSet spec, so existing environments flip to UpToDate=False / WaitingForApproval after the operator upgrade, and the fix only lands for them once a rollout is requested.
• The flag rename is safe: nothing outside this chart set --disable-statement-logging. It does break a pinned-old-operator.image.tag + new-chart combination, as any generation-affecting flag change would.

🤖 Generated with Claude Code

[def-]

QA LLM Review

1. MEDIUM -- Sample rate does not bound statement-logging storage, which is never reclaimed

misc/helm-charts/operator/values.yaml:47

Turning the default from "off" to 0.1 starts every self-managed install accumulating query-history data in persist that is never truncated, and the sample rate is the wrong lever to bound it: a byte-rate throttle already caps ingest independent of sampling, so on any moderately busy environment 0.1 and 0.99 converge to the same sustained write rate. The knob added to keep the cost bounded therefore does not bound it above a fairly low traffic threshold.

Details

Two mechanisms make this concrete.

src/storage-controller/src/collection_mgmt.rs:1188-1200 — StatementExecutionHistory, PreparedStatementHistory, SessionHistory, StatementLifecycleHistory and SqlText are explicitly excluded from partially_truncate_*, with an in-tree note that rows are never removed (MaterializeInc/database-issues#7666, closed, behavior unchanged). Every other append-only introspection collection here is trimmed on startup. So whatever gets logged stays forever, for the life of the environment, with no operator-facing way to reclaim it.

src/adapter/src/statement_logging.rs:709-735 — before a sampled statement is logged, its row bytes are charged against a token bucket refilled at statement_logging_target_data_rate (default 2071 B/s, burst credit 50 MiB). Sampling happens before this check, so the throttle is the binding constraint whenever the sampled traffic exceeds ~2 KB/s of row bytes, which is on the order of a few tens of statements per second at 0.99 and a few hundred at 0.1. Past that point the sample rate stops affecting steady-state volume entirely: both settings write ~2071 B/s ≈ 179 MB/day ≈ 65 GB/year into shards that are never compacted away. High query volume is precisely the case the removed --disable-statement-logging comment was guarding against.

Note also that operator.args.enableInternalStatementLogging already defaults to true in this chart (values.yaml:39). That setting has been inert because the sample rate was pinned to 0; this change makes it live, so internal-user statements land in the same never-truncated collections.

If the goal is a bounded cost rather than bounded sampling, the lever that actually caps bytes is statement_logging_target_data_rate. Exposing that (or lowering it for self-managed) gives a real ceiling on write rate; it still does not cap cumulative size, which needs truncation for these five collections. Worth deciding explicitly whether shipping this default before truncation exists is acceptable, rather than relying on the sample rate to do it.

2. LOW -- Out-of-range sample rate is rejected two layers away, at environmentd startup

src/orchestratord/src/controller/materialize/generation.rs:718

The new value is passed through the chart and orchestratord unvalidated, but environmentd constrains statement_logging_max_sample_rate to [0, 1] and treats a violating --system-parameter-default as a fatal catalog-open error. A plausible mistake such as statementLoggingMaxSampleRate: 10 (reading "fraction" as "percent") therefore surfaces as environmentd failing to boot rather than as a rejected Helm value.

Details

The chart has no values.schema.json, the template only checks kindIs "invalid", and #[clap(long)] statement_logging_max_sample_rate: Option<f64> (src/orchestratord/src/bin/orchestratord.rs:220) accepts any f64. src/adapter/src/catalog/open.rs:239-249 tolerates only VarError::UnknownParameter; the NUMERIC_BOUNDED_0_1_INCLUSIVE constraint violation returns Err and aborts startup.

Blast radius is limited: a rollout only starts when one is requested (src/orchestratord/src/controller/materialize.rs:398), so an existing instance keeps serving from the active generation and only the new generation fails to come up. A fresh install never becomes available. A range check in orchestratord at argument-parse time would reject it where the value was configured.

[jubrad]

Thanks, both findings verified against the code and both addressed.

1 (storage) — acted on, partly. Confirmed all three mechanics: the five collections are excluded from truncation (collection_mgmt.rs:1190-1202), sampling gates before the throttle (if !sample { return None } at statement_logging.rs:668 precedes the token-bucket check at :720), so the 2071 B/s default is the binding constraint above modest traffic and 0.1 vs 0.99 converge there.

The consequence for this PR is that my values.yaml comment was wrong: it implied the sample rate bounds storage. Corrected to state that it bounds the sampled fraction, that sustained volume is capped separately by statement_logging_target_data_rate, and that history is retained for the lifetime of the environment.

Taking your suggestion, statement_logging_target_data_rate is now also exposed as operator.args.statementLoggingTargetDataRate, defaulting to unset so environmentd's 2071 B/s applies. That gives operators the lever that actually caps write rate.

On shipping 0.1 before truncation exists: keeping it. Cloud runs these same never-truncated collections at 0.99, so self-managed at 0.1 is strictly cheaper than cloud, and the cumulative-growth gap (database-issues#7666) is pre-existing rather than introduced here. Enabling query history by default is the explicit goal of the parent issue. Bounded cumulative size still needs truncation, which is out of scope for this PR.

2 (range validation) — fixed. Added a value_parser range check on the flag, so the mistake is rejected where it was configured:

$ orchestratord --statement-logging-max-sample-rate=10
error: invalid value '10' for '--statement-logging-max-sample-rate <...>': sample rate must be between 0 and 1, got 10

1.5, -1, NaN and non-numeric input are rejected the same way; 0, 0.1 and 1 still pass.

On enableInternalStatementLogging: correct that this PR makes it live, and we're deliberately leaving it at true for now rather than changing a second default here. Flagged in the PR description as a reviewer gotcha.

[def-]

QA LLM Review

1. MEDIUM -- statementLoggingTargetDataRate >= 1000000 renders in scientific notation and crashloops the operator

misc/helm-charts/operator/templates/deployment.yaml:83

A byte-per-second knob invites values at or above 1e6, and Helm renders those unquoted from a values file in exponential form, so statementLoggingTargetDataRate: 1048576 reaches the container as --statement-logging-target-data-rate=1.048576e+07. orchestratord's clap Option<usize> rejects that at startup, so the operator Deployment goes into CrashLoopBackOff and stops reconciling every Materialize in the cluster, while helm upgrade reports success.

Details

Helm parses values files through a YAML-to-JSON round trip, so every number arrives as float64 and {{ .Values... }} prints it with Go's %v shortest-float formatting: 999999 renders as 999999, but 1000000 renders as 1e+06 (helm/helm#12195, helm/helm#11130). The value's own documentation and its sibling statement_logging_max_data_credit (50 MiB) both put the natural range for this knob well above the threshold, and the default of 2071 makes "raise it" the expected adjustment. The existing statementLoggingMaxSampleRate passthrough is not affected because it is a fraction below 1. Failure surfaces only as invalid digit found in string in the operator pod log.

-        - "--statement-logging-target-data-rate={{ .Values.operator.args.statementLoggingTargetDataRate }}"
+        - "--statement-logging-target-data-rate={{ int64 .Values.operator.args.statementLoggingTargetDataRate }}"

int64 handles both the float64 from a values file and the int64 from --set. Note that a regression test using set: will not reproduce this, since set: values are not parsed as floats. Pin it with a values: fixture file holding a value at or above 1e6 and assert the arg renders as =1048576.

[jubrad]

Confirmed and fixed in a0d9a91. Good catch, this was a real crashloop.

Reproduced the whole chain before fixing. Values file at 1048576 rendered --statement-logging-target-data-rate=1.048576e+06, and the rebuilt binary rejects it:

error: invalid value '1.048576e+06' for '--statement-logging-target-data-rate <...>': invalid digit found in string

Threshold is exactly where you said, 999999 renders literally and 1000000 becomes 1e+06. Applied your int64 diff; all of 2071 / 999999 / 1000000 / 1048576 / 100000000 now render literally from a values file, --set still works, and unset still omits the flag.

Also confirmed your note that set: can't reproduce it (those stay int64), so the regression test uses a values: fixture at tests/values/large-target-data-rate.yaml. Mutation-checked it: dropping int64 from the template fails that test and only that test. 43 tests pass.

Two clarifications on the writeup:

• The sample rate is safe, but not because it is a fraction below 1. statementLoggingMaxSampleRate: 0.0000001 does render exponentially, as 1e-07. It survives because Rust's f64 FromStr accepts exponent notation, where usize does not. So the immunity is in the parser, not the magnitude, which is worth knowing if that flag ever changes type.
• int64 truncates rather than rejecting, so a nonsensical fractional byte rate like 1.5 now silently becomes 1 B/s, which would throttle statement logging almost entirely instead of failing loudly. Accepting that tradeoff rather than adding template validation, since a fractional byte count is not a plausible input, but flagging it as a known edge.

[def-]

QA LLM Review

1. MEDIUM -- A non-numeric statementLoggingTargetDataRate now renders as 0, silently disabling statement logging

misc/helm-charts/operator/templates/deployment.yaml:83

sprig's int64 is cast.ToInt64, which discards the parse error and returns 0, so statementLoggingTargetDataRate: 10Mi renders --statement-logging-target-data-rate=0 instead of failing. A target data rate of 0 throttles every statement, so query history stays permanently empty with no error in the chart, the operator, or environmentd. Any value below 1 truncates to 0 the same way.

Details

Nothing downstream objects to the coerced value: the template's only guard is kindIs "invalid", the chart has no values.schema.json, and Option<usize> accepts Some(0), so the rendered manifest reads like a deliberate =0. In environmentd the bucket starts empty and refills at rate * elapsed (src/adapter/src/statement_logging.rs:405, :431), so at rate 0 checked_sub(cost) fails for every record forever. That is exactly the state this PR exists to remove, and the operator's symptom (empty query history after the upgrade) looks like the upgrade never took effect rather than like a bad value.

A suffixed quantity is a likely input here: every other byte-valued setting in this chart carries one (operator.resources.limits.memory: 512Mi, disk_limit: "1552MiB"), and this one is documented in bytes per second. Before this commit such a value reached clap and was rejected loudly. A one-line guard keeps the loud failure while keeping the float64 normalization:

{{- if kindIs "string" .Values.operator.args.statementLoggingTargetDataRate }}
{{- fail "operator.args.statementLoggingTargetDataRate must be a number of bytes per second, not a quantity string" }}
{{- end }}