Skip to content

Convert MZ_HISTORY_RETENTION_STRATEGIES to a builtin mv - #39473

Merged
SangJunBak merged 7 commits into
jun/sql-653-static-builtin-values-in-mv-fingerprintfrom
jun/sql-701/history-retention
Oct 14, 2026
Merged

SangJunBak merged 7 commits into
jun/sql-653-static-builtin-values-in-mv-fingerprintfrom
jun/sql-701/history-retention

Conversation

@SangJunBak

@SangJunBak SangJunBak commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor
  • Converts MZ_HISTORY_RETENTION_STRATEGIES to a builtin materialized view. Read commit messages for more details

Motivation

Closes SQL-701

Verification

  • Created a new SLT to test the SQL itself.
  • Diffed this relation using a customer's workload replay against main and this branch.

@SangJunBak
SangJunBak added this pull request to stack #38849 October 2, 2026 05:02
@linear-code

linear-code Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

SQL-701

SQL-703

@SangJunBak
SangJunBak force-pushed the jun/sql-701/history-retention branch 3 times, most recently from 26134cf to f1bbdc5 Compare October 6, 2026 03:02
@SangJunBak SangJunBak changed the title repr: move interval literal planning into Interval::from_literal Convert MZ_HISTORY_RETENTION_STRATEGIES to a builtin mv Oct 6, 2026
@SangJunBak
SangJunBak force-pushed the jun/sql-701/history-retention branch 3 times, most recently from fb3ceb9 to 1de6381 Compare October 6, 2026 04:08
@SangJunBak
SangJunBak marked this pull request as ready for review October 6, 2026 04:08
@SangJunBak
SangJunBak requested review from a team as code owners October 6, 2026 04:08

@ggevay ggevay left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just minor comments

SELECT
's' || (gm.data->'value'->>'catalog_id') AS id,
CASE
WHEN b.is_retained_metrics_object THEN coalesce(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only sees durable overrides (ALTER SYSTEM, LaunchDarkly and the config-file sync). A metrics_retention passed as --system-parameter-default never reaches mz_overridden_system_parameters: with metrics_retention=1h as a process default, SHOW metrics_retention says 1 h and the old table reported 3600000 for mz_cluster_replica_sizes, while this reports 2592000000. The underlying gap is that no relation shows effective parameter values (SQL-757), so a sentence here saying that is enough for this PR. Also worth a line in the description: these rows now follow ALTER SYSTEM SET metrics_retention immediately, where the old table kept the boot-time value until the next restart.

CREATE TABLE t_unlimited (a int) WITH (RETAIN HISTORY FOR '0')

statement ok
CREATE SOURCE lg IN CLUSTER quickstart FROM LOAD GENERATOR COUNTER WITH (RETAIN HISTORY FOR '1m')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing here creates a subsource or progress relation, so the view's 'subsource' type filter is unguarded: dropping it loses 6 of the 7 rows below and this file still passes. Appended at the end of this file, this passes on the PR's CI image (and fails when an expected value is changed); the values match the old table:

statement ok
CREATE SOURCE lg_auction IN CLUSTER quickstart FROM LOAD GENERATOR AUCTION FOR ALL TABLES WITH (RETAIN HISTORY FOR '6m')

query TT
SELECT o.name, h.value
FROM mz_internal.mz_history_retention_strategies h
JOIN mz_catalog.mz_objects o ON o.id = h.id
WHERE o.name IN ('lg_auction', 'lg_auction_progress', 'accounts', 'auctions', 'bids', 'organizations', 'users')
ORDER BY o.name
----
accounts  360000
auctions  360000
bids  360000
lg_auction  360000
lg_auction_progress  360000
organizations  360000
users  360000

@SangJunBak
SangJunBak force-pushed the jun/sql-701/history-retention branch from 1de6381 to 35c8301 Compare October 6, 2026 18:39
@SangJunBak
SangJunBak requested a review from a team as a code owner October 9, 2026 15:28
@SangJunBak
SangJunBak force-pushed the jun/sql-701/history-retention branch 2 times, most recently from f8d8470 to 27e626e Compare October 9, 2026 15:34
@SangJunBak
SangJunBak force-pushed the jun/sql-701/history-retention branch from 27e626e to 3b25c00 Compare October 13, 2026 17:38
- Move the conversion of an INTERVAL literal string from mz_sql to mz_repr. We do this because mz-catalog-decode will need to parse the interval value for SQL with RETAIN HISTORY. However, it can't reuse the function in mz-sql since it'd be a circular dependency.
- Fix stale comment for parse_interval_w_disambiguator
- We parse RETAIN HISTORY values from SQL ddl and use Interval::from_literal from the previous commit
- The `date_time_field` functions ends duplicating the same DateTimeField conversion we do in the planner. However if we didn't, we'd have to create a separate common crate which seems excessive.
- We expose the retained history properties of builtin objects into the inlined `mz_builtin_*` views
- We hardcode 'false' for builtin mvs since the rust property in a BuiltinMaterializedView exists, but it doesn't actually work
- We expose the object_type property from mz-catalog-protos such that we can match on it when referencing it in SQL
…alized view

mz_history_retention_strategies becomes a materialized view over
mz_catalog_raw. A durable item reports the retain_history_millis of its
create_sql, and the default window when it has none. The packer goes
away and the view gets a replacement migration step.
Add sqllogictests for the schema, one row per RETAIN HISTORY spelling,
builtin and introspection rows, a metrics_retention override, and
retraction.
- Say at the metrics_retention lookup that only a durable override is
  visible: a value passed as --system-parameter-default never reaches
  mz_overridden_system_parameters, so retained-metrics builtins then
  report the compiled-in default. No relation exposes effective parameter
  values yet (SQL-757).
- Cover a multi-output load generator source so that the view's
  'subsource' type filter is guarded: the subsources and the progress
  relation inherit the source's RETAIN HISTORY.
@SangJunBak
SangJunBak force-pushed the jun/sql-701/history-retention branch from 3b25c00 to 78cb181 Compare October 14, 2026 08:57
@SangJunBak
SangJunBak merged commit 7b235fe into main Oct 14, 2026
86 checks passed
@SangJunBak
SangJunBak deleted the jun/sql-701/history-retention branch October 14, 2026 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants