Repository navigation
Convert MZ_HISTORY_RETENTION_STRATEGIES to a builtin mv - #39473
SangJunBak merged 7 commits into
Conversation
26134cf to
f1bbdc5
Compare
MZ_HISTORY_RETENTION_STRATEGIES to a builtin mv
fb3ceb9 to
1de6381
Compare
ggevay
left a comment
There was a problem hiding this comment.
LGTM, just minor comments
| SELECT | ||
| 's' || (gm.data->'value'->>'catalog_id') AS id, | ||
| CASE | ||
| WHEN b.is_retained_metrics_object THEN coalesce( |
There was a problem hiding this comment.
This only sees durable overrides (ALTER SYSTEM, LaunchDarkly and the config-file sync). A metrics_retention passed as --system-parameter-default never reaches mz_overridden_system_parameters: with metrics_retention=1h as a process default, SHOW metrics_retention says 1 h and the old table reported 3600000 for mz_cluster_replica_sizes, while this reports 2592000000. The underlying gap is that no relation shows effective parameter values (SQL-757), so a sentence here saying that is enough for this PR. Also worth a line in the description: these rows now follow ALTER SYSTEM SET metrics_retention immediately, where the old table kept the boot-time value until the next restart.
| CREATE TABLE t_unlimited (a int) WITH (RETAIN HISTORY FOR '0') | ||
|
|
||
| statement ok | ||
| CREATE SOURCE lg IN CLUSTER quickstart FROM LOAD GENERATOR COUNTER WITH (RETAIN HISTORY FOR '1m') |
There was a problem hiding this comment.
Nothing here creates a subsource or progress relation, so the view's 'subsource' type filter is unguarded: dropping it loses 6 of the 7 rows below and this file still passes. Appended at the end of this file, this passes on the PR's CI image (and fails when an expected value is changed); the values match the old table:
statement ok
CREATE SOURCE lg_auction IN CLUSTER quickstart FROM LOAD GENERATOR AUCTION FOR ALL TABLES WITH (RETAIN HISTORY FOR '6m')
query TT
SELECT o.name, h.value
FROM mz_internal.mz_history_retention_strategies h
JOIN mz_catalog.mz_objects o ON o.id = h.id
WHERE o.name IN ('lg_auction', 'lg_auction_progress', 'accounts', 'auctions', 'bids', 'organizations', 'users')
ORDER BY o.name
----
accounts 360000
auctions 360000
bids 360000
lg_auction 360000
lg_auction_progress 360000
organizations 360000
users 360000
1de6381 to
35c8301
Compare
f8d8470 to
27e626e
Compare
27e626e to
3b25c00
Compare
- Move the conversion of an INTERVAL literal string from mz_sql to mz_repr. We do this because mz-catalog-decode will need to parse the interval value for SQL with RETAIN HISTORY. However, it can't reuse the function in mz-sql since it'd be a circular dependency. - Fix stale comment for parse_interval_w_disambiguator
- We parse RETAIN HISTORY values from SQL ddl and use Interval::from_literal from the previous commit - The `date_time_field` functions ends duplicating the same DateTimeField conversion we do in the planner. However if we didn't, we'd have to create a separate common crate which seems excessive.
- We expose the retained history properties of builtin objects into the inlined `mz_builtin_*` views - We hardcode 'false' for builtin mvs since the rust property in a BuiltinMaterializedView exists, but it doesn't actually work - We expose the object_type property from mz-catalog-protos such that we can match on it when referencing it in SQL
…alized view mz_history_retention_strategies becomes a materialized view over mz_catalog_raw. A durable item reports the retain_history_millis of its create_sql, and the default window when it has none. The packer goes away and the view gets a replacement migration step.
…a materialized view
Add sqllogictests for the schema, one row per RETAIN HISTORY spelling, builtin and introspection rows, a metrics_retention override, and retraction.
- Say at the metrics_retention lookup that only a durable override is visible: a value passed as --system-parameter-default never reaches mz_overridden_system_parameters, so retained-metrics builtins then report the compiled-in default. No relation exposes effective parameter values yet (SQL-757). - Cover a multi-output load generator source so that the view's 'subsource' type filter is guarded: the subsources and the progress relation inherit the source's RETAIN HISTORY.
3b25c00 to
78cb181
Compare
MZ_HISTORY_RETENTION_STRATEGIESto a builtin materialized view. Read commit messages for more detailsMotivation
Closes SQL-701
Verification