[proposal] Per-cluster etcd: give each Kubernetes control plane its own datastore - #25
[proposal] Per-cluster etcd: give each Kubernetes control plane its own datastore#25myasnikovdaniil wants to merge 2 commits into
Conversation
Fold etcd into the kubernetes app so each tenant Kubernetes cluster provisions and owns its own etcd and Kamaji DataStore. End state is one etcd per one Kubernetes cluster: consumers can start a cluster with no admin pre-step, control-plane state is physically isolated per cluster, and no etcd runs unless a cluster needs it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
📝 WalkthroughWalkthroughA new design proposal document is added at ChangesPer-cluster etcd design proposal
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~5 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a design proposal to retire the shared tenant etcd module and bind a dedicated etcd cluster directly to each Kubernetes app, improving self-service, performance isolation, and security. The review feedback highlights two important considerations: first, the risk of DNS label length violations (exceeding 63 characters) if the release name is too long, and second, a potential race condition during deletion where the etcd datastore might be reaped before the KamajiControlPlane finishes finalization.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| endpoints: | ||
| - {{ .Release.Name }}-etcd.{{ .Release.Namespace }}.svc:2379 | ||
| ``` |
There was a problem hiding this comment.
If .Release.Name is long, the service endpoint {{ .Release.Name }}-etcd might exceed the 63-character limit for Kubernetes Service names and DNS labels. To prevent deployment failures or DNS resolution issues, consider using a helper function (such as a truncated/hashed name helper) to format the etcd service name and endpoint, ensuring it strictly adheres to the 63-character limit.
|
|
||
| - **Two `Kubernetes` apps in one namespace** → each renders its own `<release>-etcd` and `<namespace>-<release>` DataStore; no name collision (the previous single shared `DataStore: <namespace>` could not represent two clusters distinctly). | ||
| - **DataStore name exceeds the cluster-scoped name length limit** → fall back to `<namespace>-<release>-<hash>`; the helper enforces the limit deterministically. | ||
| - **Cluster deletion** → the delete hook (`packages/apps/kubernetes/templates/delete.yaml`) must additionally delete the `EtcdCluster`, the cluster-scoped `DataStore`, and the per-cluster cert Secrets, *after* the existing `<release>-datastore-config` finalizer cleanup (delete.yaml already handles Kamaji's datastore-secret finalizer, issue #3062). Ordering: drain the control plane → strip the datastore-config finalizer → remove EtcdCluster + DataStore. |
There was a problem hiding this comment.
If the EtcdCluster or DataStore is deleted at the same time as the KamajiControlPlane (which is the default behavior when they are part of the same Helm release), a race condition can occur. If the etcd pods terminate before the KamajiControlPlane finishes its finalization, the Kamaji controller may fail to reconcile the control plane due to datastore connection timeouts, potentially blocking the deletion process or causing error storms in the operator logs. Consider implementing a mechanism (such as a finalizer on the DataStore or EtcdCluster that waits for the KamajiControlPlane to be gone, or explicit ordering in the deletion hook) to ensure the datastore is only reaped after the control plane has been completely deleted.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
design-proposals/etcd-per-cluster/README.md (2)
5-5: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueFuture date in a draft proposal.
The date
2026-06-30is at the end of the current month. If this is intentional as a target date, no action needed; if it's a placeholder, consider using the draft creation date or removing it until finalized.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@design-proposals/etcd-per-cluster/README.md` at line 5, The README draft currently uses a future placeholder date, so update the date field in the proposal header to a non-future value if this is meant to be a draft, or remove it until finalized. Locate the date entry in the proposal metadata section and replace the hardcoded `Date` value with the draft creation date or omit it entirely if the target date is not confirmed.
35-35: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider rephrasing "exactly" per style guidance.
Static analysis flagged this as overused. Alternatives: "precisely", "in effect", or restructure the sentence.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@design-proposals/etcd-per-cluster/README.md` at line 35, The sentence in the proposal body uses the overused word “exactly”; rephrase that line in the README text to avoid the flagged wording while keeping the same meaning. Update the prose near the per-cluster etcd description, using the existing surrounding sentence structure and terms like “per-cluster etcd” and “tenant-sprawl ceremony” to locate it, but replace “exactly” with a cleaner alternative such as “precisely” or “in effect.”Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@design-proposals/etcd-per-cluster/README.md`:
- Line 190: The Phase 4 removal instructions use the wrong key name for the
legacy etcd namespace propagation, which can mislead implementers. Update the
wording in the README section describing removal of the shared-etcd plumbing to
refer to the exact propagated Secret key and helper name, using _namespace.etcd
and cozy-lib.ns-etcd consistently with the existing references in the document.
- Line 173: Clarify the delete hook behavior in the cluster deletion section:
the current delete flow in the delete hook should only remove the per-cluster
EtcdCluster, DataStore, and cert Secrets for new per-cluster etcd clusters, and
must skip those resources for legacy shared-etcd clusters. Update the guidance
around packages/apps/kubernetes/templates/delete.yaml to explain how the hook
determines the mode (for example, by checking for the per-cluster DataStore or a
dedicated annotation) so the legacy shared EtcdCluster/DataStore owned by the
tenant module is not deleted.
---
Nitpick comments:
In `@design-proposals/etcd-per-cluster/README.md`:
- Line 5: The README draft currently uses a future placeholder date, so update
the date field in the proposal header to a non-future value if this is meant to
be a draft, or remove it until finalized. Locate the date entry in the proposal
metadata section and replace the hardcoded `Date` value with the draft creation
date or omit it entirely if the target date is not confirmed.
- Line 35: The sentence in the proposal body uses the overused word “exactly”;
rephrase that line in the README text to avoid the flagged wording while keeping
the same meaning. Update the prose near the per-cluster etcd description, using
the existing surrounding sentence structure and terms like “per-cluster etcd”
and “tenant-sprawl ceremony” to locate it, but replace “exactly” with a cleaner
alternative such as “precisely” or “in effect.”
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: d31dff8a-6a76-4ef4-9d32-ca60354f0c77
📒 Files selected for processing (1)
design-proposals/etcd-per-cluster/README.md
|
|
||
| - **Two `Kubernetes` apps in one namespace** → each renders its own `<release>-etcd` and `<namespace>-<release>` DataStore; no name collision (the previous single shared `DataStore: <namespace>` could not represent two clusters distinctly). | ||
| - **DataStore name exceeds the cluster-scoped name length limit** → fall back to `<namespace>-<release>-<hash>`; the helper enforces the limit deterministically. | ||
| - **Cluster deletion** → the delete hook (`packages/apps/kubernetes/templates/delete.yaml`) must additionally delete the `EtcdCluster`, the cluster-scoped `DataStore`, and the per-cluster cert Secrets, *after* the existing `<release>-datastore-config` finalizer cleanup (delete.yaml already handles Kamaji's datastore-secret finalizer, issue #3062). Ordering: drain the control plane → strip the datastore-config finalizer → remove EtcdCluster + DataStore. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Explicitly address delete hook behavior for legacy vs. new clusters.
The delete hook must distinguish between:
- Per-cluster etcd clusters: delete the
EtcdCluster,DataStore, and cert Secrets. - Legacy shared-etcd clusters: do not delete the shared
EtcdClusterorDataStore(owned by the tenant module/ancestor).
Without this distinction, deleting a legacy cluster could destroy the shared etcd and break all other clusters in the subtree. Add a sentence clarifying how the hook detects which mode applies (e.g., by the presence of the per-cluster DataStore or an annotation).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@design-proposals/etcd-per-cluster/README.md` at line 173, Clarify the delete
hook behavior in the cluster deletion section: the current delete flow in the
delete hook should only remove the per-cluster EtcdCluster, DataStore, and cert
Secrets for new per-cluster etcd clusters, and must skip those resources for
legacy shared-etcd clusters. Update the guidance around
packages/apps/kubernetes/templates/delete.yaml to explain how the hook
determines the mode (for example, by checking for the per-cluster DataStore or a
dedicated annotation) so the legacy shared EtcdCluster/DataStore owned by the
tenant module is not deleted.
| 1. **Phase 1 — self-provisioned etcd, opt-in.** Add the `etcd` values block and per-cluster etcd/DataStore rendering to the `kubernetes` app behind a default that self-provisions for new clusters while still honoring a legacy `_namespace.etcd` reference for existing ones. Extend `delete.yaml` to reap the per-cluster etcd/DataStore. Ship docs for the new sizing knobs. | ||
| 2. **Phase 2 — migration path.** Land and prove the shared→dedicated data migration (Kamaji datastore migration or snapshot/restore), as a numbered-migration hook where idempotent or a gated runbook otherwise. Provide a dashboard/CLI signal of which clusters are still on the shared etcd. | ||
| 3. **Phase 3 — deprecate the tenant module.** Mark `Tenant.spec.etcd` and the etcd tenant-module catalog entry deprecated; new tenants no longer offer it. Stop hardcoding `etcd: tenant-root`. | ||
| 4. **Phase 4 — remove plumbing.** Once telemetry shows no cluster on the shared etcd, delete `apps/tenant/templates/etcd.yaml`, the `etcd` tenant value, the `_namespace.etcd` propagation and `cozy-lib.ns-etcd` helper, and the `namespace.cozystack.io/etcd` label. Keep the etcd-operator and the etcd chart. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Clarify the legacy reference key name.
Line 190 says "Stop hardcoding etcd: tenant-root" but the actual value propagated is _namespace.etcd: tenant-root (line 26). Dropping the _namespace. prefix could confuse implementers about which Secret key and helper to remove. Align with the exact key name used elsewhere.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@design-proposals/etcd-per-cluster/README.md` at line 190, The Phase 4 removal
instructions use the wrong key name for the legacy etcd namespace propagation,
which can mislead implementers. Update the wording in the README section
describing removal of the shared-etcd plumbing to refer to the exact propagated
Secret key and helper name, using _namespace.etcd and cozy-lib.ns-etcd
consistently with the existing references in the document.
|
+1 on the core direction — folding etcd into the Writing as an operator of a CozyStack-based managed platform, where we run per-tenant Kubernetes clusters at scale. On the "per-cluster etcd multiplies overhead" concernIn practice, per-cluster etcd is a modest addition on top of what a
Per-cluster etcd adds ~18% to requests CPU and ~28% to CPU/memory limits on Small aside: On the proposed
|
|
Filed a scoped tracking issue for Sergei Makarov (@SerjioTT)'s minimal-API point (expose only It captures the reasoning from the review comment above — matching the existing satellite convention (hardcoded infra resources + VPA), keeping the Note it's explicitly gated on this proposal: there's nothing to expose/hide until per-cluster etcd is folded into the |
IvanHunters
left a comment
IvanHunters
left a comment
There was a problem hiding this comment.
Requesting changes — the topology change is well-motivated and the isolation win is real, but a few things should be reconciled before implementation.
- The proposal is written against
etcd.aenix.io/v1alpha1, while in-flight #2859 migrates the platform toetcd-operator.cozystack.io/v1alpha2, renames the CRDs, and switches readiness fromReadytoAvailable. The EtcdCluster apiVersion, the "wait for Ready" semantics, and the reusedextra/etcdchart shape will all be stale once #2859 lands. Please declare an explicit dependency / rebase-onto-#2859 and target v1alpha2 +Available. - Migration-number collision: the proposal targets migration 49, but 49 is already taken on main (ancestor-label backfill) and #2859 also adds its own migration 49. Any numbered migration here must claim 50+.
- Post-migration rollback is effectively one-way and paired with a stateful freeze-mode migration. Kamaji's live datastore migration puts the TenantControlPlane in read-only/freezing mode and requires restarting kubelet on every tenant worker node afterward — that worker-node kubelet restart is a real availability impact and isn't in the proposal; it belongs in the migration runbook and Testing.
- The live datastore switch is triggered by patching
.spec.dataStoreon the nativeTenantControlPlane, but cozystack uses the CAPIKamajiControlPlane, for which upstream docs don't cover a live switch. Open Question 1 is a genuine blocker, and #2859 choosing snapshot/restore is a strong signal the live CAPI path isn't available — please reconcile.
Minor: several code citations have drifted line numbers (the constructs still exist), and §2's full etcd values block should shrink to the minimal etcd.replicas surface per the direction agreed in #3179.
Dismissing the review verdict: converting this to non-blocking feedback since this is a design proposal, not code. The technical points in the review comment still stand as suggestions to reconcile before implementation.
…se all open questions Second draft of the per-cluster etcd proposal. The framing changes and every open question from the first review round is answered. Reframing: the proposal is about completing packages/apps/kubernetes, which since the control-plane / node-pool split (community#8) is the control-plane app. Every control-plane component is rendered by that chart except the one holding all of its state. Retiring the etcd tenant-module wiring is a consequence of fixing that, not the goal. This makes the relationship to community#39 explicit: #39 owns where packages live and how modules declare themselves, this owns what the control-plane app contains. Disjoint files, neither blocks the other. Resolved questions, replacing the five open ones: - Migration: the CAPI provider does re-project dataStoreName onto TenantControlPlane.spec.dataStore on every reconcile, so a live switch works, but it costs a read-only freeze window and a kubelet restart on every tenant worker node. Migration is therefore an operator-initiated maintenance runbook, never an unattended platform migration. - Standalone etcd: survives. Only the Tenant.spec.etcd bool and the _namespace.etcd propagation retire; the chart and its ApplicationDefinition stay. - Default replicas: 3. - Reuse mechanism: shared named templates in cozy-lib, not a subchart. The etcd chart hard-fails unless the release is named etcd, a subchart inherits the parent release name, and the literal etcd-*-tls names are pinned by an immutable spec.tls on adopted clusters. - Footprint: surfaced in NOTES.txt and the app README. Also folded in: the minimal values surface from cozystack#3179 (only etcd.replicas, with the StorageClass derived from it at platform level), etcd-operator v1alpha2 and the Available condition throughout, a name-length section with the real 53-character release cap and the operator's derived names, a delete hook that reaps a per-cluster etcd but never a shared one, corrected source citations, and a migration slot of 56+ rather than the stale 49. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
myasnikovdaniil
changed the title
|
IvanHunters second draft pushed. Framing changed too, it is now about completing
Citations fixed, values block cut down to |
|
Sergei Makarov (@SerjioTT) took your minimal surface as is. Only
On Q1, taking you up on that. Mechanism is settled now (kamaji live switch works, costs a read only freeze plus the kubelet restart on every tenant worker), what nobody has measured is how it behaves on talos workers at fleet scale. Migration e2e gates phase 2, so that is exactly the run that would help. |
|
scooby87 #3179 adopted. The values block now exposes only Framing changed as well, it is a companion to #39 now rather than a competing packaging change. #39 owns where packages live, this one owns what the control plane app contains. |
4 participants
Summary
Design proposal for
packages/apps/kubernetesto render its ownEtcdClusterand its own KamajiDataStore, one per cluster.Since the control plane / node pool split (#8)
apps/kubernetesis the control plane app. It renders kamaji, the certs, konnectivity, csi, kccm, the oidc wiring. The one component it does not render is the one that holds all of the control plane's state, etcd comes from outside instead, from an admin settingspec.etcd: trueon an ancestorTenant. This proposal closes that gap, so a cluster's datastore is as much part of the cluster as its api server.Two things follow from it. A consumer who can create a
Kubernetesapp gets a working cluster with no admin pre step, noawaiting-etcd. And each cluster's control plane state is physically isolated, own raft group, own disk, own quota, own CA, instead of N control planes multiplexed onto one etcd by key prefix.Relation to #39
Companion, not competitor. #39 answers where packages live and how a module declares itself. This one answers what the Kubernetes control plane app is made of, it adds one component to one chart. Disjoint files and neither blocks the other. etcd is the single tenant module whose sharing is the defect, which is why #39 leaves it out and this one picks it up.
Contents
design-proposals/etcd-per-cluster/README.md, filling the template.Second draft. Every open question from the first round is answered rather than deferred, the values surface is cut to
etcd.replicasper cozystack#3179, and the whole document is retargeted atetcd-operator.cozystack.io/v1alpha2after cozystack#2859 and cozystack#3270 merged.Status: Review