Received the following error after applying changes to the YAML definition:
[FATAL] Server encountered an error and is shutting down: store init failed to load formats after 10 retries: erasure read quorum
I have a tenant with 3 servers each having 1-drive pool. The pods came up well and were Ready.
Then I added credsSecret property referencing and existing Secret. Once changed I applied these changes. Since the default rollout strategy has MaxUnavailable: 1, it restarted one random pod, which failed.
I described the pod to see the following error:
$ kubectl describe pod prod-rustfs-mse-nvme-2 -n mse
Name: prod-rustfs-mse-nvme-2
Namespace: mse
Priority: 0
...
Status: Running
...
Controlled By: StatefulSet/prod-rustfs-finik-nvme
Containers:
rustfs:
Container ID: containerd://56a074fb2ce21688cedaf8ff5f2109b236c4337c359c20894e81892f9051075b
Image: rustfs/rustfs:1.0.0-beta.8
Image ID: docker.io/rustfs/rustfs@sha256:fa19210ac4697c79d7ccca1ec9b0eb91aebacc6691991ffb14014bb3c67e6cc3
Ports: 9000/TCP (http), 9001/TCP (console)
Host Ports: 0/TCP (http), 0/TCP (console)
State: Running
Started: Sat, 27 Jun 2026 18:09:35 +0600
Last State: Terminated
Reason: Error
Message: Initializing data directories:
Initializing log directory: /logs
Starting: /usr/bin/rustfs
[FATAL] Server encountered an error and is shutting down: store init failed to load formats after 10 retries: erasure read quorum
Exit Code: 1
Started: Sat, 27 Jun 2026 18:06:34 +0600
Finished: Sat, 27 Jun 2026 18:08:41 +0600
Ready: False
Restart Count: 4
Limits:
cpu: 2
memory: 2Gi
Requests:
cpu: 1
memory: 1Gi
Liveness: http-get http://:9000/health delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:9000/health/ready delay=0s timeout=1s period=10s #success=1 #failure=3
Startup: http-get http://:9000/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
RUSTFS_ADDRESS: 0.0.0.0:9000
RUSTFS_CONSOLE_ADDRESS: 0.0.0.0:9001
RUSTFS_CONSOLE_ENABLE: true
RUSTFS_ACCESS_KEY: <set to the key 'accesskey' in secret 'prod-rustfs-auth'> Optional: false
RUSTFS_SECRET_KEY: <set to the key 'secretkey' in secret 'prod-rustfs-auth'> Optional: false
RUST_LOG: info
RUSTFS_VOLUMES: http://prod-rustfs-mse-nvme-{0...2}.prod-rustfs-hl.mse.svc.k8s.mse.cloud:9000/data/rustfs{0...0}
Mounts:
/data/rustfs0 from vol-0 (rw)
...
Volumes:
vol-0:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: vol-0-prod-rustfs-mse-nvme-2
ReadOnly: false
kube-api-access-54sb7:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
Optional: false
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 11m default-scheduler Successfully assigned mse/prod-rustfs-mse-nvme-2 to north-10.k8s.mse.cloud
Normal Pulled 115s (x5 over 11m) kubelet Container image "rustfs/rustfs:1.0.0-beta.8" already present on machine and can be accessed by the pod
Normal Created 115s (x5 over 11m) kubelet Container created
Normal Started 115s (x5 over 11m) kubelet Container started
Warning Unhealthy 90s (x57 over 11m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
Does this mean that we cannot change the admin credentials once the cluster has been started? It should be supported to allow credentials rotation.
Received the following error after applying changes to the YAML definition:
I have a tenant with 3 servers each having 1-drive pool. The pods came up well and were
Ready.Then I added
credsSecretproperty referencing and existingSecret. Once changed I applied these changes. Since the default rollout strategy hasMaxUnavailable: 1, it restarted one random pod, which failed.I described the pod to see the following error:
Does this mean that we cannot change the admin credentials once the cluster has been started? It should be supported to allow credentials rotation.