Skip to content

docs(dist): explain pd and store recovery signals #532

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pallavithegod wants to merge 11 commits into
base: master
Choose a base branch
from
Open

docs(dist): explain pd and store recovery signals #532

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
51e8496
docs(dist): explain pd and store recovery signals
pallavithegod Oct 16, 2026
d28cbfb
docs(dist): clarify store recovery guidance
pallavithegod Oct 16, 2026
12797f3
Merge remote-tracking branch 'origin/master' into docs/pd-store-recov…
imbajin Oct 17, 2026
5a1f9c6
Address review comments on Helm recovery signals and probe behavior
imbajin Oct 17, 2026
c10cb3d
docs(dist): distinguish shutdown and restore leases
pallavithegod Oct 18, 2026
9dd9746
docs(dist): integrate recovery guidance updates
pallavithegod Oct 18, 2026
a7a1697
Merge remote-tracking branch 'origin/master' into docs/pd-store-recov…
imbajin Oct 18, 2026
b410592
docs(helm): clarify single-PD probe restart behavior and limitations …
imbajin Oct 18, 2026
0cea4cf
Merge remote-tracking branch 'origin/master' into docs/pd-store-recov…
imbajin Oct 18, 2026
248b7db
docs(dist): clarify recovery signal limits
pallavithegod Oct 19, 2026
63c0181
Merge remote-tracking branch 'origin/master' into docs/pd-store-recov…
imbajin Oct 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 30 additions & 2 deletions content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,19 @@ kubectl get pv,storageclass

PD 端口为 REST 8620、gRPC 8686、Raft 8610；Store 为 REST 8520、gRPC 8500、Raft 8510；Server HTTP 为 8080；Hubble HTTP 为 8088。
客户端 Service 隐藏 Pod 地址，NodePort 是节点上另外分配的端口，与应用监听端口不同。
PD `/v1/ready` 需要 Raft leader，`/v1/health` 只说明监听器已启动，Store 等待多数 PD peer ready 后启动。
PD `/v1/ready` 需要 Raft leader；旧镜像的 `/v1/health` 只说明监听器已启动。Store 等待多数 PD peer ready 后启动。
Server 等待存储的默认上限为 300 秒，可以用 `server.storageWaitTimeoutSeconds` 调整。
本地模板渲染不能验证集群连通性、卷挂载或 JVM 运行容量。

包含 [PD 致命错误健康检查修复](https://github.com/apache/hugegraph/pull/3358)的镜像，在 Raft 发生终止性错误或关闭后，
`/v1/health` 返回 `503`；普通选举和暂时失去 quorum 时仍返回 `200`。
快照失败后，应先修复磁盘或存储问题再手动重启 PD；仅释放空间不会重新初始化 JRaft。
下述探针可能已自动重启故障 PD；若底层存储问题仍未解决，自动重启可能反复发生，不能代替修复。
当 `pd.replicas` 为 1 时，chart 模板默认将 startup 与 liveness 探针指向 `/v1/ready`，该路径在发生终止性 Raft 错误后本就会重启单副本 PD。
当 `pd.replicas` 大于 1 时，startup 与 liveness 默认使用 `/v1/health`；包含终止状态健康检查修复的镜像也会因 `503` 而自动重启。

若配置 `pd.livenessPath=/v1/health`（同时对 startup 探针生效）改用健康检查，需注意 `/v1/health` 无法检测未能完成初始化而从未进入 ready 状态的 PD。

### Helm test 检查范围 {#helm-test-coverage}

`helm test` 逐 Server Pod 检查图绑定（默认检查 `DEFAULT-hugegraph`，可通过 `server.testGraphAlias` 指定图别名），并要求当前可见关联索引处于 `CREATED` 状态。
Expand Down Expand Up @@ -120,6 +129,20 @@ networkPolicy:
每次只删除一个 Store Pod，下一次删除前确认上一个 Pod Ready、Store 心跳新鲜，并从 PD leader 查询每个 shard group 都有完整成员及唯一 leader。
PD 的 `Up` 状态本身不证明 shard 恢复完成。PDB 限制主动驱逐，不限制 StatefulSet 控制器更新。
发现缺成员或无 leader 时停止继续替换。

包含 [Store 注册修复](https://github.com/apache/hugegraph/pull/3362)的镜像，在本地 engine 恢复后才注册为 `Up`，正常关闭时尝试报告 `Offline`；只有 PD 接受报告后才会改变记录。
应先升级 PD 再升级 Store，才能启用恢复检查。
关闭报告保留已有的 keepalive 有效期，但不会续期；Store 未返回时仍会按原有效期过期。
恢复期间，独立工作线程恢复 engine，`Offline` 注册会为已返回的 Store 续期。
关闭 RPC 要求具有内部特权的凭据：在 Store 中设置与 PD 的 `auth.secret-key` 相同的 `HG_PD_AUTH_SECRET_KEY`。
报告被拒绝或调用失败时，Store 仍会关闭；PD 保留此前记录，直到已有租约正常过期。
恢复租约只用于存活跟踪，新分配仍要求 Store 的规范状态为 `Up`。
恢复失败后，重试间隔逐步退避至 Store 心跳周期；`Offline` 注册最多每五秒续期一次。
旧版 PD 不支持专用关闭 RPC 时，Store 依赖正常过期，不会退回管理用途的 `Offline` 操作。
本地恢复成功后，心跳流重连会直接注册 `Up`，不再重复恢复。
PD 的鉴权 Store 响应新增 `heartbeatAgeMillis`（未知时为 `-1`）。注意该值由 PD 本地墙上时钟计算（`max(0, currentTimeMillis - lastHeartbeat)`）；
若发生时钟回退，未来时间戳截断为 0 可能误判为新鲜，发生时钟调整后应核对实际心跳时间戳或注册记录。
Store 心跳周期固定为 30 秒；该值达到 30000 ms 的数倍时，表示 `Up` 记录已陈旧。新鲜心跳和本地恢复都不证明 Raft 已追平，仍需保留手动升级屏障。
[运维参考](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#rolling-store-updates-and-recovery)说明这些升级条件，
但不提供完整的逐组诊断或丢盘恢复步骤。

Expand All @@ -132,8 +155,13 @@ PD 的 `Up` 状态本身不证明 shard 恢复完成。PDB 限制主动驱逐，
执行前通过鉴权的 `/v1/members` 查询 `data.pdLeader.restUrl`，并确认该 PD 的 `/v1/ready` 返回 `isLeader: true`。
GET 任务 `/v1/task/patrolPartitions` 和 `/v1/task/balanceLeaders` 会改变集群状态，应携带 PD REST 凭据请求已确认的 leader。
成员恢复和 split/move 任务完成后才能执行 leader 均衡；Store 心跳需新鲜，各组成员需完整、健康。
忙碌任务返回 body `status: 1001`。follower 可能返回空成功，leader 转移是异步的，HTTP 200 或转移计划不代表执行完成。
忙碌任务返回 body `status: 1001`。旧镜像的 follower 可能返回空成功，leader 转移是异步的，HTTP 200 或转移计划不代表执行完成。
继续维护前重新检查 `/v1/partitions`、`/v1/stores` 和应用错误。

包含[任务响应修复](https://github.com/apache/hugegraph/pull/3359)的镜像，follower 返回 body `status: 100`，并给出已知 leader 的 Raft 地址。
分区巡检还返回 `reallocatedShardGroups` 和 `offlineShardRequests`；空结果表示未调度变更，离线 shard 请求并不证明修复成功。
调用 `/v1/task/balancePartitions` 时，跳过的分区均衡返回 `status: 0` 且 `outcome: skipped`；对 `/v1/task/balanceLeaders`，
已执行但无需转移的均衡返回 `{}`。继续操作前仍需验证实际成员和 leader。
[任务操作步骤](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#pd-patrol-and-leader-balance)
说明巡检和均衡的独立操作；维护忙碌时不要循环请求任务。

Expand Down
35 changes: 33 additions & 2 deletions content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,20 @@ Store wait and Helm test reuse existing component images with `curl` already ins

PD listens on REST 8620, gRPC 8686 and Raft 8610; Store on REST 8520, gRPC 8500 and Raft 8510; Server HTTP on 8080; Hubble HTTP on 8088.
Client Services hide individual Pod addresses. NodePort is a separate allocated port on reachable nodes, not the application listener port.
PD `/v1/ready` requires a Raft leader; `/v1/health` only confirms the listener. Store startup waits for a majority of ready PD peers.
PD `/v1/ready` requires a Raft leader. On older images, `/v1/health` only confirms the listener.
Store startup waits for a majority of ready PD peers.
Server waits up to `server.storageWaitTimeoutSeconds` (default 300) before its application startup; adjust it for slow storage bootstrap.
No local Helm rendering verifies live cluster reachability, volumes or JVM capacity.

Images containing the [fatal PD health fix](https://github.com/apache/hugegraph/pull/3358) return `503` from `/v1/health`
after a terminal Raft error or shutdown; ordinary elections and quorum loss remain live (`200`).
Comment thread
pallavithegod marked this conversation as resolved.
After a failed snapshot, repair the disk or storage problem before an operator-initiated restart; freeing space alone does not reinitialize JRaft.
The probes below may already restart a failed PD automatically. If the underlying storage fault remains, those restarts can repeat without fixing it.
When `pd.replicas` is 1, chart templates default startup and liveness probes to `/v1/ready`, which already restarts a single PD after a terminal Raft error.

@imbajin imbajin Oct 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Important: Blocking: no. Summary: This restart guarantee is unqualified for older PD images, but some images that predate /v1/ready do not serve that endpoint; an unknown-path request can return HTTP 200 with an Unauthorized body, which satisfies Kubernetes' status-only httpGet probe and will not restart a terminal PD. Please limit this sentence to images that include the endpoint and identify the supported image boundary. Evidence: The merged #3358 API-reference note documents the unknown-path 200 response and warns that a status-only probe can treat it as ready.

When `pd.replicas` is greater than 1, startup and liveness default to `/v1/health`; with the fatal health fix, terminal-state `503` also triggers restarts.
Setting `pd.livenessPath=/v1/health` (which also sets the startup probe path) switches them to health checks, but `/v1/health` does not detect a PD that fails
to initialize its storage and never becomes ready.

### Helm test coverage

`helm test` validates graph binding on every Server Pod (`DEFAULT-hugegraph` by default, or configured via `server.testGraphAlias`), and requires
Expand Down Expand Up @@ -127,6 +137,21 @@ Use `store.updateStrategy.type=OnDelete` for a controlled maintenance rollout, a
Delete one Store Pod at a time. Before replacing the next, verify the previous Pod is Ready, has a fresh Store heartbeat and each shard group has its
full membership and exactly one leader, read from the PD leader. PD's `Up` status alone does not prove completed shard recovery.
PDBs limit voluntary eviction, not StatefulSet controller disruption. Stop on missing members or leaders.

Images containing the [Store registration fix](https://github.com/apache/hugegraph/pull/3362) register `Up` after local engine restore
and attempt to report `Offline` on graceful shutdown; PD changes its record only if it accepts the report. Upgrade PD before Store to enable the restore gate.
Shutdown reports preserve the existing keepalive window without renewing it; expiry still applies if the Store does not return.
During recovery, Offline registrations renew the returned Store's lease while its engines restore on an owned worker.
The shutdown RPC requires a privileged internal credential: set `HG_PD_AUTH_SECRET_KEY` in Store to match PD's `auth.secret-key`.

@bitflicker64 bitflicker64 Oct 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: On this Helm page, "set HG_PD_AUTH_SECRET_KEY in Store" reads as a manual step, but the chart in apache/hugegraph#3362 (head ebdc8386) already does it. store-statefulset.yaml adds HG_PD_AUTH_SECRET_KEY from secretKeyRef hugegraph.pd.authSecretName / hugegraph.pd.authSecretKey, the same Secret and key that pd-statefulset.yaml gives PD, and credential_contract_test.yaml asserts it for pd.auth.existingSecret and pd.auth.key. The only way to follow this sentence with the chart is store.extraEnv, which renders after the chart's entry, so a mismatched value silently replaces the chart's Secret reference and the shutdown report is refused (Store then falls back to lease expiry). The open review on #3362 also asks for this name to become reserved in store.extraEnv, which would make the step fail to render.

Please say that the chart passes PD's auth Secret (pd.auth.existingSecret / pd.auth.key) to Store automatically, and keep the manual HG_PD_AUTH_SECRET_KEY instruction only for deployments outside this chart or chart versions without #3362. Make the same change to line 137 of the Chinese page.

@imbajin imbajin Oct 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Important: Blocking: no. Summary: This is a shared privileged secret, not a per-Store identity: any Store able to read it can invoke ReportStoreShutdown with another Store ID. Please state that scope and restrict secret access to trusted internal Store workloads. Evidence: #3362 StoreShutdownAuthentication gates on the shared credential, while PDService.reportStoreShutdown forwards the request's Store ID directly to StoreNodeService.reportStoreShutdown.

If the report is refused or fails, Store still shuts down and PD retains its previous record until the existing lease expires.

@imbajin imbajin Oct 20, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Important: Blocking: no. Summary: A transport failure can occur after PD has persisted the Store as Offline but before Store receives the response, so an RPC error does not prove PD retained the prior record. Please qualify this as the outcome only when PD did not process the report; for an ambiguous result, advise checking the PD record. Evidence: #3362 PDService calls StoreNodeService.reportStoreShutdown before sending the response, and that method persists the Offline state before returning.

A recovering Store's lease keeps liveness tracking separate from placement: new placements require canonical `Up` state.
Restore retries back off to the heartbeat interval, while Offline registrations renew at most every five seconds.
Older PDs reject the dedicated shutdown RPC; Store then relies on expiry and never falls back to administrative Offline.
After successful local restore, stream reconnects register `Up` directly without repeating recovery.
PD's authenticated Store responses expose `heartbeatAgeMillis` (`-1` if unknown). Because PD calculates this from wall-clock time
(`max(0, currentTimeMillis - lastHeartbeat)`), clock rollback or skew can clamp future timestamps to 0 and falsely report fresh; verify actual timestamps
or registration records after clock shifts. Compare it with the Store heartbeat period (fixed at 30 seconds); an age of several times 30000 ms marks a stale
`Up` record. Fresh heartbeats and local restore do not prove Raft catch-up, so retain the manual barrier.
The [operations reference](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#rolling-store-updates-and-recovery)
summarizes these rollout conditions;
it does not provide a complete per-group diagnostic or disk-loss recovery procedure.
Expand All @@ -140,8 +165,14 @@ Partition patrol repairs shard membership; leader balance is a separate manual t
Before either task, identify the PD leader through authenticated `/v1/members` (`data.pdLeader.restUrl`) and confirm its `/v1/ready` reports `isLeader: true`.
The GET tasks `/v1/task/patrolPartitions` and `/v1/task/balanceLeaders` change cluster state; target that confirmed leader with PD REST credentials.
Wait for membership recovery and split/move tasks to finish before balancing leaders; Stores must have fresh heartbeats and groups full, healthy membership.
A busy task returns body `status: 1001`. A follower may return empty success, and leader transfers are asynchronous:
A busy task returns body `status: 1001`. Older images may return empty success from a follower, and leader transfers are asynchronous:
HTTP 200 or a transfer plan does not prove completion. Recheck `/v1/partitions`, `/v1/stores` and application errors before further maintenance.

With the [task response fix](https://github.com/apache/hugegraph/pull/3359), followers return body `status: 100`, naming the known leader's Raft address.
Partition patrol also reports `reallocatedShardGroups` and `offlineShardRequests`; empty results mean no changes were scheduled.
Offline-shard requests do not prove successful repairs. When calling `/v1/task/balancePartitions`, a skipped partition balance returns `status: 0` with
`outcome: skipped`; for `/v1/task/balanceLeaders`, a completed balance with no transfers returns `{}`. Continue to verify actual membership and leaders
before proceeding.
Follow the [task procedure](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#pd-patrol-and-leader-balance)
for the separate patrol and balance steps; do not loop task requests while maintenance is busy.

Expand Down