From 51e84964d3795ecf925bac9457e3aefcb6d17a57 Mon Sep 17 00:00:00 2001 From: pallavithegod Date: Fri, 16 Oct 2026 15:37:35 +0530 Subject: [PATCH 1/6] docs(dist): explain pd and store recovery signals --- .../hugegraph/hugegraph-helm-operations.md | 17 ++++++++++++++-- .../hugegraph/hugegraph-helm-operations.md | 20 +++++++++++++++++-- 2 files changed, 33 insertions(+), 4 deletions(-) diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md index 0dfaa7383..51df25fce 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -30,10 +30,15 @@ kubectl get pv,storageclass PD 端口为 REST 8620、gRPC 8686、Raft 8610；Store 为 REST 8520、gRPC 8500、Raft 8510；Server HTTP 为 8080；Hubble HTTP 为 8088。 客户端 Service 隐藏 Pod 地址，NodePort 是节点上另外分配的端口，与应用监听端口不同。 -PD `/v1/ready` 需要 Raft leader，`/v1/health` 只说明监听器已启动，Store 等待多数 PD peer ready 后启动。 +PD `/v1/ready` 需要 Raft leader；旧镜像的 `/v1/health` 只说明监听器已启动。Store 等待多数 PD peer ready 后启动。 Server 等待存储的默认上限为 300 秒，可以用 `server.storageWaitTimeoutSeconds` 调整。 本地模板渲染不能验证集群连通性、卷挂载或 JVM 运行容量。 +包含 [PD 致命错误健康检查修复](https://github.com/apache/hugegraph/pull/3358)的镜像，在 Raft 发生终止性错误或关闭后， +`/v1/health` 返回 `503`；普通选举和暂时失去 quorum 时仍返回 `200`。 +快照失败后，先修复磁盘或存储问题再重启 PD：仅释放空间不会重新初始化 JRaft。 +旧镜像仍需保留 chart 为单 PD 配置的基于 readiness 的 liveness 变通方案。 + ### Helm test 检查范围 {#helm-test-coverage} `helm test` 逐 Server Pod 检查图绑定（默认检查 `DEFAULT-hugegraph`，可通过 `server.testGraphAlias` 指定图别名），并要求当前可见关联索引处于 `CREATED` 状态。 @@ -120,6 +125,10 @@ networkPolicy: 每次只删除一个 Store Pod，下一次删除前确认上一个 Pod Ready、Store 心跳新鲜，并从 PD leader 查询每个 shard group 都有完整成员及唯一 leader。 PD 的 `Up` 状态本身不证明 shard 恢复完成。PDB 限制主动驱逐，不限制 StatefulSet 控制器更新。 发现缺成员或无 leader 时停止继续替换。 + +包含 [Store 注册修复](https://github.com/apache/hugegraph/issues/3229)的镜像，在本地 engine 恢复后才注册为 `Up`，正常关闭时报告 `Offline`。 +应先升级 PD 再升级 Store，才能启用恢复检查。PD 的鉴权 Store 响应新增 `heartbeatAgeMillis`（未知时为 `-1`）， +可与配置的心跳间隔比较，识别异常停止后残留的 `Up` 状态。新鲜心跳和本地恢复都不证明 Raft 已追平，仍需保留手动升级屏障。 [运维参考](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#rolling-store-updates-and-recovery)说明这些升级条件， 但不提供完整的逐组诊断或丢盘恢复步骤。 @@ -132,8 +141,12 @@ PD 的 `Up` 状态本身不证明 shard 恢复完成。PDB 限制主动驱逐， 执行前通过鉴权的 `/v1/members` 查询 `data.pdLeader.restUrl`，并确认该 PD 的 `/v1/ready` 返回 `isLeader: true`。 GET 任务 `/v1/task/patrolPartitions` 和 `/v1/task/balanceLeaders` 会改变集群状态，应携带 PD REST 凭据请求已确认的 leader。 成员恢复和 split/move 任务完成后才能执行 leader 均衡；Store 心跳需新鲜，各组成员需完整、健康。 -忙碌任务返回 body `status: 1001`。follower 可能返回空成功，leader 转移是异步的，HTTP 200 或转移计划不代表执行完成。 +忙碌任务返回 body `status: 1001`。旧镜像的 follower 可能返回空成功，leader 转移是异步的，HTTP 200 或转移计划不代表执行完成。 继续维护前重新检查 `/v1/partitions`、`/v1/stores` 和应用错误。 + +包含[任务响应修复](https://github.com/apache/hugegraph/issues/3231)的镜像，follower 返回 body `status: 100`，并给出已知 leader 的 Raft 地址。 +分区巡检还返回 `reallocatedShardGroups` 和 `offlineShardRequests`；空结果表示未调度变更，离线 shard 请求并不证明修复成功。 +跳过的分区均衡返回 `{"status":0,"outcome":"skipped"}`，已执行但无需转移的均衡返回 `{}`。继续操作前仍需验证实际成员和 leader。 [任务操作步骤](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#pd-patrol-and-leader-balance) 说明巡检和均衡的独立操作；维护忙碌时不要循环请求任务。 diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md index c4ceec088..80305733c 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -31,10 +31,16 @@ Store wait and Helm test reuse existing component images with `curl` already ins PD listens on REST 8620, gRPC 8686 and Raft 8610; Store on REST 8520, gRPC 8500 and Raft 8510; Server HTTP on 8080; Hubble HTTP on 8088. Client Services hide individual Pod addresses. NodePort is a separate allocated port on reachable nodes, not the application listener port. -PD `/v1/ready` requires a Raft leader; `/v1/health` only confirms the listener. Store startup waits for a majority of ready PD peers. +PD `/v1/ready` requires a Raft leader. On older images, `/v1/health` only confirms the listener. +Store startup waits for a majority of ready PD peers. Server waits up to `server.storageWaitTimeoutSeconds` (default 300) before its application startup; adjust it for slow storage bootstrap. No local Helm rendering verifies live cluster reachability, volumes or JVM capacity. +Images containing the [fatal PD health fix](https://github.com/apache/hugegraph/pull/3358) return `503` from `/v1/health` +after a terminal Raft error or shutdown; ordinary elections and quorum loss remain live (`200`). +After a failed snapshot, fix the disk or storage problem before restarting PD: freeing space alone does not reinitialize JRaft. +Keep the chart's single-PD readiness-based liveness workaround for older images. + ### Helm test coverage `helm test` validates graph binding on every Server Pod (`DEFAULT-hugegraph` by default, or configured via `server.testGraphAlias`), and requires @@ -127,6 +133,11 @@ Use `store.updateStrategy.type=OnDelete` for a controlled maintenance rollout, a Delete one Store Pod at a time. Before replacing the next, verify the previous Pod is Ready, has a fresh Store heartbeat and each shard group has its full membership and exactly one leader, read from the PD leader. PD's `Up` status alone does not prove completed shard recovery. PDBs limit voluntary eviction, not StatefulSet controller disruption. Stop on missing members or leaders. + +Images containing the [Store registration fix](https://github.com/apache/hugegraph/issues/3229) register `Up` after local engine restore +and report `Offline` on graceful shutdown. Upgrade PD before Store to enable the restore gate. +PD's authenticated Store responses expose `heartbeatAgeMillis` (`-1` if unknown); compare it with the configured heartbeat interval +to reject stale `Up` records after abrupt stops. Fresh heartbeats and local restore do not prove Raft catch-up, so retain the manual barrier. The [operations reference](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#rolling-store-updates-and-recovery) summarizes these rollout conditions; it does not provide a complete per-group diagnostic or disk-loss recovery procedure. @@ -140,8 +151,13 @@ Partition patrol repairs shard membership; leader balance is a separate manual t Before either task, identify the PD leader through authenticated `/v1/members` (`data.pdLeader.restUrl`) and confirm its `/v1/ready` reports `isLeader: true`. The GET tasks `/v1/task/patrolPartitions` and `/v1/task/balanceLeaders` change cluster state; target that confirmed leader with PD REST credentials. Wait for membership recovery and split/move tasks to finish before balancing leaders; Stores must have fresh heartbeats and groups full, healthy membership. -A busy task returns body `status: 1001`. A follower may return empty success, and leader transfers are asynchronous: +A busy task returns body `status: 1001`. Older images may return empty success from a follower, and leader transfers are asynchronous: HTTP 200 or a transfer plan does not prove completion. Recheck `/v1/partitions`, `/v1/stores` and application errors before further maintenance. + +With the [task response fix](https://github.com/apache/hugegraph/issues/3231), followers return body `status: 100`, naming the known leader's Raft address. +Partition patrol also reports `reallocatedShardGroups` and `offlineShardRequests`; empty results mean no changes were scheduled. +Offline-shard requests do not prove successful repairs. A skipped partition balance returns `{"status":0,"outcome":"skipped"}`; +a completed balance with no transfers returns `{}`. Continue to verify actual membership and leaders before proceeding. Follow the [task procedure](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#pd-patrol-and-leader-balance) for the separate patrol and balance steps; do not loop task requests while maintenance is busy. From d28cbfbcb8849d2506dbb6ea7f1e60426b7eb3e6 Mon Sep 17 00:00:00 2001 From: pallavithegod Date: Fri, 16 Oct 2026 18:50:33 +0530 Subject: [PATCH 2/6] docs(dist): clarify store recovery guidance Specify the fixed Store heartbeat period and link the implementing PRs. Explain keepalive expiry during temporary Offline reporting. Keep reconnect and restore guidance aligned in both languages. --- .../quickstart/hugegraph/hugegraph-helm-operations.md | 11 +++++++---- .../quickstart/hugegraph/hugegraph-helm-operations.md | 10 ++++++---- 2 files changed, 13 insertions(+), 8 deletions(-) diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md index 51df25fce..8c92115fd 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -126,9 +126,12 @@ networkPolicy: PD 的 `Up` 状态本身不证明 shard 恢复完成。PDB 限制主动驱逐，不限制 StatefulSet 控制器更新。 发现缺成员或无 leader 时停止继续替换。 -包含 [Store 注册修复](https://github.com/apache/hugegraph/issues/3229)的镜像，在本地 engine 恢复后才注册为 `Up`，正常关闭时报告 `Offline`。 -应先升级 PD 再升级 Store，才能启用恢复检查。PD 的鉴权 Store 响应新增 `heartbeatAgeMillis`（未知时为 `-1`）， -可与配置的心跳间隔比较，识别异常停止后残留的 `Up` 状态。新鲜心跳和本地恢复都不证明 Raft 已追平，仍需保留手动升级屏障。 +包含 [Store 注册修复](https://github.com/apache/hugegraph/pull/3362)的镜像，在本地 engine 恢复后才注册为 `Up`，正常关闭时报告 `Offline`。 +应先升级 PD 再升级 Store，才能启用恢复检查。 +临时 `Offline` 报告保留已有的 keepalive 有效期，但不会续期；Store 未恢复时仍会按原有效期过期。 +本地恢复成功后，心跳流重连会直接注册 `Up`，不再重复恢复。 +PD 的鉴权 Store 响应新增 `heartbeatAgeMillis`（未知时为 `-1`）。Store 心跳周期固定为 30 秒；该值达到 30000 ms 的数倍时，表示 `Up` 记录已陈旧。 +新鲜心跳和本地恢复都不证明 Raft 已追平，仍需保留手动升级屏障。 [运维参考](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#rolling-store-updates-and-recovery)说明这些升级条件， 但不提供完整的逐组诊断或丢盘恢复步骤。 @@ -144,7 +147,7 @@ GET 任务 `/v1/task/patrolPartitions` 和 `/v1/task/balanceLeaders` 会改变 忙碌任务返回 body `status: 1001`。旧镜像的 follower 可能返回空成功，leader 转移是异步的，HTTP 200 或转移计划不代表执行完成。 继续维护前重新检查 `/v1/partitions`、`/v1/stores` 和应用错误。 -包含[任务响应修复](https://github.com/apache/hugegraph/issues/3231)的镜像，follower 返回 body `status: 100`，并给出已知 leader 的 Raft 地址。 +包含[任务响应修复](https://github.com/apache/hugegraph/pull/3359)的镜像，follower 返回 body `status: 100`，并给出已知 leader 的 Raft 地址。 分区巡检还返回 `reallocatedShardGroups` 和 `offlineShardRequests`；空结果表示未调度变更，离线 shard 请求并不证明修复成功。 跳过的分区均衡返回 `{"status":0,"outcome":"skipped"}`，已执行但无需转移的均衡返回 `{}`。继续操作前仍需验证实际成员和 leader。 [任务操作步骤](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#pd-patrol-and-leader-balance) diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md index 80305733c..68ee486ec 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -134,10 +134,12 @@ Delete one Store Pod at a time. Before replacing the next, verify the previous P full membership and exactly one leader, read from the PD leader. PD's `Up` status alone does not prove completed shard recovery. PDBs limit voluntary eviction, not StatefulSet controller disruption. Stop on missing members or leaders. -Images containing the [Store registration fix](https://github.com/apache/hugegraph/issues/3229) register `Up` after local engine restore +Images containing the [Store registration fix](https://github.com/apache/hugegraph/pull/3362) register `Up` after local engine restore and report `Offline` on graceful shutdown. Upgrade PD before Store to enable the restore gate. -PD's authenticated Store responses expose `heartbeatAgeMillis` (`-1` if unknown); compare it with the configured heartbeat interval -to reject stale `Up` records after abrupt stops. Fresh heartbeats and local restore do not prove Raft catch-up, so retain the manual barrier. +Temporary Offline reports preserve the existing keepalive window without renewing it; expiry still applies if the Store does not return. +After successful local restore, stream reconnects register `Up` directly without repeating recovery. +PD's authenticated Store responses expose `heartbeatAgeMillis` (`-1` if unknown); compare it with the Store heartbeat period (fixed at 30 seconds). +An age of several times 30000 ms marks a stale `Up` record. Fresh heartbeats and local restore do not prove Raft catch-up, so retain the manual barrier. The [operations reference](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#rolling-store-updates-and-recovery) summarizes these rollout conditions; it does not provide a complete per-group diagnostic or disk-loss recovery procedure. @@ -154,7 +156,7 @@ Wait for membership recovery and split/move tasks to finish before balancing lea A busy task returns body `status: 1001`. Older images may return empty success from a follower, and leader transfers are asynchronous: HTTP 200 or a transfer plan does not prove completion. Recheck `/v1/partitions`, `/v1/stores` and application errors before further maintenance. -With the [task response fix](https://github.com/apache/hugegraph/issues/3231), followers return body `status: 100`, naming the known leader's Raft address. +With the [task response fix](https://github.com/apache/hugegraph/pull/3359), followers return body `status: 100`, naming the known leader's Raft address. Partition patrol also reports `reallocatedShardGroups` and `offlineShardRequests`; empty results mean no changes were scheduled. Offline-shard requests do not prove successful repairs. A skipped partition balance returns `{"status":0,"outcome":"skipped"}`; a completed balance with no transfers returns `{}`. Continue to verify actual membership and leaders before proceeding. From 5a1f9c68456c924f17a0657890db0ede5545a92e Mon Sep 17 00:00:00 2001 From: imbajin Date: Sat, 17 Oct 2026 21:43:50 +0800 Subject: [PATCH 3/6] Address review comments on Helm recovery signals and probe behavior --- .../hugegraph/hugegraph-helm-operations.md | 11 +++++++---- .../hugegraph/hugegraph-helm-operations.md | 15 ++++++++++----- 2 files changed, 17 insertions(+), 9 deletions(-) diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md index 8c92115fd..fc00fa970 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -37,7 +37,8 @@ Server 等待存储的默认上限为 300 秒，可以用 `server.storageWaitTim 包含 [PD 致命错误健康检查修复](https://github.com/apache/hugegraph/pull/3358)的镜像，在 Raft 发生终止性错误或关闭后， `/v1/health` 返回 `503`；普通选举和暂时失去 quorum 时仍返回 `200`。 快照失败后，先修复磁盘或存储问题再重启 PD：仅释放空间不会重新初始化 JRaft。 -旧镜像仍需保留 chart 为单 PD 配置的基于 readiness 的 liveness 变通方案。 +当 `pd.replicas` 为 1 时，chart 模板默认将探针指向 `/v1/ready`（无论镜像版本），因此单副本 PD 探针在 chart 调整前仍基于 readiness； +使用修复镜像时，可配置 `pd.livenessPath=/v1/health`（同时对 startup 探针生效）以启用健康检查恢复。 ### Helm test 检查范围 {#helm-test-coverage} @@ -130,8 +131,9 @@ PD 的 `Up` 状态本身不证明 shard 恢复完成。PDB 限制主动驱逐， 应先升级 PD 再升级 Store，才能启用恢复检查。 临时 `Offline` 报告保留已有的 keepalive 有效期，但不会续期；Store 未恢复时仍会按原有效期过期。 本地恢复成功后，心跳流重连会直接注册 `Up`，不再重复恢复。 -PD 的鉴权 Store 响应新增 `heartbeatAgeMillis`（未知时为 `-1`）。Store 心跳周期固定为 30 秒；该值达到 30000 ms 的数倍时，表示 `Up` 记录已陈旧。 -新鲜心跳和本地恢复都不证明 Raft 已追平，仍需保留手动升级屏障。 +PD 的鉴权 Store 响应新增 `heartbeatAgeMillis`（未知时为 `-1`）。注意该值由 PD 本地墙上时钟计算（`max(0, currentTimeMillis - lastHeartbeat)`）； +若发生时钟回退，未来时间戳截断为 0 可能误判为新鲜，发生时钟调整后应核对实际心跳时间戳或注册记录。 +Store 心跳周期固定为 30 秒；该值达到 30000 ms 的数倍时，表示 `Up` 记录已陈旧。新鲜心跳和本地恢复都不证明 Raft 已追平，仍需保留手动升级屏障。 [运维参考](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#rolling-store-updates-and-recovery)说明这些升级条件， 但不提供完整的逐组诊断或丢盘恢复步骤。 @@ -149,7 +151,8 @@ GET 任务 `/v1/task/patrolPartitions` 和 `/v1/task/balanceLeaders` 会改变 包含[任务响应修复](https://github.com/apache/hugegraph/pull/3359)的镜像，follower 返回 body `status: 100`，并给出已知 leader 的 Raft 地址。 分区巡检还返回 `reallocatedShardGroups` 和 `offlineShardRequests`；空结果表示未调度变更，离线 shard 请求并不证明修复成功。 -跳过的分区均衡返回 `{"status":0,"outcome":"skipped"}`，已执行但无需转移的均衡返回 `{}`。继续操作前仍需验证实际成员和 leader。 +调用 `/v1/task/balancePartitions` 时，跳过的分区均衡返回 `status: 0` 且 `outcome: skipped`；对 `/v1/task/balanceLeaders`， +已执行但无需转移的均衡返回 `{}`。继续操作前仍需验证实际成员和 leader。 [任务操作步骤](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#pd-patrol-and-leader-balance) 说明巡检和均衡的独立操作；维护忙碌时不要循环请求任务。 diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md index 68ee486ec..ee104eafe 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -39,7 +39,9 @@ No local Helm rendering verifies live cluster reachability, volumes or JVM capac Images containing the [fatal PD health fix](https://github.com/apache/hugegraph/pull/3358) return `503` from `/v1/health` after a terminal Raft error or shutdown; ordinary elections and quorum loss remain live (`200`). After a failed snapshot, fix the disk or storage problem before restarting PD: freeing space alone does not reinitialize JRaft. -Keep the chart's single-PD readiness-based liveness workaround for older images. +Because chart templates default probes to `/v1/ready` when `pd.replicas` is 1, single-PD deployments remain readiness-based +regardless of image until chart updates; for fixed images, override `pd.livenessPath=/v1/health` (which also sets the startup probe path) +to enable health-based probe recovery. ### Helm test coverage @@ -138,8 +140,10 @@ Images containing the [Store registration fix](https://github.com/apache/hugegra and report `Offline` on graceful shutdown. Upgrade PD before Store to enable the restore gate. Temporary Offline reports preserve the existing keepalive window without renewing it; expiry still applies if the Store does not return. After successful local restore, stream reconnects register `Up` directly without repeating recovery. -PD's authenticated Store responses expose `heartbeatAgeMillis` (`-1` if unknown); compare it with the Store heartbeat period (fixed at 30 seconds). -An age of several times 30000 ms marks a stale `Up` record. Fresh heartbeats and local restore do not prove Raft catch-up, so retain the manual barrier. +PD's authenticated Store responses expose `heartbeatAgeMillis` (`-1` if unknown). Because PD calculates this from wall-clock time +(`max(0, currentTimeMillis - lastHeartbeat)`), clock rollback or skew can clamp future timestamps to 0 and falsely report fresh; verify actual timestamps +or registration records after clock shifts. Compare it with the Store heartbeat period (fixed at 30 seconds); an age of several times 30000 ms marks a stale +`Up` record. Fresh heartbeats and local restore do not prove Raft catch-up, so retain the manual barrier. The [operations reference](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#rolling-store-updates-and-recovery) summarizes these rollout conditions; it does not provide a complete per-group diagnostic or disk-loss recovery procedure. @@ -158,8 +162,9 @@ HTTP 200 or a transfer plan does not prove completion. Recheck `/v1/partitions`, With the [task response fix](https://github.com/apache/hugegraph/pull/3359), followers return body `status: 100`, naming the known leader's Raft address. Partition patrol also reports `reallocatedShardGroups` and `offlineShardRequests`; empty results mean no changes were scheduled. -Offline-shard requests do not prove successful repairs. A skipped partition balance returns `{"status":0,"outcome":"skipped"}`; -a completed balance with no transfers returns `{}`. Continue to verify actual membership and leaders before proceeding. +Offline-shard requests do not prove successful repairs. When calling `/v1/task/balancePartitions`, a skipped partition balance returns `status: 0` with +`outcome: skipped`; for `/v1/task/balanceLeaders`, a completed balance with no transfers returns `{}`. Continue to verify actual membership and leaders +before proceeding. Follow the [task procedure](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#pd-patrol-and-leader-balance) for the separate patrol and balance steps; do not loop task requests while maintenance is busy. From c10cb3d0dd604153e94ad3d00f35dc68319dffa2 Mon Sep 17 00:00:00 2001 From: pallavithegod Date: Sun, 18 Oct 2026 16:27:50 +0530 Subject: [PATCH 4/6] docs(dist): distinguish shutdown and restore leases Explain renewal during recovery and expiry after shutdown. Document safe fallback when older PDs reject the shutdown RPC. Keep the English and Chinese operational guidance aligned. --- .../cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md | 4 +++- .../en/docs/quickstart/hugegraph/hugegraph-helm-operations.md | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md index 8c92115fd..f5bc6f31e 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -128,7 +128,9 @@ PD 的 `Up` 状态本身不证明 shard 恢复完成。PDB 限制主动驱逐， 包含 [Store 注册修复](https://github.com/apache/hugegraph/pull/3362)的镜像，在本地 engine 恢复后才注册为 `Up`，正常关闭时报告 `Offline`。 应先升级 PD 再升级 Store，才能启用恢复检查。 -临时 `Offline` 报告保留已有的 keepalive 有效期，但不会续期；Store 未恢复时仍会按原有效期过期。 +关闭报告保留已有的 keepalive 有效期，但不会续期；Store 未返回时仍会按原有效期过期。 +恢复期间，独立工作线程恢复 engine，`Offline` 注册会为已返回的 Store 续期。 +旧版 PD 不支持专用关闭 RPC 时，Store 依赖正常过期，不会退回管理用途的 `Offline` 操作。 本地恢复成功后，心跳流重连会直接注册 `Up`，不再重复恢复。 PD 的鉴权 Store 响应新增 `heartbeatAgeMillis`（未知时为 `-1`）。Store 心跳周期固定为 30 秒；该值达到 30000 ms 的数倍时，表示 `Up` 记录已陈旧。 新鲜心跳和本地恢复都不证明 Raft 已追平，仍需保留手动升级屏障。 diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md index 68ee486ec..408d823c1 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -136,7 +136,9 @@ PDBs limit voluntary eviction, not StatefulSet controller disruption. Stop on mi Images containing the [Store registration fix](https://github.com/apache/hugegraph/pull/3362) register `Up` after local engine restore and report `Offline` on graceful shutdown. Upgrade PD before Store to enable the restore gate. -Temporary Offline reports preserve the existing keepalive window without renewing it; expiry still applies if the Store does not return. +Shutdown reports preserve the existing keepalive window without renewing it; expiry still applies if the Store does not return. +During recovery, Offline registrations renew the returned Store's lease while its engines restore on an owned worker. +Older PDs reject the dedicated shutdown RPC; Store then relies on expiry and never falls back to administrative Offline. After successful local restore, stream reconnects register `Up` directly without repeating recovery. PD's authenticated Store responses expose `heartbeatAgeMillis` (`-1` if unknown); compare it with the Store heartbeat period (fixed at 30 seconds). An age of several times 30000 ms marks a stale `Up` record. Fresh heartbeats and local restore do not prove Raft catch-up, so retain the manual barrier. From b410592b97fcb2873e041400db29e746abd9ff7c Mon Sep 17 00:00:00 2001 From: imbajin Date: Sun, 18 Oct 2026 21:43:30 +0800 Subject: [PATCH 5/6] docs(helm): clarify single-PD probe restart behavior and limitations of /v1/health --- .../docs/quickstart/hugegraph/hugegraph-helm-operations.md | 4 ++-- .../docs/quickstart/hugegraph/hugegraph-helm-operations.md | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md index 7d21015b7..3c9cc9ccf 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -37,8 +37,8 @@ Server 等待存储的默认上限为 300 秒，可以用 `server.storageWaitTim 包含 [PD 致命错误健康检查修复](https://github.com/apache/hugegraph/pull/3358)的镜像，在 Raft 发生终止性错误或关闭后， `/v1/health` 返回 `503`；普通选举和暂时失去 quorum 时仍返回 `200`。 快照失败后，先修复磁盘或存储问题再重启 PD：仅释放空间不会重新初始化 JRaft。 -当 `pd.replicas` 为 1 时，chart 模板默认将探针指向 `/v1/ready`（无论镜像版本），因此单副本 PD 探针在 chart 调整前仍基于 readiness； -使用修复镜像时，可配置 `pd.livenessPath=/v1/health`（同时对 startup 探针生效）以启用健康检查恢复。 +当 `pd.replicas` 为 1 时，chart 模板默认将 startup 与 liveness 探针指向 `/v1/ready`，该路径在发生终止性 Raft 错误后本就会重启单副本 PD。 +若配置 `pd.livenessPath=/v1/health`（同时对 startup 探针生效）改用健康检查，需注意 `/v1/health` 无法检测未能完成初始化而从未进入 ready 状态的 PD。 ### Helm test 检查范围 {#helm-test-coverage} diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md index 2abf7cc50..cad2cf3f1 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -39,9 +39,9 @@ No local Helm rendering verifies live cluster reachability, volumes or JVM capac Images containing the [fatal PD health fix](https://github.com/apache/hugegraph/pull/3358) return `503` from `/v1/health` after a terminal Raft error or shutdown; ordinary elections and quorum loss remain live (`200`). After a failed snapshot, fix the disk or storage problem before restarting PD: freeing space alone does not reinitialize JRaft. -Because chart templates default probes to `/v1/ready` when `pd.replicas` is 1, single-PD deployments remain readiness-based -regardless of image until chart updates; for fixed images, override `pd.livenessPath=/v1/health` (which also sets the startup probe path) -to enable health-based probe recovery. +When `pd.replicas` is 1, chart templates default startup and liveness probes to `/v1/ready`, which already restarts a single PD after a terminal Raft error. +Setting `pd.livenessPath=/v1/health` (which also sets the startup probe path) switches them to health checks, but `/v1/health` does not detect a PD that fails +to initialize its storage and never becomes ready. ### Helm test coverage From 248b7db80e2968fd9c56fbae78ccab92429d8bc7 Mon Sep 17 00:00:00 2001 From: pallavithegod Date: Mon, 19 Oct 2026 22:39:08 +0530 Subject: [PATCH 6/6] docs(dist): clarify recovery signal limits Explain automatic single-PD and multi-PD restarts. Qualify shutdown reports with credential and expiry behavior. Keep placement eligibility and restoration retry guidance aligned. --- .../quickstart/hugegraph/hugegraph-helm-operations.md | 11 +++++++++-- .../quickstart/hugegraph/hugegraph-helm-operations.md | 10 ++++++++-- 2 files changed, 17 insertions(+), 4 deletions(-) diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md index 3c9cc9ccf..0c856cc3c 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -36,8 +36,11 @@ Server 等待存储的默认上限为 300 秒，可以用 `server.storageWaitTim 包含 [PD 致命错误健康检查修复](https://github.com/apache/hugegraph/pull/3358)的镜像，在 Raft 发生终止性错误或关闭后， `/v1/health` 返回 `503`；普通选举和暂时失去 quorum 时仍返回 `200`。 -快照失败后，先修复磁盘或存储问题再重启 PD：仅释放空间不会重新初始化 JRaft。 +快照失败后，应先修复磁盘或存储问题再手动重启 PD；仅释放空间不会重新初始化 JRaft。 +下述探针可能已自动重启故障 PD；若底层存储问题仍未解决，自动重启可能反复发生，不能代替修复。 当 `pd.replicas` 为 1 时，chart 模板默认将 startup 与 liveness 探针指向 `/v1/ready`，该路径在发生终止性 Raft 错误后本就会重启单副本 PD。 +当 `pd.replicas` 大于 1 时，startup 与 liveness 默认使用 `/v1/health`；包含终止状态健康检查修复的镜像也会因 `503` 而自动重启。 + 若配置 `pd.livenessPath=/v1/health`（同时对 startup 探针生效）改用健康检查，需注意 `/v1/health` 无法检测未能完成初始化而从未进入 ready 状态的 PD。 ### Helm test 检查范围 {#helm-test-coverage} @@ -127,10 +130,14 @@ networkPolicy: PD 的 `Up` 状态本身不证明 shard 恢复完成。PDB 限制主动驱逐，不限制 StatefulSet 控制器更新。 发现缺成员或无 leader 时停止继续替换。 -包含 [Store 注册修复](https://github.com/apache/hugegraph/pull/3362)的镜像，在本地 engine 恢复后才注册为 `Up`，正常关闭时报告 `Offline`。 +包含 [Store 注册修复](https://github.com/apache/hugegraph/pull/3362)的镜像，在本地 engine 恢复后才注册为 `Up`，正常关闭时尝试报告 `Offline`；只有 PD 接受报告后才会改变记录。 应先升级 PD 再升级 Store，才能启用恢复检查。 关闭报告保留已有的 keepalive 有效期，但不会续期；Store 未返回时仍会按原有效期过期。 恢复期间，独立工作线程恢复 engine，`Offline` 注册会为已返回的 Store 续期。 +关闭 RPC 要求具有内部特权的凭据：在 Store 中设置与 PD 的 `auth.secret-key` 相同的 `HG_PD_AUTH_SECRET_KEY`。 +报告被拒绝或调用失败时，Store 仍会关闭；PD 保留此前记录，直到已有租约正常过期。 +恢复租约只用于存活跟踪，新分配仍要求 Store 的规范状态为 `Up`。 +恢复失败后，重试间隔逐步退避至 Store 心跳周期；`Offline` 注册最多每五秒续期一次。 旧版 PD 不支持专用关闭 RPC 时，Store 依赖正常过期，不会退回管理用途的 `Offline` 操作。 本地恢复成功后，心跳流重连会直接注册 `Up`，不再重复恢复。 PD 的鉴权 Store 响应新增 `heartbeatAgeMillis`（未知时为 `-1`）。注意该值由 PD 本地墙上时钟计算（`max(0, currentTimeMillis - lastHeartbeat)`）； diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md index cad2cf3f1..6ad0bd340 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -38,8 +38,10 @@ No local Helm rendering verifies live cluster reachability, volumes or JVM capac Images containing the [fatal PD health fix](https://github.com/apache/hugegraph/pull/3358) return `503` from `/v1/health` after a terminal Raft error or shutdown; ordinary elections and quorum loss remain live (`200`). -After a failed snapshot, fix the disk or storage problem before restarting PD: freeing space alone does not reinitialize JRaft. +After a failed snapshot, repair the disk or storage problem before an operator-initiated restart; freeing space alone does not reinitialize JRaft. +The probes below may already restart a failed PD automatically. If the underlying storage fault remains, those restarts can repeat without fixing it. When `pd.replicas` is 1, chart templates default startup and liveness probes to `/v1/ready`, which already restarts a single PD after a terminal Raft error. +When `pd.replicas` is greater than 1, startup and liveness default to `/v1/health`; with the fatal health fix, terminal-state `503` also triggers restarts. Setting `pd.livenessPath=/v1/health` (which also sets the startup probe path) switches them to health checks, but `/v1/health` does not detect a PD that fails to initialize its storage and never becomes ready. @@ -137,9 +139,13 @@ full membership and exactly one leader, read from the PD leader. PD's `Up` statu PDBs limit voluntary eviction, not StatefulSet controller disruption. Stop on missing members or leaders. Images containing the [Store registration fix](https://github.com/apache/hugegraph/pull/3362) register `Up` after local engine restore -and report `Offline` on graceful shutdown. Upgrade PD before Store to enable the restore gate. +and attempt to report `Offline` on graceful shutdown; PD changes its record only if it accepts the report. Upgrade PD before Store to enable the restore gate. Shutdown reports preserve the existing keepalive window without renewing it; expiry still applies if the Store does not return. During recovery, Offline registrations renew the returned Store's lease while its engines restore on an owned worker. +The shutdown RPC requires a privileged internal credential: set `HG_PD_AUTH_SECRET_KEY` in Store to match PD's `auth.secret-key`. +If the report is refused or fails, Store still shuts down and PD retains its previous record until the existing lease expires. +A recovering Store's lease keeps liveness tracking separate from placement: new placements require canonical `Up` state. +Restore retries back off to the heartbeat interval, while Offline registrations renew at most every five seconds. Older PDs reject the dedicated shutdown RPC; Store then relies on expiry and never falls back to administrative Offline. After successful local restore, stream reconnects register `Up` directly without repeating recovery. PD's authenticated Store responses expose `heartbeatAgeMillis` (`-1` if unknown). Because PD calculates this from wall-clock time