diff --git a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md index 0dfaa7383..0c856cc3c 100644 --- a/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/cn/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -30,10 +30,19 @@ kubectl get pv,storageclass PD 端口为 REST 8620、gRPC 8686、Raft 8610;Store 为 REST 8520、gRPC 8500、Raft 8510;Server HTTP 为 8080;Hubble HTTP 为 8088。 客户端 Service 隐藏 Pod 地址,NodePort 是节点上另外分配的端口,与应用监听端口不同。 -PD `/v1/ready` 需要 Raft leader,`/v1/health` 只说明监听器已启动,Store 等待多数 PD peer ready 后启动。 +PD `/v1/ready` 需要 Raft leader;旧镜像的 `/v1/health` 只说明监听器已启动。Store 等待多数 PD peer ready 后启动。 Server 等待存储的默认上限为 300 秒,可以用 `server.storageWaitTimeoutSeconds` 调整。 本地模板渲染不能验证集群连通性、卷挂载或 JVM 运行容量。 +包含 [PD 致命错误健康检查修复](https://github.com/apache/hugegraph/pull/3358)的镜像,在 Raft 发生终止性错误或关闭后, +`/v1/health` 返回 `503`;普通选举和暂时失去 quorum 时仍返回 `200`。 +快照失败后,应先修复磁盘或存储问题再手动重启 PD;仅释放空间不会重新初始化 JRaft。 +下述探针可能已自动重启故障 PD;若底层存储问题仍未解决,自动重启可能反复发生,不能代替修复。 +当 `pd.replicas` 为 1 时,chart 模板默认将 startup 与 liveness 探针指向 `/v1/ready`,该路径在发生终止性 Raft 错误后本就会重启单副本 PD。 +当 `pd.replicas` 大于 1 时,startup 与 liveness 默认使用 `/v1/health`;包含终止状态健康检查修复的镜像也会因 `503` 而自动重启。 + +若配置 `pd.livenessPath=/v1/health`(同时对 startup 探针生效)改用健康检查,需注意 `/v1/health` 无法检测未能完成初始化而从未进入 ready 状态的 PD。 + ### Helm test 检查范围 {#helm-test-coverage} `helm test` 逐 Server Pod 检查图绑定(默认检查 `DEFAULT-hugegraph`,可通过 `server.testGraphAlias` 指定图别名),并要求当前可见关联索引处于 `CREATED` 状态。 @@ -120,6 +129,20 @@ networkPolicy: 每次只删除一个 Store Pod,下一次删除前确认上一个 Pod Ready、Store 心跳新鲜,并从 PD leader 查询每个 shard group 都有完整成员及唯一 leader。 PD 的 `Up` 状态本身不证明 shard 恢复完成。PDB 限制主动驱逐,不限制 StatefulSet 控制器更新。 发现缺成员或无 leader 时停止继续替换。 + +包含 [Store 注册修复](https://github.com/apache/hugegraph/pull/3362)的镜像,在本地 engine 恢复后才注册为 `Up`,正常关闭时尝试报告 `Offline`;只有 PD 接受报告后才会改变记录。 +应先升级 PD 再升级 Store,才能启用恢复检查。 +关闭报告保留已有的 keepalive 有效期,但不会续期;Store 未返回时仍会按原有效期过期。 +恢复期间,独立工作线程恢复 engine,`Offline` 注册会为已返回的 Store 续期。 +关闭 RPC 要求具有内部特权的凭据:在 Store 中设置与 PD 的 `auth.secret-key` 相同的 `HG_PD_AUTH_SECRET_KEY`。 +报告被拒绝或调用失败时,Store 仍会关闭;PD 保留此前记录,直到已有租约正常过期。 +恢复租约只用于存活跟踪,新分配仍要求 Store 的规范状态为 `Up`。 +恢复失败后,重试间隔逐步退避至 Store 心跳周期;`Offline` 注册最多每五秒续期一次。 +旧版 PD 不支持专用关闭 RPC 时,Store 依赖正常过期,不会退回管理用途的 `Offline` 操作。 +本地恢复成功后,心跳流重连会直接注册 `Up`,不再重复恢复。 +PD 的鉴权 Store 响应新增 `heartbeatAgeMillis`(未知时为 `-1`)。注意该值由 PD 本地墙上时钟计算(`max(0, currentTimeMillis - lastHeartbeat)`); +若发生时钟回退,未来时间戳截断为 0 可能误判为新鲜,发生时钟调整后应核对实际心跳时间戳或注册记录。 +Store 心跳周期固定为 30 秒;该值达到 30000 ms 的数倍时,表示 `Up` 记录已陈旧。新鲜心跳和本地恢复都不证明 Raft 已追平,仍需保留手动升级屏障。 [运维参考](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#rolling-store-updates-and-recovery)说明这些升级条件, 但不提供完整的逐组诊断或丢盘恢复步骤。 @@ -132,8 +155,13 @@ PD 的 `Up` 状态本身不证明 shard 恢复完成。PDB 限制主动驱逐, 执行前通过鉴权的 `/v1/members` 查询 `data.pdLeader.restUrl`,并确认该 PD 的 `/v1/ready` 返回 `isLeader: true`。 GET 任务 `/v1/task/patrolPartitions` 和 `/v1/task/balanceLeaders` 会改变集群状态,应携带 PD REST 凭据请求已确认的 leader。 成员恢复和 split/move 任务完成后才能执行 leader 均衡;Store 心跳需新鲜,各组成员需完整、健康。 -忙碌任务返回 body `status: 1001`。follower 可能返回空成功,leader 转移是异步的,HTTP 200 或转移计划不代表执行完成。 +忙碌任务返回 body `status: 1001`。旧镜像的 follower 可能返回空成功,leader 转移是异步的,HTTP 200 或转移计划不代表执行完成。 继续维护前重新检查 `/v1/partitions`、`/v1/stores` 和应用错误。 + +包含[任务响应修复](https://github.com/apache/hugegraph/pull/3359)的镜像,follower 返回 body `status: 100`,并给出已知 leader 的 Raft 地址。 +分区巡检还返回 `reallocatedShardGroups` 和 `offlineShardRequests`;空结果表示未调度变更,离线 shard 请求并不证明修复成功。 +调用 `/v1/task/balancePartitions` 时,跳过的分区均衡返回 `status: 0` 且 `outcome: skipped`;对 `/v1/task/balanceLeaders`, +已执行但无需转移的均衡返回 `{}`。继续操作前仍需验证实际成员和 leader。 [任务操作步骤](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#pd-patrol-and-leader-balance) 说明巡检和均衡的独立操作;维护忙碌时不要循环请求任务。 diff --git a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md index c4ceec088..6ad0bd340 100644 --- a/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md +++ b/content/en/docs/quickstart/hugegraph/hugegraph-helm-operations.md @@ -31,10 +31,20 @@ Store wait and Helm test reuse existing component images with `curl` already ins PD listens on REST 8620, gRPC 8686 and Raft 8610; Store on REST 8520, gRPC 8500 and Raft 8510; Server HTTP on 8080; Hubble HTTP on 8088. Client Services hide individual Pod addresses. NodePort is a separate allocated port on reachable nodes, not the application listener port. -PD `/v1/ready` requires a Raft leader; `/v1/health` only confirms the listener. Store startup waits for a majority of ready PD peers. +PD `/v1/ready` requires a Raft leader. On older images, `/v1/health` only confirms the listener. +Store startup waits for a majority of ready PD peers. Server waits up to `server.storageWaitTimeoutSeconds` (default 300) before its application startup; adjust it for slow storage bootstrap. No local Helm rendering verifies live cluster reachability, volumes or JVM capacity. +Images containing the [fatal PD health fix](https://github.com/apache/hugegraph/pull/3358) return `503` from `/v1/health` +after a terminal Raft error or shutdown; ordinary elections and quorum loss remain live (`200`). +After a failed snapshot, repair the disk or storage problem before an operator-initiated restart; freeing space alone does not reinitialize JRaft. +The probes below may already restart a failed PD automatically. If the underlying storage fault remains, those restarts can repeat without fixing it. +When `pd.replicas` is 1, chart templates default startup and liveness probes to `/v1/ready`, which already restarts a single PD after a terminal Raft error. +When `pd.replicas` is greater than 1, startup and liveness default to `/v1/health`; with the fatal health fix, terminal-state `503` also triggers restarts. +Setting `pd.livenessPath=/v1/health` (which also sets the startup probe path) switches them to health checks, but `/v1/health` does not detect a PD that fails +to initialize its storage and never becomes ready. + ### Helm test coverage `helm test` validates graph binding on every Server Pod (`DEFAULT-hugegraph` by default, or configured via `server.testGraphAlias`), and requires @@ -127,6 +137,21 @@ Use `store.updateStrategy.type=OnDelete` for a controlled maintenance rollout, a Delete one Store Pod at a time. Before replacing the next, verify the previous Pod is Ready, has a fresh Store heartbeat and each shard group has its full membership and exactly one leader, read from the PD leader. PD's `Up` status alone does not prove completed shard recovery. PDBs limit voluntary eviction, not StatefulSet controller disruption. Stop on missing members or leaders. + +Images containing the [Store registration fix](https://github.com/apache/hugegraph/pull/3362) register `Up` after local engine restore +and attempt to report `Offline` on graceful shutdown; PD changes its record only if it accepts the report. Upgrade PD before Store to enable the restore gate. +Shutdown reports preserve the existing keepalive window without renewing it; expiry still applies if the Store does not return. +During recovery, Offline registrations renew the returned Store's lease while its engines restore on an owned worker. +The shutdown RPC requires a privileged internal credential: set `HG_PD_AUTH_SECRET_KEY` in Store to match PD's `auth.secret-key`. +If the report is refused or fails, Store still shuts down and PD retains its previous record until the existing lease expires. +A recovering Store's lease keeps liveness tracking separate from placement: new placements require canonical `Up` state. +Restore retries back off to the heartbeat interval, while Offline registrations renew at most every five seconds. +Older PDs reject the dedicated shutdown RPC; Store then relies on expiry and never falls back to administrative Offline. +After successful local restore, stream reconnects register `Up` directly without repeating recovery. +PD's authenticated Store responses expose `heartbeatAgeMillis` (`-1` if unknown). Because PD calculates this from wall-clock time +(`max(0, currentTimeMillis - lastHeartbeat)`), clock rollback or skew can clamp future timestamps to 0 and falsely report fresh; verify actual timestamps +or registration records after clock shifts. Compare it with the Store heartbeat period (fixed at 30 seconds); an age of several times 30000 ms marks a stale +`Up` record. Fresh heartbeats and local restore do not prove Raft catch-up, so retain the manual barrier. The [operations reference](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#rolling-store-updates-and-recovery) summarizes these rollout conditions; it does not provide a complete per-group diagnostic or disk-loss recovery procedure. @@ -140,8 +165,14 @@ Partition patrol repairs shard membership; leader balance is a separate manual t Before either task, identify the PD leader through authenticated `/v1/members` (`data.pdLeader.restUrl`) and confirm its `/v1/ready` reports `isLeader: true`. The GET tasks `/v1/task/patrolPartitions` and `/v1/task/balanceLeaders` change cluster state; target that confirmed leader with PD REST credentials. Wait for membership recovery and split/move tasks to finish before balancing leaders; Stores must have fresh heartbeats and groups full, healthy membership. -A busy task returns body `status: 1001`. A follower may return empty success, and leader transfers are asynchronous: +A busy task returns body `status: 1001`. Older images may return empty success from a follower, and leader transfers are asynchronous: HTTP 200 or a transfer plan does not prove completion. Recheck `/v1/partitions`, `/v1/stores` and application errors before further maintenance. + +With the [task response fix](https://github.com/apache/hugegraph/pull/3359), followers return body `status: 100`, naming the known leader's Raft address. +Partition patrol also reports `reallocatedShardGroups` and `offlineShardRequests`; empty results mean no changes were scheduled. +Offline-shard requests do not prove successful repairs. When calling `/v1/task/balancePartitions`, a skipped partition balance returns `status: 0` with +`outcome: skipped`; for `/v1/task/balanceLeaders`, a completed balance with no transfers returns `{}`. Continue to verify actual membership and leaders +before proceeding. Follow the [task procedure](https://github.com/apache/hugegraph/blob/master/helm/hugegraph/docs/operations.md#pd-patrol-and-leader-balance) for the separate patrol and balance steps; do not loop task requests while maintenance is busy.