fix(store): gate up registration on restore and report offline on shutdown

docker/README.md

@@ -249,6 +249,44 @@ Open `http://localhost:8088` and sign in as `admin` with the password from `.env
 
 </details>
 
+### Store registration during restart
+
+Stores first register `Offline` to obtain their persisted identity, restore local
+partition engines, and then register `Up`. A restore failure keeps the Store
+offline and the heartbeat loop retries it. Graceful shutdown stops heartbeat
+producers before reporting `Offline` to PD; if PD is unavailable, normal heartbeat
+expiry remains the fallback. A temporary stop does not request permanent retirement.
+Shutdown reports retain the existing keepalive window without extending it, so a
+quick restart does not fail the cluster's initial-store-count gate. A Store that
+does not return is still removed by normal expiry. Stream reconnects after a
+successful restore register `Up` directly without restoring the engines again.
+Restoration attempts every local partition before reporting any failures.
+Recovery runs on an owned worker while the heartbeat thread renews the restoring
+Store's lease through Offline registrations. A slow or failed partition therefore
+does not let a returned Store expire from the initial-store-count gate.
+Partition heartbeat reports wait until the pulse stream starts after restoration.
+
+Upgrade PD before Store to enable this registration handshake. Older Stores keep
+their existing registration behavior; older PD versions ignore the restore gate.
+Temporary shutdown uses a dedicated `ReportStoreShutdown` RPC, independent of
+cluster readiness. Older PDs do not implement it, so Store logs the failed report
+and relies on expiry; it never falls back to administrative `setStore(Offline)`.
+Administrative Offline requests retain their readiness gate and active-entry removal.
+`Up` confirms local engine restore, not Raft catch-up or full shard-group health.
+PD's authenticated `/v1/stores` and `/v1/store/{id}` responses also expose
+`heartbeatAgeMillis`, computed with PD's clock (`-1` when no heartbeat is known).
+Compare it with the fixed 30-second Store heartbeat period to distinguish a stale `Up` record
+from a recent heartbeat without waiting for keep-alive expiry. A future timestamp
+caused by a PD clock adjustment reports zero age.
+Keep the cluster preset's manual rollout barrier and verify fresh heartbeats,
+membership and leaders before replacing another Store.
+The shutdown report succeeds only if PD accepts a privileged internal credential:
+set `HG_PD_AUTH_SECRET_KEY` in Store to match PD's `auth.secret-key`. Without it,
+or if the RPC fails, shutdown continues and the existing lease expires normally.
+Restoring Offline Stores retain their liveness lease but cannot receive new
+placements until `Up`; recovery retries back off to the heartbeat interval.
+
+
 ### Select image versions
 
 <details>

docker/docker-compose-3pd-3store-3server.yml

@@ -161,6 +161,7 @@ services:
     container_name: hg-store0
     hostname: store0
     environment:
+      HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md}
       HG_STORE_PD_ADDRESS: pd0:8686,pd1:8686,pd2:8686
       HG_STORE_GRPC_HOST: store0
       HG_STORE_GRPC_PORT: "8500"
@@ -176,6 +177,7 @@ services:
     container_name: hg-store1
     hostname: store1
     environment:
+      HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md}
       HG_STORE_PD_ADDRESS: pd0:8686,pd1:8686,pd2:8686
       HG_STORE_GRPC_HOST: store1
       HG_STORE_GRPC_PORT: "8500"
@@ -191,6 +193,7 @@ services:
     container_name: hg-store2
     hostname: store2
     environment:
+      HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md}
       HG_STORE_PD_ADDRESS: pd0:8686,pd1:8686,pd2:8686
       HG_STORE_GRPC_HOST: store2
       HG_STORE_GRPC_PORT: "8500"

docker/docker-compose-hstore.yml

@@ -60,6 +60,7 @@ services:
       pd:
         condition: service_healthy
     environment:
+      HG_PD_AUTH_SECRET_KEY: ${HG_PD_AUTH_SECRET_KEY:?set HG_PD_AUTH_SECRET_KEY in .env; see docker/README.md}
       HG_STORE_PD_ADDRESS: pd:8686
       HG_STORE_GRPC_HOST: store
       HG_STORE_GRPC_PORT: "8500"

helm/hugegraph/docs/operations.md

@@ -206,6 +206,25 @@ A single Store replacement can still cause request errors or timeouts while Raft
 leadership and client routes converge. Observe application errors as well as
 shard recovery before continuing; do not blindly retry writes.
 
+With PD and Store images containing #3229's registration changes, `Up` follows
+local engine restore and graceful stops report `Offline`. Use PD's
+`heartbeatAgeMillis` to reject stale `Up` records after an abrupt stop.
+Temporary shutdown reports preserve the existing keepalive window without renewing
+it; quick restarts do not trip the initial-store-count gate, while expiry still
+applies if the Store does not return. Stream reconnects do not repeat local restore.
+While recovery runs, Offline registrations renew the returned Store's lease.
+The dedicated shutdown RPC falls back to normal expiry on older PDs without
+using the administrative Offline path, so a PD rollback does not remove that lease.
+These signals do not prove Raft catch-up.
+<!-- TODO: replace the manual barrier only when Store also exposes per-group
+     Raft catch-up readiness; local restore and heartbeat freshness are insufficient. -->
+The shutdown report succeeds only if PD accepts a privileged internal credential:
+set `HG_PD_AUTH_SECRET_KEY` in Store to match PD's `auth.secret-key`. Without it,
+or if the RPC fails, shutdown continues and the existing lease expires normally.
+Restoring Offline Stores retain their liveness lease but cannot receive new
+placements until `Up`; recovery retries back off to the heartbeat interval.
+
+
 ### PD patrol and leader balance
 
 Partition patrol repairs shard membership; it does not balance shard leaders.

helm/hugegraph/templates/store-statefulset.yaml

@@ -153,6 +153,11 @@ spec:
               valueFrom:
                 fieldRef:
                   fieldPath: metadata.namespace
+            - name: HG_PD_AUTH_SECRET_KEY
+              valueFrom:
+                secretKeyRef:
+                  name: {{ include "hugegraph.pd.authSecretName" . | quote }}
+                  key: {{ include "hugegraph.pd.authSecretKey" . | quote }}
             - name: HG_STORE_PD_ADDRESS
               value: {{ include "hugegraph.pd.grpcPeersList" . | quote }}
             - name: HG_STORE_GRPC_HOST

helm/hugegraph/tests/credential_contract_test.yaml

@@ -131,3 +131,18 @@ tests:
               secretKeyRef:
                 name: provided-admin
                 key: password
+
+  - it: gives Store the configured PD credential for privileged shutdown reports
+    template: store-statefulset.yaml
+    set:
+      pd.auth.existingSecret: provided-pd
+      pd.auth.key: shutdown-key
+    asserts:
+      - contains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: HG_PD_AUTH_SECRET_KEY
+            valueFrom:
+              secretKeyRef:
+                name: provided-pd
+                key: shutdown-key

hugegraph-pd/docs/api-reference.md

@@ -200,6 +200,41 @@ channel.shutdown();
 
 Register a new Store node with PD.
 
+Updated Stores first register `Offline` to obtain their identity, restore local
+partition engines, then register `Up`. Upgrade PD before Store: older PD versions
+ignore this restore gate, and older Stores retain their previous registration
+behavior. Pending Stores still require activation. `Up` does not guarantee Raft
+catch-up; graceful shutdown reports `Offline` without requesting permanent retirement.
+Temporary shutdown reports retain the existing keepalive entry without renewing it;
+expiry still removes a Store that does not return. During recovery, repeated Offline
+registrations renew the returned Store's lease while an owned worker restores engines.
+Once local restore succeeds,
+stream reconnects register `Up` directly without repeating recovery.
+`RegisterStoreResponse.store` supplies the authoritative registration state; older
+responses without this additive field fall back to a fresh Store lookup.
+
+`ReportStoreShutdown(SetStoreRequest)` returns `SetStoreResponse` and records
+temporary Offline by Store ID, preserving PD metadata and the existing lease even
+when the cluster is not ready. Older PDs return `UNIMPLEMENTED`; updated Stores
+fall back to expiry, never to `SetStore`. Administrative `SetStore(Offline)` keeps
+its cluster-readiness guard and removes the active entry as before.
+
+Shutdown reporting requires a privileged internal Basic credential matching PD's
+`auth.secret-key`. Set the same `HG_PD_AUTH_SECRET_KEY` in each Store process;
+this shared credential authorizes internal callers across Store IDs, rather than
+identifying an individual Store. Keep it restricted to trusted internal services.
+Missing, invalid or unconfigured credentials refuse the RPC. Store still shuts
+down and PD retains its previous record until the existing lease expires if the
+report is refused, unreachable, or unsupported by an older PD.
+
+A live restoration lease is not placement readiness: new allocation, replica
+addition, balancing and capacity checks select only Stores whose canonical state
+is `Up`. Existing shard membership and liveness/expiry tracking remain separate.
+Offline renewal occurs at most every five seconds while restoration is polled;
+failed restore attempts back off from one second to the Store heartbeat interval,
+with a full failure trace only on the first or a changed partition failure set.
+
+
 **Request**:
 ```protobuf
 message RegisterStoreRequest {
@@ -923,6 +958,16 @@ curl http://localhost:8620/v1/partitions?graph_name=social_network
 
 ### Store API
 
+#### Heartbeat freshness
+
+The authenticated `/v1/stores` and `/v1/store/{id}` responses expose
+`heartbeatAgeMillis` alongside the existing `lastHeartBeat`. Age is computed with
+PD's clock, is `-1` if no heartbeat is recorded, and is clamped to zero for future
+timestamps after a clock adjustment. Compare it with the fixed 30-second Store
+heartbeat period (30000 ms) to detect stale `Up` records before keep-alive expiry.
+Read this state from the PD leader and continue to verify shard-group membership
+and leaders during rolling updates; fresh heartbeats do not prove Raft catch-up.
+
 #### List Stores
 
 ```bash

hugegraph-pd/hg-pd-client/src/main/java/org/apache/hugegraph/pd/client/PDClient.java

@@ -73,7 +73,7 @@
 @Slf4j
 public class PDClient {
 
     private static Map<String, ManagedChannel> channels = new ConcurrentHashMap();
     private static ManagedChannel channel = null;
     private final PDConfig config;
     private final Pdpb.RequestHeader header;
@@ -382,14 +382,34 @@
      * @return
      */
     public long registerStore(Metapb.Store store) throws PDException {
+        return registerStoreResponse(store).getStoreId();
+    }
+
+    public Metapb.Store registerStoreWithState(Metapb.Store store) throws PDException {
+        Pdpb.RegisterStoreResponse response = registerStoreResponse(store);
+        if (response.hasStore()) {
+            if (config.isEnableCache() && response.getStore().getId() != 0) {
+                cache.addStore(response.getStore().getId(), response.getStore());
+            }
+            return response.getStore();
+        }
+        if (response.getStoreId() == 0) {
+            return store.toBuilder().setId(0).build();
+        }
+        // Older PDs return only the identity. Avoid a stale Offline entry from an earlier handshake.
+        cache.removeStore(response.getStoreId());
+        return getStore(response.getStoreId());
+    }
+
+    private Pdpb.RegisterStoreResponse registerStoreResponse(Metapb.Store store) throws PDException {
         Pdpb.RegisterStoreRequest request = Pdpb.RegisterStoreRequest.newBuilder()
                                                                      .setHeader(header)
                                                                      .setStore(store).build();
 
         Pdpb.RegisterStoreResponse response =
                 blockingUnaryCall(PDGrpc.getRegisterStoreMethod(), request);
         handleResponseError(response.getHeader());
-        return response.getStoreId();
+        return response;
     }
 
     /**
@@ -435,6 +455,15 @@
         return store;
     }
 
+    public Metapb.Store reportStoreShutdown(long storeId) throws PDException {
+        Pdpb.SetStoreRequest request = Pdpb.SetStoreRequest.newBuilder().setHeader(header)
+                .setStore(Metapb.Store.newBuilder().setId(storeId)).build();
+        Pdpb.SetStoreResponse response = getStub().reportStoreShutdown(request);
+        handleResponseError(response.getHeader());
+        cache.removeStore(storeId);
+        return response.getStore();
+    }
+
     /**
      * Return to the active store
      *
@@ -957,7 +986,7 @@
     private <ReqT, RespT, StubT extends AbstractBlockingStub<StubT>> RespT
     blockingUnaryCall(MethodDescriptor<ReqT, RespT> method, ReqT req, int retry) throws
                                                                                  PDException {
         io.grpc.stub.AbstractBlockingStub<StubT> stub = (AbstractBlockingStub<StubT>) getStub();
         try {
             RespT resp = io.grpc.stub.ClientCalls.blockingUnaryCall(stub.getChannel(), method,
                                                                     stub.getCallOptions(), req);

hugegraph-pd/hg-pd-core/src/main/java/org/apache/hugegraph/pd/PartitionService.java

@@ -648,7 +648,7 @@ public synchronized void splitPartition(Metapb.Graph graph, int toCount) throws
         var maxShardsPerStore = pdConfig.getPartition().getMaxShardsPerStore();
         var shardCount = pdConfig.getPartition().getShardCount();
 
-        if (shardCount * toCount > storeService.getActiveStores().size() * maxShardsPerStore) {
+        if (shardCount * toCount > storeService.getPlacementStores().size() * maxShardsPerStore) {
             throw new PDException(Pdpb.ErrorType.Too_Many_Partitions_Per_Store_VALUE,
                                   "can't satisfy target shard group count, reached the upper " +
                                   "limit of the cluster");