diff --git a/docker/README.md b/docker/README.md index 9bc21b1ba7..0e9c8bb700 100644 --- a/docker/README.md +++ b/docker/README.md @@ -48,6 +48,14 @@ cd docker docker compose -f docker-compose.dev.yml up -d ``` +To run Store with the JDK 17 image profile (useful for ARM64 JNI-heavy workloads): + +```bash +cd docker +HG_STORE_DOCKERFILE=hugegraph-store/Dockerfile-jdk17 \ + docker compose -f docker-compose.dev.yml up -d --build +``` + - Images: built from source via `build: context: ..` with Dockerfiles - No `pull_policy` — builds locally, doesn't pull - Entrypoint scripts are baked into the built image (no volume mounts) diff --git a/docker/cloud-storage/.gitignore b/docker/cloud-storage/.gitignore new file mode 100644 index 0000000000..56a3ee1594 --- /dev/null +++ b/docker/cloud-storage/.gitignore @@ -0,0 +1,5 @@ +.artifacts/ +logs/ +data/ +.generated/ + diff --git a/docker/cloud-storage/README.md b/docker/cloud-storage/README.md new file mode 100644 index 0000000000..f5748f267d --- /dev/null +++ b/docker/cloud-storage/README.md @@ -0,0 +1,800 @@ +# Cloud Storage (MinIO + HStore) Local Stack + +This stack runs: + +- MinIO (S3-compatible storage) +- 1 PD node +- 3 Store nodes +- `hg-store-cloud-s3` plugin loaded from local build artifacts + +The goal is to validate that: + +1. RocksDB SST files are mirrored to S3. +2. Recovery metadata (`MANIFEST-*`, `CURRENT`, `OPTIONS-*`) is mirrored alongside SSTs. +3. Together, these uploaded files are enough for cloud recovery after local RocksDB data loss. + +## Prerequisites + +- Docker + Docker Compose +- Java 11+ +- Maven 3.5+ +- `curl` and `jq` (optional, for manual API testing) + +From repo root, build all modules: + +```bash +mvn clean package -DskipTests +``` + +This builds the necessary artifacts for the Docker images. + +## Quick Start + +Detect repo root first, then run the script: + +```bash +export REPO_ROOT="$(git rev-parse --show-toplevel)" +test -d "${REPO_ROOT}/docker/cloud-storage" + +cd "${REPO_ROOT}/docker/cloud-storage" +chmod +x ./scripts/*.sh ./entrypoints/*.sh +./scripts/test-graph-queries-and-sst.sh +``` + +```bash +# (Optional) Keep the stack running after the test for manual inspection +./scripts/test-graph-queries-and-sst.sh --keep-stack +``` + +```bash + +# (Optional) Start infrastructure only (skip data creation + validation tests) +./scripts/test-graph-queries-and-sst.sh --keep-stack --infra-only +``` + +## Test Output Files + +After running `make test` or `./scripts/test-graph-queries-and-sst.sh`, check `.generated/` for: + +- **`FULL-TEST-REPORT.txt`** — Complete test summary +- **`test-report.txt`** — Graph query test results +- **`minio-verification.txt`** — MinIO S3 object listing and SST count +- **`store-cluster.log`** — All store node logs with RocksDB/S3 events +- **`cli-load.log`** — Data load job output from hg-store-cli +- **`load-data.tsv`** — Generated test data file (200k entries) + +## Manual Verification Steps + +**Prerequisites:** Complete Step 1 first and wait for the infrastructure ready message. + +For hands-on validation with manual graph creation and queries, start the cluster using the automated script with `--keep-stack`, then follow interactive steps. **All commands use `$REPO_ROOT` to reference paths relative to the repository root.** + +**Note:** These steps verify end-to-end SST upload by: +1. Creating a graph schema +2. Adding test vertices and edges +3. Verifying data distribution across nodes +4. Restarting store nodes to flush SST files to RocksDB +5. Confirming SST files are uploaded to MinIO buckets + +### Step 0: Set Repository Root + +Before starting, set the `$REPO_ROOT` variable to point to the repository root. You can run this from anywhere: + +```bash +export REPO_ROOT=$(git rev-parse --show-toplevel) +``` + +Verify the variable is set correctly: + +```bash +echo $REPO_ROOT +``` + +### Step 1: Start Cluster with Infrastructure Only (No Auto-Load) + +Use the automated test script to build images and start the stack. For manual-only flows, +prefer `--infra-only` to skip scripted data creation and validation checks while keeping the stack up: + +```bash +$REPO_ROOT/docker/cloud-storage/scripts/test-graph-queries-and-sst.sh --keep-stack --infra-only +``` + +After Step 1 starts the stack, resolve the Docker network name for `docker run --network` commands: + +```bash +export HG_NET="${HG_NET:-cloud-storage-net}" +if docker network inspect "$HG_NET" >/dev/null 2>&1; then + echo "Using Docker network: $HG_NET" +elif docker network inspect cloud-storage-test_hg-net >/dev/null 2>&1; then + export HG_NET="cloud-storage-test_hg-net" + echo "Using Docker network: $HG_NET" +else + echo "No cloud-storage Docker network found. Ensure Step 1 completed successfully." +fi +``` + +This will: +- Build local artifacts (PD, Store, Store CLI, S3 plugin) +- Build Docker images +- Start MinIO + PD + 3 Store nodes +- Wait for all services to be healthy +- **Skip** initial data load phase +- **Keep the stack running** for manual steps + +The output will indicate when infrastructure is ready. + +### Step 2: Verify Cluster Health + +Verify all services are accessible. This may take a few minutes as the HugeGraph server needs time to initialize: + +```bash +# Wait for services to be healthy (may take 2-3 minutes on first run) +echo "Waiting for services to be healthy..." +for i in {1..60}; do + pd_ok=$(curl -fsS http://127.0.0.1:8620/v1/health >/dev/null 2>&1 && echo "yes" || echo "no") + store0_ok=$(curl -fsS http://127.0.0.1:8520/v1/health >/dev/null 2>&1 && echo "yes" || echo "no") + store1_ok=$(curl -fsS http://127.0.0.1:8521/v1/health >/dev/null 2>&1 && echo "yes" || echo "no") + store2_ok=$(curl -fsS http://127.0.0.1:8522/v1/health >/dev/null 2>&1 && echo "yes" || echo "no") + server_ok=$(curl -fsS http://127.0.0.1:8080/graphs >/dev/null 2>&1 && echo "yes" || echo "no") + + if [[ "$pd_ok" == "yes" && "$store0_ok" == "yes" && "$store1_ok" == "yes" && "$store2_ok" == "yes" && "$server_ok" == "yes" ]]; then + break + fi + + echo " Attempt $i/60: PD=$pd_ok Store0=$store0_ok Store1=$store1_ok Store2=$store2_ok Server=$server_ok" + sleep 2 +done + +# Final verification +curl -fsS http://127.0.0.1:8620/v1/health >/dev/null 2>&1 && echo "✓ PD OK" || echo "✗ PD FAILED" +curl -fsS http://127.0.0.1:8520/v1/health >/dev/null 2>&1 && echo "✓ Store0 OK" || echo "✗ Store0 FAILED" +curl -fsS http://127.0.0.1:8521/v1/health >/dev/null 2>&1 && echo "✓ Store1 OK" || echo "✗ Store1 FAILED" +curl -fsS http://127.0.0.1:8522/v1/health >/dev/null 2>&1 && echo "✓ Store2 OK" || echo "✗ Store2 FAILED" +curl -fsS http://127.0.0.1:8080/graphs >/dev/null 2>&1 && echo "✓ Server OK" || echo "✗ Server FAILED" +``` + +If all show "✓ OK", proceed to Step 3. If any show "✗ FAILED", check logs: +```bash +docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml logs --tail=50 server +``` + +**Note:** The `/graphs` endpoint returns a clean 200 OK response, making it the most reliable health check for the HugeGraph server. The `/gremlin` endpoint requires query parameters. + +### Step 3: Create Graph Schema + +Schema persists across restarts via rocksdb-cloud. If you see `ExistedException`, schema already exists. + +```bash +create_pk() { + local name="$1" dtype="$2" + local found=$(curl -s --compressed "http://localhost:8080/graphs/hugegraph/schema/propertykeys/$name" \ + | python3 -c "import sys,json; print(json.load(sys.stdin).get('name',''))" 2>/dev/null) + [[ "$found" == "$name" ]] && { echo " ✓ property key '$name' exists"; return; } + curl -s -X POST http://localhost:8080/graphs/hugegraph/schema/propertykeys \ + -H 'Content-Type: application/json' \ + -d "{\"name\":\"$name\",\"data_type\":\"$dtype\",\"cardinality\":\"SINGLE\"}" \ + | python3 -c "import sys,json; d=json.load(sys.stdin); print(' ✓ created property key:', d.get('property_key',{}).get('name','?'))" +} + +create_vl() { + local name="$1" props="$2" + local found=$(curl -s --compressed "http://localhost:8080/graphs/hugegraph/schema/vertexlabels/$name" \ + | python3 -c "import sys,json; print(json.load(sys.stdin).get('name',''))" 2>/dev/null) + [[ "$found" == "$name" ]] && { echo " ✓ vertex label '$name' exists"; return; } + curl -s -X POST http://localhost:8080/graphs/hugegraph/schema/vertexlabels \ + -H 'Content-Type: application/json' \ + -d "{\"name\":\"$name\",\"id_strategy\":\"AUTOMATIC\",\"properties\":$props}" \ + | python3 -c "import sys,json; d=json.load(sys.stdin); print(' ✓ created vertex label:', d.get('name','?'))" +} + +create_pk "name" "TEXT" +create_pk "age" "INT" +create_pk "city" "TEXT" + +create_vl "person" '["name","age","city"]' +create_vl "location" '["name"]' + +FOUND_EL=$(curl -s --compressed http://localhost:8080/graphs/hugegraph/schema/edgelabels/lives_in \ + | python3 -c "import sys,json; print(json.load(sys.stdin).get('name',''))" 2>/dev/null) +if [[ "$FOUND_EL" == "lives_in" ]]; then + echo " ✓ edge label 'lives_in' exists" +else + curl -s -X POST http://localhost:8080/graphs/hugegraph/schema/edgelabels \ + -H 'Content-Type: application/json' \ + -d '{"name":"lives_in","source_label":"person","target_label":"location"}' \ + | python3 -c "import sys,json; d=json.load(sys.stdin); print(' ✓ created edge label:', d.get('name','?'))" +fi + +echo "✓ Schema ready" +``` + +### Step 4: Add Vertices and Edges + +Add sufficient data to trigger RocksDB compaction (minimum 150+ vertices): + +```bash +insert_vertex() { + local label="$1" props="$2" + local tmpfile=$(mktemp) + local code=$(curl -s -o "$tmpfile" -w "%{http_code}" -X POST \ + http://localhost:8080/graphs/hugegraph/graph/vertices \ + -H 'Content-Type: application/json' \ + -d "{\"label\":\"$label\",\"properties\":$props}") + local body=$(cat "$tmpfile") + rm -f "$tmpfile" + if [[ "$code" == "201" || "$code" == "200" ]]; then + echo "$body" | python3 -c "import sys,json; print(json.load(sys.stdin)['id'])" + else + echo "ERROR: HTTP $code - $body" >&2 + echo "" + fi +} + +echo "Inserting test vertices (150+)..." +for i in {1..150}; do + age=$((20 + i % 50)) + city=$((i % 5)) + vid=$(insert_vertex "person" "{\"name\":\"person_$i\",\"age\":$age,\"city\":\"city_$city\"}") + if (( i % 50 == 0 )); then + echo " ✓ Inserted $i vertices" + fi +done + +echo "Adding location vertices..." +for i in {0..4}; do + insert_vertex "location" "{\"name\":\"city_$i\"}" >/dev/null +done + +echo "✓ Inserted 150+ test vertices (sufficient for RocksDB compaction)" +``` + +**Why 150+ vertices?** Smaller datasets may not trigger RocksDB compaction. 150+ vertices across 3 store nodes ensures enough write activity to generate SST files. + +### Step 5: Execute Graph Queries (Optional) + +Verify the data was stored: + +```bash +echo "=== Vertex count ===" +curl -s --compressed http://localhost:8080/graphs/hugegraph/graph/vertices | python3 -c "import sys,json; data=json.load(sys.stdin); print('Total vertices:', len(data.get('vertices',[])))" + +echo "=== Sample vertex ===" +curl -s --compressed http://localhost:8080/graphs/hugegraph/graph/vertices | python3 -c "import sys,json; data=json.load(sys.stdin); vertices=data.get('vertices', []); print(json.dumps(vertices[0], indent=2) if vertices else 'No vertices')" | head -10 +``` + +**Note:** Querying 150+ vertices can return large result sets. To keep this step fast, the above commands just check the count and show a sample. + +### Step 6: Verify Data Distribution + +Verify data has been distributed across store nodes: + +```bash +echo "=== Partition Info (before flush) ===" +for i in 0 1 2; do + port=$((8520 + i)) + echo "" + echo "Store$i (port $port):" + curl -s http://127.0.0.1:$port/v1/partitions | python3 -c "import sys,json; data=json.load(sys.stdin); print(f' Partitions: {len(data.get(\"partitions\",[])) if isinstance(data.get(\"partitions\"), list) else \"N/A\"}')" 2>/dev/null || echo " (unable to retrieve)" +done +``` + +**Expected:** Each store should show partition information, indicating data distribution across the cluster. + +### Step 7: Flush Data to S3 (Restart Store Nodes) + +To trigger SST file uploads to S3, restart the Store nodes. This forces RocksDB to: +1. Flush in-memory data to disk as SST files +2. Trigger the cloud storage event listener +3. Upload SST files to MinIO buckets + +```bash +echo "Restarting store nodes to flush data to S3..." +# Use container names so this works for both static compose and generated test compose +docker restart cloud-storage-store0 cloud-storage-store1 cloud-storage-store2 + +echo "Waiting for stores to restart and flush..." +sleep 20 + +echo "Verifying store health after restart..." +for i in 0 1 2; do + port=$((8520 + i)) + curl -fsS http://127.0.0.1:$port/v1/health >/dev/null 2>&1 && echo "✓ Store$i OK" || echo "✗ Store$i FAILED" +done +``` + +**What happens behind the scenes:** +- Docker restarts the store containers +- RocksDB initializes and detects in-memory data +- RocksDB writes all data as SST files to disk +- Cloud storage listener detects flush events +- SST files are uploaded to MinIO in parallel +- Stores become healthy and rejoin cluster + +### Step 8: Final S3 Verification (After Flush) + +Verify that SST files have been successfully uploaded to MinIO buckets: + +```bash +echo "=== Checking MinIO buckets after flush ===" +docker run --rm --entrypoint /bin/sh --network "$HG_NET" minio/mc:RELEASE.2025-08-13T08-35-41Z \ + -c 'mc alias set local http://minio:9000 minioadmin minioadmin >/dev/null && \ + for b in hugegraph-store0 hugegraph-store1 hugegraph-store2; do \ + echo ""; \ + echo "### Bucket: $b"; \ + total_count=$(mc ls --recursive local/$b | wc -l); \ + sst_count=$(mc find local/$b --name "*.sst" | wc -l); \ + printf " Total objects: %d\n" "$total_count"; \ + printf " SST files: %d\n" "$sst_count"; \ + if (( sst_count > 0 )); then \ + echo " Sample SST files (first 3):"; \ + mc find local/$b --name "*.sst" | head -3; \ + fi; \ + done' +``` + +**What successful upload looks like** (note the recovery metadata objects — `CURRENT`, `MANIFEST-*`, +`OPTIONS-*` — mirrored alongside the SSTs so the SSTs are a usable database, not orphans): +``` +### Bucket: hugegraph-store0 + Total objects: 9 + SST files: 6 + Objects under a partition prefix (paths illustrative; the file names are what matter): + local/hugegraph-store0///000009.sst + local/hugegraph-store0///000012.sst + local/hugegraph-store0///CURRENT + local/hugegraph-store0///MANIFEST-000011 + local/hugegraph-store0///OPTIONS-000013 + +### Bucket: hugegraph-store1 + Total objects: 10 + SST files: 7 + ... + +### Bucket: hugegraph-store2 + Total objects: 9 + SST files: 6 + ... +``` + +**Success criteria:** +- ✅ All three buckets show `Total objects: > 0` +- ✅ All three buckets show `SST files: > 0` +- ✅ Each partition prefix contains **exactly one** `CURRENT`, at least one `MANIFEST-*`, and at + least one `OPTIONS-*` (the consistent recovery metadata set) +- ✅ SST counts are roughly balanced across buckets +- ✅ No errors from mc command + +To assert the recovery metadata set explicitly: + +```bash +docker run --rm --entrypoint /bin/sh --network "$HG_NET" minio/mc:RELEASE.2025-08-13T08-35-41Z \ + -c 'mc alias set local http://minio:9000 minioadmin minioadmin >/dev/null && \ + for b in hugegraph-store0 hugegraph-store1 hugegraph-store2; do \ + echo "### $b: CURRENT=$(mc find local/$b --name CURRENT | wc -l)" \ + "MANIFEST=$(mc find local/$b --name "MANIFEST-*" | wc -l)" \ + "OPTIONS=$(mc find local/$b --name "OPTIONS-*" | wc -l)" \ + "SST=$(mc find local/$b --name "*.sst" | wc -l)"; \ + done' +``` + +**If buckets are still empty after flush:** +1. Check store logs for upload errors: `docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml logs store0 | grep -i "s3\|cloud\|error"` +2. Verify cloud storage was initialized: `docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml logs store0 | grep -i "Cloud storage provider.*initialized\|S3CloudStorageProvider initialized"` +3. Check that data was written: `docker exec cloud-storage-store0 sh -lc 'find /hugegraph-store/storage -name "*.sst" | wc -l'` + +### Step 8.5 (Optional): Verify Managed Delete/Clear Cleanup + +This step validates the delete path used by single-graph deployments: `graph.clear()` should clean the +remote DB prefix so stale data is not rehydrated later. + +> **Warning:** This deletes graph data. Run it after Step 8, and re-run Steps 3-8 if you want to run +> the recovery scenario in the next section. + +```bash +GRAPH_NAME=hugegraph +BUCKET=hugegraph-store0 + +# Detect one cloud DB prefix for this graph from CURRENT objects. +DB_PREFIX=$(docker run --rm --entrypoint /bin/sh --network "$HG_NET" minio/mc:RELEASE.2025-08-13T08-35-41Z \ + -c 'mc alias set local http://minio:9000 minioadmin minioadmin >/dev/null && \ + mc find local/'"$BUCKET"' --name CURRENT 2>/dev/null' \ + | grep "/${GRAPH_NAME}/" | head -n1 | sed "s#^local/${BUCKET}/##" | sed 's#/CURRENT$##') + +echo "Detected DB prefix: ${DB_PREFIX}" +[[ -n "$DB_PREFIX" ]] || { echo "Failed to detect DB prefix for ${GRAPH_NAME}"; exit 1; } + +# Add a probe object under the DB prefix so cleanup is easy to validate. +docker run --rm --entrypoint /bin/sh --network "$HG_NET" minio/mc:RELEASE.2025-08-13T08-35-41Z \ + -c 'mc alias set local http://minio:9000 minioadmin minioadmin >/dev/null && \ + printf "delete-probe\n" | mc pipe local/'"$BUCKET"'/'"$DB_PREFIX"'/manual-delete-probe.txt >/dev/null' + +echo "Objects before clear:" \ + $(docker run --rm --entrypoint /bin/sh --network "$HG_NET" minio/mc:RELEASE.2025-08-13T08-35-41Z \ + -c 'mc alias set local http://minio:9000 minioadmin minioadmin >/dev/null && \ + mc ls --recursive local/'"$BUCKET"'/'"$DB_PREFIX"' 2>/dev/null | wc -l') + +# Managed delete-equivalent path for single-graph mode. +curl -s -X DELETE "http://localhost:8080/graphs/${GRAPH_NAME}/clear" \ + --get --data-urlencode "confirm_message=I'm sure to delete all data" | head -c 200; echo +sleep 5 + +echo "Objects after clear:" \ + $(docker run --rm --entrypoint /bin/sh --network "$HG_NET" minio/mc:RELEASE.2025-08-13T08-35-41Z \ + -c 'mc alias set local http://minio:9000 minioadmin minioadmin >/dev/null && \ + mc ls --recursive local/'"$BUCKET"'/'"$DB_PREFIX"' 2>/dev/null | wc -l') + +# Optional no-orphan check: graph remains empty after store restart. +docker restart cloud-storage-store0 cloud-storage-store1 cloud-storage-store2 +sleep 20 +curl -s --compressed "http://localhost:8080/graphs/${GRAPH_NAME}/graph/vertices" \ + | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('vertices',[])))" +``` + +**Success criteria:** +- ✅ `Objects after clear` is `0` for the detected graph DB prefix +- ✅ Optional restart check returns vertex count `0` (no orphaned rehydration) + +### Step 8.6 (Optional, Advanced): Verify DB Delete Callbacks (`onDBDeleteBegin` + `onDBDeleted`) + +This step validates the DB-destroy lifecycle callbacks (not truncate/clear): + +- `onDBDeleteBegin` writes a tombstone object (`_DELETED`) for the DB prefix +- `onDBDeleted` purges the DB prefix from cloud storage + +It uses the Store test endpoint `/test/raftDelete/{groupId}` to destroy one partition engine, +which triggers `destroyGraphDB(...)` internally. + +> **Warning:** This is destructive and intended only for callback verification. Run it near the +> end of manual testing. After this step, restart from Step 1 for a fresh cluster state. + +```bash +BUCKET=hugegraph-store0 + +# Pick one partition group id from store0. +PART_ID=$(curl -s http://127.0.0.1:8520/v1/partitions \ + | python3 -c 'import sys,json; d=json.load(sys.stdin); p=d.get("partitions") or []; print(p[0].get("id", "") if p else "")') + +echo "Selected partition id: ${PART_ID}" +[[ -n "$PART_ID" ]] || { echo "No partition id found"; exit 1; } + +# RocksDB dbName is zero-padded partition id (e.g. 3 -> 00003). +DB_NAME=$(printf "%05d" "$PART_ID") + +# Detect one cloud DB prefix for this partition from CURRENT objects. +DB_PREFIX=$(docker run --rm --entrypoint /bin/sh --network "$HG_NET" minio/mc:RELEASE.2025-08-13T08-35-41Z \ + -c 'mc alias set local http://minio:9000 minioadmin minioadmin >/dev/null && \ + mc find local/'"$BUCKET"' --name CURRENT 2>/dev/null' \ + | sed "s#^local/${BUCKET}/##" \ + | grep "/${DB_NAME}/CURRENT$" \ + | head -n1 \ + | sed 's#/CURRENT$##') + +echo "Detected DB prefix: ${DB_PREFIX}" +[[ -n "$DB_PREFIX" ]] || { echo "Failed to detect cloud prefix for db ${DB_NAME}"; exit 1; } + +# Add a probe object so purge verification is deterministic. +docker run --rm --entrypoint /bin/sh --network "$HG_NET" minio/mc:RELEASE.2025-08-13T08-35-41Z \ + -c 'mc alias set local http://minio:9000 minioadmin minioadmin >/dev/null && \ + printf "db-delete-probe\n" | mc pipe local/'"$BUCKET"'/'"$DB_PREFIX"'/manual-db-delete-probe.txt >/dev/null' + +echo "Objects before db delete:" \ + $(docker run --rm --entrypoint /bin/sh --network "$HG_NET" minio/mc:RELEASE.2025-08-13T08-35-41Z \ + -c 'mc alias set local http://minio:9000 minioadmin minioadmin >/dev/null && \ + mc ls --recursive local/'"$BUCKET"'/'"$DB_PREFIX"' 2>/dev/null | wc -l') + +# Trigger partition destroy path (calls destroyGraphDB internally). +curl -s "http://127.0.0.1:8520/test/raftDelete/${PART_ID}"; echo +sleep 8 + +# Callback evidence from logs. +docker logs cloud-storage-store0 2>&1 \ + | grep -E "Cloud DB tombstone written|Cloud DB purge completed" \ + | tail -10 + +echo "Objects after db delete:" \ + $(docker run --rm --entrypoint /bin/sh --network "$HG_NET" minio/mc:RELEASE.2025-08-13T08-35-41Z \ + -c 'mc alias set local http://minio:9000 minioadmin minioadmin >/dev/null && \ + mc ls --recursive local/'"$BUCKET"'/'"$DB_PREFIX"' 2>/dev/null | wc -l') +``` + +**Success criteria:** +- ✅ Logs include `Cloud DB tombstone written` (from `onDBDeleteBegin`) +- ✅ Logs include `Cloud DB purge completed` (from `onDBDeleted`) +- ✅ `Objects after db delete` is `0` for the detected DB prefix + +## Total-Loss Recovery from Cloud + +Steps 1–8 verify that SSTs **and** a consistent metadata set (`CURRENT` / `MANIFEST-*` / +`OPTIONS-*`) reach the object store. Recovery validation is covered by the automated test: + +```bash +$REPO_ROOT/docker/cloud-storage/scripts/test-graph-queries-and-sst.sh --keep-stack +``` + +This loads 150 vertices, flushes/compacts, asserts the consistent metadata set is in every bucket, +wipes each store's RocksDB **state machine** (`db/` + `hgstore-metadata` — the cloud-mirrored +tree) while **preserving `raft/`** so the node rejoins its Raft group cleanly, then restarts and +confirms the recovered vertex count matches the pre-wipe baseline. It fails loudly if the +consistent-restore guard trips (`Cloud restore inconsistent`) or the counts differ. + +> **Full disk-loss variant (stronger test).** To simulate whole-volume loss (Raft included), +> replace the wipe with: +> ```bash +> docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml rm -sf store0 store1 store2 +> for i in 0 1 2; do docker volume rm cloud-storage-test_hg-store${i}-data 2>/dev/null || true; done +> docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml up -d store0 store1 store2 +> ``` +> Recovery then depends on Raft group re-formation in addition to cloud pre-hydration; run this +> after the standard path above passes. + +### Step 9 (Optional): Destroy the Cluster + +When done with manual verification, stop and clean up: + +```bash +docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml down +``` + +To also remove data and logs directories: + +```bash +docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml down +rm -rf $REPO_ROOT/docker/cloud-storage/data/ $REPO_ROOT/docker/cloud-storage/logs/ +``` + +To reset everything including built Docker images (for a fresh start): + +```bash +docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml down +rm -rf $REPO_ROOT/docker/cloud-storage/data/ $REPO_ROOT/docker/cloud-storage/logs/ $REPO_ROOT/docker/cloud-storage/.artifacts/ +docker rmi \ + hugegraph-pd-cloud-storage:local \ + hugegraph-store-cloud-storage:local \ + 2>/dev/null || true +``` + + +## Architecture Notes + +- **Entrypoints**: Store containers inject `cloud.storage.*` config via `SPRING_APPLICATION_JSON` in `entrypoints/store-entrypoint.sh`. +- **Config precedence**: values injected by `SPRING_APPLICATION_JSON` override defaults in `/hugegraph-store/conf/application.yml`. +- **Plugin Loading**: Store startup uses `PropertiesLauncher` with `-Dloader.path=/hugegraph-store/plugins` to discover S3 provider JAR via `ServiceLoader` +- **Logging**: Store containers run with `DEBUG` level for: + - `org.apache.hugegraph.store.node.cloud` + - `org.apache.hugegraph.rocksdb.access` +- **Networking**: Containers run on `cloud-storage-net` (static compose) or `cloud-storage-test_hg-net` (default generated test stack) +- **Storage**: Local bind mounts under `data/` and `logs/` for inspection +- **Bucket layout**: each Store node writes SSTs to a dedicated bucket (`store0 -> hugegraph-store0`, `store1 -> hugegraph-store1`, `store2 -> hugegraph-store2`) + +### Troubleshooting Manual Verification Steps + +**Step 2: Server not becoming healthy** +```bash +# Check if server container is running +docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml ps server + +# Check server logs +docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml logs --tail=100 server + +# Verify port 8080 is accessible and /graphs endpoint works +curl -v http://127.0.0.1:8080/graphs +``` + +**Note:** The server health check uses `/graphs` endpoint (HTTP 200 OK), not `/gremlin` (requires query parameter). + +**Step 3-4: Graph API errors (HTTP errors)** +```bash +# Verify server is healthy +curl -fsS http://127.0.0.1:8080/graphs && echo "Server OK" || echo "Server not ready" + +# Check if graph exists +curl -s http://localhost:8080/graphs/hugegraph/graph/vertices | python3 -c "import sys,json; print(json.load(sys.stdin).keys())" + +# Check server logs for errors +docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml logs server | grep -i "error\|exception" | tail -20 +``` + +**Step 6: Partition info showing empty** +```bash +# Data may still be in RocksDB memory, not yet in partitions +# This is normal - proceed to Step 7 to flush + +# Or check if data was actually written +docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml logs store0 | grep -i "vertex\|edge\|insert" | tail -10 +``` + +**Step 7: Stores not coming back healthy after restart** +```bash +# Check individual store health +for i in 0 1 2; do + port=$((8520 + i)) + echo "Store$i:" + curl -v http://127.0.0.1:$port/v1/health 2>&1 | grep -E "HTTP/|Connection refused" +done + +# Check store logs for startup errors +docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml logs store0 | tail -50 | grep -i "error\|exception" +``` + +**Step 8: Buckets showing 0 files after flush** + +This is the most common issue. Debug step-by-step: + +1. **Verify stores have data on disk:** + ```bash + docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml exec store0 \ + find /hugegraph-store/storage -name "*.sst" | wc -l + ``` + If output is 0, no SST files were created. Go back to Step 4 and ensure data was inserted. + +2. **Verify cloud storage was initialized:** + ```bash + docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml logs store0 | \ + grep -i "S3CloudStorageProvider\|cloud storage" | head -5 + ``` + If no output, plugin didn't load. Check `/hugegraph-store/plugins/` for `hg-store-cloud-s3.jar`. + +3. **Check for S3 upload errors:** + ```bash + docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml logs store0 | \ + grep -i "upload\|s3\|error" | tail -20 + ``` + +4. **Verify MinIO connectivity:** + ```bash + docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml exec store0 \ + curl -v http://minio:9000/minio/health/live + ``` + +5. **Manually check MinIO buckets:** + ```bash + export HG_NET="cloud-storage-net" + docker run --rm --entrypoint /bin/sh --network "$HG_NET" minio/mc:RELEASE.2025-08-13T08-35-41Z \ + -c 'mc alias set local http://minio:9000 minioadmin minioadmin && mc ls local/' + ``` + +### Troubleshooting: General Infrastructure Issues + +### Store crashes with SIGSEGV on ARM64 (libjvm.so) + +If store logs show a fatal JVM crash like: + +```text +SIGSEGV ... libjvm.so ... linux-aarch64 +``` + +this is typically seen on ARM64 hosts with JVM + RocksDB startup. + +**On ARM64 hosts, the script automatically applies safe JVM defaults:** +- Uses Java 17 runtime image (not Java 11) +- Applies conservative JVM flags: `-XX:+UseSerialGC -XX:-UseCompressedOops -XX:-UseCompressedClassPointers` + +Simply run without special flags: + +```bash +$REPO_ROOT/docker/cloud-storage/scripts/test-graph-queries-and-sst.sh +``` + +The script detects your host architecture and applies appropriate defaults automatically. + +### Custom overrides on ARM64 + +If you need different JVM flags or runtime images: + +```bash +HG_DOCKER_DEFAULT_PLATFORM=linux/amd64 \ +HG_PD_JAVA_RUNTIME_IMAGE=eclipse-temurin:17-jre \ +HG_STORE_JAVA_RUNTIME_IMAGE=eclipse-temurin:17-jre \ +HG_STORE_JAVA_OPTS="-XX:+UseSerialGC -XX:-UseCompressedOops -XX:-UseCompressedClassPointers" \ +$REPO_ROOT/docker/cloud-storage/scripts/test-graph-queries-and-sst.sh +``` + +But in most cases, running without overrides should work: + +```bash +$REPO_ROOT/docker/cloud-storage/scripts/test-graph-queries-and-sst.sh +``` + +If old containers are still running, force recreate with fresh env: + +```bash +docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml down +$REPO_ROOT/docker/cloud-storage/scripts/test-graph-queries-and-sst.sh + +# Verify container runtime JDK actually changed +docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml run --rm --entrypoint java store0 -version +``` + +If the crash persists, inspect the generated JVM error file from store logs/data path: + +```bash +docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml logs store0 | tail -200 +docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml exec store0 ls -lah /hugegraph-store/hs_err_pid*.log +``` + +### No SST files appear in MinIO + +- Short test runs may not trigger compaction; SST deletion especially requires heavier/longer load +- Check store logs for upload errors: + ```bash + docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml logs store0 | grep -i "cloud\|s3" + ``` +- Verify MinIO is accessible: + ```bash + docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml exec minio mc ls local/ 2>&1 | head -20 + ``` + +### Query tests show FAILED + +- Ensure all store nodes are healthy: check `/v1/health` endpoints +- Manually test: `curl -fsS http://127.0.0.1:8520/v1/partitions | jq` +- Check store logs with: + ```bash + docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml logs store0 store1 store2 + ``` + +### Keep stack running for debugging + +```bash +# Run test but keep stack up +KEEP_STACK=1 $REPO_ROOT/docker/cloud-storage/scripts/test-graph-queries-and-sst.sh +``` + +Then inspect manually: +```bash +docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml logs store0 store1 store2 +docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml exec store0 ls -la /hugegraph-store/storage +``` + +Stop manually when done: +```bash +docker compose -f $REPO_ROOT/docker/cloud-storage/docker-compose.yml down +``` + +## Summary of Manual Verification Workflow + +This manual verification process validates the complete SST upload pipeline: + +1. **Step 1:** Start the infrastructure (MinIO, PD, 3 Store nodes, HugeGraph Server) +2. **Step 2:** Wait for all services to become healthy +3. **Step 3:** Create graph schema (property keys, vertex labels, edge labels) +4. **Step 4:** Load 150+ test vertices to trigger compaction +5. **Step 5:** (Optional) Verify data was stored +6. **Step 6:** Verify data distribution across store nodes +7. **Step 7:** Restart store nodes to flush SST files to disk and upload to MinIO +8. **Step 8:** Verify SST files are present in MinIO buckets +9. **Step 8.5 (optional):** Verify managed delete/clear cleans graph cloud prefix (and optional no-orphan restart check) +10. **Step 8.6 (optional, advanced):** Verify DB delete callbacks (`onDBDeleteBegin` + `onDBDeleted`) on one partition +11. **Recovery (optional):** Run `test-graph-queries-and-sst.sh --keep-stack` (without `--infra-only`) + — the script wipes local RocksDB state, restarts, and confirms the recovered vertex count matches + the pre-wipe baseline (see [Total-Loss Recovery from Cloud](#total-loss-recovery-from-cloud)) +12. **Step 9:** Cleanup when done + +- **Note:** For manual Step 3+ workflows, run Step 1 with `--infra-only` so the script starts infrastructure but skips scripted data creation/validation that can mutate graph state. + +**Success = Non-zero SST file counts in all three buckets after Step 8** (and, for recovery, +`AFTER == BEFORE` vertex count after the total-loss recovery) + +## What Gets Verified + +✅ Graph schema creation works +✅ Vertex/edge insertion works +✅ Data distribution across 3 store nodes +✅ RocksDB SST file generation on restart +✅ Cloud storage plugin uploads SST files to MinIO +✅ Multiple buckets receive files consistently +✅ A consistent `CURRENT` + `MANIFEST-*` + `OPTIONS-*` metadata set is mirrored alongside SSTs +✅ Managed delete/clear prunes the graph cloud prefix (optional step) +✅ DB delete callbacks write tombstone and purge remote DB prefix (optional advanced step) +✅ A store recovers all data from cloud after losing its local RocksDB state (pre-hydration) + +## Notes + +- S3 provider JAR must be in `/hugegraph-store/plugins/` for `ServiceLoader` discovery +- Plugin dependency staging filters extra SLF4J binding jars to avoid duplicate binding warnings at runtime +- Cloud settings are read from environment variables at store startup time +- **Minimum data size:** 150+ vertices is recommended to trigger RocksDB SST file generation. Very small datasets may not create any SST files. +- MinIO buckets are pre-created by `minio-init` service: `hugegraph-store0`, `hugegraph-store1`, `hugegraph-store2` + + + diff --git a/docker/cloud-storage/docker-compose.yml b/docker/cloud-storage/docker-compose.yml new file mode 100644 index 0000000000..269fc2c3c7 --- /dev/null +++ b/docker/cloud-storage/docker-compose.yml @@ -0,0 +1,237 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +name: cloud-storage + +networks: + hg-net: + name: cloud-storage-net + driver: bridge + +volumes: + minio-data: + +services: + minio: + image: minio/minio:latest + container_name: cloud-storage-minio + command: server /data --console-address ':9001' + environment: + MINIO_ROOT_USER: minioadmin + MINIO_ROOT_PASSWORD: minioadmin + ports: + - "9000:9000" + - "9001:9001" + volumes: + - minio-data:/data + networks: [hg-net] + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] + interval: 10s + timeout: 5s + retries: 20 + + minio-init: + image: minio/mc:RELEASE.2025-08-13T08-35-41Z + container_name: cloud-storage-minio-init + depends_on: + minio: + condition: service_healthy + entrypoint: > + /bin/sh -c ' + mc alias set local http://minio:9000 minioadmin minioadmin && + mc mb -p local/hugegraph-store0 || true && + mc mb -p local/hugegraph-store1 || true && + mc mb -p local/hugegraph-store2 || true && + mc anonymous set none local/hugegraph-store0 || true && + mc anonymous set none local/hugegraph-store1 || true && + mc anonymous set none local/hugegraph-store2 || true + ' + networks: [hg-net] + + pd: + build: + context: ../.. + dockerfile: docker/cloud-storage/images/pd.Dockerfile + args: + JAVA_RUNTIME_IMAGE: ${HG_PD_JAVA_RUNTIME_IMAGE:-eclipse-temurin:11-jre} + image: hugegraph/pd:cloud-storage-local + container_name: cloud-storage-pd + depends_on: + minio-init: + condition: service_completed_successfully + environment: + HG_PD_GRPC_HOST: pd + HG_PD_GRPC_PORT: "8686" + HG_PD_REST_PORT: "8620" + HG_PD_RAFT_ADDRESS: pd:8610 + HG_PD_RAFT_PEERS_LIST: pd:8610 + HG_PD_INITIAL_STORE_LIST: store0:8500,store1:8500,store2:8500 + HG_PD_DATA_PATH: /hugegraph-pd/pd_data + HG_PD_INITIAL_STORE_COUNT: "3" + ports: + - "8620:8620" + - "8686:8686" + volumes: + - ./data/pd:/hugegraph-pd/pd_data + - ./logs/pd:/hugegraph-pd/logs + networks: [hg-net] + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:8620/v1/health >/dev/null || exit 1"] + interval: 15s + timeout: 10s + retries: 30 + start_period: 60s + + store0: + build: + context: ../.. + dockerfile: docker/cloud-storage/images/store.Dockerfile + args: + JAVA_RUNTIME_IMAGE: ${HG_STORE_JAVA_RUNTIME_IMAGE:-eclipse-temurin:11-jre} + image: hugegraph/store:cloud-storage-local + container_name: cloud-storage-store0 + depends_on: + pd: + condition: service_healthy + environment: + HG_STORE_PD_ADDRESS: pd:8686 + HG_STORE_GRPC_HOST: store0 + HG_STORE_GRPC_PORT: "8500" + HG_STORE_REST_PORT: "8520" + HG_STORE_RAFT_ADDRESS: store0:8510 + HG_CLOUD_STORAGE_ENDPOINT: http://minio:9000 + HG_CLOUD_STORAGE_BUCKET: hugegraph-store0 + JAVA_OPTS: ${HG_STORE_JAVA_OPTS:--Xms4g -Xmx4g -XX:+UseG1GC} + ports: + - "8500:8500" + - "8510:8510" + - "8520:8520" + volumes: + - ./data/store0:/hugegraph-store/storage + - ./logs/store0:/hugegraph-store/logs + networks: [hg-net] + deploy: + resources: + limits: + memory: 8G + reservations: + memory: 6G + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:8520/v1/health >/dev/null || exit 1"] + interval: 15s + timeout: 10s + retries: 40 + start_period: 120s + + store1: + image: hugegraph/store:cloud-storage-local + container_name: cloud-storage-store1 + depends_on: + pd: + condition: service_healthy + environment: + HG_STORE_PD_ADDRESS: pd:8686 + HG_STORE_GRPC_HOST: store1 + HG_STORE_GRPC_PORT: "8500" + HG_STORE_REST_PORT: "8520" + HG_STORE_RAFT_ADDRESS: store1:8510 + HG_CLOUD_STORAGE_ENDPOINT: http://minio:9000 + HG_CLOUD_STORAGE_BUCKET: hugegraph-store1 + JAVA_OPTS: ${HG_STORE_JAVA_OPTS:--Xms4g -Xmx4g -XX:+UseG1GC} + ports: + - "8501:8500" + - "8511:8510" + - "8521:8520" + volumes: + - ./data/store1:/hugegraph-store/storage + - ./logs/store1:/hugegraph-store/logs + networks: [hg-net] + deploy: + resources: + limits: + memory: 8G + reservations: + memory: 6G + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:8520/v1/health >/dev/null || exit 1"] + interval: 15s + timeout: 10s + retries: 40 + start_period: 120s + + store2: + image: hugegraph/store:cloud-storage-local + container_name: cloud-storage-store2 + depends_on: + pd: + condition: service_healthy + environment: + HG_STORE_PD_ADDRESS: pd:8686 + HG_STORE_GRPC_HOST: store2 + HG_STORE_GRPC_PORT: "8500" + HG_STORE_REST_PORT: "8520" + HG_STORE_RAFT_ADDRESS: store2:8510 + HG_CLOUD_STORAGE_ENDPOINT: http://minio:9000 + HG_CLOUD_STORAGE_BUCKET: hugegraph-store2 + JAVA_OPTS: ${HG_STORE_JAVA_OPTS:--Xms4g -Xmx4g -XX:+UseG1GC} + ports: + - "8502:8500" + - "8512:8510" + - "8522:8520" + volumes: + - ./data/store2:/hugegraph-store/storage + - ./logs/store2:/hugegraph-store/logs + networks: [hg-net] + deploy: + resources: + limits: + memory: 8G + reservations: + memory: 6G + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:8520/v1/health >/dev/null || exit 1"] + interval: 15s + timeout: 10s + retries: 40 + start_period: 120s + + server: + image: hugegraph/server:1.7.0 + container_name: cloud-storage-server + depends_on: + store0: + condition: service_healthy + store1: + condition: service_healthy + store2: + condition: service_healthy + environment: + GREMLIN_SERVER_HOST: 0.0.0.0 + GREMLIN_SERVER_PORT: "8182" + ports: + - "8080:8080" + - "8182:8182" + volumes: + - ./logs/server:/hugegraph-server/logs + networks: [hg-net] + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:8080/gremlin >/dev/null || exit 1"] + interval: 15s + timeout: 10s + retries: 40 + start_period: 120s diff --git a/docker/cloud-storage/entrypoints/pd-entrypoint.sh b/docker/cloud-storage/entrypoints/pd-entrypoint.sh new file mode 100755 index 0000000000..8ad2159d5e --- /dev/null +++ b/docker/cloud-storage/entrypoints/pd-entrypoint.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +set -euo pipefail + +: "${HG_PD_GRPC_HOST:=pd}" +: "${HG_PD_GRPC_PORT:=8686}" +: "${HG_PD_REST_PORT:=8620}" +: "${HG_PD_RAFT_ADDRESS:=pd:8610}" +: "${HG_PD_RAFT_PEERS_LIST:=pd:8610}" +: "${HG_PD_DATA_PATH:=/hugegraph-pd/pd_data}" +: "${HG_PD_INITIAL_STORE_LIST:=store0:8500,store1:8500,store2:8500}" +: "${HG_PD_INITIAL_STORE_COUNT:=3}" + +json_escape() { + local s="$1" + s=${s//\\/\\\\} + s=${s//\"/\\\"} + s=${s//$'\n'/} + printf "%s" "$s" +} + +export SPRING_APPLICATION_JSON="$(cat <&2 + exit 2 + fi +} + +json_escape() { + local s="$1" + s=${s//\\/\\\\} + s=${s//\"/\\\"} + s=${s//$'\n'/} + printf "%s" "$s" +} + +require_env "HG_STORE_PD_ADDRESS" +require_env "HG_STORE_GRPC_HOST" +require_env "HG_STORE_RAFT_ADDRESS" + +: "${HG_STORE_GRPC_PORT:=8500}" +: "${HG_STORE_REST_PORT:=8520}" +: "${HG_STORE_DATA_PATH:=/hugegraph-store/storage}" +: "${HG_CLOUD_STORAGE_PROVIDER:=s3}" +: "${HG_CLOUD_STORAGE_ENABLED:=true}" +: "${HG_CLOUD_STORAGE_BUCKET:=hugegraph-store}" +: "${HG_CLOUD_STORAGE_REGION:=us-east-1}" +: "${HG_CLOUD_STORAGE_ENDPOINT:=http://minio:9000}" +: "${HG_CLOUD_STORAGE_ACCESS_KEY:=minioadmin}" +: "${HG_CLOUD_STORAGE_SECRET_KEY:=minioadmin}" +: "${HG_CLOUD_STORAGE_PATH_PREFIX:=hugegraph}" + +export SPRING_APPLICATION_JSON="$(cat < flush+compact -> verify a consistent + {CURRENT, MANIFEST, OPTIONS, SST} set in MinIO -> + wipe each store's local RocksDB state (raft/ preserved) -> + restart -> confirm data is recovered from cloud (not empty DB). + + 2. DB Deletion Cleanup Test: + create test graph -> load data -> sync to cloud -> + delete graph -> verify cloud storage prefix is cleaned up + (tests onDBDeleted() -> purgeRemotePrefix() behavior). + + --keep-stack Leave the stack running on exit (same as KEEP_UP=true). + --skip-smoke-tests Start infrastructure only; skip data load + validation tests. + --infra-only Alias of --skip-smoke-tests. +USAGE + exit 0 ;; + *) echo "unknown arg: $1 (see --help)" >&2; exit 2 ;; + esac + shift +done +log() { printf "[cloud-storage] %s\n" "$*"; } +need_cmd() { command -v "$1" >/dev/null 2>&1 || { echo "ERROR: $1 not found" >&2; exit 2; }; } + +report_line() { + local file="$1" + shift + printf "%s\n" "$*" >> "$file" +} + +report_event() { + local category="$1" + shift + report_line "$FULL_TEST_REPORT" "[$(date -u +"%Y-%m-%dT%H:%M:%SZ")] [$category] $*" +} + +record_phase() { + local phase="$1" + local status="$2" + local details="$3" + report_line "$TEST_REPORT" "$(date -u +"%Y-%m-%dT%H:%M:%SZ")\t${phase}\t${status}\t${details}" + report_event "$phase" "${status} - ${details}" +} + +init_reports() { + mkdir -p "$GENERATED_DIR" + : > "$FULL_TEST_REPORT" + : > "$TEST_REPORT" + : > "$MINIO_REPORT" + : > "$STORE_CLUSTER_LOG" + : > "$CLI_LOAD_LOG" + : > "$LOAD_DATA_FILE" + + report_line "$FULL_TEST_REPORT" "Cloud Storage E2E Test Report" + report_line "$FULL_TEST_REPORT" "started_at=${SCRIPT_START_TS}" + report_line "$FULL_TEST_REPORT" "script=${BASH_SOURCE[0]}" + report_line "$FULL_TEST_REPORT" "compose_project=${COMPOSE_PROJECT_NAME}" + report_line "$FULL_TEST_REPORT" "" + + report_line "$TEST_REPORT" "timestamp_utc phase status details" + report_line "$MINIO_REPORT" "MinIO Verification Report" + report_line "$MINIO_REPORT" "started_at=${SCRIPT_START_TS}" + report_line "$MINIO_REPORT" "" + report_line "$CLI_LOAD_LOG" "CLI/Data Load Report" + report_line "$CLI_LOAD_LOG" "started_at=${SCRIPT_START_TS}" + report_line "$CLI_LOAD_LOG" "" + report_line "$LOAD_DATA_FILE" "name age city" +} + +collect_store_logs() { + if [[ -f "$COMPOSE_FILE" ]]; then + docker compose -f "$COMPOSE_FILE" logs store0 store1 store2 > "$STORE_CLUSTER_LOG" 2>&1 || true + fi +} + +finalize_reports() { + local exit_code="$1" + local end_ts + end_ts="$(date -u +"%Y-%m-%dT%H:%M:%SZ")" + + report_line "$FULL_TEST_REPORT" "" + report_line "$FULL_TEST_REPORT" "Summary" + report_line "$FULL_TEST_REPORT" "ended_at=${end_ts}" + report_line "$FULL_TEST_REPORT" "exit_code=${exit_code}" + report_line "$FULL_TEST_REPORT" "infra_ready=${INFRA_READY}" + report_line "$FULL_TEST_REPORT" "smoke_tests=${SMOKE_STATUS}" + report_line "$FULL_TEST_REPORT" "recovery_test=${RECOVERY_STATUS}" + report_line "$FULL_TEST_REPORT" "db_deletion_cleanup_test=${DELETE_CLEANUP_STATUS}" + report_line "$FULL_TEST_REPORT" "db_recreation_no_orphan_test=${RECREATE_STATUS}" + report_line "$FULL_TEST_REPORT" "" + report_line "$FULL_TEST_REPORT" "Generated artifacts:" + report_line "$FULL_TEST_REPORT" "- ${FULL_TEST_REPORT}" + report_line "$FULL_TEST_REPORT" "- ${TEST_REPORT}" + report_line "$FULL_TEST_REPORT" "- ${MINIO_REPORT}" + report_line "$FULL_TEST_REPORT" "- ${STORE_CLUSTER_LOG}" + report_line "$FULL_TEST_REPORT" "- ${CLI_LOAD_LOG}" + report_line "$FULL_TEST_REPORT" "- ${LOAD_DATA_FILE}" +} + +assert_all_reports_present() { + local missing=0 + for report_file in "$FULL_TEST_REPORT" "$TEST_REPORT" "$MINIO_REPORT" "$STORE_CLUSTER_LOG" "$CLI_LOAD_LOG" "$LOAD_DATA_FILE"; do + if [[ ! -f "$report_file" ]]; then + echo "ERROR: expected report file missing: $report_file" >&2 + missing=$((missing + 1)) + fi + done + if (( missing > 0 )); then + echo "ERROR: $missing expected report file(s) are missing; CI contract violated" >&2 + return 1 + fi + return 0 +} + +find_dist_dir() { + local glob="$1" + local d + for d in $glob; do + [[ -d "$d" ]] && { echo "$d"; return 0; } + done + return 1 +} + +find_plugin_jar() { + local f + for f in "${REPO_ROOT}"/hugegraph-store/hg-store-cloud-s3/target/hg-store-cloud-s3-*.jar; do + [[ -f "$f" ]] || continue + case "$f" in + *-sources.jar|*-javadoc.jar|*original*) continue ;; + esac + echo "$f" + return 0 + done + return 1 +} + +prepare_artifacts() { + local pd_src store_src plugin_jar plugin_dep_dir dep_base + + pd_src="$(find_dist_dir "${REPO_ROOT}/hugegraph-pd/apache-hugegraph-pd-*")" || { + echo "ERROR: PD dist not found under ${REPO_ROOT}/hugegraph-pd/apache-hugegraph-pd-*" >&2 + echo "Run: mvn clean package -DskipTests" >&2 + exit 2 + } + store_src="$(find_dist_dir "${REPO_ROOT}/hugegraph-store/apache-hugegraph-store-*")" || { + echo "ERROR: Store dist not found under ${REPO_ROOT}/hugegraph-store/apache-hugegraph-store-*" >&2 + echo "Run: mvn clean package -DskipTests" >&2 + exit 2 + } + plugin_jar="$(find_plugin_jar)" || { + echo "ERROR: S3 plugin jar not found under ${REPO_ROOT}/hugegraph-store/hg-store-cloud-s3/target/" >&2 + echo "Run: mvn clean package -DskipTests" >&2 + exit 2 + } + + log "preparing Docker artifacts in ${ARTIFACTS_DIR}" + rm -rf "${ARTIFACTS_DIR}" + mkdir -p "${ARTIFACTS_DIR}/pd-dist" "${ARTIFACTS_DIR}/store-dist" "${ARTIFACTS_DIR}/plugins" + + cp -R "${pd_src}/." "${ARTIFACTS_DIR}/pd-dist/" + cp -R "${store_src}/." "${ARTIFACTS_DIR}/store-dist/" + cp "${plugin_jar}" "${ARTIFACTS_DIR}/plugins/" + + plugin_dep_dir="${REPO_ROOT}/hugegraph-store/hg-store-cloud-s3/target/dependency" + if [[ -d "${plugin_dep_dir}" ]]; then + # Keep only external plugin deps; internal HugeGraph jars must come from /hugegraph-store/lib. + for dep in "${plugin_dep_dir}"/*.jar; do + [[ -f "${dep}" ]] || continue + dep_base="$(basename "${dep}")" + case "${dep_base}" in + hg-*.jar|hugegraph-*.jar) continue ;; + esac + cp "${dep}" "${ARTIFACTS_DIR}/plugins/" + done + else + log "warning: plugin dependency dir not found at ${plugin_dep_dir}; continuing with plugin jar only" + fi + + # Prevent duplicate SLF4J binding clashes from plugin dependency staging. + rm -f "${ARTIFACTS_DIR}"/plugins/log4j-slf4j-impl-*.jar "${ARTIFACTS_DIR}"/plugins/slf4j-log4j12-*.jar || true +} + +ensure_image() { + local img="$1" + docker image inspect "$img" >/dev/null 2>&1 && return 0 + # Check if it's a local build image (contains "cloud-storage-local") + if [[ "$img" == *"cloud-storage-local"* ]]; then + log "image $img is local-build only (will be built via docker compose build)" + return 0 + fi + log "pulling $img..." + docker pull "$img" >/dev/null || exit 3 +} +ensure_minio_buckets() { + for bucket in "$S3_BUCKET_STORE0" "$S3_BUCKET_STORE1" "$S3_BUCKET_STORE2"; do + docker run --rm --network "$1" --entrypoint /bin/sh "$MINIO_MC_IMAGE" -c "mc alias set local http://minio:9000 $MINIO_ROOT_USER $MINIO_ROOT_PASSWORD >/dev/null && mc mb --ignore-existing local/$bucket >/dev/null" + done +} +wait_svc() { + local svc max=120 i=0 + svc="$1" + while [[ $i -lt $max ]]; do + local cid status + cid=$(docker compose -f "$COMPOSE_FILE" ps -q "$svc" 2>/dev/null || true) + [[ -z "$cid" ]] && { sleep 2; i=$((i+1)); continue; } + status=$(docker inspect --format '{{if .State.Health}}{{.State.Health.Status}}{{else}}{{.State.Status}}{{end}}' "$cid" 2>/dev/null || true) + [[ "$status" == "healthy" || "$status" == "running" ]] && { log "✓ $svc"; return 0; } + sleep 2 + i=$((i+1)) + done + echo "ERROR: $svc timeout" >&2 + return 1 +} +wait_http() { + local url max=120 i=0 + url="$1" + while [[ $i -lt $max ]]; do + [[ $(curl -so /dev/null -w "%{http_code}" "$url" 2>/dev/null) == "200" ]] && { log "✓ $url ready"; return 0; } + sleep 2 + i=$((i+1)) + done + echo "ERROR: $url timeout" >&2 + return 1 +} + +delete_graph_with_confirm() { + local graph_api="$1" + local resp_file status + resp_file="/tmp/hg-delete-$$.json" + status=$(curl -s -o "$resp_file" -w "%{http_code}" -X DELETE "$graph_api" \ + --get --data-urlencode "confirm_message=I'm sure to drop the graph") + if [[ "$status" != 2* ]]; then + echo "ERROR: graph delete failed with HTTP ${status}: ${graph_api}" >&2 + head -80 "$resp_file" >&2 || true + rm -f "$resp_file" || true + return 1 + fi + rm -f "$resp_file" || true +} + +clear_graph_with_confirm() { + local graph_api="$1" + local resp_file status clear_api + resp_file="/tmp/hg-clear-$$.json" + clear_api="${graph_api}/clear" + status=$(curl -s -o "$resp_file" -w "%{http_code}" -X DELETE "$clear_api" \ + --get --data-urlencode "confirm_message=I'm sure to delete all data") + if [[ "$status" != 2* ]]; then + echo "ERROR: graph clear failed with HTTP ${status}: ${clear_api}" >&2 + head -80 "$resp_file" >&2 || true + rm -f "$resp_file" || true + return 1 + fi + rm -f "$resp_file" || true +} + +on_exit() { + local status=$? + local assertion_rc + collect_store_logs + finalize_reports "$status" + assert_all_reports_present + assertion_rc=$? + # Preserve original exit code unless test passed but assertion failed + [[ $status -eq 0 && $assertion_rc -ne 0 ]] && status=$assertion_rc + [[ "$KEEP_UP" == "true" ]] || (docker compose -f "$COMPOSE_FILE" down -v 2>/dev/null || true) +} +trap on_exit EXIT + +# ----------------------------------------------------------------------------- +# Total-loss recovery E2E helpers +# ----------------------------------------------------------------------------- + +# Current vertex count via the server graph API (0 if the query fails). +graph_vertex_count() { + curl -s --compressed "${GRAPH_API_BASE}/graph/vertices" 2>/dev/null \ + | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('vertices',[])))" \ + 2>/dev/null || echo 0 +} + +# Create a minimal schema and insert enough vertices to generate SST files. +load_test_data() { + log "loading test data (schema + 150 vertices)..." + report_event "graph-load" "starting schema + vertex load" + local insert_ok=0 insert_fail=0 + + # Create deterministic input file promised by README. + for i in $(seq 1 150); do + printf "person_%s\t%s\tcity_%s\n" "$i" "$((20 + i % 50))" "$((i % 5))" >> "$LOAD_DATA_FILE" + done + + for pk in '{"name":"name","data_type":"TEXT","cardinality":"SINGLE"}' \ + '{"name":"age","data_type":"INT","cardinality":"SINGLE"}' \ + '{"name":"city","data_type":"TEXT","cardinality":"SINGLE"}'; do + curl -s -o /dev/null -X POST "${GRAPH_API_BASE}/schema/propertykeys" \ + -H 'Content-Type: application/json' -d "$pk" || true + done + curl -s -o /dev/null -X POST "${GRAPH_API_BASE}/schema/vertexlabels" \ + -H 'Content-Type: application/json' \ + -d '{"name":"person","id_strategy":"AUTOMATIC","properties":["name","age","city"]}' || true + + while IFS=$'\t' read -r name age city; do + [[ "$name" == "name" ]] && continue + local payload code + payload="{\"label\":\"person\",\"properties\":{\"name\":\"${name}\",\"age\":${age},\"city\":\"${city}\"}}" + code=$(curl -s -o /dev/null -w "%{http_code}" -X POST "${GRAPH_API_BASE}/graph/vertices" \ + -H 'Content-Type: application/json' -d "$payload" || true) + if [[ "$code" =~ ^2 ]]; then + insert_ok=$((insert_ok + 1)) + if (( insert_ok % 25 == 0 )); then + report_line "$CLI_LOAD_LOG" "[$(date -u +"%Y-%m-%dT%H:%M:%SZ")] inserted=${insert_ok} failed=${insert_fail}" + fi + else + insert_fail=$((insert_fail + 1)) + report_line "$CLI_LOAD_LOG" "[$(date -u +"%Y-%m-%dT%H:%M:%SZ")] vertex_insert_failed name=${name} http=${code}" + fi + done < "$LOAD_DATA_FILE" + + report_line "$CLI_LOAD_LOG" "[$(date -u +"%Y-%m-%dT%H:%M:%SZ")] load_complete inserted=${insert_ok} failed=${insert_fail}" + + if (( insert_ok == 0 )); then + record_phase "graph-load" "FAIL" "no vertices inserted" + echo "ERROR: failed to insert any vertices" >&2 + return 1 + fi + + log " ✓ inserted ${insert_ok} vertices (count now = $(graph_vertex_count))" + record_phase "graph-load" "PASS" "inserted=${insert_ok};failed=${insert_fail};vertex_count=$(graph_vertex_count)" +} + +# Assert every bucket holds a consistent {CURRENT, MANIFEST, OPTIONS, SST} set. +# This deterministic check verifies metadata objects from ordered +# CURRENT/MANIFEST/OPTIONS mirroring ran, so the SSTs are no longer orphans. +verify_metadata_in_minio() { + local out rc + set +e + out=$(docker run --rm --entrypoint /bin/sh --network "$NETWORK" "$MINIO_MC_IMAGE" -c ' + mc alias set local http://minio:9000 '"$MINIO_ROOT_USER $MINIO_ROOT_PASSWORD"' >/dev/null 2>&1 + rc=0 + for b in '"$S3_BUCKET_STORE0 $S3_BUCKET_STORE1 $S3_BUCKET_STORE2"'; do + sst=$(mc find local/$b --name "*.sst" 2>/dev/null | wc -l | tr -d " ") + cur=$(mc find local/$b --name "CURRENT" 2>/dev/null | wc -l | tr -d " ") + man=$(mc find local/$b --name "MANIFEST-*" 2>/dev/null | wc -l | tr -d " ") + opt=$(mc find local/$b --name "OPTIONS-*" 2>/dev/null | wc -l | tr -d " ") + printf "### %s: sst=%s CURRENT=%s MANIFEST=%s OPTIONS=%s\n" "$b" "$sst" "$cur" "$man" "$opt" + if [ "$cur" -lt 1 ] || [ "$man" -lt 1 ] || [ "$sst" -lt 1 ]; then + echo " ✗ recovery metadata/SST set incomplete for $b"; rc=1 + else + echo " ✓ consistent {CURRENT, MANIFEST, OPTIONS, SST} set present" + fi + done + exit $rc + ' 2>&1) + rc=$? + set -e + + printf "%s\n" "$out" | tee -a "$MINIO_REPORT" + return $rc +} + +# Stop a store, wipe its RocksDB state-machine data (db/ + metadata graph) while +# preserving raft/ and snapshot/, then start it again. This models local +# state-machine loss where the node still rejoins its raft group but must +# re-hydrate its RocksDB data from cloud on open (pre-hydration path). +wipe_store_rocksdb_state() { + local idx="$1" + local vol="${COMPOSE_PROJECT_NAME}_hg-store${idx}-data" + docker compose -f "$COMPOSE_FILE" stop "store${idx}" >/dev/null 2>&1 || true + docker run --rm --entrypoint /bin/sh -v "${vol}:/s" "$MINIO_MC_IMAGE" -c ' + cd /s 2>/dev/null || exit 0 + for d in *; do + case "$d" in + raft|snapshot) ;; # keep raft log + snapshot so the node rejoins cleanly + *) rm -rf "$d" ;; # drop db/ and the metadata graph -> must come back from cloud + esac + done + echo " store'"${idx}"' storage after wipe: $(ls -1 | tr "\n" " ")" + ' || true + docker compose -f "$COMPOSE_FILE" start "store${idx}" >/dev/null 2>&1 || true +} + +count_objects_in_prefix() { + local bucket="$1" + local prefix="$2" + docker run --rm --entrypoint /bin/sh --network "$NETWORK" "$MINIO_MC_IMAGE" -c ' + mc alias set local http://minio:9000 '"$MINIO_ROOT_USER $MINIO_ROOT_PASSWORD"' >/dev/null 2>&1 + target="local/'"$bucket"'/'"$prefix"'" + if mc ls --recursive "$target" >/tmp/prefix_ls.txt 2>/dev/null; then + wc -l < /tmp/prefix_ls.txt | tr -d " " + else + echo 0 + fi + ' + } + +put_probe_object_in_prefix() { + local bucket="$1" + local prefix="$2" + local probe_key="${prefix%/}/marker-$(date +%s).txt" + docker run --rm --entrypoint /bin/sh --network "$NETWORK" "$MINIO_MC_IMAGE" -c ' + mc alias set local http://minio:9000 '"$MINIO_ROOT_USER $MINIO_ROOT_PASSWORD"' >/dev/null 2>&1 + printf "db-delete-probe\n" | mc pipe local/'"$bucket"'/'"$probe_key"' >/dev/null + ' + log " ✓ wrote cloud probe object: s3://${bucket}/${probe_key}" +} + +find_graph_db_prefix() { + local bucket="$1" + local graph_name="$2" + local candidate rel + + candidate=$(docker run --rm --entrypoint /bin/sh --network "$NETWORK" "$MINIO_MC_IMAGE" -c ' + mc alias set local http://minio:9000 '"$MINIO_ROOT_USER $MINIO_ROOT_PASSWORD"' >/dev/null 2>&1 + mc find local/'"$bucket"' --name "CURRENT" 2>/dev/null + ' | grep "/${graph_name}/" | head -n1 || true) + + if [[ -z "$candidate" ]]; then + candidate=$(docker run --rm --entrypoint /bin/sh --network "$NETWORK" "$MINIO_MC_IMAGE" -c ' + mc alias set local http://minio:9000 '"$MINIO_ROOT_USER $MINIO_ROOT_PASSWORD"' >/dev/null 2>&1 + mc find local/'"$bucket"' --name "*.sst" 2>/dev/null + ' | grep "/${graph_name}/" | head -n1 || true) + fi + + [[ -z "$candidate" ]] && return 1 + rel="${candidate#local/${bucket}/}" + + if [[ "$rel" == "${graph_name}/"* ]]; then + : + elif [[ "$rel" == *"/${graph_name}/"* ]]; then + rel="${rel#*/${graph_name}/}" + rel="${graph_name}/${rel}" + else + return 1 + fi + + if [[ "$rel" == */CURRENT ]]; then + echo "${rel%/CURRENT}/" + else + echo "${rel%/*}/" + fi +} + +run_recovery_test() { + log "=== Total-loss recovery E2E ===" + local before after + + before=$(graph_vertex_count) + log "baseline vertex count = ${before}" + if [[ "$before" == "0" ]]; then + echo "ERROR: no data to recover (baseline is 0); load data first" >&2 + return 1 + fi + + log "forcing flush + compaction (restart stores) so data lands in SST files..." + docker restart cloud-storage-store0 cloud-storage-store1 cloud-storage-store2 >/dev/null + wait_svc "store0" 240; wait_svc "store1" 240; wait_svc "store2" 240 + wait_http "${GRAPH_API_BASE}/graph/vertices" 180 + + + log "verifying a consistent metadata set is durable in MinIO..." + verify_metadata_in_minio || { echo "ERROR: recovery metadata not durable in cloud" >&2; return 1; } + + log "simulating local RocksDB state loss on all stores (raft/ preserved)..." + for i in 0 1 2; do wipe_store_rocksdb_state "$i"; done + wait_svc "store0" 240; wait_svc "store1" 240; wait_svc "store2" 240 + wait_http "${GRAPH_API_BASE}/graph/vertices" 240 + + log "checking store logs for pre-hydration / restore-consistency..." + if docker compose -f "$COMPOSE_FILE" logs store0 store1 store2 2>/dev/null \ + | grep -qi "Cloud restore inconsistent"; then + echo "ERROR: consistent-restore guard tripped (CURRENT referenced a missing manifest)" >&2 + return 1 + fi + docker compose -f "$COMPOSE_FILE" logs store0 store1 store2 2>/dev/null \ + | grep -i "Cloud pre-hydration finished" | tail -3 \ + || log " (no pre-hydration log lines matched; check DEBUG logs manually)" + + after=$(graph_vertex_count) + log "recovered vertex count = ${after} (baseline was ${before})" + if [[ "$after" == "$before" ]]; then + log "✓ RECOVERY SUCCESS: data fully recovered from cloud after local state loss" + else + echo "ERROR: recovery mismatch (before=${before}, after=${after})" >&2 + return 1 + fi + } + +run_db_deletion_cleanup_test() { + log "=== DB deletion + cloud storage prefix cleanup E2E ===" + local graph_name test_api count_before count_after probe_prefix db_cloud_prefix + graph_name="${GRAPH_API_BASE##*/}" + test_api="${GRAPH_API_BASE}" + + log "using existing graph '${graph_name}' created/populated by recovery test" + + db_cloud_prefix=$(find_graph_db_prefix "$S3_BUCKET_STORE0" "$graph_name" || true) + if [[ -z "$db_cloud_prefix" ]]; then + echo "ERROR: failed to detect cloud DB prefix for graph '${graph_name}' in bucket '${S3_BUCKET_STORE0}'" >&2 + log " sample objects in bucket for debugging:" + docker run --rm --entrypoint /bin/sh --network "$NETWORK" "$MINIO_MC_IMAGE" -c ' + mc alias set local http://minio:9000 '"$MINIO_ROOT_USER $MINIO_ROOT_PASSWORD"' >/dev/null 2>&1 + mc ls --recursive local/'"$S3_BUCKET_STORE0"' | head -20 + ' || true + return 1 + fi + probe_prefix="${db_cloud_prefix}" + log "detected graph DB cloud prefix: ${db_cloud_prefix}" + + # Write a deterministic probe object so the delete-prefix test always has cloud data to prune. + log "writing probe object into cloud probe prefix '${probe_prefix}'..." + put_probe_object_in_prefix "$S3_BUCKET_STORE0" "${probe_prefix}" + + count_before=$(count_objects_in_prefix "$S3_BUCKET_STORE0" "${probe_prefix}" || echo "0") + count_before="${count_before//[^0-9]/}" + if [[ -z "$count_before" || "$count_before" -eq 0 ]]; then + echo "ERROR: failed to create/observe probe object in cloud probe prefix '${probe_prefix}'" >&2 + return 1 + fi + log " ✓ objects in probe prefix before delete: ${count_before}" + + log "clearing graph data for '${graph_name}'..." + clear_graph_with_confirm "${test_api}" || { + echo "ERROR: failed to clear graph '${graph_name}'" >&2 + return 1 + } + sleep 5 + + log "verifying cloud probe prefix has been cleaned up..." + count_after=$(count_objects_in_prefix "$S3_BUCKET_STORE0" "${probe_prefix}" || echo "0") + log " objects in probe prefix after deletion: ${count_after}" + + if [[ "$count_after" == "0" || "$count_after" == "" ]]; then + log "✓ DB DELETION CLEANUP SUCCESS: cloud probe prefix pruned after DB deletion" + else + echo "ERROR: cloud probe prefix not cleaned up (objects remaining: ${count_after})" >&2 + log " listing remaining objects:" + docker run --rm --entrypoint /bin/sh --network "$NETWORK" "$MINIO_MC_IMAGE" -c ' + mc alias set local http://minio:9000 '"$MINIO_ROOT_USER $MINIO_ROOT_PASSWORD"' >/dev/null 2>&1 + mc ls --recursive local/'"$S3_BUCKET_STORE0"'/'"$probe_prefix"' | head -20 + ' || true + return 1 + fi + + log "skip graph delete in single-graph deployment; clear() is the deletion-equivalent path under test" + } + + run_db_recreation_no_orphan_test() { + log "=== Post-clear no-orphan rehydration E2E ===" + local graph_name test_api + graph_name="${GRAPH_API_BASE##*/}" + test_api="${GRAPH_API_BASE%/graphs/*}/graphs/${graph_name}" + + log "simulating local RocksDB loss after clear to verify no stale cloud rehydration..." + for i in 0 1 2; do wipe_store_rocksdb_state "$i"; done + wait_svc "store0" 240; wait_svc "store1" 240; wait_svc "store2" 240 + wait_http "${GRAPH_API_BASE}/graph/vertices" 240 + + log "verifying graph remains empty (no orphaned data rehydrated from cloud)..." + local vertex_count + vertex_count=$(curl -s --compressed "${test_api}/graph/vertices" 2>/dev/null \ + | python3 -c "import sys,json; print(len(json.load(sys.stdin).get('vertices',[])))" \ + 2>/dev/null || echo "0") + + log " vertex count in recreated '${graph_name}': ${vertex_count}" + + if [[ "$vertex_count" == "0" ]]; then + log "✓ RECREATION NO-ORPHAN SUCCESS: deleted graph marker prevents rehydration of old data" + else + echo "ERROR: orphaned data found in recreated graph (vertices: ${vertex_count})" >&2 + log " this suggests deletion markers were not properly written or preserved" + return 1 + fi + + log "leaving graph '${graph_name}' in place" + } + +need_cmd docker curl python3 +init_reports +report_event "bootstrap" "initializing stack and reports" +log "pulling images..." +ensure_image "$MINIO_IMAGE" +ensure_image "$MINIO_MC_IMAGE" +ensure_image "$HG_PD_IMAGE" +ensure_image "$HG_STORE_IMAGE" +ensure_image "$HG_SERVER_IMAGE" +mkdir -p "$GENERATED_DIR" +mkdir -p "$SERVER_GRAPHS_DIR" +cat > "$SERVER_GRAPH_CONF" << 'PROPS' +gremlin.graph=org.apache.hugegraph.HugeFactory +backend=hstore +serializer=binary +store=hugegraph +task.scheduler_type=local +pd.peers=pd:8686 +PROPS +mkdir -p "$(dirname "$COMPOSE_FILE")" +docker compose -f "$COMPOSE_FILE" down -v 2>/dev/null || true +log "generating docker-compose.yml..." +cat > "$COMPOSE_FILE" << 'YAML' +services: + minio: + image: minio/minio:latest + container_name: cloud-storage-minio + command: server /data --console-address ":9001" + environment: + MINIO_ROOT_USER: minioadmin + MINIO_ROOT_PASSWORD: minioadmin + ports: ["9000:9000", "9001:9001"] + volumes: [hg-minio-data:/data] + networks: [hg-net] + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:9000/minio/health/live >/dev/null || exit 1"] + interval: 5s + timeout: 5s + retries: 40 + pd: + build: + context: ../../../ + dockerfile: docker/cloud-storage/images/pd.Dockerfile + image: hugegraph/pd:cloud-storage-local + container_name: cloud-storage-pd + hostname: pd + depends_on: + minio: + condition: service_healthy + environment: + HG_PD_GRPC_HOST: pd + HG_PD_GRPC_PORT: "8686" + HG_PD_REST_PORT: "8620" + HG_PD_RAFT_ADDRESS: pd:8610 + HG_PD_RAFT_PEERS_LIST: pd:8610 + HG_PD_INITIAL_STORE_LIST: store0:8500,store1:8500,store2:8500 + HG_PD_INITIAL_STORE_COUNT: "3" + HG_PD_DATA_PATH: /hugegraph-pd/pd_data + ports: ["8620:8620", "8686:8686"] + volumes: [hg-pd-data:/hugegraph-pd/pd_data] + networks: [hg-net] + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:8620/v1/health >/dev/null || exit 1"] + interval: 10s + timeout: 5s + retries: 30 + store0: + build: + context: ../../../ + dockerfile: docker/cloud-storage/images/store.Dockerfile + image: hugegraph/store:cloud-storage-local + container_name: cloud-storage-store0 + hostname: store0 + depends_on: + pd: + condition: service_healthy + environment: + HG_STORE_PD_ADDRESS: pd:8686 + HG_STORE_GRPC_HOST: store0 + HG_STORE_GRPC_PORT: "8500" + HG_STORE_REST_PORT: "8520" + HG_STORE_RAFT_ADDRESS: store0:8510 + HG_STORE_DATA_PATH: /hugegraph-store/storage + HG_CLOUD_STORAGE_ENABLED: "true" + HG_CLOUD_STORAGE_BUCKET: "hugegraph-store0" + HG_CLOUD_STORAGE_ENDPOINT: "http://minio:9000" + HG_CLOUD_STORAGE_REGION: "us-east-1" + HG_CLOUD_STORAGE_ACCESS_KEY: "minioadmin" + HG_CLOUD_STORAGE_SECRET_KEY: "minioadmin" + HG_CLOUD_STORAGE_PATH_STYLE: "true" + HG_CLOUD_STORAGE_SYNC_INTERVAL_SECONDS: "30" + HG_CLOUD_STORAGE_SYNCHRONOUS_SST_UPLOAD_MODE: "true" + ports: ["8520:8520"] + volumes: [hg-store0-data:/hugegraph-store/storage] + networks: [hg-net] + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:8520/v1/health >/dev/null || exit 1"] + interval: 10s + timeout: 10s + retries: 40 + start_period: 60s + store1: + build: + context: ../../../ + dockerfile: docker/cloud-storage/images/store.Dockerfile + image: hugegraph/store:cloud-storage-local + container_name: cloud-storage-store1 + hostname: store1 + depends_on: + pd: + condition: service_healthy + environment: + HG_STORE_PD_ADDRESS: pd:8686 + HG_STORE_GRPC_HOST: store1 + HG_STORE_GRPC_PORT: "8500" + HG_STORE_REST_PORT: "8520" + HG_STORE_RAFT_ADDRESS: store1:8510 + HG_STORE_DATA_PATH: /hugegraph-store/storage + HG_CLOUD_STORAGE_ENABLED: "true" + HG_CLOUD_STORAGE_BUCKET: "hugegraph-store1" + HG_CLOUD_STORAGE_ENDPOINT: "http://minio:9000" + HG_CLOUD_STORAGE_REGION: "us-east-1" + HG_CLOUD_STORAGE_ACCESS_KEY: "minioadmin" + HG_CLOUD_STORAGE_SECRET_KEY: "minioadmin" + HG_CLOUD_STORAGE_PATH_STYLE: "true" + HG_CLOUD_STORAGE_SYNC_INTERVAL_SECONDS: "30" + HG_CLOUD_STORAGE_SYNCHRONOUS_SST_UPLOAD_MODE: "true" + ports: ["8521:8520"] + volumes: [hg-store1-data:/hugegraph-store/storage] + networks: [hg-net] + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:8520/v1/health >/dev/null || exit 1"] + interval: 10s + timeout: 10s + retries: 40 + start_period: 60s + store2: + build: + context: ../../../ + dockerfile: docker/cloud-storage/images/store.Dockerfile + image: hugegraph/store:cloud-storage-local + container_name: cloud-storage-store2 + hostname: store2 + depends_on: + pd: + condition: service_healthy + environment: + HG_STORE_PD_ADDRESS: pd:8686 + HG_STORE_GRPC_HOST: store2 + HG_STORE_GRPC_PORT: "8500" + HG_STORE_REST_PORT: "8520" + HG_STORE_RAFT_ADDRESS: store2:8510 + HG_STORE_DATA_PATH: /hugegraph-store/storage + HG_CLOUD_STORAGE_ENABLED: "true" + HG_CLOUD_STORAGE_BUCKET: "hugegraph-store2" + HG_CLOUD_STORAGE_ENDPOINT: "http://minio:9000" + HG_CLOUD_STORAGE_REGION: "us-east-1" + HG_CLOUD_STORAGE_ACCESS_KEY: "minioadmin" + HG_CLOUD_STORAGE_SECRET_KEY: "minioadmin" + HG_CLOUD_STORAGE_PATH_STYLE: "true" + HG_CLOUD_STORAGE_SYNC_INTERVAL_SECONDS: "30" + HG_CLOUD_STORAGE_SYNCHRONOUS_SST_UPLOAD_MODE: "true" + ports: ["8522:8520"] + volumes: [hg-store2-data:/hugegraph-store/storage] + networks: [hg-net] + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:8520/v1/health >/dev/null || exit 1"] + interval: 10s + timeout: 10s + retries: 40 + start_period: 60s + server: + image: hugegraph/server:1.7.0 + container_name: cloud-storage-server + hostname: server + depends_on: + store0: + condition: service_healthy + store1: + condition: service_healthy + store2: + condition: service_healthy + environment: + STORE_REST: store0:8520 + ports: ["8080:8080"] + volumes: + - ${SERVER_GRAPHS_DIR}:/hugegraph-server/conf/graphs + networks: [hg-net] + healthcheck: + test: ["CMD-SHELL", "curl -fsS http://localhost:8080/versions >/dev/null || exit 1"] + interval: 10s + timeout: 5s + retries: 40 + start_period: 60s +networks: + hg-net: + driver: bridge +volumes: + hg-minio-data: + hg-pd-data: + hg-store0-data: + hg-store1-data: + hg-store2-data: +YAML +prepare_artifacts +log "building local cloud-storage images..." +docker compose -f "$COMPOSE_FILE" build --no-cache 2>&1 | grep -E "^(Building|FINISHED|\[|Successfully|Error)" || true +log "starting minio and pd first..." +docker compose -f "$COMPOSE_FILE" up -d minio pd +wait_svc "minio" 120 +wait_svc "pd" 180 +NETWORK="${COMPOSE_PROJECT_NAME}_hg-net" +log "creating MinIO buckets before store startup..." +ensure_minio_buckets "$NETWORK" +log "starting stores and server..." +docker compose -f "$COMPOSE_FILE" up -d store0 store1 store2 server +log "waiting for stores..." +wait_svc "store0" 180 +wait_svc "store1" 180 +wait_svc "store2" 180 +wait_svc "server" 180 +log "waiting for graph backend..." +wait_http "$GRAPH_API_BASE/graph/vertices" 60 +INFRA_READY="true" +record_phase "infra-health" "PASS" "minio,pd,store0,store1,store2,server healthy" +log "✓ SUCCESS: Cloud storage infrastructure ready" + +if [[ "$SKIP_SMOKE_TESTS" == "true" ]]; then + SMOKE_STATUS="SKIPPED" + record_phase "smoke-tests" "SKIPPED" "SKIP_SMOKE_TESTS=true" + log "SKIP_SMOKE_TESTS=true -> skipping data creation and validation phases" + log "✓ SUCCESS: infrastructure-only mode complete" + exit 0 +fi + +SMOKE_STATUS="RUNNING" +if ! load_test_data; then + SMOKE_STATUS="FAILED" + exit 1 +fi + +if run_recovery_test; then + RECOVERY_STATUS="PASS" + record_phase "recovery-test" "PASS" "vertex count recovered after local state loss" +else + RECOVERY_STATUS="FAIL" + record_phase "recovery-test" "FAIL" "recovery workflow failed" + SMOKE_STATUS="FAILED" + exit 1 +fi + +if run_db_deletion_cleanup_test; then + DELETE_CLEANUP_STATUS="PASS" + record_phase "db-deletion-cleanup" "PASS" "cloud prefix cleaned after clear()" +else + DELETE_CLEANUP_STATUS="FAIL" + record_phase "db-deletion-cleanup" "FAIL" "cloud prefix cleanup failed" + SMOKE_STATUS="FAILED" + exit 1 +fi + +if run_db_recreation_no_orphan_test; then + RECREATE_STATUS="PASS" + record_phase "db-recreation-no-orphan" "PASS" "recreated graph remained empty" +else + RECREATE_STATUS="FAIL" + record_phase "db-recreation-no-orphan" "FAIL" "orphan data was rehydrated" + SMOKE_STATUS="FAILED" + exit 1 +fi + +SMOKE_STATUS="PASS" +record_phase "smoke-tests" "PASS" "all cloud storage E2E tests passed" +log "✓ SUCCESS: all cloud storage E2E tests passed" diff --git a/docker/docker-compose.dev.yml b/docker/docker-compose.dev.yml index aa0736a38b..31a8be72d9 100644 --- a/docker/docker-compose.dev.yml +++ b/docker/docker-compose.dev.yml @@ -56,7 +56,7 @@ services: store: build: context: .. - dockerfile: hugegraph-store/Dockerfile + dockerfile: ${HG_STORE_DOCKERFILE:-hugegraph-store/Dockerfile} container_name: hg-store hostname: store restart: unless-stopped diff --git a/hugegraph-store/docs/pluggable-cloud-storage-architecture.md b/hugegraph-store/docs/pluggable-cloud-storage-architecture.md new file mode 100644 index 0000000000..5ac5eee3bf --- /dev/null +++ b/hugegraph-store/docs/pluggable-cloud-storage-architecture.md @@ -0,0 +1,998 @@ +# Pluggable Cloud Storage Architecture (HStore) + +This document explains: + +- The existing HStore workflow (without cloud storage) +- The new pluggable cloud storage workflow (cloud-backed SST lifecycle) +- Write path and read path behavior +- Failure handling and operational recovery scenarios + +## Overview + +HStore keeps partition data in local RocksDB on each Store node. The pluggable cloud-storage +feature extends this by mirroring SST file lifecycle events (create/delete) to an external +object store and hydrating missing files back when needed (startup and read-miss). + +- Existing path: local RocksDB is the primary online read/write path. +- New extension: cloud object storage acts as a pluggable SST mirror and recovery source. +- Provider model: runtime-selected SPI provider via `CloudStorageProviderFactory`. + +## Architecture Diagram + +```text ++---------------------------+ +| Client Layer | +| Gremlin/REST/Cypher | ++-------------|-------------+ + v ++---------------------------+ +----------------+ +| HugeGraph Server |-->| PD Cluster | ++-------------|-------------+ +----------------+ + v ++----------------------------------------------+ +| Store Cluster (Raft replication) | +| | +| +-----------------------------------------+ | +| | WAL + MemTable -> Local RocksDB SST | | +| +--------------------+--------------------+ | +| | | +| v | +| +-----------------------------------------+ | +| | Pluggable Cloud Storage Workflow (NEW) | | +| | CloudStorageEventListener (NEW) | | +| | -> CloudStorageProviderFactory (NEW) | | +| | -> CloudStorageProvider (NEW) | | +| +-----------------------------------------+ | ++----------------------|-----------------------+ + v + +-------------------------------+ + | Cloud Storage (NEW) | + | +-------------+ +----+ +----+ | + | |S3 compatible| |ADLS| |GCP | | + | +-------------+ +----+ +----+ | + +-------------------------------+ + +``` + +- `Client Layer`: External clients issuing Gremlin/REST/Cypher read-write requests. +- `HugeGraph Server`: API/query layer that routes graph requests to PD and Store. +- `PD Cluster`: Placement/metadata control plane (partition mapping, leader scheduling). +- `Store Cluster`: Raft-based data plane where writes are replicated and persisted. +- `WAL + MemTable -> Local RocksDB SST`: Local durability and compaction pipeline in each Store node. +- `Pluggable Cloud Storage Workflow (NEW)`: SST lifecycle hook path for upload/delete/download/list. +- `CloudStorageEventListener (NEW)`: Captures RocksDB file events and triggers cloud operations. +- `CloudStorageProviderFactory (NEW)`: SPI loader that selects and initializes the active provider. +- `CloudStorageProvider (NEW)`: Provider implementation (`s3`, etc.) used for object operations. +- `Cloud Storage (NEW)`: Remote object backend options (S3 compatible, ADSL, GCP). + + +## New Configuration Options + +The pluggable cloud-storage behavior is controlled from `application.yml` under +`cloud.storage`. + +| Configuration | Default | Description | +|------------------------------------------------------|-------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `cloud.storage.enabled` | `false` | Enables/disables cloud storage integration. | +| `cloud.storage.provider` | `s3` | Active provider name. Must match `CloudStorageProvider#providerName()`. | +| `cloud.storage.path-prefix` | `hugegraph` | Prefix prepended to all remote object keys. | +| `cloud.storage.startup-hydration-enabled` | `true` | Downloads missing remote files on DB opening before normal serving. | +| `cloud.storage.read-miss-guard-window-ms` | `3000` | Guard window to throttle repeated read-miss hydration attempts per db/table. Values `<= 0` disable throttling. | +| `cloud.storage.upload-retry-max-attempts` | `3` | Whole-file retry attempts after a first upload failure. Default `3` retries are enabled to protect against transient network errors.
Set to `0` to disable whole-file retries (failures go directly to DLQ) when the provider already has sufficient internal retry logic. `CloudStorageNonRetryableException` always bypasses retries and goes directly to DLQ regardless of this value. | +| `cloud.storage.upload-retry-initial-delay-ms` | `1000` | Delay before first whole-file retry; subsequent retries use exponential backoff. Only used when `upload-retry-max-attempts > 0`. | +| `cloud.storage.upload-retry-max-delay-ms` | `60000` | Upper bound for exponential backoff delay between whole-file retry attempts. Only used when `upload-retry-max-attempts > 0`. | +| `cloud.storage.upload-backpressure-high-watermark` | `64` | When `> 0`, slows the RocksDB flush/compaction thread in `onTableFileCreated` while the pending-upload backlog (retry-queue in-flight + DLQ size) exceeds this value, bounding the amount of local-only at-risk data. Blocked at most 30 s per event. `0` disables backpressure. | +| `cloud.storage.wal-mode` | `flush` | WAL durability mode for metadata mirroring. `flush` forces a MemTable flush before each capture; at most the un-flushed tail since the last sync is lost on an uncontrolled crash. `wal` also mirrors active `*.log` WAL segments alongside metadata and replays them on restore (lower RPO, at the cost of more frequent small uploads). | + +Notes: + +- Keep `path-prefix` stable across restarts to preserve object-key continuity. +- If `provider` is not found on classpath, initialization fails fast in `CloudStorageProviderFactory`. +- Upload retry uses a two-layer model: S3 part-level retries (inside one `uploadFile()` call) are handled by the provider; whole-file retries are handled by `CloudUploadRetryQueue` when `upload-retry-max-attempts > 0` (default `3`). +- `CloudStorageNonRetryableException` thrown by a provider bypasses immediate retries. The file remains unconfirmed and is automatically retried on the next compaction via the delete guard. +- Failed uploads are tracked via metrics (`cloud_storage_upload_failures_total`) and structured logs. Automatic retry occurs on subsequent compactions; no manual recovery needed. +- Backpressure blocks the RocksDB compaction thread for at most 30 s (`BACKPRESSURE_MAX_WAIT_MS`) even if the backlog remains above the watermark after that window. +- Metadata sync publishes a consistent `CURRENT`/`MANIFEST`/`OPTIONS` (plus WAL tail when `wal-mode: wal`) snapshot so a full-disk-loss node can reopen from cloud. +- Metadata sync is always enabled when cloud storage is enabled. +- Metadata sync is event-triggered by storage events; there is no background interval scheduler. +- Provider-specific properties (e.g., bucket, region, credentials) are configured under `cloud.storage..*` namespace. +- **Operational observability**: Monitor metrics and logs to track sync progress (see "Observability: Metrics & Logs" section above). + +### S3 provider-specific options (`cloud.storage.s3.*`) + +For provider `s3`, configure these properties under `cloud.storage.s3`: + +| Configuration | Default | Description | +|---------------------------------------------------------|-----------|------------------------------------------------------------------------------------------------------| +| `cloud.storage.s3.bucket` | _(none)_ | Target S3 bucket name. Required when S3 provider is enabled. | +| `cloud.storage.s3.region` | _(none)_ | AWS region (for example `us-east-1`). | +| `cloud.storage.s3.endpoint` | _(none)_ | Optional custom endpoint for S3-compatible stores (MinIO/Ceph). | +| `cloud.storage.s3.access-key` | _(none)_ | Access key / AWS Access ID credential. Omit to use AWS default credentials chain. | +| `cloud.storage.s3.secret-key` | _(none)_ | Secret key credential. Omit to use AWS default credentials chain. | +| `cloud.storage.s3.multipart-part-retry-max-attempts` | `3` | Max retries for each multipart upload part before the whole file upload fails. | +| `cloud.storage.s3.multipart-part-retry-base-backoff-ms` | `1000` | Base backoff for part retries (exponential: 1x/2x/4x...). | +| `cloud.storage.s3.multipart-exhausted-direct-dlq` | `false` | If `true`, part-retry exhaustion is marked non-retryable so outer SST retry can go directly to DLQ. | + +### Sample `application.yml` (`cloud.storage`) + +If you want to configure Store directly through `application.yml` (instead of env injection), +use a snippet like this: + +```yaml +cloud: + storage: + enabled: true + provider: s3 + path-prefix: hugegraph + + # Hydration controls + startup-hydration-enabled: true + read-miss-guard-window-ms: 3000 + + # Upload retry & dead-letter queue (whole-file level; common for all providers) + upload-retry-max-attempts: 3 # 0 to disable whole-file retries (DLQ-only) + upload-retry-initial-delay-ms: 1000 + upload-retry-max-delay-ms: 60000 + + # Backpressure: slow RocksDB flush/compaction thread while pending-upload backlog > watermark + # Blocks at most 30 s per event; 0 disables + upload-backpressure-high-watermark: 64 + + # Metadata durability (CURRENT/MANIFEST/OPTIONS[/WAL]) + # Metadata sync is always enabled when cloud storage is enabled + wal-mode: flush # 'flush' or 'wal' (lower RPO, more uploads) + + # S3 provider-specific configuration (cloud.storage.s3.*) + s3: + bucket: hugegraph-store0 + region: us-east-1 + endpoint: http://minio:9000 + access-key: minioadmin + secret-key: minioadmin + + # Multipart upload retry configuration (S3 part-level, inside one uploadFile() call) + multipart-part-retry-max-attempts: 3 + multipart-part-retry-base-backoff-ms: 1000 + multipart-exhausted-direct-dlq: false +``` + +Notes: + +- `upload-retry-max-attempts` defaults to `3` (immediate retries enabled after first failure). Set to `0` only if the provider has sufficient built-in retry logic. Automatic retry also occurs on subsequent compactions via the delete guard, so failed uploads are never silently lost. +- In this docker stack, each Store node uses its own bucket (`hugegraph-store0/1/2`). +- For local MinIO, endpoint is typically `http://minio:9000` inside the docker network. +- Provider-specific properties are configured under `cloud.storage..*` namespace. +- **Monitor these metrics**: `cloud_storage_unconfirmed_files_total` (should trend to 0), `cloud_storage_upload_failures_total` (should be low), `cloud_storage_sync_latency_ms` (should be sub-second for typical files). + + +## 3) Write Path + +Two sub-flows exist: **SST upload** (on flush/compaction) and **SST delete** (on compaction cleanup). + +### 3a) SST Upload Flow + +```text +SST UPLOAD (onTableFileCreated) + +Client + | + | 1) Write request + v +HugeGraph Server + | + | 2) Route to partition leader + v +Store Node (RocksDB) + | + | 3) WAL append + MemTable update + | 4) Flush/compaction creates *.sst + v +CloudStorageEventListener + | + | 5) onTableFileCreated(db, cf, file) + | 6) toRelativeKey(file) -> / + | 7) uploadFile(localPath, remoteKey) + v +CloudStorageProvider + | + | 8) PUT object (full key: //) + v +Object Storage + | + | 9) ACK + v +CloudStorageEventListener + | + | 10) syncTracker.markConfirmed(dbName, fileNumber) <- bitmap updated + | 11) applyBackpressure(dbName) <- optional throttle if backlog > watermark +``` + +### 3b) SST Delete Flow + +```text +SST DELETE (onTableFileDeleted) — runs on the RocksDB compaction thread + +RocksDB compaction completes: SST1 + SST2 -> MERGED_SST + | + | 1) onTableFileDeleted(db, cf, filePath=SST1 or SST2) + v +CloudStorageEventListener + | + | 2) ensureLiveSetUploaded(provider, dbName) + | -> syncTracker.allConfirmed(dbName, liveFiles) <- single lock, short-circuit on first miss + | If NOT all confirmed: + | -> for each unconfirmed live file: + | upload now (idempotent PUT, no existence probe) + | syncTracker.markConfirmed on success + | If any live file still not durable: return false -> skip delete (hold old SST) + | + | 3) [live set confirmed durable] + | syncMetadataSnapshotInline(dbName) <- publish MANIFEST/CURRENT BEFORE delete + | -> RocksDBFactory.captureMetadataSnapshot() + | -> uploadMetadataSnapshot(...) + | * verifies all manifest-referenced SSTs are in cloud + | * uploads OPTIONS / MANIFEST / CURRENT atomically + | If snapshot fails: return false -> skip delete (hold old SST) + | + | 4) [MANIFEST published] provider.deleteFile(remoteKey) <- safe to remove old SST + v +Object Storage: old SST deleted +``` + +### Write-path notes + +- On DB creation (`onDBCreated`), existing local SST files are scanned and uploaded if missing in cloud. + A MemTable flush is triggered via `onDBCreated` (event-driven, not a background async task) so + WAL-recovered/in-memory data materializes into SST files and gets mirrored to cloud. +- Remote object key is always scoped per store node: + **`//`** e.g. `hugegraph/store-127.0.0.1_8501/0/000001.sst`. + `store-scope-prefix` is derived from `raft.address` at startup (see `AppConfig.buildCloudStoreScopePrefix()`). + This guarantees key uniqueness across nodes even when multiple Store nodes share the same bucket. +- After every successful upload, `syncTracker.markConfirmed(dbName, fileNumber)` sets the corresponding + bit in a per-`dbName` in-memory bitmap (`CloudSyncTracker`). This bitmap is the fast path used by the + delete guard (`allConfirmed`) to avoid per-file cloud API calls on the hot compaction thread. +- The delete guard (`ensureLiveSetUploaded`) checks the **entire current live SST set**, not just the file + being deleted. Reason: the old SST being confirmed says nothing about whether its compaction outputs + (replacement files) are durable. Only when every live file is confirmed does the delete proceed. +- MANIFEST/CURRENT is published to cloud **before** the old SST is deleted, so that a recovery attempt + always has a consistent metadata + data state to restore from. + +## 4) Read Path + +Two hydration modes exist: + +1. **Startup pre-hydration (`onDBOpening`)**: download missing remote files before serving, + then seed the sync-tracker bitmap from the remote listing so the first post-restart compaction + can use the fast-path bitmap check instead of issuing per-file cloud API calls. +2. **Read-miss hydration (`onReadMiss`)**: if local read misses, fetch missing SST from cloud, ingest, and retry. + +```text +READ PATH (ANSI) + +Read request + | + | 1) get(key) + v +RocksDB + | + | 2) miss -> onReadMiss(session, table, key) + v +CloudStorageEventListener + | + | 3) listFiles(//) + v +CloudStorageProvider + | + | 4) LIST objects + v +Object Storage + | + | 5) return key list (scoped to this store node's prefix) + v +CloudStorageProvider + | + | 6) for each missing *.sst: + | downloadFile(remoteKey, localPath) + | GET object -> receive object bytes + v +CloudStorageEventListener + | + | 7) ingestSstFile(downloaded) + v +RocksDB + | + | 8) retry get(key) + v +Read request result +``` + +### Startup pre-hydration detail (`onDBOpening`) + +```text +onDBOpening(dbName) + | + | 1) listFiles(///) + | -> remoteFiles (scoped to this store node) + | + | 2) for each remoteFile not present locally: + | downloadFile(remoteKey, localPath) + | + | 3) [after download loop] + | for each *.sst in remoteFiles: + | syncTracker.markConfirmed(dbName, fileNumber) <- bitmap seeded from remote listing + | zero extra cloud API calls +``` + +### Read-path notes + +- All `listFiles` calls use the **`//`** namespace so each store node + only sees and downloads its own SST files, even when multiple nodes share the same bucket. +- A guard window (`read-miss-guard-window-ms`) throttles repeated hydration attempts for the same db/table pair. +- Only missing local SST files are downloaded. +- Non-SST objects are ignored for read-miss ingestion. +- After startup pre-hydration, `syncTracker` is fully seeded from the remote listing. Subsequent + compaction delete guards use the in-memory bitmap (`allConfirmed`) and do **not** issue + `provider.fileExists()` round-trips on the hot compaction thread. + +### Scope: managed delete vs accidental local loss + +- Current behavior intentionally treats a missing local DB directory as a **recovery** case and hydrates from cloud when remote objects exist. +- Intentional delete/reset is recognized only through the managed deletion flow (`RocksDBFactory.destroyGraphDB()` -> `onDBDeleteBegin`/`onDBDeleted`) that writes and then purges the DB tombstone marker. +- If a local DB directory is removed outside the managed flow, no tombstone callback is emitted; hydration may restore data from cloud for the same DB path. +- This is an accepted scope tradeoff for now to prioritize accidental local-loss recovery. +- Future hardening can add explicit generation/epoch markers to distinguish external manual delete from disaster recovery. + +## 5) Failure Handling + +### Upload failures (write-critical) + +#### Two-layer retry model + +Upload failures use a two-layer retry strategy: + +| Layer | Where | What it retries | Config keys | +|----------------------------|----------------------------------------------|--------------------------------------------|---------------------------------------------------------------------------------------------------------| +| **Part-level** (S3 only) | Inside `S3CloudStorageProvider.uploadFile()` | Individual 512 MB multipart chunks | `cloud.storage.s3.multipart-part-retry-max-attempts`, `multipart-part-retry-base-backoff-ms` | +| **File-level** (common) | `CloudUploadRetryQueue` (async) | Whole SST file after `uploadFile()` throws | `cloud.storage.upload-retry-max-attempts`, `upload-retry-initial-delay-ms`, `upload-retry-max-delay-ms` | + +The default `upload-retry-max-attempts=3` enables whole-file retries for transient failures. The S3 provider also handles part-level retries internally. Both layers can be tuned independently. + +#### Failure flow (default `upload-retry-max-attempts=3`) + +- `onTableFileCreated` calls `provider.uploadFile()`. +- S3 retries individual parts internally (via `multipart-part-retry-*`). +- If `uploadFile()` still throws a regular `IOException`, `CloudUploadRetryQueue.submit()` enqueues the task for up to `upload-retry-max-attempts` whole-file retries with exponential backoff. +- If `uploadFile()` throws `CloudStorageNonRetryableException`, the task bypasses retries and skips the upload queue. +- After all retry attempts are exhausted (or if non-retryable), the file remains unconfirmed in the bitmap. The file stays on disk and is **automatically retried on the next compaction** via the delete guard (`ensureLiveSetUploaded`). + +#### Observability: Metrics & Logs + +Instead of a persistent DLQ, the system provides real-time observability through metrics and structured logs: + +**Metrics (exposed to Prometheus/monitoring):** + +- `cloud_storage_unconfirmed_files_total` (gauge, labeled by `db_name`) — Count of SST files not yet confirmed in cloud bitmap. High value → uploads are failing or slow. +- `cloud_storage_upload_failures_total` (counter, labeled by `db_name`, `cf_name`, `error_type`) — Total upload failures (transient + permanent). Tracks upload problem frequency. +- `cloud_storage_retry_queue_size` (gauge) — Files waiting in the upload retry queue. Indicates backlog of pending uploads. +- `cloud_storage_sync_latency_ms` (histogram, labeled by `db_name`) — Time from SST file creation (onTableFileCreated) to bitmap confirmation (markConfirmed). Measures sync speed. +- `cloud_storage_delete_guard_reupload_count` (counter, labeled by `db_name`) — Number of files re-uploaded by the delete guard when live set was not fully durable. Indicates frequent upload failures. + +**Structured Logs:** + +Log entries provide visibility into sync progress and failures: + +- **Startup (bitmap seeding)**: `"Seeded sync-tracker bitmap with N confirmed files from remote listing: dbName=..."` + - Indicates: bitmap is warm after restart, delete guard can use fast path. + +- **Delete guard checking**: `"Checking live set durability: dbName=..., liveFileCount=..., unconfirmedCount=..."` + - Indicates: delete guard evaluated M files, found K unconfirmed. + +- **Delete guard re-upload**: `"Re-uploading M unconfirmed live files (not yet confirmed in cloud): dbName=..., files=[...], reason=live_set_not_durable"` + - Indicates: files failed to sync on first attempt, being re-attempted now. + +- **Delete guard re-upload success**: `"Successfully confirmed L previously unconfirmed files; delete proceeding: dbName=..., oldSstFile=..."` + - Indicates: re-upload succeeded, old SST can now be safely deleted. + +- **Delete guard skipped**: `"Delete skipped (live set not fully durable in cloud): dbName=..., oldSstFile=..., unconfirmedFiles=[...], nextRetryAt=..."` + - Indicates: at least one live file is still not durable. Delete is held. Retry will occur on next compaction. + +- **Upload failure (first attempt)**: `"Upload failed (will retry): dbName=..., cfName=..., filePath=..., attempt=1/3, error=..."` + - Indicates: transient failure, will be retried. + +- **Upload failure (exhausted retries)**: `"Upload failed permanently after 3 retries: dbName=..., cfName=..., filePath=..., error=..., nextRetryOnCompaction=..."` + - Indicates: all retries exhausted. File stays on disk; will retry automatically on next compaction. + +**Example: Finding unsynced files** + +Query metrics in your monitoring system: + +```promql +# Find databases with unconfirmed files +cloud_storage_unconfirmed_files_total > 0 + +# Alert if unconfirmed count grows over time +rate(cloud_storage_unconfirmed_files_total[5m]) > 0 +``` + +Or search logs: + +```bash +# Find all delete-guard re-uploads in the past hour +kubectl logs -l app=hugegraph-store --since=1h | grep "Re-uploading.*unconfirmed" + +# Find all permanently-failed uploads +kubectl logs -l app=hugegraph-store | grep "Upload failed permanently" + +# Track a specific file +kubectl logs -l app=hugegraph-store | grep "filePath=/path/to/000123.sst" +``` + +**Automatic retry without DLQ:** + +- Files that fail to upload remain unconfirmed in the bitmap. +- On the next compaction that touches the same DB, the delete guard calls `ensureLiveSetUploaded`, which: + 1. Checks bitmap for all current live files + 2. For any unconfirmed file, attempts upload again (idempotent PUT) + 3. On success, updates bitmap; on failure, logs and holds delete +- This cycle repeats automatically; no manual DLQ replay needed. +- Upload success rate and retry frequency are tracked via metrics and logs. + +**Tuning:** + +To control retry behavior: + +```yaml +cloud.storage.upload-retry-max-attempts: 3 # Immediate retries on first failure +cloud.storage.upload-retry-initial-delay-ms: 1000 # Start backoff at 1s +cloud.storage.upload-retry-max-delay-ms: 60000 # Cap backoff at 60s +``` + +If the provider handles all retry logic internally, disable whole-file retries: + +```yaml +cloud.storage.upload-retry-max-attempts: 0 # No immediate retries; rely on delete guard auto-retry +``` + +### Delete failures (non-fatal) + +- `onTableFileDeleted` delete failure is logged and processing continues. +- Impact is stale/orphaned cloud objects, not immediate read/write unavailability. + +### Hydration failures + +- Pre-hydration/list/download failures throw `IllegalStateException` and stop the flow for that DB open attempt. +- Read-miss hydration failures are logged and return `false`; caller falls back to original miss behavior. + +### Provider lifecycle / config failures + +- Unknown provider name or missing plugin JAR fails initialization in `CloudStorageProviderFactory`. +- Provider switching/re-init is handled with close-and-reinitialize semantics. +- If no active provider is found at runtime (provider is `null`), all event callbacks (`onTableFileCreated`, `onTableFileDeleted`, `onReadMiss`) return immediately without error, so RocksDB continues normally without cloud mirroring. + +### Non-retryable upload failures (`CloudStorageNonRetryableException`) + +- Providers can signal a permanently failed upload by throwing `CloudStorageNonRetryableException` from `uploadFile()`. +- This exception bypasses the whole-file retry loop and marks the file as unconfirmed (no bitmap entry). +- The file remains on disk and is **automatically retried on the next compaction** via the delete guard's re-upload logic. +- The S3 provider uses this when `multipart-exhausted-direct-dlq: true` and all multipart part retries are exhausted, preventing pointless full-file re-attempts for a part that consistently fails. +- Metrics (`cloud_storage_upload_failures_total`) and logs track the failure for operational visibility. + +### Backpressure timeout + +- When `upload-backpressure-high-watermark > 0` and the pending-upload backlog (retry-queue in-flight + DLQ size) exceeds the watermark, `onTableFileCreated` parks the RocksDB flush/compaction thread in 50 ms increments. +- After `30 000 ms` (`BACKPRESSURE_MAX_WAIT_MS`) the backpressure wait exits unconditionally and the new SST upload is attempted regardless, to prevent a permanent RocksDB stall. +- A warning log is emitted when backpressure starts, and an info log when it is released. + +### Delete guard failure (live set not fully durable) + +- Before deleting a superseded SST from cloud, `onTableFileDeleted` verifies the entire current live SST set is confirmed present in cloud (`ensureLiveSetUploaded`). +- If any live SST cannot be confirmed durable (e.g. a prior upload failed), the delete is skipped with a warning log. +- The unconfirmed file remains on disk. On the next compaction, the delete guard will re-attempt upload for all unconfirmed files. +- **Observability**: Metrics `cloud_storage_unconfirmed_files_total` and `cloud_storage_delete_guard_reupload_count` track the frequency. Logs show which files are unconfirmed and why the delete was held. +- **Impact**: Orphaned (but safe) old object may remain in cloud temporarily. Data integrity is preserved because the replacement files are not yet confirmed durable. + +### Metadata snapshot capture failure + +- `syncMetadataSnapshotInline` calls `RocksDBFactory.captureMetadataSnapshot()`. If this returns `null` (e.g. the DB is not open or the checkpoint could not be acquired), the metadata sync is skipped and returns `false`. +- When called from `onTableFileDeleted`, a `null` snapshot causes the delete to be held with a warning log, preserving the old SST in cloud until a valid snapshot can be published. + +### Metadata publish failure (SST durability check fails) + +- `uploadMetadataSnapshot` first ensures all manifest-referenced SSTs are present in cloud. If any SST cannot be made durable, the method aborts before uploading `MANIFEST` or `CURRENT`, returns `false`, and logs a warning. +- The previous durable generation (prior `CURRENT`/`MANIFEST`) is left intact in cloud, so a recovery attempt still has a valid consistent state to restore from. +- An `IOException` during upload of `OPTIONS`/`MANIFEST`/`CURRENT` also aborts the publish with a warning. + +### S3 multipart upload failures + +- **Initiation failure**: if `CreateMultipartUpload` fails, `uploadFile()` throws `IOException` immediately (no part retries). The whole-file retry queue then handles it if `upload-retry-max-attempts > 0`. +- **Part retry exhaustion**: individual part upload failures are retried up to `multipart-part-retry-max-attempts` times with exponential backoff (`multipart-part-retry-base-backoff-ms`). When all part retries are exhausted, the multipart upload is aborted. +- **Abort failure**: if the `AbortMultipartUpload` call itself fails, the failure is logged as a warning. The incomplete multipart upload may remain in the bucket (incurring storage cost) until an S3 lifecycle rule or manual cleanup removes it. +- **Direct-DLQ on part exhaustion**: when `multipart-exhausted-direct-dlq: true`, part-retry exhaustion throws `CloudStorageNonRetryableException`, bypassing whole-file retries. + + +### Local file compacted away during retry + +- When a local SST file is compacted into a new merged SST (e.g., SST1 → MERGED_SST), the old SST1 may still be unconfirmed in the bitmap if its upload failed. +- On the next compaction involving MERGED_SST, the delete guard checks if MERGED_SST is confirmed. If not, it re-attempts upload. +- If SST1 was compacted away, its local file no longer exists, so no retry is attempted for SST1 itself. +- However, the data from SST1 is now present in MERGED_SST in cloud (or will be on next attempt). Once MERGED_SST is confirmed, SST1 can be safely deleted from cloud. +- Metrics track `cloud_storage_delete_guard_reupload_count` to monitor how often the delete guard needs to re-upload files. + +## 6) Recovery Scenarios + +Failure-mode and RPO-oriented recovery summary: + +| Failure scenario | Data loss? | RPO | Recovery mechanism | Mitigation | +|-----------------------------------------------|--------------------------------------------|--------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------| +| Single Store crash | No | 0s | 2/3 Raft quorum survives. Leader re-election continues service. In-flight (not SST-flushed) data is recovered from Raft logs; flushed data remains available via local/cloud SSTs. | Keep 3+ replicas across zones, alert on replica loss, and use durable PV-backed store nodes. | +| 2 of 3 Stores crash | No (service stalls until quorum restored) | 0s | Surviving replica Raft log bootstraps recovering nodes after restart. In-flight data is recovered from Raft log replay. | Enforce anti-affinity and failure-domain isolation to prevent correlated failures. | +| Catastrophic loss: all Store disks destroyed | Yes | Seconds to minutes (depends on SST sync mode/interval) | Raft logs and local SSTs are lost. Nodes recover from last completed cloud SST sync; data after that sync is unrecoverable. | Use durable disks, scheduled volume snapshots/backups, and synchronous SST upload mode for tighter RPO. | + +Notes: + +- In-flight data (not yet flushed to SST) is recovered from Raft logs when quorum-replicated. +- Cloud storage recovery primarily protects flushed SST state and disaster cases involving local disk loss. +- Synchronous SST upload reduces catastrophic-loss RPO compared with periodic upload mode. +- Recovery correctness is protected by the metadata-before-delete invariant: before deleting superseded SSTs, Store confirms manifest-referenced SST durability and publishes updated `MANIFEST`/`CURRENT`, preventing stale or missing-manifest references after compaction retries. +- Policy note: for how managed delete vs accidental local-loss is distinguished during hydration, see `## 4) Read Path` -> `Scope: managed delete vs accidental local loss`. + +## 7) Operational Notes + +### Monitoring & Alerting + +Cloud storage health is exposed through metrics and logs. Set up monitoring for: + +**Alerts (suggest these thresholds):** + +```yaml +# Alert if any database has unconfirmed files for > 5 minutes (sync is stuck) +- alert: CloudStorageUnsyncedFilesHigh + expr: cloud_storage_unconfirmed_files_total > 0 and increase(cloud_storage_unconfirmed_files_total[5m]) > 0 + annotations: + summary: "Store node {{ $labels.instance }} has {{ $value }} unconfirmed files in {{ $labels.db_name }}" + action: "Check upload logs for permission errors or network issues" + +# Alert if upload failures are increasing (transient or persistent failures) +- alert: CloudStorageUploadFailuresIncreasing + expr: rate(cloud_storage_upload_failures_total[5m]) > 0 + annotations: + summary: "Store node {{ $labels.instance }} is experiencing {{ $value }} upload failures/sec" + action: "Review logs for details; check S3/cloud provider status" + +# Alert if retry queue is growing (backlog of pending uploads) +- alert: CloudStorageRetryQueueBacklog + expr: cloud_storage_retry_queue_size > 10 + annotations: + summary: "Store node {{ $labels.instance }} has {{ $value }} files in retry queue" + action: "Monitor until queue drains; backpressure may be slowing compaction" + +# Alert if sync latency is high (uploads taking too long) +- alert: CloudStorageSyncLatencyHigh + expr: histogram_quantile(0.95, cloud_storage_sync_latency_ms) > 30000 + annotations: + summary: "Store node {{ $labels.instance }} 95th percentile sync latency is {{ $value }}ms" + action: "Check network link to cloud provider; consider tuning upload concurrency" +``` + +**Logs to monitor:** + +```bash +# Watch for re-uploads (delete guard re-attempting failed uploads) +kubectl logs -f -l app=hugegraph-store | grep "Re-uploading.*unconfirmed" + +# Watch for permanently-failed uploads (still retried, but worth investigation) +kubectl logs -f -l app=hugegraph-store | grep "Upload failed permanently" + +# Watch for delete-skipped events (delete guard holding old SSTs) +kubectl logs -f -l app=hugegraph-store | grep "Delete skipped.*not fully durable" + +# Trace a specific unconfirmed file +kubectl logs -f -l app=hugegraph-store | grep "filePath=/path/to/000123.sst" +``` + +### Troubleshooting + +**Symptom: `cloud_storage_unconfirmed_files_total` stays > 0** + +- **Cause**: Uploads are failing and not recovering. +- **Action**: + 1. Check logs for "Upload failed permanently" messages — what's the error? + 2. Verify cloud provider credentials and connectivity: `curl https://s3-endpoint/bucket-name` + 3. Check if bucket exists and Store node has put/get/delete permissions. + 4. Verify `cloud.storage.path-prefix` matches actual S3 prefix where Store expects to write. + 5. If a specific file is stuck, manually trigger sync via admin API or wait for next compaction. + +**Symptom: Delete operations are slow or stalling** + +- **Cause**: Delete guard is re-uploading many files; backpressure may be active. +- **Action**: + 1. Check logs for "Re-uploading M unconfirmed files" — how many? + 2. If backpressure is active, monitor `cloud_storage_retry_queue_size`. Once it drains, delete resumes. + 3. Increase `upload-backpressure-high-watermark` if you want to allow more local-only data during cloud outages. + +**Symptom: High `cloud_storage_sync_latency_ms`** + +- **Cause**: Uploads are slow. Could be network, object size, or cloud provider latency. +- **Action**: + 1. Check average file size: `ls -lh /data/raft/*/db-*/` (larger files take longer) + 2. Measure network throughput: `iperf3` or cloud provider bandwidth test. + 3. Check if S3 multipart upload is enabled and tuned (reduces latency for large files). + +### General Best Practices + +- Prefer stable `path-prefix` and bucket naming; changing them affects object lookup continuity. +- Keep plugin JAR versions aligned with Store version to avoid SPI/API mismatch. +- Track logs around upload, hydration, and provider init to detect divergence early. +- For DR drills, test both node-level restart and full-cluster restart with cloud hydration enabled. +- Set up alerts on the key metrics (`unconfirmed_files_total`, `upload_failures_total`, `sync_latency_ms`) to catch issues early. + +## Plugin Development: Adding Custom Cloud Storage Providers + +This section guides developers on building and deploying new cloud storage provider plugins (e.g., Azure Blob Storage, ADLS, GCP). + +### Plugin Architecture Overview + +HugeGraph uses a **Service Provider Interface (SPI)** pattern to discover and load cloud storage providers at runtime: + +1. **CloudStorageProvider interface**: Located in `hugegraph-store/hg-store-common`, defines the contract all providers must implement. +2. **ServiceLoader discovery**: Store node uses `java.util.ServiceLoader` to find all `CloudStorageProvider` implementations on the classpath. +3. **SPI selection**: At Store startup, `CloudStorageProviderFactory` loads the provider specified in `cloud.storage.provider` config. +4. **Plugin packaging/runtime classpath**: Provider implementations are packaged as separate JAR modules and must be available on Store runtime classpath. + +### CloudStorageProvider Interface + +All custom providers must implement: + +```java +public interface CloudStorageProvider { + + /** + * Unique name for this provider (e.g., "s3", "azure", "adls"). + * Must match the `cloud.storage.provider` config value to be activated. + */ + String providerName(); + + /** + * Initialize the provider with configuration. + * Called once at Store startup. Throw exception if init fails. + */ + void init(CloudStorageConfig config) throws IOException; + + /** + * Upload a local file to cloud storage with the given remote key. + */ + void uploadFile(String localPath, String remoteKey) throws IOException; + + /** + * Download a remote file from cloud storage to the local path. + */ + void downloadFile(String remoteKey, String localPath) throws IOException; + + /** + * Delete a remote file from cloud storage. + */ + void deleteFile(String remoteKey) throws IOException; + + /** + * Check if a remote file exists. + */ + boolean fileExists(String remoteKey) throws IOException; + + /** + * List all remote files under the given directory prefix. + * Returns list of keys (with prefix stripped). + */ + List listFiles(String remoteDirPrefix) throws IOException; + + /** + * Close and release all provider resources. + */ + void close() throws IOException; +} +``` + +**Location:** `hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageProvider.java` + +### Module Structure for a New Provider + +``` +hugegraph-store/ +├── hg-store-cloud-newprovider/ # New provider module +│ ├── pom.xml # Maven module definition +│ ├── src/main/java/org/apache/hugegraph/store/cloud/newprovider/ +│ │ └── NewProviderCloudStorageProvider.java # Implementation +│ ├── src/main/resources/META-INF/services/ +│ │ └── org.apache.hugegraph.store.cloud.CloudStorageProvider +│ └── src/test/java/... # Unit tests +``` + +Recommendation: + +- For providers maintained in this repository, prefer adding them as submodules under `hugegraph-store/` (same model as `hg-store-cloud-s3`). +- External providers are also supported if their jars are placed on runtime classpath. + +### Step 1: Create Module POM + +File: `hugegraph-store/hg-store-cloud-newprovider/pom.xml` + +```xml + + + 4.0.0 + + + org.apache.hugegraph + hugegraph-store + ${revision} + ../pom.xml + + + hg-store-cloud-newprovider + HugeGraph Store Cloud NewProvider + NewProvider Storage plugin for HugeGraph Store cloud storage integration + + + + + org.apache.hugegraph + hg-store-common + ${revision} + + + + + NewProvider + NewProvider + 1.0 + + + + +``` + +### Step 2: Implement the Provider + +File: `hugegraph-store/hg-store-cloud-newprovider/src/main/java/org/apache/hugegraph/store/cloud/newprovider/NewProviderCloudStorageProvider.java` + +```java +package org.apache.hugegraph.store.cloud.newprovider; + +import org.apache.hugegraph.store.cloud.CloudStorageConfig; +import org.apache.hugegraph.store.cloud.CloudStorageProvider; +import lombok.extern.slf4j.Slf4j; + +import java.io.IOException; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.List; + +@SuppressWarnings("RedundantThrows") +@Slf4j +public class NewProviderCloudStorageProvider implements CloudStorageProvider { + + public static final String PROVIDER_NAME = "newprovider"; + + private Object providerClient; + private String pathPrefix; + + @Override + public String providerName() { + return PROVIDER_NAME; + } + + @Override + public void init(CloudStorageConfig config) throws IOException { + //Steps to initialize the NewProvider client using config parameters + } + + @Override + public void uploadFile(String localPath, String remoteKey) throws IOException { + //Steps to upload file to NewProvider cloud storage + } + + @Override + public void downloadFile(String remoteKey, String localPath) throws IOException { + //Steps to download file from NewProvider cloud storage + } + + @Override + public void deleteFile(String remoteKey) throws IOException { + //Steps to delete file from NewProvider cloud storage + } + + @Override + public boolean fileExists(String remoteKey) throws IOException { + //Steps to check if file exists in NewProvider cloud storage + return false; + } + + @Override + public List listFiles(String remoteDirPrefix) throws IOException { + //Steps to list files in NewProvider cloud storage + return new ArrayList<>(); + } + + @Override + public void close() throws IOException { + //Steps to close and cleanup NewProvider client resources + } +} +``` + +### Step 3: Register via SPI + +File: `hugegraph-store/hg-store-cloud-newprovider/src/main/resources/META-INF/services/org.apache.hugegraph.store.cloud.CloudStorageProvider` + +``` +org.apache.hugegraph.store.cloud.newprovider.NewProviderCloudStorageProvider +``` + +This file tells Java's `ServiceLoader` to discover this provider at runtime. + +### Step 4: Build and Test + +```bash +# Build the provider module +cd hugegraph-store +mvn clean package -pl hg-store-cloud-newprovider -am -DskipTests + +# Find the generated JAR +find . -name "hg-store-cloud-newprovider-*.jar" +# Output: hugegraph-store/hg-store-cloud-newprovider/target/hg-store-cloud-newprovider-1.0.jar +``` + +### Step 5: Package for Runtime Classpath + +Use one of these runtime models: + +**Option A (recommended for in-repo providers): add as Store submodule dependency** + +1. Add module `hg-store-cloud-newprovider` under `hugegraph-store/`. +2. Add dependency from `hg-store-node` to `hg-store-cloud-newprovider`. +3. Rebuild distribution so provider + transitive dependencies are packaged with Store runtime. + +**Option B (external provider): provide jars on runtime classpath** + +- Supply `hg-store-cloud-newprovider-*.jar` **and** all required dependency jars, or +- supply a single shaded/uber provider jar that already contains transitive dependencies. + +Notes: + +- Simply copying a provider jar without its runtime dependencies can cause `ClassNotFoundException` during SPI loading. +- The exact classpath injection method depends on your launch model (custom startup script, container image, or JVM args). + +### Step 6: Configure and Activate + +In Store `application.yml`: + +```yaml +cloud: + storage: + enabled: true + provider: newprovider # Must match NewProviderCloudStorageProvider.providerName() + path-prefix: hugegraph + + # Provider-specific configuration (cloud.storage.newprovider.*) + newprovider: + bucket: my-container + region: myaccount # e.g., Azure account name + endpoint: https://myaccount.blob.core.windows.net + access-key: ${NEWPROVIDER_KEY} + secret-key: ${NEWPROVIDER_SECRET} +``` + +Or via Docker env (for S3 example): + +```bash +HG_CLOUD_STORAGE_ENABLED=true \ +HG_CLOUD_STORAGE_PROVIDER=s3 \ +HG_CLOUD_STORAGE_PATH_PREFIX=hugegraph \ +HG_CLOUD_STORAGE_S3_BUCKET=hugegraph-store0 \ +HG_CLOUD_STORAGE_S3_REGION=us-east-1 \ +HG_CLOUD_STORAGE_S3_ENDPOINT=http://minio:9000 \ +HG_CLOUD_STORAGE_S3_ACCESS_KEY=minioadmin \ +HG_CLOUD_STORAGE_S3_SECRET_KEY=minioadmin \ +docker run hugegraph-store:latest +``` + +Or via Docker env (for custom newprovider): + +```bash +HG_CLOUD_STORAGE_ENABLED=true \ +HG_CLOUD_STORAGE_PROVIDER=newprovider \ +HG_CLOUD_STORAGE_PATH_PREFIX=hugegraph \ +HG_CLOUD_STORAGE_NEWPROVIDER_BUCKET=my-container \ +HG_CLOUD_STORAGE_NEWPROVIDER_REGION=myaccount \ +HG_CLOUD_STORAGE_NEWPROVIDER_ENDPOINT=https://myaccount.blob.core.windows.net \ +HG_CLOUD_STORAGE_NEWPROVIDER_ACCESS_KEY=${NEWPROVIDER_KEY} \ +HG_CLOUD_STORAGE_NEWPROVIDER_SECRET_KEY=${NEWPROVIDER_SECRET} \ +docker run hugegraph-store:latest +``` + +Notes on property naming conventions: + +- **YAML properties**: Use kebab-case with provider-specific namespacing (e.g., `cloud.storage.s3.bucket`, `cloud.storage.newprovider.access-key`) +- **Environment variables**: Use uppercase with underscores and `HG_CLOUD_STORAGE_*` prefix. Provider-specific options use `HG_CLOUD_STORAGE__*` pattern +- Kebab-case YAML properties are converted to uppercase with underscores (e.g., `multipart-part-retry-max-attempts` → `MULTIPART_PART_RETRY_MAX_ATTEMPTS`) + +### Testing Your Provider + +Add unit tests in `hg-store-cloud-newprovider/src/test/`: + +```java +@Test +public void testInit() { + // Test provider initialization with mock config +} + +@Test +public void testUploadDownload() { + // Test upload and download of a sample file +} + +@Test +public void testDeleteAndList() { + // Test delete and list operations +} + +@Test +public void testFileExists() { + // Test file existence check +} + +@Test +public void testClose() { + // Test provider close and resource cleanup +} +``` + +### Troubleshooting Provider Discovery + +If your provider isn't loaded: + +1. **Check SPI registration file exists:** + ```bash + jar tf hg-store-cloud-newprovider-1.0.jar | grep "META-INF/services" + ``` + Should show: `META-INF/services/org.apache.hugegraph.store.cloud.CloudStorageProvider` + +2. **Check provider name matches config:** + ```bash + jar xf hg-store-cloud-newprovider-1.0.jar META-INF/services/org.apache.hugegraph.store.cloud.CloudStorageProvider + cat META-INF/services/org.apache.hugegraph.store.cloud.CloudStorageProvider + ``` + +3. **Enable debug logging:** + ```yaml + logging: + level: + org.apache.hugegraph.store.cloud: DEBUG + org.apache.hugegraph.store.node.cloud: DEBUG + ``` + +4. **Verify provider jars are on classpath:** + ```bash + # Check Store startup logs for provider discovery / class-loading errors + docker logs hugegraph-store | grep -i "ServiceLoader\|provider" + ``` + +### Key Implementation Tips + +1. **Thread safety**: Ensure provider is thread-safe for concurrent upload/download/delete operations. +2. **Connection pooling**: Reuse client connections; initialize once in `init()`, close in `close()`. +3. **Path normalization**: Always use `pathPrefix` correctly (see S3 example for reference). +4. **Error handling**: Throw `IOException` for operational issues. Implement your own internal retry logic (see `S3CloudStorageProvider` for reference). The common `CloudUploadRetryQueue` provides a DLQ safety net but does not retry by default (`upload-retry-max-attempts=0`). +5. **Logging**: Use SLF4J (via `@Slf4j`) for consistent log levels with Store. +6. **Configuration validation**: Validate all required fields in `init()` and fail fast. + +### Contributing Your Provider + +To upstream your new provider (e.g., `hg-store-cloud-newprovider`): + +1. Create PR to `hugegraph-store/` with new provider module +2. Add tests in `hg-store-test/` that validate provider behavior +3. Update `docker/cloud-storage/` examples with new provider setup steps +4. Update `install-dist/` license/notice if adding third-party dependencies diff --git a/hugegraph-store/hg-store-cloud-s3/pom.xml b/hugegraph-store/hg-store-cloud-s3/pom.xml new file mode 100644 index 0000000000..a151cc771b --- /dev/null +++ b/hugegraph-store/hg-store-cloud-s3/pom.xml @@ -0,0 +1,116 @@ + + + + + 4.0.0 + + + org.apache.hugegraph + hugegraph-store + ${revision} + ../pom.xml + + + hg-store-cloud-s3 + + + S3 cloud storage provider plugin for HugeGraph Store. + Add this JAR to the classpath and set cloud.storage.provider=s3 to enable S3 offload. + + + + 2.33.8 + + + + + + org.apache.hugegraph + hg-store-common + + + + + org.rocksdb + rocksdbjni + 7.7.3 + runtime + + + + + software.amazon.awssdk + s3 + ${aws.sdk.version} + + + + + software.amazon.awssdk + url-connection-client + ${aws.sdk.version} + + + + org.projectlombok + lombok + provided + + + + + junit + junit + 4.13.2 + test + + + org.slf4j + slf4j-simple + 1.7.36 + test + + + + + + + + org.apache.maven.plugins + maven-surefire-plugin + + + + **/*Benchmark.java + + + -Xmx512m + + 21600 + + + + + diff --git a/hugegraph-store/hg-store-cloud-s3/src/main/java/org/apache/hugegraph/store/cloud/s3/S3CloudStorageConfig.java b/hugegraph-store/hg-store-cloud-s3/src/main/java/org/apache/hugegraph/store/cloud/s3/S3CloudStorageConfig.java new file mode 100644 index 0000000000..cce310551c --- /dev/null +++ b/hugegraph-store/hg-store-cloud-s3/src/main/java/org/apache/hugegraph/store/cloud/s3/S3CloudStorageConfig.java @@ -0,0 +1,110 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.cloud.s3; + +import lombok.Data; + +/** + * Amazon S3 / S3-compatible provider configuration, bound from + * {@code cloud.storage.s3.*} in application.yml. + * + *

This is a plain Java POJO (no Spring annotations) so it can be used + * both as a Spring {@code @ConfigurationProperties} target (via standard setters) + * and directly in tests without a Spring context. + * + * 

+ * cloud:
+ *   storage:
+ *     s3:
+ *       bucket: hugegraph-store
+ *       region: us-east-1
+ *       endpoint:                             # optional custom endpoint (MinIO, Ceph…)
+ *       access-key: AKIAIOSFODNN7EXAMPLE      # omit to use AWS default credentials chain
+ *       secret-key: wJalrXUtnFEMI/…           # omit to use AWS default credentials chain
+ *       multipart-part-retry-max-attempts: 3
+ *       multipart-part-retry-base-backoff-ms: 1000
+ *       multipart-exhausted-direct-dlq: false
+ *
+ */ +@Data +public class S3CloudStorageConfig { + + public static final int DEFAULT_MULTIPART_PART_RETRY_MAX_ATTEMPTS = 3; + public static final long DEFAULT_MULTIPART_PART_RETRY_BASE_BACKOFF_MS = 1000L; + + public static final String KEY_BUCKET = "bucket"; + public static final String KEY_REGION = "region"; + public static final String KEY_ENDPOINT = "endpoint"; + public static final String KEY_ACCESS_KEY = "access-key"; + public static final String KEY_SECRET_KEY = "secret-key"; + public static final String KEY_MULTIPART_RETRY_MAX_ATTEMPTS = + "multipart-part-retry-max-attempts"; + public static final String KEY_MULTIPART_RETRY_BASE_BACKOFF_MS = + "multipart-part-retry-base-backoff-ms"; + public static final String KEY_MULTIPART_EXHAUSTED_DIRECT_DLQ = + "multipart-exhausted-direct-dlq"; + + /** + * S3 bucket name. + */ + private String bucket; + + /** + * AWS region (e.g. "us-east-1"). + */ + private String region; + + /** + * Optional custom endpoint URL for S3-compatible stores (e.g. MinIO, Ceph). + * Leave empty to use the default AWS endpoint. + */ + private String endpoint; + + /** + * AWS access key ID. Omit to use the default AWS credentials chain + * (env vars, instance profile, ~/.aws/credentials, etc.). + */ + private String accessKey; + + /** + * AWS secret access key. Omit to use the default AWS credentials chain. + */ + private String secretKey; + + /** + * Maximum retry attempts for a single multipart chunk upload (part-level retry). + * Default: {@value DEFAULT_MULTIPART_PART_RETRY_MAX_ATTEMPTS}. + */ + private int multipartPartRetryMaxAttempts = + DEFAULT_MULTIPART_PART_RETRY_MAX_ATTEMPTS; + + /** + * Base backoff in milliseconds for multipart chunk retry (1x/2x/4x…). + * Default: {@value DEFAULT_MULTIPART_PART_RETRY_BASE_BACKOFF_MS} ms. + */ + private long multipartPartRetryBaseBackoffMs = + DEFAULT_MULTIPART_PART_RETRY_BASE_BACKOFF_MS; + + /** + * If true, multipart chunk retry exhaustion is treated as non-retryable so outer + * SST retry can move directly to DLQ without further whole-file attempts. + */ + private boolean multipartExhaustedDirectDlq = false; + +} + diff --git a/hugegraph-store/hg-store-cloud-s3/src/main/java/org/apache/hugegraph/store/cloud/s3/S3CloudStorageProvider.java b/hugegraph-store/hg-store-cloud-s3/src/main/java/org/apache/hugegraph/store/cloud/s3/S3CloudStorageProvider.java new file mode 100644 index 0000000000..f1c0ebe9a7 --- /dev/null +++ b/hugegraph-store/hg-store-cloud-s3/src/main/java/org/apache/hugegraph/store/cloud/s3/S3CloudStorageProvider.java @@ -0,0 +1,864 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.cloud.s3; + +import java.io.FileInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.net.URI; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.List; +import java.util.Locale; +import java.util.Map; + +import org.apache.hugegraph.store.cloud.CloudStorageConfig; +import org.apache.hugegraph.store.cloud.CloudStorageNonRetryableException; +import org.apache.hugegraph.store.cloud.CloudStorageProvider; + +import lombok.extern.slf4j.Slf4j; + +import org.jetbrains.annotations.NotNull; + +import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; +import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider; +import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; +import software.amazon.awssdk.awscore.exception.AwsServiceException; +import software.amazon.awssdk.core.exception.SdkException; +import software.amazon.awssdk.core.sync.RequestBody; +import software.amazon.awssdk.http.ContentStreamProvider; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.s3.S3Client; +import software.amazon.awssdk.services.s3.S3ClientBuilder; +import software.amazon.awssdk.services.s3.model.AbortMultipartUploadRequest; +import software.amazon.awssdk.services.s3.model.CompletedMultipartUpload; +import software.amazon.awssdk.services.s3.model.CompletedPart; +import software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest; +import software.amazon.awssdk.services.s3.model.CreateMultipartUploadRequest; +import software.amazon.awssdk.services.s3.model.CreateMultipartUploadResponse; +import software.amazon.awssdk.services.s3.model.DeleteObjectRequest; +import software.amazon.awssdk.services.s3.model.DeleteObjectsRequest; +import software.amazon.awssdk.services.s3.model.DeleteObjectsResponse; +import software.amazon.awssdk.services.s3.model.GetObjectRequest; +import software.amazon.awssdk.services.s3.model.HeadObjectRequest; +import software.amazon.awssdk.services.s3.model.ListObjectsV2Request; +import software.amazon.awssdk.services.s3.model.ListObjectsV2Response; +import software.amazon.awssdk.services.s3.model.NoSuchKeyException; +import software.amazon.awssdk.services.s3.model.ObjectIdentifier; +import software.amazon.awssdk.services.s3.model.PutObjectRequest; +import software.amazon.awssdk.services.s3.model.S3Object; +import software.amazon.awssdk.services.s3.model.UploadPartRequest; +import software.amazon.awssdk.services.s3.model.UploadPartResponse; + +/** + * Amazon S3 (and S3-compatible) implementation of {@link CloudStorageProvider}. + * + *

Activation

+ * Place {@code hg-store-cloud-s3-*.jar} on the classpath and configure: + * 
+ * cloud:
+ *   storage:
+ *     enabled: true
+ *     provider: s3
+ *     s3:
+ *       bucket: my-bucket
+ *       region: us-east-1
+ *
+ * + *

Credentials

+ *
    + *
  • If {@code cloud.storage.s3.access-key} / {@code secret-key} are set, + * they are used directly.
    • + *
  • Otherwise the standard AWS Default Credentials chain is followed + * (env vars, instance profile, ~/.aws/credentials, etc.).
    • + *
+ * + *

S3-compatible endpoints (MinIO, Ceph, etc.)

+ * Set {@code cloud.storage.s3.endpoint} to the custom HTTP/HTTPS endpoint URL. + * + *

Large-file (multipart) uploads

+ * S3 limits a single PUT to 5 GB. Files larger than + * {@link #MULTIPART_THRESHOLD_BYTES} ({@value #MULTIPART_THRESHOLD_BYTES} MB) + * are automatically split into {@link #PART_SIZE_BYTES} ({@value #PART_SIZE_MB} MB) + * chunks and uploaded using the S3 Multipart Upload API. + * Each chunk is logged individually so progress is visible for very large files. + * + *

Multipart part retry tuning

+ * Tune part-level retry behavior via typed S3 keys: + * 
+ * cloud:
+ *   storage:
+ *     s3:
+ *       multipart-part-retry-max-attempts: 5
+ *       multipart-part-retry-base-backoff-ms: 1500
+ *       multipart-exhausted-direct-dlq: false
+ *
+ * These options apply only to multipart chunks, not to whole-file retry/DLQ policy + * in {@code CloudUploadRetryQueue}. + * + *

Timing metrics

+ * Every upload logs the file size, elapsed time, and throughput at INFO level: + * 
+ *   S3 upload complete: db/000042.sst | size=64.0 MB | elapsed=830 ms | throughput=77.11 MB/s
+ *
+ */ +@Slf4j +public class S3CloudStorageProvider implements CloudStorageProvider { + + /** Provider name as referenced in {@link CloudStorageConfig#getProvider()}. */ + public static final String PROVIDER_NAME = "s3"; + + /** + * Files larger than this are uploaded via multipart. + * S3's hard per-PUT limit is 5 GB; we start multipart well below that. + */ + static final long MULTIPART_THRESHOLD_BYTES = 512L * 1024 * 1024; // 512 MB + + /** + * Size of each multipart chunk. + * S3 minimum part size is 5 MB (except for the last part). + */ + static final long PART_SIZE_BYTES = 512L * 1024 * 1024; // 512 MB + static final int PART_SIZE_MB = 512; + + private S3Client s3Client; + private String bucket; + private String pathPrefix; + private int partUploadMaxRetries = S3CloudStorageConfig.DEFAULT_MULTIPART_PART_RETRY_MAX_ATTEMPTS; + private long partUploadRetryBaseBackoffMs = + S3CloudStorageConfig.DEFAULT_MULTIPART_PART_RETRY_BASE_BACKOFF_MS; + private boolean multipartExhaustedDirectDlq = false; + + // ----------------------------------------------------------------------- + // CloudStorageProvider + // ----------------------------------------------------------------------- + + @Override + public String providerName() { + return PROVIDER_NAME; + } + + @Override + public void init(CloudStorageConfig config) { + Map props = config.getProviderProperties(); + if (props == null || props.isEmpty()) { + throw new IllegalArgumentException("S3 provider selected but providerProperties are empty"); + } + + this.bucket = props.get(S3CloudStorageConfig.KEY_BUCKET); + if (this.bucket == null || this.bucket.isBlank()) { + throw new IllegalArgumentException("S3 bucket is required: cloud.storage.s3.bucket"); + } + this.pathPrefix = config.getPathPrefix(); + this.initRetryConfig(props); + + S3ClientBuilder builder = S3Client.builder(); + + // Credentials + String ak = props.get(S3CloudStorageConfig.KEY_ACCESS_KEY); + String sk = props.get(S3CloudStorageConfig.KEY_SECRET_KEY); + if (ak != null && !ak.isEmpty() && sk != null && !sk.isEmpty()) { + builder.credentialsProvider( + StaticCredentialsProvider.create(AwsBasicCredentials.create(ak, sk))); + } else { + builder.credentialsProvider(DefaultCredentialsProvider.builder().build()); + } + + // Region + String region = props.get(S3CloudStorageConfig.KEY_REGION); + if (region != null && !region.isEmpty()) { + builder.region(Region.of(region)); + } + + // Custom endpoint (MinIO, Ceph, LocalStack …) + String endpoint = props.get(S3CloudStorageConfig.KEY_ENDPOINT); + if (endpoint != null && !endpoint.isEmpty()) { + builder.endpointOverride(URI.create(endpoint)); + // Path-style required for most non-AWS S3 services + builder.serviceConfiguration( + software.amazon.awssdk.services.s3.S3Configuration.builder() + .pathStyleAccessEnabled(true) + .build()); + } + + this.s3Client = builder.build(); + log.info("S3CloudStorageProvider initialized: bucket='{}', region='{}', endpoint='{}', " + + "partRetryMaxAttempts={}, partRetryBaseBackoffMs={}, " + + "multipartExhaustedDirectDlq={}", + bucket, region, endpoint, + this.partUploadMaxRetries, + this.partUploadRetryBaseBackoffMs, + this.multipartExhaustedDirectDlq); + } + + private void initRetryConfig(Map props) { + int retryMaxAttempts = parseIntOrDefault( + props.get(S3CloudStorageConfig.KEY_MULTIPART_RETRY_MAX_ATTEMPTS) + ); + if (retryMaxAttempts <= 0) { + log.warn("Invalid cloud.storage.s3.multipart-part-retry-max-attempts={} " + + "(must be > 0), using default {}", + retryMaxAttempts, + S3CloudStorageConfig.DEFAULT_MULTIPART_PART_RETRY_MAX_ATTEMPTS); + this.partUploadMaxRetries = S3CloudStorageConfig.DEFAULT_MULTIPART_PART_RETRY_MAX_ATTEMPTS; + } else { + this.partUploadMaxRetries = retryMaxAttempts; + } + + long retryBaseBackoffMs = parseLongOrDefault( + props.get(S3CloudStorageConfig.KEY_MULTIPART_RETRY_BASE_BACKOFF_MS) + ); + if (retryBaseBackoffMs <= 0L) { + log.warn("Invalid cloud.storage.s3.multipart-part-retry-base-backoff-ms={} " + + "(must be > 0), using default {}", + retryBaseBackoffMs, + S3CloudStorageConfig.DEFAULT_MULTIPART_PART_RETRY_BASE_BACKOFF_MS); + this.partUploadRetryBaseBackoffMs = + S3CloudStorageConfig.DEFAULT_MULTIPART_PART_RETRY_BASE_BACKOFF_MS; + } else { + this.partUploadRetryBaseBackoffMs = retryBaseBackoffMs; + } + + this.multipartExhaustedDirectDlq = parseBooleanOrDefault( + props.get(S3CloudStorageConfig.KEY_MULTIPART_EXHAUSTED_DIRECT_DLQ)); + } + + private static int parseIntOrDefault(String value) { + if (value == null || value.isBlank()) { + return S3CloudStorageConfig.DEFAULT_MULTIPART_PART_RETRY_MAX_ATTEMPTS; + } + try { + return Integer.parseInt(value.trim()); + } catch (NumberFormatException e) { + log.warn("Invalid {}={}, using default {}", + "cloud.storage.s3.multipart-part-retry-max-attempts", value, + S3CloudStorageConfig.DEFAULT_MULTIPART_PART_RETRY_MAX_ATTEMPTS); + return S3CloudStorageConfig.DEFAULT_MULTIPART_PART_RETRY_MAX_ATTEMPTS; + } + } + + private static long parseLongOrDefault(String value) { + if (value == null || value.isBlank()) { + return S3CloudStorageConfig.DEFAULT_MULTIPART_PART_RETRY_BASE_BACKOFF_MS; + } + try { + return Long.parseLong(value.trim()); + } catch (NumberFormatException e) { + log.warn("Invalid {}={}, using default {}", + "cloud.storage.s3.multipart-part-retry-base-backoff-ms", value, + S3CloudStorageConfig.DEFAULT_MULTIPART_PART_RETRY_BASE_BACKOFF_MS); + return S3CloudStorageConfig.DEFAULT_MULTIPART_PART_RETRY_BASE_BACKOFF_MS; + } + } + + private static boolean parseBooleanOrDefault(String value) { + if (value == null || value.isBlank()) { + return false; + } + return Boolean.parseBoolean(value.trim()); + } + + /** + * Uploads a local file to S3. + * + *

Files ≤ {@link #MULTIPART_THRESHOLD_BYTES} use a single PUT request. + * Larger files are split into {@link #PART_SIZE_BYTES} chunks and uploaded via + * the S3 Multipart Upload API, which is required for files larger than 5 GB. + * + *

Timing and throughput are always logged at INFO level after the upload + * completes (or each part for multipart uploads). + */ + @Override + public void uploadFile(String localPath, String remoteKey) throws IOException { + java.nio.file.Path path = Paths.get(localPath); + long fileSize; + try { + fileSize = Files.size(path); + } catch (IOException e) { + throw new IOException("Cannot stat local file: " + localPath, e); + } + + String fullKey = buildKey(remoteKey); + long startNs = System.nanoTime(); + + if (fileSize > MULTIPART_THRESHOLD_BYTES) { + uploadMultipart(path, fileSize, fullKey); + } else { + uploadSinglePart(path, fullKey, localPath); + } + + long elapsedMs = (System.nanoTime() - startNs) / 1_000_000; + double throughputMBps = elapsedMs > 0 + ? (fileSize / 1_048_576.0) / (elapsedMs / 1000.0) + : 0.0; + log.info("S3 upload complete: {} | size={} | elapsed={} ms | throughput={} MB/s", + remoteKey, + humanSize(fileSize), + elapsedMs, + String.format(Locale.US, "%.2f", throughputMBps)); + } + + @Override + public void deleteFile(String remoteKey) throws IOException { + String fullKey = buildKey(remoteKey); + try { + s3Client.deleteObject( + DeleteObjectRequest.builder().bucket(bucket).key(fullKey).build()); + log.debug("S3 delete: s3://{}/{}", bucket, fullKey); + } catch (SdkException e) { + throw classifySdkException("deleteObject", fullKey, e); + } + } + + /** + * Deletes all objects under a prefix using S3's DeleteObjects (batch delete) API. + * + *

Much more efficient than individual deletes, especially for prefixes with many objects. + * Handles pagination internally if the prefix contains more than 1000 objects. + * + * @param remoteDirPrefix directory/prefix inside bucket (without provider pathPrefix) + * @return number of objects deleted + * @throws IOException on I/O or network failure + */ + @Override + public int deletePrefix(String remoteDirPrefix) throws IOException { + String fullPrefix = buildKey(remoteDirPrefix == null ? "" : remoteDirPrefix); + int totalDeleted = 0; + + try { + String token = null; + do { + ListObjectsV2Request.Builder listReq = + ListObjectsV2Request.builder().bucket(bucket).prefix(fullPrefix); + if (token != null) { + listReq.continuationToken(token); + } + ListObjectsV2Response listResp = s3Client.listObjectsV2(listReq.build()); + + List toDelete = new ArrayList<>(); + for (S3Object obj : listResp.contents()) { + String key = obj.key(); + if (key != null && !key.endsWith("/")) { + toDelete.add(ObjectIdentifier.builder().key(key).build()); + } + } + + if (!toDelete.isEmpty()) { + try { + DeleteObjectsResponse deleteResp = s3Client.deleteObjects( + DeleteObjectsRequest.builder() + .bucket(bucket) + .delete(software.amazon.awssdk.services.s3.model.Delete.builder() + .objects(toDelete) + .build()) + .build()); + totalDeleted += deleteResp.deleted().size(); + log.debug("S3 batch delete: deleted {} objects from prefix {}", + deleteResp.deleted().size(), remoteDirPrefix); + } catch (SdkException e) { + log.warn("S3 batch delete failed for prefix='{}': {}", + fullPrefix, e.getMessage()); + // Fall back to individual deletes for any remaining objects + for (ObjectIdentifier obj : toDelete) { + try { + s3Client.deleteObject(DeleteObjectRequest.builder() + .bucket(bucket) + .key(obj.key()) + .build()); + totalDeleted++; + } catch (SdkException ex) { + log.debug("S3 fallback delete failed for key='{}': {}", + obj.key(), ex.getMessage()); + } + } + } + } + + token = listResp.nextContinuationToken(); + } while (token != null && !token.isEmpty()); + + if (totalDeleted > 0) { + log.info("S3 prefix delete completed: prefix={}, deleted={}", remoteDirPrefix, totalDeleted); + } + return totalDeleted; + + } catch (SdkException e) { + throw classifySdkException("deletePrefix", fullPrefix, e); + } + } + + @Override + public boolean fileExists(String remoteKey) throws IOException { + String fullKey = buildKey(remoteKey); + try { + s3Client.headObject(HeadObjectRequest.builder().bucket(bucket).key(fullKey).build()); + return true; + } catch (NoSuchKeyException e) { + return false; + } catch (AwsServiceException e) { + // Some S3-compatible providers return generic service exceptions for 404. + if (e.statusCode() == 404) { + return false; + } + throw classifySdkException("headObject", fullKey, e); + } catch (SdkException e) { + throw classifySdkException("headObject", fullKey, e); + } + } + + @Override + public List listFiles(String remoteDirPrefix) throws IOException { + String fullPrefix = buildKey(remoteDirPrefix == null ? "" : remoteDirPrefix); + List keys = new ArrayList<>(); + try { + String token = null; + do { + ListObjectsV2Request.Builder req = + ListObjectsV2Request.builder().bucket(bucket).prefix(fullPrefix); + if (token != null) { + req.continuationToken(token); + } + ListObjectsV2Response resp = s3Client.listObjectsV2(req.build()); + for (S3Object obj : resp.contents()) { + String key = obj.key(); + if (key == null || key.endsWith("/")) { + continue; + } + keys.add(stripPathPrefix(key)); + } + token = resp.nextContinuationToken(); + } while (token != null && !token.isEmpty()); + return keys; + } catch (SdkException e) { + throw classifySdkException("listObjectsV2", fullPrefix, e); + } + } + + @Override + public void downloadFile(String remoteKey, String localPath) throws IOException { + String fullKey = buildKey(remoteKey); + long startNs = System.nanoTime(); + Path destinationPath = Paths.get(localPath); + try { + + s3Client.getObject( + GetObjectRequest.builder().bucket(bucket).key(fullKey).build(), + destinationPath); + } catch (SdkException e) { + throw classifySdkException("getObject", fullKey, e); + } + long elapsedMs = (System.nanoTime() - startNs) / 1_000_000; + long fileSize = 0; + try { + fileSize = Files.size(destinationPath); + } catch (IOException ignored) { + // best-effort; don't fail download reporting + } + double throughputMBps = elapsedMs > 0 + ? (fileSize / 1_048_576.0) / (elapsedMs / 1000.0) + : 0.0; + log.info("S3 download complete: {} | size={} | elapsed={} ms | throughput={} MB/s", + remoteKey, + humanSize(fileSize), + elapsedMs, + String.format(Locale.US, "%.2f", throughputMBps)); + } + + @Override + public void close() throws IOException { + if (s3Client != null) { + s3Client.close(); + s3Client = null; + log.info("S3CloudStorageProvider closed"); + } + } + + // ----------------------------------------------------------------------- + // Internal – upload strategies + // ----------------------------------------------------------------------- + + /** + * Single-PUT upload for files ≤ {@link #MULTIPART_THRESHOLD_BYTES}, with bounded + * exponential-backoff retry on transient failures (using the same tuning as multipart parts). + * Without this, a single transient network blip on the common small-SST path would surface + * immediately and, with whole-file retries disabled, go straight to the DLQ. + */ + private void uploadSinglePart(java.nio.file.Path path, String fullKey, + String localPath) throws IOException { + IOException last = null; + for (int attempt = 1; attempt <= this.partUploadMaxRetries; attempt++) { + try { + s3Client.putObject( + PutObjectRequest.builder().bucket(bucket).key(fullKey).build(), + path); + return; + } catch (SdkException e) { + IOException classified = classifySdkException("putObject", fullKey, e); + if (classified instanceof CloudStorageNonRetryableException) { + throw classified; + } + last = classified; + if (attempt >= this.partUploadMaxRetries) { + break; + } + long backoffMs = this.partUploadRetryBaseBackoffMs * (1L << (attempt - 1)); + log.warn("S3 single-PUT retry: attempt={}/{} key={} reason={} nextBackoffMs={}", + attempt, this.partUploadMaxRetries, fullKey, + classified.getMessage(), backoffMs); + sleepQuietly(backoffMs); + } + } + throw new IOException( + "S3 upload failed for local='" + localPath + "' key='" + fullKey + "' after " + + this.partUploadMaxRetries + " attempt(s)", last); + } + + /** + * Multipart upload for files > {@link #MULTIPART_THRESHOLD_BYTES}. + * + *

Each part is logged individually so that progress of multi-hour uploads + * is visible in the server log: + * 

+     *   S3 multipart part 1/41 uploaded: size=512.0 MB | elapsed=6 230 ms | throughput=82.18 MB/s
+     *   S3 multipart part 2/41 uploaded: size=512.0 MB | elapsed=6 050 ms | throughput=84.63 MB/s
+     *   ...
+     *   S3 multipart upload completed: key=hugegraph/hgstore-data/000099.sst | parts=41
+     *
+ * + *

If any part fails the multipart upload is aborted (to avoid incomplete-upload storage + * charges) and an {@link IOException} is thrown. + */ + private void uploadMultipart(java.nio.file.Path path, long fileSize, + String fullKey) throws IOException { + int totalParts = (int) Math.ceil((double) fileSize / PART_SIZE_BYTES); + log.info("S3 multipart upload started: key={} | size={} | parts={} | partSize={} MB", + fullKey, humanSize(fileSize), totalParts, PART_SIZE_MB); + + // Step 1 – initiate + CreateMultipartUploadResponse initResp; + try { + initResp = s3Client.createMultipartUpload( + CreateMultipartUploadRequest.builder().bucket(bucket).key(fullKey).build()); + } catch (SdkException e) { + throw classifySdkException("createMultipartUpload", fullKey, e); + } + String uploadId = initResp.uploadId(); + + List completedParts = new ArrayList<>(totalParts); + try { + // Step 2 – upload each part + for (int partNum = 1; partNum <= totalParts; partNum++) { + long offset = (long) (partNum - 1) * PART_SIZE_BYTES; + long partLen = Math.min(PART_SIZE_BYTES, fileSize - offset); + + long partStartNs = System.nanoTime(); + String eTag = uploadOnePartWithRetry(path, fullKey, uploadId, + partNum, totalParts, + offset, partLen); + long partElapsedMs = (System.nanoTime() - partStartNs) / 1_000_000; + double partThroughput = partElapsedMs > 0 + ? (partLen / 1_048_576.0) / (partElapsedMs / 1000.0) + : 0.0; + log.info("S3 multipart part {}/{} uploaded: size={} | elapsed={} ms | " + + "throughput={} MB/s", + partNum, totalParts, + humanSize(partLen), + partElapsedMs, + String.format(Locale.US, "%.2f", partThroughput)); + + completedParts.add(CompletedPart.builder() + .partNumber(partNum) + .eTag(eTag) + .build()); + } + + // Step 3 – complete + s3Client.completeMultipartUpload( + CompleteMultipartUploadRequest.builder() + .bucket(bucket).key(fullKey) + .uploadId(uploadId) + .multipartUpload( + CompletedMultipartUpload.builder() + .parts(completedParts) + .build()) + .build()); + log.info("S3 multipart upload completed: key={} | parts={}", fullKey, totalParts); + + } catch (Exception e) { + // Abort to avoid partial-upload storage charges + try { + s3Client.abortMultipartUpload( + AbortMultipartUploadRequest.builder() + .bucket(bucket).key(fullKey) + .uploadId(uploadId).build()); + log.warn("S3 multipart upload aborted: key={} uploadId={}", fullKey, uploadId); + } catch (Exception abortEx) { + log.warn("S3 multipart abort failed: key={} uploadId={} reason={}", + fullKey, uploadId, abortEx.getMessage()); + } + if (e instanceof CloudStorageNonRetryableException) { + throw (CloudStorageNonRetryableException) e; + } + throw new IOException( + "S3 multipart upload failed for key='" + fullKey + "'", e); + } + } + + /** + * Uploads a single part of a multipart upload and returns its ETag. + * Supplies a replayable stream provider so AWS SDK retries can reopen the + * exact byte range for each attempt. + */ + private String uploadOnePart(java.nio.file.Path path, String fullKey, + String uploadId, int partNumber, + long offset, long partLen) throws IOException { + try { + ContentStreamProvider partStreamProvider = + () -> openBoundedPartStream(path, offset, partLen); + UploadPartResponse resp = s3Client.uploadPart( + UploadPartRequest.builder() + .bucket(bucket).key(fullKey) + .uploadId(uploadId).partNumber(partNumber) + .contentLength(partLen) + .build(), + RequestBody.fromContentProvider(partStreamProvider, + partLen, + "application/octet-stream")); + return resp.eTag(); + } catch (SdkException e) { + throw classifySdkException("uploadPart(part=" + partNumber + ")", fullKey, e); + } + } + + /** + * Opens a fresh stream for the exact byte-range of one multipart part. + * + *

Returned stream starts at {@code offset} and is bounded to {@code partLen} + * bytes, allowing AWS SDK to recreate request bodies for retries. + */ + private InputStream openBoundedPartStream(java.nio.file.Path path, + long offset, + long partLen) { + try { + FileInputStream fis = new FileInputStream(path.toFile()); + long remaining = offset; + while (remaining > 0) { + long skipped = fis.skip(remaining); + if (skipped <= 0) { + fis.close(); + throw new IOException( + "Unexpected EOF while seeking to offset " + offset + " in " + path); + } + remaining -= skipped; + } + return new LimitedInputStream(fis, partLen); + } catch (IOException e) { + throw new IllegalStateException( + "Failed to open multipart part stream at offset=" + offset + + " length=" + partLen + " path=" + path, e); + } + } + + /** + * Uploads one multipart chunk with local retries. This avoids restarting the whole + * SST upload when only one or two parts fail due to transient network/S3 errors. + */ + private String uploadOnePartWithRetry(java.nio.file.Path path, + String fullKey, + String uploadId, + int partNumber, + int totalParts, + long offset, + long partLen) throws IOException { + IOException last = null; + for (int attempt = 1; attempt <= this.partUploadMaxRetries; attempt++) { + try { + return uploadOnePart(path, fullKey, uploadId, partNumber, offset, partLen); + } catch (IOException e) { + if (e instanceof CloudStorageNonRetryableException) { + throw e; + } + last = e; + if (attempt >= this.partUploadMaxRetries) { + break; + } + long backoffMs = this.partUploadRetryBaseBackoffMs * (1L << (attempt - 1)); + log.warn("S3 multipart part retry: part={}/{} attempt={}/{} key={} " + + "reason={} nextBackoffMs={}", + partNumber, totalParts, + attempt, this.partUploadMaxRetries, + fullKey, + e.getMessage(), + backoffMs); + sleepQuietly(backoffMs); + } + } + String message = String.format( + "S3 multipart part failed after %d attempt(s): key=%s part=%d/%d", + this.partUploadMaxRetries, fullKey, partNumber, totalParts); + if (this.multipartExhaustedDirectDlq) { + throw new CloudStorageNonRetryableException(message, last); + } + throw new IOException(message, last); + } + + private static void sleepQuietly(long ms) { + try { + Thread.sleep(ms); + } catch (InterruptedException ie) { + Thread.currentThread().interrupt(); + } + } + + /** + * Classifies SDK exceptions into retryable/non-retryable provider exceptions. + * + *

Retryable: + *

    + *
  • client-side transport failures ({@link SdkException});
    • + *
  • service throttling / transient statuses (408/425/429/500/502/503/504).
    • + *
+ * Non-retryable: + *
    + *
  • permanent service-side statuses (e.g. auth/permission/not-found/validation).
    • + *
+ */ + private IOException classifySdkException(String operation, String key, SdkException e) { + if (e instanceof AwsServiceException) { + AwsServiceException ase = (AwsServiceException) e; + int status = ase.statusCode(); + String code = ase.awsErrorDetails() != null ? ase.awsErrorDetails().errorCode() : ""; + String requestId = ase.requestId(); + String message = String.format(Locale.US, + "S3 %s failed: key=%s status=%d code=%s requestId=%s", + operation, key, status, code, requestId); + if (isRetryableServiceFailure(ase)) { + return new IOException(message, ase); + } + return new CloudStorageNonRetryableException(message, ase); + } + + // Client-side network/IO/timeout failures are retryable by default. + return new IOException("S3 " + operation + " failed for key='" + key + "'", e); + } + + private static boolean isRetryableServiceFailure(AwsServiceException e) { + if (e.isThrottlingException()) { + return true; + } + int status = e.statusCode(); + return status == 408 || status == 425 || status == 429 || + status == 500 || status == 502 || status == 503 || status == 504; + } + + // ----------------------------------------------------------------------- + // Internal – key helpers + // ----------------------------------------------------------------------- + + /** + * Prepends {@link #pathPrefix} to the supplied key, using "/" as separator. + * If the prefix is null or empty, the key is returned unchanged. + */ + private String buildKey(String key) { + if (pathPrefix == null || pathPrefix.isEmpty()) { + return key; + } + // Normalise leading slashes + String normalKey = key.startsWith("/") ? key.substring(1) : key; + return pathPrefix.endsWith("/") + ? pathPrefix + normalKey + : pathPrefix + "/" + normalKey; + } + + private String stripPathPrefix(String fullKey) { + if (fullKey == null) { + return ""; + } + if (pathPrefix == null || pathPrefix.isEmpty()) { + return fullKey.startsWith("/") ? fullKey.substring(1) : fullKey; + } + String normalizedPrefix = pathPrefix.endsWith("/") ? pathPrefix : pathPrefix + "/"; + if (fullKey.startsWith(normalizedPrefix)) { + return fullKey.substring(normalizedPrefix.length()); + } + return fullKey; + } + + // ----------------------------------------------------------------------- + // Internal – formatting + // ----------------------------------------------------------------------- + + static String humanSize(long bytes) { + if (bytes < 1024L) return bytes + " B"; + if (bytes < 1024L * 1024) return String.format(Locale.US, "%.1f KB", bytes / 1024.0); + if (bytes < 1024L * 1024 * 1024) return String.format(Locale.US, "%.1f MB", + bytes / (1024.0 * 1024)); + return String.format(Locale.US, "%.2f GB", bytes / (1024.0 * 1024 * 1024)); + } + + // ----------------------------------------------------------------------- + // Inner types + // ----------------------------------------------------------------------- + + /** + * An {@link InputStream} wrapper that limits reading to exactly {@code limit} bytes. + * Used to feed each multipart chunk to the S3 SDK without loading it into memory. + */ + private static final class LimitedInputStream extends InputStream { + + private final InputStream wrapped; + private long remaining; + + LimitedInputStream(InputStream wrapped, long limit) { + this.wrapped = wrapped; + this.remaining = limit; + } + + @Override + public int read() throws IOException { + if (remaining <= 0) { + return -1; + } + int b = wrapped.read(); + if (b >= 0) { + remaining--; + } + return b; + } + + @Override + public int read(@NotNull byte[] buf, int off, int len) throws IOException { + if (remaining <= 0) { + return -1; + } + int toRead = (int) Math.min(len, remaining); + int n = wrapped.read(buf, off, toRead); + if (n > 0) { + remaining -= n; + } + return n; + } + + @Override + public void close() throws IOException { + wrapped.close(); + } + } +} diff --git a/hugegraph-store/hg-store-cloud-s3/src/main/resources/META-INF/services/org.apache.hugegraph.store.cloud.CloudStorageProvider b/hugegraph-store/hg-store-cloud-s3/src/main/resources/META-INF/services/org.apache.hugegraph.store.cloud.CloudStorageProvider new file mode 100644 index 0000000000..241d82992b --- /dev/null +++ b/hugegraph-store/hg-store-cloud-s3/src/main/resources/META-INF/services/org.apache.hugegraph.store.cloud.CloudStorageProvider @@ -0,0 +1,2 @@ +org.apache.hugegraph.store.cloud.s3.S3CloudStorageProvider + diff --git a/hugegraph-store/hg-store-cloud-s3/src/test/java/org/apache/hugegraph/store/cloud/s3/S3CloudStorageProviderClassificationTest.java b/hugegraph-store/hg-store-cloud-s3/src/test/java/org/apache/hugegraph/store/cloud/s3/S3CloudStorageProviderClassificationTest.java new file mode 100644 index 0000000000..9289c156c5 --- /dev/null +++ b/hugegraph-store/hg-store-cloud-s3/src/test/java/org/apache/hugegraph/store/cloud/s3/S3CloudStorageProviderClassificationTest.java @@ -0,0 +1,184 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.cloud.s3; + +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.assertFalse; + +import java.io.IOException; +import java.lang.reflect.Field; +import java.lang.reflect.InvocationTargetException; +import java.lang.reflect.Method; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.concurrent.atomic.AtomicBoolean; + +import org.apache.hugegraph.store.cloud.CloudStorageNonRetryableException; +import org.junit.Test; + +import software.amazon.awssdk.awscore.exception.AwsErrorDetails; +import software.amazon.awssdk.awscore.exception.AwsServiceException; +import software.amazon.awssdk.core.exception.SdkClientException; +import software.amazon.awssdk.core.exception.SdkException; +import software.amazon.awssdk.services.s3.S3Client; +import software.amazon.awssdk.services.s3.model.AbortMultipartUploadResponse; +import software.amazon.awssdk.services.s3.model.CreateMultipartUploadResponse; +import software.amazon.awssdk.services.s3.model.S3Exception; + +public class S3CloudStorageProviderClassificationTest { + + @Test + public void classifySdkException_retryableServiceStatus_returnsIOException() throws Exception { + S3CloudStorageProvider provider = new S3CloudStorageProvider(); + AwsServiceException retryable503 = s3ServiceException(503, "SlowDown", "req-503"); + + IOException classified = invokeClassify(provider, retryable503); + + assertFalse("503 should be treated as retryable IOException", + classified instanceof CloudStorageNonRetryableException); + } + + @Test + public void classifySdkException_nonRetryableServiceStatus_returnsNonRetryable() throws Exception { + S3CloudStorageProvider provider = new S3CloudStorageProvider(); + AwsServiceException forbidden403 = s3ServiceException(403, "AccessDenied", "req-403"); + + IOException classified = invokeClassify(provider, forbidden403); + + assertTrue("403 should be treated as non-retryable", + classified instanceof CloudStorageNonRetryableException); + } + + @Test + public void classifySdkException_retryable429Status_isRetryable() throws Exception { + S3CloudStorageProvider provider = new S3CloudStorageProvider(); + AwsServiceException throttled = s3ServiceException(429, "TooManyRequests", "req-throttle"); + + IOException classified = invokeClassify(provider, throttled); + + assertFalse("429 should be treated as retryable", + classified instanceof CloudStorageNonRetryableException); + } + + @Test + public void classifySdkException_clientSideFailure_isRetryableIOException() throws Exception { + S3CloudStorageProvider provider = new S3CloudStorageProvider(); + SdkClientException clientFailure = + SdkClientException.builder().message("connection reset").build(); + + IOException classified = invokeClassify(provider, clientFailure); + + assertFalse("Client-side failures should stay retryable", + classified instanceof CloudStorageNonRetryableException); + } + + @Test + public void uploadMultipart_nonRetryablePartFailure_rethrowsNonRetryableAfterAbort() + throws Exception { + S3CloudStorageProvider provider = new S3CloudStorageProvider(); + AtomicBoolean aborted = new AtomicBoolean(false); + + // Dynamic proxy keeps this test lightweight without extra mocking deps. + S3Client s3 = (S3Client) java.lang.reflect.Proxy.newProxyInstance( + S3Client.class.getClassLoader(), + new Class[]{S3Client.class}, + (proxy, method, args) -> { + String name = method.getName(); + switch (name) { + case "createMultipartUpload": + return CreateMultipartUploadResponse.builder() + .uploadId("u-1") + .build(); + case "uploadPart": + throw s3ServiceException(403, "AccessDenied", "req-upload-part"); + case "abortMultipartUpload": + aborted.set(true); + return AbortMultipartUploadResponse.builder().build(); + case "close": + return null; + } + throw new UnsupportedOperationException("Unexpected S3Client method: " + name); + }); + + setField(provider, "s3Client", s3); + setField(provider, "bucket", "test-bucket"); + setField(provider, "partUploadMaxRetries", 2); + + Path tmp = Files.createTempFile("hg-s3-classify", ".bin"); + Files.write(tmp, new byte[]{1}); + try { + try { + invokeUploadMultipart(provider, tmp); + } catch (CloudStorageNonRetryableException e) { + // expected path + } + assertTrue("Multipart failure must abort upload before rethrowing", aborted.get()); + } finally { + Files.deleteIfExists(tmp); + } + } + + private static IOException invokeClassify(S3CloudStorageProvider provider, SdkException e) + throws Exception { + Method classify = S3CloudStorageProvider.class.getDeclaredMethod( + "classifySdkException", String.class, String.class, SdkException.class); + classify.setAccessible(true); + return (IOException) classify.invoke(provider, "uploadPart", "k.sst", e); + } + + private static void invokeUploadMultipart(S3CloudStorageProvider provider, + Path path) throws Exception { + Method method = S3CloudStorageProvider.class.getDeclaredMethod( + "uploadMultipart", java.nio.file.Path.class, long.class, String.class); + method.setAccessible(true); + try { + method.invoke(provider, path, 1L, "k.sst"); + } catch (InvocationTargetException ite) { + Throwable cause = ite.getCause(); + if (cause instanceof Exception) { + throw (Exception) cause; + } + throw ite; + } + } + + private static void setField(S3CloudStorageProvider provider, + String fieldName, + Object value) throws Exception { + Field field = S3CloudStorageProvider.class.getDeclaredField(fieldName); + field.setAccessible(true); + field.set(provider, value); + } + + private static AwsServiceException s3ServiceException(int statusCode, + String errorCode, + String requestId) { + AwsErrorDetails details = AwsErrorDetails.builder() + .serviceName("S3") + .errorCode(errorCode) + .errorMessage("simulated") + .build(); + return S3Exception.builder() + .statusCode(statusCode) + .requestId(requestId) + .awsErrorDetails(details) + .message("simulated") + .build(); + } +} + diff --git a/hugegraph-store/hg-store-cloud-s3/src/test/java/org/apache/hugegraph/store/cloud/s3/S3SingleLargeFileE2ETest.java b/hugegraph-store/hg-store-cloud-s3/src/test/java/org/apache/hugegraph/store/cloud/s3/S3SingleLargeFileE2ETest.java new file mode 100644 index 0000000000..2decdcc998 --- /dev/null +++ b/hugegraph-store/hg-store-cloud-s3/src/test/java/org/apache/hugegraph/store/cloud/s3/S3SingleLargeFileE2ETest.java @@ -0,0 +1,420 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.cloud.s3; + +import java.io.BufferedOutputStream; +import java.io.FileOutputStream; +import java.io.IOException; +import java.net.Socket; +import java.net.URI; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Locale; +import java.util.UUID; + +import org.apache.hugegraph.store.cloud.CloudStorageConfig; +import org.junit.After; +import org.junit.Assert; +import org.junit.Assume; +import org.junit.Before; +import org.junit.Test; + +import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; +import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider; +import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.s3.S3Client; +import software.amazon.awssdk.services.s3.S3ClientBuilder; +import software.amazon.awssdk.services.s3.model.CreateBucketRequest; +import software.amazon.awssdk.services.s3.model.HeadBucketRequest; + +/** + * End-to-end large-file S3 test focused on one realistic scenario: + * generate one 20 GB SST-like file, upload (multipart), check remote existence, + * download it back, and validate size. + * + *

Setup: The test requires a running S3-compatible service (e.g., MinIO). + * It automatically skips if the endpoint is unreachable or required properties are missing. + * + *

Zero-config local run: If no VM options are set, the test defaults to a local + * MinIO instance at {@code http://localhost:9000} with credentials {@code minioadmin/minioadmin}. + * Simply start MinIO and run the test directly from your IDE: + * 

+ * docker run -p 9000:9000 \
+ *   -e MINIO_ROOT_USER=minioadmin \
+ *   -e MINIO_ROOT_PASSWORD=minioadmin \
+ *   minio/minio server /data
+ *
+ * + *

Custom run with explicit properties: + * 

+ * # Start MinIO in Docker:
+ * docker run -p 9000:9000 -e MINIO_ROOT_USER=minioadmin \
+ *   -e MINIO_ROOT_PASSWORD=minioadmin minio/minio server /data
+ *
+ * # Run test:
+ * mvn test -pl hugegraph-store/hg-store-cloud-s3 \
+ *   -Dtest=S3SingleLargeFileE2ETest \
+ *   -Ds3.perf.endpoint=... \
+ *   -Ds3.perf.bucket=hugegraph-perf \
+ *   -Ds3.perf.accessKey=minioadmin \
+ *   -Ds3.perf.secretKey=minioadmin \
+ *   -Ds3.perf.region=us-east-1 \
+ *   -Ds3.perf.singleFileSizeGb=20
+ *
+ * + *

Properties: + *

    + *
  • {@code s3.perf.bucket} (required) — S3 bucket name + *
  • {@code s3.perf.endpoint} (required if MinIO) — Service endpoint URL + *
  • {@code s3.perf.accessKey} — Access key (default: empty, uses default credentials) + *
  • {@code s3.perf.secretKey} — Secret key (default: empty, uses default credentials) + *
  • {@code s3.perf.region} — AWS region (default: us-east-1) + *
  • {@code s3.perf.pathPrefix} — Path prefix in bucket (default: hugegraph-perf) + *
  • {@code s3.perf.singleFileSizeGb} — File size in GB for test (default: 20) + *
  • {@code s3.perf.skipCleanup} — Skip remote file cleanup after test (default: false) + *
+ * + *

The test auto-skips if: + *

    + *
  • The endpoint (default: {@code http://localhost:9000}) is not reachable + *
+ */ +@SuppressWarnings("SameParameterValue") +public class S3SingleLargeFileE2ETest { + + private static final String PROP_BUCKET = "s3.perf.bucket"; + private static final String PROP_REGION = "s3.perf.region"; + private static final String PROP_ENDPOINT = "s3.perf.endpoint"; + private static final String PROP_ACCESS_KEY = "s3.perf.accessKey"; + private static final String PROP_SECRET_KEY = "s3.perf.secretKey"; + private static final String PROP_PATH_PREFIX = "s3.perf.pathPrefix"; + private static final String PROP_SINGLE_FILE_GB = "s3.perf.singleFileSizeGb"; + private static final String PROP_SKIP_CLEANUP = "s3.perf.skipCleanup"; + + // Defaults for zero-config local MinIO runs (e.g. running the test directly from an IDE). + // These values match the standard MinIO Docker quickstart: + // docker run -p 9000:9000 -e MINIO_ROOT_USER=minioadmin \ + // -e MINIO_ROOT_PASSWORD=minioadmin minio/minio server /data + private static final String DEFAULT_ENDPOINT = "http://localhost:9000"; + private static final String DEFAULT_BUCKET = "hugegraph-perf"; + private static final String DEFAULT_REGION = "us-east-1"; + private static final String DEFAULT_ACCESS_KEY = "minioadmin"; + private static final String DEFAULT_SECRET_KEY = "minioadmin"; + + private S3CloudStorageProvider provider; + private Path tmpDir; + private String remoteKey; + private boolean skipCleanup; + + /** S3 client used exclusively for bucket-lifecycle management in tests. */ + private S3Client adminS3Client; + + @Before + public void setUp() throws IOException { + String endpoint = System.getProperty(PROP_ENDPOINT, DEFAULT_ENDPOINT); + String bucket = System.getProperty(PROP_BUCKET, DEFAULT_BUCKET); + String region = System.getProperty(PROP_REGION, DEFAULT_REGION); + String ak = System.getProperty(PROP_ACCESS_KEY, DEFAULT_ACCESS_KEY); + String sk = System.getProperty(PROP_SECRET_KEY, DEFAULT_SECRET_KEY); + + // Verify the endpoint is reachable — skip gracefully if MinIO / S3 is not running + Assume.assumeTrue( + "Skipping S3SingleLargeFileE2ETest: S3 endpoint not reachable at " + endpoint + + ". Start MinIO with: docker run -p 9000:9000 " + + "-e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin " + + "minio/minio server /data", + isEndpointReachable(endpoint)); + + // Build an admin S3 client for bucket management + this.adminS3Client = buildAdminS3Client(endpoint, region, ak, sk); + + // Auto-create the bucket if it does not exist yet + ensureBucketExists(this.adminS3Client, bucket, region, endpoint); + + CloudStorageConfig s3Cfg = new CloudStorageConfig(); + s3Cfg.setEnabled(true); + s3Cfg.setProvider("s3"); + s3Cfg.setPathPrefix(System.getProperty(PROP_PATH_PREFIX, "hugegraph-perf")); + s3Cfg.getProviderProperties().put("bucket", bucket); + s3Cfg.getProviderProperties().put("region", region); + s3Cfg.getProviderProperties().put("endpoint", endpoint); + s3Cfg.getProviderProperties().put("access-key", ak); + s3Cfg.getProviderProperties().put("secret-key", sk); + + this.skipCleanup = boolProp(PROP_SKIP_CLEANUP, false); + this.provider = new S3CloudStorageProvider(); + this.provider.init(s3Cfg); + this.tmpDir = Files.createTempDirectory("hg-s3-e2e-"); + } + + @After + public void tearDown() throws Exception { + if (this.provider != null) { + if (!this.skipCleanup && this.remoteKey != null) { + try { + this.provider.deleteFile(this.remoteKey); + } catch (Exception ignored) { + // Best-effort cleanup only. + } + } + this.provider.close(); + } + if (this.adminS3Client != null) { + this.adminS3Client.close(); + } + if (this.tmpDir != null) { + deleteDirectory(this.tmpDir.toFile()); + } + } + + @Test + public void uploadDownloadLargeFileEndToEnd() throws Exception { + long fileSizeGb = longProp(PROP_SINGLE_FILE_GB, 20L); + long fileSizeBytes = fileSizeGb * 1024L * 1024L * 1024L; + + Assert.assertTrue("file size must trigger multipart upload", + fileSizeBytes > S3CloudStorageProvider.MULTIPART_THRESHOLD_BYTES); + + Path localSrc = this.tmpDir.resolve("src-" + UUID.randomUUID() + ".sst"); + Path localDst = this.tmpDir.resolve("dst-" + UUID.randomUUID() + ".sst"); + this.remoteKey = "perf-single-file/hugegraph-large-" + UUID.randomUUID() + ".sst"; + + System.out.printf(Locale.US, + "%n[E2E] generating file: size=%d GB (%s)%n", + fileSizeGb, humanSize(fileSizeBytes)); + generateLargeFile(localSrc, fileSizeBytes); + + long uploadStart = System.nanoTime(); + this.provider.uploadFile(localSrc.toString(), this.remoteKey); + long uploadMs = (System.nanoTime() - uploadStart) / 1_000_000; + + Assert.assertTrue("uploaded object not found in remote storage", + this.provider.fileExists(this.remoteKey)); + + long downloadStart = System.nanoTime(); + this.provider.downloadFile(this.remoteKey, localDst.toString()); + long downloadMs = (System.nanoTime() - downloadStart) / 1_000_000; + + long srcSize = Files.size(localSrc); + long dstSize = Files.size(localDst); + Assert.assertEquals("downloaded file size mismatch", srcSize, dstSize); + + double uploadMBps = srcSize > 0 && uploadMs > 0 + ? (srcSize / 1_048_576.0) / (uploadMs / 1000.0) + : 0.0; + double downloadMBps = dstSize > 0 && downloadMs > 0 + ? (dstSize / 1_048_576.0) / (downloadMs / 1000.0) + : 0.0; + + System.out.printf(Locale.US, + "[E2E] upload=%s (%.2f MB/s), download=%s (%.2f MB/s)%n", + humanDuration(uploadMs), uploadMBps, + humanDuration(downloadMs), downloadMBps); + } + + private static void generateLargeFile(Path dest, long sizeBytes) throws IOException { + final int blockSize = 64 * 1024 * 1024; + byte[] block = buildSstBlock(blockSize); + long written = 0L; + + try (BufferedOutputStream out = new BufferedOutputStream( + new FileOutputStream(dest.toFile()), blockSize)) { + while (written < sizeBytes) { + long toWrite = Math.min(blockSize, sizeBytes - written); + block[0] = (byte) (written >>> 20); + out.write(block, 0, (int) toWrite); + written += toWrite; + } + } + } + + private static byte[] buildSstBlock(int size) { + byte[] block = new byte[size]; + block[0] = (byte) 0x57; + block[1] = (byte) 0xfb; + block[2] = (byte) 0x80; + block[3] = (byte) 0x8b; + for (int i = 4; i < size; i++) { + block[i] = (byte) ((i * 1103515245 + 12345) & 0xFF); + } + return block; + } + + private static String humanSize(long bytes) { + if (bytes < 1024L) { + return bytes + " B"; + } + if (bytes < 1024L * 1024L) { + return String.format(Locale.US, "%.1f KB", bytes / 1024.0); + } + if (bytes < 1024L * 1024L * 1024L) { + return String.format(Locale.US, "%.1f MB", bytes / (1024.0 * 1024.0)); + } + return String.format(Locale.US, "%.2f GB", bytes / (1024.0 * 1024.0 * 1024.0)); + } + + private static String humanDuration(long millis) { + if (millis < 1000L) { + return millis + " ms"; + } + if (millis < 60_000L) { + return String.format(Locale.US, "%.1f s", millis / 1000.0); + } + long m = millis / 60_000L; + long s = (millis % 60_000L) / 1000L; + return String.format(Locale.US, "%dm %02ds", m, s); + } + + private static long longProp(String key, long def) { + String v = System.getProperty(key); + if (v == null || v.isBlank()) { + return def; + } + try { + return Long.parseLong(v.trim()); + } catch (NumberFormatException e) { + return def; + } + } + + private static boolean boolProp(String key, boolean def) { + String v = System.getProperty(key); + if (v == null || v.isBlank()) { + return def; + } + return Boolean.parseBoolean(v.trim()); + } + + /** + * Builds an S3 admin client for bucket management during tests. + * Mirrors the same credential / endpoint / region logic used by {@link S3CloudStorageProvider}. + */ + private static S3Client buildAdminS3Client(String endpoint, String region, + String ak, String sk) { + S3ClientBuilder builder = S3Client.builder(); + + if (ak != null && !ak.isEmpty() && sk != null && !sk.isEmpty()) { + builder.credentialsProvider( + StaticCredentialsProvider.create(AwsBasicCredentials.create(ak, sk))); + } else { + builder.credentialsProvider(DefaultCredentialsProvider.builder().build()); + } + + if (region != null && !region.isEmpty()) { + builder.region(Region.of(region)); + } + + if (endpoint != null && !endpoint.isEmpty()) { + builder.endpointOverride(URI.create(endpoint)); + builder.serviceConfiguration( + software.amazon.awssdk.services.s3.S3Configuration.builder() + .pathStyleAccessEnabled(true) + .build()); + } + + return builder.build(); + } + + /** + * Creates the bucket if it does not already exist. + * For AWS S3, {@code CreateBucketRequest} needs a {@code LocationConstraint} for + * regions other than {@code us-east-1}; for MinIO (path-style) no constraint is needed. + * + * @param s3 admin S3 client + * @param bucket bucket name to ensure + * @param region AWS region (used for location constraint on real AWS) + * @param endpoint custom endpoint (empty for real AWS) + */ + private static void ensureBucketExists(S3Client s3, String bucket, + String region, String endpoint) { + try { + s3.headBucket(HeadBucketRequest.builder().bucket(bucket).build()); + System.out.printf("[E2E] bucket already exists: %s%n", bucket); + } catch (software.amazon.awssdk.services.s3.model.S3Exception e) { + // Bucket does not exist — create it + System.out.printf("[E2E] bucket '%s' not found, creating...%n", bucket); + + CreateBucketRequest.Builder createReq = CreateBucketRequest.builder().bucket(bucket); + + // Real AWS: us-east-1 must NOT have a LocationConstraint; every other region must. + boolean isRealAws = endpoint == null || endpoint.isEmpty(); + if (isRealAws && region != null && !region.isEmpty() + && !region.equals("us-east-1")) { + createReq.createBucketConfiguration( + software.amazon.awssdk.services.s3.model.CreateBucketConfiguration.builder() + .locationConstraint(region) + .build()); + } + + s3.createBucket(createReq.build()); + System.out.printf("[E2E] bucket created: %s%n", bucket); + } + } + + /** + * Checks if the S3 endpoint (e.g., ...) is reachable. + * Parses the URL to extract host and port, then attempts a socket connection. + * + * @param endpoint the endpoint URL (e.g., ...) + * @return true if reachable, false otherwise + */ + private static boolean isEndpointReachable(String endpoint) { + try { + // Parse endpoint URL to extract host and port + // Expected format: http://host:port or https://host:port + java.net.URL url = new java.net.URL(endpoint); + String host = url.getHost(); + int port = url.getPort(); + if (port == -1) { + port = url.getDefaultPort(); // 80 for http, 443 for https + } + + if (host == null || host.isBlank() || port <= 0) { + return false; + } + + // Attempt socket connection with 2-second timeout + try (Socket socket = new Socket()) { + socket.connect(new java.net.InetSocketAddress(host, port), 2000); + return true; + } + } catch (Exception e) { + return false; + } + } + + @SuppressWarnings("ResultOfMethodCallIgnored") + private static void deleteDirectory(java.io.File dir) { + if (!dir.exists()) { + return; + } + java.io.File[] children = dir.listFiles(); + if (children != null) { + for (java.io.File child : children) { + if (child.isDirectory()) { + deleteDirectory(child); + } else { + child.delete(); + } + } + } + dir.delete(); + } +} + diff --git a/hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageConfig.java b/hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageConfig.java new file mode 100644 index 0000000000..81ff9a3efe --- /dev/null +++ b/hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageConfig.java @@ -0,0 +1,125 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.cloud; + +import java.util.HashMap; +import java.util.Map; + +import lombok.Data; + +/** + * Configuration for pluggable cloud storage providers. + * + *

Mapped from application.yml under the {@code cloud.storage} prefix. + 

+ * cloud:
+ *   storage:
+ *     enabled: true
+ *     provider: s3
+ *     path-prefix: hugegraph
+ *     s3:
+ *       bucket: hugegraph-store
+ *       region: us-east-1
+ *       access-key: AKIAIOSFODNN7EXAMPLE
+ *       secret-key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
+ *
+ */ +@Data +public class CloudStorageConfig { + + /** + * Whether cloud storage is enabled. Defaults to false. + */ + private boolean enabled = false; + + /** + * Name of the provider to activate. Must match {@link CloudStorageProvider#providerName()}. + * The provider JAR must be on the classpath. Defaults to "s3". + */ + private String provider = "s3"; + + /** + * Key prefix prepended to every object stored in the bucket. + * Defaults to "hugegraph". + */ + private String pathPrefix = "hugegraph"; + + /** + * Whether startup pre-hydration (cloud -> local before DB open) is enabled. + */ + private boolean startupHydrationEnabled = true; + + /** + * Guard window in milliseconds for repeated read-miss hydration attempts on the + * same db/table pair. Values <= 0 disable the guard. + */ + private long readMissGuardWindowMs = 3000L; + + + /** + * Maximum number of whole-file upload retries after a first failure. Default is {@code 5}. + * Under the primary-durability model an SST that never reaches cloud is at risk on the + * ephemeral local disk, so whole-file retries are enabled by default. After all attempts are + * exhausted the task is moved to the DLQ for operational visibility / replay. + * + *

Set to {@code 0} to disable whole-file retries (failures go straight to the DLQ) when the + * provider already implements a sufficient internal retry strategy. + */ + private int uploadRetryMaxAttempts = 3; + + /** + * Delay before the first whole-file retry attempt, in milliseconds. + * Subsequent retries use exponential back-off up to {@link #uploadRetryMaxDelayMs}. + * Only used when {@link #uploadRetryMaxAttempts} > 0. Default: 1 000 ms. + */ + private long uploadRetryInitialDelayMs = 1_000L; + + /** + * Upper bound for the exponential back-off delay for whole-file retries, in milliseconds. + * Only used when {@link #uploadRetryMaxAttempts} > 0. Default: 60 000 ms. + */ + private long uploadRetryMaxDelayMs = 60_000L; + + /** + * Backpressure high-watermark on the pending-upload backlog (retry-queue in-flight + DLQ). + * When {@code > 0} and the backlog exceeds this value, RocksDB's flush/compaction thread is + * briefly slowed in {@code onTableFileCreated} so ingestion cannot outrun the cloud mirror, + * bounding the amount of local-only (at-risk) data. Default: {@code 64}. {@code 0} disables it. + */ + private int uploadBackpressureHighWatermark = 64; + + + /** + * WAL durability mode for metadata mirroring: + *

    + *
  • {@code flush} (default): a flush is forced before each metadata capture so the durable + * state is fully in SST files; at most the un-flushed in-memory tail written since the + * last sync is lost on an uncontrolled crash.
    • + *
  • {@code wal}: the active WAL {@code *.log} segments are mirrored alongside the metadata + * and replayed on restore (lower RPO, at the cost of more frequent small uploads).
    • + *
+ */ + private String walMode = "flush"; + + + /** + * Provider-specific properties, mapped from {@code cloud.storage..*} in the configuration. + * These are passed verbatim to the provider implementation and may include credentials, + */ + private Map providerProperties = new HashMap<>(); +} diff --git a/hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageNonRetryableException.java b/hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageNonRetryableException.java new file mode 100644 index 0000000000..91859418e2 --- /dev/null +++ b/hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageNonRetryableException.java @@ -0,0 +1,34 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.cloud; + +import java.io.IOException; + +/** + * Marker exception indicating a cloud upload failure should not be retried at + * the whole-SST level and can be moved directly to DLQ. + */ +public class CloudStorageNonRetryableException extends IOException { + + private static final long serialVersionUID = 1L; + + public CloudStorageNonRetryableException(String message, Throwable cause) { + super(message, cause); + } +} + diff --git a/hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageProvider.java b/hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageProvider.java new file mode 100644 index 0000000000..4314466f3b --- /dev/null +++ b/hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageProvider.java @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.cloud; + +import java.io.Closeable; +import java.io.IOException; +import java.util.Collections; +import java.util.List; + +/** + * SPI interface for pluggable cloud storage backends. + * + *

Implementations are discovered via {@link java.util.ServiceLoader}: + * each provider JAR must contain + * {@code META-INF/services/org.apache.hugegraph.store.cloud.CloudStorageProvider} + * listing the fully-qualified implementation class. + * + *

The active provider is selected at runtime through + * {@link CloudStorageConfig#getProvider()} and initialized once by + * {@link CloudStorageProviderFactory#initialize(CloudStorageConfig)}. + */ +public interface CloudStorageProvider extends Closeable { + + /** + * Returns the unique, lower-case name of this provider (e.g. {@code "s3"}, {@code "gcs"}). + * Must match the value set in {@link CloudStorageConfig#getProvider()}. + */ + String providerName(); + + /** + * Initializes the provider with the supplied configuration. + * Called once before any upload/download/delete operations. + * + * @param config cloud storage configuration + */ + void init(CloudStorageConfig config); + + /** + * Uploads a local file to cloud storage. + * + * @param localPath absolute path of the local source file + * @param remoteKey destination key / path inside the bucket (without prefix) + * @throws IOException on I/O or network failure + */ + void uploadFile(String localPath, String remoteKey) throws IOException; + + /** + * Deletes an object from cloud storage. + * + * @param remoteKey key / path of the object to delete (without prefix) + * @throws IOException on I/O or network failure + */ + void deleteFile(String remoteKey) throws IOException; + + /** + * Checks whether an object exists in cloud storage. + * + * @param remoteKey key / path to check (without prefix) + * @return {@code true} if the object exists + * @throws IOException on I/O or network failure + */ + boolean fileExists(String remoteKey) throws IOException; + + /** + * Lists object keys under a remote directory/prefix. + * + *

Default implementation returns an empty list so existing providers remain compatible. + * + * @param remoteDirPrefix directory/prefix inside bucket (without provider pathPrefix) + * @return object keys relative to provider root (without provider pathPrefix) + * @throws IOException on I/O or network failure + */ + default List listFiles(String remoteDirPrefix) throws IOException { + return Collections.emptyList(); + } + + /** + * Downloads an object from cloud storage to a local file. + * + * @param remoteKey destination key / path inside the bucket (without prefix) + * @param localPath absolute path of the local destination file + * @throws IOException on I/O or network failure + */ + void downloadFile(String remoteKey, String localPath) throws IOException; + + /** + * Deletes all objects under a prefix in a single best-effort operation. + * + *

Default implementation lists and deletes individually for backward compatibility. + * Implementations should override to provide more efficient bulk-delete semantics + * where supported by the underlying storage backend (e.g. S3 DeleteObjects API). + * + * @param remoteDirPrefix directory/prefix inside bucket (without provider pathPrefix) + * @return number of objects deleted + * @throws IOException on I/O or network failure + */ + default int deletePrefix(String remoteDirPrefix) throws IOException { + List keys = listFiles(remoteDirPrefix); + for (String key : keys) { + try { + deleteFile(key); + } catch (IOException e) { + // Best-effort: continue deleting remaining files + } + } + return keys.size(); + } + + /** + * Releases resources held by the provider (e.g. HTTP clients, connections). + */ + @Override + void close() throws IOException; +} + diff --git a/hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageProviderFactory.java b/hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageProviderFactory.java new file mode 100644 index 0000000000..a6e63b06a9 --- /dev/null +++ b/hugegraph-store/hg-store-common/src/main/java/org/apache/hugegraph/store/cloud/CloudStorageProviderFactory.java @@ -0,0 +1,179 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.cloud; + +import java.io.IOException; +import java.util.Map; +import java.util.ServiceLoader; +import java.util.concurrent.ConcurrentHashMap; + +import lombok.Getter; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Factory for {@link CloudStorageProvider} instances. + * + *

Providers are discovered at class-loading time via {@link ServiceLoader}. + * Any JAR that includes + * {@code META-INF/services/org.apache.hugegraph.store.cloud.CloudStorageProvider} + * is automatically picked up when it is present on the classpath. + * + *

Usage: + * 

+ *   CloudStorageConfig cfg = ...; // populated from application.yml
+ *   CloudStorageProvider provider = CloudStorageProviderFactory.initialize(cfg);
+ *   // later:
+ *   CloudStorageProvider active = CloudStorageProviderFactory.getActiveProvider();
+ *
+ */ +public final class CloudStorageProviderFactory { + + private static final Logger log = LoggerFactory.getLogger(CloudStorageProviderFactory.class); + + /** All discovered providers keyed by {@link CloudStorageProvider#providerName()}. */ + private static final Map REGISTRY = new ConcurrentHashMap<>(); + + /** The currently active (initialized) provider; null when disabled or not yet initialized. + * -- GETTER -- + * Returns the currently active provider, or + * if cloud storage is + * disabled or + * has not yet been called. + */ + @Getter + private static volatile CloudStorageProvider activeProvider; + + static { + loadProviders(); + } + + private CloudStorageProviderFactory() { + } + + // ----------------------------------------------------------------------- + // Public API + // ----------------------------------------------------------------------- + + /** + * Initializes and activates the cloud storage provider described by {@code config}. + * + *

The method is idempotent: if called multiple times, the existing active + * provider is closed before a new one is initialized. + * + * @param config cloud storage configuration + * @return the initialized provider, or {@code null} when + * {@link CloudStorageConfig#isEnabled()} is {@code false} + * @throws IllegalArgumentException if no provider matching {@code config.getProvider()} + * was found on the classpath + */ + public static synchronized CloudStorageProvider initialize(CloudStorageConfig config) { + if (!config.isEnabled()) { + log.info("Cloud storage is disabled (cloud.storage.enabled=false)"); + return null; + } + + String name = config.getProvider(); + CloudStorageProvider provider = REGISTRY.get(name); + if (provider == null) { + throw new IllegalArgumentException( + "No cloud storage provider found for name '" + name + "'. " + + "Available providers: " + REGISTRY.keySet() + ". " + + "Make sure the provider JAR (e.g. hg-store-cloud-s3) is on the classpath."); + } + + // Close any previously active provider + if (activeProvider != null && activeProvider != provider) { + try { + activeProvider.close(); + } catch (IOException e) { + log.warn("Error closing previous cloud storage provider", e); + } + } + + provider.init(config); + activeProvider = provider; + log.info("Cloud storage provider '{}' initialized. bucket={}", + name, config.getProviderProperties().getOrDefault("bucket", "N/A")); + return provider; + } + + /** + * Shuts down and deregisters the active provider. + * Subsequent calls to {@link #getActiveProvider()} return {@code null}. + */ + public static synchronized void shutdown() { + if (activeProvider != null) { + try { + activeProvider.close(); + log.info("Cloud storage provider '{}' closed", activeProvider.providerName()); + } catch (IOException e) { + log.warn("Error closing cloud storage provider", e); + } + activeProvider = null; + } + } + + /** + * Resets the active provider to {@code null} without closing it. + * + *

For testing only. Use {@link #shutdown()} in production code. + */ + public static synchronized void reset() { + activeProvider = null; + } + + /** + * Directly injects an active provider, bypassing SPI discovery and {@link #initialize}. + * + *

For testing only. Allows tests to supply a stub/mock provider without + * placing a real JAR on the classpath. + * + * @param provider the provider instance to activate (may be {@code null} to clear) + */ + public static synchronized void setActiveProviderForTest(CloudStorageProvider provider) { + activeProvider = provider; + } + + // ----------------------------------------------------------------------- + // Internal helpers + // ----------------------------------------------------------------------- + + /** Scans the classpath for {@link CloudStorageProvider} implementations via SPI. */ + private static void loadProviders() { + ServiceLoader loader = + ServiceLoader.load(CloudStorageProvider.class, + CloudStorageProviderFactory.class.getClassLoader()); + for (CloudStorageProvider p : loader) { + String name = p.providerName(); + if (REGISTRY.containsKey(name)) { + log.warn("Duplicate cloud storage provider name '{}' – keeping first registration", + name); + } else { + REGISTRY.put(name, p); + log.info("Discovered cloud storage provider: '{}' ({})", + name, p.getClass().getName()); + } + } + if (REGISTRY.isEmpty()) { + log.debug("No cloud storage providers found on classpath; cloud storage is unavailable"); + } + } +} + diff --git a/hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/CloudStorageConfigTest.java b/hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/CloudStorageConfigTest.java new file mode 100644 index 0000000000..e1c9f09593 --- /dev/null +++ b/hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/CloudStorageConfigTest.java @@ -0,0 +1,289 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.cloud; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +import java.util.HashMap; +import java.util.Map; + +import org.junit.Before; +import org.junit.Test; + +public class CloudStorageConfigTest { + + private CloudStorageConfig config; + + @Before + public void setUp() { + config = new CloudStorageConfig(); + } + + // ---- Common fields ---- + + @Test + public void testDefaults() { + assertFalse(config.isEnabled()); + assertEquals("s3", config.getProvider()); + assertEquals("hugegraph", config.getPathPrefix()); + assertTrue(config.isStartupHydrationEnabled()); + assertEquals(3000L, config.getReadMissGuardWindowMs()); + assertNotNull(config.getProviderProperties()); + } + + @Test + public void testEnabledFlag() { + assertFalse(config.isEnabled()); + + config.setEnabled(true); + assertTrue(config.isEnabled()); + + config.setEnabled(false); + assertFalse(config.isEnabled()); + } + + @Test + public void testProvider() { + assertEquals("s3", config.getProvider()); + config.setProvider("gcs"); + assertEquals("gcs", config.getProvider()); + config.setProvider("adls"); + assertEquals("adls", config.getProvider()); + } + + @Test + public void testPathPrefix() { + assertEquals("hugegraph", config.getPathPrefix()); + + config.setPathPrefix("myapp/data"); + assertEquals("myapp/data", config.getPathPrefix()); + } + + @Test + public void testStartupHydrationEnabled() { + assertTrue(config.isStartupHydrationEnabled()); + + config.setStartupHydrationEnabled(false); + assertFalse(config.isStartupHydrationEnabled()); + + config.setStartupHydrationEnabled(true); + assertTrue(config.isStartupHydrationEnabled()); + } + + @Test + public void testReadMissGuardWindowMs() { + assertEquals(3000L, config.getReadMissGuardWindowMs()); + + config.setReadMissGuardWindowMs(5000L); + assertEquals(5000L, config.getReadMissGuardWindowMs()); + + config.setReadMissGuardWindowMs(0L); + assertEquals(0L, config.getReadMissGuardWindowMs()); + + config.setReadMissGuardWindowMs(-1L); + assertEquals(-1L, config.getReadMissGuardWindowMs()); + } + + @Test + public void testUploadRetryDefaults() { + // Default 3: whole-file retries enabled under the primary-durability model. + assertEquals(3, config.getUploadRetryMaxAttempts()); + assertEquals(1_000L, config.getUploadRetryInitialDelayMs()); + assertEquals(60_000L, config.getUploadRetryMaxDelayMs()); + // Backpressure enabled by default. + assertEquals(64, config.getUploadBackpressureHighWatermark()); + } + + @Test + public void testMetadataSyncDefaults() { + // Metadata mirroring defaults to flush mode. + assertEquals("flush", config.getWalMode()); + } + + @Test + public void testMetadataSyncSetters() { + + config.setWalMode("wal"); + assertEquals("wal", config.getWalMode()); + } + + @Test + public void testUploadRetryMaxAttempts() { + config.setUploadRetryMaxAttempts(3); + assertEquals(3, config.getUploadRetryMaxAttempts()); + + config.setUploadRetryMaxAttempts(0); + assertEquals(0, config.getUploadRetryMaxAttempts()); + + config.setUploadRetryMaxAttempts(10); + assertEquals(10, config.getUploadRetryMaxAttempts()); + } + + @Test + public void testUploadRetryInitialDelayMs() { + config.setUploadRetryInitialDelayMs(500L); + assertEquals(500L, config.getUploadRetryInitialDelayMs()); + + config.setUploadRetryInitialDelayMs(2_000L); + assertEquals(2_000L, config.getUploadRetryInitialDelayMs()); + } + + @Test + public void testUploadRetryMaxDelayMs() { + config.setUploadRetryMaxDelayMs(30_000L); + assertEquals(30_000L, config.getUploadRetryMaxDelayMs()); + + config.setUploadRetryMaxDelayMs(120_000L); + assertEquals(120_000L, config.getUploadRetryMaxDelayMs()); + } + + @Test + public void testUploadBackpressureHighWatermark() { + config.setUploadBackpressureHighWatermark(128); + assertEquals(128, config.getUploadBackpressureHighWatermark()); + + config.setUploadBackpressureHighWatermark(0); + assertEquals(0, config.getUploadBackpressureHighWatermark()); + } + + // ---- Provider properties (cloud.storage..* flattened) ---- + + @Test + public void testProviderPropertiesDefaults() { + assertNotNull(config.getProviderProperties()); + assertTrue(config.getProviderProperties().isEmpty()); + } + + @Test + public void testProviderPropertiesSetAndGet() { + Map props = new HashMap<>(); + props.put("bucket", "my-bucket"); + props.put("region", "us-east-1"); + props.put("endpoint", "https://s3.amazonaws.com"); + props.put("access-key", "AKIAIOSFODNN7EXAMPLE"); + props.put("secret-key", "secret"); + props.put("multipart-part-retry-max-attempts", "7"); + + config.setProviderProperties(props); + + assertEquals("my-bucket", config.getProviderProperties().get("bucket")); + assertEquals("us-east-1", config.getProviderProperties().get("region")); + assertEquals("7", config.getProviderProperties() + .get("multipart-part-retry-max-attempts")); + } + + @Test + public void testCompleteConfiguration() { + config.setEnabled(true); + config.setProvider("s3"); + config.setPathPrefix("production/data"); + config.setStartupHydrationEnabled(true); + config.setReadMissGuardWindowMs(5000L); + config.setUploadRetryMaxAttempts(3); + config.setUploadRetryInitialDelayMs(500L); + config.setUploadRetryMaxDelayMs(30_000L); + + Map providerProps = new HashMap<>(); + providerProps.put("bucket", "hugegraph-backup"); + providerProps.put("region", "us-west-2"); + providerProps.put("endpoint", "https://s3-us-west-2.amazonaws.com"); + providerProps.put("access-key", "test-access-key"); + providerProps.put("secret-key", "test-secret-key"); + providerProps.put("multipart-part-retry-max-attempts", "5"); + providerProps.put("multipart-part-retry-base-backoff-ms", "1500"); + providerProps.put("multipart-exhausted-direct-dlq", "false"); + config.setProviderProperties(providerProps); + + // Common fields + assertTrue(config.isEnabled()); + assertEquals("s3", config.getProvider()); + assertEquals("production/data", config.getPathPrefix()); + assertTrue(config.isStartupHydrationEnabled()); + assertEquals(5000L, config.getReadMissGuardWindowMs()); + assertEquals(3, config.getUploadRetryMaxAttempts()); + assertEquals(500L, config.getUploadRetryInitialDelayMs()); + assertEquals(30_000L, config.getUploadRetryMaxDelayMs()); + + // Provider-specific fields (cloud.storage..*) + assertEquals("hugegraph-backup", config.getProviderProperties().get("bucket")); + assertEquals("us-west-2", config.getProviderProperties().get("region")); + assertEquals("https://s3-us-west-2.amazonaws.com", + config.getProviderProperties().get("endpoint")); + assertEquals("test-access-key", config.getProviderProperties().get("access-key")); + assertEquals("test-secret-key", config.getProviderProperties().get("secret-key")); + assertEquals("5", config.getProviderProperties().get("multipart-part-retry-max-attempts")); + assertEquals("1500", + config.getProviderProperties().get("multipart-part-retry-base-backoff-ms")); + assertEquals("false", config.getProviderProperties().get("multipart-exhausted-direct-dlq")); + } + + @Test + public void testMultiplePropertyUpdates() { + Map props1 = new HashMap<>(); + props1.put("key1", "value1"); + config.setProviderProperties(props1); + assertEquals("value1", config.getProviderProperties().get("key1")); + + Map props2 = new HashMap<>(); + props2.put("key2", "value2"); + config.setProviderProperties(props2); + assertEquals("value2", config.getProviderProperties().get("key2")); + } + + @Test + public void testWalModeDefaults() { + assertEquals("flush", config.getWalMode()); + } + + @Test + public void testWalModeTransition() { + config.setWalMode("wal"); + assertEquals("wal", config.getWalMode()); + + config.setWalMode("flush"); + assertEquals("flush", config.getWalMode()); + } + + @Test + public void testBackpressureDisabled() { + config.setUploadBackpressureHighWatermark(0); + assertEquals(0, config.getUploadBackpressureHighWatermark()); + } + + @Test + public void testNegativeReadMissGuardWindow() { + config.setReadMissGuardWindowMs(-5000L); + assertEquals(-5000L, config.getReadMissGuardWindowMs()); + } + + @Test + public void testProviderPropertiesImmutabilityBehavior() { + Map props = new HashMap<>(); + props.put("bucket", "original"); + config.setProviderProperties(props); + + // Modify the original map + props.put("bucket", "modified"); + + // The config should have the modified value since it references the same map + assertEquals("modified", config.getProviderProperties().get("bucket")); + } +} diff --git a/hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/CloudStorageNonRetryableExceptionTest.java b/hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/CloudStorageNonRetryableExceptionTest.java new file mode 100644 index 0000000000..d1e3b24830 --- /dev/null +++ b/hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/CloudStorageNonRetryableExceptionTest.java @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.cloud; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertSame; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; + +import org.junit.Test; + +public class CloudStorageNonRetryableExceptionTest { + + @Test + public void testConstructorPreservesMessageAndCause() { + IOException cause = new IOException("root cause"); + + CloudStorageNonRetryableException exception = + new CloudStorageNonRetryableException("non-retryable", cause); + + assertEquals("non-retryable", exception.getMessage()); + assertSame(cause, exception.getCause()); + } + + @Test + public void testDirectSuperclassIsIOException() { + assertSame(IOException.class, CloudStorageNonRetryableException.class.getSuperclass()); + } + + @Test + public void testSerialVersionUID() { + try { + java.lang.reflect.Field field = + CloudStorageNonRetryableException.class.getDeclaredField("serialVersionUID"); + field.setAccessible(true); + Long serialVersionUID = (Long) field.get(null); + assertEquals(Long.valueOf(1L), serialVersionUID); + } catch (NoSuchFieldException | IllegalAccessException e) { + // serialVersionUID field exists as expected + } + } + + @Test + public void testConstructorWithNullMessage() { + IOException cause = new IOException("cause"); + + CloudStorageNonRetryableException exception = + new CloudStorageNonRetryableException(null, cause); + + assertNull(exception.getMessage()); + assertSame(cause, exception.getCause()); + } + + @Test + public void testConstructorWithNullCause() { + CloudStorageNonRetryableException exception = + new CloudStorageNonRetryableException("message", null); + + assertEquals("message", exception.getMessage()); + assertNull(exception.getCause()); + } + + @Test + public void testConstructorWithBothNull() { + CloudStorageNonRetryableException exception = + new CloudStorageNonRetryableException(null, null); + + assertNull(exception.getMessage()); + assertNull(exception.getCause()); + } + + @Test + public void testExceptionThrowableAndCatch() { + IOException cause = new IOException("original cause"); + CloudStorageNonRetryableException exception = + new CloudStorageNonRetryableException("cloud error", cause); + + try { + throw exception; + } catch (IOException e) { + assertEquals("cloud error", e.getMessage()); + assertSame(cause, e.getCause()); + } + } + + @Test + public void testExceptionStackTrace() { + IOException cause = new IOException("root cause"); + CloudStorageNonRetryableException exception = + new CloudStorageNonRetryableException("non-retryable", cause); + + assertTrue(exception.toString().contains("CloudStorageNonRetryableException")); + assertTrue(exception.toString().contains("non-retryable")); + } +} diff --git a/hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/CloudStorageProviderFactoryTest.java b/hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/CloudStorageProviderFactoryTest.java new file mode 100644 index 0000000000..ff7fdfa481 --- /dev/null +++ b/hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/CloudStorageProviderFactoryTest.java @@ -0,0 +1,441 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.cloud; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertSame; +import static org.junit.Assert.fail; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import java.io.IOException; +import java.lang.reflect.Field; +import java.lang.reflect.Method; +import java.util.HashMap; +import java.util.Map; +import java.util.ServiceLoader; + +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +public class CloudStorageProviderFactoryTest { + + private CloudStorageConfig config; + private CloudStorageProvider mockProvider; + private Map registryBackup; + + @Before + public void setUp() { + config = new CloudStorageConfig(); + mockProvider = mock(CloudStorageProvider.class); + when(mockProvider.providerName()).thenReturn("s3"); + + // Keep each test isolated from static registry state. + registryBackup = new HashMap<>(registry()); + registry().clear(); + } + + @After + public void tearDown() { + // Reset static mutable state to prevent test interference. + CloudStorageProviderFactory.reset(); + registry().clear(); + registry().putAll(registryBackup); + } + + @SuppressWarnings("unchecked") + private static Map registry() { + try { + Field field = CloudStorageProviderFactory.class.getDeclaredField("REGISTRY"); + field.setAccessible(true); + return (Map) field.get(null); + } catch (ReflectiveOperationException e) { + throw new AssertionError("Unable to access CloudStorageProviderFactory.REGISTRY", e); + } + } + + private static void invokeLoadProviders() { + try { + Method method = CloudStorageProviderFactory.class.getDeclaredMethod("loadProviders"); + method.setAccessible(true); + method.invoke(null); + } catch (ReflectiveOperationException e) { + throw new AssertionError("Unable to invoke CloudStorageProviderFactory.loadProviders", e); + } + } + + /** + * Test initialize with disabled cloud storage returns null + */ + @Test + public void testInitializeDisabled() { + config.setEnabled(false); + + CloudStorageProvider result = CloudStorageProviderFactory.initialize(config); + + assertNull(result); + assertNull(CloudStorageProviderFactory.getActiveProvider()); + } + + /** + * Test that setActiveProviderForTest allows setting a provider + */ + @Test + public void testInitializeWithMockProvider() { + config.setEnabled(true); + config.setProvider("s3"); + config.getProviderProperties().put("bucket", "test-bucket"); + + // Inject the mock provider for testing (bypass SPI discovery) + CloudStorageProviderFactory.setActiveProviderForTest(mockProvider); + + // Verify it's set + assertEquals(mockProvider, CloudStorageProviderFactory.getActiveProvider()); + } + + /** + * Test that multiple setActiveProviderForTest calls work + */ + @Test + public void testMultipleSetActiveProvider() { + CloudStorageProvider oldProvider = mock(CloudStorageProvider.class); + when(oldProvider.providerName()).thenReturn("s3"); + + // Set first provider + CloudStorageProviderFactory.setActiveProviderForTest(oldProvider); + assertEquals(oldProvider, CloudStorageProviderFactory.getActiveProvider()); + + // Switch to new provider + CloudStorageProviderFactory.setActiveProviderForTest(mockProvider); + assertEquals(mockProvider, CloudStorageProviderFactory.getActiveProvider()); + } + + /** + * Test that initialize with unknown provider (no SPI) throws IllegalArgumentException + */ + @Test + public void testInitializeUnknownProviderNoSpi() { + config.setEnabled(true); + config.setProvider("unknown-provider"); + + try { + CloudStorageProviderFactory.initialize(config); + fail("Should have thrown IllegalArgumentException"); + } catch (IllegalArgumentException e) { + assertNotNull(e.getMessage()); + // Expected - no providers are available via SPI in tests + } + } + + /** + * Test shutdown closes the active provider + */ + @Test + public void testShutdown() throws IOException { + // Inject mock provider without calling initialize + CloudStorageProviderFactory.setActiveProviderForTest(mockProvider); + + assertNotNull(CloudStorageProviderFactory.getActiveProvider()); + + CloudStorageProviderFactory.shutdown(); + + assertNull(CloudStorageProviderFactory.getActiveProvider()); + verify(mockProvider, times(1)).close(); + } + + /** + * Test reset clears active provider without closing it + */ + @Test + public void testReset() throws IOException { + CloudStorageProviderFactory.setActiveProviderForTest(mockProvider); + + assertNotNull(CloudStorageProviderFactory.getActiveProvider()); + + CloudStorageProviderFactory.reset(); + + assertNull(CloudStorageProviderFactory.getActiveProvider()); + // close should NOT be called + verify(mockProvider, times(0)).close(); + } + + /** + * Test shutdown handles IOException gracefully + */ + @Test + public void testShutdownWithException() throws IOException { + // Inject mock provider without calling initialize + doThrow(new IOException("Mock close error")).when(mockProvider).close(); + CloudStorageProviderFactory.setActiveProviderForTest(mockProvider); + + // Should not throw exception, error is logged + CloudStorageProviderFactory.shutdown(); + + assertNull(CloudStorageProviderFactory.getActiveProvider()); + } + + /** + * Test setActiveProviderForTest allows injection + */ + @Test + public void testSetActiveProviderForTest() { + assertNull(CloudStorageProviderFactory.getActiveProvider()); + + CloudStorageProviderFactory.setActiveProviderForTest(mockProvider); + + assertEquals(mockProvider, CloudStorageProviderFactory.getActiveProvider()); + + CloudStorageProviderFactory.setActiveProviderForTest(null); + + assertNull(CloudStorageProviderFactory.getActiveProvider()); + } + + /** + * Test that initialize is idempotent (calling multiple times should work) + */ + @Test + public void testInitializeIdempotent() { + // Test the disabled case which doesn't require SPI providers + config.setEnabled(false); + + CloudStorageProvider result1 = CloudStorageProviderFactory.initialize(config); + CloudStorageProvider result2 = CloudStorageProviderFactory.initialize(config); + + assertNull(result1); + assertNull(result2); + } + + /** + * Test that initialize closes previous provider when switching providers + */ + @Test + public void testInitializeClosesPreviousProvider() { + // Set up first provider + CloudStorageProvider firstProvider = mock(CloudStorageProvider.class); + when(firstProvider.providerName()).thenReturn("s3"); + CloudStorageProviderFactory.setActiveProviderForTest(firstProvider); + + // Set up second provider to replace it + CloudStorageProvider secondProvider = mock(CloudStorageProvider.class); + when(secondProvider.providerName()).thenReturn("gcs"); + + config.setEnabled(true); + config.setProvider("gcs"); + + // Inject second provider for this test + CloudStorageProviderFactory.setActiveProviderForTest(secondProvider); + + // Verify we can close without exception + CloudStorageProviderFactory.shutdown(); + assertNull(CloudStorageProviderFactory.getActiveProvider()); + } + + /** + * Test that initialize with close exception on previous provider handles error + */ + @Test + public void testInitializeClosePreviousProviderThrows() throws IOException { + // Set up first provider that throws when closing + CloudStorageProvider firstProvider = mock(CloudStorageProvider.class); + when(firstProvider.providerName()).thenReturn("s3"); + doThrow(new IOException("Mock error")).when(firstProvider).close(); + CloudStorageProviderFactory.setActiveProviderForTest(firstProvider); + + // Should not throw, error is logged + CloudStorageProviderFactory.shutdown(); + assertNull(CloudStorageProviderFactory.getActiveProvider()); + } + + /** + * Test shutdown when no provider is active + */ + @Test + public void testShutdownNoActiveProvider() { + CloudStorageProviderFactory.reset(); + + // Should not throw exception + CloudStorageProviderFactory.shutdown(); + + assertNull(CloudStorageProviderFactory.getActiveProvider()); + } + + @Test + public void testInitializeUsesRegistryProviderAndSetsActive() { + registry().put("s3", mockProvider); + config.setEnabled(true); + config.setProvider("s3"); + config.getProviderProperties().put("bucket", "unit-bucket"); + + CloudStorageProvider result = CloudStorageProviderFactory.initialize(config); + + assertSame(mockProvider, result); + assertSame(mockProvider, CloudStorageProviderFactory.getActiveProvider()); + verify(mockProvider, times(1)).init(config); + } + + @Test + public void testInitializeWithSameActiveProviderDoesNotCloseIt() throws IOException { + registry().put("s3", mockProvider); + CloudStorageProviderFactory.setActiveProviderForTest(mockProvider); + config.setEnabled(true); + config.setProvider("s3"); + + CloudStorageProviderFactory.initialize(config); + + verify(mockProvider, never()).close(); + verify(mockProvider, times(1)).init(config); + } + + @Test + public void testInitializeSwitchProviderClosesPreviousProvider() throws IOException { + CloudStorageProvider oldProvider = mock(CloudStorageProvider.class); + when(oldProvider.providerName()).thenReturn("old"); + + CloudStorageProvider newProvider = mock(CloudStorageProvider.class); + when(newProvider.providerName()).thenReturn("gcs"); + + registry().put("gcs", newProvider); + CloudStorageProviderFactory.setActiveProviderForTest(oldProvider); + + config.setEnabled(true); + config.setProvider("gcs"); + + CloudStorageProvider result = CloudStorageProviderFactory.initialize(config); + + assertSame(newProvider, result); + assertSame(newProvider, CloudStorageProviderFactory.getActiveProvider()); + verify(oldProvider, times(1)).close(); + verify(newProvider, times(1)).init(config); + } + + @Test + public void testInitializeContinuesWhenClosingPreviousProviderFails() throws IOException { + CloudStorageProvider oldProvider = mock(CloudStorageProvider.class); + when(oldProvider.providerName()).thenReturn("old"); + doThrow(new IOException("close failed")).when(oldProvider).close(); + + CloudStorageProvider newProvider = mock(CloudStorageProvider.class); + when(newProvider.providerName()).thenReturn("gcs"); + + registry().put("gcs", newProvider); + CloudStorageProviderFactory.setActiveProviderForTest(oldProvider); + + config.setEnabled(true); + config.setProvider("gcs"); + + CloudStorageProvider result = CloudStorageProviderFactory.initialize(config); + + assertSame(newProvider, result); + assertSame(newProvider, CloudStorageProviderFactory.getActiveProvider()); + verify(oldProvider, times(1)).close(); + verify(newProvider, times(1)).init(config); + } + + @Test + public void testLoadProvidersKeepsFirstRegistrationOnDuplicateName() { + CloudStorageProvider firstRegistration = mock(CloudStorageProvider.class); + when(firstRegistration.providerName()) + .thenReturn(TestDuplicateNamedProvider.PROVIDER_NAME); + + boolean duplicateProviderDiscoverable = + ServiceLoader.load(CloudStorageProvider.class, + CloudStorageProviderFactory.class.getClassLoader()) + .stream() + .map(ServiceLoader.Provider::get) + .anyMatch(p -> TestDuplicateNamedProvider.PROVIDER_NAME + .equals(p.providerName())); + if (!duplicateProviderDiscoverable) { + fail("Test duplicate provider is not discoverable via ServiceLoader"); + } + + registry().put(TestDuplicateNamedProvider.PROVIDER_NAME, firstRegistration); + + invokeLoadProviders(); + + assertSame(firstRegistration, + registry().get(TestDuplicateNamedProvider.PROVIDER_NAME)); + } + + @SuppressWarnings("DataFlowIssue") + @Test + public void testInitializeWithNullConfigThrowsException() { + try { + CloudStorageProviderFactory.initialize(null); + fail("Should have thrown exception"); + } catch (Exception e) { + // Expected - null config should cause an error + } + } + + @Test + public void testShutdownMultipleTimes() { + CloudStorageProviderFactory.setActiveProviderForTest(mockProvider); + + CloudStorageProviderFactory.shutdown(); + CloudStorageProviderFactory.shutdown(); // Should not throw + + assertNull(CloudStorageProviderFactory.getActiveProvider()); + } + + @Test + public void testResetMultipleTimes() { + CloudStorageProviderFactory.setActiveProviderForTest(mockProvider); + + CloudStorageProviderFactory.reset(); + CloudStorageProviderFactory.reset(); // Should not throw + + assertNull(CloudStorageProviderFactory.getActiveProvider()); + } + + @Test + public void testSetActiveProviderForTestWithNull() { + CloudStorageProviderFactory.setActiveProviderForTest(mockProvider); + assertEquals(mockProvider, CloudStorageProviderFactory.getActiveProvider()); + + CloudStorageProviderFactory.setActiveProviderForTest(null); + assertNull(CloudStorageProviderFactory.getActiveProvider()); + } + + @Test + public void testInitializeLogsWhenDisabled() { + config.setEnabled(false); + + CloudStorageProvider result = CloudStorageProviderFactory.initialize(config); + + assertNull(result); + } + + @Test + public void testProviderInitializationCalledWithCorrectConfig() { + registry().put("s3", mockProvider); + config.setEnabled(true); + config.setProvider("s3"); + config.getProviderProperties().put("bucket", "test-bucket"); + + CloudStorageProviderFactory.initialize(config); + + verify(mockProvider, times(1)).init(config); + } + } diff --git a/hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/CloudStorageProviderTest.java b/hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/CloudStorageProviderTest.java new file mode 100644 index 0000000000..c7346365e1 --- /dev/null +++ b/hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/CloudStorageProviderTest.java @@ -0,0 +1,353 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.cloud; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.util.List; + +import org.junit.Test; + +/** + * Tests for CloudStorageProvider interface default implementations. + */ +@SuppressWarnings("resource") +public class CloudStorageProviderTest { + + /** + * Test that the default listFiles() implementation returns an empty list + */ + @Test + public void testDefaultListFilesReturnsEmptyList() throws IOException { + // Create a minimal implementation that uses default listFiles + CloudStorageProvider provider = new CloudStorageProvider() { + @Override + public String providerName() { + return "test"; + } + + @Override + public void init(CloudStorageConfig config) { + // No-op + } + + @Override + public void uploadFile(String localPath, String remoteKey) { + // No-op + } + + @Override + public void deleteFile(String remoteKey) { + // No-op + } + + @Override + public boolean fileExists(String remoteKey) { + return false; + } + + @Override + public void downloadFile(String remoteKey, String localPath) { + // No-op + } + + @Override + public void close() { + // No-op + } + }; + + // Call the default listFiles method + List result = provider.listFiles("some/prefix"); + + // Verify it returns an empty list (the default implementation) + assertNotNull(result); + assertTrue(result.isEmpty()); + } + + /** + * Test that listFiles default implementation can be overridden + */ + @Test + public void testListFilesCanBeOverridden() throws IOException { + CloudStorageProvider provider = new CloudStorageProvider() { + @Override + public String providerName() { + return "test"; + } + + @Override + public void init(CloudStorageConfig config) { + // No-op + } + + @Override + public void uploadFile(String localPath, String remoteKey) { + // No-op + } + + @Override + public void deleteFile(String remoteKey) { + // No-op + } + + @Override + public boolean fileExists(String remoteKey) { + return false; + } + + @Override + public List listFiles(String remoteDirPrefix) { + // Override with custom implementation + java.util.List result = new java.util.ArrayList<>(); + result.add("file1.sst"); + result.add("file2.sst"); + return result; + } + + @Override + public void downloadFile(String remoteKey, String localPath) { + // No-op + } + + @Override + public void close() { + // No-op + } + }; + + List result = provider.listFiles("some/prefix"); + + assertEquals(2, result.size()); + assertTrue(result.contains("file1.sst")); + assertTrue(result.contains("file2.sst")); + } + + @Test + public void testDefaultDeletePrefixDeletesAllListedKeys() throws IOException { + java.util.List deleted = new java.util.ArrayList<>(); + + CloudStorageProvider provider = new CloudStorageProvider() { + @Override + public String providerName() { + return "test"; + } + + @Override + public void init(CloudStorageConfig config) { + // No-op + } + + @Override + public void uploadFile(String localPath, String remoteKey) { + // No-op + } + + @Override + public void deleteFile(String remoteKey) { + deleted.add(remoteKey); + } + + @Override + public boolean fileExists(String remoteKey) { + return false; + } + + @Override + public List listFiles(String remoteDirPrefix) { + return java.util.Arrays.asList("a.sst", "b.sst", "c.sst"); + } + + @Override + public void downloadFile(String remoteKey, String localPath) { + // No-op + } + + @Override + public void close() { + // No-op + } + }; + + int deletedCount = provider.deletePrefix("some/prefix"); + + assertEquals(3, deletedCount); + assertEquals(3, deleted.size()); + assertTrue(deleted.contains("a.sst")); + assertTrue(deleted.contains("b.sst")); + assertTrue(deleted.contains("c.sst")); + } + + @Test + public void testDefaultDeletePrefixContinuesOnDeleteFailure() throws IOException { + java.util.List attempted = new java.util.ArrayList<>(); + + CloudStorageProvider provider = new CloudStorageProvider() { + @Override + public String providerName() { + return "test"; + } + + @Override + public void init(CloudStorageConfig config) { + // No-op + } + + @Override + public void uploadFile(String localPath, String remoteKey) { + // No-op + } + + @Override + public void deleteFile(String remoteKey) throws IOException { + attempted.add(remoteKey); + if ("b.sst".equals(remoteKey)) { + throw new IOException("simulated delete failure"); + } + } + + @Override + public boolean fileExists(String remoteKey) { + return false; + } + + @Override + public List listFiles(String remoteDirPrefix) { + return java.util.Arrays.asList("a.sst", "b.sst", "c.sst"); + } + + @Override + public void downloadFile(String remoteKey, String localPath) { + // No-op + } + + @Override + public void close() { + // No-op + } + }; + + int deletedCount = provider.deletePrefix("some/prefix"); + + // Default impl is best-effort and returns list size even if one delete fails. + assertEquals(3, deletedCount); + assertEquals(3, attempted.size()); + assertEquals("a.sst", attempted.get(0)); + assertEquals("b.sst", attempted.get(1)); + assertEquals("c.sst", attempted.get(2)); + } + + @Test + public void testDeletePrefixWithEmptyList() throws IOException { + CloudStorageProvider provider = new CloudStorageProvider() { + @Override + public String providerName() { + return "test"; + } + + @Override + public void init(CloudStorageConfig config) { + } + + @Override + public void uploadFile(String localPath, String remoteKey) { + } + + @Override + public void deleteFile(String remoteKey) { + } + + @Override + public boolean fileExists(String remoteKey) { + return false; + } + + @Override + public List listFiles(String remoteDirPrefix) { + return java.util.Collections.emptyList(); + } + + @Override + public void downloadFile(String remoteKey, String localPath) { + } + + @Override + public void close() { + } + }; + + int deletedCount = provider.deletePrefix("some/prefix"); + + assertEquals(0, deletedCount); + } + + @Test + public void testListFilesDefaultWithLargeDataset() throws IOException { + CloudStorageProvider provider = new CloudStorageProvider() { + @Override + public String providerName() { + return "test"; + } + + @Override + public void init(CloudStorageConfig config) { + } + + @Override + public void uploadFile(String localPath, String remoteKey) { + } + + @Override + public void deleteFile(String remoteKey) { + } + + @Override + public boolean fileExists(String remoteKey) { + return false; + } + + @Override + public List listFiles(String remoteDirPrefix) { + // Simulate large dataset + java.util.List files = new java.util.ArrayList<>(); + for (int i = 0; i < 1000; i++) { + files.add("file" + i + ".sst"); + } + return files; + } + + @Override + public void downloadFile(String remoteKey, String localPath) { + } + + @Override + public void close() { + } + }; + + List result = provider.listFiles("some/prefix"); + + assertEquals(1000, result.size()); + assertTrue(result.contains("file0.sst")); + assertTrue(result.contains("file999.sst")); + } +} + diff --git a/hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/TestDuplicateNamedProvider.java b/hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/TestDuplicateNamedProvider.java new file mode 100644 index 0000000000..1df7492f40 --- /dev/null +++ b/hugegraph-store/hg-store-common/src/test/java/org/apache/hugegraph/store/cloud/TestDuplicateNamedProvider.java @@ -0,0 +1,59 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.cloud; + +/** Test-only SPI provider used to exercise CloudStorageProviderFactory.loadProviders(). */ +public class TestDuplicateNamedProvider implements CloudStorageProvider { + + public static final String PROVIDER_NAME = "test-duplicate-provider"; + + @Override + public String providerName() { + return PROVIDER_NAME; + } + + @Override + public void init(CloudStorageConfig config) { + // no-op for test provider + } + + @Override + public void uploadFile(String localPath, String remoteKey) { + // no-op for test provider + } + + @Override + public void deleteFile(String remoteKey) { + // no-op for test provider + } + + @Override + public boolean fileExists(String remoteKey) { + return false; + } + + @Override + public void downloadFile(String remoteKey, String localPath) { + // no-op for test provider + } + + @Override + public void close() { + // no-op for test provider + } +} diff --git a/hugegraph-store/hg-store-common/src/test/resources/META-INF/services/org.apache.hugegraph.store.cloud.CloudStorageProvider b/hugegraph-store/hg-store-common/src/test/resources/META-INF/services/org.apache.hugegraph.store.cloud.CloudStorageProvider new file mode 100644 index 0000000000..49ac6bbca7 --- /dev/null +++ b/hugegraph-store/hg-store-common/src/test/resources/META-INF/services/org.apache.hugegraph.store.cloud.CloudStorageProvider @@ -0,0 +1,2 @@ +org.apache.hugegraph.store.cloud.TestDuplicateNamedProvider + diff --git a/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/business/BusinessHandlerImpl.java b/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/business/BusinessHandlerImpl.java index 9287bfe267..80a38f028e 100644 --- a/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/business/BusinessHandlerImpl.java +++ b/hugegraph-store/hg-store-core/src/main/java/org/apache/hugegraph/store/business/BusinessHandlerImpl.java @@ -1008,17 +1008,27 @@ public void batchGet(String graph, String table, SupplierFires {@code notifyTruncateBegin} before the key-range delete so that + * registered listeners can prepare for truncate and suppress racing callbacks, + * and {@code notifyTruncate} afterwards so that listeners can finalize any + * post-truncate cleanup — matching the same callback pair fired by + * {@link org.apache.hugegraph.rocksdb.access.RocksDBSession#truncate()}. */ @Override public void truncate(String graphName, int partId) throws HgStoreException { // Each partition corresponds to a rocksdb instance, so the rocksdb instance name is // rocksdb + partId try (RocksDBSession dbSession = getSession(graphName, partId)) { + String dbName = dbSession.getGraphName(); + String dbPath = dbSession.getDbPath(); + factory.notifyTruncateBegin(dbName, dbPath); dbSession.sessionOp().deleteRange(keyCreator.getStartKey(partId, graphName), keyCreator.getEndKey(partId, graphName)); // Release map ID keyCreator.delGraphId(partId, graphName); + factory.notifyTruncate(dbName, dbPath); } } diff --git a/hugegraph-store/hg-store-core/src/test/java/org/apache/hugegraph/store/business/BusinessHandlerImplTest.java b/hugegraph-store/hg-store-core/src/test/java/org/apache/hugegraph/store/business/BusinessHandlerImplTest.java new file mode 100644 index 0000000000..1e989c85a5 --- /dev/null +++ b/hugegraph-store/hg-store-core/src/test/java/org/apache/hugegraph/store/business/BusinessHandlerImplTest.java @@ -0,0 +1,473 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.business; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +import org.apache.hugegraph.rocksdb.access.RocksDBSession; +import org.apache.hugegraph.rocksdb.access.SessionOperator; +import org.apache.hugegraph.pd.grpc.Metapb; +import org.apache.hugegraph.store.meta.Partition; +import org.apache.hugegraph.store.meta.PartitionManager; +import org.apache.hugegraph.store.pd.PdProvider; +import org.apache.hugegraph.store.util.HgStoreException; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.Test; +import org.rocksdb.Cache; +import org.rocksdb.ColumnFamilyHandle; +import org.rocksdb.MemoryUsageType; + +/** + * Comprehensive unit tests for BusinessHandlerImpl covering static utility methods, + * partition operations, and core business logic. + */ +public class BusinessHandlerImplTest { + + private static class SessionOverridingBusinessHandler extends BusinessHandlerImpl { + + private final RocksDBSession session; + + SessionOverridingBusinessHandler(PartitionManager partitionManager, RocksDBSession session) { + super(partitionManager); + this.session = session; + } + + @Override + public RocksDBSession getSession(int partId) throws HgStoreException { + return this.session; + } + } + + private BusinessHandlerImpl handler; + private PartitionManager mockPartitionManager; + private RocksDBSession mockSession; + + @BeforeClass + public static void setUpClass() { + // Initialize static resources if needed + } + + @Before + public void setUp() { + mockPartitionManager = mock(PartitionManager.class); + mockSession = mock(RocksDBSession.class); + + handler = new BusinessHandlerImpl(mockPartitionManager); + + // Setup common mock behaviors + when(mockSession.sessionOp()).thenReturn(mock(SessionOperator.class)); + } + + @Test + public void testFnvHashDeterministicAndSensitiveToInput() { + byte[] keyA = "abc".getBytes(StandardCharsets.UTF_8); + byte[] keyB = "abd".getBytes(StandardCharsets.UTF_8); + + Long hashA1 = BusinessHandlerImpl.fnvHash(keyA); + Long hashA2 = BusinessHandlerImpl.fnvHash(keyA); + Long hashB = BusinessHandlerImpl.fnvHash(keyB); + + assertEquals(hashA1, hashA2); + assertNotEquals(hashA1, hashB); + } + + @Test + public void testFnvHashEmptyInputUsesOffsetBasis() { + Long hash = BusinessHandlerImpl.fnvHash(new byte[0]); + + assertEquals(Long.valueOf(0xcbf29ce484222325L), hash); + } + + @Test + public void testFnvHashSensitiveToByteOrder() { + byte[] key1 = new byte[]{1, 2, 3}; + byte[] key2 = new byte[]{3, 2, 1}; + + Long hash1 = BusinessHandlerImpl.fnvHash(key1); + Long hash2 = BusinessHandlerImpl.fnvHash(key2); + + assertNotEquals(hash1, hash2); + } + + @Test + public void testFnvHashLargeInput() { + byte[] largeInput = new byte[10000]; + for (int i = 0; i < largeInput.length; i++) { + largeInput[i] = (byte) (i % 256); + } + + Long hash = BusinessHandlerImpl.fnvHash(largeInput); + + assertNotNull(hash); + } + + @Test + public void testGetDbNameFormatsAndCachesPartitionId() { + String dbName = BusinessHandlerImpl.getDbName(12); + + assertEquals("00012", dbName); + assertEquals(dbName, BusinessHandlerImpl.getDbName(12)); + } + + @Test + public void testGetDbNameFormatsPadding() { + assertEquals("00001", BusinessHandlerImpl.getDbName(1)); + assertEquals("00010", BusinessHandlerImpl.getDbName(10)); + assertEquals("00100", BusinessHandlerImpl.getDbName(100)); + assertEquals("01000", BusinessHandlerImpl.getDbName(1000)); + assertEquals("10000", BusinessHandlerImpl.getDbName(10000)); + } + + @Test + public void testGetDbNameCachingBehavior() { + String first = BusinessHandlerImpl.getDbName(999); + String second = BusinessHandlerImpl.getDbName(999); + String third = BusinessHandlerImpl.getDbName(999); + + assertEquals(first, second); + assertEquals(second, third); + assertEquals("00999", first); + } + + @Test + public void testSetIndexDataSizeAcceptsPositiveAndIgnoresNonPositive() { + try { + BusinessHandlerImpl.setIndexDataSize(1L); + BusinessHandlerImpl.setIndexDataSize(1024L); + BusinessHandlerImpl.setIndexDataSize(Long.MAX_VALUE); + + // Non-positive values are documented as no-op and should not throw. + BusinessHandlerImpl.setIndexDataSize(0L); + BusinessHandlerImpl.setIndexDataSize(-10L); + BusinessHandlerImpl.setIndexDataSize(Long.MIN_VALUE); + } catch (Exception e) { + fail("setIndexDataSize should not throw for tested inputs: " + e.getMessage()); + } + } + + @Test + public void testSetIndexDataSizeSmallPositiveValue() { + try { + BusinessHandlerImpl.setIndexDataSize(1L); + BusinessHandlerImpl.setIndexDataSize(1024L); + } catch (Exception e) { + fail("setIndexDataSize should accept small positive values: " + e.getMessage()); + } + } + + @Test + public void testGetCompactionPoolIsNotNull() { + var pool = BusinessHandlerImpl.getCompactionPool(); + + assertNotNull(pool); + assertTrue(pool.getCorePoolSize() > 0); + assertTrue(pool.getMaximumPoolSize() > 0); + } + + @Test + public void testGetCompactionPoolConsistency() { + var pool1 = BusinessHandlerImpl.getCompactionPool(); + var pool2 = BusinessHandlerImpl.getCompactionPool(); + + assertEquals(pool1, pool2); + } + + // ========== Tests for doPut/doGet ========== + + @Test + public void testDoPutSuccessfully() throws HgStoreException { + // This test verifies the method signature exists and is callable + // Complete testing would require mocking internal RocksDB sessions + // which requires complex setup with actual RocksDB structures + } + + @Test + public void testDoPutThrowsExceptionOnInternalError() { + PartitionManager partitionManager = mock(PartitionManager.class); + PdProvider pdProvider = mock(PdProvider.class); + when(partitionManager.getPdProvider()).thenReturn(pdProvider); + + Metapb.Partition partition = Metapb.Partition.newBuilder().setId(10).build(); + when(pdProvider.getPartitionByCode("g", 1)).thenReturn(partition); + + RocksDBSession session = mock(RocksDBSession.class); + SessionOperator op = mock(SessionOperator.class); + when(session.sessionOp()).thenReturn(op); + doThrow(new RuntimeException("prepare boom")).when(op).prepare(); + + BusinessHandlerImpl localHandler = + new SessionOverridingBusinessHandler(partitionManager, session); + + try { + localHandler.doPut("g", 1, "g+v", new byte[]{1}, new byte[]{2}); + fail("Expected HgStoreException"); + } catch (HgStoreException e) { + assertEquals(HgStoreException.EC_RKDB_DOPUT_FAIL, e.getCode()); + assertTrue(e.getMessage().contains("prepare boom")); + verify(op, times(1)).rollback(); + } + } + + @Test + public void testDoGetReturnsNullWhenPartitionNotManaged() throws HgStoreException { + when(mockPartitionManager.hasPartition("test-graph", 1)).thenReturn(false); + + // Full test would need complete partition manager setup + } + + // ========== Tests for getLeaderPartitionIds ========== + + @Test + public void testGetLeaderPartitionIds() { + String graph = "test-graph"; + List expectedIds = Arrays.asList(1, 2, 3); + when(mockPartitionManager.getLeaderPartitionIds(graph)).thenReturn(expectedIds); + + List result = handler.getLeaderPartitionIds(graph); + + assertEquals(expectedIds, result); + verify(mockPartitionManager, times(1)).getLeaderPartitionIds(graph); + } + + @Test + public void testGetLeaderPartitionIdsReturnsEmptyList() { + String graph = "empty-graph"; + when(mockPartitionManager.getLeaderPartitionIds(graph)).thenReturn(Collections.emptyList()); + + List result = handler.getLeaderPartitionIds(graph); + + assertTrue(result.isEmpty()); + verify(mockPartitionManager, times(1)).getLeaderPartitionIds(graph); + } + + // ========== Tests for getLeaderPartitionIdSet ========== + + @Test + public void testGetLeaderPartitionIdSet() { + when(mockPartitionManager.getLeaderPartitionIdSet()).thenReturn( + Collections.singleton(1)); + + var result = handler.getLeaderPartitionIdSet(); + + assertNotNull(result); + assertTrue(result.contains(1)); + verify(mockPartitionManager, times(1)).getLeaderPartitionIdSet(); + } + + // ========== Tests for Table operations ========== + + @Test + public void testExistsTableReturnsTrue() { + String table = "g+v"; + + when(mockSession.tableIsExist(table)).thenReturn(true); + + // Full test would require session mocking + } + + @Test + public void testGetTableNames() { + List expectedTables = Arrays.asList("g+v", "g+e", "g+index"); + Map tableMap = new HashMap<>(); + expectedTables.forEach(t -> tableMap.put(t, mock(ColumnFamilyHandle.class))); + + when(mockSession.getTables()).thenReturn(tableMap); + + // Full test would require session mocking and setup + } + + // ========== Tests for Partition operations ========== + + @Test + public void testCleanPartitionCallsPartitionManager() { + String graph = "test-graph"; + int partId = 1; + + Partition mockPartition = mock(Partition.class); + when(mockPartitionManager.getPartitionFromPD(graph, partId)).thenReturn(mockPartition); + when(mockPartition.getStartKey()).thenReturn(0L); + when(mockPartition.getEndKey()).thenReturn(100L); + + // This tests that the method properly delegates to partition manager + // Full implementation requires extensive mocking + } + + @Test + public void testDeletePartition() { + // Verifies method exists and can be called + // Full testing requires RocksDB session management + } + + // ========== Tests for Metric operations ========== + + @Test + public void testGetApproximateMemoryUsageByType() { + List caches = new ArrayList<>(); + + Map result = handler.getApproximateMemoryUsageByType(caches); + + assertNotNull(result); + // Empty map on exception is expected behavior + } + + // ========== Tests for additional FNV Hash scenarios ========== + + @Test + public void testFnvHashConsistency() { + byte[] input = "test-data".getBytes(); + Long hash1 = BusinessHandlerImpl.fnvHash(input); + Long hash2 = BusinessHandlerImpl.fnvHash(input); + + assertEquals(hash1, hash2); + } + + @Test + public void testFnvHashDifferentInputs() { + byte[] input1 = "test1".getBytes(); + byte[] input2 = "test2".getBytes(); + + Long hash1 = BusinessHandlerImpl.fnvHash(input1); + Long hash2 = BusinessHandlerImpl.fnvHash(input2); + + assertNotEquals(hash1, hash2); + } + + // ========== Tests for additional index data size scenarios ========== + + @Test + public void testSetIndexDataSizePositive() { + long newSize = 100 * 1024L; + BusinessHandlerImpl.setIndexDataSize(newSize); + // Verify through reflection or static state inspection + } + + @Test + public void testSetIndexDataSizeNegativeIgnored() { + long originalSize = 50 * 1024L; + BusinessHandlerImpl.setIndexDataSize(originalSize); + + // Setting negative value should be ignored + BusinessHandlerImpl.setIndexDataSize(-1); + // Size should remain unchanged + } + + @Test + public void testSetIndexDataSizeZeroIgnored() { + long originalSize = 50 * 1024L; + BusinessHandlerImpl.setIndexDataSize(originalSize); + + // Setting zero should be ignored + BusinessHandlerImpl.setIndexDataSize(0); + // Size should remain unchanged + } + + // ========== Tests for transaction operations ========== + + @Test + public void testTxBuilderCreatesBuilder() { + // This verifies the method exists and returns a TxBuilder + // Full testing requires RocksDB session setup + } + + // ========== Tests for database operations ========== + + @Test + public void testCloseDB() { + int partId = 1; + + // Verifies the method can be called + // Full testing requires RocksDBFactory setup + handler.closeDB(partId); + } + + @Test + public void testFlushAll() { + // Verifies the method can be called without throwing + handler.flushAll(); + } + + @Test + public void testCloseAll() { + // Verifies the method can be called without throwing + handler.closeAll(); + } + + @Test + public void testGetPartitionIds() { + String graph = "test-graph"; + List expectedIds = Arrays.asList(1, 2, 3); + when(mockPartitionManager.getPartitionIds(graph)).thenReturn(expectedIds); + + List result = handler.getPartitionIds(graph); + + assertEquals(expectedIds, result); + } + + // ========== Tests for state management ========== + + @Test + public void testSetAndNotifyState() { + // Verifies method exists and basic functionality + // Full testing requires proper initialization of compactionState + } + + @Test + public void testGetState() { + // Verifies method exists + // Full testing requires proper state setup + } + + // ========== Tests for lock operations ========== + + @Test + public void testUnlock() { + // Verifies method exists + // Full testing requires proper pathLock setup + } + + @Test + public void testGetLockPath() { + int partitionId = 1; + when(mockPartitionManager.getDbDataPath(partitionId)) + .thenReturn("/data/partition/00001"); + + String lockPath = handler.getLockPath(partitionId); + + assertNotNull(lockPath); + verify(mockPartitionManager, times(1)).getDbDataPath(partitionId); + } +} + diff --git a/hugegraph-store/hg-store-dist/src/assembly/static/conf/application.yml b/hugegraph-store/hg-store-dist/src/assembly/static/conf/application.yml index 8e2b8d4f74..67e56ec6e4 100644 --- a/hugegraph-store/hg-store-dist/src/assembly/static/conf/application.yml +++ b/hugegraph-store/hg-store-dist/src/assembly/static/conf/application.yml @@ -62,3 +62,64 @@ logging: config: 'file:./conf/log4j2.xml' level: root: info + +# --------------------------------------------------------------------------- +# Cloud Storage (pluggable provider – disabled by default) +# --------------------------------------------------------------------------- +# To enable, set enabled=true and add the desired provider JAR to the +# classpath (e.g. hg-store-cloud-s3-*.jar for Amazon S3 / S3-compatible). +# +# cloud: +# storage: +# enabled: true +# provider: s3 # must match CloudStorageProvider.providerName() +# path-prefix: hugegraph # key prefix inside the bucket +# # --- Upload retry & dead-letter queue (common for all providers) --- +# upload-retry-max-attempts: 3 # whole-file retries after first failure; default 3 (0 disables) +# upload-retry-initial-delay-ms: 1000 # delay before first retry (ms); default 1000 +# upload-retry-max-delay-ms: 60000 # exponential-backoff ceiling (ms); default 60000 +# upload-backpressure-high-watermark: 64 # slow ingestion while pending-upload backlog exceeds this; 0 disables +# wal-mode: flush # 'flush' (lose un-flushed tail on crash) or 'wal' (mirror+replay WAL tail) +# # Metadata sync is event-triggered by storage events; no background interval scheduler. +# # --- S3 provider settings (cloud.storage.s3.*) --- +# s3: +# bucket: hugegraph-store # S3 bucket name +# region: us-east-1 # AWS region +# endpoint: # custom S3-compatible endpoint (MinIO, Ceph...) +# access-key: AKIAIOSFODNN7EXAMPLE # omit to use the AWS default credentials chain +# secret-key: wJalrXUtnFEMI/... # omit to use the AWS default credentials chain +# multipart-part-retry-max-attempts: 3 +# multipart-part-retry-base-backoff-ms: 1000 +# multipart-exhausted-direct-dlq: false +cloud: + storage: + enabled: false + provider: s3 + path-prefix: hugegraph + # Hydration controls + startup-hydration-enabled: true # download missing remote files on DB opening before serving + read-miss-guard-window-ms: 3000 # guard window (ms) to throttle repeated read-miss hydration attempts + # Whole-file upload retries after a first failure (default 3). Set 0 to disable + # (failures go straight to the DLQ) when the provider has sufficient internal retry. + upload-retry-max-attempts: 3 + upload-retry-initial-delay-ms: 1000 + upload-retry-max-delay-ms: 60000 + # Backpressure high-watermark on the pending-upload backlog; 0 disables. + upload-backpressure-high-watermark: 64 + # Mirror a consistent CURRENT/MANIFEST/OPTIONS[/WAL] snapshot so cloud objects stay + # recoverable (a node that lost its local disk reopens from cloud, not an empty DB). + # Metadata sync is always enabled when cloud storage is enabled. + # Sync is event-triggered by storage events; there is no background interval scheduler. + # WAL durability: 'flush' (force flush before capture; lose only the un-flushed tail on crash) + # or 'wal' (also mirror + replay the WAL tail; lower RPO, more frequent small uploads). + wal-mode: flush + s3: + bucket: + region: + endpoint: + access-key: + secret-key: + multipart-part-retry-max-attempts: 3 + multipart-part-retry-base-backoff-ms: 1000 + multipart-exhausted-direct-dlq: false + diff --git a/hugegraph-store/hg-store-node/pom.xml b/hugegraph-store/hg-store-node/pom.xml index aff68d0db0..255c6bf858 100644 --- a/hugegraph-store/hg-store-node/pom.xml +++ b/hugegraph-store/hg-store-node/pom.xml @@ -120,6 +120,15 @@ org.apache.hugegraph hg-store-core + + org.apache.hugegraph + hg-store-cloud-s3 + + + org.roaringbitmap + RoaringBitmap + 0.9.38 + com.taobao.arthas diff --git a/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/AppConfig.java b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/AppConfig.java index 3f1624c087..899c750da4 100644 --- a/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/AppConfig.java +++ b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/AppConfig.java @@ -17,21 +17,41 @@ package org.apache.hugegraph.store.node; +import java.nio.file.Paths; +import java.util.Arrays; import java.util.HashMap; +import java.util.LinkedHashMap; import java.util.Map; import javax.annotation.PostConstruct; - +import javax.annotation.PreDestroy; + +import org.apache.hugegraph.rocksdb.access.RocksDBFactory; +import org.apache.hugegraph.store.node.cloud.CloudStorageEventListener; +import org.apache.hugegraph.store.node.cloud.CloudStorageMetrics; +import org.apache.hugegraph.store.node.cloud.CloudSyncTracker; +import org.apache.hugegraph.store.node.cloud.CloudUploadRetryQueue; +import org.apache.hugegraph.store.cloud.CloudStorageConfig; +import org.apache.hugegraph.store.cloud.CloudStorageProviderFactory; import org.apache.hugegraph.store.options.JobOptions; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Value; import org.springframework.boot.context.properties.ConfigurationProperties; import org.springframework.context.annotation.Configuration; +import org.springframework.core.env.AbstractEnvironment; +import org.springframework.core.env.EnumerablePropertySource; +import org.springframework.core.env.Environment; import org.springframework.stereotype.Component; +import io.micrometer.core.instrument.MeterRegistry; + import lombok.Data; +import lombok.EqualsAndHashCode; +import lombok.ToString; +import lombok.extern.slf4j.Slf4j; @Data +@Slf4j @Component public class AppConfig { @@ -86,6 +106,15 @@ public class AppConfig { @Autowired private QueryPushDownConfig queryPushDownConfig; + @Autowired + private CloudStorageSpringConfig cloudStorageSpringConfig; + + @Autowired(required = false) + private MeterRegistry meterRegistry; + + /** Retry queue created during {@link #initCloudStorage()}; closed on {@link #onDestroy()}. */ + private volatile CloudUploadRetryQueue cloudUploadRetryQueue; + public String getRaftPath() { if (raftPath == null || raftPath.length() == 0) { return dataPath; @@ -116,6 +145,119 @@ public void init() { "0".equals(rocksdb.get("write_buffer_size"))) { rocksdb.put("write_buffer_size", Long.toString(totalMemory / 1000)); } + + // ---- Cloud storage initialization ---- + initCloudStorage(); + } + + /** + * Initialises the cloud storage provider (if enabled) and registers + * {@link CloudStorageEventListener} with {@link RocksDBFactory} so that + * SST file creation/deletion events are forwarded to cloud storage. + * + *

The resolved absolute data-path is passed to the listener so that + * S3 object keys are relative to the store's storage root rather than + * being full container-specific absolute paths. + */ + private void initCloudStorage() { + CloudStorageConfig cfg = cloudStorageSpringConfig.toCloudStorageConfig(); + if (!cfg.isEnabled()) { + log.info("Cloud storage disabled (cloud.storage.enabled=false)"); + return; + } + try { + CloudStorageProviderFactory.initialize(cfg); + String resolvedDataRoot = + Paths.get(dataPath).toAbsolutePath().normalize().toString(); + + // Shared sync tracker: the listener's delete guard and the retry queue's success + // callback both update it, so a superseded cloud object is deleted only once every + // live SST file of that DB is confirmed present in cloud. + CloudSyncTracker syncTracker = new CloudSyncTracker(); + + CloudUploadRetryQueue retryQueue = new CloudUploadRetryQueue( + cfg.getUploadRetryMaxAttempts(), + cfg.getUploadRetryInitialDelayMs(), + cfg.getUploadRetryMaxDelayMs(), + resolvedDataRoot, + syncTracker::markConfirmed); + this.cloudUploadRetryQueue = retryQueue; + + String storeScopePrefix = buildCloudStoreScopePrefix(); + + CloudStorageEventListener listener = new CloudStorageEventListener( + resolvedDataRoot, + cfg.isStartupHydrationEnabled(), + cfg.getReadMissGuardWindowMs(), + retryQueue, + syncTracker, + cfg.getUploadBackpressureHighWatermark(), + "wal".equalsIgnoreCase(cfg.getWalMode()), + storeScopePrefix); + + // Initialize metrics if MeterRegistry is available + if (meterRegistry != null) { + CloudStorageMetrics.init(meterRegistry, syncTracker); + } + + RocksDBFactory.getInstance().addRocksdbChangedListener(listener); + log.info("Cloud storage provider '{}' registered with RocksDBFactory " + + "(dataRoot='{}', storeScopePrefix='{}', startupHydration={}, " + + "readMissHydration=true, " + + "readMissGuardWindowMs={}, uploadRetryMaxAttempts={}, " + + "uploadRetryInitialDelayMs={}, uploadRetryMaxDelayMs={})", + cfg.getProvider(), resolvedDataRoot, storeScopePrefix, + cfg.isStartupHydrationEnabled(), + cfg.getReadMissGuardWindowMs(), + cfg.getUploadRetryMaxAttempts(), + cfg.getUploadRetryInitialDelayMs(), + cfg.getUploadRetryMaxDelayMs()); + } catch (Exception e) { + log.error("Failed to initialize cloud storage provider '{}': {}", + cfg.getProvider(), e.getMessage(), e); + } + } + + /** + * Builds a deterministic per-store cloud key prefix so distributed store nodes can safely + * share the same bucket/path-prefix without key collisions. + */ + private String buildCloudStoreScopePrefix() { + String identity = raft != null ? raft.getAddress() : null; + if (identity == null || identity.trim().isEmpty()) { + identity = getStoreServerAddress(); + } + return "store-" + sanitizeCloudKeySegment(identity); + } + + /** Converts an address-like identifier into a cloud-key-safe path segment. */ + private static String sanitizeCloudKeySegment(String raw) { + if (raw == null || raw.trim().isEmpty()) { + return "unknown"; + } + StringBuilder sb = new StringBuilder(raw.length()); + for (int i = 0; i < raw.length(); i++) { + char ch = raw.charAt(i); + if ((ch >= 'a' && ch <= 'z') || (ch >= 'A' && ch <= 'Z') || + (ch >= '0' && ch <= '9') || ch == '-' || ch == '_' || ch == '.') { + sb.append(ch); + } else { + sb.append('_'); + } + } + return sb.toString(); + } + + /** + * Gracefully shuts down the cloud upload retry queue on application stop. + */ + @PreDestroy + public void onDestroy() { + if (cloudUploadRetryQueue != null) { + log.info("Shutting down CloudUploadRetryQueue (dlqSize={}) …", + cloudUploadRetryQueue.getDlqSize()); + cloudUploadRetryQueue.close(); + } } @Override @@ -312,6 +454,100 @@ public class RocksdbConfig { private Map rocksdb = new HashMap<>(); } + /** + * Spring {@link ConfigurationProperties} wrapper for the common cloud storage properties. + * + *

Provider-specific keys (e.g. {@code cloud.storage.s3.bucket}) are read + * directly from the Spring {@link Environment} at conversion time, so this class + * has zero knowledge of any specific cloud provider. + * + * 

+     * cloud:
+     *   storage:
+     *     enabled: true
+     *     provider: s3          # selects which sub-namespace to forward to the provider
+     *     path-prefix: hugegraph
+     *     s3:                   # all keys here are forwarded as-is to the S3 provider
+     *       bucket: my-bucket
+     *       region: us-east-1
+     *       access-key: ${AWS_ACCESS_KEY_ID}
+     *       secret-key: ${AWS_SECRET_ACCESS_KEY}
+     *
+ */ + @Data + @Configuration + @ConfigurationProperties(prefix = "cloud.storage") + public class CloudStorageSpringConfig { + + private boolean enabled = false; + private String provider = "s3"; + private String pathPrefix = "hugegraph"; + private boolean startupHydrationEnabled = true; + private long readMissGuardWindowMs = 3000L; + // Whole-file upload retries after a first failure. Default 3 under the primary-durability + private int uploadRetryMaxAttempts = 3; + private long uploadRetryInitialDelayMs = 1_000L; + private long uploadRetryMaxDelayMs = 60_000L; + // Backpressure high-watermark on the pending-upload backlog; 0 disables. + private int uploadBackpressureHighWatermark = 64; + private String walMode = "flush"; + + /** + * Injected by Spring; used to read {@code cloud.storage..*} properties + * without coupling this class to any specific provider. + */ + @Autowired + @EqualsAndHashCode.Exclude + @ToString.Exclude + private Environment environment; + + /** Converts this Spring-bound config into a plain {@link CloudStorageConfig} POJO. */ + public CloudStorageConfig toCloudStorageConfig() { + CloudStorageConfig cfg = new CloudStorageConfig(); + cfg.setEnabled(enabled); + cfg.setProvider(provider); + cfg.setPathPrefix(pathPrefix); + cfg.setStartupHydrationEnabled(startupHydrationEnabled); + cfg.setReadMissGuardWindowMs(readMissGuardWindowMs); + cfg.setUploadRetryMaxAttempts(uploadRetryMaxAttempts); + cfg.setUploadRetryInitialDelayMs(uploadRetryInitialDelayMs); + cfg.setUploadRetryMaxDelayMs(uploadRetryMaxDelayMs); + cfg.setUploadBackpressureHighWatermark(uploadBackpressureHighWatermark); + cfg.setWalMode(walMode); + cfg.setProviderProperties(readProviderProperties()); + return cfg; + } + + /** + * Reads all {@code cloud.storage..*} keys from the Spring Environment + * and returns them as a flat map with the provider sub-prefix stripped. + * + *

For example, with {@code provider=s3}, the YAML key + * {@code cloud.storage.s3.bucket} becomes {@code bucket} in the returned map. + */ + private Map readProviderProperties() { + Map props = new LinkedHashMap<>(); + if (!(environment instanceof AbstractEnvironment)) { + return props; + } + String prefix = "cloud.storage." + provider + "."; + ((AbstractEnvironment) environment).getPropertySources().stream() + .filter(ps -> ps instanceof EnumerablePropertySource) + .map(ps -> (EnumerablePropertySource) ps) + .flatMap(ps -> Arrays.stream(ps.getPropertyNames())) + .filter(key -> key.startsWith(prefix)) + .distinct() + .forEach(key -> { + String shortKey = key.substring(prefix.length()); + String value = environment.getProperty(key); + if (value != null) { + props.put(shortKey, value); + } + }); + return props; + } + } + public JobOptions getJobOptions() { JobOptions jobOptions = new JobOptions(); jobOptions.setCore(jobConfig.getCore() == 0 ? cpus : jobConfig.getCore()); diff --git a/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudStorageEventListener.java b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudStorageEventListener.java new file mode 100644 index 0000000000..6204ebb75a --- /dev/null +++ b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudStorageEventListener.java @@ -0,0 +1,1318 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.node.cloud; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.ArrayBlockingQueue; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.RejectedExecutionException; +import java.util.concurrent.ThreadPoolExecutor; +import java.util.concurrent.ThreadFactory; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.locks.LockSupport; +import java.util.stream.Stream; + +import org.apache.hugegraph.rocksdb.access.RocksDBFactory; +import org.apache.hugegraph.rocksdb.access.RocksDBFactory.LiveSstFile; +import org.apache.hugegraph.rocksdb.access.RocksDBFactory.MetadataSnapshot; +import org.apache.hugegraph.rocksdb.access.RocksDBFactory.RocksdbChangedListener; +import org.apache.hugegraph.rocksdb.access.RocksDBSession; +import org.apache.hugegraph.store.cloud.CloudStorageProvider; +import org.apache.hugegraph.store.cloud.CloudStorageProviderFactory; + +import lombok.extern.slf4j.Slf4j; + +/** + * {@link RocksdbChangedListener} that bridges RocksDB table-file lifecycle events + * to the active {@link CloudStorageProvider}. + * + *

When cloud storage is enabled: + *

    + *
  • {@link #onDBCreated} uploads any SST files that already exist in the DB directory + * (e.g. surviving from a previous run), triggers a non-blocking MemTable flush so + * WAL-recovered or recently-written data is written to SST files (completion is signalled + * event-driven via {@link #onTableFileCreated}), then mirrors metadata inline.
    • + *
  • {@link #onTableFileCreated} pins newly created SST files via a hard link, dispatches + * upload work to a bounded background executor, and mirrors metadata after upload completes. + * If the hard link fails (e.g. cross-device mount, filesystem limits), the upload is routed + * directly to the retry queue using the original SST path — no copy is made, so no extra + * disk space is consumed.
    • + *
  • {@link #onTableFileDeleted} mirrors metadata first, then removes the superseded SST object.
    • + *
+ * + *

Remote key construction

+ * The remote key is derived by stripping the {@code dataRoot} prefix from the absolute + * local file path. This keeps the object layout clean and independent of the container + * filesystem layout: + * 
+ *   dataRoot  = /hugegraph-store/storage
+ *   filePath  = /hugegraph-store/storage/hgstore-metadata/000008.sst
+ *   remoteKey = store-127.0.0.1_8501/hgstore-metadata/000008.sst
+ *   (with path-prefix "hugegraph") → hugegraph/store-127.0.0.1_8501/hgstore-metadata/000008.sst
+ *
+ * + * This listener is registered with {@link RocksDBFactory} during application startup + * (see {@link org.apache.hugegraph.store.node.AppConfig}). + */ +@Slf4j +public class CloudStorageEventListener implements RocksdbChangedListener { + + /** Absolute, normalised path of the store's data root directory. */ + private final String dataRoot; + + /** Optional per-store namespace prefix prepended to every remote cloud key. */ + private final String storeScopePrefix; + + private static final long DEFAULT_READ_MISS_GUARD_WINDOW_MS = 3000L; + + private final boolean startupHydrationEnabled; + private final long readMissGuardWindowMs; + private final Map readMissAttemptTs; + + /** + * Optional retry queue; when non-null, upload failures are submitted here instead + * of just being logged. When null, failures are only logged (no retry). + */ + private final CloudUploadRetryQueue retryQueue; + + /** Tracks which SST files are confirmed present in cloud (per-DB Roaring bitmap). */ + private final CloudSyncTracker syncTracker; + + /** + * When {@code > 0}, {@link #onTableFileCreated} slows the RocksDB flush/compaction thread while + * the number of not-yet-durable uploads (retry-queue in-flight + DLQ) exceeds this watermark, + * so ingestion cannot outrun the cloud mirror. {@code 0} disables backpressure. + */ + private final int backpressureHighWatermark; + + + /** Upper bound on how long a single {@link #onTableFileCreated} call will block for backpressure. */ + private static final long BACKPRESSURE_MAX_WAIT_MS = 30_000L; + private static final long BACKPRESSURE_POLL_MS = 50L; + + /** Bounded async upload dispatcher so RocksDB callbacks return quickly. */ + private static final int ASYNC_UPLOAD_THREADS = 2; + private static final int ASYNC_UPLOAD_QUEUE_CAPACITY = 256; + private static final ThreadPoolExecutor SHARED_UPLOAD_EXECUTOR = + new ThreadPoolExecutor( + ASYNC_UPLOAD_THREADS, + ASYNC_UPLOAD_THREADS, + 60L, + TimeUnit.SECONDS, + new ArrayBlockingQueue<>(ASYNC_UPLOAD_QUEUE_CAPACITY), + newUploadThreadFactory(), + new ThreadPoolExecutor.AbortPolicy()); + private final ThreadPoolExecutor uploadExecutor; + private final Path uploadStagingDir; + + // ----------------------------------------------------------------------- + // Metadata (CURRENT/MANIFEST/OPTIONS[/WAL]) mirroring & consistent restore + // ----------------------------------------------------------------------- + + + /** + * When {@code true} ({@code wal-mode: wal}), the active WAL {@code *.log} segments are mirrored + * alongside the metadata and replayed on restore. When {@code false} ({@code wal-mode: flush}), + * no WAL is mirrored. + */ + private final boolean walModeEnabled; + + /** Resolved DB directory -> logical DB name, so {@link #onCompacted} resolves path events. */ + private final Map dbNameByDir = new ConcurrentHashMap<>(); + + /** + * Tracks which DBs are currently being truncated. While a DB is in this set, + * metadata sync operations are skipped to allow the purge to complete cleanly + * without new metadata files being re-uploaded. + */ + private final Set truncatingDbs = ConcurrentHashMap.newKeySet(); + + /** + * Tracks the timestamp of recent truncations (DB name -> truncation time in ms). + * Used to suppress metadata syncs for a grace period after truncation, allowing + * pending RocksDB background operations and callbacks to complete without + * re-uploading metadata that was just purged. + */ + private final Map truncationTimes = new ConcurrentHashMap<>(); + + /** + * Grace period (ms) after truncation during which metadata syncs are suppressed. + * This allows pending RocksDB background callbacks to complete without re-uploading + * metadata that was purged during truncation. + */ + private static final long TRUNCATION_GRACE_PERIOD_MS = 5_000L; + + /** + * Sentinel object key written to the DB prefix during database deletion. + * {@link #preHydrateDbFiles} checks for this key and skips hydration when present, preventing + * a newly-recreated DB from ingesting data that belonged to a previous deleted generation. + */ + static final String DB_TOMBSTONE_FILE = "_DELETED"; + + /** + * @param dataRoot absolute path of the store's data directory + * (value of {@code app.data-path}, resolved to an absolute path). + */ + public CloudStorageEventListener(String dataRoot) { + this(dataRoot, true, DEFAULT_READ_MISS_GUARD_WINDOW_MS, null); + } + + public CloudStorageEventListener(String dataRoot, + boolean startupHydrationEnabled) { + this(dataRoot, startupHydrationEnabled, DEFAULT_READ_MISS_GUARD_WINDOW_MS, null); + } + + /** + * @param readMissGuardWindowMs guard window in ms for repeated read-miss hydration attempts + * for the same db/table pair (cloud.storage.read-miss-guard-window-ms) + */ + public CloudStorageEventListener(String dataRoot, + boolean startupHydrationEnabled, + long readMissGuardWindowMs) { + this(dataRoot, startupHydrationEnabled, readMissGuardWindowMs, null); + } + + /** + * @param retryQueue optional {@link CloudUploadRetryQueue}; when non-null, upload failures + * are retried asynchronously and eventually moved to the dead-letter queue. + * Pass {@code null} to disable retries (failures are only logged). + */ + public CloudStorageEventListener(String dataRoot, + boolean startupHydrationEnabled, + long readMissGuardWindowMs, + CloudUploadRetryQueue retryQueue) { + this(dataRoot, startupHydrationEnabled, readMissGuardWindowMs, retryQueue, + new CloudSyncTracker(), 0); + } + + /** + * Full constructor. + * + * @param syncTracker tracks SST files confirmed present in cloud; the delete guard + * uses it to avoid deleting a superseded object before its + * replacements are durable. Must be shared with the retry queue. + * @param backpressureHighWatermark {@code > 0} to slow ingestion while the pending-upload backlog + * exceeds this value; {@code 0} disables backpressure. + */ + public CloudStorageEventListener(String dataRoot, + boolean startupHydrationEnabled, + long readMissGuardWindowMs, + CloudUploadRetryQueue retryQueue, + CloudSyncTracker syncTracker, + int backpressureHighWatermark) { + this(dataRoot, startupHydrationEnabled, readMissGuardWindowMs, retryQueue, syncTracker, + backpressureHighWatermark, false); + } + + /** + * Full constructor including metadata-mirroring parameters. + * + * @param walModeEnabled {@code true} for {@code wal-mode: wal} (mirror + replay WAL); + * {@code false} for {@code wal-mode: flush} (force flush, no WAL) + */ + public CloudStorageEventListener(String dataRoot, + boolean startupHydrationEnabled, + long readMissGuardWindowMs, + CloudUploadRetryQueue retryQueue, + CloudSyncTracker syncTracker, + int backpressureHighWatermark, + boolean walModeEnabled) { + this(dataRoot, startupHydrationEnabled, readMissGuardWindowMs, retryQueue, syncTracker, + backpressureHighWatermark, walModeEnabled, null); + } + + /** + * Full constructor including metadata-mirroring and per-store key namespace parameters. + * + * @param storeScopePrefix optional per-store key prefix to isolate cloud objects across + * distributed store nodes sharing the same bucket/path-prefix. + */ + public CloudStorageEventListener(String dataRoot, + boolean startupHydrationEnabled, + long readMissGuardWindowMs, + CloudUploadRetryQueue retryQueue, + CloudSyncTracker syncTracker, + int backpressureHighWatermark, + boolean walModeEnabled, + String storeScopePrefix) { + String normalised = Paths.get(dataRoot).toAbsolutePath().normalize().toString(); + // Strip trailing separator so substring arithmetic is consistent. + this.dataRoot = normalised.endsWith(File.separator) + ? normalised.substring(0, normalised.length() - 1) + : normalised; + this.startupHydrationEnabled = startupHydrationEnabled; + this.readMissGuardWindowMs = Math.max(0L, readMissGuardWindowMs); + this.readMissAttemptTs = new ConcurrentHashMap<>(); + this.retryQueue = retryQueue; + this.syncTracker = syncTracker != null ? syncTracker : new CloudSyncTracker(); + this.backpressureHighWatermark = Math.max(0, backpressureHighWatermark); + this.walModeEnabled = walModeEnabled; + this.storeScopePrefix = normaliseKeyPrefix(storeScopePrefix); + this.uploadStagingDir = Paths.get(this.dataRoot, ".cloud-upload-staging"); + this.uploadExecutor = SHARED_UPLOAD_EXECUTOR; + } + + private static ThreadFactory newUploadThreadFactory() { + return r -> { + Thread t = new Thread(r, "cloud-upload-dispatch"); + t.setDaemon(true); + return t; + }; + } + + // ----------------------------------------------------------------------- + // RocksdbChangedListener + // ----------------------------------------------------------------------- + + @Override + public void onDBOpening(String dbName, String dbPath) { + if (!startupHydrationEnabled) { + return; + } + CloudStorageProvider provider = CloudStorageProviderFactory.getActiveProvider(); + if (provider == null) { + return; + } + preHydrateDbFiles(provider, dbName, dbPath); + } + + /** + * Called when a read returns null in RocksDB. + * + *

We restore only the SST files that RocksDB references as live in its manifest but + * that are physically missing on local disk, downloading each back to its exact original + * path so RocksDB finds it on the next access. This deliberately avoids + * {@code ingestExternalFile}, which would (a) risk placing a file into the wrong column family + * and (b) assign a fresh sequence number that can resurrect deleted keys. Restricting to the + * live set also guarantees superseded / compacted-away objects are never resurrected. + * + *

Note: a genuine key-not-found also arrives here as {@code value == null}; in that case no + * live file is missing and we return {@code false} without any cloud I/O. + */ + @Override + public boolean onReadMiss(RocksDBSession session, String table, byte[] key) { + String dbName = session.getGraphName(); + if (!shouldAttemptReadMissHydration(dbName, table)) { + return false; + } + CloudStorageProvider provider = CloudStorageProviderFactory.getActiveProvider(); + if (provider == null) { + return false; + } + int restored = restoreMissingLiveFiles(provider, dbName, + RocksDBFactory.getInstance().getLiveSstFiles(dbName)); + if (restored > 0) { + log.info("Cloud read-miss hydration: restored {} missing live SST file(s) for db={}", + restored, dbName); + return true; + } + return false; + } + + /** + * Downloads back any SST files that are live in RocksDB's manifest but missing on local disk, + * writing each to its original path. Returns the number of files restored. + * + *

Package-private testable seam: caller supplies the live-file set. + */ + int restoreMissingLiveFiles(CloudStorageProvider provider, String dbName, + List liveFiles) { + int restored = 0; + for (LiveSstFile live : liveFiles) { + Path localPath = Paths.get(live.getAbsolutePath()); + if (Files.exists(localPath)) { + continue; + } + String remoteKey = toRelativeKey(live.getAbsolutePath()); + try { + if (!provider.fileExists(remoteKey)) { + log.warn("Cloud read-miss: live file missing locally AND absent in cloud: " + + "db={}, key={}", dbName, remoteKey); + continue; + } + Files.createDirectories(localPath.getParent()); + // Download to a temp file then atomically move into place so a crash mid-download + // never leaves RocksDB reading a truncated SST at the expected path. + Path tmp = localPath.resolveSibling(localPath.getFileName() + ".hydrate"); + provider.downloadFile(remoteKey, tmp.toString()); + Files.move(tmp, localPath, java.nio.file.StandardCopyOption.ATOMIC_MOVE); + syncTracker.markConfirmed(dbName, live.getAbsolutePath()); + restored++; + } catch (IOException e) { + log.warn("Cloud read-miss restore failed: db={}, key={}, reason={}", + dbName, remoteKey, e.getMessage()); + } + } + return restored; + } + + /** + * Called when a new RocksDB instance is opened for the first time. + * + *

Uploads any SST files that already exist in {@code dbPath} (e.g. from a previous run) + * and then triggers a MemTable flush so that WAL-recovered data is also written to + * SST files and eventually forwarded here via {@link #onTableFileCreated}. + * + * @param dbName logical name of the graph / partition + * @param dbPath absolute path of the RocksDB directory + */ + @Override + public void onDBCreated(String dbName, String dbPath) { + recordDb(dbName, dbPath); + CloudStorageProvider provider = CloudStorageProviderFactory.getActiveProvider(); + if (provider == null) { + return; + } + uploadExistingSstFiles(provider, dbName, dbPath); + flushDb(dbName); + // Mirror metadata immediately after initial upload/flush to keep cloud state recoverable. + syncMetadataSnapshotInline(provider, dbName); + } + + /** + * Called just before the local RocksDB directory is removed. Writes a small tombstone object + * ({@value #DB_TOMBSTONE_FILE}) to the DB's remote prefix so that any subsequent + * {@link #preHydrateDbFiles} call for the same path will detect the deleted generation and + * skip hydration rather than re-ingesting stale objects. + * + *

This callback fires while the session is still in a pending-destroy list (refcount may + * be non-zero). The tombstone write is best-effort: a failure is logged but does not block + * the deletion. The cloud purge in {@link #onDBDeleted} provides a second line of defence. + * + * @param dbName logical graph/partition name + * @param dbPath absolute path of the RocksDB directory being destroyed + */ + @Override + public void onDBDeleteBegin(String dbName, String dbPath) { + CloudStorageProvider provider = CloudStorageProviderFactory.getActiveProvider(); + if (provider == null) { + return; + } + String tombstoneKey = dbPrefix(dbPath) + "/" + DB_TOMBSTONE_FILE; + Path tmp = null; + try { + tmp = Files.createTempFile("hgstore-tombstone-", ".tmp"); + Files.write(tmp, "deleted".getBytes(java.nio.charset.StandardCharsets.UTF_8)); + provider.uploadFile(tmp.toString(), tombstoneKey); + log.info("Cloud DB tombstone written: db={}, key={}", dbName, tombstoneKey); + } catch (Exception e) { + log.warn("Cloud DB tombstone write failed (onDBDeleted will still purge): " + + "db={}, key={}, reason={}", dbName, tombstoneKey, e.getMessage()); + } finally { + if (tmp != null) { + try { + Files.deleteIfExists(tmp); + } catch (IOException ignore) { + // best-effort temp-file cleanup + } + } + } + } + + /** + * Called after the local RocksDB directory has been physically removed. Purges all cloud + * objects under the DB prefix (SSTs, metadata, tombstone) so a future creation at the same + * path starts with a clean remote state. Also clears all in-memory state for this DB. + * + *

The purge is best-effort: individual delete failures are logged at DEBUG level and do + * not throw. Any objects that survive the purge are neutralised by the tombstone check in + * {@link #preHydrateDbFiles}: the next open will find the tombstone (or an empty prefix if + * the purge was complete), skip hydration, and clean up any leftovers. + * + * @param dbName logical graph/partition name + * @param dbPath absolute path of the now-deleted RocksDB directory + */ + @Override + public void onDBDeleted(String dbName, String dbPath) { + // Clear in-memory tracking so no stale state bleeds into a recreated DB. + syncTracker.clearDb(dbName); + readMissAttemptTs.entrySet().removeIf(e -> e.getKey().startsWith(dbName + "::")); + dbNameByDir.values().removeIf(dbName::equals); + + CloudStorageProvider provider = CloudStorageProviderFactory.getActiveProvider(); + if (provider == null) { + return; + } + purgeRemotePrefix(provider, dbName, dbPrefix(dbPath)); + } + + /** + * Called after a RocksDB has been truncated (all data cleared but directory preserved). + * Purges all cloud objects under the DB prefix (SSTs, metadata) so the remote state matches + * the now-empty local state. Also clears all in-memory sync tracking for this DB. + * + *

This is triggered by graph.clear() operations to ensure cloud storage is cleaned up + * when the graph data is cleared. + * + * @param dbName logical graph/partition name + * @param dbPath absolute path of the RocksDB directory + */ + @Override + public void onDBTruncateBegin(String dbName, String dbPath) { + truncatingDbs.add(dbName); + truncationTimes.put(dbName, System.currentTimeMillis()); + syncTracker.clearDb(dbName); + readMissAttemptTs.entrySet().removeIf(e -> e.getKey().startsWith(dbName + "::")); + } + + @Override + public void onDBTruncated(String dbName, String dbPath) { + truncatingDbs.add(dbName); + try { + CloudStorageProvider provider = CloudStorageProviderFactory.getActiveProvider(); + if (provider == null) { + return; + } + purgeRemotePrefix(provider, dbName, dbPrefix(dbPath)); + } finally { + truncationTimes.put(dbName, System.currentTimeMillis()); + truncatingDbs.remove(dbName); + } + } + + /** + * Checks if a DB is within the grace period after truncation, during which + * metadata syncs should be suppressed to prevent re-uploading purged data. + */ + private boolean isInTruncationGracePeriod(String dbName) { + Long truncationTime = truncationTimes.get(dbName); + if (truncationTime == null) { + return false; + } + long elapsed = System.currentTimeMillis() - truncationTime; + if (elapsed > TRUNCATION_GRACE_PERIOD_MS) { + // Grace period expired, remove the record + truncationTimes.remove(dbName); + return false; + } + return true; + } + + /** + * Deletes every remote object under {@code prefix} using an optimized prefix-level delete + * if available, falling back to individual file deletion if necessary. + * This is called during DB destruction to prevent a recreated DB from hydrating stale data. + */ + private void purgeRemotePrefix(CloudStorageProvider provider, String dbName, String prefix) { + String normalizedPrefix = prefix.endsWith("/") ? prefix : prefix + "/"; + try { + int deleted = provider.deletePrefix(normalizedPrefix); + if (deleted > 0) { + log.info("Cloud DB purge completed: db={}, prefix={}, deleted={}", + dbName, prefix, deleted); + } + } catch (IOException e) { + log.warn("Cloud DB purge failed for db={}, prefix={}: {}", + dbName, prefix, e.getMessage()); + } + } + + /** + * Pins and asynchronously uploads the newly created SST file to the active cloud + * storage provider. + * + * @param dbName RocksDB instance name (partition id) + * @param cfName column-family name + * @param filePath absolute local path of the new SST file + * @param fileSize file size in bytes (informational) + */ + @Override + public void onTableFileCreated(String dbName, String cfName, + String filePath, long fileSize) { + CloudStorageProvider provider = CloudStorageProviderFactory.getActiveProvider(); + if (provider == null) { + return; + } + recordDb(dbName, parentDir(filePath)); + CloudStorageMetrics.registerDatabaseMetrics(dbName); + String remoteKey = toRelativeKey(filePath); + + Path pinned; + try { + pinned = pinForAsyncUpload(filePath); + } catch (Exception e) { + String errorType = e.getClass().getSimpleName(); + CloudStorageMetrics.recordUploadFailure(dbName, cfName, errorType); + // Hard link failed — no copy fallback, no extra disk use. Route original SST path + // to retry queue; retry will upload directly from the original file if still present. + log.warn("Cloud upload staging (hard link failed): db={}, cf={}, path={} " + + "— routing original SST to retry queue: {}", + dbName, cfName, filePath, e.getMessage()); + if (retryQueue != null) { + retryQueue.submit(dbName, cfName, filePath, remoteKey, e); + } + applyBackpressure(dbName); + return; + } + + try { + uploadExecutor.execute(() -> { + long startTimeMs = System.currentTimeMillis(); + try { + provider.uploadFile(pinned.toString(), remoteKey); + syncTracker.markConfirmed(dbName, filePath); + long syncLatencyMs = System.currentTimeMillis() - startTimeMs; + CloudStorageMetrics.recordSyncLatency(dbName, syncLatencyMs); + // Skip metadata sync if DB is being truncated or in grace period after truncation + // to allow purge to complete cleanly without metadata files being re-uploaded. + if (!truncatingDbs.contains(dbName) && !isInTruncationGracePeriod(dbName)) { + syncMetadataSnapshotInline(provider, dbName); + } + log.debug("Cloud upload success: db={}, cf={}, path={}, size={}, latencyMs={}", + dbName, cfName, filePath, fileSize, syncLatencyMs); + } catch (Exception e) { + String errorType = e.getClass().getSimpleName(); + CloudStorageMetrics.recordUploadFailure(dbName, cfName, errorType); + log.error("Cloud upload failed (will retry on next compaction): " + + "db={}, cf={}, path={}, error={}", dbName, cfName, + filePath, e.getMessage()); + if (retryQueue != null) { + // Keep retry semantics unchanged: retries target the original SST path. + retryQueue.submit(dbName, cfName, filePath, remoteKey, e); + } + } finally { + try { + Files.deleteIfExists(pinned); + } catch (IOException e) { + log.debug("Failed to cleanup staged upload file {}: {}", + pinned, e.getMessage()); + } + } + }); + } catch (RejectedExecutionException e) { + try { + Files.deleteIfExists(pinned); + } catch (IOException ioe) { + log.debug("Failed to cleanup staged upload file {}: {}", pinned, ioe.getMessage()); + } + CloudStorageMetrics.recordUploadFailure(dbName, cfName, "UploadQueueFull"); + log.error("Cloud upload dispatch rejected (queue full): db={}, cf={}, path={}", + dbName, cfName, filePath); + if (retryQueue != null) { + retryQueue.submit(dbName, cfName, filePath, remoteKey, + new IOException("cloud upload dispatch queue full", e)); + } + } + + // Apply backpressure AFTER handling this file so the flush/compaction thread slows down + // while the cloud mirror is behind, preventing ingestion from outrunning durability. + applyBackpressure(dbName); + } + + /** + * Creates a stable hard-link snapshot of the SST file for async upload, so the upload worker + * can read a consistent source even after RocksDB deletes the original during compaction. + * + *

Hard links share the same inode — no extra data blocks are consumed. If the original is + * deleted by compaction before the worker runs, the hard-link still holds the inode alive so + * the upload can proceed normally. + * + *

If the hard link fails (e.g. cross-device mount, filesystem hard-link limits), an + * {@link IOException} is thrown. The caller ({@link #onTableFileCreated}) catches this and + * routes the upload to the retry queue using the original SST path — no copy is made and no + * extra disk space is consumed. The retry will succeed as long as the original file still + * exists when it fires; if it has been compacted away the retry queue silently drops it. + */ + private Path pinForAsyncUpload(String filePath) throws IOException { + Path source = Paths.get(filePath); + Files.createDirectories(uploadStagingDir); + String fileName = source.getFileName().toString(); + Path staged = uploadStagingDir.resolve(fileName + ".upload-" + System.nanoTime()); + try { + Files.createLink(staged, source); + return staged; + } catch (Exception linkEx) { + throw new IOException( + "Hard link failed; upload will be retried from original SST path: " + + linkEx.getMessage(), linkEx); + } + } + + /** + * Blocks the calling (RocksDB flush/compaction) thread while the pending-upload backlog exceeds + * {@link #backpressureHighWatermark}, up to {@link #BACKPRESSURE_MAX_WAIT_MS}. This is the + * durability-tier backpressure: it keeps at-risk local-only data bounded. + */ + private void applyBackpressure(String dbName) { + if (backpressureHighWatermark <= 0 || retryQueue == null) { + return; + } + long waited = 0L; + boolean logged = false; + while (pendingUploadBacklog() > backpressureHighWatermark + && waited < BACKPRESSURE_MAX_WAIT_MS) { + if (!logged) { + log.warn("Cloud upload backpressure: db={}, backlog={} > watermark={}, " + + "slowing ingestion", dbName, pendingUploadBacklog(), + backpressureHighWatermark); + logged = true; + } + LockSupport.parkNanos(TimeUnit.MILLISECONDS.toNanos(BACKPRESSURE_POLL_MS)); + if (Thread.currentThread().isInterrupted()) { + return; + } + waited += BACKPRESSURE_POLL_MS; + } + if (logged) { + log.info("Cloud upload backpressure released: db={}, backlog={}, waitedMs={}", + dbName, pendingUploadBacklog(), waited); + } + } + + private int pendingUploadBacklog() { + int backlog = uploadExecutor.getQueue().size() + uploadExecutor.getActiveCount(); + if (retryQueue != null) { + backlog += retryQueue.getInFlightCount() + retryQueue.getDlqSize(); + } + return backlog; + } + + /** + * Removes the deleted SST file from the active cloud storage provider. + * + * @param dbName RocksDB instance name (partition id) + * @param cfName column-family name + * @param filePath absolute local path of the deleted SST file + */ + @Override + public void onTableFileDeleted(String dbName, String cfName, String filePath) { + CloudStorageProvider provider = CloudStorageProviderFactory.getActiveProvider(); + if (provider == null) { + return; + } + // Skip file deletion during truncation or grace period; the entire DB prefix will be purged anyway + if (truncatingDbs.contains(dbName) || isInTruncationGracePeriod(dbName)) { + log.debug("Skipping delete during truncation: db={}, path={}", dbName, filePath); + return; + } + // DATA-LOSS GUARD: never delete a superseded cloud object until every SST file currently + // live in this DB is confirmed present in cloud. + if (!ensureLiveSetUploaded(provider, dbName)) { + log.warn("Delete skipped (live set not fully durable in cloud): db={}, filePath={}", + dbName, filePath); + return; + } + + // MANIFEST-BEFORE-DELETE INVARIANT: publish an updated MANIFEST+CURRENT that reflects + // the post-compaction live set before removing the old SST from cloud. Without this, a + // crash between the SST delete and the next metadata-sync attempt leaves a cloud + // MANIFEST that references an object that no longer exists, making recovery impossible. + // + // We use syncMetadataSnapshotInline (no MemTable flush) because: + // (a) this callback fires on a RocksDB compaction thread — calling flushSession(wait=true) + // here would deadlock against RocksDB background execution; + // (b) a flush is not needed: by the time onTableFileDeleted fires the compaction is + // complete, new SSTs are already on disk and uploaded, and the MANIFEST already + // reflects the post-compaction state. + if (!syncMetadataSnapshotInline(provider, dbName)) { + log.warn("Delete skipped (metadata sync failed, MANIFEST not yet updated): " + + "db={}, filePath={}", dbName, filePath); + return; + } + + String remoteKey = toRelativeKey(filePath); + try { + provider.deleteFile(remoteKey); + syncTracker.clearConfirmed(dbName, filePath); + log.debug("Cloud delete success: db={}, cf={}, path={}", dbName, cfName, filePath); + } catch (Exception e) { + // Non-fatal: log and continue. + log.error("Cloud delete failed: db={}, cf={}, path={}", dbName, cfName, filePath, e); + } + } + + /** + * Ensures every live SST file of {@code dbName} is confirmed present in cloud, uploading any + * that are not yet confirmed. Returns {@code true} only when the entire live set is durable. + * + *

Fast path (common case)

+ * {@link #preHydrateDbFiles} seeds {@link #syncTracker} from the cloud listing on startup, + * and {@link #onTableFileCreated} marks every successful upload confirmed. By the time + * {@link #onTableFileDeleted} fires the entire live set is normally already confirmed, so + * {@link CloudSyncTracker#allConfirmed} returns {@code true} after one lock acquisition with + * zero cloud I/O — regardless of live-set size. + * + *

Why we must check the whole live set, not just the deleted {@code filePath}

+ * {@code filePath} is the old compaction input being removed from cloud. Its bit + * being set in the bitmap only confirms it was uploaded previously — it says nothing about + * whether the compaction outputs (the replacement files, which form the new live + * set) are also in cloud. Deleting the input before the outputs are durable would leave + * data irretrievably lost on a node crash. + * + *

Slow path (rare)

+ * Entered only when {@code allConfirmed} returns {@code false} (e.g. a previous upload + * failed). Files present locally are uploaded directly without a {@code fileExists} probe + * (idempotent PUT). Files absent locally and not confirmed are treated as not-yet-durable + * and the delete is held. + */ + private boolean ensureLiveSetUploaded(CloudStorageProvider provider, String dbName) { + return ensureLiveSetUploaded(provider, dbName, + RocksDBFactory.getInstance().getLiveSstFiles(dbName)); + } + + /** Testable seam: caller supplies the live-file set instead of reading the RocksDB singleton. */ + boolean ensureLiveSetUploaded(CloudStorageProvider provider, String dbName, + List liveFiles) { + // Fast path: single lock acquisition, zero cloud I/O — the common case. + if (syncTracker.allConfirmed(dbName, liveFiles)) { + return true; + } + + // Slow path: one or more live files not yet confirmed — find and upload them. + log.info("Checking live set durability: db={}, liveFileCount={}", dbName, liveFiles.size()); + + java.util.List unconfirmedFiles = new java.util.ArrayList<>(); + boolean allDurable = true; + + for (LiveSstFile live : liveFiles) { + String localPath = live.getAbsolutePath(); + if (syncTracker.isConfirmed(dbName, localPath)) { + continue; + } + unconfirmedFiles.add(localPath); + Path localFile = Paths.get(localPath); + String remoteKey = toRelativeKey(localPath); + if (!Files.exists(localFile)) { + // File absent locally and absent from bitmap. Startup hydration (preHydrateDbFiles) + // seeds the bitmap from the cloud listing, so reaching here means this file is + // genuinely not durable in cloud — hold the delete. + allDurable = false; + log.warn("Delete guard: live file absent locally and not confirmed in cloud: " + + "db={}, filePath={}", dbName, localPath); + continue; + } + try { + // Upload directly — no fileExists probe (idempotent PUT is cheaper than a probe). + provider.uploadFile(localPath, remoteKey); + syncTracker.markConfirmed(dbName, localPath); + } catch (Exception e) { + allDurable = false; + log.warn("Delete guard: failed to upload live file: db={}, filePath={}, error={}", + dbName, localPath, e.getMessage()); + if (retryQueue != null) { + retryQueue.submit(dbName, live.getCfName(), localPath, remoteKey, e); + } + } + } + + if (!unconfirmedFiles.isEmpty()) { + CloudStorageMetrics.incrementDeleteGuardReupload(dbName); + if (allDurable) { + log.info("Re-uploaded {} previously unconfirmed live files (delete proceeding): " + + "db={}, files={}", unconfirmedFiles.size(), dbName, unconfirmedFiles); + } else { + log.warn("Re-upload attempted for {} unconfirmed live files (some still not durable, " + + "delete held): db={}, files={}", unconfirmedFiles.size(), dbName, + unconfirmedFiles); + } + } + + return allDurable; + } + + /** + * A compaction changed this DB's live SST set; mirror metadata immediately (event-driven). + * + *

Note: RocksDB delivers the DB directory path here (from {@code db.getName()}), + * not the logical name, so we resolve it against {@link #dbNameByDir}. + */ + @Override + public void onCompacted(String dbNameOrPath) { + CloudStorageProvider provider = CloudStorageProviderFactory.getActiveProvider(); + if (provider == null) { + return; + } + String normalised = Paths.get(dbNameOrPath).toAbsolutePath().normalize().toString(); + String dbName = dbNameByDir.getOrDefault(normalised, dbNameOrPath); + // Skip metadata sync during truncation or grace period to allow purge to complete cleanly + if (!truncatingDbs.contains(dbName) && !isInTruncationGracePeriod(dbName)) { + syncMetadataSnapshotInline(provider, dbName); + } + } + + // ----------------------------------------------------------------------- + // Metadata mirroring + // ----------------------------------------------------------------------- + + /** Records the resolved DB directory for a logical DB name (idempotent). */ + private void recordDb(String dbName, String dbDir) { + if (dbName == null || dbDir == null) { + return; + } + String normalised = Paths.get(dbDir).toAbsolutePath().normalize().toString(); + dbNameByDir.put(normalised, dbName); + } + + /** Absolute, normalised parent directory of a file path (the DB dir of an SST). */ + private String parentDir(String filePath) { + Path parent = Paths.get(filePath).toAbsolutePath().normalize().getParent(); + return parent == null ? null : parent.toString(); + } + + + /** + * Captures a consistent metadata snapshot and mirrors it to cloud without first flushing the + * MemTable. Safe to call from any thread, including a RocksDB event/compaction callback. + * + *

By the time a compaction event fires, the MANIFEST already reflects the post-compaction + * live SST set — a checkpoint captures that consistent state without needing a flush. In + * {@code wal} mode the WAL tail is included in the checkpoint, so un-flushed data is still + * recoverable. In {@code flush} mode un-flushed MemTable data written since the last sync is + * not captured, but that is acceptable here because the goal is to publish a MANIFEST that + * does not reference any cloud object that is about to be deleted. + * + *

Package-private for testability. + */ + boolean syncMetadataSnapshotInline(CloudStorageProvider provider, String dbName) { + MetadataSnapshot snapshot = RocksDBFactory.getInstance().captureMetadataSnapshot(dbName); + if (snapshot == null) { + return false; + } + try { + return uploadMetadataSnapshot(provider, dbName, snapshot); + } finally { + snapshot.cleanup(); + } + } + + /** + * Uploads a captured metadata snapshot to cloud in strict consistency order: + *

    + *
  1. every SST the captured manifest references (from the checkpoint's hard-links);
    2. + *
  2. the mirrored WAL {@code *.log} tail ({@code wal} mode only);
    3. + *
  3. the {@code OPTIONS-*} and {@code MANIFEST-} blobs;
    4. + *
  4. last, {@code CURRENT} — the pointer — so a restore that fetches {@code CURRENT} + * never sees it referencing a manifest whose SSTs are not all in cloud;
    5. + *
  5. only then prune superseded remote {@code MANIFEST-*}/{@code OPTIONS-*}/{@code *.log}.
    6. + *
+ * If any referenced SST cannot be made durable, the method aborts before publishing + * {@code MANIFEST}/{@code CURRENT} and returns {@code false}, leaving the previous durable + * generation intact. + */ + boolean uploadMetadataSnapshot(CloudStorageProvider provider, String dbName, + MetadataSnapshot snapshot) { + String dbDir = snapshot.getDbDir(); + String tempDir = snapshot.getTempDir(); + + // (1) Confirm every manifest-referenced SST is present in cloud. + if (!ensureSnapshotSstsUploaded(provider, dbName, snapshot)) { + log.warn("Cloud metadata-sync: SST set not fully durable, holding metadata publish " + + "for db={}", dbName); + return false; + } + + try { + // (2) WAL tail (wal mode only). + if (walModeEnabled) { + for (String walName : snapshot.getWalFileNames()) { + uploadMetaFile(provider, dbDir, tempDir, walName); + } + } + // (3) OPTIONS-* then MANIFEST-. + for (String optionsName : snapshot.getOptionsFileNames()) { + uploadMetaFile(provider, dbDir, tempDir, optionsName); + } + if (snapshot.getManifestFileName() != null) { + uploadMetaFile(provider, dbDir, tempDir, snapshot.getManifestFileName()); + } + // (4) CURRENT last — the atomic pointer publish. + if (snapshot.getCurrentFileName() != null) { + uploadMetaFile(provider, dbDir, tempDir, snapshot.getCurrentFileName()); + } + } catch (IOException e) { + log.warn("Cloud metadata-sync: failed to publish metadata for db={}: {}", + dbName, e.getMessage()); + return false; + } + + // (5) Prune superseded remote metadata now that the new CURRENT is durable. + pruneRemoteMetadata(provider, dbDir, snapshot); + log.debug("Cloud metadata-sync published: db={}, manifest={}, options={}, wal={}", + dbName, snapshot.getManifestFileName(), snapshot.getOptionsFileNames().size(), + walModeEnabled ? snapshot.getWalFileNames().size() : 0); + return true; + } + + /** + * Ensures every SST the captured manifest references (the checkpoint's hard-linked {@code *.sst} + * set) is present in cloud, uploading from the hard-link when not. The hard-link pins the + * content even if compaction removed the original, so this cannot race a concurrent delete. + */ + private boolean ensureSnapshotSstsUploaded(CloudStorageProvider provider, String dbName, + MetadataSnapshot snapshot) { + String dbDir = snapshot.getDbDir(); + String tempDir = snapshot.getTempDir(); + boolean allDurable = true; + for (String sstName : snapshot.getSstFileNames()) { + String realPath = joinPath(dbDir, sstName); + String remoteKey = toRelativeKey(realPath); + if (syncTracker.isConfirmed(dbName, realPath)) { + continue; + } + try { + if (provider.fileExists(remoteKey)) { + syncTracker.markConfirmed(dbName, realPath); + continue; + } + provider.uploadFile(joinPath(tempDir, sstName), remoteKey); + syncTracker.markConfirmed(dbName, realPath); + } catch (Exception e) { + allDurable = false; + log.warn("Cloud metadata-sync: failed to confirm SST db={}, key={}, reason={}", + dbName, remoteKey, e.getMessage()); + if (retryQueue != null && Files.exists(Paths.get(realPath))) { + retryQueue.submit(dbName, null, realPath, remoteKey, e); + } + } + } + return allDurable; + } + + /** Uploads one metadata file from the checkpoint temp dir to its real-path-derived remote key. */ + private void uploadMetaFile(CloudStorageProvider provider, String dbDir, String tempDir, + String fileName) throws IOException { + provider.uploadFile(joinPath(tempDir, fileName), toRelativeKey(joinPath(dbDir, fileName))); + } + + /** + * Deletes remote {@code MANIFEST-*}/{@code OPTIONS-*} (and, always, {@code *.log}) objects for + * this DB that are not part of the just-published snapshot, bounding remote metadata growth. + * {@code CURRENT} and {@code *.sst} are never pruned here (SST lifecycle is the delete guard's). + */ + private void pruneRemoteMetadata(CloudStorageProvider provider, String dbDir, + MetadataSnapshot snapshot) { + Set keep = new HashSet<>(); + if (snapshot.getManifestFileName() != null) { + keep.add(snapshot.getManifestFileName()); + } + keep.addAll(snapshot.getOptionsFileNames()); + if (walModeEnabled) { + keep.addAll(snapshot.getWalFileNames()); + } + String prefix = dbPrefix(dbDir); + List remoteKeys; + try { + remoteKeys = provider.listFiles(prefix.endsWith("/") ? prefix : prefix + "/"); + } catch (IOException e) { + log.debug("Cloud metadata-sync prune: list failed for prefix={}: {}", + prefix, e.getMessage()); + return; + } + for (String remoteKey : remoteKeys) { + int slash = remoteKey.lastIndexOf('/'); + String base = slash >= 0 ? remoteKey.substring(slash + 1) : remoteKey; + boolean isSupersededMeta = (base.startsWith("MANIFEST-") || base.startsWith("OPTIONS-") + || base.endsWith(".log")) && !keep.contains(base); + if (!isSupersededMeta) { + continue; + } + try { + provider.deleteFile(remoteKey); + log.debug("Cloud metadata-sync prune: deleted superseded {}", remoteKey); + } catch (IOException e) { + log.debug("Cloud metadata-sync prune: delete failed for {}: {}", + remoteKey, e.getMessage()); + } + } + } + + /** Joins a directory and a file name with the platform separator. */ + private static String joinPath(String dir, String name) { + return dir.endsWith(File.separator) || dir.endsWith("/") + ? dir + name + : dir + File.separator + name; + } + + // ----------------------------------------------------------------------- + // Internal helpers + // ----------------------------------------------------------------------- + + /** + * Converts an absolute local file path to a remote key by stripping the data-root prefix. + * + * 
+     *   dataRoot = /hugegraph-store/storage
+     *   filePath = /hugegraph-store/storage/hgstore-metadata/000008.sst
+     *   result   = hgstore-metadata/000008.sst
+     *
+ * + * If {@code filePath} does not start with {@code dataRoot} the leading slash is simply + * stripped so the key is still valid (though possibly not ideallyformatted). + */ + String toRelativeKey(String filePath) { + String relative; + if (filePath.startsWith(dataRoot)) { + String rel = filePath.substring(dataRoot.length()); + // Strip leading separator produced by the substring. + relative = rel.startsWith("/") || rel.startsWith(File.separator) + ? rel.substring(1) + : rel; + return withStoreScope(relative); + } + // Fallback: strip any leading slash so the key does not start with '/'. + relative = filePath.startsWith("/") ? filePath.substring(1) : filePath; + return withStoreScope(relative); + } + + /** + * Walks {@code dbPath} and uploads every {@code *.sst} file that is not already + * present in cloud storage. This handles restarts where SST files from a previous + * run were never uploaded (e.g. cloud storage was enabled after the last shutdown). + */ + private void uploadExistingSstFiles(CloudStorageProvider provider, String dbName, + String dbPath) { + Path root = Paths.get(dbPath); + if (!root.toFile().isDirectory()) { + return; + } + try (Stream paths = Files.walk(root)) { + paths.filter(p -> p.toString().endsWith(".sst")) + .forEach(p -> { + String localPath = p.toString(); + String remoteKey = toRelativeKey(localPath); + try { + if (!provider.fileExists(remoteKey)) { + provider.uploadFile(localPath, remoteKey); + log.info("Cloud initial-upload: {} -> {}", localPath, remoteKey); + } + } catch (IOException e) { + throw new IllegalStateException( + String.format("Cloud initial-upload failed for db=%s path=%s", + dbName, localPath), e); + } + }); + } catch (IOException e) { + throw new IllegalStateException( + String.format("Cloud initial-upload scan failed for db=%s dbPath=%s", + dbName, dbPath), e); + } + } + + private void preHydrateDbFiles(CloudStorageProvider provider, String dbName, String dbPath) { + Path root = Paths.get(dbPath); + try { + Files.createDirectories(root); + } catch (IOException e) { + throw new IllegalStateException( + String.format("Cloud pre-hydration mkdir failed for db=%s path=%s", + dbName, dbPath), e); + } + + CloudStorageMetrics.registerDatabaseMetrics(dbName); + + String prefix = dbPrefix(dbPath); + + // STALE-DATA GUARD: if the remote prefix carries a _DELETED tombstone, the previous + // generation of this DB was destroyed but cloud objects were not fully removed. Hydrating + // them would silently resurrect deleted data in the new DB. Instead, skip hydration, + // trigger a best-effort remote purge so the prefix is clean, and start fresh. + String tombstoneKey = prefix + "/" + DB_TOMBSTONE_FILE; + try { + if (provider.fileExists(tombstoneKey)) { + log.warn("Cloud pre-hydration skipped: tombstone found for db={} — previous " + + "generation was deleted. Purging stale remote objects.", dbName); + purgeRemotePrefix(provider, dbName, prefix); + return; + } + } catch (IOException e) { + // Tombstone check itself failed — cannot safely determine generation; skip hydration. + log.warn("Cloud pre-hydration: tombstone check failed for db={}, skipping to avoid " + + "stale-data risk: {}", dbName, e.getMessage()); + return; + } + + List remoteFiles = listRemoteKeys(provider, prefix); + if (remoteFiles.isEmpty()) { + log.debug("Cloud pre-hydration skipped: no remote files for db={} prefix={}", + dbName, prefix); + return; + } + + int downloaded = 0; + for (String remoteKey : remoteFiles) { + Path localPath = resolveLocalPath(remoteKey); + if (Files.exists(localPath)) { + continue; + } + try { + Files.createDirectories(localPath.getParent()); + provider.downloadFile(remoteKey, localPath.toString()); + downloaded++; + } catch (IOException e) { + throw new IllegalStateException( + String.format("Cloud pre-hydration failed for db=%s key=%s", + dbName, remoteKey), e); + } + } + if (downloaded > 0) { + log.info("Pre-hydration completed: db={}, downloadedFiles={}", dbName, downloaded); + } + + int confirmed = 0; + for (String remoteKey : remoteFiles) { + if (remoteKey.endsWith(".sst")) { + syncTracker.markConfirmed(dbName, resolveLocalPath(remoteKey).toString()); + confirmed++; + } + } + if (confirmed > 0) { + log.info("Seeded sync-tracker bitmap with {} confirmed files from remote listing: " + + "db={}", confirmed, dbName); + } + + verifyMetadataConsistency(dbName, root); + } + + /** + * Consistent-restore guard: if a {@code CURRENT} pointer was mirrored, the {@code MANIFEST-} + * it references must be present locally after pre-hydration. Opening RocksDB with a + * {@code CURRENT} pointing at a missing manifest would silently yield an empty / partial DB, so + * we fail loudly instead. (A manifest-referenced SST that is absent is caught loudly by + * RocksDB's own open, which refuses to start on a missing referenced file.) + */ + private void verifyMetadataConsistency(String dbName, Path dbRoot) { + Path current = dbRoot.resolve("CURRENT"); + if (!Files.exists(current)) { + // No mirrored metadata (or a genuinely empty prefix) — nothing to verify. + return; + } + String manifestName; + try { + manifestName = Files.readString(current).trim(); + } catch (IOException e) { + throw new IllegalStateException( + String.format("Cloud restore: unreadable CURRENT for db=%s", dbName), e); + } + if (manifestName.isEmpty() || !Files.exists(dbRoot.resolve(manifestName))) { + throw new IllegalStateException(String.format( + "Cloud restore inconsistent for db=%s: CURRENT references manifest '%s' which is " + + "absent locally and in cloud. Refusing to open a partial DB.", + dbName, manifestName)); + } + } + + private List listRemoteKeys(CloudStorageProvider provider, String prefix) { + try { + return provider.listFiles(prefix.endsWith("/") ? prefix : prefix + "/"); + } catch (IOException e) { + throw new IllegalStateException( + String.format("Cloud list failed for prefix=%s", prefix), e); + } + } + + private String dbPrefix(String dbPath) { + String relative = toRelativeKey(dbPath); + return relative.endsWith("/") ? relative.substring(0, relative.length() - 1) : relative; + } + + private Path resolveLocalPath(String remoteKey) { + String relative = stripStoreScope(remoteKey); + Path root = Paths.get(this.dataRoot); + Path local = root.resolve(relative).normalize(); + if (!local.startsWith(root)) { + throw new IllegalArgumentException("Invalid remote key outside data root: " + remoteKey); + } + return local; + } + + private String withStoreScope(String relativeKey) { + if (storeScopePrefix.isEmpty()) { + return relativeKey; + } + if (relativeKey == null || relativeKey.isEmpty()) { + return storeScopePrefix; + } + return storeScopePrefix + "/" + relativeKey; + } + + private String stripStoreScope(String remoteKey) { + if (storeScopePrefix.isEmpty()) { + return remoteKey; + } + String scopedPrefix = storeScopePrefix + "/"; + if (remoteKey.startsWith(scopedPrefix)) { + return remoteKey.substring(scopedPrefix.length()); + } + if (remoteKey.equals(storeScopePrefix)) { + return ""; + } + throw new IllegalArgumentException(String.format( + "Remote key '%s' does not belong to store scope '%s'", remoteKey, storeScopePrefix)); + } + + private static String normaliseKeyPrefix(String prefix) { + if (prefix == null) { + return ""; + } + String trimmed = prefix.trim(); + if (trimmed.isEmpty()) { + return ""; + } + String normalized = trimmed.replace('\\', '/'); + while (normalized.startsWith("/")) { + normalized = normalized.substring(1); + } + while (normalized.endsWith("/")) { + normalized = normalized.substring(0, normalized.length() - 1); + } + return normalized; + } + + /** + * Triggers a non-blocking MemTable flush for the named DB via {@link RocksDBFactory}. + * The call returns immediately (fire-and-forget); when RocksDB completes the flush it + * fires {@link #onTableFileCreated} via its event-listener mechanism, which then uploads + * the resulting SST file to cloud storage. The overall flow is event-driven, not async + * in the sense of a background thread managed by this class. + */ + private void flushDb(String dbName) { + try { + RocksDBFactory.getInstance().flushSession(dbName, false); + log.debug("Cloud storage: triggered flush (non-blocking, event-driven) for db={}", dbName); + } catch (Exception e) { + log.warn("Cloud storage: flush failed for db={}: {}", dbName, e.getMessage()); + } + } + + boolean shouldAttemptReadMissHydration(String dbName, String table) { + if (readMissGuardWindowMs <= 0) { + return true; + } + long now = System.currentTimeMillis(); + String guardKey = dbName + "::" + table; + Long prev = readMissAttemptTs.put(guardKey, now); + if (prev == null) { + return true; + } + long elapsed = now - prev; + if (elapsed >= readMissGuardWindowMs) { + return true; + } + log.debug("Skip read-miss hydration due to guard window: db={}, table={}, elapsedMs={}", + dbName, table, elapsed); + return false; + } +} diff --git a/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudStorageMetrics.java b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudStorageMetrics.java new file mode 100644 index 0000000000..8dc6f425cf --- /dev/null +++ b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudStorageMetrics.java @@ -0,0 +1,182 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.node.cloud; + +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.atomic.AtomicLong; + +import org.apache.hugegraph.store.node.util.HgAssert; + +import io.micrometer.core.instrument.Counter; +import io.micrometer.core.instrument.Gauge; +import io.micrometer.core.instrument.MeterRegistry; +import io.micrometer.core.instrument.Timer; +import lombok.extern.slf4j.Slf4j; + +/** + * Cloud Storage Metrics registration and management. + * Exposes metrics for cloud storage operations including upload success/failure, + * sync latency, confirmed files (SSTs synced to cloud), and delete guard re-upload activity. + */ +@Slf4j +public final class CloudStorageMetrics { + + private static MeterRegistry registry; + private static CloudSyncTracker syncTracker; + + // Per-database counters for delete guard re-uploads + private static final ConcurrentHashMap deleteGuardReuploadPerDb = + new ConcurrentHashMap<>(); + + // Overall upload failure counter (will be incremented with tags) + private static Counter uploadFailuresCounter; + + // Sync latency timer + private static Timer syncLatencyTimer; + + private CloudStorageMetrics() { + } + + /** + * Initialize cloud storage metrics. + * Must be called once at application startup with MeterRegistry and CloudSyncTracker instances. + */ + public static void init(final MeterRegistry meterRegistry, final CloudSyncTracker tracker) { + HgAssert.isArgumentNotNull(meterRegistry, "meterRegistry"); + HgAssert.isArgumentNotNull(tracker, "CloudSyncTracker"); + + if (registry != null) { + return; + } + + registry = meterRegistry; + syncTracker = tracker; + + // Register global metrics + uploadFailuresCounter = Counter.builder(CloudStorageMetricsConst.UPLOAD_FAILURES) + .description("Total upload failures (transient and permanent)") + .register(registry); + + syncLatencyTimer = Timer.builder(CloudStorageMetricsConst.SYNC_LATENCY_MS) + .description("Time in milliseconds from SST file creation to cloud confirmation") + .publishPercentiles(0.5, 0.95, 0.99) + .register(registry); + + // Register gauge for retry queue size (monitored at runtime) + Gauge.builder(CloudStorageMetricsConst.RETRY_QUEUE_SIZE, () -> 0) + .description("Number of files waiting in the upload retry queue") + .register(registry); + + log.info("Cloud storage metrics initialized"); + } + + /** + * Register a per-database metrics gauges. + * Should be called when a database opens for the first time. + */ + public static void registerDatabaseMetrics(final String dbName) { + if (registry == null) { + return; + } + + try { + // Register confirmed files gauge for this database (files successfully synced to cloud) + Gauge.builder(CloudStorageMetricsConst.CONFIRMED_FILES, + () -> getConfirmedFileCount(dbName)) + .description("Number of SST files confirmed present in cloud storage") + .tag(CloudStorageMetricsConst.TAG_DB_NAME, dbName) + .register(registry); + + // Register delete guard re-upload counter for this database + deleteGuardReuploadPerDb.putIfAbsent(dbName, new AtomicLong(0)); + + Gauge.builder(CloudStorageMetricsConst.DELETE_GUARD_REUPLOAD_COUNT, + () -> getDeleteGuardReuploadCount(dbName)) + .description("Number of times delete guard re-uploaded unconfirmed files during " + + "compaction") + .tag(CloudStorageMetricsConst.TAG_DB_NAME, dbName) + .register(registry); + + log.debug("Registered cloud storage metrics for database: {}", dbName); + } catch (IllegalArgumentException e) { + // Gauge already registered for this database + log.debug("Cloud storage metrics already registered for database: {}", dbName); + } + } + + /** + * Record a sync latency measurement (time to confirm upload to cloud). + */ + @SuppressWarnings("unused") // dbName reserved for future per-db latency tracking + public static void recordSyncLatency(final String dbName, final long latencyMs) { + if (syncLatencyTimer == null) { + return; + } + syncLatencyTimer.record(latencyMs, java.util.concurrent.TimeUnit.MILLISECONDS); + } + + /** + * Record an upload failure. + */ + @SuppressWarnings("unused") // Parameters reserved for future per-db/cf failure tracking + public static void recordUploadFailure(final String dbName, final String cfName, + final String errorType) { + if (uploadFailuresCounter == null) { + return; + } + uploadFailuresCounter.increment(); + } + + /** + * Get the current count of confirmed SST files for a database (from bitmap). + */ + public static long getConfirmedFileCount(final String dbName) { + if (syncTracker == null) { + return 0; + } + return syncTracker.confirmedCount(dbName); + } + + /** + * Increment the delete guard re-upload count for a database. + */ + public static void incrementDeleteGuardReupload(final String dbName) { + deleteGuardReuploadPerDb.computeIfAbsent(dbName, k -> new AtomicLong(0)).incrementAndGet(); + } + + /** + * Get the delete guard re-upload count for a database. + */ + public static long getDeleteGuardReuploadCount(final String dbName) { + AtomicLong count = deleteGuardReuploadPerDb.get(dbName); + return count != null ? count.get() : 0; + } + + /** + * Update retry queue size (called by CloudUploadRetryQueue). + */ + @SuppressWarnings("unused") // Placeholder for future dynamic retry queue size tracking + public static void setRetryQueueSize(final int size) { + // This is exposed as a gauge; the actual value is updated externally + } +} + + + + + diff --git a/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudStorageMetricsConst.java b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudStorageMetricsConst.java new file mode 100644 index 0000000000..a614209527 --- /dev/null +++ b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudStorageMetricsConst.java @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.node.cloud; + +/** + * Constants for cloud storage metrics. + * Defines metric names and tags used for monitoring cloud storage operations. + */ +public final class CloudStorageMetricsConst { + + // Prefix used for all cloud storage metrics + public static final String PREFIX = "cloud_storage"; + + // Metric names + public static final String CONFIRMED_FILES = PREFIX + "_confirmed_files_total"; + public static final String UPLOAD_FAILURES = PREFIX + "_upload_failures_total"; + public static final String RETRY_QUEUE_SIZE = PREFIX + "_retry_queue_size"; + public static final String SYNC_LATENCY_MS = PREFIX + "_sync_latency_ms"; + public static final String DELETE_GUARD_REUPLOAD_COUNT = PREFIX + "_delete_guard_reupload_count"; + + // Tag names + public static final String TAG_DB_NAME = "db_name"; + + private CloudStorageMetricsConst() { + } +} + + diff --git a/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudSyncTracker.java b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudSyncTracker.java new file mode 100644 index 0000000000..6d32e9fcfd --- /dev/null +++ b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudSyncTracker.java @@ -0,0 +1,165 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.node.cloud; + +import java.io.File; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; + +import org.apache.hugegraph.rocksdb.access.RocksDBFactory.LiveSstFile; +import org.roaringbitmap.longlong.Roaring64NavigableMap; + +/** + * Tracks which RocksDB SST files are confirmed present in cloud storage. + * + *

SST file names are monotonic per-DB file numbers ({@code 000123.sst}), so the natural key is + * the integer file number rather than the string path. Sync state is kept as one + * {@link Roaring64NavigableMap} per DB — each set bit means "SST file number N for this DB is + * confirmed present in cloud". Roaring bitmaps stay compact even as low numbers are cleared after + * old files are deleted, and file numbers are unique across column families within a DB, so a + * single bitmap per DB is sufficient (which fits RocksDB delete events not carrying a CF name) + * + *

This tracker is the linchpin of the delete-guard invariant: a superseded cloud object is + * deleted only once every live SST file of that DB is confirmed here. + * + *

{@link Roaring64NavigableMap} is not thread-safe, so all access to a per-DB bitmap is + * synchronized on the bitmap instance. + */ +public final class CloudSyncTracker { + + private final Map confirmedByDb = new ConcurrentHashMap<>(); + + /** + * Parses the SST file number from a file path such as {@code /data/db/000123.sst}. + * + * @return the file number, or {@code -1} if the path is not a parseable {@code *.sst} file + */ + public static long parseSstFileNumber(String filePath) { + if (filePath == null || !filePath.endsWith(".sst")) { + return -1L; + } + int slash = Math.max(filePath.lastIndexOf('/'), filePath.lastIndexOf(File.separatorChar)); + String name = filePath.substring(slash + 1, filePath.length() - ".sst".length()); + if (name.isEmpty()) { + return -1L; + } + // RocksDB SST base names are pure digits (e.g. 000123). Reject anything else. + try { + return Long.parseLong(name); + } catch (NumberFormatException e) { + return -1L; + } + } + + private Roaring64NavigableMap bitmap(String dbName) { + return confirmedByDb.computeIfAbsent(dbName, k -> new Roaring64NavigableMap()); + } + + /** Marks the SST file as confirmed present in cloud. No-op for non-SST paths. */ + public void markConfirmed(String dbName, String filePath) { + long number = parseSstFileNumber(filePath); + if (number < 0) { + return; + } + Roaring64NavigableMap bm = bitmap(dbName); + synchronized (bm) { + bm.addLong(number); + } + } + + /** Returns whether the SST file is confirmed present in cloud. */ + public boolean isConfirmed(String dbName, String filePath) { + long number = parseSstFileNumber(filePath); + if (number < 0) { + return false; + } + Roaring64NavigableMap bm = confirmedByDb.get(dbName); + if (bm == null) { + return false; + } + synchronized (bm) { + return bm.contains(number); + } + } + + /** Clears the confirmed bit for a file (called after the cloud object is deleted). */ + public void clearConfirmed(String dbName, String filePath) { + long number = parseSstFileNumber(filePath); + if (number < 0) { + return; + } + Roaring64NavigableMap bm = confirmedByDb.get(dbName); + if (bm == null) { + return; + } + synchronized (bm) { + bm.removeLong(number); + } + } + + /** + * Returns {@code true} if every file in {@code liveFiles} is confirmed present + * in cloud. + * + *

This is the fast-path check used by the delete guard: it acquires the per-DB lock + * once for the entire set and short-circuits on the first miss, so a DB whose + * live set is fully confirmed costs one lock acquisition regardless of set size — compared + * to N acquisitions with N individual {@link #isConfirmed} calls. + * + * @param liveFiles the current live SST file set obtained from RocksDB's manifest + * @return {@code true} iff every SST path in {@code liveFiles} has its confirmed bit set + */ + public boolean allConfirmed(String dbName, List liveFiles) { + if (liveFiles.isEmpty()) { + return true; + } + Roaring64NavigableMap bm = confirmedByDb.get(dbName); + if (bm == null) { + return false; + } + synchronized (bm) { + for (LiveSstFile live : liveFiles) { + long num = parseSstFileNumber(live.getAbsolutePath()); + if (num >= 0 && !bm.contains(num)) { + return false; // short-circuit on first unconfirmed file + } + } + } + return true; + } + + /** + * Removes all confirmed-sync state for the named database. + * Called when a DB is destroyed so the bitmap does not leak memory or bleed into a recreated DB. + */ + public void clearDb(String dbName) { + confirmedByDb.remove(dbName); + } + + /** Number of confirmed SST files for a DB (testing / monitoring). */ + public long confirmedCount(String dbName) { + Roaring64NavigableMap bm = confirmedByDb.get(dbName); + if (bm == null) { + return 0L; + } + synchronized (bm) { + return bm.getLongCardinality(); + } + } +} diff --git a/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudUploadRetryQueue.java b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudUploadRetryQueue.java new file mode 100644 index 0000000000..e4322bb49a --- /dev/null +++ b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/CloudUploadRetryQueue.java @@ -0,0 +1,547 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.node.cloud; + +import java.io.BufferedReader; +import java.io.Closeable; +import java.io.IOException; +import java.io.PrintWriter; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.nio.file.StandardOpenOption; +import java.util.ArrayList; +import java.util.Deque; +import java.util.List; +import java.util.concurrent.ConcurrentLinkedDeque; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +import org.apache.hugegraph.store.cloud.CloudStorageProvider; +import org.apache.hugegraph.store.cloud.CloudStorageNonRetryableException; +import org.apache.hugegraph.store.cloud.CloudStorageProviderFactory; + +import lombok.extern.slf4j.Slf4j; + +/** + * Asynchronous retry queue with a file-backed dead-letter queue (DLQ) for failed + * cloud SST uploads. + * + *

Retry strategy

+ *
    + *
  • When {@code maxAttempts == 0} (default), failures go directly to the DLQ with no + * whole-file retry. The provider is expected to handle its own retries internally + * (e.g. S3 multipart-part-retry).
    • + *
  • When {@code maxAttempts > 0}, the first retry is scheduled after + * {@code initialDelayMs} milliseconds.
    • + *
  • Subsequent retries use exponential backoff: {@code delay = initialDelayMs * 2^(attempt-1)}, + * capped at {@code maxDelayMs}.
    • + *
  • After {@code maxAttempts} consecutive failures the task is moved to the DLQ.
    • + *
  • If the local SST file no longer exists at retry time (compacted away by RocksDB) + * the retry is silently dropped — there is nothing to upload.
    • + *
+ * + *

Dead-letter queue (DLQ)

+ *
    + *
  • In-memory: {@link #getDlqEntries()} returns a snapshot.
    • + *
  • On-disk: entries are appended to {@code /.cloud-upload-dlq.tsv} in a + * tab-separated format so they survive JVM restarts. Existing entries are loaded + * back into memory at construction time.
    • + *
  • {@link #replayDlq()} retries all DLQ entries immediately; successful uploads are + * removed and the file is rewritten. Failed entries are re-submitted to the retry + * queue with a fresh attempt cycle.
    • + *
+ * + *

Thread safety

+ * All public methods are thread-safe. The in-memory DLQ is a {@link ConcurrentLinkedDeque}; + * disk I/O is serialised onto the internal {@link ScheduledExecutorService} (for appends) + * or synchronised explicitly (for rewrites). + * + *

Lifecycle

+ * Call {@link #close()} (e.g. from a Spring {@code @PreDestroy} method) to shut down the + * background executor gracefully. + */ +@Slf4j +public class CloudUploadRetryQueue implements Closeable { + + // ----------------------------------------------------------------------- + // Constants + // ----------------------------------------------------------------------- + + /** Name of the on-disk DLQ file, relative to {@code dataRoot}. */ + static final String DLQ_FILE_NAME = ".cloud-upload-dlq.tsv"; + + private static final String DLQ_COMMENT = + "# Cloud SST upload dead-letter queue – tab-separated: " + + "failedAt\\tattemptCount\\tdbName\\tcfName\\tfilePath\\tremoteKey\\tlastError"; + + // ----------------------------------------------------------------------- + // Configuration + // ----------------------------------------------------------------------- + + private final int maxAttempts; + private final long initialDelayMs; + private final long maxDelayMs; + + /** + * Invoked with {@code (dbName, filePath)} whenever an upload succeeds via retry or DLQ replay, + * so the caller can mark the file confirmed-present in cloud. May be a no-op. + */ + private final java.util.function.BiConsumer onUploadConfirmed; + + // ----------------------------------------------------------------------- + // State + // ----------------------------------------------------------------------- + + /** In-memory dead-letter queue – holds tasks that exhausted all retries. */ + private final Deque dlq = new ConcurrentLinkedDeque<>(); + + /** Absolute path of the on-disk DLQ file. */ + private final Path dlqFile; + + /** Background thread pool used to schedule retries. */ + private final ScheduledExecutorService scheduler; + + /** Counter of in-flight retry tasks (for testing / monitoring). */ + private final AtomicInteger inFlightCount = new AtomicInteger(0); + + // ----------------------------------------------------------------------- + // Constructor + // ----------------------------------------------------------------------- + + /** + * @param maxAttempts maximum whole-file upload retries after a first failure. + * {@code 0} means no retries – failures go directly to DLQ + * (provider is expected to handle its own retries internally). + * Positive values enable exponential-backoff whole-file retries. + * @param initialDelayMs delay before the first retry, in milliseconds; clamped to ≥ 100. + * Ignored when {@code maxAttempts == 0}. + * @param maxDelayMs upper bound for exponential backoff delay; clamped to + * ≥ {@code initialDelayMs}. Ignored when {@code maxAttempts == 0}. + * @param dataRoot absolute path of the store's data directory – the DLQ file is + * written here + */ + public CloudUploadRetryQueue(int maxAttempts, long initialDelayMs, + long maxDelayMs, String dataRoot) { + this(maxAttempts, initialDelayMs, maxDelayMs, dataRoot, null); + } + + /** + * @param onUploadConfirmed callback invoked with {@code (dbName, filePath)} on every successful + * retry / DLQ-replay upload; pass {@code null} for none. + */ + public CloudUploadRetryQueue(int maxAttempts, long initialDelayMs, + long maxDelayMs, String dataRoot, + java.util.function.BiConsumer onUploadConfirmed) { + this.maxAttempts = Math.max(0, maxAttempts); + this.initialDelayMs = Math.max(100L, initialDelayMs); + this.maxDelayMs = Math.max(this.initialDelayMs, maxDelayMs); + this.onUploadConfirmed = onUploadConfirmed != null + ? onUploadConfirmed + : (db, path) -> { }; + this.dlqFile = Paths.get(dataRoot, DLQ_FILE_NAME); + + this.scheduler = Executors.newScheduledThreadPool(2, r -> { + Thread t = new Thread(r, "cloud-upload-retry"); + t.setDaemon(true); + return t; + }); + + loadDlqFromDisk(); + } + + // ----------------------------------------------------------------------- + // Public API + // ----------------------------------------------------------------------- + + /** + * Submits a failed upload to the retry queue or directly to the DLQ. + * + *

If {@code maxAttempts == 0} (default), the task goes directly to the DLQ — the + * provider is expected to handle its own retries internally. + * + *

If {@code maxAttempts > 0}, the first retry is scheduled after {@code initialDelayMs} + * ms with exponential backoff. After all attempts are exhausted the task is moved to the DLQ. + * + *

If the file no longer exists at retry time (compacted away by RocksDB) the task is + * silently dropped. + * + * @param dbName RocksDB instance name (partition id) + * @param cfName column-family name + * @param filePath absolute local path of the SST file + * @param remoteKey remote object key (relative to bucket root) + * @param cause the exception from the first upload attempt + */ + public void submit(String dbName, String cfName, String filePath, + String remoteKey, Exception cause) { + if (cause instanceof CloudStorageNonRetryableException || maxAttempts == 0) { + // Non-retryable exception, or retry disabled: go straight to DLQ. + moveToDlq(dbName, cfName, filePath, remoteKey, 0, + cause.getMessage() != null ? cause.getMessage() : "non-retryable"); + return; + } + scheduleRetry(dbName, cfName, filePath, remoteKey, 1); + } + + /** + * Returns an unmodifiable snapshot of all current DLQ entries. + */ + public List getDlqEntries() { + return List.copyOf(dlq); + } + + /** + * Returns the number of entries currently in the DLQ. + */ + public int getDlqSize() { + return dlq.size(); + } + + /** + * Returns the number of retry tasks currently scheduled or executing. + */ + public int getInFlightCount() { + return inFlightCount.get(); + } + + /** + * Replays all DLQ entries immediately by attempting to re-upload each file. + * + *

    + *
  • Entries that succeed are removed from the in-memory DLQ and the DLQ file is + * rewritten without them.
    • + *
  • Entries whose local file no longer exists are silently dropped.
    • + *
  • Entries that fail are re-submitted to the retry queue with a fresh attempt cycle + * (they are removed from the DLQ for now and will re-enter it only if all retries + * fail again).
    • + *
+ */ + public void replayDlq() { + // Drain the whole DLQ atomically so concurrent appends are not affected. + List snapshot = new ArrayList<>(); + FailedUploadTask entry; + while ((entry = dlq.pollFirst()) != null) { + snapshot.add(entry); + } + if (snapshot.isEmpty()) { + log.info("DLQ replay: queue is empty, nothing to do"); + return; + } + log.info("DLQ replay started: {} entries", snapshot.size()); + + int succeeded = 0; + int dropped = 0; + int requeued = 0; + + CloudStorageProvider provider = CloudStorageProviderFactory.getActiveProvider(); + + for (FailedUploadTask task : snapshot) { + if (!Files.exists(Paths.get(task.getFilePath()))) { + log.info("DLQ replay: local file gone, dropping: path={}", task.getFilePath()); + dropped++; + continue; + } + if (provider == null) { + // No provider – re-queue and bail out early. + scheduleRetry(task.getDbName(), task.getCfName(), task.getFilePath(), + task.getRemoteKey(), 1); + requeued++; + continue; + } + try { + provider.uploadFile(task.getFilePath(), task.getRemoteKey()); + onUploadConfirmed.accept(task.getDbName(), task.getFilePath()); + log.info("DLQ replay succeeded: db={}, cf={}, path={}", + task.getDbName(), task.getCfName(), task.getFilePath()); + succeeded++; + } catch (Exception e) { + log.warn("DLQ replay failed, re-queuing: db={}, cf={}, path={}, reason={}", + task.getDbName(), task.getCfName(), task.getFilePath(), e.getMessage()); + scheduleRetry(task.getDbName(), task.getCfName(), task.getFilePath(), + task.getRemoteKey(), 1); + requeued++; + } + } + + // Rewrite the DLQ file to reflect removals (only currently queued DLQ entries remain). + rewriteDlqFile(); + + log.info("DLQ replay finished: succeeded={}, dropped={}, requeued={}", + succeeded, dropped, requeued); + } + + // ----------------------------------------------------------------------- + // Lifecycle + // ----------------------------------------------------------------------- + + /** + * Shuts down the background retry executor, waiting up to 10 seconds for + * in-flight tasks to complete. + */ + @Override + public void close() { + scheduler.shutdown(); + try { + if (!scheduler.awaitTermination(10, TimeUnit.SECONDS)) { + log.warn("CloudUploadRetryQueue: executor did not terminate within 10 s; " + + "forcing shutdown ({} tasks still in flight)", inFlightCount.get()); + scheduler.shutdownNow(); + } + } catch (InterruptedException e) { + scheduler.shutdownNow(); + Thread.currentThread().interrupt(); + } + } + + // ----------------------------------------------------------------------- + // Internal – retry scheduling + // ----------------------------------------------------------------------- + + private void scheduleRetry(String dbName, String cfName, String filePath, + String remoteKey, int attempt) { + long delayMs = computeDelay(attempt); + log.info("Cloud upload retry scheduled: db={}, cf={}, path={}, attempt={}/{}, delayMs={}", + dbName, cfName, filePath, attempt, maxAttempts, delayMs); + inFlightCount.incrementAndGet(); + scheduler.schedule( + () -> executeRetry(dbName, cfName, filePath, remoteKey, attempt), + delayMs, TimeUnit.MILLISECONDS); + } + + private void executeRetry(String dbName, String cfName, String filePath, + String remoteKey, int attempt) { + try { + // If the local SST file is gone (compacted), drop silently – nothing to upload. + if (!Files.exists(Paths.get(filePath))) { + log.info("Cloud upload retry: local file no longer exists (compacted?), " + + "dropping: db={}, cf={}, path={}", dbName, cfName, filePath); + return; + } + + CloudStorageProvider provider = CloudStorageProviderFactory.getActiveProvider(); + if (provider == null) { + log.warn("Cloud upload retry: no active provider, giving up: " + + "db={}, cf={}, path={}", dbName, cfName, filePath); + moveToDlq(dbName, cfName, filePath, remoteKey, attempt, "No active provider"); + return; + } + + provider.uploadFile(filePath, remoteKey); + onUploadConfirmed.accept(dbName, filePath); + log.info("Cloud upload retry succeeded: db={}, cf={}, path={}, attempt={}", + dbName, cfName, filePath, attempt); + + } catch (Exception e) { + log.warn("Cloud upload retry failed: db={}, cf={}, path={}, attempt={}/{}, reason={}", + dbName, cfName, filePath, attempt, maxAttempts, e.getMessage()); + if (e instanceof CloudStorageNonRetryableException) { + moveToDlq(dbName, cfName, filePath, remoteKey, attempt, + e.getMessage() != null ? e.getMessage() : "non-retryable"); + return; + } + if (attempt >= maxAttempts) { + moveToDlq(dbName, cfName, filePath, remoteKey, attempt, e.getMessage()); + } else { + scheduleRetry(dbName, cfName, filePath, remoteKey, attempt + 1); + } + } finally { + inFlightCount.decrementAndGet(); + } + } + + private void moveToDlq(String dbName, String cfName, String filePath, + String remoteKey, int attemptCount, String lastError) { + FailedUploadTask task = new FailedUploadTask(dbName, cfName, filePath, + remoteKey, attemptCount, lastError); + dlq.addLast(task); + appendDlqEntryToDisk(task); + log.error("Cloud upload moved to DLQ after {} attempt(s) – file is local-only: " + + "db={}, cf={}, path={}, remoteKey={}", + attemptCount, dbName, cfName, filePath, remoteKey); + } + + /** + * Exponential back-off: {@code initialDelayMs * 2^(attempt-1)}, capped at {@code maxDelayMs}. + */ + private long computeDelay(int attempt) { + // Guard against overflow for large attempt numbers. + int exp = Math.min(attempt - 1, 30); + long delay = initialDelayMs * (1L << exp); + return Math.min(delay, maxDelayMs); + } + + // ----------------------------------------------------------------------- + // Internal – DLQ persistence + // ----------------------------------------------------------------------- + + /** Loads any persisted DLQ entries from disk at startup. */ + private void loadDlqFromDisk() { + if (!Files.exists(dlqFile)) { + return; + } + int loaded = 0; + try (BufferedReader reader = Files.newBufferedReader(dlqFile, StandardCharsets.UTF_8)) { + String line; + while ((line = reader.readLine()) != null) { + if (line.isBlank() || line.startsWith("#")) { + continue; + } + FailedUploadTask task = deserialize(line); + if (task != null) { + dlq.addLast(task); + loaded++; + } + } + } catch (IOException e) { + log.warn("DLQ: failed to load persisted entries from {}: {}", + dlqFile, e.getMessage()); + } + if (loaded > 0) { + log.warn("DLQ: loaded {} pending failed upload(s) from disk – " + + "call replayDlq() to retry them (file={})", loaded, dlqFile); + } + } + + /** Appends a single DLQ entry to the on-disk file. */ + private void appendDlqEntryToDisk(FailedUploadTask task) { + try { + boolean newFile = !Files.exists(dlqFile); + try (PrintWriter pw = new PrintWriter(Files.newBufferedWriter( + dlqFile, StandardCharsets.UTF_8, + StandardOpenOption.CREATE, + StandardOpenOption.APPEND))) { + if (newFile) { + pw.println(DLQ_COMMENT); + } + pw.println(serialize(task)); + } + } catch (IOException e) { + log.warn("DLQ: failed to persist entry to {}: {}", dlqFile, e.getMessage()); + } + } + + /** Rewrites the DLQ file from the current in-memory DLQ (used after replay). */ + private void rewriteDlqFile() { + try (PrintWriter pw = new PrintWriter(Files.newBufferedWriter( + dlqFile, StandardCharsets.UTF_8, + StandardOpenOption.CREATE, + StandardOpenOption.TRUNCATE_EXISTING))) { + pw.println(DLQ_COMMENT); + dlq.forEach(t -> pw.println(serialize(t))); + } catch (IOException e) { + log.warn("DLQ: failed to rewrite {}: {}", dlqFile, e.getMessage()); + } + } + + // ----------------------------------------------------------------------- + // Internal – serialisation (tab-separated, with escape for special chars) + // ----------------------------------------------------------------------- + + /** + * Serialises a {@link FailedUploadTask} to a single tab-separated line. + * Fields: failedAt, attemptCount, dbName, cfName, filePath, remoteKey, lastError. + */ + String serialize(FailedUploadTask task) { + return task.getFailedAt() + "\t" + + task.getAttemptCount() + "\t" + + escape(task.getDbName()) + "\t" + + escape(task.getCfName()) + "\t" + + escape(task.getFilePath()) + "\t" + + escape(task.getRemoteKey()) + "\t" + + escape(task.getLastError()); + } + + /** Deserialises a line; returns {@code null} and logs a warning on parse errors. */ + FailedUploadTask deserialize(String line) { + String[] parts = line.split("\t", 7); + if (parts.length < 7) { + log.warn("DLQ: skipping malformed line (expected 7 fields, got {}): {}", + parts.length, line); + return null; + } + try { + long failedAt = Long.parseLong(parts[0].trim()); + int attemptCount = Integer.parseInt(parts[1].trim()); + return new FailedUploadTask( + unescape(parts[2]), // dbName + unescape(parts[3]), // cfName + unescape(parts[4]), // filePath + unescape(parts[5]), // remoteKey + failedAt, + attemptCount, + unescape(parts[6]) // lastError + ); + } catch (NumberFormatException e) { + log.warn("DLQ: failed to parse numeric field in line '{}': {}", line, e.getMessage()); + return null; + } + } + + /** Escapes backslash, tab and newline so the serialised form is safe in TSV lines. */ + private static String escape(String s) { + if (s == null || s.isEmpty()) { + return ""; + } + return s.replace("\\", "\\\\") + .replace("\t", "\\t") + .replace("\n", "\\n") + .replace("\r", "\\r"); + } + + private static String unescape(String s) { + if (s == null || s.isEmpty()) { + return ""; + } + // Process escape sequences left-to-right, handling \\ first to avoid double-decode. + StringBuilder sb = new StringBuilder(s.length()); + for (int i = 0; i < s.length(); i++) { + char c = s.charAt(i); + if (c == '\\' && i + 1 < s.length()) { + char next = s.charAt(i + 1); + switch (next) { + case '\\': + sb.append('\\'); + i++; + break; + case 't': + sb.append('\t'); + i++; + break; + case 'n': + sb.append('\n'); + i++; + break; + case 'r': + sb.append('\r'); + i++; + break; + default: + sb.append(c); + break; + } + } else { + sb.append(c); + } + } + return sb.toString(); + } +} + diff --git a/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/FailedUploadTask.java b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/FailedUploadTask.java new file mode 100644 index 0000000000..320fe3cddb --- /dev/null +++ b/hugegraph-store/hg-store-node/src/main/java/org/apache/hugegraph/store/node/cloud/FailedUploadTask.java @@ -0,0 +1,54 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.node.cloud; + +import lombok.Getter; + +/** + * Immutable dead-letter queue entry for a failed SST cloud upload. + */ +@Getter +public final class FailedUploadTask { + + private final String dbName; + private final String cfName; + private final String filePath; + private final String remoteKey; + private final long failedAt; + private final int attemptCount; + private final String lastError; + + public FailedUploadTask(String dbName, String cfName, String filePath, + String remoteKey, int attemptCount, String lastError) { + this(dbName, cfName, filePath, remoteKey, System.currentTimeMillis(), + attemptCount, lastError); + } + + public FailedUploadTask(String dbName, String cfName, String filePath, + String remoteKey, long failedAt, + int attemptCount, String lastError) { + this.dbName = dbName; + this.cfName = cfName; + this.filePath = filePath; + this.remoteKey = remoteKey; + this.failedAt = failedAt; + this.attemptCount = attemptCount; + this.lastError = lastError != null ? lastError : ""; + } + +} diff --git a/hugegraph-store/hg-store-node/src/main/resources/application.yml b/hugegraph-store/hg-store-node/src/main/resources/application.yml index 0b65270608..4c830f2349 100644 --- a/hugegraph-store/hg-store-node/src/main/resources/application.yml +++ b/hugegraph-store/hg-store-node/src/main/resources/application.yml @@ -1,3 +1,4 @@ +#file: noinspection SpringBootApplicationYaml # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with @@ -49,3 +50,34 @@ logging: config: classpath:log4j2-dev.xml level: root: info + +# Cloud storage (disabled by default; see hg-store-dist application.yml for full docs) +cloud: + storage: + enabled: false + provider: s3 + path-prefix: hugegraph + # Whole-file upload retries after a first failure (default 5). Set 0 to disable + # (failures go straight to the DLQ) when the provider has sufficient internal retry. + upload-retry-max-attempts: 3 + upload-retry-initial-delay-ms: 1000 + upload-retry-max-delay-ms: 60000 + # Backpressure high-watermark on the pending-upload backlog; 0 disables. + upload-backpressure-high-watermark: 64 + # Mirror a consistent CURRENT/MANIFEST/OPTIONS[/WAL] snapshot so cloud objects stay + # recoverable (a node that lost its local disk reopens from cloud, not an empty DB). + # Metadata sync is always enabled when cloud storage is enabled. + # Sync is event-triggered by storage events; there is no background interval scheduler. + # WAL durability: 'flush' (force flush before capture; lose only the un-flushed tail on crash) + # or 'wal' (also mirror + replay the WAL tail; lower RPO, more frequent small uploads). + wal-mode: flush + s3: + bucket: + region: + endpoint: + access-key: + secret-key: + multipart-part-retry-max-attempts: 3 + multipart-part-retry-base-backoff-ms: 1000 + multipart-exhausted-direct-dlq: false + diff --git a/hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/AppConfigCloudStorageTest.java b/hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/AppConfigCloudStorageTest.java new file mode 100644 index 0000000000..5cf499f9e4 --- /dev/null +++ b/hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/AppConfigCloudStorageTest.java @@ -0,0 +1,185 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.node; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +import org.apache.hugegraph.store.cloud.CloudStorageConfig; +import org.junit.Before; +import org.junit.Test; +import org.springframework.mock.env.MockEnvironment; + +public class AppConfigCloudStorageTest { + + private AppConfig.CloudStorageSpringConfig springConfig; + private MockEnvironment mockEnv; + + @Before + public void setUp() { + AppConfig appConfig = new AppConfig(); + springConfig = appConfig.new CloudStorageSpringConfig(); + mockEnv = new MockEnvironment(); + springConfig.setEnvironment(mockEnv); + } + + /** + * Common fields are bound from {@code cloud.storage.*} and converted correctly. + * Provider-specific properties flow from the environment through the provider sub-namespace. + */ + @Test + public void testCloudStorageSpringConfigConversion() { + springConfig.setEnabled(true); + springConfig.setProvider("s3"); + springConfig.setPathPrefix("test-prefix"); + springConfig.setStartupHydrationEnabled(false); + springConfig.setReadMissGuardWindowMs(5000L); + + // Simulate cloud.storage.s3.* properties being present in the Spring Environment + mockEnv.setProperty("cloud.storage.s3.bucket", "test-bucket"); + mockEnv.setProperty("cloud.storage.s3.region", "us-west-2"); + mockEnv.setProperty("cloud.storage.s3.endpoint", "https://s3.example.com"); + mockEnv.setProperty("cloud.storage.s3.access-key", "test-access-key"); + mockEnv.setProperty("cloud.storage.s3.secret-key", "test-secret-key"); + mockEnv.setProperty("cloud.storage.s3.multipart-part-retry-max-attempts", "5"); + mockEnv.setProperty("cloud.storage.s3.multipart-part-retry-base-backoff-ms", "1500"); + mockEnv.setProperty("cloud.storage.s3.multipart-exhausted-direct-dlq", "true"); + + CloudStorageConfig cfg = springConfig.toCloudStorageConfig(); + + // Common fields + assertTrue(cfg.isEnabled()); + assertEquals("s3", cfg.getProvider()); + assertEquals("test-prefix", cfg.getPathPrefix()); + assertFalse(cfg.isStartupHydrationEnabled()); + assertEquals(5000L, cfg.getReadMissGuardWindowMs()); + + // Provider-specific properties forwarded verbatim + assertEquals("test-bucket", cfg.getProviderProperties().get("bucket")); + assertEquals("us-west-2", cfg.getProviderProperties().get("region")); + assertEquals("https://s3.example.com", cfg.getProviderProperties().get("endpoint")); + assertEquals("test-access-key", cfg.getProviderProperties().get("access-key")); + assertEquals("test-secret-key", cfg.getProviderProperties().get("secret-key")); + assertEquals("5", cfg.getProviderProperties().get("multipart-part-retry-max-attempts")); + assertEquals("1500", + cfg.getProviderProperties().get("multipart-part-retry-base-backoff-ms")); + assertEquals("true", + cfg.getProviderProperties().get("multipart-exhausted-direct-dlq")); + } + + /** + * When no provider env properties exist, providerProperties is empty but not null. + */ + @Test + public void testCloudStorageSpringConfigDefaults() { + CloudStorageConfig cfg = springConfig.toCloudStorageConfig(); + + assertFalse(cfg.isEnabled()); + assertEquals("s3", cfg.getProvider()); + assertEquals("hugegraph", cfg.getPathPrefix()); + assertTrue(cfg.isStartupHydrationEnabled()); + assertEquals(3000L, cfg.getReadMissGuardWindowMs()); + assertEquals(3, cfg.getUploadRetryMaxAttempts()); + assertEquals(64, cfg.getUploadBackpressureHighWatermark()); + assertEquals("flush", cfg.getWalMode()); + assertNotNull(cfg.getProviderProperties()); + assertTrue(cfg.getProviderProperties().isEmpty()); + } + + /** + * Common property setters on CloudStorageSpringConfig work correctly. + */ + @Test + public void testCloudStorageSpringConfigCommonGettersSetters() { + springConfig.setEnabled(true); + assertTrue(springConfig.isEnabled()); + + springConfig.setProvider("gcs"); + assertEquals("gcs", springConfig.getProvider()); + + springConfig.setPathPrefix("my-prefix"); + assertEquals("my-prefix", springConfig.getPathPrefix()); + + springConfig.setStartupHydrationEnabled(false); + assertFalse(springConfig.isStartupHydrationEnabled()); + + springConfig.setReadMissGuardWindowMs(7000L); + assertEquals(7000L, springConfig.getReadMissGuardWindowMs()); + + springConfig.setUploadRetryMaxAttempts(3); + assertEquals(3, springConfig.getUploadRetryMaxAttempts()); + + springConfig.setUploadRetryInitialDelayMs(500L); + assertEquals(500L, springConfig.getUploadRetryInitialDelayMs()); + + springConfig.setUploadRetryMaxDelayMs(30_000L); + assertEquals(30_000L, springConfig.getUploadRetryMaxDelayMs()); + + springConfig.setUploadBackpressureHighWatermark(128); + assertEquals(128, springConfig.getUploadBackpressureHighWatermark()); + assertEquals(128, + springConfig.toCloudStorageConfig().getUploadBackpressureHighWatermark()); + + + springConfig.setWalMode("wal"); + assertEquals("wal", springConfig.getWalMode()); + assertEquals("wal", springConfig.toCloudStorageConfig().getWalMode()); + } + + /** + * Provider-specific sub-namespace is driven by the {@code provider} field, + * so a different provider name reads from a different env sub-namespace. + */ + @Test + public void testProviderNamespaceIsDynamic() { + springConfig.setProvider("gcs"); + mockEnv.setProperty("cloud.storage.gcs.bucket", "gcs-bucket"); + mockEnv.setProperty("cloud.storage.gcs.credentials-file-path", "/path/creds.json"); + // S3 keys should NOT appear + mockEnv.setProperty("cloud.storage.s3.bucket", "s3-bucket"); + + CloudStorageConfig cfg = springConfig.toCloudStorageConfig(); + + assertEquals("gcs-bucket", cfg.getProviderProperties().get("bucket")); + assertEquals("/path/creds.json", + cfg.getProviderProperties().get("credentials-file-path")); + assertFalse("s3 keys must not bleed into gcs namespace", + cfg.getProviderProperties().containsKey("cloud.storage.s3.bucket")); + } + + /** + * Test AppConfig getRaftPath. + */ + @Test + public void testGetRaftPath() { + AppConfig config = new AppConfig(); + // getRaftPath should not throw; actual value depends on initialization + config.getRaftPath(); + } + + /** + * Test CloudStorageSpringConfig creation and conversion with empty env. + */ + @Test + public void testCloudStorageSpringConfigCreation() { + assertNotNull(springConfig); + assertNotNull(springConfig.toCloudStorageConfig()); + } +} diff --git a/hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/cloud/CloudStorageEventListenerTest.java b/hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/cloud/CloudStorageEventListenerTest.java new file mode 100644 index 0000000000..6e76ae77bf --- /dev/null +++ b/hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/cloud/CloudStorageEventListenerTest.java @@ -0,0 +1,1140 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.node.cloud; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +import java.io.File; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.concurrent.locks.LockSupport; +import java.util.function.BooleanSupplier; + +import org.apache.hugegraph.rocksdb.access.RocksDBFactory.LiveSstFile; +import org.apache.hugegraph.rocksdb.access.RocksDBFactory.MetadataSnapshot; +import org.apache.hugegraph.store.cloud.CloudStorageConfig; +import org.apache.hugegraph.store.cloud.CloudStorageProvider; +import org.apache.hugegraph.store.cloud.CloudStorageProviderFactory; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +/** + * Unit tests for {@link CloudStorageEventListener}. + * + *

Focuses on the relative-key computation ({@code toRelativeKey}) and + * the {@link CloudStorageEventListener#onTableFileCreated} / {@link + * CloudStorageEventListener#onTableFileDeleted} delegate calls to the + * active {@link CloudStorageProvider}. + */ +public class CloudStorageEventListenerTest { + + private static final String DATA_ROOT = "/hugegraph-store/storage"; + + private CloudStorageEventListener listener; + + @Before + public void setUp() { + listener = new CloudStorageEventListener(DATA_ROOT); + } + + @After + public void tearDown() { + // Reset active provider between tests + CloudStorageProviderFactory.reset(); + } + + // ----------------------------------------------------------------------- + // toRelativeKey + // ----------------------------------------------------------------------- + + @Test + public void toRelativeKey_stripsDataRootPrefix() { + String filePath = DATA_ROOT + "/hgstore-metadata/000008.sst"; + assertEquals("hgstore-metadata/000008.sst", listener.toRelativeKey(filePath)); + } + + @Test + public void toRelativeKey_stripsDataRootPrefixForPartitionDb() { + String filePath = DATA_ROOT + "/0/000042.sst"; + assertEquals("0/000042.sst", listener.toRelativeKey(filePath)); + } + + @Test + public void toRelativeKey_fallsBackToStripLeadingSlash_whenNotUnderDataRoot() { + String filePath = "/some/other/path/000001.sst"; + assertEquals("some/other/path/000001.sst", listener.toRelativeKey(filePath)); + } + + @Test + public void toRelativeKey_handlesDataRootWithTrailingSlash() { + CloudStorageEventListener l = + new CloudStorageEventListener(DATA_ROOT + File.separator); + assertEquals("hgstore-metadata/000008.sst", + l.toRelativeKey(DATA_ROOT + "/hgstore-metadata/000008.sst")); + } + + @Test + public void toRelativeKey_appliesStoreScopePrefix() { + CloudStorageEventListener l = new CloudStorageEventListener( + DATA_ROOT, true, 0L, null, new CloudSyncTracker(), 0, + false, "store-127.0.0.1_8501"); + String filePath = DATA_ROOT + "/0/000042.sst"; + assertEquals("store-127.0.0.1_8501/0/000042.sst", l.toRelativeKey(filePath)); + } + + // ----------------------------------------------------------------------- + // onTableFileCreated / onTableFileDeleted – no active provider (no-op) + // ----------------------------------------------------------------------- + + @Test + public void onTableFileCreated_noActiveProvider_doesNotThrow() { + // No provider registered → method must be a silent no-op. + listener.onTableFileCreated("hgstore-metadata", "default", + DATA_ROOT + "/hgstore-metadata/000008.sst", 1234L); + } + + @Test + public void onTableFileDeleted_noActiveProvider_doesNotThrow() { + listener.onTableFileDeleted("hgstore-metadata", "default", + DATA_ROOT + "/hgstore-metadata/000008.sst"); + } + + // ----------------------------------------------------------------------- + // onTableFileCreated / onTableFileDeleted – with stub provider + // ----------------------------------------------------------------------- + + @Test + public void onTableFileCreated_delegatesToProvider_withRelativeKey() throws Exception { + Path tmpRoot = Files.createTempDirectory("hgstore-test-created"); + Path dbDir = tmpRoot.resolve("hgstore-metadata"); + Files.createDirectories(dbDir); + Path sst = dbDir.resolve("000008.sst"); + Files.write(sst, "sst".getBytes()); + + CapturingProvider provider = new CapturingProvider(); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + CloudStorageEventListener l = new CloudStorageEventListener(tmpRoot.toString()); + + try { + l.onTableFileCreated("hgstore-metadata", "default", sst.toString(), 512L); + + waitForCondition(() -> provider.uploads.size() == 1, + "expected exactly one async upload"); + + assertEquals(1, provider.uploads.size()); + assertTrue(provider.uploads.get(0)[0].contains("/.cloud-upload-staging/")); + assertEquals("hgstore-metadata/000008.sst", provider.uploads.get(0)[1]); + } finally { + deleteRecursively(tmpRoot.toFile()); + } + } + + @Test + public void onTableFileCreated_delegatesToProvider_withStoreScopePrefix() throws Exception { + Path tmpRoot = Files.createTempDirectory("hgstore-test-scoped"); + Path dbDir = tmpRoot.resolve("0"); + Files.createDirectories(dbDir); + Path sst = dbDir.resolve("000008.sst"); + Files.write(sst, "sst".getBytes()); + + CapturingProvider provider = new CapturingProvider(); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + CloudStorageEventListener l = new CloudStorageEventListener( + tmpRoot.toString(), true, 0L, null, new CloudSyncTracker(), 0, + false, "store-127.0.0.1_8501"); + try { + l.onTableFileCreated("0", "default", sst.toString(), 512L); + + waitForCondition(() -> provider.uploads.size() == 1, + "expected exactly one async upload with store scope prefix"); + + assertEquals(1, provider.uploads.size()); + assertEquals("store-127.0.0.1_8501/0/000008.sst", provider.uploads.get(0)[1]); + } finally { + deleteRecursively(tmpRoot.toFile()); + } + } + + @Test + public void onTableFileCreated_uploadFailure_doesNotThrow_andSubmitsToRetryQueue() + throws Exception { + // A listener wired with a retry queue: failure must not throw and must submit to queue. + Path tmpRoot = Files.createTempDirectory("hgstore-test-retry"); + try (CloudUploadRetryQueue retryQueue = new CloudUploadRetryQueue( + 1, 50L, 50L, tmpRoot.toString())) { + CloudStorageEventListener l = new CloudStorageEventListener( + DATA_ROOT, true, 0L, retryQueue); + CloudStorageProviderFactory.setActiveProviderForTest(new FailingUploadProvider()); + // Must NOT throw – failure is handled asynchronously. + l.onTableFileCreated("hgstore-metadata", "default", + DATA_ROOT + "/hgstore-metadata/000008.sst", 512L); + // Queue should have one in-flight retry submitted. + assertTrue("Expected at least one in-flight retry", + retryQueue.getInFlightCount() > 0 || retryQueue.getDlqSize() >= 0); + } finally { + deleteRecursively(tmpRoot.toFile()); + } + } + + @Test + public void onTableFileCreated_uploadFailure_noRetryQueue_doesNotThrow() { + // A listener without a retry queue: failure must still not throw (just logs). + CloudStorageProviderFactory.setActiveProviderForTest(new FailingUploadProvider()); + // listener is created in setUp() with no retry queue. + listener.onTableFileCreated("hgstore-metadata", "default", + DATA_ROOT + "/hgstore-metadata/000008.sst", 512L); + // If we reached here without an exception the test passes. + } + + @Test + public void onTableFileCreated_isNonBlocking_returnsQuicklyWithSlowProvider() throws Exception { + // Verify that onTableFileCreated does not block on provider.uploadFile(), + // even when the provider sleeps for a long duration. + Path tmpRoot = Files.createTempDirectory("hgstore-test-nonblocking"); + Path metadataDir = tmpRoot.resolve("hgstore-metadata"); + Files.createDirectories(metadataDir); + Path sst = metadataDir.resolve("000008.sst"); + Files.write(sst, "sst".getBytes()); + + try { + CloudStorageEventListener l = new CloudStorageEventListener(tmpRoot.toString()); + long sleepDurationMs = 1000L; // Provider will sleep for 1 second + CloudStorageProviderFactory.setActiveProviderForTest( + new SlowUploadProvider(sleepDurationMs)); + + // Measure how long onTableFileCreated takes to return + long startNs = System.nanoTime(); + l.onTableFileCreated("hgstore-metadata", "default", sst.toString(), 512L); + long callDurationMs = (System.nanoTime() - startNs) / 1_000_000; + + // The callback must return in a fraction of the provider's sleep time. + // Allow a small buffer (100ms) for overhead, but the bulk of the sleep + // should happen asynchronously in the background executor. + assertTrue("onTableFileCreated should return quickly without blocking on upload; " + + "call took " + callDurationMs + "ms (provider sleeps for " + + sleepDurationMs + "ms)", + callDurationMs < 200L); + } finally { + deleteRecursively(tmpRoot.toFile()); + } + } + + @Test + public void onTableFileDeleted_delegatesToProvider_withRelativeKey() { + CapturingProvider provider = new CapturingProvider(); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + CloudStorageEventListener l = new CloudStorageEventListener(DATA_ROOT) { + @Override + boolean syncMetadataSnapshotInline(CloudStorageProvider p, String dbName) { + return true; + } + }; + + l.onTableFileDeleted("hgstore-metadata", "default", + DATA_ROOT + "/hgstore-metadata/000008.sst"); + + assertEquals(1, provider.deletes.size()); + assertEquals("hgstore-metadata/000008.sst", provider.deletes.get(0)); + } + + // ----------------------------------------------------------------------- + // onDBCreated – uploads existing SST files from the DB directory + // ----------------------------------------------------------------------- + + @Test + public void onDBCreated_uploadsExistingSstFiles() throws Exception { + // Create a temporary directory that mimics a partition DB directory. + Path tmpRoot = Files.createTempDirectory("hgstore-test-storage"); + Path partitionDir = tmpRoot.resolve("0"); + Files.createDirectories(partitionDir); + Files.createFile(partitionDir.resolve("000001.sst")); + Files.createFile(partitionDir.resolve("000002.sst")); + + CloudStorageEventListener l = + new CloudStorageEventListener(tmpRoot.toString()); + CapturingProvider provider = new CapturingProvider(); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + try { + // Trigger onDBCreated; startup backfill should upload both SST files synchronously. + l.onDBCreated("0", partitionDir.toString()); + + assertEquals(2, provider.uploads.size()); + List remoteKeys = new ArrayList<>(); + for (String[] u : provider.uploads) { + remoteKeys.add(u[1]); + } + // Keys should be relative (e.g. "0/000001.sst") + for (String key : remoteKeys) { + assert !key.startsWith("/") : "Remote key must not start with '/': " + key; + assert key.endsWith(".sst") : "Remote key must end with .sst: " + key; + } + } finally { + deleteRecursively(tmpRoot.toFile()); + } + } + + @Test + public void onDBCreated_existingUploadFailure_throwsRuntimeException() throws Exception { + Path tmpRoot = Files.createTempDirectory("hgstore-test-storage"); + Path partitionDir = tmpRoot.resolve("0"); + Files.createDirectories(partitionDir); + Files.createFile(partitionDir.resolve("000001.sst")); + + CloudStorageEventListener l = + new CloudStorageEventListener(tmpRoot.toString()); + CloudStorageProviderFactory.setActiveProviderForTest(new FailingUploadProvider()); + + try { + try { + l.onDBCreated("0", partitionDir.toString()); + fail("Expected startup backfill failure to be rethrown"); + } catch (IllegalStateException e) { + assertTrue(e.getMessage().contains("Cloud initial-upload failed")); + } + } finally { + deleteRecursively(tmpRoot.toFile()); + } + } + + // ----------------------------------------------------------------------- + // metadata mirroring (upload order, CURRENT-last, prune, consistent restore) + // ----------------------------------------------------------------------- + private static CloudStorageEventListener metadataListener(boolean walMode) { + return new CloudStorageEventListener(DATA_ROOT, false, 0L, null, + new CloudSyncTracker(), 0, walMode); + } + + private static MetadataSnapshot snapshot(List options, List ssts, + List wals) { + return new MetadataSnapshot("/hugegraph-store/storage/0", "/hugegraph-store/storage/0" + "_tmp", + "CURRENT", "MANIFEST-000005", options, ssts, wals); + } + + @Test + public void uploadMetadataSnapshot_uploadsReferencedSstsThenMetadataThenCurrentLast() { + CloudStorageEventListener l = metadataListener(false); + CapturingProvider provider = new CapturingProvider(); + // The referenced SST is already durable in cloud → no re-upload, but metadata must follow. + provider.putRemoteFile("0/000003.sst", "sst".getBytes()); + + MetadataSnapshot snap = snapshot( + List.of("OPTIONS-000004"), List.of("000003.sst"), + List.of()); + + boolean durable = l.uploadMetadataSnapshot(provider, "0", snap); + + assertTrue(durable); + List keys = new ArrayList<>(); + for (String[] up : provider.uploads) { + keys.add(up[1]); + } + // OPTIONS + MANIFEST are published before CURRENT; CURRENT is published last of all. + assertEquals(List.of("0/OPTIONS-000004", "0/MANIFEST-000005", "0/CURRENT"), keys); + } + + @Test + public void uploadMetadataSnapshot_holdsManifestAndCurrentWhenReferencedSstUploadFails() { + CloudStorageEventListener l = metadataListener(false); + // Referenced SST is neither in cloud nor uploadable → the whole publish must be held. + FailingUploadProvider provider = new FailingUploadProvider(); + + MetadataSnapshot snap = snapshot( + List.of("OPTIONS-000004"), List.of("000003.sst"), + List.of()); + + boolean durable = l.uploadMetadataSnapshot(provider, "0", snap); + + assertFalse(durable); + // No MANIFEST/OPTIONS/CURRENT may be published while a referenced SST is not durable. + for (String[] up : provider.uploads) { + fail("nothing should have been published, but uploaded: " + up[1]); + } + } + + @Test + public void uploadMetadataSnapshot_prunesSupersededRemoteMetadataButKeepsCurrentAndSst() { + CloudStorageEventListener l = metadataListener(false); + CapturingProvider provider = new CapturingProvider(); + provider.putRemoteFile("0/000003.sst", "sst".getBytes()); + // Superseded remote metadata from a previous generation, plus files that must be kept. + provider.putRemoteFile("0/MANIFEST-000001", "old".getBytes()); + provider.putRemoteFile("0/OPTIONS-000000", "old".getBytes()); + provider.putRemoteFile("0/CURRENT", "old".getBytes()); + + MetadataSnapshot snap = snapshot( + List.of("OPTIONS-000004"), List.of("000003.sst"), + List.of()); + + assertTrue(l.uploadMetadataSnapshot(provider, "0", snap)); + + assertTrue(provider.deletes.contains("0/MANIFEST-000001")); + assertTrue(provider.deletes.contains("0/OPTIONS-000000")); + // CURRENT (the pointer) and SST objects are never pruned by the metadata sync. + assertFalse(provider.deletes.contains("0/CURRENT")); + assertFalse(provider.deletes.contains("0/000003.sst")); + } + + @Test + public void uploadMetadataSnapshot_walMode_mirrorsWalBeforeCurrent() { + CloudStorageEventListener l = metadataListener(true); + CapturingProvider provider = new CapturingProvider(); + provider.putRemoteFile("0/000003.sst", "sst".getBytes()); + + MetadataSnapshot snap = snapshot( + List.of("OPTIONS-000004"), List.of("000003.sst"), + List.of("000002.log")); + + assertTrue(l.uploadMetadataSnapshot(provider, "0", snap)); + + List keys = new ArrayList<>(); + for (String[] up : provider.uploads) { + keys.add(up[1]); + } + assertEquals(List.of("0/000002.log", "0/OPTIONS-000004", "0/MANIFEST-000005", "0/CURRENT"), + keys); + } + + @Test + public void preHydration_failsLoudlyWhenCurrentReferencesMissingManifest() throws Exception { + Path tmpRoot = Files.createTempDirectory("hgstore-test-storage"); + Path partitionDir = tmpRoot.resolve("0"); + Files.createDirectories(partitionDir); + + CloudStorageEventListener l = new CloudStorageEventListener(tmpRoot.toString(), true); + CapturingProvider provider = new CapturingProvider(); + // CURRENT points at a manifest that was never mirrored → restore must refuse to open. + provider.putRemoteFile("0/CURRENT", "MANIFEST-000009\n".getBytes()); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + try { + l.onDBOpening("0", partitionDir.toString()); + fail("expected IllegalStateException for inconsistent restore"); + } catch (IllegalStateException e) { + assertTrue(e.getMessage().contains("Cloud restore inconsistent")); + } finally { + deleteRecursively(tmpRoot.toFile()); + } + } + + // ----------------------------------------------------------------------- + // Helpers + // ----------------------------------------------------------------------- + + @SuppressWarnings("ResultOfMethodCallIgnored") + private void deleteRecursively(File f) { + if (f.isDirectory()) { + for (File child : Objects.requireNonNull(f.listFiles())) { + deleteRecursively(child); + } + } + f.delete(); + } + + private void waitForCondition(BooleanSupplier condition, String timeoutMessage) + throws InterruptedException { + long deadline = System.currentTimeMillis() + 2000L; + while (System.currentTimeMillis() < deadline) { + if (condition.getAsBoolean()) { + return; + } + LockSupport.parkNanos(10_000_000L); + if (Thread.currentThread().isInterrupted()) { + throw new InterruptedException("Interrupted while waiting for async cloud callback"); + } + } + fail(timeoutMessage); + } + + /** + * Minimal {@link CloudStorageProvider} that records upload and delete calls. + */ + static class CapturingProvider implements CloudStorageProvider { + + final List uploads = Collections.synchronizedList(new ArrayList<>()); + final List deletes = Collections.synchronizedList(new ArrayList<>()); + final Map remoteFiles = new HashMap<>(); + int listFilesCalls = 0; + + @Override + public String providerName() { + return "capturing"; + } + + @Override + public void init(CloudStorageConfig config) { + } + + @Override + public void uploadFile(String localPath, String remoteKey) throws IOException { + uploads.add(new String[]{localPath, remoteKey}); + } + + @SuppressWarnings("RedundantThrows") + @Override + public void deleteFile(String remoteKey) throws IOException { + deletes.add(remoteKey); + } + + @SuppressWarnings("RedundantThrows") + @Override + public boolean fileExists(String remoteKey) throws IOException { + return remoteFiles.containsKey(remoteKey); + } + + @SuppressWarnings("RedundantThrows") + @Override + public List listFiles(String remoteDirPrefix) throws IOException { + listFilesCalls++; + List result = new ArrayList<>(); + for (String key : remoteFiles.keySet()) { + if (key.startsWith(remoteDirPrefix)) { + result.add(key); + } + } + return result; + } + + @Override + public void downloadFile(String remoteKey, String localPath) throws IOException { + byte[] data = remoteFiles.get(remoteKey); + if (data == null) { + throw new IOException("remote key not found: " + remoteKey); + } + Path p = Paths.get(localPath); + Files.createDirectories(p.getParent()); + Files.write(p, data); + } + + @SuppressWarnings("RedundantThrows") + @Override + public void close() throws IOException { + } + + @SuppressWarnings("RedundantThrows") + @Override + public int deletePrefix(String remoteDirPrefix) throws IOException { + List result = new ArrayList<>(); + for (String key : remoteFiles.keySet()) { + if (key.startsWith(remoteDirPrefix)) { + result.add(key); + } + } + for (String key : result) { + deletes.add(key); + remoteFiles.remove(key); + } + return result.size(); + } + + void putRemoteFile(String key, byte[] content) { + remoteFiles.put(key, content); + } + } + + static class FailingUploadProvider extends CapturingProvider { + + @Override + public void uploadFile(String localPath, String remoteKey) throws IOException { + throw new IOException("simulated upload failure"); + } + } + + /** + * A {@link CloudStorageProvider} that sleeps during upload to simulate a slow provider. + * Used to verify that {@code onTableFileCreated} is non-blocking and does not wait for + * the provider's upload to complete. + */ + static class SlowUploadProvider extends CapturingProvider { + + private final long sleepMs; + + SlowUploadProvider(long sleepMs) { + this.sleepMs = sleepMs; + } + + @Override + public void uploadFile(String localPath, String remoteKey) throws IOException { + // Record the upload call (parent class behavior) + super.uploadFile(localPath, remoteKey); + // Simulate a slow operation that would block if called synchronously + try { + Thread.sleep(sleepMs); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new IOException("upload sleep interrupted", e); + } + } + } + + @Test + public void onDBOpening_downloadsMissingRemoteFiles() throws Exception { + Path tmpRoot = Files.createTempDirectory("hgstore-test-storage"); + Path partitionDir = tmpRoot.resolve("0"); + Files.createDirectories(partitionDir); + + CloudStorageEventListener l = + new CloudStorageEventListener(tmpRoot.toString(), true); + CapturingProvider provider = new CapturingProvider(); + provider.putRemoteFile("0/CURRENT", "MANIFEST-000001".getBytes()); + provider.putRemoteFile("0/MANIFEST-000001", "manifest-body".getBytes()); + provider.putRemoteFile("0/000001.sst", "sst-body".getBytes()); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + try { + l.onDBOpening("0", partitionDir.toString()); + assertTrue(Files.exists(partitionDir.resolve("CURRENT"))); + assertTrue(Files.exists(partitionDir.resolve("MANIFEST-000001"))); + assertTrue(Files.exists(partitionDir.resolve("000001.sst"))); + } finally { + deleteRecursively(tmpRoot.toFile()); + } + } + + @Test + public void onDBOpening_downloadsMissingRemoteFiles_withStoreScopePrefix() throws Exception { + Path tmpRoot = Files.createTempDirectory("hgstore-test-storage"); + Path partitionDir = tmpRoot.resolve("0"); + Files.createDirectories(partitionDir); + + CloudStorageEventListener l = new CloudStorageEventListener( + tmpRoot.toString(), true, 0L, null, new CloudSyncTracker(), 0, + false, "store-127.0.0.1_8501"); + CapturingProvider provider = new CapturingProvider(); + provider.putRemoteFile("store-127.0.0.1_8501/0/CURRENT", "MANIFEST-000001".getBytes()); + provider.putRemoteFile("store-127.0.0.1_8501/0/MANIFEST-000001", "manifest-body".getBytes()); + provider.putRemoteFile("store-127.0.0.1_8501/0/000001.sst", "sst-body".getBytes()); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + try { + l.onDBOpening("0", partitionDir.toString()); + assertTrue(Files.exists(partitionDir.resolve("CURRENT"))); + assertTrue(Files.exists(partitionDir.resolve("MANIFEST-000001"))); + assertTrue(Files.exists(partitionDir.resolve("000001.sst"))); + } finally { + deleteRecursively(tmpRoot.toFile()); + } + } + + // ----------------------------------------------------------------------- + // Read-miss restores missing LIVE files to their original path (no ingest) + // ----------------------------------------------------------------------- + + @Test + public void restoreMissingLiveFiles_restoresMissingLiveFileToOriginalPath() throws Exception { + Path tmpRoot = Files.createTempDirectory("hgstore-test-storage"); + Path partitionDir = tmpRoot.resolve("0"); + Files.createDirectories(partitionDir); + + CloudStorageEventListener l = new CloudStorageEventListener(tmpRoot.toString(), true); + CapturingProvider provider = new CapturingProvider(); + provider.putRemoteFile("0/000001.sst", "sst-body".getBytes()); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + String localPath = partitionDir.resolve("000001.sst").toString(); + List live = List.of(new LiveSstFile(localPath, "default")); + + try { + int restored = l.restoreMissingLiveFiles(provider, "0", live); + assertEquals(1, restored); + // Restored to the EXACT original path so RocksDB finds it; no ingest is performed. + assertTrue(Files.exists(partitionDir.resolve("000001.sst"))); + } finally { + deleteRecursively(tmpRoot.toFile()); + } + } + + @Test + public void restoreMissingLiveFiles_routesEachFileToItsOwnPath_crossCf() throws Exception { + Path tmpRoot = Files.createTempDirectory("hgstore-test-storage"); + Path partitionDir = tmpRoot.resolve("0"); + Files.createDirectories(partitionDir); + + CloudStorageEventListener l = new CloudStorageEventListener(tmpRoot.toString(), true); + CapturingProvider provider = new CapturingProvider(); + provider.putRemoteFile("0/000001.sst", "g-body".getBytes()); + provider.putRemoteFile("0/000002.sst", "q-body".getBytes()); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + // Two live files belonging to different column families. + List live = List.of( + new LiveSstFile(partitionDir.resolve("000001.sst").toString(), "g"), + new LiveSstFile(partitionDir.resolve("000002.sst").toString(), "q")); + + try { + int restored = l.restoreMissingLiveFiles(provider, "0", live); + assertEquals(2, restored); + // Each file lands at its own path; content is not mixed across CFs. + assertEquals("g-body", + new String(Files.readAllBytes(partitionDir.resolve("000001.sst")))); + assertEquals("q-body", + new String(Files.readAllBytes(partitionDir.resolve("000002.sst")))); + } finally { + deleteRecursively(tmpRoot.toFile()); + } + } + + @Test + public void restoreMissingLiveFiles_skipsFilesPresentLocally() throws Exception { + Path tmpRoot = Files.createTempDirectory("hgstore-test-storage"); + Path partitionDir = tmpRoot.resolve("0"); + Files.createDirectories(partitionDir); + Files.write(partitionDir.resolve("000001.sst"), "already-here".getBytes()); + + CloudStorageEventListener l = new CloudStorageEventListener(tmpRoot.toString(), true); + CapturingProvider provider = new CapturingProvider(); + provider.putRemoteFile("0/000001.sst", "cloud-body".getBytes()); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + List live = List.of( + new LiveSstFile(partitionDir.resolve("000001.sst").toString(), "default")); + + try { + int restored = l.restoreMissingLiveFiles(provider, "0", live); + assertEquals(0, restored); + // Present local file is untouched (not overwritten from cloud). + assertEquals("already-here", + new String(Files.readAllBytes(partitionDir.resolve("000001.sst")))); + } finally { + deleteRecursively(tmpRoot.toFile()); + } + } + + @Test + public void readMissGuard_skipsRepeatedAttemptsWithinWindow() { + CloudStorageEventListener l = + new CloudStorageEventListener(DATA_ROOT, true, 60_000L); + assertTrue(l.shouldAttemptReadMissHydration("0", "default")); + assertFalse(l.shouldAttemptReadMissHydration("0", "default")); + // A different table is not throttled by the first table's attempt. + assertTrue(l.shouldAttemptReadMissHydration("0", "other")); + } + + @Test + public void deleteGuard_holdsWhenLiveFileNotConfirmedAndUploadFails() throws Exception { + Path tmpRoot = Files.createTempDirectory("hgstore-test-guard"); + Path partitionDir = tmpRoot.resolve("0"); + Files.createDirectories(partitionDir); + // A live file that exists locally but whose upload will fail. + Path liveLocal = partitionDir.resolve("000009.sst"); + Files.write(liveLocal, "live".getBytes()); + + CloudSyncTracker tracker = new CloudSyncTracker(); + CloudStorageEventListener l = new CloudStorageEventListener( + tmpRoot.toString(), true, 0L, null, tracker, 0); + FailingUploadProvider provider = new FailingUploadProvider(); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + List live = List.of(new LiveSstFile(liveLocal.toString(), "default")); + + try { + boolean durable = l.ensureLiveSetUploaded(provider, "0", live); + assertFalse("Guard must report the live set as NOT durable", durable); + } finally { + deleteRecursively(tmpRoot.toFile()); + } + } + + @Test + public void deleteGuard_passesWhenAllLiveFilesConfirmed() throws Exception { + Path tmpRoot = Files.createTempDirectory("hgstore-test-guard"); + Path partitionDir = tmpRoot.resolve("0"); + Files.createDirectories(partitionDir); + Path liveLocal = partitionDir.resolve("000009.sst"); + Files.write(liveLocal, "live".getBytes()); + + CloudSyncTracker tracker = new CloudSyncTracker(); + // Pre-mark the live file as already confirmed in cloud. + tracker.markConfirmed("0", liveLocal.toString()); + CloudStorageEventListener l = new CloudStorageEventListener( + tmpRoot.toString(), true, 0L, null, tracker, 0); + CapturingProvider provider = new CapturingProvider(); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + List live = List.of(new LiveSstFile(liveLocal.toString(), "default")); + + try { + boolean durable = l.ensureLiveSetUploaded(provider, "0", live); + assertTrue("Guard must pass when the whole live set is confirmed", durable); + // No upload needed since it was already confirmed. + assertEquals(0, provider.uploads.size()); + } finally { + deleteRecursively(tmpRoot.toFile()); + } + } + + @Test + public void deleteGuard_uploadsUnconfirmedLiveFileThenPasses() throws Exception { + Path tmpRoot = Files.createTempDirectory("hgstore-test-guard"); + Path partitionDir = tmpRoot.resolve("0"); + Files.createDirectories(partitionDir); + Path liveLocal = partitionDir.resolve("000009.sst"); + Files.write(liveLocal, "live".getBytes()); + + CloudSyncTracker tracker = new CloudSyncTracker(); + CloudStorageEventListener l = new CloudStorageEventListener( + tmpRoot.toString(), true, 0L, null, tracker, 0); + CapturingProvider provider = new CapturingProvider(); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + List live = List.of(new LiveSstFile(liveLocal.toString(), "default")); + + try { + boolean durable = l.ensureLiveSetUploaded(provider, "0", live); + assertTrue(durable); + // The unconfirmed-but-present live file was uploaded and then marked confirmed. + assertEquals(1, provider.uploads.size()); + assertEquals("0/000009.sst", provider.uploads.get(0)[1]); + assertTrue(tracker.isConfirmed("0", liveLocal.toString())); + } finally { + deleteRecursively(tmpRoot.toFile()); + } + } + + @Test + public void deleteGuard_blocksOldSstDelete_whenInputsConfirmedButMergedOutputMissing() + throws Exception { + Path tmpRoot = Files.createTempDirectory("hgstore-test-guard"); + Path partitionDir = tmpRoot.resolve("0"); + Files.createDirectories(partitionDir); + + // Compaction inputs SST1/SST2 were uploaded previously and are still present locally. + Path sst1 = partitionDir.resolve("000001.sst"); + Path sst2 = partitionDir.resolve("000002.sst"); + Files.write(sst1, "sst1".getBytes()); + Files.write(sst2, "sst2".getBytes()); + + // Compaction output exists in RocksDB live-set metadata but is missing both locally and + // in cloud (upload failed/lost). This is the dangerous window we must guard. + Path merged = partitionDir.resolve("000010.sst"); + + CloudSyncTracker tracker = new CloudSyncTracker(); + tracker.markConfirmed("0", sst1.toString()); + tracker.markConfirmed("0", sst2.toString()); + + CloudStorageEventListener l = new CloudStorageEventListener( + tmpRoot.toString(), true, 0L, null, tracker, 0); + CapturingProvider provider = new CapturingProvider(); + provider.putRemoteFile("0/000001.sst", "sst1".getBytes()); + provider.putRemoteFile("0/000002.sst", "sst2".getBytes()); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + // Post-compaction live set contains only MERGED_SST. + List live = List.of(new LiveSstFile(merged.toString(), "default")); + + try { + boolean durable = l.ensureLiveSetUploaded(provider, "0", live); + assertFalse("Merged output is not durable; old SST delete must be blocked", durable); + // MERGED_SST is absent locally, so guard cannot self-heal by upload. + assertEquals(0, provider.uploads.size()); + } finally { + deleteRecursively(tmpRoot.toFile()); + } + } + + // ----------------------------------------------------------------------- + // MANIFEST-before-delete invariant + // ----------------------------------------------------------------------- + + /** + * Captures call-order between {@code syncMetadataSnapshotInline} and {@code provider.deleteFile}. + * Overrides {@code syncMetadataSnapshotInline} so no live RocksDB session is required. + */ + static class OrderingListener extends CloudStorageEventListener { + + final List callOrder = new ArrayList<>(); + private final boolean syncResult; + + OrderingListener(String dataRoot, CloudSyncTracker tracker, boolean syncResult) { + super(dataRoot, false, 0L, null, tracker, 0, + /* walModeEnabled */ false); + this.syncResult = syncResult; + } + + @Override + boolean syncMetadataSnapshotInline(CloudStorageProvider provider, String dbName) { + callOrder.add("sync"); + return syncResult; + } + } + + /** + * Verifies the MANIFEST-before-delete invariant: + * {@code syncMetadataSnapshotInline} must be called and succeed before the superseded SST is + * deleted from cloud. + * + *

Scenario: compaction merged SST1+SST2 into MERGED_SST. MERGED_SST was already uploaded + * and confirmed. When {@code onTableFileDeleted} fires for SST1 an updated MANIFEST+CURRENT + * must be published first, then SST1 deleted. A crash between those two steps leaves the + * cluster in a recoverable state. + */ + @Test + public void onTableFileDeleted_publishesUpdatedManifestBeforeDeletingSupersededSst() { + String sst1Remote = "db0/000001.sst"; + // No real RocksDB session for "db0" → getLiveSstFiles() returns empty + // → ensureLiveSetUploaded passes trivially. + CapturingProvider provider = new CapturingProvider() { + @Override + public void deleteFile(String remoteKey) throws IOException { + // Record delete after any sync call already recorded by the listener. + super.deleteFile(remoteKey); + } + }; + provider.putRemoteFile(sst1Remote, "sst1".getBytes()); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + CloudSyncTracker tracker = new CloudSyncTracker(); + OrderingListener l = new OrderingListener(DATA_ROOT, tracker, /* syncResult */ true) { + @Override + boolean syncMetadataSnapshotInline(CloudStorageProvider p, String dbName) { + boolean result = super.syncMetadataSnapshotInline(p, dbName); + // Verify no delete has happened yet when sync fires. + assertTrue("delete must not precede sync", provider.deletes.isEmpty()); + return result; + } + }; + + l.onTableFileDeleted("db0", "default", DATA_ROOT + "/db0/000001.sst"); + + assertEquals("sync must have been called once", List.of("sync"), l.callOrder); + assertTrue("SST1 must be deleted from cloud", provider.deletes.contains(sst1Remote)); + } + + /** + * When {@code syncMetadataSnapshotInline} fails (MANIFEST not updated), the SST delete must be + * skipped entirely to avoid leaving an irrecoverable state in cloud. + */ + @Test + public void onTableFileDeleted_skipsDeleteWhenMetadataSyncFails() { + String sst1Remote = "db0/000001.sst"; + CapturingProvider provider = new CapturingProvider(); + provider.putRemoteFile(sst1Remote, "sst1".getBytes()); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + CloudSyncTracker tracker = new CloudSyncTracker(); + // syncResult=false: metadata sync fails → delete must be suppressed. + OrderingListener l = new OrderingListener(DATA_ROOT, tracker, /* syncResult */ false); + + l.onTableFileDeleted("db0", "default", DATA_ROOT + "/db0/000001.sst"); + + assertEquals("sync must have been called", List.of("sync"), l.callOrder); + assertTrue("SST must NOT be deleted when metadata sync fails", provider.deletes.isEmpty()); + } + + // ----------------------------------------------------------------------- + // onDBDeleteBegin / onDBDeleted — stale-data guard + // ----------------------------------------------------------------------- + + @Test + public void onDBDeleteBegin_writesTombstoneToCloud() { + CapturingProvider provider = new CapturingProvider(); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + CloudStorageEventListener l = new CloudStorageEventListener(DATA_ROOT); + l.onDBDeleteBegin("mydb", DATA_ROOT + "/mydb"); + + String expectedTombstone = "mydb/" + CloudStorageEventListener.DB_TOMBSTONE_FILE; + boolean tombstoneUploaded = provider.uploads.stream() + .anyMatch(pair -> expectedTombstone.equals(pair[1])); + assertTrue("Tombstone must be uploaded to cloud prefix", tombstoneUploaded); + } + + @Test + public void onDBDeleteBegin_tombstoneKeyRespectsStoreScopePrefix() { + CapturingProvider provider = new CapturingProvider(); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + CloudStorageEventListener l = new CloudStorageEventListener( + DATA_ROOT, true, 0L, null, new CloudSyncTracker(), 0, false, + "store-127.0.0.1_8501"); + l.onDBDeleteBegin("mydb", DATA_ROOT + "/mydb"); + + String expectedTombstone = "store-127.0.0.1_8501/mydb/" + + CloudStorageEventListener.DB_TOMBSTONE_FILE; + boolean tombstoneUploaded = provider.uploads.stream() + .anyMatch(pair -> expectedTombstone.equals(pair[1])); + assertTrue("Tombstone must include store scope prefix", tombstoneUploaded); + } + + @Test + public void onDBDeleted_purgesAllRemoteObjectsUnderPrefix() { + CapturingProvider provider = new CapturingProvider(); + provider.putRemoteFile("mydb/000001.sst", "sst".getBytes()); + provider.putRemoteFile("mydb/CURRENT", "MANIFEST-1".getBytes()); + provider.putRemoteFile("mydb/MANIFEST-000001", "body".getBytes()); + provider.putRemoteFile("mydb/" + CloudStorageEventListener.DB_TOMBSTONE_FILE, + "deleted".getBytes()); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + CloudStorageEventListener l = new CloudStorageEventListener(DATA_ROOT); + l.onDBDeleted("mydb", DATA_ROOT + "/mydb"); + + // All four remote objects must have been submitted for deletion. + List expectedDeleted = List.of( + "mydb/000001.sst", "mydb/CURRENT", "mydb/MANIFEST-000001", + "mydb/" + CloudStorageEventListener.DB_TOMBSTONE_FILE); + for (String key : expectedDeleted) { + assertTrue("Remote object must be deleted after DB destroy: " + key, + provider.deletes.contains(key)); + } + } + + @Test + public void onDBDeleted_clearsSyncTrackerState() { + CloudSyncTracker tracker = new CloudSyncTracker(); + tracker.markConfirmed("mydb", DATA_ROOT + "/mydb/000001.sst"); + assertEquals(1L, tracker.confirmedCount("mydb")); + + CapturingProvider provider = new CapturingProvider(); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + CloudStorageEventListener l = new CloudStorageEventListener( + DATA_ROOT, true, 0L, null, tracker, 0); + l.onDBDeleted("mydb", DATA_ROOT + "/mydb"); + + assertEquals("Sync tracker must be cleared for the deleted DB", + 0L, tracker.confirmedCount("mydb")); + } + + @Test + public void onDBOpening_skipsHydrationWhenTombstonePresent() throws Exception { + Path tmpRoot = Files.createTempDirectory("hgstore-test-storage"); + Path partitionDir = tmpRoot.resolve("mydb"); + Files.createDirectories(partitionDir); + + CloudStorageEventListener l = new CloudStorageEventListener(tmpRoot.toString(), true); + CapturingProvider provider = new CapturingProvider(); + // Populate cloud with stale data from a previous deleted generation + tombstone. + provider.putRemoteFile("mydb/000001.sst", "stale-sst".getBytes()); + provider.putRemoteFile("mydb/CURRENT", "MANIFEST-000001".getBytes()); + provider.putRemoteFile("mydb/" + CloudStorageEventListener.DB_TOMBSTONE_FILE, + "deleted".getBytes()); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + try { + l.onDBOpening("mydb", partitionDir.toString()); + + // Hydration must be skipped — stale SST and CURRENT must NOT be downloaded. + assertFalse("Stale SST must not be hydrated when tombstone is present", + Files.exists(partitionDir.resolve("000001.sst"))); + assertFalse("Stale CURRENT must not be hydrated when tombstone is present", + Files.exists(partitionDir.resolve("CURRENT"))); + // All stale remote objects must be submitted for deletion during tombstone purge. + assertTrue("Stale SST must be purged", + provider.deletes.contains("mydb/000001.sst")); + assertTrue("Stale CURRENT must be purged", + provider.deletes.contains("mydb/CURRENT")); + assertTrue("Tombstone itself must be purged", + provider.deletes.contains( + "mydb/" + CloudStorageEventListener.DB_TOMBSTONE_FILE)); + } finally { + deleteRecursively(tmpRoot.toFile()); + } + } + + @Test + public void onDBOpening_proceedsNormallyWhenNoTombstone() throws Exception { + Path tmpRoot = Files.createTempDirectory("hgstore-test-storage"); + Path partitionDir = tmpRoot.resolve("mydb"); + Files.createDirectories(partitionDir); + + CloudStorageEventListener l = new CloudStorageEventListener(tmpRoot.toString(), true); + CapturingProvider provider = new CapturingProvider(); + // Normal cloud state — no tombstone. + provider.putRemoteFile("mydb/000001.sst", "sst-body".getBytes()); + provider.putRemoteFile("mydb/CURRENT", "MANIFEST-000001".getBytes()); + provider.putRemoteFile("mydb/MANIFEST-000001", "manifest".getBytes()); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + try { + l.onDBOpening("mydb", partitionDir.toString()); + + assertTrue("SST must be hydrated in normal open", + Files.exists(partitionDir.resolve("000001.sst"))); + assertTrue("CURRENT must be hydrated in normal open", + Files.exists(partitionDir.resolve("CURRENT"))); + } finally { + deleteRecursively(tmpRoot.toFile()); + } + } + + @Test + public void deleteAndRecreate_newDbDoesNotIngestDeletedData() throws Exception { + // Simulate the full delete-then-recreate lifecycle: + // 1. DB is created and writes data to cloud. + // 2. DB is deleted → tombstone written, cloud purged. + // 3. DB is recreated at the same path → hydration must find empty prefix, not stale data. + Path tmpRoot = Files.createTempDirectory("hgstore-test-storage"); + Path dbDir = tmpRoot.resolve("graph0"); + Files.createDirectories(dbDir); + + CapturingProvider provider = new CapturingProvider(); + // Stale objects from the old generation. + provider.putRemoteFile("graph0/000001.sst", "old-data".getBytes()); + provider.putRemoteFile("graph0/CURRENT", "MANIFEST-000001".getBytes()); + CloudStorageProviderFactory.setActiveProviderForTest(provider); + + CloudStorageEventListener l = new CloudStorageEventListener(tmpRoot.toString(), true); + + // Step 1: begin delete — tombstone uploaded. + l.onDBDeleteBegin("graph0", dbDir.toString()); + boolean tombstoneUploaded = provider.uploads.stream() + .anyMatch(p -> p[1].equals("graph0/" + CloudStorageEventListener.DB_TOMBSTONE_FILE)); + assertTrue("Tombstone must be uploaded during deleteBegin", tombstoneUploaded); + + // Simulate tombstone appearing in cloud (as it would after a real upload). + provider.putRemoteFile("graph0/" + CloudStorageEventListener.DB_TOMBSTONE_FILE, + "deleted".getBytes()); + + // Step 2: deletion complete — cloud purged. + l.onDBDeleted("graph0", dbDir.toString()); + // All three objects must be deleted (including tombstone itself). + assertTrue("Old SST must be deleted", provider.deletes.contains("graph0/000001.sst")); + assertTrue("Old CURRENT must be deleted", provider.deletes.contains("graph0/CURRENT")); + assertTrue("Tombstone must be deleted after purge", + provider.deletes.contains( + "graph0/" + CloudStorageEventListener.DB_TOMBSTONE_FILE)); + + // Step 3: cloud is now clean; recreate at same path. + // Remove all objects from "cloud" to simulate a clean state. + provider.remoteFiles.clear(); + Files.createDirectories(dbDir); + l.onDBOpening("graph0", dbDir.toString()); + + // Nothing to hydrate; recreated DB dir must be empty. + assertFalse("Stale SST must not be present in recreated DB dir", + Files.exists(dbDir.resolve("000001.sst"))); + assertFalse("Stale CURRENT must not be present in recreated DB dir", + Files.exists(dbDir.resolve("CURRENT"))); + + deleteRecursively(tmpRoot.toFile()); + } + +} diff --git a/hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/cloud/CloudStorageMetricsTest.java b/hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/cloud/CloudStorageMetricsTest.java new file mode 100644 index 0000000000..e823a396b4 --- /dev/null +++ b/hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/cloud/CloudStorageMetricsTest.java @@ -0,0 +1,364 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.node.cloud; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNotNull; + +import java.lang.reflect.Field; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicLong; + +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +import io.micrometer.core.instrument.Counter; +import io.micrometer.core.instrument.Gauge; +import io.micrometer.core.instrument.Timer; +import io.micrometer.core.instrument.simple.SimpleMeterRegistry; + +/** + * Unit tests for {@link CloudStorageMetrics}. + * + *

Because {@link CloudStorageMetrics} uses static singleton state, each test + * resets all static fields via reflection in {@link #tearDown()} so that tests + * remain independent. + */ +public class CloudStorageMetricsTest { + + private SimpleMeterRegistry registry; + private CloudSyncTracker syncTracker; + + @Before + public void setUp() { + registry = new SimpleMeterRegistry(); + syncTracker = new CloudSyncTracker(); + resetStaticState(); + } + + @After + public void tearDown() { + resetStaticState(); + } + + // ----------------------------------------------------------------------- + // init() + // ----------------------------------------------------------------------- + + @Test + public void init_registersGlobalMetrics() { + CloudStorageMetrics.init(registry, syncTracker); + + assertNotNull("upload failures counter must be registered", + registry.find(CloudStorageMetricsConst.UPLOAD_FAILURES).counter()); + assertNotNull("sync latency timer must be registered", + registry.find(CloudStorageMetricsConst.SYNC_LATENCY_MS).timer()); + assertNotNull("retry queue size gauge must be registered", + registry.find(CloudStorageMetricsConst.RETRY_QUEUE_SIZE).gauge()); + } + + @Test + public void init_isIdempotent_secondCallIsNoOp() { + CloudStorageMetrics.init(registry, syncTracker); + + // A second registry should be ignored + SimpleMeterRegistry secondRegistry = new SimpleMeterRegistry(); + CloudStorageMetrics.init(secondRegistry, syncTracker); + + // Upload failures counter must still be registered on the first registry + assertNotNull(registry.find(CloudStorageMetricsConst.UPLOAD_FAILURES).counter()); + // Nothing should be registered on the second registry + assertEquals(0, secondRegistry.getMeters().size()); + } + + @Test(expected = IllegalArgumentException.class) + public void init_nullRegistry_throwsIllegalArgument() { + CloudStorageMetrics.init(null, syncTracker); + } + + @Test(expected = IllegalArgumentException.class) + public void init_nullTracker_throwsIllegalArgument() { + CloudStorageMetrics.init(registry, null); + } + + // ----------------------------------------------------------------------- + // registerDatabaseMetrics() + // ----------------------------------------------------------------------- + + @Test + public void registerDatabaseMetrics_beforeInit_isNoOp() { + // No init — should not throw and registry must remain empty + CloudStorageMetrics.registerDatabaseMetrics("db1"); + assertEquals(0, registry.getMeters().size()); + } + + @Test + public void registerDatabaseMetrics_registersConfirmedFilesGauge() { + CloudStorageMetrics.init(registry, syncTracker); + CloudStorageMetrics.registerDatabaseMetrics("mydb"); + + Gauge gauge = registry.find(CloudStorageMetricsConst.CONFIRMED_FILES) + .tag(CloudStorageMetricsConst.TAG_DB_NAME, "mydb") + .gauge(); + assertNotNull("confirmed_files gauge must be registered for the database", gauge); + } + + @Test + public void registerDatabaseMetrics_registersDeleteGuardReuploadGauge() { + CloudStorageMetrics.init(registry, syncTracker); + CloudStorageMetrics.registerDatabaseMetrics("mydb"); + + Gauge gauge = registry.find(CloudStorageMetricsConst.DELETE_GUARD_REUPLOAD_COUNT) + .tag(CloudStorageMetricsConst.TAG_DB_NAME, "mydb") + .gauge(); + assertNotNull("delete_guard_reupload_count gauge must be registered for the database", + gauge); + } + + @Test + public void registerDatabaseMetrics_isIdempotent_secondCallIsNoOp() { + CloudStorageMetrics.init(registry, syncTracker); + CloudStorageMetrics.registerDatabaseMetrics("mydb"); + + // Count meters after first registration + int metersAfterFirst = registry.getMeters().size(); + + // A second call for the same database should not add new meters + CloudStorageMetrics.registerDatabaseMetrics("mydb"); + assertEquals(metersAfterFirst, registry.getMeters().size()); + } + + @Test + public void registerDatabaseMetrics_multipleDatabases_eachGetsSeparateGauges() { + CloudStorageMetrics.init(registry, syncTracker); + CloudStorageMetrics.registerDatabaseMetrics("db-alpha"); + CloudStorageMetrics.registerDatabaseMetrics("db-beta"); + + assertNotNull(registry.find(CloudStorageMetricsConst.CONFIRMED_FILES) + .tag(CloudStorageMetricsConst.TAG_DB_NAME, "db-alpha").gauge()); + assertNotNull(registry.find(CloudStorageMetricsConst.CONFIRMED_FILES) + .tag(CloudStorageMetricsConst.TAG_DB_NAME, "db-beta").gauge()); + } + + // ----------------------------------------------------------------------- + // recordUploadFailure() + // ----------------------------------------------------------------------- + + @Test + public void recordUploadFailure_beforeInit_isNoOp() { + // Must not throw + CloudStorageMetrics.recordUploadFailure("db1", "default", "timeout"); + } + + @Test + public void recordUploadFailure_incrementsCounter() { + CloudStorageMetrics.init(registry, syncTracker); + + CloudStorageMetrics.recordUploadFailure("db1", "default", "timeout"); + CloudStorageMetrics.recordUploadFailure("db1", "default", "auth"); + + Counter counter = registry.find(CloudStorageMetricsConst.UPLOAD_FAILURES).counter(); + assertNotNull(counter); + assertEquals(2.0, counter.count(), 0.001); + } + + @Test + public void recordUploadFailure_multipleCallsAccumulate() { + CloudStorageMetrics.init(registry, syncTracker); + + for (int i = 0; i < 5; i++) { + CloudStorageMetrics.recordUploadFailure("db1", "cf1", "error-" + i); + } + + Counter counter = registry.find(CloudStorageMetricsConst.UPLOAD_FAILURES).counter(); + assertNotNull(counter); + assertEquals(5.0, counter.count(), 0.001); + } + + // ----------------------------------------------------------------------- + // recordSyncLatency() + // ----------------------------------------------------------------------- + + @Test + public void recordSyncLatency_beforeInit_isNoOp() { + // Must not throw + CloudStorageMetrics.recordSyncLatency("db1", 250L); + } + + @Test + public void recordSyncLatency_recordsMeasurement() { + CloudStorageMetrics.init(registry, syncTracker); + + CloudStorageMetrics.recordSyncLatency("db1", 100L); + CloudStorageMetrics.recordSyncLatency("db1", 200L); + + Timer timer = registry.find(CloudStorageMetricsConst.SYNC_LATENCY_MS).timer(); + assertNotNull(timer); + assertEquals(2, timer.count()); + assertEquals(300.0, timer.totalTime(TimeUnit.MILLISECONDS), 1.0); + } + + @Test + public void recordSyncLatency_zeroLatency_isRecorded() { + CloudStorageMetrics.init(registry, syncTracker); + CloudStorageMetrics.recordSyncLatency("db1", 0L); + + Timer timer = registry.find(CloudStorageMetricsConst.SYNC_LATENCY_MS).timer(); + assertNotNull(timer); + assertEquals(1, timer.count()); + } + + // ----------------------------------------------------------------------- + // getConfirmedFileCount() + // ----------------------------------------------------------------------- + + @Test + public void getConfirmedFileCount_beforeInit_returnsZero() { + assertEquals(0L, CloudStorageMetrics.getConfirmedFileCount("db1")); + } + + @Test + public void getConfirmedFileCount_afterInit_delegatesToSyncTracker() { + // Confirm a file in the tracker then query via metrics + syncTracker.markConfirmed("db1", "db1/000001.sst"); + syncTracker.markConfirmed("db1", "db1/000002.sst"); + + CloudStorageMetrics.init(registry, syncTracker); + + assertEquals(2L, CloudStorageMetrics.getConfirmedFileCount("db1")); + } + + @Test + public void getConfirmedFileCount_unknownDb_returnsZero() { + CloudStorageMetrics.init(registry, syncTracker); + assertEquals(0L, CloudStorageMetrics.getConfirmedFileCount("no-such-db")); + } + + // ----------------------------------------------------------------------- + // incrementDeleteGuardReupload() / getDeleteGuardReuploadCount() + // ----------------------------------------------------------------------- + + @Test + public void getDeleteGuardReuploadCount_unknownDb_returnsZero() { + assertEquals(0L, CloudStorageMetrics.getDeleteGuardReuploadCount("unknown-db")); + } + + @Test + public void incrementDeleteGuardReupload_incrementsCountForDb() { + CloudStorageMetrics.incrementDeleteGuardReupload("db1"); + CloudStorageMetrics.incrementDeleteGuardReupload("db1"); + CloudStorageMetrics.incrementDeleteGuardReupload("db1"); + + assertEquals(3L, CloudStorageMetrics.getDeleteGuardReuploadCount("db1")); + } + + @Test + public void incrementDeleteGuardReupload_countsAreIsolatedPerDb() { + CloudStorageMetrics.incrementDeleteGuardReupload("db-a"); + CloudStorageMetrics.incrementDeleteGuardReupload("db-a"); + CloudStorageMetrics.incrementDeleteGuardReupload("db-b"); + + assertEquals(2L, CloudStorageMetrics.getDeleteGuardReuploadCount("db-a")); + assertEquals(1L, CloudStorageMetrics.getDeleteGuardReuploadCount("db-b")); + } + + @Test + public void incrementDeleteGuardReupload_beforeRegisterDb_stillWorks() { + // Incrementing before registerDatabaseMetrics() should still track the count + CloudStorageMetrics.incrementDeleteGuardReupload("db-new"); + assertEquals(1L, CloudStorageMetrics.getDeleteGuardReuploadCount("db-new")); + } + + @Test + public void deleteGuardReuploadGauge_reflectsLiveCount() { + CloudStorageMetrics.init(registry, syncTracker); + CloudStorageMetrics.registerDatabaseMetrics("mydb"); + + CloudStorageMetrics.incrementDeleteGuardReupload("mydb"); + CloudStorageMetrics.incrementDeleteGuardReupload("mydb"); + + Gauge gauge = registry.find(CloudStorageMetricsConst.DELETE_GUARD_REUPLOAD_COUNT) + .tag(CloudStorageMetricsConst.TAG_DB_NAME, "mydb") + .gauge(); + assertNotNull(gauge); + assertEquals(2.0, gauge.value(), 0.001); + } + + @Test + public void confirmedFilesGauge_reflectsTrackerState() { + CloudStorageMetrics.init(registry, syncTracker); + CloudStorageMetrics.registerDatabaseMetrics("mydb"); + + syncTracker.markConfirmed("mydb", "mydb/000001.sst"); + syncTracker.markConfirmed("mydb", "mydb/000002.sst"); + syncTracker.markConfirmed("mydb", "mydb/000003.sst"); + + Gauge gauge = registry.find(CloudStorageMetricsConst.CONFIRMED_FILES) + .tag(CloudStorageMetricsConst.TAG_DB_NAME, "mydb") + .gauge(); + assertNotNull(gauge); + assertEquals(3.0, gauge.value(), 0.001); + } + + // ----------------------------------------------------------------------- + // setRetryQueueSize() — placeholder method, must not throw + // ----------------------------------------------------------------------- + + @Test + public void setRetryQueueSize_doesNotThrow() { + CloudStorageMetrics.setRetryQueueSize(0); + CloudStorageMetrics.setRetryQueueSize(42); + CloudStorageMetrics.setRetryQueueSize(Integer.MAX_VALUE); + } + + // ----------------------------------------------------------------------- + // Helpers + // ----------------------------------------------------------------------- + + /** + * Resets all static fields in {@link CloudStorageMetrics} to {@code null} / empty + * via reflection so that each test starts from a clean slate. + */ + @SuppressWarnings("unchecked") + private static void resetStaticState() { + try { + setStaticField("registry"); + setStaticField("syncTracker"); + setStaticField("uploadFailuresCounter"); + setStaticField("syncLatencyTimer"); + + Field mapField = CloudStorageMetrics.class + .getDeclaredField("deleteGuardReuploadPerDb"); + mapField.setAccessible(true); + ((ConcurrentHashMap) mapField.get(null)).clear(); + } catch (Exception e) { + throw new RuntimeException("Failed to reset CloudStorageMetrics static state", e); + } + } + + private static void setStaticField(String fieldName) + throws NoSuchFieldException, IllegalAccessException { + Field field = CloudStorageMetrics.class.getDeclaredField(fieldName); + field.setAccessible(true); + field.set(null, (Object) null); + } +} + + diff --git a/hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/cloud/CloudSyncTrackerTest.java b/hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/cloud/CloudSyncTrackerTest.java new file mode 100644 index 0000000000..143103743b --- /dev/null +++ b/hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/cloud/CloudSyncTrackerTest.java @@ -0,0 +1,75 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.node.cloud; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import org.junit.Test; + +public class CloudSyncTrackerTest { + + @Test + public void parseSstFileNumber_parsesDigitBaseName() { + assertEquals(123L, CloudSyncTracker.parseSstFileNumber("/data/db/000123.sst")); + assertEquals(8L, CloudSyncTracker.parseSstFileNumber("0/000008.sst")); + assertEquals(0L, CloudSyncTracker.parseSstFileNumber("000000.sst")); + } + + @Test + public void parseSstFileNumber_rejectsNonSstAndNonNumeric() { + assertEquals(-1L, CloudSyncTracker.parseSstFileNumber("/data/db/CURRENT")); + assertEquals(-1L, CloudSyncTracker.parseSstFileNumber("/data/db/MANIFEST-000001")); + assertEquals(-1L, CloudSyncTracker.parseSstFileNumber("/data/db/abc.sst")); + assertEquals(-1L, CloudSyncTracker.parseSstFileNumber(null)); + assertEquals(-1L, CloudSyncTracker.parseSstFileNumber(".sst")); + } + + @Test + public void markIsConfirmedAndClear_roundTrip() { + CloudSyncTracker tracker = new CloudSyncTracker(); + String path = "0/000042.sst"; + + assertFalse(tracker.isConfirmed("0", path)); + tracker.markConfirmed("0", path); + assertTrue(tracker.isConfirmed("0", path)); + assertEquals(1L, tracker.confirmedCount("0")); + + tracker.clearConfirmed("0", path); + assertFalse(tracker.isConfirmed("0", path)); + assertEquals(0L, tracker.confirmedCount("0")); + } + + @Test + public void confirmedBitmapsAreScopedPerDb() { + CloudSyncTracker tracker = new CloudSyncTracker(); + tracker.markConfirmed("0", "0/000001.sst"); + // Same file number, different DB, must be independent. + assertFalse(tracker.isConfirmed("1", "1/000001.sst")); + assertTrue(tracker.isConfirmed("0", "0/000001.sst")); + } + + @Test + public void nonSstPathsAreIgnored() { + CloudSyncTracker tracker = new CloudSyncTracker(); + tracker.markConfirmed("0", "0/CURRENT"); + assertEquals(0L, tracker.confirmedCount("0")); + assertFalse(tracker.isConfirmed("0", "0/CURRENT")); + } +} diff --git a/hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/cloud/CloudUploadRetryQueueTest.java b/hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/cloud/CloudUploadRetryQueueTest.java new file mode 100644 index 0000000000..c17764d633 --- /dev/null +++ b/hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/cloud/CloudUploadRetryQueueTest.java @@ -0,0 +1,354 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.node.cloud; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.ArrayList; +import java.util.List; +import java.util.Objects; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; + +import org.apache.hugegraph.store.cloud.CloudStorageConfig; +import org.apache.hugegraph.store.cloud.CloudStorageProvider; +import org.apache.hugegraph.store.cloud.CloudStorageProviderFactory; +import org.junit.After; +import org.junit.Before; +import org.junit.Test; + +/** + * Unit tests for {@link CloudUploadRetryQueue}. + */ +@SuppressWarnings("BusyWait") +public class CloudUploadRetryQueueTest { + + private Path tmpRoot; + + @Before + public void setUp() throws IOException { + tmpRoot = Files.createTempDirectory("hgstore-retry-queue-test"); + CloudStorageProviderFactory.reset(); + } + + @After + public void tearDown() { + CloudStorageProviderFactory.reset(); + deleteRecursively(tmpRoot.toFile()); + } + + // ----------------------------------------------------------------------- + // submit → retry succeeds on second attempt + // ----------------------------------------------------------------------- + + @Test + public void submit_retriesAndSucceeds() throws Exception { + CountDownLatch successLatch = new CountDownLatch(1); + AtomicInteger callCount = new AtomicInteger(0); + + CloudStorageProviderFactory.setActiveProviderForTest(new CapturingProvider() { + @Override + public void uploadFile(String localPath, String remoteKey) throws IOException { + if (callCount.incrementAndGet() == 1) { + throw new IOException("transient failure"); + } + // Second call succeeds. + successLatch.countDown(); + } + }); + + // Create a real SST file so the retry doesn't drop it as "compacted". + Path sstFile = tmpRoot.resolve("000001.sst"); + Files.createFile(sstFile); + + try (CloudUploadRetryQueue queue = + new CloudUploadRetryQueue(3, 50L, 200L, tmpRoot.toString())) { + + queue.submit("db1", "default", sstFile.toString(), + "db1/000001.sst", new IOException("initial failure")); + + assertTrue("Upload should have succeeded within 2 s", + successLatch.await(2, TimeUnit.SECONDS)); + assertEquals(0, queue.getDlqSize()); + } + } + + // ----------------------------------------------------------------------- + // submit → all attempts fail → moved to DLQ + // ----------------------------------------------------------------------- + + @Test + public void submit_exhaustsRetriesAndMovesToDlq() throws Exception { + AtomicInteger uploadCalls = new AtomicInteger(0); + + CloudStorageProviderFactory.setActiveProviderForTest(new CapturingProvider() { + @Override + public void uploadFile(String localPath, String remoteKey) throws IOException { + uploadCalls.incrementAndGet(); + throw new IOException("permanent failure"); + } + }); + + Path sstFile = tmpRoot.resolve("000002.sst"); + Files.createFile(sstFile); + + // maxAttempts=2, initial+retry delay 50ms → DLQ after ~100ms total + try (CloudUploadRetryQueue queue = + new CloudUploadRetryQueue(2, 50L, 50L, tmpRoot.toString())) { + + queue.submit("db1", "default", sstFile.toString(), + "db1/000002.sst", new IOException("initial")); + + // Wait for the retry cycle to finish. + long deadline = System.currentTimeMillis() + 3_000L; + while (queue.getDlqSize() == 0 && System.currentTimeMillis() < deadline) { + Thread.sleep(50); + } + + assertEquals("Task should be in the DLQ after max retries", 1, queue.getDlqSize()); + FailedUploadTask entry = queue.getDlqEntries().get(0); + assertEquals("db1", entry.getDbName()); + assertEquals("db1/000002.sst", entry.getRemoteKey()); + assertTrue(entry.getAttemptCount() >= 1); + } + } + + // ----------------------------------------------------------------------- + // DLQ persistence – entries survive queue reconstruction + // ----------------------------------------------------------------------- + + @Test + public void dlq_persistsAndLoadsFromDisk() throws Exception { + CloudStorageProviderFactory.setActiveProviderForTest(new CapturingProvider() { + @Override + public void uploadFile(String localPath, String remoteKey) throws IOException { + throw new IOException("always fails"); + } + }); + + Path sstFile = tmpRoot.resolve("000003.sst"); + Files.createFile(sstFile); + + // First queue instance: run until task hits DLQ. + try (CloudUploadRetryQueue queue = + new CloudUploadRetryQueue(1, 50L, 50L, tmpRoot.toString())) { + queue.submit("db2", "cf1", sstFile.toString(), + "db2/000003.sst", new IOException("fail")); + + long deadline = System.currentTimeMillis() + 3_000L; + while (queue.getDlqSize() == 0 && System.currentTimeMillis() < deadline) { + Thread.sleep(50); + } + assertEquals(1, queue.getDlqSize()); + } + + // Verify the DLQ file was written. + assertTrue("DLQ file should exist", + Files.exists(tmpRoot.resolve(CloudUploadRetryQueue.DLQ_FILE_NAME))); + + // Second queue instance: should load the persisted entry. + try (CloudUploadRetryQueue queue2 = + new CloudUploadRetryQueue(1, 50L, 50L, tmpRoot.toString())) { + assertEquals("DLQ entry should have been loaded from disk", 1, queue2.getDlqSize()); + FailedUploadTask loaded = queue2.getDlqEntries().get(0); + assertEquals("db2", loaded.getDbName()); + assertEquals("cf1", loaded.getCfName()); + assertEquals("db2/000003.sst", loaded.getRemoteKey()); + } + } + + // ----------------------------------------------------------------------- + // replayDlq – successful replay clears the DLQ + // ----------------------------------------------------------------------- + + @Test + public void replayDlq_successfulUploadClearsDlq() throws Exception { + // Step 1: force a task into the DLQ. + AtomicInteger uploadCalls = new AtomicInteger(0); + CloudStorageProviderFactory.setActiveProviderForTest(new CapturingProvider() { + @Override + public void uploadFile(String localPath, String remoteKey) throws IOException { + if (uploadCalls.incrementAndGet() <= 1) { + throw new IOException("fail during initial + retry"); + } + // Subsequent calls succeed (replay). + } + }); + + Path sstFile = tmpRoot.resolve("000004.sst"); + Files.createFile(sstFile); + + try (CloudUploadRetryQueue queue = + new CloudUploadRetryQueue(1, 50L, 50L, tmpRoot.toString())) { + queue.submit("db3", "default", sstFile.toString(), + "db3/000004.sst", new IOException("initial fail")); + + // Wait for DLQ. + long deadline = System.currentTimeMillis() + 3_000L; + while (queue.getDlqSize() == 0 && System.currentTimeMillis() < deadline) { + Thread.sleep(50); + } + assertEquals(1, queue.getDlqSize()); + + // Now replay: the provider will succeed. + queue.replayDlq(); + + assertEquals("DLQ should be empty after successful replay", 0, queue.getDlqSize()); + } + } + + // ----------------------------------------------------------------------- + // replayDlq – local file gone → dropped silently, not re-queued + // ----------------------------------------------------------------------- + + @Test + public void replayDlq_dropsEntryWhenLocalFileGone() throws Exception { + CloudStorageProviderFactory.setActiveProviderForTest(new CapturingProvider() { + @Override + public void uploadFile(String localPath, String remoteKey) throws IOException { + throw new IOException("fail"); + } + }); + + // File path that does NOT exist (deleted SST). + String nonExistentFile = tmpRoot.resolve("gone.sst").toString(); + + try (CloudUploadRetryQueue queue = + new CloudUploadRetryQueue(1, 50L, 50L, tmpRoot.toString())) { + // Submit; the retry will run but the local file doesn't exist → drop silently. + queue.submit("db4", "default", nonExistentFile, + "db4/gone.sst", new IOException("initial")); + + // Wait for the retry to drain (file-not-found path → task dropped). + long deadline = System.currentTimeMillis() + 3_000L; + while (queue.getInFlightCount() > 0 && System.currentTimeMillis() < deadline) { + Thread.sleep(50); + } + + // Task dropped (not in DLQ) because local file was gone. + assertEquals("Task with non-existent file should be silently dropped", + 0, queue.getDlqSize()); + } + } + + // ----------------------------------------------------------------------- + // Serialisation round-trip + // ----------------------------------------------------------------------- + + @Test + public void serialize_deserialize_roundTrip() { + try (CloudUploadRetryQueue queue = + new CloudUploadRetryQueue(3, 100L, 1000L, tmpRoot.toString())) { + + FailedUploadTask original = new FailedUploadTask( + "db\twith-tab", "cf-name", "/data/root/file.sst", + "data/root/file.sst", 1234567890123L, 3, + "Error with\nnewline and\\backslash"); + + String line = queue.serialize(original); + assertFalse("Serialised line must not contain raw tab in field values", + hasUnescapedTab(line)); + + FailedUploadTask restored = queue.deserialize(line); + assertNotNull(restored); + assertEquals(original.getDbName(), restored.getDbName()); + assertEquals(original.getCfName(), restored.getCfName()); + assertEquals(original.getFilePath(), restored.getFilePath()); + assertEquals(original.getRemoteKey(), restored.getRemoteKey()); + assertEquals(original.getFailedAt(), restored.getFailedAt()); + assertEquals(original.getAttemptCount(), restored.getAttemptCount()); + assertEquals(original.getLastError(), restored.getLastError()); + } + } + + // ----------------------------------------------------------------------- + // Helpers + // ----------------------------------------------------------------------- + + /** Returns true if the serialised line has the wrong number of tab-separated fields. */ + private boolean hasUnescapedTab(String serialised) { + String[] parts = serialised.split("\t", -1); + return parts.length != 7; + } + + @SuppressWarnings("ResultOfMethodCallIgnored") + private void deleteRecursively(java.io.File f) { + if (f.isDirectory()) { + for (java.io.File child : Objects.requireNonNull(f.listFiles())) { + deleteRecursively(child); + } + } + f.delete(); + } + + // ----------------------------------------------------------------------- + // Stub providers + // ----------------------------------------------------------------------- + + static class CapturingProvider implements CloudStorageProvider { + + final List uploads = new ArrayList<>(); + + @Override + public String providerName() { + return "capturing"; + } + + @Override + public void init(CloudStorageConfig config) { + } + + @Override + public void uploadFile(String localPath, String remoteKey) throws IOException { + uploads.add(new String[]{localPath, remoteKey}); + } + + @Override + public void deleteFile(String remoteKey) { + } + + @Override + public boolean fileExists(String remoteKey) { + return false; + } + + @Override + public void downloadFile(String remoteKey, String localPath) { + } + + @Override + public void close() { + } + } +} + + + + + + + + + diff --git a/hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/cloud/RocksDBCallbackExceptionBehaviorTest.java b/hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/cloud/RocksDBCallbackExceptionBehaviorTest.java new file mode 100644 index 0000000000..65b42b5a88 --- /dev/null +++ b/hugegraph-store/hg-store-node/src/test/java/org/apache/hugegraph/store/node/cloud/RocksDBCallbackExceptionBehaviorTest.java @@ -0,0 +1,241 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hugegraph.store.node.cloud; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import java.util.concurrent.locks.LockSupport; +import java.util.function.BooleanSupplier; + +import org.apache.hugegraph.store.cloud.CloudStorageNonRetryableException; +import org.apache.hugegraph.store.cloud.CloudStorageProvider; +import org.apache.hugegraph.store.cloud.CloudStorageProviderFactory; +import org.junit.After; +import org.junit.Assert; +import org.junit.Before; +import org.junit.Test; +import org.mockito.Mockito; + +/** + * Test that verifies RocksDB behavior when exceptions are thrown from + * the {@link CloudStorageEventListener#onTableFileCreated} callback. + * + *

Key scenarios: + *

    + *
  • Upload provider throws an exception → captured and logged, retry queue engaged
    • + *
  • RocksDB does NOT crash when listener throws
    • + *
  • Multiple listeners can be registered; one failure doesn't prevent others
    • + *
  • Exception crosses JNI boundary safely (RocksDB swallows it)
    • + *
+ * + *

Run with: + * 

+ * mvn test -pl hugegraph-store/hg-store-node \
+ *   -Dtest=RocksDBCallbackExceptionBehaviorTest \
+ *   -am -DskipTests=false
+ *
+ */ +public class RocksDBCallbackExceptionBehaviorTest { + + private CloudStorageEventListener listener; + private CloudStorageProvider mockProvider; + private CloudUploadRetryQueue retryQueue; + private Path tmpDir; + + @Before + public void setUp() throws IOException { + // Create temp directory for DLQ file + this.tmpDir = Files.createTempDirectory("hgstore-test-"); + + // Create a mock provider that we can control (throw/succeed) + this.mockProvider = Mockito.mock(CloudStorageProvider.class); + + // Create a minimal retry queue (in-memory + optional disk DLQ) + this.retryQueue = new CloudUploadRetryQueue( + 3, // maxAttempts + 100L, // initialDelayMs + 5000L, // maxDelayMs + this.tmpDir.toString() // dataRoot + ); + + // Create the listener with retry queue + this.listener = new CloudStorageEventListener( + this.tmpDir.toString(), + true, // startupHydrationEnabled + 3000L, // readMissGuardWindowMs + this.retryQueue + ); + + // Set the active provider for testing + CloudStorageProviderFactory.setActiveProviderForTest(this.mockProvider); + } + + @After + public void tearDown() { + if (this.retryQueue != null) { + this.retryQueue.close(); + } + if (this.tmpDir != null) { + deleteDirectory(this.tmpDir.toFile()); + } + CloudStorageProviderFactory.reset(); + } + + @SuppressWarnings("ResultOfMethodCallIgnored") + private static void deleteDirectory(java.io.File dir) { + if (!dir.exists()) { + return; + } + java.io.File[] children = dir.listFiles(); + if (children != null) { + for (java.io.File child : children) { + if (child.isDirectory()) { + deleteDirectory(child); + } else { + child.delete(); + } + } + } + dir.delete(); + } + + private String createSstFile(String dbName, String fileName) throws IOException { + Path dbDir = this.tmpDir.resolve(dbName); + Files.createDirectories(dbDir); + Path sst = dbDir.resolve(fileName); + Files.write(sst, new byte[]{1}); + return sst.toString(); + } + + private void waitForCondition(BooleanSupplier condition, String timeoutMessage) + throws InterruptedException { + long deadline = System.currentTimeMillis() + 2000L; + while (System.currentTimeMillis() < deadline) { + if (condition.getAsBoolean()) { + return; + } + LockSupport.parkNanos(10_000_000L); + if (Thread.currentThread().isInterrupted()) { + throw new InterruptedException("Interrupted while waiting for async cloud callback"); + } + } + Assert.fail(timeoutMessage); + } + + @Test + public void uploadThrowsException_doesNotCrashRocksDB_submitsToRetryQueue() throws Exception { + String dbName = "hgstore-metadata"; + String cfName = "default"; + String filePath = createSstFile(dbName, "000001.sst"); + long fileSize = 64 * 1024 * 1024; // 64 MB + + // Step 1: Configure provider to throw an exception + IOException uploadFailure = new IOException("Network timeout: S3 unavailable"); + Mockito.doThrow(uploadFailure) + .when(this.mockProvider) + .uploadFile(Mockito.anyString(), Mockito.anyString()); + + // Step 2: Invoke callback (simulating RocksDB calling us after flush) + // This should NOT throw, even though the provider throws + try { + this.listener.onTableFileCreated(dbName, cfName, filePath, fileSize); + } catch (Exception e) { + Assert.fail("onTableFileCreated should NOT throw exception; caught: " + e); + } + + // Step 3: Verify retry queue captured the failure + waitForCondition(() -> this.retryQueue.getInFlightCount() > 0 + || !this.retryQueue.getDlqEntries().isEmpty(), + "task should be enqueued for retry (either in-flight or in DLQ)"); + + // Step 4: Verify provider.uploadFile was actually called + Mockito.verify(this.mockProvider, Mockito.timeout(1500).times(1)) + .uploadFile(filePath, dbName + "/000001.sst"); + } + + @Test + public void uploadThrowsNonRetryableException_submitsDirectlyToDLQ() throws Exception { + String dbName = "hgstore-metadata"; + String cfName = "default"; + String filePath = createSstFile(dbName, "000002.sst"); + long fileSize = 64 * 1024 * 1024; + + // Configure provider to throw a non-retryable exception + CloudStorageNonRetryableException nonRetryable = + new CloudStorageNonRetryableException( + "Authentication failed; credentials invalid", + null + ); + Mockito.doThrow(nonRetryable) + .when(this.mockProvider) + .uploadFile(Mockito.anyString(), Mockito.anyString()); + + // Invoke callback + try { + this.listener.onTableFileCreated(dbName, cfName, filePath, fileSize); + } catch (Exception e) { + Assert.fail("onTableFileCreated should not throw; caught: " + e); + } + + waitForCondition(() -> !this.retryQueue.getDlqEntries().isEmpty(), + "non-retryable exception should land in DLQ immediately"); + + // Verify task ended up in DLQ (not retrying) + List dlqEntries = this.retryQueue.getDlqEntries(); + Assert.assertFalse("non-retryable exception should land in DLQ immediately", + dlqEntries.isEmpty()); + + FailedUploadTask task = dlqEntries.get(0); + Assert.assertEquals(dbName, task.getDbName()); + Assert.assertTrue(task.getLastError().contains("credentials")); + } + + @Test + public void callbackInvokedWithoutActiveProvider_doesNotCrash() { + String dbName = "hgstore-metadata"; + String cfName = "default"; + String filePath = "/data/hgstore/" + dbName + "/000004.sst"; + long fileSize = 64 * 1024 * 1024; + + // Deactivate provider + CloudStorageProviderFactory.reset(); + + // Invoke callback — should gracefully no-op + try { + this.listener.onTableFileCreated(dbName, cfName, filePath, fileSize); + } catch (Exception e) { + Assert.fail("callback should handle null provider gracefully; caught: " + e); + } + + // Verify no crash and no DLQ entries (no-op) + Assert.assertEquals("no-op when provider is null", 0, + this.retryQueue.getDlqEntries().size()); + } +} + + + + + + + + + + diff --git a/hugegraph-store/hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/access/RocksDBFactory.java b/hugegraph-store/hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/access/RocksDBFactory.java index 2e8e0bae68..6fbdf82d16 100644 --- a/hugegraph-store/hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/access/RocksDBFactory.java +++ b/hugegraph-store/hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/access/RocksDBFactory.java @@ -38,9 +38,13 @@ import org.rocksdb.AbstractEventListener; import org.rocksdb.Cache; import org.rocksdb.CompactionJobInfo; +import org.rocksdb.LiveFileMetaData; import org.rocksdb.MemoryUsageType; import org.rocksdb.MemoryUtil; import org.rocksdb.RocksDB; +import org.rocksdb.Status; +import org.rocksdb.TableFileCreationInfo; +import org.rocksdb.TableFileDeletionInfo; import lombok.extern.slf4j.Slf4j; @@ -49,6 +53,8 @@ public final class RocksDBFactory { private static final List rocksdbChangedListeners = new ArrayList<>(); private static RocksDBFactory dbFactory; + /** Singleton event listener wired into every new RocksDB instance. */ + private final RocksdbEventListener rocksdbEventListener = new RocksdbEventListener(); static { RocksDB.loadLibrary(); @@ -178,6 +184,46 @@ public void onCompactionCompleted(RocksDB db, CompactionJobInfo compactionJobInf public void onCompactionBegin(final RocksDB db, final CompactionJobInfo compactionJobInfo) { log.info("RocksdbEventListener onCompactionBegin"); } + + /** + * Invoked by RocksDB after a new SST (table) file has been successfully created. + * Propagates the event to all registered {@link RocksdbChangedListener}s so that + * cloud-storage providers can upload the new file. + */ + @Override + public void onTableFileCreated(final TableFileCreationInfo info) { + if (info.getStatus().getCode() != Status.Code.Ok) { + log.warn("onTableFileCreated: skipping failed file creation, path={}, status={}", + info.getFilePath(), info.getStatus().getCodeString()); + return; + } + log.debug("onTableFileCreated: db={}, cf={}, path={}, size={}", + info.getDbName(), info.getColumnFamilyName(), + info.getFilePath(), info.getFileSize()); + rocksdbChangedListeners.forEach(listener -> + listener.onTableFileCreated(info.getDbName(), info.getColumnFamilyName(), + info.getFilePath(), info.getFileSize()) + ); + } + + /** + * Invoked by RocksDB after an SST (table) file has been deleted. + * Propagates the event to all registered {@link RocksdbChangedListener}s so that + * cloud-storage providers can remove the corresponding remote object. + */ + @Override + public void onTableFileDeleted(final TableFileDeletionInfo info) { + if (info.getStatus().getCode() != Status.Code.Ok) { + log.warn("onTableFileDeleted: skipping failed file deletion, path={}, status={}", + info.getFilePath(), info.getStatus().getCodeString()); + return; + } + log.debug("onTableFileDeleted: db={}, path={}", + info.getDbName(), info.getFilePath()); + rocksdbChangedListeners.forEach(listener -> + listener.onTableFileDeleted(info.getDbName(), null, info.getFilePath()) + ); + } } public RocksDBSession createGraphDB(String dbPath, String dbName) { @@ -189,16 +235,30 @@ public RocksDBSession createGraphDB(String dbPath, String dbName, long version) throw new RuntimeException("db closed"); } operateLock.writeLock().lock(); + boolean isNew = false; + RocksDBSession dbSession = null; try { - RocksDBSession dbSession = dbSessionMap.get(dbName); + dbSession = dbSessionMap.get(dbName); if (dbSession == null) { + String dbOpenPath = dbPath.endsWith(File.separator) ? dbPath + dbName : + dbPath + File.separator + dbName; + rocksdbChangedListeners.forEach(listener -> listener.onDBOpening(dbName, dbOpenPath)); log.info("create rocksdb for {}", dbName); dbSession = new RocksDBSession(this.hugeConfig, dbPath, dbName, version); dbSessionMap.put(dbName, dbSession); + isNew = true; } return dbSession.clone(); } finally { operateLock.writeLock().unlock(); + if (isNew && dbSession != null) { + // Notify listeners so they can upload any pre-existing SST files + // and flush the MemTable (which may hold WAL-recovered data). + final String finalDbName = dbSession.getGraphName(); + final String finalDbPath = dbSession.getDbPath(); + rocksdbChangedListeners.forEach(listener -> + listener.onDBCreated(finalDbName, finalDbPath)); + } } } @@ -314,19 +374,314 @@ public void addRocksdbChangedListener(RocksdbChangedListener listener) { rocksdbChangedListeners.add(listener); } + /** + * Notifies all registered listeners that a RocksDB truncate operation is about to start. + * + * @param dbName the graph / partition name + * @param dbPath the absolute path of the RocksDB directory + */ + public void notifyTruncateBegin(String dbName, String dbPath) { + rocksdbChangedListeners.forEach(listener -> listener.onDBTruncateBegin(dbName, dbPath)); + } + + /** + * Notifies all registered listeners that a RocksDB has been truncated. + * This should be called after the database content has been cleared to allow + * listeners (e.g., cloud storage) to clean up remote state. + * + * @param dbName the graph / partition name + * @param dbPath the absolute path of the RocksDB directory + */ + public void notifyTruncate(String dbName, String dbPath) { + rocksdbChangedListeners.forEach(listener -> listener.onDBTruncated(dbName, dbPath)); + } + + /** + * Flushes the MemTable of the named RocksDB session to disk, creating an SST file. + * This triggers {@link RocksdbChangedListener#onTableFileCreated} for every registered + * listener (including cloud-storage upload). + * + * @param dbName the graph / partition name + * @param wait if {@code true} the call blocks until the flush completes + */ + public void flushSession(String dbName, boolean wait) { + RocksDBSession session = dbSessionMap.get(dbName); + if (session != null) { + try { + session.flush(wait); + log.debug("Flushed RocksDB session for db={}", dbName); + } catch (Exception e) { + log.warn("Failed to flush RocksDB session for db={}: {}", dbName, e.getMessage()); + } + } + } + + /** + * Returns the live SST files currently referenced by the named RocksDB session, with the + * owning column-family name for each. Used by cloud storage to (a) confirm the full live set + * is present in cloud before deleting a superseded object, and (b) route read-miss hydration + * to the correct column family. + * + *

The returned metadata reflects RocksDB's manifest, so it lists files that RocksDB believes + * are live even if the local file has since been evicted from disk. + * + * @param dbName the graph / partition name + * @return live SST files (possibly empty); never {@code null} + */ + public List getLiveSstFiles(String dbName) { + RocksDBSession session = dbSessionMap.get(dbName); + if (session == null) { + return List.of(); + } + RocksDB db = session.getDB(); + if (db == null) { + return List.of(); + } + List result = new ArrayList<>(); + for (LiveFileMetaData md : db.getLiveFilesMetaData()) { + String dir = md.path(); + String name = md.fileName(); + if (name == null || !name.endsWith(".sst")) { + continue; + } + String absolutePath; + if (name.startsWith(File.separator) || name.startsWith("/")) { + absolutePath = dir.endsWith(File.separator) || dir.endsWith("/") + ? dir.substring(0, dir.length() - 1) + name + : dir + name; + } else { + absolutePath = dir.endsWith(File.separator) || dir.endsWith("/") + ? dir + name + : dir + File.separator + name; + } + String cfName = new String(md.columnFamilyName(), java.nio.charset.StandardCharsets.UTF_8); + result.add(new LiveSstFile(absolutePath, cfName)); + } + return result; + } + + /** A live SST file and the column family it belongs to. */ + public static final class LiveSstFile { + + private final String absolutePath; + private final String cfName; + + public LiveSstFile(String absolutePath, String cfName) { + this.absolutePath = absolutePath; + this.cfName = cfName; + } + + public String getAbsolutePath() { + return absolutePath; + } + + public String getCfName() { + return cfName; + } + } + + /** + * Captures a point-in-time, internally-consistent copy of the named DB's RocksDB metadata + * ({@code CURRENT}, {@code MANIFEST-*}, {@code OPTIONS-*}, the WAL {@code *.log} tail, and + * hard-links to the live {@code *.sst} set) into a temporary sibling directory of the DB, via + * RocksDB {@link org.rocksdb.Checkpoint} (the same primitive {@code saveSnapshot} uses). + * + *

This is the capture primitive behind metadata durability: the returned snapshot exposes + * the exact {@code {manifest, live-SST-set}} pair as of the checkpoint instant, so a caller can + * mirror a consistent set of objects to cloud storage without racing live compaction. The + * hard-linked SSTs also pin their content for the lifetime of the snapshot, so an SST cannot be + * physically removed by compaction between capture and upload. + * + *

The caller must call {@link MetadataSnapshot#cleanup()} when done to remove the + * temporary directory (which only contains metadata copies and SST hard-links — deleting it + * never touches the real SST files). + * + * @param dbName the graph / partition name + * @return the captured snapshot, or {@code null} if the session is not open + */ + public MetadataSnapshot captureMetadataSnapshot(String dbName) { + RocksDBSession session = dbSessionMap.get(dbName); + if (session == null || session.getDB() == null) { + return null; + } + return session.captureMetadataCheckpoint(); + } + + /** + * A consistent snapshot of a RocksDB instance's metadata plus hard-links to its live SST set, + * materialised under {@link #getTempDir()} by {@link #captureMetadataSnapshot(String)}. + * + *

File names are relative to the checkpoint directory. {@link #getDbDir()} is the real DB + * directory the metadata belongs to — remote keys must be derived from {@code dbDir + name} (not + * the temp directory) so a restore lands each file back at its original path. + */ + public static final class MetadataSnapshot { + + private final String dbDir; + private final String tempDir; + private final String currentFileName; + private final String manifestFileName; + private final List optionsFileNames; + private final List sstFileNames; + private final List walFileNames; + + public MetadataSnapshot(String dbDir, String tempDir, String currentFileName, + String manifestFileName, List optionsFileNames, + List sstFileNames, List walFileNames) { + this.dbDir = dbDir; + this.tempDir = tempDir; + this.currentFileName = currentFileName; + this.manifestFileName = manifestFileName; + this.optionsFileNames = optionsFileNames; + this.sstFileNames = sstFileNames; + this.walFileNames = walFileNames; + } + + /** The real DB directory the captured metadata belongs to (for remote-key derivation). */ + public String getDbDir() { + return dbDir; + } + + /** The temporary checkpoint directory holding the metadata copies and SST hard-links. */ + public String getTempDir() { + return tempDir; + } + + /** The {@code CURRENT} file name (always {@code "CURRENT"}), or {@code null} if absent. */ + public String getCurrentFileName() { + return currentFileName; + } + + /** The {@code MANIFEST-} file name referenced by {@code CURRENT}, or {@code null}. */ + public String getManifestFileName() { + return manifestFileName; + } + + /** {@code OPTIONS-} file names present in the checkpoint. */ + public List getOptionsFileNames() { + return optionsFileNames; + } + + /** {@code *.sst} file names the captured manifest references (as hard-links). */ + public List getSstFileNames() { + return sstFileNames; + } + + /** WAL {@code *.log} file names captured (used by {@code wal} mode). */ + public List getWalFileNames() { + return walFileNames; + } + + /** Removes the temporary checkpoint directory. Never touches the real SST files. */ + public void cleanup() { + if (tempDir == null) { + return; + } + try { + FileUtils.deleteDirectory(new File(tempDir)); + } catch (Exception e) { + log.warn("Failed to clean up metadata checkpoint temp dir {}: {}", + tempDir, e.getMessage()); + } + } + } + + public boolean onReadMiss(RocksDBSession session, String table, byte[] key) { + if (session == null) { + return false; + } + boolean hydrated = false; + for (RocksdbChangedListener listener : rocksdbChangedListeners) { + try { + if (listener.onReadMiss(session, table, key)) { + hydrated = true; + } + } catch (Exception e) { + log.warn("onReadMiss listener failed for db={}, table={}: {}", + session.getGraphName(), table, e.getMessage()); + } + } + return hydrated; + } + + /** + * Returns the singleton {@link RocksdbEventListener} that should be registered + * with every new {@link org.rocksdb.DBOptions} via + * {@link org.rocksdb.DBOptions#setListeners}. + */ + public RocksdbEventListener getEventListener() { + return rocksdbEventListener; + } + public interface RocksdbChangedListener { default void onCompacted(String dbName) { } + default void onDBOpening(String dbName, String dbPath) { + } + + /** + * Called immediately after a new RocksDB instance has been opened for the first time. + * + *

Implementations can use this callback to upload any SST files that already exist + * in {@code dbPath} (e.g. from a previous run) and to trigger a MemTable flush so that + * any WAL-recovered data is also written to SST files and forwarded to cloud storage. + * + * @param dbName logical name of the graph / partition + * @param dbPath absolute path of the RocksDB directory + */ + default void onDBCreated(String dbName, String dbPath) { + } + default void onDBDeleteBegin(String dbName, String filePath) { } default void onDBDeleted(String dbName, String filePath) { } + default void onDBTruncateBegin(String dbName, String filePath) { + } + + default void onDBTruncated(String dbName, String filePath) { + } + default void onDBSessionReleased(RocksDBSession dbSession) { } + + /** + * Called after a new SST file has been successfully created by RocksDB. + * + * @param dbName RocksDB instance name + * @param cfName column-family name + * @param filePath absolute path of the new SST file + * @param fileSize size of the file in bytes + */ + default void onTableFileCreated(String dbName, String cfName, + String filePath, long fileSize) { + } + + /** + * Called after an SST file has been deleted by RocksDB. + * + * @param dbName RocksDB instance name + * @param cfName column-family name + * @param filePath absolute path of the deleted SST file + */ + default void onTableFileDeleted(String dbName, String cfName, String filePath) { + } + + /** + * Called when a get() operation returns null so listeners can attempt cloud hydration. + * + * @param session RocksDB session where miss happened + * @param table target column-family/table + * @param key requested key + * @return true if listener hydrated new local data and caller should retry get() + */ + default boolean onReadMiss(RocksDBSession session, String table, byte[] key) { + return false; + } } class DBSessionWatcher { diff --git a/hugegraph-store/hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/access/RocksDBSession.java b/hugegraph-store/hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/access/RocksDBSession.java index f4e7605a7f..d8e771e0fd 100644 --- a/hugegraph-store/hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/access/RocksDBSession.java +++ b/hugegraph-store/hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/access/RocksDBSession.java @@ -26,6 +26,7 @@ import java.nio.file.Paths; import java.util.ArrayList; import java.util.Arrays; +import java.util.Collections; import java.util.List; import java.util.Map; import java.util.Set; @@ -431,6 +432,11 @@ private void openRocksDB(String dbDataPath, long version) { RocksDBSession.initOptions(hugeConfig, opts, opts, opts, opts); dbOptions = new DBOptions(opts); dbOptions.setStatistics(rocksDbStats); + // Register the factory-level event listener so that table-file events + // (onTableFileCreated / onTableFileDeleted) are forwarded to all + // RocksdbChangedListener implementations (e.g. cloud storage providers). + dbOptions.setListeners( + Collections.singletonList(RocksDBFactory.getInstance().getEventListener())); try { List columnFamilyDescriptorList = @@ -598,8 +604,13 @@ public synchronized void truncate() { tableNames.remove(defaultCF); log.info("truncate table: {}", String.join(",", tableNames)); + // Notify listeners before table recreation so they can suppress cloud sync callbacks. + RocksDBFactory.getInstance().notifyTruncateBegin(this.graphName, this.dbPath); this.dropTables(tableNames.toArray(new String[0])); this.createTables(tableNames.toArray(new String[0])); + + // Notify listeners that truncate has completed so they can purge remote state. + RocksDBFactory.getInstance().notifyTruncate(this.graphName, this.dbPath); } public void flush(boolean wait) { @@ -693,6 +704,70 @@ public void saveSnapshot(String snapshotPath) throws DBStoreException { System.currentTimeMillis() - startTime); } + /** + * Captures a consistent copy of this DB's metadata (CURRENT / MANIFEST-* / OPTIONS-* / WAL) plus + * hard-links to the live SST set into a temporary directory, using a RocksDB + * {@link Checkpoint}. See {@link RocksDBFactory#captureMetadataSnapshot(String)} for the metadata + * rationale. + * + *

The temporary directory is a sibling of the DB directory (same filesystem), so SST files + * are hard-linked rather than copied. The caller owns cleanup via + * {@link RocksDBFactory.MetadataSnapshot#cleanup()}. + * + * @return the captured snapshot; never {@code null} + * @throws DBStoreException if the checkpoint cannot be created + */ + RocksDBFactory.MetadataSnapshot captureMetadataCheckpoint() throws DBStoreException { + String tempDir = this.dbPath + "_cloudmeta_" + System.nanoTime(); + cfHandleLock.readLock().lock(); + try (final Checkpoint checkpoint = Checkpoint.create(this.rocksDB)) { + final File tempFile = new File(tempDir); + FileUtils.deleteDirectory(tempFile); + checkpoint.createCheckpoint(tempDir); + } catch (final Exception e) { + try { + FileUtils.deleteDirectory(new File(tempDir)); + } catch (IOException ignore) { + // best-effort cleanup of a partial checkpoint + } + log.error("Fail to create metadata checkpoint at {}", tempDir, e); + throw new DBStoreException( + String.format("Fail to create metadata checkpoint at %s", tempDir)); + } finally { + cfHandleLock.readLock().unlock(); + } + return categoriseCheckpoint(tempDir); + } + + /** Classifies the files a checkpoint materialised into the metadata snapshot descriptor. */ + private RocksDBFactory.MetadataSnapshot categoriseCheckpoint(String tempDir) { + String currentFileName = null; + String manifestFileName = null; + List optionsFileNames = new ArrayList<>(); + List sstFileNames = new ArrayList<>(); + List walFileNames = new ArrayList<>(); + File[] files = new File(tempDir).listFiles(); + if (files != null) { + for (File f : files) { + String name = f.getName(); + if (name.equals("CURRENT")) { + currentFileName = name; + } else if (name.startsWith("MANIFEST-")) { + manifestFileName = name; + } else if (name.startsWith("OPTIONS-")) { + optionsFileNames.add(name); + } else if (name.endsWith(".sst")) { + sstFileNames.add(name); + } else if (name.endsWith(".log")) { + walFileNames.add(name); + } + } + } + return new RocksDBFactory.MetadataSnapshot(this.dbPath, tempDir, currentFileName, + manifestFileName, optionsFileNames, + sstFileNames, walFileNames); + } + private boolean verifySnapshot(String snapshotPath) { try { try (final Options options = new Options(); diff --git a/hugegraph-store/hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/access/SessionOperatorImpl.java b/hugegraph-store/hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/access/SessionOperatorImpl.java index b8259e5220..b1b79c211c 100644 --- a/hugegraph-store/hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/access/SessionOperatorImpl.java +++ b/hugegraph-store/hg-store-rocksdb/src/main/java/org/apache/hugegraph/rocksdb/access/SessionOperatorImpl.java @@ -208,7 +208,14 @@ public void deleteRange(byte[] keyFrom, byte[] keyTo) throws DBStoreException { @Override public byte[] get(String table, byte[] key) throws DBStoreException { try (CFHandleLock cf = this.getLock(table)) { - return rocksdb().get(cf.get(), key); + byte[] value = rocksdb().get(cf.get(), key); + if (value != null) { + return value; + } + if (RocksDBFactory.getInstance().onReadMiss(this.session, table, key)) { + return rocksdb().get(cf.get(), key); + } + return null; } catch (RocksDBException e) { throw new DBStoreException(e); } diff --git a/hugegraph-store/pom.xml b/hugegraph-store/pom.xml index 9ff1e933e5..97e2391c14 100644 --- a/hugegraph-store/pom.xml +++ b/hugegraph-store/pom.xml @@ -21,7 +21,6 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 hugegraph-store - ${revision} pom @@ -41,6 +40,7 @@ hg-store-node hg-store-dist hg-store-cli + hg-store-cloud-s3 @@ -80,6 +80,11 @@ hg-store-core ${project.version} + + org.apache.hugegraph + hg-store-cloud-s3 + ${project.version} + org.apache.hugegraph hg-store-transfer @@ -302,5 +307,13 @@ + + + + cloud-s3 + + hg-store-cloud-s3 + + diff --git a/install-dist/release-docs/LICENSE b/install-dist/release-docs/LICENSE index 0014317824..6b90c62ac2 100644 --- a/install-dist/release-docs/LICENSE +++ b/install-dist/release-docs/LICENSE @@ -245,6 +245,14 @@ The text of each license is also included in licenses/LICENSE-[project].txt. https://central.sonatype.com/artifact/org.hdrhistogram/HdrHistogram/2.1.12 -> CC0 1.0 https://central.sonatype.com/artifact/org.hdrhistogram/HdrHistogram/2.1.9 -> CC0 1.0 +======================================================================== +Third party MIT-0 licenses +======================================================================== +The following components are provided under the MIT-0 License. See project link for details. +The text of each license is also included in licenses/LICENSE-[project].txt. + + https://central.sonatype.com/artifact/org.reactivestreams/reactive-streams/1.0.4 -> MIT-0 + ======================================================================== Third party BSD-2-Clause licenses ======================================================================== @@ -444,28 +452,38 @@ non-Apache-2.0 license text is required for a bundled component. https://central.sonatype.com/artifact/io.netty/netty-all/4.1.42.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-all/4.1.44.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-all/4.1.61.Final -> Apache 2.0 + https://central.sonatype.com/artifact/io.netty/netty-buffer/4.1.108.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-buffer/4.1.52.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-buffer/4.1.72.Final -> Apache 2.0 + https://central.sonatype.com/artifact/io.netty/netty-codec-http/4.1.108.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-codec-http/4.1.52.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-codec-http/4.1.72.Final -> Apache 2.0 + https://central.sonatype.com/artifact/io.netty/netty-codec-http2/4.1.108.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-codec-http2/4.1.52.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-codec-http2/4.1.72.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-codec-socks/4.1.52.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-codec-socks/4.1.72.Final -> Apache 2.0 + https://central.sonatype.com/artifact/io.netty/netty-codec/4.1.108.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-codec/4.1.52.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-codec/4.1.72.Final -> Apache 2.0 + https://central.sonatype.com/artifact/io.netty/netty-common/4.1.108.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-common/4.1.52.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-common/4.1.72.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-handler-proxy/4.1.52.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-handler-proxy/4.1.72.Final -> Apache 2.0 + https://central.sonatype.com/artifact/io.netty/netty-handler/4.1.108.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-handler/4.1.52.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-handler/4.1.72.Final -> Apache 2.0 + https://central.sonatype.com/artifact/io.netty/netty-resolver/4.1.108.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-resolver/4.1.52.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-resolver/4.1.72.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-tcnative-boringssl-static/2.0.25.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-tcnative-boringssl-static/2.0.36.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-tcnative-classes/2.0.46.Final -> Apache 2.0 + https://central.sonatype.com/artifact/io.netty/netty-transport-classes-epoll/4.1.108.Final -> Apache 2.0 + https://central.sonatype.com/artifact/io.netty/netty-transport-native-unix-common/4.1.108.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-transport-native-unix-common/4.1.72.Final -> Apache 2.0 + https://central.sonatype.com/artifact/io.netty/netty-transport/4.1.108.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-transport/4.1.52.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.netty/netty-transport/4.1.72.Final -> Apache 2.0 https://central.sonatype.com/artifact/io.opentracing/opentracing-api/0.22.0 -> Apache 2.0 @@ -740,6 +758,33 @@ non-Apache-2.0 license text is required for a bundled component. https://central.sonatype.com/artifact/org.yaml/snakeyaml/1.28 -> Apache 2.0 https://central.sonatype.com/artifact/org.yaml/snakeyaml/2.2 -> Apache 2.0 https://central.sonatype.com/artifact/org.zeroturnaround/zt-zip/1.14 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/annotations/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/apache-client/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/arns/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/auth/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/aws-core/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/aws-query-protocol/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/aws-xml-protocol/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/checksums-spi/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/checksums/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/crt-core/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/endpoints-spi/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/http-auth-aws/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/http-auth-spi/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/http-auth/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/http-client-spi/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/identity-spi/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/json-utils/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/metrics-spi/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/netty-nio-client/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/profiles/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/protocol-core/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/regions/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/s3/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/sdk-core/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/third-party-jackson-core/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.awssdk/utils/2.33.8 -> Apache 2.0 + https://central.sonatype.com/artifact/software.amazon.eventstream/eventstream/1.0.1 -> Apache 2.0 ======================================================================== Third party BSD licenses diff --git a/install-dist/release-docs/NOTICE b/install-dist/release-docs/NOTICE index 775f03241c..6cddb47b85 100644 --- a/install-dist/release-docs/NOTICE +++ b/install-dist/release-docs/NOTICE @@ -1890,6 +1890,38 @@ Copyright 2020-2021 SmartBear Software Inc. ======================================================================== +AWS SDK for Java 2.0 NOTICE + +======================================================================== + +AWS SDK for Java 2.0 +Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +This product includes software developed by +Amazon Technologies, Inc (http://www.amazon.com/). + +********************** +THIRD PARTY COMPONENTS +********************** +This software includes third party software subject to the following copyrights: +- XML parsing and utility functions from JetS3t - Copyright 2006-2009 James Murty. +- PKCS#1 PEM encoded private key parsing and utility functions from oauth.googlecode.com - Copyright 1998-2010 AOL Inc. +- Apache Commons Lang - https://github.com/apache/commons-lang +- Netty Reactive Streams - https://github.com/playframework/netty-reactive-streams +- Jackson-core - https://github.com/FasterXML/jackson-core +- Jackson-dataformat-cbor - https://github.com/FasterXML/jackson-dataformats-binary + +The licenses for these third party components are included in LICENSE.txt + +- For Apache Commons Lang see also this required NOTICE: + Apache Commons Lang + Copyright 2001-2020 The Apache Software Foundation + + This product includes software developed at + The Apache Software Foundation (https://www.apache.org/). + +======================================================================== + # # jansi 2.4.0 (https://github.com/fusesource/jansi) # diff --git a/install-dist/release-docs/licenses/LICENSE-reactive-streams-1.0.4.txt b/install-dist/release-docs/licenses/LICENSE-reactive-streams-1.0.4.txt new file mode 100644 index 0000000000..cfcf3bd994 --- /dev/null +++ b/install-dist/release-docs/licenses/LICENSE-reactive-streams-1.0.4.txt @@ -0,0 +1,17 @@ +MIT No Attribution + +Copyright (c) 2015-2022 Reactive Streams contributors + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies +of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + diff --git a/install-dist/scripts/dependency/known-dependencies.txt b/install-dist/scripts/dependency/known-dependencies.txt index 326ac3dd76..8936446ca8 100644 --- a/install-dist/scripts/dependency/known-dependencies.txt +++ b/install-dist/scripts/dependency/known-dependencies.txt @@ -10,12 +10,15 @@ animal-sniffer-annotations-1.14.jar animal-sniffer-annotations-1.18.jar animal-sniffer-annotations-1.19.jar annotations-13.0.jar +annotations-2.33.8.jar annotations-24.0.1.jar annotations-4.1.1.4.jar ansj_seg-5.1.6.jar antlr-runtime-3.5.2.jar aopalliance-repackaged-3.0.1.jar +apache-client-2.33.8.jar apiguardian-api-1.1.0.jar +arns-2.33.8.jar arthas-agent-attach-3.6.4.jar arthas-agent-attach-3.7.1.jar arthas-packaging-3.6.4.jar @@ -33,8 +36,12 @@ asm-util-5.0.3.jar assertj-core-3.19.0.jar ast-9.0-9.0.20190305.jar audience-annotations-0.13.0.jar +auth-2.33.8.jar auto-service-annotations-1.0.jar automaton-1.11-8.jar +aws-core-2.33.8.jar +aws-query-protocol-2.33.8.jar +aws-xml-protocol-2.33.8.jar bolt-1.6.2.jar bolt-1.6.4.jar byte-buddy-1.10.20.jar @@ -51,6 +58,8 @@ checker-qual-2.0.0.jar checker-qual-3.12.0.jar checker-qual-3.33.0.jar checker-qual-3.5.0.jar +checksums-2.33.8.jar +checksums-spi-2.33.8.jar chronicle-bytes-2.20.111.jar chronicle-core-2.20.126.jar chronicle-queue-5.20.123.jar @@ -63,6 +72,7 @@ commons-cli-1.5.0.jar commons-codec-1.11.jar commons-codec-1.13.jar commons-codec-1.15.jar +commons-codec-1.17.1.jar commons-codec-1.9.jar commons-collections-3.2.2.jar commons-collections4-4.4.jar @@ -85,6 +95,7 @@ commons-pool2-2.0.jar commons-text-1.10.0.jar commons-text-1.9.jar concurrent-trees-2.4.0.jar +crt-core-2.33.8.jar cypher-gremlin-extensions-1.0.4.jar disruptor-3.3.7.jar disruptor-3.4.1.jar @@ -92,12 +103,14 @@ eclipse-collections-10.4.0.jar eclipse-collections-11.1.0.jar eclipse-collections-api-10.4.0.jar eclipse-collections-api-11.1.0.jar +endpoints-spi-2.33.8.jar error_prone_annotations-2.1.3.jar error_prone_annotations-2.10.0.jar error_prone_annotations-2.18.0.jar error_prone_annotations-2.3.4.jar error_prone_annotations-2.4.0.jar error_prone_annotations-2.48.0.jar +eventstream-1.0.1.jar exp4j-0.4.8.jar expressions-9.0-9.0.20190305.jar failsafe-2.4.1.jar @@ -193,8 +206,15 @@ hk2-utils-3.0.1.jar hppc-0.7.1.jar hppc-0.8.1.jar htrace-core4-4.1.0-incubating.jar +http-auth-2.33.8.jar +http-auth-aws-2.33.8.jar +http-auth-aws-eventstream-2.33.8.jar +http-auth-spi-2.33.8.jar +http-client-spi-2.33.8.jar httpclient-4.5.13.jar httpcore-4.4.13.jar +httpcore-4.4.16.jar +identity-spi-2.33.8.jar ikanalyzer-2012_u6.jar ivy-2.4.0.jar j2objc-annotations-1.1.jar @@ -324,6 +344,7 @@ jraft-core-1.3.9.jar json-path-2.5.0.jar json-simple-1.1.jar json-smart-2.3.jar +json-utils-2.33.8.jar jsonassert-1.5.0.jar jsr305-3.0.1.jar jsr305-3.0.2.jar @@ -421,6 +442,7 @@ metrics-core-4.2.4.jar metrics-jersey3-4.2.4.jar metrics-jvm-3.1.5.jar metrics-logback-3.1.5.jar +metrics-spi-2.33.8.jar micrometer-core-1.7.12.jar micrometer-registry-prometheus-1.7.12.jar mmseg4j-core-1.10.0.jar @@ -431,33 +453,44 @@ mxdump-0.14.jar netty-all-4.1.42.Final.jar netty-all-4.1.44.Final.jar netty-all-4.1.61.Final.jar +netty-buffer-4.1.126.Final.jar netty-buffer-4.1.52.Final.jar netty-buffer-4.1.72.Final.jar +netty-codec-4.1.126.Final.jar netty-codec-4.1.52.Final.jar netty-codec-4.1.72.Final.jar +netty-codec-http-4.1.126.Final.jar netty-codec-http-4.1.52.Final.jar netty-codec-http-4.1.72.Final.jar +netty-codec-http2-4.1.126.Final.jar netty-codec-http2-4.1.52.Final.jar netty-codec-http2-4.1.72.Final.jar netty-codec-socks-4.1.52.Final.jar netty-codec-socks-4.1.72.Final.jar +netty-common-4.1.126.Final.jar netty-common-4.1.52.Final.jar netty-common-4.1.72.Final.jar +netty-handler-4.1.126.Final.jar netty-handler-4.1.130.Final.jar netty-handler-4.1.52.Final.jar netty-handler-4.1.72.Final.jar netty-handler-proxy-4.1.52.Final.jar netty-handler-proxy-4.1.72.Final.jar +netty-nio-client-2.33.8.jar +netty-resolver-4.1.126.Final.jar netty-resolver-4.1.130.Final.jar netty-resolver-4.1.52.Final.jar netty-resolver-4.1.72.Final.jar netty-tcnative-boringssl-static-2.0.25.Final.jar netty-tcnative-boringssl-static-2.0.36.Final.jar netty-tcnative-classes-2.0.46.Final.jar +netty-transport-4.1.126.Final.jar netty-transport-4.1.52.Final.jar netty-transport-4.1.72.Final.jar +netty-transport-classes-epoll-4.1.126.Final.jar netty-transport-classes-epoll-4.1.130.Final.jar netty-transport-native-epoll-4.1.130.Final.jar +netty-transport-native-unix-common-4.1.126.Final.jar netty-transport-native-unix-common-4.1.72.Final.jar nimbus-jose-jwt-4.41.2.jar nlp-lang-1.7.7.jar @@ -496,6 +529,7 @@ powermock-module-junit4-2.0.0-RC.3.jar powermock-module-junit4-common-2.0.0-RC.3.jar powermock-module-junit4-rule-2.0.0-RC.3.jar powermock-reflect-2.0.0-RC.3.jar +profiles-2.33.8.jar proto-google-common-protos-1.17.0.jar proto-google-common-protos-2.0.1.jar protobuf-java-3.11.0.jar @@ -503,20 +537,27 @@ protobuf-java-3.17.2.jar protobuf-java-3.21.7.jar protobuf-java-3.5.1.jar protobuf-java-util-3.17.2.jar +protocol-core-2.33.8.jar protostuff-api-1.6.0.jar protostuff-collectionschema-1.6.0.jar protostuff-core-1.6.0.jar protostuff-runtime-1.6.0.jar psjava-0.1.19.jar +reactive-streams-1.0.4.jar +regions-2.33.8.jar reporter-config-base-3.0.3.jar reporter-config3-3.0.3.jar +retries-2.33.8.jar +retries-spi-2.33.8.jar rewriting-9.0-9.0.20190305.jar rocksdbjni-6.29.5.jar rocksdbjni-7.7.3.jar rocksdbjni-8.10.2.jar +s3-2.33.8.jar scala-java8-compat_2.12-0.8.0.jar scala-library-2.12.7.jar scala-reflect-2.12.7.jar +sdk-core-2.33.8.jar shims-0.9.38.jar sigar-1.6.4.jar simpleclient-0.10.0.jar @@ -539,6 +580,7 @@ slf4j-api-1.7.21.jar slf4j-api-1.7.25.jar slf4j-api-1.7.31.jar slf4j-api-1.7.32.jar +slf4j-api-1.7.36.jar slf4j-api-2.0.9.jar snakeyaml-1.18.jar snakeyaml-1.26.jar @@ -591,12 +633,15 @@ swagger-integration-jakarta-2.2.18.jar swagger-jaxrs2-jakarta-2.2.18.jar swagger-models-1.5.18.jar swagger-models-jakarta-2.2.18.jar +third-party-jackson-core-2.33.8.jar tinkergraph-gremlin-3.5.1.jar token-provider-2.0.0.jar tomcat-embed-el-9.0.63.jar tracer-core-3.0.8.jar translation-1.0.4.jar +url-connection-client-2.33.8.jar util-9.0-9.0.20190305.jar +utils-2.33.8.jar validation-api-1.1.0.Final.jar websocket-api-9.4.46.v20220331.jar websocket-client-9.4.46.v20220331.jar diff --git a/install-dist/scripts/dependency/regenerate_known_dependencies.sh b/install-dist/scripts/dependency/regenerate_known_dependencies.sh old mode 100644 new mode 100755 diff --git a/pom.xml b/pom.xml index 850ac99fa8..0b5044ba81 100644 --- a/pom.xml +++ b/pom.xml @@ -217,6 +217,10 @@ **/pid **/tmp/** + + docker/cloud-storage/** + + **/.generated/** **/src/main/java/org/apache/hugegraph/pd/grpc/** **/src/main/java/org/apache/hugegraph/store/grpc/** @@ -311,6 +315,9 @@ **/*.txt **/.flattened-pom.xml **/apache-hugegraph-*/**/* + + docker/cloud-storage/** + **/.generated/**