objstoresoak

A multi-day soak test for TidesDB running in object-store mode, with a real-time dashboard of whats going on. It hammers a single instance with four concurrent workloads, writes deterministic sequential keys so reads can verify correctness (not just latency), and records everything so you can watch the LSM tree and the object-store pipeline evolve over days.

What it does

Six cooperative workloads share one TidesDB handle (object-store mode forces unified-memtable, and TidesDB takes an exclusive directory lock, so one shared handle is the only option, TidesDB's own flush/compaction/upload threads provide the real parallelism):

Workload	Behaviour
Writer	Sequential keys key:<20-digit seq>, deterministic values, token-bucket paced to a steady target rate.
Point reader	Mixes hot (recent) and cold (old, fetched from object store) keys; verifies each value against the mutation model.
Range reader	Seeks a random start and scans forward, asserting keys come back strictly increasing and each value matches the model.
Mutator	Delete and update phases that switch on and off on random timers; while active they tombstone or rewrite random live keys on the main CF, recorded in an in-memory model so reads stay verifiable.
CF scaler	Creates and drops auxiliary CFs (soak_aux_N) over time toward a random target, driving live put/get/delete traffic onto the ones that exist. The verified main CF is never touched.
Stats probe	Samples getDbStats, cf.getStats and getCacheStats once a second and folds engine internals into each metrics frame.

Every value on the main CF is fully determined by its key and update generation, so the readers verify exact content, not just latency. A mismatch, or a key that should exist but is missing, is a real correctness failure, surfaced on the dashboard as verify errors, the single most important number in a multi-day run.

Architecture

soak process (Node, one tidesdb handle)
  writer · point · range · mutator · CF scaler   (cooperative paced loops)
        └───────────────► TidesDB ──► object store (GCS-S3 or local FS)
                             │        engine log → data/db/LOG (DEBUG, truncated at 512 MB)
  stats probe ──► getDbStats / cf.getStats / getCacheStats
        │
        ├─ metrics.jsonl   full history, survives restart
        └─ HTTP :8080
             /            uPlot dashboard (dark / light themes)
             /api/history backlog on page load (downsampled)
             /api/stream  SSE, one frame/sec live push

Each workload runs as a self-rescheduling timer with a token-bucket pacer, so load is a steady rate over days rather than a burst that burns out in an hour. Every tick is bounded so the HTTP server is never starved.

Quick start (local, filesystem connector)

npm install
npm run dev          # TIDES_OBJSTORE=fs, writes to ./data
# open http://localhost:8080

Requires libtidesdb installed (ldconfig -p | grep tidesdb). The tidesdb npm dep resolves to ../tidesdb-ts.

Running against Google Cloud Storage

TidesDB's TypeScript binding speaks the S3-compatible connector, and GCS exposes an S3 XML API at storage.googleapis.com. So "object store on GCS" = the S3 connector pointed there with HMAC keys and path-style addressing.

gsutil mb gs://my-tidesdb-soak
gsutil hmac create SERVICE_ACCOUNT_EMAIL   # prints access key + secret

export TIDES_OBJSTORE=gcs
export TIDES_GCS_BUCKET=my-tidesdb-soak
export TIDES_GCS_ACCESS_KEY=GOOG1E...
export TIDES_GCS_SECRET_KEY=...
npm start

Deploying on a GCE instance

1. Create a VM (e.g. e2-standard-4, Ubuntu), install Node 20+ and libtidesdb.
2. Copy this repo to /opt/objstoresoak, then npm install and npm install tidesdb (the published binding; locally it resolves to ../tidesdb-ts).
3. cp deploy/objstoresoak.env.example /etc/objstoresoak.env, fill in GCS creds, chmod 600.
4. cp deploy/objstoresoak.service /etc/systemd/system/ && systemctl enable --now objstoresoak.
5. Open the dashboard port to the world:

   gcloud compute firewall-rules create objstoresoak-dash \
     --allow tcp:8080 --target-tags=objstoresoak --source-ranges=0.0.0.0/0
   gcloud compute instances add-tags INSTANCE --tags=objstoresoak

The dashboard is read-only, it exposes no controls, so it's safe to make public.

Config (env vars)

Var	Default	Meaning
TIDES_OBJSTORE	fs	fs or gcs
TIDES_DB_PATH	./data/db	Local DB / cache directory
TIDES_FS_OBJ	./data/objstore	FS connector object root
TIDES_CF	soak	Main (verified) column family name
TIDES_GCS_BUCKET / _ACCESS_KEY / _SECRET_KEY	-	GCS S3 credentials
TIDES_GCS_ENDPOINT / _PREFIX / _REGION	storage.googleapis.com / objstoresoak/ / auto	GCS S3 endpoint, key prefix, region
TIDES_CACHE_BYTES	512 MiB	Local object-store cache budget (forces refetch when exceeded)
TIDES_WBUF	0 (lib default)	Unified memtable flush threshold (bytes); lower it to make the LSM flush/compact - and the LSM panels move - sooner
TIDES_WRITE_RATE	2000	Target committed puts/sec
TIDES_WRITE_BATCH	250	Puts per transaction
TIDES_VALUE_SIZE	256	Bytes per value
TIDES_GET_RATE / TIDES_GET_BATCH	1500 / 100	Target point gets/sec, gets per reader tick
TIDES_HOT_FRACTION	0.3	Share of gets aimed at recent keys
TIDES_RANGE_RATE / TIDES_RANGE_LEN	30 / 200	Range scans/sec, entries per scan
TIDES_MUTATE	true	Enable the delete/update phases
TIDES_DELETE_RATE / TIDES_UPDATE_RATE	400 / 600	Deletes/updates per sec while that phase is active
TIDES_MUTATE_CAP	1000000	Max distinct mutated keys tracked in memory
TIDES_SCALE_CFS	true	Enable auxiliary CF create/drop
TIDES_MAX_AUX_CFS	6	Upper bound on live auxiliary CFs
TIDES_CF_WRITE_RATE	1500	Ops/sec spread across the live auxiliary CFs
TIDES_CF_SCALE_MS	20000	How often a new aux-CF target is chosen
TIDES_LOG_LEVEL / TIDES_LOG_FILE / TIDES_LOG_TRUNC	debug / true / 512 MiB	Engine log level, log-to-file, auto-truncation size
TIDES_STATS_MS	1000	Stats-probe + frame-emit interval
PORT / HTTP_HOST	8080 / 0.0.0.0	Dashboard bind
TIDES_RING_SECONDS	86400	In-memory history at 1 Hz
TIDES_METRICS_FILE / TIDES_SEQ_FILE	./data/metrics.jsonl / ./data/seq.json	History + write-cursor files

Restart safety

The write cursor is persisted to data/seq.json each second; on restart the writer resumes past it instead of rewriting from zero, and the dashboard warms its history from data/metrics.jsonl. The local DB dir doubles as the object-store cache, so a restart is a warm start; deleting it triggers TidesDB's cold-start recovery from the object store.

The mutation model is in-memory only. On restart, keys written in earlier runs fall below a modelFloor and are verified weakly (any value present must belong to that key), while keys written in the new run are verified strictly (exact generation, deletes expected to be absent). This keeps verification sound across restarts without persisting a large model.

Logging

The engine logs at DEBUG to data/db/LOG rather than stderr, so the console/journal stays clean over a multi-day run, and truncates that file in place once it reaches 512 MB. One library line still prints to stderr during early init, before the log config takes effect. Tune with TIDES_LOG_LEVEL / TIDES_LOG_FILE / TIDES_LOG_TRUNC.

Dashboard

Two themes - a dark phosphor mission-control console and a light printed-telemetry look - toggled in the header (or via ?theme=dark|light).

Panels: throughput · range scans · point/write/range latency (p50/p99) · reader & block-cache hit rate · LSM shape (total + L0 SSTables, data size) · read amplification / levels · persisted vs logical keys (the gap is memtable backlog) · mutations (delete/update ops/sec) · tombstones (count, ratio, worst per-SSTable density) · column families (live count + aux ops/sec) · CF lifecycle (created/dropped) · flush/compaction counts · flush/compaction backlog · object-store upload queue, total uploads, failures · local cache usage · write amplification · memory (RSS, pressure).

The header shows backend, M.E.T. (uptime), keys written, write target, live aux-CF count, the active mutation phase, and the verify/op error counters - which turn red on any nonzero count.