mongo-hourly-monitor

Apify Actor that takes an hourly snapshot of a sharded MongoDB Atlas cluster. We use it to watch the system and to check that our query/index/write optimizations actually help. It reuses the query-shape logic from the offline analysis pipeline in the parent directory, so results are comparable.

Run it on an Apify Schedule with cron 15 * * * * (UTC). A run at 10:15 collects:

Phase	Source	Window
Slow queries	every mongod's log via atlas logs download	09:00–10:00
Metrics	atlas metrics processes + atlas metrics disks, all measurements	09:00–10:00, smallest granularity that returns data (PT10S, else PT1M)
Oplog	local.oplog.rs on one secondary per shard	last 20 minutes
Indexes	$indexStats + $collStats via mongos	point-in-time

Outputs

Two datasets (defined in .actor/actor.json under storages.datasets):

• queryShapes — one item per slow-query shape: load totals (wall time, CPU, docs examined, bytes sent, blocking sorts, COLLSCANs) and one real example operation (the slowest one, with appName and host). getMore batches are counted into the query that opened the cursor.
• indexes — one item per index: key, size (summed over shards), document count, bytes per document, per-node usage counters, ops/day and flags (hidden / unused / near-unused / bloated).

One key-value store (the default one) with two collections:

• report-* — the human-facing reports: report-top-slowest, report-top-most-total-time, report-top-most-docs-examined, report-top-most-blocking-sorts, report-top-heaviest-cpu, report-top-most-data-sent, report-top-most-frequent, report-top-worst-scan-ratio (top 50 shapes per category), report-collscan-operations (every COLLSCAN op), report-top-largest-indexes (top 50), report-oplog-top-by-bytes and report-oplog-top-by-count (top 20 write groups, grouped by namespace + op + fields written, not just by collection), report-oplog-by-namespace, report-collections-storage (one entry per collection: docs, data/storage/index bytes, reusable free space, per-shard split + sharded flag — from the same $collStats pass as the index sweep), report-metrics-charts (HTML dashboard), report-query-shapes-overview and report-summary.
• metric-* — every raw metrics download, one file per process or disk, plus metric-files-index listing them.

Permissions & secrets (environment variables)

Set these as secret environment variables on the Actor:

Env var	What
MONGODB_ATLAS_PUBLIC_API_KEY	Atlas API key, public part
MONGODB_ATLAS_PRIVATE_API_KEY	Atlas API key, private part
MONGODB_ATLAS_PROJECT_ID	Atlas project id
MONGO_USERNAME	Atlas database user
MONGO_PASSWORD	its password

Atlas API key — project roles

• Project Read Only — needed for atlas processes list and all atlas metrics calls.
• Project Data Access Read Only — needed for atlas logs download (Project Read Only is NOT enough for logs).

Give the key both roles. If your organization requires an access list on API keys, add the Actor's outgoing IPs (0.0.0.0/0 on the shared Apify platform, or a static IP if you use one).

MongoDB database user — roles

Create the user in Atlas → Database Access with these privileges:

• clusterMonitor@admin — listDatabases, $indexStats, $collStats.
• backup@admin — read access to local.oplog.rs. Regular users cannot read the local database on Atlas; backup is the smallest role that can.
• read@apifier-prod — listCollections on the analyzed database.

("Atlas admin" also works but grants much more than needed.)

The cluster's IP access list must allow the Actor too. The Actor connects to mongos (port 27016) for the index sweep and directly to one secondary mongod per shard (port 27017) for the oplog.

Input

All optional: clusterName (apify-prod), databaseName (apifier-prod), windowMinutes (60 — the window always ends at the last full hour), oplogWindowMinutes (20), skipOplog (false — skips the oplog write-attribution phase entirely).

Development

pnpm install
pnpm run build          # tsc
pnpm run lint           # oxlint
pnpm run format:check   # oxfmt

# local run (needs the atlas CLI on PATH and the env vars above)
export MONGODB_ATLAS_PUBLIC_API_KEY=… MONGODB_ATLAS_PRIVATE_API_KEY=… \
       MONGODB_ATLAS_PROJECT_ID=… MONGO_USERNAME=… MONGO_PASSWORD=…
apify run

# deploy, then create the Schedule (cron "15 * * * *", timezone UTC)
apify push

Suggested run options: 4 GB memory, timeout 30 minutes (a normal run finishes well under 10). Phases run in parallel and a failing phase does not stop the others; the run is marked failed at the end if anything was incomplete, with per-phase status in report-summary.

Things to know

• The slow log is a floor, not a census. Only operations slower than each node's slowms get logged (200 ms on prod since 2026-07-06, set by set-profiler.sh in the parent repo). Comparisons across runs assume the same slowms on every node.
• PT10S granularity is not available on every tier/window. The Actor falls back to PT1M per host; metric-files-index records which one each file used.
• Oplog "fields written" means top-level fields of the update diff. A $set of stats.x and of stats.y both count as stats.
• Batched inserts are unwrapped. MongoDB 8.0 replicates a batch of inserts as one applyOps oplog entry; the pipeline unwraps it so every insert is counted. A naive find({ns: …}) on the oplog misses these.
• Hidden indexes always show 0 ops (the query planner cannot see them), so they are flagged hidden, not unused. Usage counters reset when a node restarts — check counterWindowDays before trusting ops/day.
• Known-noise sources are skipped in the slow-query analysis: all mongot / Atlas Search operations (its search queries and its own change-stream reads, matched by appName), the batched $cmd update commands of the MONGO_UPDATE_THROTTLED write-behind daemons, MongoDB's own mirrored reads (command.mirrored: true cache warming), and interactive human clients (MongoDB Shell / Compass / mongosh). The analytics node — the last member of each shard — is left out of log processing altogether. Per-node skip counts and the skipped hosts are in report-query-shapes-overview.
• Oplog namespace totals carry a per-shard split (perShard) and a sharded flag read from config.collections via mongos, so reports can show which shard receives each collection's writes.