Your data fits on one machine. Justify the distributed system.
Licensing note: every tool below is tagged with its license so you can filter for genuine FOSS at a glance — (proprietary, hosted) marks closed/SaaS-only offerings included for completeness, not endorsement of lock-in.
A curated, decision-guide awesome-list of tools, patterns, and evidence for staying on a single node — one fast box with a good engine on it — before reaching for a distributed data or compute system. This is not an anti-distributed-systems manifesto. It's a corrective: most teams reach for Spark, Hadoop, or a distributed-by-default architecture long before their data or workload actually requires it, and pay the operational tax (cluster management, network partition handling, coordination overhead, harder debugging) for scale they never hit.
Modern single-node engines are dramatically more capable than the ones that made "just use Hadoop" the default advice a decade ago: vectorized execution, columnar formats, and cheap large-memory/NVMe instances mean a single well-chosen machine can comfortably chew through hundreds of gigabytes to low terabytes of data that would previously have "needed a cluster." The sentiment behind pieces like Adam Drake's "Command-line tools can be 235x faster than your Hadoop cluster" and Frank McSherry, Michael Isard, and Derek Murray's COST paper ("Scalability! But at what COST?", HotOS 2015) still holds: a lot of "distributed" jobs never beat a competent single-threaded or single-node implementation, because the distributed version spends its budget on coordination rather than doing more useful work per second.
- Decision Heuristics
- Single-Node OLAP / Analytics Engines
- Single-Node ETL & Orchestration
- Embedded / Single-Node Search
- Single-Node Queues & Streaming
- Vertical Scaling: Instance Options
- When to Actually Go Distributed
- Further Reading
Rough guidance, not gospel — the right threshold depends on your engine, query patterns, and team.
| Signal | Single-node is probably fine | Consider distributed |
|---|---|---|
| Dataset size | Fits on local NVMe / fits in RAM with room to spare (roughly up to a few hundred GB – low TB for analytics workloads with a columnar engine) | Genuinely multi-TB to PB, or growing faster than a single box's disk/RAM can track |
| Team size / ops maturity | Small team, no dedicated platform/SRE function | Dedicated infra team that can own cluster operations |
| Query pattern | Batch analytics, ad hoc queries, scheduled jobs | Massive concurrent multi-tenant query load that saturates one machine's cores/IO |
| Failure tolerance | A single machine restarting is an acceptable blip | You need to keep serving through a single machine's hardware failure with no downtime |
| Cost sensitivity | You'd rather pay for one bigger box than manage a fleet | Cluster cost is genuinely lower than the biggest single-box option available to you |
If none of the right-hand column is true yet, a single node is very likely the right call — and you can always outgrow it later; a well-designed single-node pipeline (e.g. Parquet files + a columnar engine) tends to port to a distributed engine reading the same files with much less pain than the reverse migration.
- DuckDB
(MIT)— in-process, columnar, vectorized SQL OLAP engine; the flagship "just embed it" analytics database. - ClickHouse
(Apache-2.0)— column-oriented DBMS; runs superbly as a single powerful node, distributed mode is opt-in, not mandatory. - Polars
(MIT)— Rust-based, multi-threaded DataFrame library with lazy query optimization; a single-machine alternative to distributed DataFrame APIs. - Apache DataFusion
(Apache-2.0)— extensible, vectorized query engine (Rust) usable embedded in a single process. - chDB
(Apache-2.0)— embedded, in-process version of ClickHouse, DuckDB-style ergonomics. - DataFrame.jl
(MIT)— high-performance single-machine tabular data manipulation for Julia workloads.
All FOSS — this category has no proprietary-only gap to fill.
- dbt
(Apache-2.0)— SQL transformation tool; pairs naturally with DuckDB for a fully single-node ELT stack. - dbt-duckdb
(Apache-2.0)— the dbt adapter that makes "dbt + DuckDB, no warehouse" a first-class workflow. - Dagster
(Apache-2.0)— can run as a single-process orchestrator for small-to-mid pipelines before you need a distributed executor. - Prefect
(Apache-2.0)— similarly scales down to a single machine/process cleanly. - SQLMesh
(Apache-2.0)— transformation framework with a strong single-engine (DuckDB) local dev/prod story.
- Meilisearch
(MIT)— fast, single-binary search engine; no cluster required to get production-grade relevance. - Typesense
(GPL-3.0)— typo-tolerant search, runs great as a single node for small-to-mid corpora. - Tantivy
(MIT)— Rust full-text search library (the engine under Quickwit); embed it directly, no server. - SQLite FTS5
(public domain)— full-text search built directly into SQLite; zero extra infrastructure.
- Redis Streams
(AGPL-3.0 as of Redis 8; RSALv2/SSPL for the 2024–2025 releases in between — not FOSS during that window)— durable, consumer-group log semantics without standing up Kafka. Redis relicensed away from its original BSD-3-Clause license in March 2024 to source-available RSALv2/SSPL, then returned to open source under AGPL-3.0 with Redis 8 in 2025 — check which version/license you're actually pulling. - Valkey
(BSD-3-Clause)— the Linux Foundation–backed fork of pre-relicensing Redis; the unambiguous FOSS choice if license certainty matters more than tracking upstream Redis. - NATS
(Apache-2.0)(core, non-clustered mode) — lightweight pub/sub and request-reply messaging, trivial single-node deploy. - SQLite-based queues
(MIT)(e.g. litequeue) — a job queue backed by a single SQLite file; no broker process at all. - Faktory
(open source — verify current license before depending on it; historically AGPL-3.0-style)— background job server that runs comfortably as a single instance for most workloads.
Reach for Apache Kafka (Apache-2.0) or a distributed streaming platform when you genuinely need multi-producer/multi-consumer durability at a throughput and retention scale a single broker can't hold, or cross-datacenter replication — not by default.
Before splitting a workload across machines, check how far "one bigger machine" actually goes. Everything in this section is proprietary cloud/hosting infrastructure, not software — there's no FOSS equivalent to "a bigger machine," but the engines you'd run on it (above) are all open source:
- High-memory cloud instance families (e.g. AWS
r/xseries, GCPm1/m2, AzureM-series)(proprietary cloud infrastructure)— now offer multiple TB of RAM on a single node. - NVMe-backed instance families
(proprietary cloud infrastructure)— give single-node IO throughput that rivals small clusters from a few years ago. - Bare-metal / dedicated single-box providers (e.g. Hetzner dedicated servers)
(proprietary hosting provider)— often dramatically cheaper per core/GB than an equivalent managed cluster. If you're running your own physical hardware instead, that's the fully self-hosted/FOSS-adjacent end of this spectrum.
Being single-node-first doesn't mean single-node-always. Genuine reasons to go distributed:
- You've measured it, not guessed it. You've actually run the workload on the biggest reasonable single box and it doesn't fit or doesn't finish in your required window.
- High availability across hardware failure is a hard requirement, not a nice-to-have — a single node is a single point of failure.
- Sustained multi-tenant concurrency that saturates one machine's CPU/IO even after query optimization and indexing.
- Data genuinely exceeds what a single machine's disk (or a realistic cluster of local disks you'd attach) can hold, and is still growing.
- Organizational reality: you already operate a distributed platform (e.g. Kubernetes + object storage) and the incremental cost of using it is lower than standing up a separate single-node pattern.
- Adam Drake — "Command-line Tools can be 235x Faster than your Hadoop Cluster"
- Frank McSherry, Michael Isard, Derek G. Murray — "Scalability! But at what COST?", HotOS 2015
- Jordan Tigani — "Big Data is Dead" — on how much of what got called "big data" fits comfortably on one modern machine
See CONTRIBUTING.md. PRs and issues suggesting resources are welcome.
To the extent possible under law, the contributors to this list have waived all copyright and related or neighboring rights to this work. See LICENSE (CC0 1.0 Universal).