From 06a44bb0abbb72f15664bcf5e4d1b2c511ed34ae Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Sun, 25 Oct 2026 16:01:18 -0400 Subject: [PATCH 1/7] docs: add point queries blog to good reads and tuning guide Co-Authored-By: Claude Fable 5.1 --- dev/update_config_docs.sh | 32 +++++++++++++++++++ .../user-guide/concepts-readings-events.md | 2 ++ docs/source/user-guide/configs.md | 32 +++++++++++++++++++ 3 files changed, 66 insertions(+) diff --git a/dev/update_config_docs.sh b/dev/update_config_docs.sh index 5bba39625f606..3b6291eb7cda6 100755 --- a/dev/update_config_docs.sh +++ b/dev/update_config_docs.sh @@ -198,6 +198,38 @@ SET datafusion.execution.target_partitions = '1'; [`ListingTable`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html +## Point Queries (Single Row Lookups) + +Point queries look up a small number of rows by key (for example, +`SELECT * FROM t WHERE id = 42`). For these queries the fixed costs of planning, +opening files, and coordinating partitions often dominate the actual work, so +the goal is to read as little data as possible and to avoid unnecessary +parallelism. + +For Parquet data, the most effective techniques are: + +- Sort the data by the lookup key so min/max statistics prune most row groups + and pages, and use relatively small row groups and pages (for example, ~128k + rows per row group and ~1k rows per page) so that pruning is fine grained. +- Set `datafusion.execution.target_partitions` to `1` to avoid the overhead of + parallelism, as described in [Short Queries](#short-queries). +- Enable `datafusion.execution.parquet.pushdown_filters` so filters are + evaluated while decoding Parquet rather than after the whole row group is read. +- Keep Parquet metadata caching enabled (the default) and size the cache with + `datafusion.runtime.metadata_cache_limit` so it holds the metadata for the + files you query repeatedly. +- Reuse prepared statements to avoid re-planning each query. + +```sql +SET datafusion.execution.target_partitions = '1'; +SET datafusion.execution.parquet.pushdown_filters = true; +``` + +See the [Best Practices for Single Row Lookups (Point Queries)] blog post for +benchmarks and a detailed walk through of each of these settings. + +[Best Practices for Single Row Lookups (Point Queries)]: https://datafusion.apache.org/blog/2026/10/24/point-queries/ + ## Memory-limited Queries When executing a memory-consuming query under a tight memory limit, DataFusion diff --git a/docs/source/user-guide/concepts-readings-events.md b/docs/source/user-guide/concepts-readings-events.md index 2024dc6474caf..5d7a066b3e658 100644 --- a/docs/source/user-guide/concepts-readings-events.md +++ b/docs/source/user-guide/concepts-readings-events.md @@ -37,6 +37,8 @@ This is a list of DataFusion related blog posts, articles, and other resources. Please open a PR to add any new resources you create or find +- **2026-10-24** [Blog: Best Practices for Single Row Lookups (Point Queries)](https://datafusion.apache.org/blog/2026/10/24/point-queries/) + - **2026-10-01** [Blog: How we extended Apache DataFusion to execute one query across many machines](https://www.datadoghq.com/blog/engineering/distributed-datafusion/) - **2026-07-29** [Blog: Bringing DuckLake to DataFusion](https://ducklake.select/2026/07/29/bringing-ducklake-to-datafusion/) diff --git a/docs/source/user-guide/configs.md b/docs/source/user-guide/configs.md index 27356ab8c1b77..3a161b1bb8899 100644 --- a/docs/source/user-guide/configs.md +++ b/docs/source/user-guide/configs.md @@ -284,6 +284,38 @@ SET datafusion.execution.target_partitions = '1'; [`listingtable`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html +## Point Queries (Single Row Lookups) + +Point queries look up a small number of rows by key (for example, +`SELECT * FROM t WHERE id = 42`). For these queries the fixed costs of planning, +opening files, and coordinating partitions often dominate the actual work, so +the goal is to read as little data as possible and to avoid unnecessary +parallelism. + +For Parquet data, the most effective techniques are: + +- Sort the data by the lookup key so min/max statistics prune most row groups + and pages, and use relatively small row groups and pages (for example, ~128k + rows per row group and ~1k rows per page) so that pruning is fine grained. +- Set `datafusion.execution.target_partitions` to `1` to avoid the overhead of + parallelism, as described in [Short Queries](#short-queries). +- Enable `datafusion.execution.parquet.pushdown_filters` so filters are + evaluated while decoding Parquet rather than after the whole row group is read. +- Keep Parquet metadata caching enabled (the default) and size the cache with + `datafusion.runtime.metadata_cache_limit` so it holds the metadata for the + files you query repeatedly. +- Reuse prepared statements to avoid re-planning each query. + +```sql +SET datafusion.execution.target_partitions = '1'; +SET datafusion.execution.parquet.pushdown_filters = true; +``` + +See the [Best Practices for Single Row Lookups (Point Queries)] blog post for +benchmarks and a detailed walk through of each of these settings. + +[best practices for single row lookups (point queries)]: https://datafusion.apache.org/blog/2026/10/24/point-queries/ + ## Memory-limited Queries When executing a memory-consuming query under a tight memory limit, DataFusion From 1f5c13c6b154e7dbf153b1ab549df3eee23d9414 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Mon, 26 Oct 2026 14:58:32 -0400 Subject: [PATCH 2/7] fixup --- dev/update_config_docs.sh | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/dev/update_config_docs.sh b/dev/update_config_docs.sh index 3b6291eb7cda6..be04dc79b76a0 100755 --- a/dev/update_config_docs.sh +++ b/dev/update_config_docs.sh @@ -202,31 +202,31 @@ SET datafusion.execution.target_partitions = '1'; Point queries look up a small number of rows by key (for example, `SELECT * FROM t WHERE id = 42`). For these queries the fixed costs of planning, -opening files, and coordinating partitions often dominate the actual work, so -the goal is to read as little data as possible and to avoid unnecessary +opening files, and coordinating partitions often dominate the actual work. You can +optimize such queries by clustering your data on the predicate, and avoiding +unnecessary parallelism. For Parquet data, the most effective techniques are: -- Sort the data by the lookup key so min/max statistics prune most row groups - and pages, and use relatively small row groups and pages (for example, ~128k - rows per row group and ~1k rows per page) so that pruning is fine grained. +- Sort the data by the lookup key so min/max statistics. - Set `datafusion.execution.target_partitions` to `1` to avoid the overhead of - parallelism, as described in [Short Queries](#short-queries). + parallelism. - Enable `datafusion.execution.parquet.pushdown_filters` so filters are - evaluated while decoding Parquet rather than after the whole row group is read. -- Keep Parquet metadata caching enabled (the default) and size the cache with + evaluated while decoding Parquet. +- Set the Parquet metadata cache size via `datafusion.runtime.metadata_cache_limit` so it holds the metadata for the files you query repeatedly. - Reuse prepared statements to avoid re-planning each query. +Example configuration: ```sql SET datafusion.execution.target_partitions = '1'; SET datafusion.execution.parquet.pushdown_filters = true; ``` See the [Best Practices for Single Row Lookups (Point Queries)] blog post for -benchmarks and a detailed walk through of each of these settings. +more details. [Best Practices for Single Row Lookups (Point Queries)]: https://datafusion.apache.org/blog/2026/10/24/point-queries/ From 4286bf7a63b4f5999c2c9139bce1ba7a9d5db5e5 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Mon, 26 Oct 2026 15:00:43 -0400 Subject: [PATCH 3/7] docs: merge point queries guidance into Short Queries tuning section Co-Authored-By: Claude Fable 5.1 --- dev/update_config_docs.sh | 28 ++++++++----------------- docs/source/user-guide/configs.md | 34 +++++++++++-------------------- 2 files changed, 21 insertions(+), 41 deletions(-) diff --git a/dev/update_config_docs.sh b/dev/update_config_docs.sh index be04dc79b76a0..3bd4f510ac40f 100755 --- a/dev/update_config_docs.sh +++ b/dev/update_config_docs.sh @@ -172,12 +172,14 @@ cat <<'EOF' >> "$TARGET_FILE" # Tuning Guide -## Short Queries +## Short Queries and Point Lookups By default DataFusion will attempt to maximize parallelism and use all cores -- For example, if you have 32 cores, each plan will split the data into 32 -partitions. However, if your data is small, the overhead of splitting the data -to enable parallelization can dominate the actual computation. +partitions. However, if your data is small, or the query only reads a few rows +(for example, a point lookup such as `SELECT * FROM t WHERE id = 42`), the +overhead of planning, opening files, and splitting the data to enable +parallelization can dominate the actual computation. You can find out how many cores are being used via the [`EXPLAIN`] command and look at the number of partitions in the plan. @@ -190,7 +192,7 @@ does not apply to user defined data sources and only works when DataFusion has a If you know your data is small, you can set the `datafusion.execution.target_partitions` option to a smaller number to reduce the overhead of repartitioning. For very small datasets (e.g. less -than 1MB), we recommend setting `target_partitions` to 1 to avoid repartitioning altogether. +than 1MB) or point lookups, we recommend setting `target_partitions` to 1 to avoid repartitioning altogether. ```sql SET datafusion.execution.target_partitions = '1'; @@ -198,20 +200,10 @@ SET datafusion.execution.target_partitions = '1'; [`ListingTable`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html -## Point Queries (Single Row Lookups) +For point lookups on Parquet data, the following techniques also help: -Point queries look up a small number of rows by key (for example, -`SELECT * FROM t WHERE id = 42`). For these queries the fixed costs of planning, -opening files, and coordinating partitions often dominate the actual work. You can -optimize such queries by clustering your data on the predicate, and avoiding -unnecessary -parallelism. - -For Parquet data, the most effective techniques are: - -- Sort the data by the lookup key so min/max statistics. -- Set `datafusion.execution.target_partitions` to `1` to avoid the overhead of - parallelism. +- Sort the data by the lookup key so min/max statistics can prune row groups + and pages. - Enable `datafusion.execution.parquet.pushdown_filters` so filters are evaluated while decoding Parquet. - Set the Parquet metadata cache size via @@ -219,9 +211,7 @@ For Parquet data, the most effective techniques are: files you query repeatedly. - Reuse prepared statements to avoid re-planning each query. -Example configuration: ```sql -SET datafusion.execution.target_partitions = '1'; SET datafusion.execution.parquet.pushdown_filters = true; ``` diff --git a/docs/source/user-guide/configs.md b/docs/source/user-guide/configs.md index 3a161b1bb8899..ad21a9369f409 100644 --- a/docs/source/user-guide/configs.md +++ b/docs/source/user-guide/configs.md @@ -258,12 +258,14 @@ The following runtime configuration settings are available: # Tuning Guide -## Short Queries +## Short Queries and Point Lookups By default DataFusion will attempt to maximize parallelism and use all cores -- For example, if you have 32 cores, each plan will split the data into 32 -partitions. However, if your data is small, the overhead of splitting the data -to enable parallelization can dominate the actual computation. +partitions. However, if your data is small, or the query only reads a few rows +(for example, a point lookup such as `SELECT * FROM t WHERE id = 42`), the +overhead of planning, opening files, and splitting the data to enable +parallelization can dominate the actual computation. You can find out how many cores are being used via the [`EXPLAIN`] command and look at the number of partitions in the plan. @@ -276,7 +278,7 @@ does not apply to user defined data sources and only works when DataFusion has a If you know your data is small, you can set the `datafusion.execution.target_partitions` option to a smaller number to reduce the overhead of repartitioning. For very small datasets (e.g. less -than 1MB), we recommend setting `target_partitions` to 1 to avoid repartitioning altogether. +than 1MB) or point lookups, we recommend setting `target_partitions` to 1 to avoid repartitioning altogether. ```sql SET datafusion.execution.target_partitions = '1'; @@ -284,35 +286,23 @@ SET datafusion.execution.target_partitions = '1'; [`listingtable`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html -## Point Queries (Single Row Lookups) +For point lookups on Parquet data, the following techniques also help: -Point queries look up a small number of rows by key (for example, -`SELECT * FROM t WHERE id = 42`). For these queries the fixed costs of planning, -opening files, and coordinating partitions often dominate the actual work, so -the goal is to read as little data as possible and to avoid unnecessary -parallelism. - -For Parquet data, the most effective techniques are: - -- Sort the data by the lookup key so min/max statistics prune most row groups - and pages, and use relatively small row groups and pages (for example, ~128k - rows per row group and ~1k rows per page) so that pruning is fine grained. -- Set `datafusion.execution.target_partitions` to `1` to avoid the overhead of - parallelism, as described in [Short Queries](#short-queries). +- Sort the data by the lookup key so min/max statistics can prune row groups + and pages. - Enable `datafusion.execution.parquet.pushdown_filters` so filters are - evaluated while decoding Parquet rather than after the whole row group is read. -- Keep Parquet metadata caching enabled (the default) and size the cache with + evaluated while decoding Parquet. +- Set the Parquet metadata cache size via `datafusion.runtime.metadata_cache_limit` so it holds the metadata for the files you query repeatedly. - Reuse prepared statements to avoid re-planning each query. ```sql -SET datafusion.execution.target_partitions = '1'; SET datafusion.execution.parquet.pushdown_filters = true; ``` See the [Best Practices for Single Row Lookups (Point Queries)] blog post for -benchmarks and a detailed walk through of each of these settings. +more details. [best practices for single row lookups (point queries)]: https://datafusion.apache.org/blog/2026/10/24/point-queries/ From f3abd117476ba0145e1263f09b6e31654c4bb93b Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Mon, 26 Oct 2026 15:03:43 -0400 Subject: [PATCH 4/7] fixup --- dev/update_config_docs.sh | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/dev/update_config_docs.sh b/dev/update_config_docs.sh index 3bd4f510ac40f..f2aac8f9afbf1 100755 --- a/dev/update_config_docs.sh +++ b/dev/update_config_docs.sh @@ -200,17 +200,18 @@ SET datafusion.execution.target_partitions = '1'; [`ListingTable`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html -For point lookups on Parquet data, the following techniques also help: +For point lookups, the following techniques also help: -- Sort the data by the lookup key so min/max statistics can prune row groups - and pages. +- Sort the data by the lookup key so the data that is fetched is physically nearby (clustered). + This makes techniques such as Parquet min/max metadata pruning more effective. - Enable `datafusion.execution.parquet.pushdown_filters` so filters are evaluated while decoding Parquet. - Set the Parquet metadata cache size via - `datafusion.runtime.metadata_cache_limit` so it holds the metadata for the + `datafusion.runtime.metadata_cache_limit` so it is large enough to hold all the metadata for the files you query repeatedly. - Reuse prepared statements to avoid re-planning each query. +Here is how to enable pushdown filters: ```sql SET datafusion.execution.parquet.pushdown_filters = true; ``` From b69c55438e631afafe5057ae98708bdec6ae63c7 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Mon, 26 Oct 2026 15:04:35 -0400 Subject: [PATCH 5/7] docs: regenerate configs.md for point lookup wording Co-Authored-By: Claude Fable 5.1 --- dev/update_config_docs.sh | 1 + docs/source/user-guide/configs.md | 10 ++++++---- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/dev/update_config_docs.sh b/dev/update_config_docs.sh index f2aac8f9afbf1..8c82de8e3a45f 100755 --- a/dev/update_config_docs.sh +++ b/dev/update_config_docs.sh @@ -212,6 +212,7 @@ For point lookups, the following techniques also help: - Reuse prepared statements to avoid re-planning each query. Here is how to enable pushdown filters: + ```sql SET datafusion.execution.parquet.pushdown_filters = true; ``` diff --git a/docs/source/user-guide/configs.md b/docs/source/user-guide/configs.md index ad21a9369f409..6c173f7cf7500 100644 --- a/docs/source/user-guide/configs.md +++ b/docs/source/user-guide/configs.md @@ -286,17 +286,19 @@ SET datafusion.execution.target_partitions = '1'; [`listingtable`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html -For point lookups on Parquet data, the following techniques also help: +For point lookups, the following techniques also help: -- Sort the data by the lookup key so min/max statistics can prune row groups - and pages. +- Sort the data by the lookup key so the data that is fetched is physically nearby (clustered). + This makes techniques such as Parquet min/max metadata pruning more effective. - Enable `datafusion.execution.parquet.pushdown_filters` so filters are evaluated while decoding Parquet. - Set the Parquet metadata cache size via - `datafusion.runtime.metadata_cache_limit` so it holds the metadata for the + `datafusion.runtime.metadata_cache_limit` so it is large enough to hold all the metadata for the files you query repeatedly. - Reuse prepared statements to avoid re-planning each query. +Here is how to enable pushdown filters: + ```sql SET datafusion.execution.parquet.pushdown_filters = true; ``` From c4850a3130b197cc7f0cad50f2e42114d1e28c9c Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Tue, 27 Oct 2026 12:28:36 -0400 Subject: [PATCH 6/7] feedback --- dev/update_config_docs.sh | 15 ++++++++++++++- docs/source/user-guide/configs.md | 17 ++++++++++++++++- 2 files changed, 30 insertions(+), 2 deletions(-) diff --git a/dev/update_config_docs.sh b/dev/update_config_docs.sh index 8c82de8e3a45f..f08ad673ee5de 100755 --- a/dev/update_config_docs.sh +++ b/dev/update_config_docs.sh @@ -204,12 +204,25 @@ For point lookups, the following techniques also help: - Sort the data by the lookup key so the data that is fetched is physically nearby (clustered). This makes techniques such as Parquet min/max metadata pruning more effective. +- Write Parquet files with smaller row groups and data pages than the default + so that less data needs to be read and decoded for each lookup. - Enable `datafusion.execution.parquet.pushdown_filters` so filters are evaluated while decoding Parquet. - Set the Parquet metadata cache size via `datafusion.runtime.metadata_cache_limit` so it is large enough to hold all the metadata for the files you query repeatedly. -- Reuse prepared statements to avoid re-planning each query. +- Reuse physical plans to avoid re-planning each query. + +Here is an example of how to write Parquet files with smaller row groups and +data pages: + +```sql +COPY (SELECT * FROM t) TO 't.parquet' +OPTIONS ( + MAX_ROW_GROUP_SIZE 131072, + DATA_PAGE_ROW_COUNT_LIMIT 2048 +); +``` Here is how to enable pushdown filters: diff --git a/docs/source/user-guide/configs.md b/docs/source/user-guide/configs.md index 4b9fa14c6759a..a8fb17fdcf7ea 100644 --- a/docs/source/user-guide/configs.md +++ b/docs/source/user-guide/configs.md @@ -291,12 +291,27 @@ For point lookups, the following techniques also help: - Sort the data by the lookup key so the data that is fetched is physically nearby (clustered). This makes techniques such as Parquet min/max metadata pruning more effective. +- Write Parquet files with smaller row groups and data pages than the defaults + (1M rows per row group and 20k rows per page) so that less data needs to be + read and decoded for each lookup. For example, use `datafusion.execution.parquet.max_row_group_size` + and `datafusion.execution.parquet.data_page_row_count_limit` to target around + 128k rows per row group and 2k-4k rows per page. - Enable `datafusion.execution.parquet.pushdown_filters` so filters are evaluated while decoding Parquet. - Set the Parquet metadata cache size via `datafusion.runtime.metadata_cache_limit` so it is large enough to hold all the metadata for the files you query repeatedly. -- Reuse prepared statements to avoid re-planning each query. +- Reuse physical plans to avoid re-planning each query. + +Here is how to write Parquet files with smaller row groups and data pages: + +```sql +COPY (SELECT * FROM t) TO 't.parquet' +OPTIONS ( + MAX_ROW_GROUP_SIZE 131072, + DATA_PAGE_ROW_COUNT_LIMIT 2048 +); +``` Here is how to enable pushdown filters: From 3272ce2afdd3a7a4287464e54f7902bed5919294 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Tue, 27 Oct 2026 12:29:20 -0400 Subject: [PATCH 7/7] docs: regenerate configs.md Co-Authored-By: Claude Fable 5.1 --- docs/source/user-guide/configs.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/source/user-guide/configs.md b/docs/source/user-guide/configs.md index a8fb17fdcf7ea..b5f1cd5870222 100644 --- a/docs/source/user-guide/configs.md +++ b/docs/source/user-guide/configs.md @@ -291,11 +291,8 @@ For point lookups, the following techniques also help: - Sort the data by the lookup key so the data that is fetched is physically nearby (clustered). This makes techniques such as Parquet min/max metadata pruning more effective. -- Write Parquet files with smaller row groups and data pages than the defaults - (1M rows per row group and 20k rows per page) so that less data needs to be - read and decoded for each lookup. For example, use `datafusion.execution.parquet.max_row_group_size` - and `datafusion.execution.parquet.data_page_row_count_limit` to target around - 128k rows per row group and 2k-4k rows per page. +- Write Parquet files with smaller row groups and data pages than the default + so that less data needs to be read and decoded for each lookup. - Enable `datafusion.execution.parquet.pushdown_filters` so filters are evaluated while decoding Parquet. - Set the Parquet metadata cache size via @@ -303,7 +300,8 @@ For point lookups, the following techniques also help: files you query repeatedly. - Reuse physical plans to avoid re-planning each query. -Here is how to write Parquet files with smaller row groups and data pages: +Here is an example of how to write Parquet files with smaller row groups and +data pages: ```sql COPY (SELECT * FROM t) TO 't.parquet'