Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 41 additions & 4 deletions dev/update_config_docs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -172,12 +172,14 @@ cat <<'EOF' >> "$TARGET_FILE"

# Tuning Guide

## Short Queries
## Short Queries and Point Lookups

By default DataFusion will attempt to maximize parallelism and use all cores --
For example, if you have 32 cores, each plan will split the data into 32
partitions. However, if your data is small, the overhead of splitting the data
to enable parallelization can dominate the actual computation.
partitions. However, if your data is small, or the query only reads a few rows
(for example, a point lookup such as `SELECT * FROM t WHERE id = 42`), the
overhead of planning, opening files, and splitting the data to enable
parallelization can dominate the actual computation.

You can find out how many cores are being used via the [`EXPLAIN`] command and look
at the number of partitions in the plan.
Expand All @@ -190,14 +192,49 @@ does not apply to user defined data sources and only works when DataFusion has a

If you know your data is small, you can set the `datafusion.execution.target_partitions`
option to a smaller number to reduce the overhead of repartitioning. For very small datasets (e.g. less
than 1MB), we recommend setting `target_partitions` to 1 to avoid repartitioning altogether.
than 1MB) or point lookups, we recommend setting `target_partitions` to 1 to avoid repartitioning altogether.

```sql
SET datafusion.execution.target_partitions = '1';
```

[`ListingTable`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html

For point lookups, the following techniques also help:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think about highlighting the file configuration here as well (i.e., smaller pages and row groups)?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is a good idea -- I pushed some changes

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Specifically something like

  • Write Parquet files with smaller row groups and data pages than the default
    so that less data needs to be read and decoded for each lookup.

...

Here is an example of how to write Parquet files with smaller row groups and
data pages:

COPY (SELECT * FROM t) TO 't.parquet'
OPTIONS (
  MAX_ROW_GROUP_SIZE 131072,
  DATA_PAGE_ROW_COUNT_LIMIT 2048
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks.


- Sort the data by the lookup key so the data that is fetched is physically nearby (clustered).
This makes techniques such as Parquet min/max metadata pruning more effective.
- Write Parquet files with smaller row groups and data pages than the default
so that less data needs to be read and decoded for each lookup.
- Enable `datafusion.execution.parquet.pushdown_filters` so filters are
evaluated while decoding Parquet.
- Set the Parquet metadata cache size via
`datafusion.runtime.metadata_cache_limit` so it is large enough to hold all the metadata for the
files you query repeatedly.
- Reuse physical plans to avoid re-planning each query.

Here is an example of how to write Parquet files with smaller row groups and
data pages:

```sql
COPY (SELECT * FROM t) TO 't.parquet'
OPTIONS (
MAX_ROW_GROUP_SIZE 131072,
DATA_PAGE_ROW_COUNT_LIMIT 2048
);
```

Here is how to enable pushdown filters:

```sql
SET datafusion.execution.parquet.pushdown_filters = true;
```

See the [Best Practices for Single Row Lookups (Point Queries)] blog post for
more details.

[Best Practices for Single Row Lookups (Point Queries)]: https://datafusion.apache.org/blog/2026/10/24/point-queries/

## Memory-limited Queries

When executing a memory-consuming query under a tight memory limit, DataFusion
Expand Down
2 changes: 2 additions & 0 deletions docs/source/user-guide/concepts-readings-events.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,8 @@

This is a list of DataFusion related blog posts, articles, and other resources. Please open a PR to add any new resources you create or find

- **2026-10-24** [Blog: Best Practices for Single Row Lookups (Point Queries)](https://datafusion.apache.org/blog/2026/10/24/point-queries/)

- **2026-10-01** [Blog: How we extended Apache DataFusion to execute one query across many machines](https://www.datadoghq.com/blog/engineering/distributed-datafusion/)

- **2026-07-29** [Blog: Bringing DuckLake to DataFusion](https://ducklake.select/2026/07/29/bringing-ducklake-to-datafusion/)
Expand Down
45 changes: 41 additions & 4 deletions docs/source/user-guide/configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -259,12 +259,14 @@ The following runtime configuration settings are available:

# Tuning Guide

## Short Queries
## Short Queries and Point Lookups

By default DataFusion will attempt to maximize parallelism and use all cores --
For example, if you have 32 cores, each plan will split the data into 32
partitions. However, if your data is small, the overhead of splitting the data
to enable parallelization can dominate the actual computation.
partitions. However, if your data is small, or the query only reads a few rows
(for example, a point lookup such as `SELECT * FROM t WHERE id = 42`), the
overhead of planning, opening files, and splitting the data to enable
parallelization can dominate the actual computation.

You can find out how many cores are being used via the [`EXPLAIN`] command and look
at the number of partitions in the plan.
Expand All @@ -277,14 +279,49 @@ does not apply to user defined data sources and only works when DataFusion has a

If you know your data is small, you can set the `datafusion.execution.target_partitions`
option to a smaller number to reduce the overhead of repartitioning. For very small datasets (e.g. less
than 1MB), we recommend setting `target_partitions` to 1 to avoid repartitioning altogether.
than 1MB) or point lookups, we recommend setting `target_partitions` to 1 to avoid repartitioning altogether.

```sql
SET datafusion.execution.target_partitions = '1';
```

[`listingtable`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html

For point lookups, the following techniques also help:

- Sort the data by the lookup key so the data that is fetched is physically nearby (clustered).
This makes techniques such as Parquet min/max metadata pruning more effective.
- Write Parquet files with smaller row groups and data pages than the default
so that less data needs to be read and decoded for each lookup.
- Enable `datafusion.execution.parquet.pushdown_filters` so filters are
evaluated while decoding Parquet.
- Set the Parquet metadata cache size via
`datafusion.runtime.metadata_cache_limit` so it is large enough to hold all the metadata for the
files you query repeatedly.
- Reuse physical plans to avoid re-planning each query.

Here is an example of how to write Parquet files with smaller row groups and
data pages:

```sql
COPY (SELECT * FROM t) TO 't.parquet'
OPTIONS (
MAX_ROW_GROUP_SIZE 131072,
DATA_PAGE_ROW_COUNT_LIMIT 2048
);
```

Here is how to enable pushdown filters:

```sql
SET datafusion.execution.parquet.pushdown_filters = true;
```

See the [Best Practices for Single Row Lookups (Point Queries)] blog post for
more details.

[best practices for single row lookups (point queries)]: https://datafusion.apache.org/blog/2026/10/24/point-queries/

## Memory-limited Queries

When executing a memory-consuming query under a tight memory limit, DataFusion
Expand Down
Loading