From 8f17e773bf1274e78a3b3e4edb1255f3f750a2f4 Mon Sep 17 00:00:00 2001 From: Louis Chu Date: Thu, 6 Aug 2026 00:33:09 +0800 Subject: [PATCH 1/3] Add DataFusion backend support column to PPL command reference Add a 'DataFusion Backend' column to the Commands table in docs/user/ppl/index.md, indicating whether each PPL command runs on the analytics engine (DataFusion) execution path used for parquet/composite indices (Yes / Partial / No / N/A), with a legend defining the values. Signed-off-by: Louis Chu --- docs/user/ppl/index.md | 113 +++++++++++++++++++++-------------------- 1 file changed, 57 insertions(+), 56 deletions(-) diff --git a/docs/user/ppl/index.md b/docs/user/ppl/index.md index 3e5a08d990d..2708ff46bef 100644 --- a/docs/user/ppl/index.md +++ b/docs/user/ppl/index.md @@ -34,63 +34,64 @@ source=accounts The following commands are available in PPL: **Note:** Experimental commands are ready for use, but specific parameters may change based on feedback. + **Note:** The **DataFusion Backend** column indicates support on the analytics engine (DataFusion) execution path used for `parquet`/composite-format indices: **Yes** = supported; **Partial** = supported with a known limitation on specific modes or variants (see the command page); **No** = operates on index data but is not supported on the DataFusion path (runs on the default engine, is rejected, or has no plan translation); **N/A** = does not touch the parquet/AE data path by design (planning, metadata, or management commands). -| Command Name | Version Introduced | Current Status | Command Description | -| --- | --- | --- | --- | -| [search command](cmd/search.md) | 1.0 | stable (since 1.0) | Retrieve documents from the index. | -| [where command](cmd/where.md) | 1.0 | stable (since 1.0) | Filter the search result using boolean expressions. | -| [subquery command](cmd/subquery.md) | 3.0 | experimental (since 3.0) | Embed one PPL query inside another for complex filtering and data retrieval operations. | -| [fields command](cmd/fields.md) | 1.0 | stable (since 1.0) | Keep or remove fields from the search result. | -| [rename command](cmd/rename.md) | 1.0 | stable (since 1.0) | Rename one or more fields in the search result. | -| [eval command](cmd/eval.md) | 1.0 | stable (since 1.0) | Evaluate an expression and append the result to the search result. | -| [foreach command](cmd/foreach.md) | 3.8 | experimental (since 3.8) | Run a templated evaluation for each selected field or collection element. | -| [convert command](cmd/convert.md) | 3.5 | experimental (since 3.5) | Transform field values to numeric values using specialized conversion functions. | -| [replace command](cmd/replace.md) | 3.4 | experimental (since 3.4) | Replace text in one or more fields in the search result | -| [fillnull command](cmd/fillnull.md) | 3.0 | experimental (since 3.0) | Fill null with provided value in one or more fields in the search result. | -| [expand command](cmd/expand.md) | 3.1 | experimental (since 3.1) | Transform a single document into multiple documents by expanding a nested array field. | -| [flatten command](cmd/flatten.md) | 3.1 | experimental (since 3.1) | Flatten a struct or an object field into separate fields in a document. | -| [table command](cmd/table.md) | 3.3 | experimental (since 3.3) | Keep or remove fields from the search result using enhanced syntax options. | -| [stats command](cmd/stats.md) | 1.0 | stable (since 1.0) | Calculate aggregation from search results. | -| [eventstats command](cmd/eventstats.md) | 3.1 | experimental (since 3.1) | Calculate aggregation statistics and add them as new fields to each event. | -| [streamstats command](cmd/streamstats.md) | 3.4 | experimental (since 3.4) | Calculate cumulative or rolling statistics as events are processed in order. | -| [bin command](cmd/bin.md) | 3.3 | experimental (since 3.3) | Group numeric values into buckets of equal intervals. | -| [timechart command](cmd/timechart.md) | 3.3 | experimental (since 3.3) | Create time-based charts and visualizations. | -| [chart command](cmd/chart.md) | 3.4 | experimental (since 3.4) | Apply statistical aggregations to search results and group the data for visualizations. | -| [trendline command](cmd/trendline.md) | 3.0 | experimental (since 3.0) | Calculate moving averages of fields. | -| [sort command](cmd/sort.md) | 1.0 | stable (since 1.0) | Sort all the search results by the specified fields. | -| [reverse command](cmd/reverse.md) | 3.2 | experimental (since 3.2) | Reverse the display order of search results. | -| [head command](cmd/head.md) | 1.0 | stable (since 1.0) | Return the first N number of specified results after an optional offset in search order. | -| [dedup command](cmd/dedup.md) | 1.0 | stable (since 1.0) | Remove identical documents defined by the field from the search result. | -| [top command](cmd/top.md) | 1.0 | stable (since 1.0) | Find the most common tuple of values of all fields in the field list. | -| [rare command](cmd/rare.md) | 1.0 | stable (since 1.0) | Find the least common tuple of values of all fields in the field list. | -| [parse command](cmd/parse.md) | 1.3 | stable (since 1.3) | Parse a text field with a regular expression and append the result to the search result. | -| [grok command](cmd/grok.md) | 2.4 | stable (since 2.4) | Parse a text field with a grok pattern and append the results to the search result. | -| [rex command](cmd/rex.md) | 3.3 | experimental (since 3.3) | Extract fields from a raw text field using regular expression named capture groups. | -| [regex command](cmd/regex.md) | 3.3 | experimental (since 3.3) | Filter search results by matching field values against a regular expression pattern. | -| [spath command](cmd/spath.md) | 3.3 | experimental (since 3.3) | Extract fields from structured text data. | -| [patterns command](cmd/patterns.md) | 2.4 | stable (since 2.4) | Extract log patterns from a text field and append the results to the search result. | -| [join command](cmd/join.md) | 3.0 | stable (since 3.0) | Combine two datasets together. | -| [append command](cmd/append.md) | 3.3 | experimental (since 3.3) | Append the result of a sub-search to the bottom of the input search results. | -| [appendcol command](cmd/appendcol.md) | 3.1 | experimental (since 3.1) | Append the result of a sub-search and attach it alongside the input search results. | -| [lookup command](cmd/lookup.md) | 3.0 | experimental (since 3.0) | Add or replace data from a lookup index. | -| [multisearch command](cmd/multisearch.md) | 3.4 | experimental (since 3.4) | Execute multiple search queries and combine their results. | -| [union command](cmd/union.md) | 3.7 | experimental (since 3.7) | Combine results from multiple datasets using UNION ALL semantics. | -| [rest command](cmd/rest.md) | 3.9 | experimental (since 3.9) | Read an allow-listed, read-only in-cluster management endpoint (cluster/cat/nodes) as rows. Calcite engine only. | -| [ml command](cmd/ml.md) | 2.5 | stable (since 2.5) | Apply machine learning algorithms to analyze data. | -| [kmeans command](cmd/kmeans.md) | 1.3 | stable (since 1.3) | Apply the kmeans algorithm on the search result returned by a PPL command. | -| [ad command](cmd/ad.md) | 1.3 | deprecated (since 2.5) | Apply Random Cut Forest algorithm on the search result returned by a PPL command. | -| [describe command](cmd/describe.md) | 2.1 | stable (since 2.1) | Query the metadata of an index. | -| [explain command](cmd/explain.md) | 3.1 | stable (since 3.1) | Explain the plan of query. | -| [show datasources command](cmd/showdatasources.md) | 2.4 | stable (since 2.4) | Query datasources configured in the PPL engine. | -| [makeresults command](cmd/makeresults.md) | 3.8 | experimental (since 3.8) | Generate in-memory rows for testing and seeding, optionally from inline CSV/JSON data. | -| [addtotals command](cmd/addtotals.md) | 3.5 | stable (since 3.5) | Adds row and column values and appends a totals column and row. | -| [addcoltotals command](cmd/addcoltotals.md) | 3.5 | stable (since 3.5) | Adds column values and appends a totals row. | -| [transpose command](cmd/transpose.md) | 3.5 | stable (since 3.5) | Transpose rows to columns. | -| [mvcombine command](cmd/mvcombine.md) | 3.5 | stable (since 3.4) | Combines values of a specified field across rows identical on all other fields. | -| [nomv command](cmd/nomv.md) | 3.6 | stable (since 3.6) | Converts a multivalue field to a single-value string by joining elements with newlines. | -| [mvexpand command](cmd/mvexpand.md) | 3.6 | stable (since 3.6) | Expand a multi-valued field into separate documents (one per value). | -| [graphlookup command](cmd/graphlookup.md) | 3.6 | experimental (since 3.6) | Performs recursive graph traversal on a collection using a BFS algorithm.| -| [xyseries command](cmd/xyseries.md) | 3.8 | experimental (since 3.8) | Converts row-oriented grouped results into a wide table format suitable for chart visualizations. One field serves as the X axis (row key), one field provides pivot values for generating output column names, and one or more data fields fill the pivoted cells. Only rows matching the explicitly provided pivot values in the `in` clause are included. | +| Command Name | Version Introduced | Current Status | DataFusion Backend | Command Description | +| --- | --- | --- | --- | --- | +| [search command](cmd/search.md) | 1.0 | stable (since 1.0) | Yes | Retrieve documents from the index. | +| [where command](cmd/where.md) | 1.0 | stable (since 1.0) | Yes | Filter the search result using boolean expressions. | +| [subquery command](cmd/subquery.md) | 3.0 | experimental (since 3.0) | Yes | Embed one PPL query inside another for complex filtering and data retrieval operations. | +| [fields command](cmd/fields.md) | 1.0 | stable (since 1.0) | Yes | Keep or remove fields from the search result. | +| [rename command](cmd/rename.md) | 1.0 | stable (since 1.0) | Yes | Rename one or more fields in the search result. | +| [eval command](cmd/eval.md) | 1.0 | stable (since 1.0) | Yes | Evaluate an expression and append the result to the search result. | +| [foreach command](cmd/foreach.md) | 3.8 | experimental (since 3.8) | No | Run a templated evaluation for each selected field or collection element. | +| [convert command](cmd/convert.md) | 3.5 | experimental (since 3.5) | Yes | Transform field values to numeric values using specialized conversion functions. | +| [replace command](cmd/replace.md) | 3.4 | experimental (since 3.4) | Yes | Replace text in one or more fields in the search result | +| [fillnull command](cmd/fillnull.md) | 3.0 | experimental (since 3.0) | Yes | Fill null with provided value in one or more fields in the search result. | +| [expand command](cmd/expand.md) | 3.1 | experimental (since 3.1) | No | Transform a single document into multiple documents by expanding a nested array field. | +| [flatten command](cmd/flatten.md) | 3.1 | experimental (since 3.1) | Yes | Flatten a struct or an object field into separate fields in a document. | +| [table command](cmd/table.md) | 3.3 | experimental (since 3.3) | Yes | Keep or remove fields from the search result using enhanced syntax options. | +| [stats command](cmd/stats.md) | 1.0 | stable (since 1.0) | Yes | Calculate aggregation from search results. | +| [eventstats command](cmd/eventstats.md) | 3.1 | experimental (since 3.1) | Yes | Calculate aggregation statistics and add them as new fields to each event. | +| [streamstats command](cmd/streamstats.md) | 3.4 | experimental (since 3.4) | Partial | Calculate cumulative or rolling statistics as events are processed in order. | +| [bin command](cmd/bin.md) | 3.3 | experimental (since 3.3) | Yes | Group numeric values into buckets of equal intervals. | +| [timechart command](cmd/timechart.md) | 3.3 | experimental (since 3.3) | Yes | Create time-based charts and visualizations. | +| [chart command](cmd/chart.md) | 3.4 | experimental (since 3.4) | Yes | Apply statistical aggregations to search results and group the data for visualizations. | +| [trendline command](cmd/trendline.md) | 3.0 | experimental (since 3.0) | Yes | Calculate moving averages of fields. | +| [sort command](cmd/sort.md) | 1.0 | stable (since 1.0) | Yes | Sort all the search results by the specified fields. | +| [reverse command](cmd/reverse.md) | 3.2 | experimental (since 3.2) | Yes | Reverse the display order of search results. | +| [head command](cmd/head.md) | 1.0 | stable (since 1.0) | Yes | Return the first N number of specified results after an optional offset in search order. | +| [dedup command](cmd/dedup.md) | 1.0 | stable (since 1.0) | Yes | Remove identical documents defined by the field from the search result. | +| [top command](cmd/top.md) | 1.0 | stable (since 1.0) | Yes | Find the most common tuple of values of all fields in the field list. | +| [rare command](cmd/rare.md) | 1.0 | stable (since 1.0) | Yes | Find the least common tuple of values of all fields in the field list. | +| [parse command](cmd/parse.md) | 1.3 | stable (since 1.3) | Partial | Parse a text field with a regular expression and append the result to the search result. | +| [grok command](cmd/grok.md) | 2.4 | stable (since 2.4) | Yes | Parse a text field with a grok pattern and append the results to the search result. | +| [rex command](cmd/rex.md) | 3.3 | experimental (since 3.3) | Yes | Extract fields from a raw text field using regular expression named capture groups. | +| [regex command](cmd/regex.md) | 3.3 | experimental (since 3.3) | Yes | Filter search results by matching field values against a regular expression pattern. | +| [spath command](cmd/spath.md) | 3.3 | experimental (since 3.3) | Yes | Extract fields from structured text data. | +| [patterns command](cmd/patterns.md) | 2.4 | stable (since 2.4) | Partial | Extract log patterns from a text field and append the results to the search result. | +| [join command](cmd/join.md) | 3.0 | stable (since 3.0) | Yes | Combine two datasets together. | +| [append command](cmd/append.md) | 3.3 | experimental (since 3.3) | Yes | Append the result of a sub-search to the bottom of the input search results. | +| [appendcol command](cmd/appendcol.md) | 3.1 | experimental (since 3.1) | Yes | Append the result of a sub-search and attach it alongside the input search results. | +| [lookup command](cmd/lookup.md) | 3.0 | experimental (since 3.0) | Partial | Add or replace data from a lookup index. | +| [multisearch command](cmd/multisearch.md) | 3.4 | experimental (since 3.4) | Partial | Execute multiple search queries and combine their results. | +| [union command](cmd/union.md) | 3.7 | experimental (since 3.7) | Yes | Combine results from multiple datasets using UNION ALL semantics. | +| [rest command](cmd/rest.md) | 3.9 | experimental (since 3.9) | N/A | Read an allow-listed, read-only in-cluster management endpoint (cluster/cat/nodes) as rows. Calcite engine only. | +| [ml command](cmd/ml.md) | 2.5 | stable (since 2.5) | No | Apply machine learning algorithms to analyze data. | +| [kmeans command](cmd/kmeans.md) | 1.3 | stable (since 1.3) | No | Apply the kmeans algorithm on the search result returned by a PPL command. | +| [ad command](cmd/ad.md) | 1.3 | deprecated (since 2.5) | No | Apply Random Cut Forest algorithm on the search result returned by a PPL command. | +| [describe command](cmd/describe.md) | 2.1 | stable (since 2.1) | N/A | Query the metadata of an index. | +| [explain command](cmd/explain.md) | 3.1 | stable (since 3.1) | N/A | Explain the plan of query. | +| [show datasources command](cmd/showdatasources.md) | 2.4 | stable (since 2.4) | N/A | Query datasources configured in the PPL engine. | +| [makeresults command](cmd/makeresults.md) | 3.8 | experimental (since 3.8) | Yes | Generate in-memory rows for testing and seeding, optionally from inline CSV/JSON data. | +| [addtotals command](cmd/addtotals.md) | 3.5 | stable (since 3.5) | Yes | Adds row and column values and appends a totals column and row. | +| [addcoltotals command](cmd/addcoltotals.md) | 3.5 | stable (since 3.5) | Yes | Adds column values and appends a totals row. | +| [transpose command](cmd/transpose.md) | 3.5 | stable (since 3.5) | Yes | Transpose rows to columns. | +| [mvcombine command](cmd/mvcombine.md) | 3.5 | stable (since 3.4) | No | Combines values of a specified field across rows identical on all other fields. | +| [nomv command](cmd/nomv.md) | 3.6 | stable (since 3.6) | No | Converts a multivalue field to a single-value string by joining elements with newlines. | +| [mvexpand command](cmd/mvexpand.md) | 3.6 | stable (since 3.6) | No | Expand a multi-valued field into separate documents (one per value). | +| [graphlookup command](cmd/graphlookup.md) | 3.6 | experimental (since 3.6) | No | Performs recursive graph traversal on a collection using a BFS algorithm.| +| [xyseries command](cmd/xyseries.md) | 3.8 | experimental (since 3.8) | Partial | Converts row-oriented grouped results into a wide table format suitable for chart visualizations. One field serves as the X axis (row key), one field provides pivot values for generating output column names, and one or more data fields fill the pivoted cells. Only rows matching the explicitly provided pivot values in the `in` clause are included. | - [Syntax](cmd/syntax.md) - PPL query structure and command syntax formatting * **Functions** From 6d2bf9c0f2606af2b96fbcaa0f1fd0ffd9ac4fcf Mon Sep 17 00:00:00 2001 From: Louis Chu Date: Thu, 6 Aug 2026 15:35:16 +0800 Subject: [PATCH 2/3] Mark makeresults DataFusion Backend as No makeresults lowers to a Values node whose DataFusion pushdown (VirtualTableScan, esp. CHAR/VARCHAR string literals) is not yet supported on main -- that fix is tracked in the (unmerged) core PR opensearch-project/OpenSearch#22554. Mark No until it lands. Signed-off-by: Louis Chu --- docs/user/ppl/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/user/ppl/index.md b/docs/user/ppl/index.md index 2708ff46bef..ffc58973ee6 100644 --- a/docs/user/ppl/index.md +++ b/docs/user/ppl/index.md @@ -83,7 +83,7 @@ source=accounts | [describe command](cmd/describe.md) | 2.1 | stable (since 2.1) | N/A | Query the metadata of an index. | | [explain command](cmd/explain.md) | 3.1 | stable (since 3.1) | N/A | Explain the plan of query. | | [show datasources command](cmd/showdatasources.md) | 2.4 | stable (since 2.4) | N/A | Query datasources configured in the PPL engine. | -| [makeresults command](cmd/makeresults.md) | 3.8 | experimental (since 3.8) | Yes | Generate in-memory rows for testing and seeding, optionally from inline CSV/JSON data. | +| [makeresults command](cmd/makeresults.md) | 3.8 | experimental (since 3.8) | No | Generate in-memory rows for testing and seeding, optionally from inline CSV/JSON data. | | [addtotals command](cmd/addtotals.md) | 3.5 | stable (since 3.5) | Yes | Adds row and column values and appends a totals column and row. | | [addcoltotals command](cmd/addcoltotals.md) | 3.5 | stable (since 3.5) | Yes | Adds column values and appends a totals row. | | [transpose command](cmd/transpose.md) | 3.5 | stable (since 3.5) | Yes | Transpose rows to columns. | From cbf53017c714f867d5937f6177fd99259a7e9b5c Mon Sep 17 00:00:00 2001 From: Louis Chu Date: Fri, 7 Aug 2026 02:04:49 +0800 Subject: [PATCH 3/3] Apply reviewer feedback: refine DataFusion Backend values Per review by @ahkcs on PR #5679: - search -> Partial (relevance/full-text predicates are Lucene-delegated; base scan runs on DataFusion) - convert -> Partial (some conversion functions are not wired on the DataFusion path) - flatten -> No (struct/object flattening is not supported on parquet/composite) - bin -> Partial (span bucketing runs; other bin variants are not supported) expand and mvexpand were already changed to No in a prior commit per the same review. Signed-off-by: Louis Chu --- docs/user/ppl/index.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/user/ppl/index.md b/docs/user/ppl/index.md index ffc58973ee6..ec632728388 100644 --- a/docs/user/ppl/index.md +++ b/docs/user/ppl/index.md @@ -38,23 +38,23 @@ source=accounts | Command Name | Version Introduced | Current Status | DataFusion Backend | Command Description | | --- | --- | --- | --- | --- | -| [search command](cmd/search.md) | 1.0 | stable (since 1.0) | Yes | Retrieve documents from the index. | +| [search command](cmd/search.md) | 1.0 | stable (since 1.0) | Partial | Retrieve documents from the index. | | [where command](cmd/where.md) | 1.0 | stable (since 1.0) | Yes | Filter the search result using boolean expressions. | | [subquery command](cmd/subquery.md) | 3.0 | experimental (since 3.0) | Yes | Embed one PPL query inside another for complex filtering and data retrieval operations. | | [fields command](cmd/fields.md) | 1.0 | stable (since 1.0) | Yes | Keep or remove fields from the search result. | | [rename command](cmd/rename.md) | 1.0 | stable (since 1.0) | Yes | Rename one or more fields in the search result. | | [eval command](cmd/eval.md) | 1.0 | stable (since 1.0) | Yes | Evaluate an expression and append the result to the search result. | | [foreach command](cmd/foreach.md) | 3.8 | experimental (since 3.8) | No | Run a templated evaluation for each selected field or collection element. | -| [convert command](cmd/convert.md) | 3.5 | experimental (since 3.5) | Yes | Transform field values to numeric values using specialized conversion functions. | +| [convert command](cmd/convert.md) | 3.5 | experimental (since 3.5) | Partial | Transform field values to numeric values using specialized conversion functions. | | [replace command](cmd/replace.md) | 3.4 | experimental (since 3.4) | Yes | Replace text in one or more fields in the search result | | [fillnull command](cmd/fillnull.md) | 3.0 | experimental (since 3.0) | Yes | Fill null with provided value in one or more fields in the search result. | | [expand command](cmd/expand.md) | 3.1 | experimental (since 3.1) | No | Transform a single document into multiple documents by expanding a nested array field. | -| [flatten command](cmd/flatten.md) | 3.1 | experimental (since 3.1) | Yes | Flatten a struct or an object field into separate fields in a document. | +| [flatten command](cmd/flatten.md) | 3.1 | experimental (since 3.1) | No | Flatten a struct or an object field into separate fields in a document. | | [table command](cmd/table.md) | 3.3 | experimental (since 3.3) | Yes | Keep or remove fields from the search result using enhanced syntax options. | | [stats command](cmd/stats.md) | 1.0 | stable (since 1.0) | Yes | Calculate aggregation from search results. | | [eventstats command](cmd/eventstats.md) | 3.1 | experimental (since 3.1) | Yes | Calculate aggregation statistics and add them as new fields to each event. | | [streamstats command](cmd/streamstats.md) | 3.4 | experimental (since 3.4) | Partial | Calculate cumulative or rolling statistics as events are processed in order. | -| [bin command](cmd/bin.md) | 3.3 | experimental (since 3.3) | Yes | Group numeric values into buckets of equal intervals. | +| [bin command](cmd/bin.md) | 3.3 | experimental (since 3.3) | Partial | Group numeric values into buckets of equal intervals. | | [timechart command](cmd/timechart.md) | 3.3 | experimental (since 3.3) | Yes | Create time-based charts and visualizations. | | [chart command](cmd/chart.md) | 3.4 | experimental (since 3.4) | Yes | Apply statistical aggregations to search results and group the data for visualizations. | | [trendline command](cmd/trendline.md) | 3.0 | experimental (since 3.0) | Yes | Calculate moving averages of fields. |