Support PPL format command

[songkant-aws]

Description

Adds the PPL format command, which collapses tabular results into a single search-expression string. It supports configurable row, column, and multivalue delimiters; maxresults; emptystr; null handling; and quote/backslash escaping.

source=web_logs | fields status, host | format

The command also supports runtime search predicates produced by subsearches:

search source=web_logs status>=500 [ search source=rules | fields host ]

The subsearch result is formatted into one scalar search string and combined with the static parent predicate before the OpenSearch request is executed. Explicit format output remains a normal one-row result and can continue through later pipeline commands.

Design

• The grammar and AST represent format options explicitly, including the all-or-none positional delimiter group.
• FormatPlanner lowers each input row to a formatted expression, aggregates rows globally, applies the row wrapper and fallback, and projects the single search field.
• Search subqueries remain structured through parsing. When parent search requires an implicit formatted predicate, the planner builds an implicit Format scalar subquery.
• RuntimeSearchCorrelator rewrites only scalar subqueries registered as implicit Format inputs into the left side of a LogicalCorrelate. Other scalar, IN, and EXISTS subqueries remain untouched.
• The correlated right scan references the formatted string through its correlation variable. DynamicQueryStringSpec defers query-string compilation until the left input has produced its value.
• The runtime query_string filter is appended through the Calcite pushdown path, so it is combined with an existing pushed filter using AND semantics rather than replacing it.
• The legacy execution engine reports the command as Calcite-only.

Testing

• Lexer, parser, AST, anonymizer, and search predicate compiler unit tests.
• Full Calcite logical-plan tests for default options, custom delimiters, multivalue fields, escaping, empty input, explicit format, implicit format, multiple subsearches, and nested subsearches.
• Integration tests for explicit format output, continued pipeline processing, static and dynamic predicate composition, NOT subsearches, multiple subsearches, nested subsearches, explain output, and analytics-engine execution.
• Complete logical and physical explain-plan YAML golden files.
• Scan-level coverage proving that an existing pushed filter and a late-bound dynamic query_string filter are conjoined.

Related Issues

Related to #5233

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request is not applicable because this change does not alter the REST API shape.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 6c89411)

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ Recommended focus areas for review Possible Issue The planImplicitSearchField method limits the input to 1 row with builder.limit(0, 1) at line 102, but then aggregates with COUNT, MAX, and ARRAY_AGG at lines 119-121. If the input has zero rows, COUNT returns 0, MAX returns null, and ARRAY_AGG returns an empty array. The subsequent CASE at lines 123-133 checks if RAW_SEARCH_COUNT_FIELD > 0 to decide whether to use RAW_SEARCH_VALUE_FIELD or formatAggregatedRows. However, when the input is empty, formatAggregatedRows is called with an empty FORMAT_ROWS_FIELD array. The formatAggregatedRows method at line 201 calls ARRAY_COMPACT on this empty array, then ARRAY_JOIN, which produces an empty string. The isNotEmpty check at line 215 will fail, so the result becomes emptyString. This is correct behavior for an empty subsearch, but the logic path is not immediately obvious and could be fragile if formatAggregatedRows changes. Consider adding a comment explaining the empty-input case or explicitly handling it before aggregation. private RelNode planImplicitSearchField( Format node, CalcitePlanContext context, List<RelDataTypeField> fields, RelDataTypeField searchField) { RelBuilder builder = context.relBuilder; builder.limit(0, 1); RelDataType varchar = builder.getTypeFactory().createSqlType(SqlTypeName.VARCHAR); RexNode rawSearch = castFieldValue(searchField, builder.field(searchField.getIndex()), varchar, node, context); List<RexNode> fallbackFields = fields.stream() .filter(field -> field != searchField) .map(field -> formatField(field, node, context)) .toList(); RexNode fallbackRow = formatRow(fallbackFields, node, context); builder.project(List.of(rawSearch, fallbackRow), List.of(RAW_SEARCH_FIELD, FORMAT_ROW_FIELD)); RexNode rawSearchRef = builder.field(RAW_SEARCH_FIELD); RexNode rowRef = builder.field(FORMAT_ROW_FIELD); builder.aggregate( builder.groupKey(), builder.aggregateCall(SqlStdOperatorTable.COUNT, rawSearchRef).as(RAW_SEARCH_COUNT_FIELD), builder.aggregateCall(SqlStdOperatorTable.MAX, rawSearchRef).as(RAW_SEARCH_VALUE_FIELD), builder.aggregateCall(SqlLibraryOperators.ARRAY_AGG, rowRef).as(FORMAT_ROWS_FIELD)); RexNode hasRawSearch = builder.call( SqlStdOperatorTable.GREATER_THAN, builder.field(RAW_SEARCH_COUNT_FIELD), builder.literal(0)); RexNode result = builder.call( SqlStdOperatorTable.CASE, hasRawSearch, builder.field(RAW_SEARCH_VALUE_FIELD), formatAggregatedRows(node, context)); builder.project(List.of(result), List.of(SEARCH_FIELD), true); return builder.peek(); } Possible Issue The correlate method at line 42 throws IllegalStateException with message SUBSEARCH_APPLICATION_ERROR if the input is not a Filter. However, the method is called from CalciteRelNodeVisitor.visitSearch at line 360 only when node.hasImplicitSubquery() is true. If the planner produces a non-Filter node (e.g., a Project or Aggregate) before the filter, this will fail with a generic error message. The error message "The search command could not apply the subsearch result." does not indicate that the issue is an unexpected node type. Consider checking the node type earlier or providing a more specific error message that mentions the expected Filter node. /** * Moves the implicit-format scalar subqueries out of a {@code query_string} filter and makes them * the left input of a correlate. The right scan consumes their single-row output through a * correlation variable, so its OpenSearch request is not built until every runtime search * subquery result is available. Other Calcite subquery kinds remain in their original expression. */ public static RelNode correlate( RelNode filterNode, SearchPredicateCompiler searchPredicateCompiler, Predicate<RexSubQuery> isImplicitFormatSubquery) { if (!(filterNode instanceof Filter filter)) { throw new IllegalStateException(SUBSEARCH_APPLICATION_ERROR); } Possible Issue The buildRuntimeQuery method at line 96 checks if values.length != queryParts.size() and throws IllegalStateException with ASSEMBLY_ERROR. However, if a runtime predicate appears multiple times in the query (as tested in DynamicQueryStringSpecTest.marksEveryOccurrenceOfTheSameRuntimePredicate), the runtimePredicateParts set will contain multiple indices pointing to the same RexNode. The values array is expected to have one entry per queryParts element, not one per unique runtime predicate. If the same predicate is used twice, the code at lines 104-110 will call compiler.compile(value) twice on the same input value, which is correct. However, if the correlation variable produces different values for different occurrences (which should not happen in the current design), this could lead to incorrect results. The current implementation assumes that all occurrences of the same RexNode will have the same runtime value, which is enforced by the correlation mechanism. This is correct, but the assumption is not documented. Consider adding a comment explaining that the same RexNode in multiple positions will always have the same runtime value. public String buildRuntimeQuery(String[] values) { Objects.requireNonNull(values, ASSEMBLY_ERROR); if (values.length != queryParts.size()) { throw new IllegalStateException(ASSEMBLY_ERROR); } StringBuilder query = new StringBuilder(); for (int i = 0; i < values.length; i++) { String value = values[i]; if (runtimePredicateParts.contains(i)) { value = compiler.compile(value); } else if (value == null) { throw new IllegalStateException(ASSEMBLY_ERROR); } query.append(value); } return query.toString(); }

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to 6c89411

Explore these optional code suggestions:

Category	Suggestion	Impact
General	Validate compiled predicate is non-null The method doesn't validate that runtime predicate parts produce non-null values after compilation. If compiler.compile(value) returns null, it will be appended to the query, potentially causing issues. Add a null check after compilation to ensure robustness. opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/context/DynamicQueryStringSpec.java [96-113] public String buildRuntimeQuery(String[] values) { Objects.requireNonNull(values, ASSEMBLY_ERROR); if (values.length != queryParts.size()) { throw new IllegalStateException(ASSEMBLY_ERROR); } StringBuilder query = new StringBuilder(); for (int i = 0; i < values.length; i++) { String value = values[i]; if (runtimePredicateParts.contains(i)) { value = compiler.compile(value); + if (value == null) { + throw new IllegalStateException(ASSEMBLY_ERROR); + } } else if (value == null) { throw new IllegalStateException(ASSEMBLY_ERROR); } query.append(value); } return query.toString(); } Suggestion importance[1-10]: 7 __ Why: This is a valid concern about ensuring compiler.compile(value) doesn't return null. Adding this check would prevent potential issues where a null value is appended to the query, improving robustness.	Medium
	Add node type to error message The error message SUBSEARCH_APPLICATION_ERROR is generic and doesn't provide context about what type of node was expected versus what was received. Include the actual node type in the error message to aid debugging. core/src/main/java/org/opensearch/sql/calcite/RuntimeSearchCorrelator.java [53-55] if (!(filterNode instanceof Filter filter)) { - throw new IllegalStateException(SUBSEARCH_APPLICATION_ERROR); + throw new IllegalStateException(SUBSEARCH_APPLICATION_ERROR + " Expected Filter but got: " + filterNode.getClass().getSimpleName()); } Suggestion importance[1-10]: 6 __ Why: Adding the actual node type to the error message would improve debugging by providing more context. This is a valid improvement for error handling, though it's a minor enhancement.	Low
	Return empty string instead of null When formattedFields is empty, returning a null literal may cause issues in aggregation or concatenation operations. Consider returning an empty string literal instead to ensure consistent behavior in downstream operations and avoid potential null pointer exceptions. core/src/main/java/org/opensearch/sql/calcite/FormatPlanner.java [140-143] if (formattedFields.isEmpty()) { - RelDataType varchar = context.rexBuilder.getTypeFactory().createSqlType(SqlTypeName.VARCHAR); - return context.rexBuilder.makeNullLiteral(varchar); + return stringLiteral("", context); } Suggestion importance[1-10]: 5 __ Why: The suggestion to return an empty string instead of null is reasonable for consistency, but the current implementation may be intentional. The makeNullLiteral is used in a context where null handling is expected in ifNotEmpty, so this change has moderate impact.	Low
	Clear builder state before empty result After filtering metadata fields, if all remaining fields are empty, the code creates a values node with emptyString. However, this bypasses the maxResults limit that was applied earlier. Consider checking if fields are empty before applying the limit to avoid unnecessary processing. core/src/main/java/org/opensearch/sql/calcite/FormatPlanner.java [54-63] List<RelDataTypeField> fields = builder.peek().getRowType().getFieldList().stream() .filter(field -> !METADATAFIELD_TYPE_MAP.containsKey(field.getName())) .sorted(Comparator.comparing(RelDataTypeField::getName)) .toList(); if (fields.isEmpty()) { + builder.build(); builder.values(new String[] {SEARCH_FIELD}, node.getEmptyString()); return builder.peek(); } Suggestion importance[1-10]: 4 __ Why: The suggestion to call builder.build() before creating the empty values node is questionable. The current code applies maxResults limit before checking if fields are empty, which seems intentional. Adding builder.build() might disrupt the builder state unnecessarily.	Low

---

Previous suggestions

Suggestions up to commit 669bee1

Category	Suggestion	Impact
General	Validate before state modification The validation that checks getFieldCount() != 1 occurs after the subquery is already identified as an implicit format subquery. If this validation fails, it throws an exception but the seen set has already been modified. Consider performing the field count validation before adding to seen to maintain consistency. core/src/main/java/org/opensearch/sql/calcite/RuntimeSearchCorrelator.java [112-130] static List<RexSubQuery> findImplicitFormatSubqueries( RexNode condition, Predicate<RexSubQuery> isImplicitFormatSubquery) { List<RexSubQuery> subqueries = new ArrayList<>(); Set<RexSubQuery> seen = Collections.newSetFromMap(new IdentityHashMap<>()); condition.accept( new RexVisitorImpl<Void>(true) { @Override public Void visitSubQuery(RexSubQuery subquery) { - if (isImplicitFormatSubquery.test(subquery) && seen.add(subquery)) { + if (isImplicitFormatSubquery.test(subquery)) { if (subquery.rel.getRowType().getFieldCount() != 1) { throw new SemanticCheckException( "Implicit format subsearch must return exactly one column"); } - subqueries.add(subquery); + if (seen.add(subquery)) { + subqueries.add(subquery); + } } return null; } }); return subqueries; } Suggestion importance[1-10]: 7 __ Why: Performing validation before modifying the seen set ensures consistency in error handling. If validation fails, the state remains unchanged, which is a better practice for maintaining data integrity.	Medium
	Cache compiled regex pattern The regex pattern should be compiled once and reused to avoid repeated compilation overhead. Consider declaring a static Pattern field and using pattern.matcher(fieldName).matches() for better performance when this method is called frequently. core/src/main/java/org/opensearch/sql/calcite/FormatPlanner.java [270-275] +private static final Pattern FIELD_NAME_PATTERN = + Pattern.compile("[A-Za-z_@][A-Za-z0-9_@-]*(\\.[A-Za-z_@][A-Za-z0-9_@-]*)*"); + private String formatFieldName(String fieldName) { - if (fieldName.matches("[A-Za-z_@][A-Za-z0-9_@-]*(\\.[A-Za-z_@][A-Za-z0-9_@-]*)*")) { + if (FIELD_NAME_PATTERN.matcher(fieldName).matches()) { return fieldName; } return "`" + fieldName.replace("`", "``") + "`"; } Suggestion importance[1-10]: 6 __ Why: Compiling the regex pattern once as a static field improves performance when formatFieldName is called frequently. This is a good optimization for a method that processes field names repeatedly.	Low
	Extract repeated method calls The contains(i) check on a Set is performed inside the loop for every iteration. Consider extracting runtimePredicateParts() and compiler() to local variables before the loop to avoid repeated method calls and improve readability. opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteEnumerableIndexScan.java [163-173] private String buildRuntimeQuery(String[] queryParts) { StringBuilder query = new StringBuilder(); + Set<Integer> runtimeParts = pushDownContext.getDynamicQueryString().runtimePredicateParts(); + SearchPredicateCompiler compiler = pushDownContext.getDynamicQueryString().compiler(); for (int i = 0; i < queryParts.length; i++) { String part = queryParts[i]; - if (pushDownContext.getDynamicQueryString().runtimePredicateParts().contains(i)) { - part = pushDownContext.getDynamicQueryString().compiler().compile(part); + if (runtimeParts.contains(i)) { + part = compiler.compile(part); } query.append(part); } return query.toString(); } Suggestion importance[1-10]: 5 __ Why: Extracting runtimePredicateParts() and compiler() to local variables before the loop reduces repeated method calls and improves code readability, though the performance gain is modest.	Low
	Use fully qualified class name The error message should include the actual class name for better debugging. Consider using filterNode.getClass().getName() instead of getSimpleName() to provide the fully qualified class name, which is more helpful when diagnosing issues across different packages. core/src/main/java/org/opensearch/sql/calcite/RuntimeSearchCorrelator.java [50-54] if (!(filterNode instanceof Filter filter)) { throw new IllegalStateException( "Runtime search query must produce a filter, but got " - + filterNode.getClass().getSimpleName()); + + filterNode.getClass().getName()); } Suggestion importance[1-10]: 4 __ Why: Using getName() instead of getSimpleName() provides more context in error messages, but this is a minor improvement that doesn't significantly impact functionality or debugging in most cases.	Low

Suggestions up to commit da0ec85

Category	Suggestion	Impact
Possible issue	Prevent potential NoSuchElementException The method uses getFirst() which throws NoSuchElementException if the list is empty. Add a guard to verify the list is non-empty before accessing elements, or handle the empty case explicitly. core/src/main/java/org/opensearch/sql/calcite/RuntimeSearchCorrelator.java [133-146] private static RelNode combineSubqueries(List<RexSubQuery> subqueries, RexBuilder rexBuilder) { - RelNode result = subqueries.getFirst().rel; + if (subqueries.isEmpty()) { + throw new IllegalArgumentException("Cannot combine empty subquery list"); + } + RelNode result = subqueries.get(0).rel; for (int i = 1; i < subqueries.size(); i++) { result = LogicalJoin.create( result, subqueries.get(i).rel, List.of(), rexBuilder.makeLiteral(true), Set.of(), JoinRelType.INNER); } return result; } Suggestion importance[1-10]: 7 __ Why: The method uses getFirst() which can throw NoSuchElementException if the list is empty. However, the caller correlate() already validates that subqueries is non-empty before calling this method (line 58), so this is a defensive improvement rather than a critical bug fix.	Medium
	Add null safety for query parts The method doesn't validate that queryParts is non-null or that its length matches expectations. If runtimeQueryParts is null when passed to scan(), this will cause a NullPointerException. Add null checks before processing. opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteEnumerableIndexScan.java [163-173] private String buildRuntimeQuery(String[] queryParts) { + if (queryParts == null || queryParts.length == 0) { + throw new IllegalArgumentException("Query parts cannot be null or empty"); + } StringBuilder query = new StringBuilder(); for (int i = 0; i < queryParts.length; i++) { String part = queryParts[i]; if (pushDownContext.getDynamicQueryString().runtimePredicateParts().contains(i)) { part = pushDownContext.getDynamicQueryString().compiler().compile(part); } query.append(part); } return query.toString(); } Suggestion importance[1-10]: 6 __ Why: The method buildRuntimeQuery() is only called from scan() at line 148, where runtimeQueryParts is already checked for null before calling this method. However, adding explicit validation improves defensive programming and makes the method more robust if called from other contexts in the future.	Low
General	Validate implicit format field requirements When no fields are present, the method returns early without checking if node.isImplicit() is true. For implicit format subsearches, this could bypass necessary correlation setup. Verify that early return is safe for implicit format scenarios. core/src/main/java/org/opensearch/sql/calcite/FormatPlanner.java [53-56] if (fields.isEmpty()) { + if (node.isImplicit()) { + throw new IllegalStateException("Implicit format subsearch requires at least one field"); + } builder.values(new String[] {SEARCH_FIELD}, node.getEmptyString()); return builder.peek(); } Suggestion importance[1-10]: 5 __ Why: The suggestion raises a valid concern about early return for implicit format, but the current implementation appears intentional. The emptyString fallback is a valid result even for implicit subsearches. The suggestion may be overly restrictive without clear evidence of a bug.	Low

[github-actions]

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit 669bee1.

Path	Line	Severity	Description
ppl/src/main/java/org/opensearch/sql/ppl/parser/PPLSearchPredicateCompiler.java	30	medium	Empty or null predicate compiles to '*:*' (match-all). If a subsearch returns an empty or null 'search' field, the implicit format will produce a predicate that matches every document in the parent index, potentially exposing data the caller did not intend to retrieve. This is an unguarded default that could be exploited by crafting a subsearch that always returns an empty search value.
core/src/main/java/org/opensearch/sql/calcite/FormatPlanner.java	89	medium	A scalar field named 'search' in a subsearch result is injected verbatim as an OpenSearch query_string predicate after only PPL parse validation. A principal who can write arbitrary values into an indexed 'search' field (or control eval expressions in a subsearch) can craft search predicates that influence which parent-index documents are returned — for example using OpenSearch query_string wildcards or field-level queries against fields they should not be able to filter on directly.

The table above displays the top 10 most important findings.

Total: 2 | Critical: 0 | High: 0 | Medium: 2 | Low: 0

---

Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.

---

⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

[github-actions]

Persistent review updated to latest commit 669bee1

[penghuo]

enable doc-test for format command.

[songkant-aws]

Enabled.

[penghuo]

Test query not releated to format command.

[songkant-aws]

Removed.

[penghuo]

Add query results.

[songkant-aws]

Added.

[penghuo]

Add query results.

[songkant-aws]

Added.

[penghuo]

Why mention calcite engine specific? It is our default execution engine.

[songkant-aws]

Removed the engine-specific wording.

[penghuo]

This is not a real limitation.

[songkant-aws]

Removed.

[penghuo]

Polish limitation section. make it user friendly.

[songkant-aws]

Rewritten in user-facing terms with concrete valid and invalid examples.

[penghuo]

What is upper-bound?

[songkant-aws]

Clarified maxresults. In case of implicit subsearch, subsearch will formats result rows capped by subsearch.maxout settings. But even this is legit, the OpenSearch query_string execution could error out due to indices.query.bool.max_clause_countlimit

[penghuo]

this is enhancement of search command, please update search command doc.

[penghuo]

Does this feature works with append + search command?

[songkant-aws]

Updated search.md with bracketed subsearch behavior and examples.

[songkant-aws]

Yes, this feature works with append command combination. Added tests for append inside the subsearch and after the parent dynamic search.

[penghuo]

Does this feature works with append + search command?

[penghuo]

avoid using record. hard to backport.

[songkant-aws]

Replaced it with a regular final class.

[penghuo]

buildRuntimeQuery is a function of DynamicQueryStringSpec

[songkant-aws]

Moved buildRuntimeQuery into DynamicQueryStringSpec.

[penghuo]

add UT for new class.

[songkant-aws]

Added UTs.

[penghuo]

what if the results can not be convert to string? what is customer facing error message?

[songkant-aws]

Added validation and user-facing errors for non-text results and invalid generated predicates.

[songkant-aws]

Generally, the scalar type fields could be always converted to string. But the multiset type or array type are not supported, the error message will tell user to avoid it.

[penghuo]

queryString and subsearch can not co-exist?

[songkant-aws]

They can coexist in the original search expression. Detection now inspects the expression tree, so static and subsearch predicates are handled together.

[penghuo]

source = outer a in [ source = inner | fields a ] is as same as source = outer [ source = inner | fields a ]. the dynamic string pushdown works for both cases?

[songkant-aws]

search a IN [subquery] is not valid search syntax; search IN accepts a literal list. Relational IN [subquery] is supported in where, and an integration test now covers it after an implicit-format search.

[github-actions]

Persistent review updated to latest commit 6c89411