feat: Distribute exact-match lucene variable references - #2987
Conversation
🦋 Changeset detectedLatest commit: 39963bf The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Greptile SummaryThis PR adds exact-match expansion for quoted Lucene variable references while preserving substring semantics for unquoted references.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains.
|
| Filename | Overview |
|---|---|
| packages/common-utils/src/variables.ts | Adds context-aware Lucene substitution and AST-based distribution of quoted field references, including the prior-thread special-syntax fix. |
| packages/common-utils/src/queryParser.ts | Centralizes parser-sensitive token encoding and adds source-preserving decoding used by variable rewriting. |
| packages/app/src/components/SQLEditor/variableCompletions.tsx | Uses language-aware substitution for previews and documents quoted exact-match behavior. |
| packages/common-utils/src/tests/variables.test.ts | Covers exact-match distribution, escaping, negation, empty selections, and the URL and escaped-colon cases from the previous finding. |
| packages/common-utils/src/tests/queryParser.test.ts | Verifies special-token source round-tripping and the SQL semantics of distributed exact-match terms. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
Input[Lucene template] --> Tokenize[Tokenize variables and macros]
Tokenize --> Encode[Encode parser-sensitive text]
Encode --> Sentinel[Replace references with sentinels]
Sentinel --> Parse[Parse Lucene AST]
Parse --> Rewrite{Quoted field reference?}
Rewrite -->|Yes| Exact[Distribute field exact matches]
Rewrite -->|No| Standard[Apply standard variable expansion]
Exact --> Restore[Restore source spelling]
Standard --> Restore
Restore --> Output[Expanded Lucene query]
Reviews (2): Last reviewed commit: "fix: Encode special values before lucene..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
E2E Test Results✅ All tests passed • 308 passed • 1 skipped • 1206s
Tests ran across 4 shards in parallel. |
🟡 Tier 3 — StandardIntroduces new logic, modifies core functionality, or touches areas with non-trivial risk. Why this tier:
Review process: Full human review — logic, architecture, edge cases. Stats
|
Deep Review✅ No critical issues found. This PR rewrites quoted Lucene variable references ( 🟡 P2 -- recommended
🔵 P3 nitpicks (5)
Reviewers (7): correctness (independent trace), security, testing, maintainability, api-contract, project-standards, previous-comments, performance. Testing gaps:
Note: the adversarial and TypeScript reviewer sub-agents did not return before synthesis; their lanes (break-input construction and AST type-safety) were partially covered by the correctness trace and security review. |
| export function decodeSpecialTokensToSource(query: string): string { | ||
| return SPECIAL_TOKEN_ENCODINGS.reduce( | ||
| (decoded, { decodePattern, source }) => | ||
| decoded.replace(decodePattern, source), | ||
| query, | ||
| ); | ||
| } |
There was a problem hiding this comment.
The changes here were introduced so that a lossless encode/decoding could be used to (a) encode before parsing lucene in variables.ts and (b) decode back to the exact original input instead of what decodeSpecialTokens returns, which is slightly lossy in some cases because it returns unescaped values.
1 participant
Summary
This PR enhances support for variables in lucene by rewriting
Field:"$var"to(Field:"A" OR Field"B")(when A and B are selected for $var. This preserves exact-match semantics. The basic expansionField:("A" OR "B")is a substring condition `Field ILIKE "%A%" OR Field ILIKE "%B%".The quoted form was chosen for exact-match semantics because
Field:"A"is exact match.Field:$varcontinues using the basic expansion, matchingField:Asubstring semantics.How
The rewrite is done through the following process:
__hdx_sentinel_Nin a sentinelString, and track the locations of the placeholders within the sentinelString. We do this to ensure that the sentinelString can be parsed as valid lucene (${var} reference are not valid single lucene terms).The transform is a lucene --> lucene transformation, so the resulting lucene still goes through the existing lucene --> SQL transpiler and inherits all of its optimizations.
Screenshots or video
Some examples:
How to test on Vercel preview
References