feat(ai): give the schema prompt to the tables the question is about by J2TeamNNL · Pull Request #3750 · TableProApp/TablePro

J2TeamNNL · 2026-10-21T08:44:01Z

Summary

AI chat and inline suggestions send columns for at most maxSchemaTables tables (20 by default), and took the first ones listed. Each turn now gives that budget to the tables it is about.
First come the tables the editor's query reads, through the new SQLContextAnalyzer.tableReferences(in:grammar:). In chat this covers every statement in the 2,000 characters the prompt carries, while Include current query is on and the editor is SQL. Inline suggestions use the statement at the cursor.
A referenced name puts first every table it could mean, matched without case and the default schema's table first, since engines fold unquoted names differently.
The question's matches follow: BM25 over identifier words with trigram matching for typos (SchemaRelevanceRanker in TableProAutocomplete). The ranker reads table names and comments. It reads columns only once every table has them, so a table loaded on an earlier turn never outranks one whose columns were never read.
With at most 300 tables and a driver whose fetchAllColumns is one query (new DatabaseDriver.providesBulkColumnFetch), the first ranked turn reads every column in that one query.
A custom slash command ranks by its body, not by the schema it renders. Before the connection's AI policy allows access, it reads no more than it did before.
Tables past the budget stay a count line. The chat model reaches them with list_tables and describe_table.
The chat's column cache is dropped on a database or schema switch. Before, a same-named table in the new database kept the old database's columns.
ALTER TABLE schema.table now resolves to table in schema, for completion too, where it resolved to a table named schema.

What reaches the model is unchanged: the same settings and budget decide it. Only which tables fill the budget changes.

Docs: TableProApp/docs#122

Test plan

SchemaRelevanceRankerTests and SQLContextAnalyzerTableReferenceTests (TableProCore, swift test): ranking, a statement ending in a comment or open string, a CTE that shares a table's name, a qualified ALTER target.
AISchemaContextRelevanceTests: question order, table comments, columns only when every table has them, editor tables first, SQL keywords match nothing, every statement counts, default schema first, case collisions keep both, the count line.
SchemaContextForAITests: the tables the cursor's statement reads take the inline budget, including a table named after the first 2,000 characters.
AIChatViewModelSlashTests: a custom command ranks by its body; a tool result turn does not replace the question.

J2TeamNNL

Review notes on the schema relevance ranking. Each comment is a place where the behavior can go wrong.

J2TeamNNL · 2026-10-21T09:28:54Z

        }
    }

+    private func prioritizeSchema(for question: String) {


Ranking only sees columns for tables that are already loaded, and those are the first tables in default order. On the first question no columns are loaded, so a question like "which rows have an email" cannot find a table at position 300 by its column. On later turns the tables loaded earlier outrank better matches that are not loaded yet.

J2TeamNNL · 2026-10-21T09:28:54Z

        }
    }

+    private func prioritizeSchema(for question: String) {


A follow-up that matches no table ("and by month?") sets relevantTableNames to an empty list. The next prompt then falls back to alphabetical order and the tables the chat is about can drop out of the first maxTables. Keep the previous ranking when the new one is empty.

J2TeamNNL · 2026-10-21T09:28:54Z

    }

+    private func prioritizeSchema(for question: String) {
+        let text = [question, currentQuery ?? ""].joined(separator: "\n")


currentQuery is joined into the ranking text. SQL words like order, group, user and table then match table names at full weight. SELECT * FROM x ORDER BY y boosts orders and order_items.

J2TeamNNL · 2026-10-21T09:28:54Z

            await inFlight.value
-            return
+            // The load in flight fetched for the previous order; a new order may need other tables.
+            guard question != nil else { return }


Race in ensureSchemaLoaded. Two callers that wait on the same in-flight load both resume afterwards. The first sets inFlightSchemaLoad = nil and can clear the task the second caller just stored, so a third caller starts a duplicate load. Clear it only when it is still your own task.

J2TeamNNL · 2026-10-21T09:28:54Z

+                        counts[word, default: 0] += 1
+                    }
+                }
+                for term in counts.keys {


Fuzzy matching builds trigram sets for the query word and for every vocabulary term on each comparison. With thousands of tables this allocates a lot, and it runs on the main actor. Compute term trigrams once per index.

J2TeamNNL · 2026-10-21T09:28:54Z

+            var score = 0.0
+            for match in matches {
+                guard let frequency = terms[match.term] else { continue }
+                score += match.weight * index.bm25(frequency: frequency, idfTerm: match.term, length: index.lengths[offset])


The full-name boost uses substring containment on the joined query words. A table data gets +5 from "metadata", and user from "username". The +5 outweighs the BM25 score. Match whole words in order instead.

J2TeamNNL · 2026-10-21T09:28:54Z

+        }
+        return word
+    }
+


The stemmer maps "statuses" to "statuse" but leaves "status" alone, and "caches" to "cach". It also skips non-ASCII words, so a question in CJK or Cyrillic shares no words with English identifiers and nothing ranks.

J2TeamNNL · 2026-10-21T09:28:54Z

-            lines.append(
-                "\n… and \(tables.count - maxTables) more tables (not shown)"
-            )
+        let hiddenTables = tables.dropFirst(maxTables)


Up to 200 hidden table names are now added to every prompt, where before it was a count line. There is no token budget for them, so a database with 1000+ tables pays this on every message.

J2TeamNNL · 2026-10-21T09:28:54Z

+        guard !text.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty else { return }
+        relevantTableNames = AISchemaContext.relevantTableNames(
+            in: tables,
+            columnsByTable: columnsByTable,


relevantTableNames is shared mutable state set at the start of a turn and read later by orderedTables(). A slash command or an attachment prime that runs in between can render with another turn's ordering.

J2TeamNNL · 2026-10-21T09:28:54Z

            guard let self else { return }
            if settings.includeSchema {
-                await self.ensureSchemaLoaded()
+                let question = await MainActor.run {


The question is read from messages inside the detached task after an actor hop, not when the turn is created. If the history changes in between (regenerate, edit), the ranking uses the wrong question. Capture it before the task starts.

… clear the schema load only when it is ours A follow-up that matches no table no longer resets the ranking. The full-name boost needs the name's words in order in the question. Term trigrams are built once per index. The in-flight schema load is cleared only by the task that set it, and the question is read when the turn starts.

J2TeamNNL · 2026-10-21T09:29:12Z

Follow-up to the review above. Fixes are in J2TeamNNL@8cf9ddf.

Columns only for loaded tables: skipped. Ranking by columns of tables that are not loaded needs a column fetch for every table, which this PR avoids on purpose.
Empty rank wipes the ranking: fixed.
currentQuery in the ranking text: skipped. Table names in the editor query are the strongest signal, and stripping SQL words adds a keyword list for a small gain.
Race in ensureSchemaLoaded: fixed.
Fuzzy matching cost: fixed for the allocations (term trigrams built once). Moving it off the main actor is skipped, the work is now one pass over the vocabulary.
Name boost on substrings: fixed, whole words in order, with a test.
Stemmer: skipped. The trigram match already covers statuses and status, and a non-Latin question ranking nothing is the same as before this PR.
200 hidden table names: skipped. The cap is deliberate and bounded.
Shared relevantTableNames: skipped. It runs on the main actor, is set only at turn start, and the load that follows uses the same order.
Question read in the detached task: fixed.

Checks: swiftlint --strict on ViewModels and Core/AI is clean. The ranker tests pass in a scratch package (8 of 8). The app target was not built here, so CI has to confirm it.

…y, and keep hidden tables a count

… AI policy, and read table references past comments

…t table spelling first

… shares its name with

… an ALTER target's schema

…ulk entry only when it means one table

J2TeamNNL added 2 commits October 21, 2026 15:43

feat(ai): give the schema prompt to the tables the question is about

9e61895

docs(changelog): note schema relevance in AI chat

ca6296f

J2TeamNNL marked this pull request as ready for review October 21, 2026 08:46

J2TeamNNL commented Oct 21, 2026

View reviewed changes

datlechin added 8 commits October 22, 2026 17:29

Merge remote-tracking branch 'origin/main' into feat/ai-schema-relevance

1fb0382

fix(ai): rank by the editor's table references, read columns uniforml…

16cf838

…y, and keep hidden tables a count

fix(ai): scope the chat column cache, gate slash command reads on the…

afe04b1

… AI policy, and read table references past comments

fix(ai): keep tables a bulk read left out unknown, and match the exac…

6c87923

…t table spelling first

fix(ai): load foreign keys apart from columns, and keep a table a CTE…

5477979

… shares its name with

fix(ai): put every table a referenced name could mean first, and keep…

364f523

… an ALTER target's schema

fix(ai): rank a custom command by its body, and trust a case-folded b…

525b97d

…ulk entry only when it means one table

fix(ai): rank by the user's question, not a tool result turn

9bc07fd

datlechin mentioned this pull request Oct 22, 2026

docs(features): AI schema context fills its table budget by relevance TableProApp/docs#122

Merged

datlechin merged commit b9b1170 into TableProApp:main Oct 22, 2026
57 of 59 checks passed

datlechin mentioned this pull request Oct 22, 2026

chore(release): bring release/0.83 up to main for 0.83.0-beta.2 #3798

Merged

Navigation Menu

Repository navigation

feat(ai): give the schema prompt to the tables the question is about - #3750

feat(ai): give the schema prompt to the tables the question is about#3750
datlechin merged 11 commits into
TableProApp:mainfrom
J2TeamNNL:feat/ai-schema-relevance

J2TeamNNL commented Oct 21, 2026 •

edited by datlechin

Loading

Uh oh!

J2TeamNNL left a comment

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL commented Oct 21, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

J2TeamNNL commented Oct 21, 2026 • edited by datlechin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

J2TeamNNL left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

J2TeamNNL commented Oct 21, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

J2TeamNNL commented Oct 21, 2026 •

edited by datlechin

Loading