perf(tables): keep the batch unique check on per-value GIN probes by waleedlatif1 · Pull Request #8978 · simstudioai/sim

waleedlatif1 · 2026-10-13T18:34:51Z

Summary

checkBatchUniqueConstraintsDb runs inside the batch insert and import write transactions, after the unique locks are taken. It built a single query, table_id = $1 AND (data @> $a OR data @> $b …) LIMIT n, with up to 1000 ORed values per column. Because of the LIMIT, the planner assumed it could stop early. So instead of probing the tenant GIN index (user_table_rows_tenant_data_gin_idx), it walked the table's rows through a btree on table_id and checked every value as a filter on each row. A slow check holds the unique-value and schema locks for its whole duration, so writers to the same table queue up behind it.

This PR splits the values into UNION ALL branches. Each branch ORs at most 25 predicates and has no LIMIT. Each value now plans as its own GIN bitmap probe.

Predicates are unchanged. Every value is still built by uniqueValuePredicate, including the composite object/array equality.
Cast-compared (ttl) columns stay one pass. Their COALESCE(cast = …) predicate is a filter that no index serves, so splitting it into groups would mean one table walk per group.
New equalityCast(column) in column-types/comparison-sql.ts. It decides whether a column compares by containment or by cast, and fieldPredicate, columnTextForEquality and the batch check now all use it.
Unchanged: null and in-batch duplicate handling, error messages, the result shape, and lock order (lock, then check, then insert).

Plan evidence

Production, plain EXPLAIN with synthetic values against a large tenant:

Query shape	Plan
200 ORed values + `LIMIT 200` (before)	`Limit -> Index Scan using user_table_rows_workspace_table_idx`, with the 200 values as a `Filter`
200 ORed values, no `LIMIT`	BitmapOr of 200 GIN probes, but ANDed with a GIN scan of the tenant's whole `table_id` entry
50 ORed values, no `LIMIT`	Plain BitmapOr of GIN probes (planning 3.3 ms)
`UNION ALL` of single-value branches	One GIN bitmap probe per branch

Local Postgres 17 replica of the table and its indexes, with a 20k-row tenant and 1000 values:

	Plan	Execution time
Before	Btree walk	889 ms
After (40 groups)	1000 GIN probes	34 ms (planning 11 ms)

Failure modes covered

A value already stored: the error names its one-based row position.
A duplicate within the batch: reported once, against the first row.
Composite values:
- an object whose stored row is a superset does not conflict;
- the same object with a different key order does conflict;
- an array with a different order does not conflict, while the same array does.
Null and missing cells never conflict.
Conflicts in the first, middle and last of 11 probe groups are all found (260 rows).
ttl (valueForEquality): the same instant written with a different offset conflicts, and the check makes exactly one scan of the table rather than one per group.
Plan: on a 40k-row tenant with 260 values, the transaction-local scan counters show exactly 260 GIN index scans, no other index scans and no sequential scans. On the pre-fix query this shows one btree walk instead (reproduced locally).
Two concurrent batches above the value-lock cap that share a value still serialize: one succeeds and the value is stored once.

Risks

No LIMIT. The result is bounded by the rows that actually match. A unique column cannot hold duplicates, because enabling unique rejects them, so this matches the old row bound. A legacy duplicate would now be reported for every matching row rather than an arbitrary subset.
Group size is a planner heuristic. 25 stays well under the size where a no-LIMIT OR started adding the whole-tenant table_id intersection (seen at 200, absent at 50).
Planning cost. At most 40 branches per column, measured at about 11 ms for 1000 values.
The plan test depends on cost estimates. It relies on the planner choosing GIN probes for a 40k-row tenant, which held across 10k–40k rows in local runs.

The batch unique check ORed every value into one query under LIMIT n. The LIMIT made the planner expect an early exit and walk the table's rows through a btree with every value as a filter, so large batches read the whole table inside the write transaction while holding the unique and schema locks. Split the values into UNION ALL branches of at most 25 ORed predicates with no LIMIT, so each value plans as its own GIN probe. Cast-compared columns (ttl), which no index serves, stay one pass. The predicates, null and in-batch handling, error messages, result shape, and lock order are unchanged.

vercel · 2026-10-13T18:34:57Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Oct 13, 2026 6:56pm UTC

cubic-dev-ai

No issues found across 4 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-13T18:38:13Z

@greptile

waleedlatif1 · 2026-10-13T18:38:15Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-13T18:38:20Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

greptile-apps · 2026-10-13T18:39:50Z

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no blocking issue remains.

Summary

The PR splits batch uniqueness checks into small UNION ALL groups to favor GIN probes. Cast-based checks stay in one query, and equalityCast centralizes the choice.

Large batch checks now probe containment values through GIN.

_{Reviews (4) · Last reviewed commit: "fix(tables): measure the unique-check sc..." · Reviewed by Greptile}

cubic-dev-ai

No issues found across 4 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-13T18:45:51Z

@greptile

waleedlatif1 · 2026-10-13T18:45:53Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-13T18:45:58Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 4 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

Turn on auto-fix | Re-trigger cubic}

… index

waleedlatif1 · 2026-10-13T18:56:32Z

@greptile

waleedlatif1 · 2026-10-13T18:56:34Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-13T18:56:40Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 4 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

Turn on auto-fix | Re-trigger cubic}

cubic-dev-ai Bot reviewed Oct 13, 2026

View reviewed changes

chore(tables): fold the unique-check test helpers into one

7474048

vercel Bot temporarily deployed to Preview October 13, 2026 18:38 Inactive

greptile-apps Bot reviewed Oct 13, 2026

View reviewed changes

Comment thread apps/sim/lib/table/column-types/comparison-sql.ts Outdated

cubic-dev-ai Bot reviewed Oct 13, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 13, 2026

View reviewed changes

Comment thread apps/sim/lib/table/column-types/comparison-sql.ts Outdated

fix(tables): coalesce a null equality cast to undefined

b21d12f

vercel Bot temporarily deployed to Preview October 13, 2026 18:45 Inactive

cubic-dev-ai Bot reviewed Oct 13, 2026

View reviewed changes

fix(tables): measure the unique-check scans by delta on a flushed GIN…

d619895

… index

vercel Bot temporarily deployed to Preview October 13, 2026 18:56 Inactive

cubic-dev-ai Bot reviewed Oct 13, 2026

View reviewed changes

waleedlatif1 merged commit fa5f0db into staging Oct 13, 2026
51 checks passed

waleedlatif1 mentioned this pull request Oct 13, 2026

v0.9.21: gemini 4 argon, db improvements, fivetran integration, search speedups #8989

Merged

waleedlatif1 deleted the perf/table-unique-probe branch October 13, 2026 21:19

Conversation

waleedlatif1 commented Oct 13, 2026

Summary

Plan evidence

Failure modes covered

Risks

Uh oh!

vercel Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Oct 13, 2026

Uh oh!

waleedlatif1 commented Oct 13, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 13, 2026

Uh oh!

greptile-apps Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Oct 13, 2026

Uh oh!

waleedlatif1 commented Oct 13, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 13, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Oct 13, 2026

Uh oh!

waleedlatif1 commented Oct 13, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 13, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Oct 13, 2026 •

edited

Loading

greptile-apps Bot commented Oct 13, 2026 •

edited

Loading