Skip to content

Make SQLite primary keys AUTOINCREMENT #1390

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
brandur wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Make SQLite primary keys AUTOINCREMENT #1390

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

⚠️ This release contains a new database migration, version 8, that only affects SQLite. It rebuilds `river_job` with `AUTOINCREMENT` to prevent automatically generated job IDs from being reused after deletion. The migration is a no-op for PostgreSQL.

### Added

- Added `Config.LeaderElectionDisabled` to let a client work jobs without participating in leader election or running maintenance services. Other eligible clients in the same database and schema continue handling scheduling, retries, periodic enqueueing, rescue, and cleanup. [PR #1382](https://github.com/riverqueue/river/pull/1382).
Expand All @@ -22,6 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Improved PostgreSQL job listing performance when filtering by one finalized state (`completed`, `cancelled`, or `discarded`) and sorting by finalized time, including in River UI. [PR #1374](https://github.com/riverqueue/river/pull/1374).
- Fixed `JobRescuer` overwriting jobs that complete, leave the running state, or are claimed again by another worker after being fetched for rescue, preserving their state, errors, metadata, and timestamps across PostgreSQL and SQLite drivers. Fixes [#1302](https://github.com/riverqueue/river/issues/1302). [PR #1373](https://github.com/riverqueue/river/pull/1373).
- Fixed SQLite notification listeners delivering notifications from before a subscription or from an unsubscribe gap. Notification reads now fetch subscribed topics in bounded batches, and cleanup deletes expired notifications in batches of 10,000 rows (reduced to 1,000 after repeated timeouts), with pauses between batches to reduce write lock contention. [PR #1381](https://github.com/riverqueue/river/pull/1381).
- Fixed SQLite reusing the ID of a deleted job when that job held the largest ID, which could cause an ID observed earlier to refer to an unrelated job later. [PR #1390](https://github.com/riverqueue/river/pull/1390).

## [0.47.0] - 2026-09-01

Expand Down
2 changes: 1 addition & 1 deletion riverdriver/river_driver_interface.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -964,7 +964,7 @@ func MigrationLineMainTruncateTables(version int) []string {
return []string{"river_job", "river_leader", "river_queue"}
case 5, 6:
return []string{"river_job", "river_leader", "river_queue", "river_client", "river_client_queue"}
case 0, 7:
case 0, 7, 8:
return []string{"river_job", "river_leader", "river_queue", "river_notification"}
}

Expand Down
3 changes: 3 additions & 0 deletions riverdriver/riverdatabasesql/migration/main/008_job_id_autoincrement.down.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
-- No-op. PostgreSQL sequences already prevent automatically generated job IDs
-- from being reused.
SELECT 1;
3 changes: 3 additions & 0 deletions riverdriver/riverdatabasesql/migration/main/008_job_id_autoincrement.up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
-- No-op. PostgreSQL sequences already prevent automatically generated job IDs
-- from being reused.
SELECT 1;
14 changes: 14 additions & 0 deletions riverdriver/riverdrivertest/job_insert.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -583,6 +583,20 @@ func exerciseJobInsert[TTx any](ctx context.Context, t *testing.T,
t.Run("JobInsertFull", func(t *testing.T) {
t.Parallel()

t.Run("DoesNotReuseAutomaticallyGeneratedID", func(t *testing.T) {
t.Parallel()

exec, _ := setup(ctx, t)

job := testfactory.Job(ctx, t, exec, &testfactory.JobOpts{})

_, err := exec.JobDelete(ctx, &riverdriver.JobDeleteParams{ID: job.ID})
require.NoError(t, err)

jobAfter := testfactory.Job(ctx, t, exec, &testfactory.JobOpts{})
require.Greater(t, jobAfter.ID, job.ID)
})

t.Run("MinimalArgsWithDefaults", func(t *testing.T) {
t.Parallel()

Expand Down
37 changes: 36 additions & 1 deletion riverdriver/riverdrivertest/migration.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,8 @@ func exerciseMigration[TTx any](ctx context.Context, t *testing.T,
driver.GetMigrationTruncateTables(riverdriver.MigrationLineMain, 6))
require.Equal(t, expectedLatestTables,
driver.GetMigrationTruncateTables(riverdriver.MigrationLineMain, 7))
require.Equal(t, expectedLatestTables,
driver.GetMigrationTruncateTables(riverdriver.MigrationLineMain, 8))
require.Equal(t, expectedLatestTables,
driver.GetMigrationTruncateTables(riverdriver.MigrationLineMain, 0))
})
Expand Down Expand Up @@ -144,7 +146,7 @@ func exerciseMigration[TTx any](ctx context.Context, t *testing.T,
}
})

t.Run("MigrateDownFromVersionSevenWithJobData", func(t *testing.T) {
t.Run("MigrateDownFromVersionEightWithJobData", func(t *testing.T) {
t.Parallel()

driver, schema := driverWithSchema(ctx, t, &riverdbtest.TestSchemaOpts{
Expand All @@ -171,6 +173,39 @@ func exerciseMigration[TTx any](ctx context.Context, t *testing.T,
require.NotZero(t, job.ID)
})

t.Run("MigrateUpFromVersionSevenWithJobData", func(t *testing.T) {
t.Parallel()

driver, schema := driverWithSchema(ctx, t, &riverdbtest.TestSchemaOpts{
DisableReuse: true,
LineTargetVersions: map[string]int{
riverdriver.MigrationLineMain: 7,
},
})
exec := driver.GetExecutor()

job := testfactory.Job(ctx, t, exec, &testfactory.JobOpts{Schema: schema})

migrator, err := rivermigrate.New(driver, &rivermigrate.Config{
Line: riverdriver.MigrationLineMain,
Logger: riversharedtest.Logger(t),
Schema: schema,
})
require.NoError(t, err)

_, err = migrator.Migrate(ctx, rivermigrate.DirectionUp, nil)
require.NoError(t, err)

job, err = exec.JobGetByID(ctx, &riverdriver.JobGetByIDParams{ID: job.ID, Schema: schema})
require.NoError(t, err)

_, err = exec.JobDelete(ctx, &riverdriver.JobDeleteParams{ID: job.ID, Schema: schema})
require.NoError(t, err)

jobAfter := testfactory.Job(ctx, t, exec, &testfactory.JobOpts{Schema: schema})
require.Greater(t, jobAfter.ID, job.ID)
})

t.Run("MigrateUpFromVersionSixWithQueueData", func(t *testing.T) {
t.Parallel()

Expand Down
3 changes: 3 additions & 0 deletions riverdriver/riverpgxv5/migration/main/008_job_id_autoincrement.down.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
-- No-op. PostgreSQL sequences already prevent automatically generated job IDs
-- from being reused.
SELECT 1;
3 changes: 3 additions & 0 deletions riverdriver/riverpgxv5/migration/main/008_job_id_autoincrement.up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
-- No-op. PostgreSQL sequences already prevent automatically generated job IDs
-- from being reused.
SELECT 1;
3 changes: 2 additions & 1 deletion riverdriver/riversqlite/internal/dbsqlc/river_job.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
CREATE TABLE river_job (
id integer PRIMARY KEY, -- SQLite makes this autoincrementing automatically
-- AUTOINCREMENT prevents SQLite from reusing the IDs of deleted jobs.
id integer PRIMARY KEY AUTOINCREMENT,
args jsonb NOT NULL DEFAULT (jsonb('{}')),
attempt integer NOT NULL DEFAULT 0,
attempted_at timestamp,
Expand Down
4 changes: 2 additions & 2 deletions riverdriver/riversqlite/migration/main/006_bulk_unique.up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
DROP TABLE /* TEMPLATE: schema */river_job;

CREATE TABLE /* TEMPLATE: schema */river_job (
id integer PRIMARY KEY, -- SQLite makes this autoincrementing automatically
id integer PRIMARY KEY, -- SQLite aliases this to ROWID, which may reuse deleted IDs.
args blob NOT NULL DEFAULT '{}',
attempt integer NOT NULL DEFAULT 0,
attempted_at timestamp,
Expand Down Expand Up @@ -60,4 +60,4 @@ CREATE UNIQUE INDEX /* TEMPLATE: schema */river_job_unique_idx ON river_job (uni
WHEN 'running' THEN unique_states & (1 << 6)
WHEN 'scheduled' THEN unique_states & (1 << 7)
ELSE 0
END >= 1;
END >= 1;
2 changes: 1 addition & 1 deletion .../riversqlite/migration/main/007_notification_outbox_sqlite_jsonb_and_sql_cleanup.down.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ DROP INDEX /* TEMPLATE: schema */river_job_unique_idx;
ALTER TABLE /* TEMPLATE: schema */river_job RENAME TO river_job_old;

CREATE TABLE /* TEMPLATE: schema */river_job (
id integer PRIMARY KEY, -- SQLite makes this autoincrementing automatically
id integer PRIMARY KEY, -- SQLite aliases this to ROWID, which may reuse deleted IDs.
args blob NOT NULL DEFAULT '{}',
attempt integer NOT NULL DEFAULT 0,
attempted_at timestamp,
Expand Down
2 changes: 1 addition & 1 deletion ...er/riversqlite/migration/main/007_notification_outbox_sqlite_jsonb_and_sql_cleanup.up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ DROP INDEX /* TEMPLATE: schema */river_job_unique_idx;
ALTER TABLE /* TEMPLATE: schema */river_job RENAME TO river_job_old;

CREATE TABLE /* TEMPLATE: schema */river_job (
id integer PRIMARY KEY, -- SQLite makes this autoincrementing automatically
id integer PRIMARY KEY, -- SQLite aliases this to ROWID, which may reuse deleted IDs.
args blob NOT NULL DEFAULT (jsonb('{}')),
attempt integer NOT NULL DEFAULT 0,
attempted_at timestamp,
Expand Down
98 changes: 98 additions & 0 deletions riverdriver/riversqlite/migration/main/008_job_id_autoincrement.down.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
-- Rebuild river_job to restore SQLite's default ROWID allocation behavior.

DROP INDEX /* TEMPLATE: schema */river_job_kind;
DROP INDEX /* TEMPLATE: schema */river_job_state_and_finalized_at_index;
DROP INDEX /* TEMPLATE: schema */river_job_prioritized_fetching_index;
DROP INDEX /* TEMPLATE: schema */river_job_unique_idx;

ALTER TABLE /* TEMPLATE: schema */river_job RENAME TO river_job_old;

CREATE TABLE /* TEMPLATE: schema */river_job (
id integer PRIMARY KEY,
args blob NOT NULL DEFAULT (jsonb('{}')),
attempt integer NOT NULL DEFAULT 0,
attempted_at timestamp,
attempted_by blob, -- json
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
errors blob, -- json
finalized_at timestamp,
kind text NOT NULL,
max_attempts integer NOT NULL DEFAULT 25,
metadata blob NOT NULL DEFAULT (jsonb('{}')),
priority integer NOT NULL DEFAULT 1,
queue text NOT NULL DEFAULT 'default',
state text NOT NULL DEFAULT 'available',
scheduled_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
tags blob NOT NULL DEFAULT (jsonb('[]')),
unique_key blob,
unique_states integer,
CONSTRAINT finalized_or_finalized_at_null CHECK (
(finalized_at IS NULL AND state NOT IN ('cancelled', 'completed', 'discarded')) OR
(finalized_at IS NOT NULL AND state IN ('cancelled', 'completed', 'discarded'))
),
CONSTRAINT priority_in_range CHECK (priority >= 1 AND priority <= 4),
CONSTRAINT queue_length CHECK (length(queue) > 0 AND length(queue) < 128),
CONSTRAINT kind_length CHECK (length(kind) > 0 AND length(kind) < 128),
CONSTRAINT state_valid CHECK (state IN ('available', 'cancelled', 'completed', 'discarded', 'pending', 'retryable', 'running', 'scheduled'))
);

INSERT INTO /* TEMPLATE: schema */river_job (
id,
args,
attempt,
attempted_at,
attempted_by,
created_at,
errors,
finalized_at,
kind,
max_attempts,
metadata,
priority,
queue,
state,
scheduled_at,
tags,
unique_key,
unique_states
)
SELECT
id,
args,
attempt,
attempted_at,
attempted_by,
created_at,
errors,
finalized_at,
kind,
max_attempts,
metadata,
priority,
queue,
state,
scheduled_at,
tags,
unique_key,
unique_states
FROM /* TEMPLATE: schema */river_job_old;

DROP TABLE /* TEMPLATE: schema */river_job_old;

CREATE INDEX /* TEMPLATE: schema */river_job_kind ON river_job (kind);
CREATE INDEX /* TEMPLATE: schema */river_job_state_and_finalized_at_index ON river_job (state, finalized_at) WHERE finalized_at IS NOT NULL;
CREATE INDEX /* TEMPLATE: schema */river_job_prioritized_fetching_index ON river_job (state, queue, priority, scheduled_at, id);
CREATE UNIQUE INDEX /* TEMPLATE: schema */river_job_unique_idx ON river_job (unique_key)
WHERE unique_key IS NOT NULL
AND unique_states IS NOT NULL
AND CASE state
WHEN 'available' THEN unique_states & (1 << 0)
WHEN 'cancelled' THEN unique_states & (1 << 1)
WHEN 'completed' THEN unique_states & (1 << 2)
WHEN 'discarded' THEN unique_states & (1 << 3)
WHEN 'pending' THEN unique_states & (1 << 4)
WHEN 'retryable' THEN unique_states & (1 << 5)
WHEN 'running' THEN unique_states & (1 << 6)
WHEN 'scheduled' THEN unique_states & (1 << 7)
ELSE 0
END >= 1;
100 changes: 100 additions & 0 deletions riverdriver/riversqlite/migration/main/008_job_id_autoincrement.up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
-- Rebuild river_job so automatically generated IDs are never reused after the
-- job holding the largest ID is deleted. Unlike PostgreSQL sequences, SQLite's
-- default ROWID allocator may otherwise reuse that deleted ID.

DROP INDEX /* TEMPLATE: schema */river_job_kind;
DROP INDEX /* TEMPLATE: schema */river_job_state_and_finalized_at_index;
DROP INDEX /* TEMPLATE: schema */river_job_prioritized_fetching_index;
DROP INDEX /* TEMPLATE: schema */river_job_unique_idx;

ALTER TABLE /* TEMPLATE: schema */river_job RENAME TO river_job_old;

CREATE TABLE /* TEMPLATE: schema */river_job (
id integer PRIMARY KEY AUTOINCREMENT,

@bgentry bgentry Sep 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AUTOINCREMENT starts tracking IDs from the rows copied into the new river_job, so an ID that exists only in Pro's dead letter table can still be handed out again after this migration. That leaves the original dead letter conflict possible for an upgraded database.

Example and suggested fix

Suppose the highest live job ID is 41 and river_job_dead_letter contains an earlier job with ID 42. Copying the live rows initializes sqlite_sequence to 41; the next job gets 42, and moving it to dead letter fails on the existing row's primary key. I reproduced this allocation and conflict in SQLite. An empty live job table has the same problem for every archived ID.

If existing Pro SQLite databases are supported, their upgrade should advance the job sequence past the maximum ID in both live and dead letter tables, in the same transaction, and test the subsequent dead letter move. For OSS-only databases, IDs that were issued and then deleted without any surviving record cannot be recovered; the migration can only guarantee no reuse going forward from the highest ID it can observe.

args blob NOT NULL DEFAULT (jsonb('{}')),
attempt integer NOT NULL DEFAULT 0,
attempted_at timestamp,
attempted_by blob, -- json
created_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
errors blob, -- json
finalized_at timestamp,
kind text NOT NULL,
max_attempts integer NOT NULL DEFAULT 25,
metadata blob NOT NULL DEFAULT (jsonb('{}')),
priority integer NOT NULL DEFAULT 1,
queue text NOT NULL DEFAULT 'default',
state text NOT NULL DEFAULT 'available',
scheduled_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
tags blob NOT NULL DEFAULT (jsonb('[]')),
unique_key blob,
unique_states integer,
CONSTRAINT finalized_or_finalized_at_null CHECK (
(finalized_at IS NULL AND state NOT IN ('cancelled', 'completed', 'discarded')) OR
(finalized_at IS NOT NULL AND state IN ('cancelled', 'completed', 'discarded'))
),
CONSTRAINT priority_in_range CHECK (priority >= 1 AND priority <= 4),
CONSTRAINT queue_length CHECK (length(queue) > 0 AND length(queue) < 128),
CONSTRAINT kind_length CHECK (length(kind) > 0 AND length(kind) < 128),
CONSTRAINT state_valid CHECK (state IN ('available', 'cancelled', 'completed', 'discarded', 'pending', 'retryable', 'running', 'scheduled'))
);

INSERT INTO /* TEMPLATE: schema */river_job (
id,
args,
attempt,
attempted_at,
attempted_by,
created_at,
errors,
finalized_at,
kind,
max_attempts,
metadata,
priority,
queue,
state,
scheduled_at,
tags,
unique_key,
unique_states
)
SELECT
id,
args,
attempt,
attempted_at,
attempted_by,
created_at,
errors,
finalized_at,
kind,
max_attempts,
metadata,
priority,
queue,
state,
scheduled_at,
tags,
unique_key,
unique_states
FROM /* TEMPLATE: schema */river_job_old;

DROP TABLE /* TEMPLATE: schema */river_job_old;

@bgentry bgentry Sep 26, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropping river_job_old also drops Pro-owned columns, indexes, and triggers if this database already installed Pro on main 007. The new Pro prerequisite runs only in migration 001, so it cannot protect a database that has already passed that migration.

Upgrade scenario and suggested fix

Pro's migrations add partition_key, workflow_id, and workflow_task to river_job, along with indexes and synchronization triggers. This rebuild copies only OSS columns into the new table; Pro's columns are omitted, while its indexes and triggers stay attached to the renamed table and disappear here. The Pro migration records remain, so rerunning Pro migrations does not restore them. The 008 down migration has the same exposure after Pro has been installed.

If existing Pro SQLite databases are intentionally unsupported, please make this migration fail before the rebuild when it finds an installed Pro schema, and document the reset/upgrade path. If they are supported, the two migration lines need a coordinated upgrade that preserves or reconstructs Pro's data and schema in both allowed directions. A regression test should start at main 007 with Pro installed and populated, then migrate main to 008 and exercise Pro operations.


CREATE INDEX /* TEMPLATE: schema */river_job_kind ON river_job (kind);
CREATE INDEX /* TEMPLATE: schema */river_job_state_and_finalized_at_index ON river_job (state, finalized_at) WHERE finalized_at IS NOT NULL;
CREATE INDEX /* TEMPLATE: schema */river_job_prioritized_fetching_index ON river_job (state, queue, priority, scheduled_at, id);
CREATE UNIQUE INDEX /* TEMPLATE: schema */river_job_unique_idx ON river_job (unique_key)
WHERE unique_key IS NOT NULL
AND unique_states IS NOT NULL
AND CASE state
WHEN 'available' THEN unique_states & (1 << 0)
WHEN 'cancelled' THEN unique_states & (1 << 1)
WHEN 'completed' THEN unique_states & (1 << 2)
WHEN 'discarded' THEN unique_states & (1 << 3)
WHEN 'pending' THEN unique_states & (1 << 4)
WHEN 'retryable' THEN unique_states & (1 << 5)
WHEN 'running' THEN unique_states & (1 << 6)
WHEN 'scheduled' THEN unique_states & (1 << 7)
ELSE 0
END >= 1;
Loading