Release 2026.3.0

.github/instructions/api.instructions.md

@@ -94,10 +94,12 @@ async def publish_score_set(
     worker: ArqRedis = Depends(get_worker),
 ):
     # ... validation and DB updates ...
-    await worker.enqueue_job(
-        "create_variants_for_score_set",
-        score_set.id,
-        correlation_id,
+    await enqueue_pipeline_for_score_set(
+        db=db,
+        redis=worker,
+        pipeline_name="publish_score_set",
+        score_set=item,
+        user=user_data.user,
     )
 ```
 
@@ -114,24 +116,20 @@ responses=shared_responses  # Defines 4xx/5xx response schemas
 Most write operations trigger a multi-step pipeline via the worker:
 
 ```python
-from mavedb.lib.workflow.pipeline_factory import PipelineFactory
+from mavedb.lib.workflow.kickoff import enqueue_pipeline_for_score_set
 
 # In a router endpoint:
-pipeline, entrypoint_job_run = PipelineFactory.create_pipeline(
+pipeline, job = await enqueue_pipeline_for_score_set(
     db=db,
-    name="validate_map_annotate_score_set",
-    pipeline_params={
-        "score_set_id": score_set.id,
-        "updater_id": user_data.user.id,
-        "correlation_id": logging_context().get("correlation_id"),
-    },
+    redis=worker,
+    pipeline_name="validate_map_annotate_score_set",
+    score_set=item,
+    user=user_data.user,
+    extra_params={"scores_file_key": scores_file_key, ...},
 )
-db.commit()
-
-await worker.enqueue_job("start_pipeline", entrypoint_job_run.id)
 ```
 
-This creates a `Pipeline` with multiple `JobRun` records and `JobDependency` records, then enqueues the pipeline's `start_pipeline` entrypoint in ARQ. The worker coordinates the rest — each job runs after its dependencies complete.
+`enqueue_pipeline_for_score_set` is the single kickoff seam shared by routers and scripts. It fills `score_set_id`, `updater_id` and `correlation_id`, uses `PipelineFactory` to create a `Pipeline` with its `JobRun` and `JobDependency` records, then enqueues the pipeline's `start_pipeline` entrypoint in ARQ. A failed enqueue discards the just-created pipeline records. The worker coordinates the rest — each job runs after its dependencies complete.
 
 ### Job Function Signature
 
@@ -155,18 +153,11 @@ Callers pass only `ctx` and `job_id` when enqueueing. The decorator creates the
 Correlation IDs flow from the API request through the pipeline to each job:
 
 ```python
-# In the router — capture correlation ID from starlette-context
-from mavedb.lib.logging.context import save_to_logging_context, logging_context
+# In the router — save request context; the kickoff seam reads the correlation ID from it
+from mavedb.lib.logging.context import save_to_logging_context
 
 save_to_logging_context({"score_set_urn": urn})
-correlation_id = logging_context().get("correlation_id")
-
-# Pass to pipeline via pipeline_params
-pipeline, entrypoint = PipelineFactory.create_pipeline(
-    db=db,
-    name="validate_map_annotate_score_set",
-    pipeline_params={"correlation_id": correlation_id, ...},
-)
+await enqueue_pipeline_for_score_set(db=db, redis=worker, pipeline_name="validate_map_annotate_score_set", ...)
 ```
 
 Each job retrieves the correlation ID from its `job_params` and uses `job_manager.save_to_context()` for structured logging.

.github/instructions/copilot-instructions.md

@@ -12,8 +12,8 @@ MaveDB is a bioinformatics database API for Multiplex Assays of Variant Effect (
 ### Service Architecture
 - **FastAPI application** — `src/mavedb/server_main.py`, router-based endpoint organization
 - **Background worker** — ARQ/Redis for async processing (variant creation, mapping, publication, annotation)
-- **Multi-container setup** — API server, worker, PostgreSQL, Redis, plus external services (cdot-rest, dcd-mapping, seqrepo)
-- **Docker config** — `docker-compose-dev.yml` (6 services), multi-stage `Dockerfile`
+- **Multi-container setup** — API server, worker, PostgreSQL 15, Redis, moto (local S3), plus external services (cdot-rest, dcd-mapping, seqrepo)
+- **Docker config** — `docker-compose-dev.yml` (9 services), multi-stage `Dockerfile`
 
 ### Core Domain Model
 - **Hierarchical URN system**: ExperimentSet (`urn:mavedb:00000001`) → Experiment (`00000001-a`) → ScoreSet (`00000001-a-1`) → Variant (ScoreSet URN + `#` + number)
@@ -41,8 +41,8 @@ src/mavedb/
 ├── worker/                 # ARQ background worker system
 │   ├── jobs/               # Job function implementations (by category)
 │   │   ├── registry.py     # Central registry of all jobs, cron definitions
-│   │   ├── variant_processing/  # Variant creation and mapping
-│   │   ├── external_services/   # ClinGen, ClinVar, gnomAD, UniProt
+│   │   ├── variant_processing/  # Variant creation, mapping, reverse translation
+│   │   ├── external_services/   # ClinGen, ClinVar, gnomAD, UniProt, VEP
 │   │   ├── pipeline_management/ # Pipeline entrypoint (start_pipeline)
 │   │   └── system/         # Cron maintenance (cleanup stalled jobs)
 │   ├── lib/                # Infrastructure layer
@@ -143,7 +143,7 @@ poetry run python -m mavedb.scripts.<script_name>
 
 - [score_set.py](src/mavedb/models/score_set.py) — Primary data model patterns
 - [score_sets.py](src/mavedb/routers/score_sets.py) — Complex router with worker integration
-- [jobs.py](src/mavedb/worker/jobs.py) — Background processing patterns
+- [registry.py](src/mavedb/worker/jobs/registry.py) — Background job registration
 - [score_set.py](src/mavedb/view_models/score_set.py) — Pydantic model hierarchy
 - [server_main.py](src/mavedb/server_main.py) — App setup and dependency injection
 - [authentication.py](src/mavedb/lib/authentication.py) — Auth patterns
@@ -222,9 +222,10 @@ poetry run python -m mavedb.scripts.<script_name>
 - **Key job types**:
   - `create_variants_for_score_set` - Parse uploaded CSV, create variant records
   - `map_variants_for_score_set` - Map variants via DCD Mapping / VRS
+  - `reverse_translate_variants_for_score_set` - Derive equivalent nucleotide/protein alleles and projection groups via variant-annotation
   - `submit_score_set_mappings_to_car/ldh` - Submit to ClinGen services
   - `cleanup_stalled_jobs` - Cron job for recovering stuck jobs
-- **Enqueueing pipelines**: Routers call `PipelineFactory.create_pipeline()` then `ArqRedis.enqueue_job("start_pipeline", ...)` with the pipeline's entrypoint JobRun ID
+- **Enqueueing pipelines**: Routers and scripts call `enqueue_pipeline_for_score_set()` (`src/mavedb/lib/workflow/kickoff.py`), which creates the pipeline via `PipelineFactory.create_pipeline()` and enqueues its `start_pipeline` entrypoint JobRun
 - **Detailed documentation**: See `src/mavedb/worker/README.md` and `.github/instructions/worker.instructions.md`
 
 ### View Models (Pydantic)

.github/instructions/database.instructions.md

@@ -8,13 +8,12 @@ applyTo: 'src/mavedb/models/**/*.py,alembic/**/*.py'
 ## Model Structure
 
 ### Base Class
-All models use `@as_declarative()` with a shared `Base` class providing:
-- `id`: Integer primary key (auto-increment)
-- `creation_date` / `modification_date`: Auto-managed timestamps
-- `created_by_id` / `modified_by_id`: Foreign keys to users table
+All models subclass the `@as_declarative()` `Base` in `src/mavedb/db/base.py`. It declares no columns; it only defaults `__tablename__` to the lowercased class name. Table models declare their own `id` primary key and set `__tablename__` explicitly; view models map `__table__` to a view instead.
 
 ### Common Columns
 Most resource models include:
+- `creation_date` / `modification_date`: `Date` columns defaulting to today (`modification_date` updates on write)
+- `created_by_id` / `modified_by_id`: Foreign keys to the users table (user-owned resources: experiment sets, experiments, score sets, calibrations, calibration controls, collections)
 - `urn`: Unique string identifier (temporary `tmp:<uuid>` or permanent `urn:mavedb:...`)
 - `private`: Boolean flag for visibility control
 - `approved`: Boolean for moderation status
@@ -50,6 +49,16 @@ class ScoreSetPublicationIdentifierAssociation(Base):
     primary = Column(Boolean, default=False)
 ```
 
+### Valid-Time Versioning (`ValidTime`)
+Rows that change over time mix in `ValidTime` (`src/mavedb/db/mixins.py`) instead of being updated or deleted in place:
+- `valid_from` / `valid_to`: a row is live while `valid_to` is NULL; history is the half-open window `[valid_from, valid_to)`.
+- Reads: filter explicitly with `Model.current` or `Model.live_at(as_of)` (`None` means current; also works on `aliased()` classes). Never assume a query returns only live rows.
+- Replacing a row: `supersede_with` (one row) or `supersede_live_where` (bulk) retire the old rows and stamp the new rows' `valid_from` with the same timestamp, so the handoff has no gap. Never retire and then insert separately.
+- Withdrawing with no successor: `retire` (one row) or `retire_live_where` (bulk UPDATE).
+- `__retire_cascade__`: relationship names whose live `ValidTime` children are retired when the parent is (one level deep). `supersede_live_where` raises on a class that declares it; use `supersede_with` per row.
+- Each table adds a partial unique index over its natural key `WHERE valid_to IS NULL`, so a second live row raises `IntegrityError`.
+- Content-addressed, immutable rows (e.g. alleles) do not use the mixin.
+
 ### Enum Columns
 Use non-native enums stored as strings:
 ```python
@@ -121,4 +130,4 @@ if item.private and not user_has_access(user, item):
 ```
 
 ## Materialized Views
-Used for pre-computed aggregations. Refresh via worker cron jobs defined in `src/mavedb/worker/settings.py`.
+Used for pre-computed aggregations. Refresh via worker cron jobs defined in `src/mavedb/worker/jobs/registry.py`.

.github/instructions/worker.instructions.md

@@ -86,7 +86,7 @@ Always return using factory methods:
 
 ```python
 return JobExecutionOutcome.succeeded(data={"variants_created": count})
-return JobExecutionOutcome.failed(reason="No mapped variants found", data={...})
+return JobExecutionOutcome.failed(reason="No mapping records found", data={...})
 return JobExecutionOutcome.failed(reason="HGVS parse error", failure_category=FailureCategory.DATA_ERROR)
 return JobExecutionOutcome.skipped(data={"reason": "Feature disabled"})
 # For unhandled exceptions: let them propagate — the decorator catches and creates .errored()
@@ -124,11 +124,12 @@ Parameters with `None` values in pipeline definitions are filled at runtime from
 - **Slack safety**: `send_slack_job_failure()` and `send_slack_job_error()` catch their own exceptions internally (logging critical on failure), so Slack outages never interfere with job lifecycle management or error recovery in the decorators.
 - **Stale RUNNING recovery**: `start_job()` accepts RUNNING as a startable status (alongside QUEUED and PENDING). When ARQ re-delivers a job after a worker crash, `start_job()` logs a warning and resets the start timestamp rather than raising `JobTransitionError`.
 - **Concurrency limit**: `max_jobs = 2` in `ArqWorkerSettings` prevents event loop starvation from sync psycopg2 DB calls. With the default `max_jobs=10`, multiple concurrent jobs issuing blocking DB operations can starve the asyncio event loop.
+- **Fast lane**: `start_pipeline` and `create_variants_for_score_set` run on a separate queue (`FAST_LANE_QUEUE_NAME`) consumed by a second worker process (`ArqFastLaneWorkerSettings`), so long mapping jobs can't block new pipelines. Every `enqueue_job` and every `arq.jobs.Job(...)` status/abort call must pass `_queue_name=arq_queue_name(job_run)`.
 
 ## Pipeline Lifecycle (Brief)
 
-1. Router calls `PipelineFactory.create_pipeline()` → creates `Pipeline`, `JobRun`, and `JobDependency` records
-2. Router enqueues the `start_pipeline` entrypoint job in ARQ
+1. Router or script calls `enqueue_pipeline_for_score_set()` (`lib/workflow/kickoff.py`) → `PipelineFactory.create_pipeline()` creates `Pipeline`, `JobRun`, and `JobDependency` records
+2. The same call enqueues the `start_pipeline` entrypoint job in ARQ
 3. `start_pipeline` runs → its `@with_pipeline_management` decorator starts the pipeline and calls `coordinate_pipeline()`
 4. `coordinate_pipeline()` finds PENDING jobs whose dependencies are met → marks them QUEUED → enqueues in ARQ
 5. Each job runs → after completion, its decorator calls `coordinate_pipeline()` again
@@ -181,7 +182,7 @@ Pipeline definitions live in `src/mavedb/lib/workflow/definitions.py`. The `Pipe
 | `lib/managers/pipeline_manager.py` | Pipeline coordination, dependency resolution, job enqueueing |
 | `lib/managers/constants.py` | Status groupings (`TERMINAL_JOB_STATUSES`, `STARTABLE_JOB_STATUSES`, etc.) |
 | `lib/managers/exceptions.py` | Exception hierarchy (`JobStateError`, `PipelineCoordinationError`, etc.) |
-| `settings/worker.py` | `ArqWorkerSettings` class (ARQ worker configuration) |
+| `settings/worker.py` | `ArqWorkerSettings` (main) and `ArqFastLaneWorkerSettings` classes (ARQ worker configuration) |
 | `settings/lifecycle.py` | Worker startup/shutdown hooks, `standalone_ctx()` |
 | `src/mavedb/lib/workflow/definitions.py` | Pipeline and job definitions (`PIPELINE_DEFINITIONS`) |
 | `src/mavedb/lib/workflow/pipeline_factory.py` | Creates Pipeline + JobRun + JobDependency records |

Makefile

@@ -0,0 +1,23 @@
+VENV := .venv/bin
+
+.DEFAULT_GOAL := help
+
+.PHONY: help
+help:
+	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "  %-12s %s\n", $$1, $$2}'
+
+.PHONY: dev
+dev: ## Install deps including local editable variant-annotation
+	poetry install --extras server
+
+.PHONY: test
+test: ## Run the test suite
+	$(VENV)/pytest tests/
+
+.PHONY: lint
+lint: ## Check code with ruff
+	$(VENV)/ruff check src/ tests/
+
+.PHONY: format
+format: ## Format code with ruff
+	$(VENV)/ruff format src/ tests/

README.md

@@ -67,9 +67,10 @@ To stop the application when it is running as a daemon, run
 docker-compose -f docker-compose-local.yml down
 ```
 
-`docker-compose-local.yml` configures four containers: one for the API server, one for the PostgreSQL database, one for the
-worker node and one for the Redis cache which acts as the job queue for the worker node. The worker node stores data in a Docker
-volume named `mavedb-redis` and the database stores data in a Docker volume named `mavedb-data`. Both these volumes will persist
+`docker-compose-local.yml` configures five containers: one for the API server, one for the PostgreSQL 15 database, two worker
+nodes and one for the Redis cache which acts as the job queue for the workers. The fast worker runs only pipeline starts and
+variant creation, on its own queue, so new score sets get variants while mapping jobs occupy the main worker. The worker node stores data in a Docker
+volume named `mavedb-redis` and the database stores data in a Docker volume named `mavedb-data-15`. Both these volumes will persist
 after running `docker-compose down`.
 
 **Notes**
@@ -88,6 +89,21 @@ after running `docker-compose down`.
     The database username and password should be edited for production deployments. `NCBI_API_KEY` will be removed in
     the future. **TODO** Move these to an .env file.
 
+2. Optional database and CSV export tuning variables. Times are in seconds, and setting a timeout or `CSV_BUILD_SLOTS`
+   to `0` disables it. All of them apply only to the API server; the worker, scripts and migrations run without them.
+
+    | Variable | Default | Purpose |
+    |---|---|---|
+    | `DB_POOL_SIZE` / `DB_MAX_OVERFLOW` | 5 / 10 | Persistent and burst connections per process |
+    | `DB_POOL_TIMEOUT` | 10 | Longest wait for a free pooled connection before the request returns 503 |
+    | `DB_STATEMENT_TIMEOUT_SECONDS` | 30 | Longest single statement before the request returns 503 |
+    | `DB_CSV_STATEMENT_TIMEOUT_SECONDS` | 45 | The same limit for the score set CSV routes |
+    | `DB_LOCK_TIMEOUT_SECONDS` | 5 | Longest wait for a row or table lock before the request returns 503 |
+    | `DB_IDLE_IN_TRANSACTION_TIMEOUT_SECONDS` | 60 | Longest idle gap inside an open transaction before Postgres ends the session |
+    | `DB_SLOW_STATEMENT_SECONDS` | 5 | Statements slower than this are logged with their route |
+    | `CSV_BUILD_SLOTS` | 2 | Score set CSV builds of `CSV_BUILD_MIN_ROWS` or more rows each process runs at once |
+    | `CSV_BUILD_MIN_ROWS` | 10000 | Smaller builds skip the cap. A large build waits up to 10s for a slot, then returns 503 with `Retry-After` |
+
 ### Running the API server in Docker for development
 
 A similar procedure can be followed to run the API server in development mode on your local machine. There are a couple