You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Page score set downloads without walking every skipped row #901
Scripts page through large score sets with start and limit, and each page makes the database read and discard every row before start. On a 648,000-variant score set a page near the end reads more than half a million full rows to return a few thousand. This issue finds each page's first row through the index alone and then reads only the page's own rows, with no change to the API.
Problem
fetch_variant_csv_data (lib/csv/fetch.py) applies start as SQL OFFSET. With the variant-number index the rows come out in order, but OFFSET still fetches each skipped row from the table, including its data column, before dropping it.
Scope
Select the page's variant ids in a subquery that reads only the index: SELECT id FROM variants WHERE scoreset_id = :id ORDER BY <variant number>, id OFFSET :start LIMIT :limit. The index on (scoreset_id, variant number, id) covers it.
Fetch the full rows, and their joined mappings, for those ids only.
Apply it wherever start or limit is set; unpaged requests keep today's query.
start and limit keep their meaning, and the output doesn't change.
Acceptance criteria
EXPLAIN of the id subquery shows an index-only scan.
In staging, EXPLAIN ANALYZE of the queries for start=600000&limit=50000 on urn:mavedb:00000053-a-1 totals under 1 second.
Paged output is byte-for-byte identical to the current output for the same start and limit.
Background
In the week to 2026-09-29, deep pages of urn:mavedb:00000053-a-1 took 5 to 23 minutes each, from several unrelated clients. Most of that was the unindexed sort, which the variant-number index removes; this issue removes the rest.
Paging by variant_number > start would be simpler, but it assumes a score set's variant numbers are exactly 1 to N with no gaps, which nothing enforces.
Activity