Skip to content

Page score set downloads without walking every skipped row #901

Open

Description

@bencap

Summary

Scripts page through large score sets with start and limit, and each page makes the database read and discard every row before start. On a 648,000-variant score set a page near the end reads more than half a million full rows to return a few thousand. This issue finds each page's first row through the index alone and then reads only the page's own rows, with no change to the API.

Problem

fetch_variant_csv_data (lib/csv/fetch.py) applies start as SQL OFFSET. With the variant-number index the rows come out in order, but OFFSET still fetches each skipped row from the table, including its data column, before dropping it.

Scope

  • Select the page's variant ids in a subquery that reads only the index: SELECT id FROM variants WHERE scoreset_id = :id ORDER BY <variant number>, id OFFSET :start LIMIT :limit. The index on (scoreset_id, variant number, id) covers it.
  • Fetch the full rows, and their joined mappings, for those ids only.
  • Apply it wherever start or limit is set; unpaged requests keep today's query.

start and limit keep their meaning, and the output doesn't change.

Acceptance criteria

  • EXPLAIN of the id subquery shows an index-only scan.
  • In staging, EXPLAIN ANALYZE of the queries for start=600000&limit=50000 on urn:mavedb:00000053-a-1 totals under 1 second.
  • Paged output is byte-for-byte identical to the current output for the same start and limit.
Background

In the week to 2026-09-29, deep pages of urn:mavedb:00000053-a-1 took 5 to 23 minutes each, from several unrelated clients. Most of that was the unindexed sort, which the variant-number index removes; this issue removes the rest.

Paging by variant_number > start would be simpler, but it assumes a score set's variant numbers are exactly 1 to N with no gaps, which nothing enforces.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    app: backendTask implementation touches the backend

    Type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions