Skip to content

Segment membership: Bitmap sharding and parallel backfill #7419

Closed as not planned

Description

@khvn26

Together with Flagsmith/flagsmith-engine#299, this unlocks:

  • read-latency SLO tightening for 100M+ environments
  • backfill SLO bucket tightening for 10M+ environments

Scope

  • Migrate AtomBitmap from OneToOneField to (atom, shard_id) unique. Background job to split existing blobs by ord range.
  • Read path fetches all shards for an atom in one query and unions in memory.
  • process_segment_canonical_changed becomes a fan-out task spawning one child task per shard.

Acceptance criteria

  • Per-shard blob caps at ~128 KB at the 870M-identity scale.
  • Read latency p95 enters the "≤ 10M" SLO envelope even for the 870M env after sharding.
  • Backfill of the 870M env completes in less than 24h on N workers (N TBD by review).

Activity

  1. added
    apiIssue related to the REST API
    on May 4, 2026
  2. changed the title [-]Bitmap sharding and parallel backfill[/-] [+]Segment membership: Bitmap sharding and parallel backfill[/+] on May 4, 2026
  3. khvn26 commented on May 9, 2026

    @khvn26
    MemberAuthor

    Closing as not planned. The backlog epic #7416 has been reshaped against the columnar design (RFC: Segment Membership Inspection (columnar) (private to Flagsmith team)). This issue was specific to the prior bitmap design and no longer applies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    apiIssue related to the REST API

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions