[WIP] Parallel linclust - #1124

Draft

bbuschkaemper wants to merge 29 commits into

soedinglab:masterfrom

bbuschkaemper:parallel-linclust

Commits on Jul 29, 2026

Add shared-filesystem parallel linclust foundations.
bbuschkaemper
committed
2cf26ce
Add distributed k-mer shuffle.
bbuschkaemper
committed
53efff4
Add distributed reduce.
bbuschkaemper
committed
ec7c139

Commits on Jul 30, 2026

Add distributed alignment.
bbuschkaemper
committed
67fce30
Add distributed align2clust.
bbuschkaemper
committed
21a07be
Add k-mer extraction waves (to manage within scratch budget).
bbuschkaemper
committed
fa62a16
Add clusterstsv conversion.
bbuschkaemper
committed
0842bb4
Seed the pass-2 filter gate with the pair's diagonal.
bbuschkaemper
committed
1962152

Commits on Jul 31, 2026

Various smaller bugfixes.
bbuschkaemper
committed
9cbe59c

Commits on Aug 3, 2026

Fix handling of duplicate edges across blocks.
bbuschkaemper
committed
2c31978
Fix invalid inline documentation.
bbuschkaemper
committed
bc53556
Changed parallel linclust param from mpi-runner to runner.
bbuschkaemper
committed
3742f78
Fix candidate edges header bug.
bbuschkaemper
committed
01d70bf
Update in-line comments.
bbuschkaemper
committed
8cb0a4c
Fix name spelling.
bbuschkaemper
committed
ac83fe3
Fix cicd for greedycluster and kmer partition test.
bbuschkaemper
committed
80aded2
Fix resume crashes. Fix smaller bugs.
bbuschkaemper
committed
3f1a0aa
Revert unnecessary fsync fix (out of scope, requires too much resources).
bbuschkaemper
committed
e957f2d
Fix silent error in merge cluster parallel. Fix memory budget over-allocation during translate keys.
bbuschkaemper
committed
0b78481

Commits on Aug 4, 2026

More fixes.
bbuschkaemper
committed
f5b53de
More fixes.
bbuschkaemper
committed
8fd9702

Commits on Aug 5, 2026

Add varint codec and length-rank table
bbuschkaemper
committed
db1f30e
Pack the k-mer and candidate-edge bucket formats
bbuschkaemper
committed
6f0cfcd
Fix quadratic growth when decoding packed blocks
bbuschkaemper
committed
e612926
Group oversized partitions in k-mer slices
bbuschkaemper
committed
648e438
Wake the work-queue heartbeat on a condition variable
bbuschkaemper
committed
a014722
Add packed-edge codec tests and align-bucket skew reporting
bbuschkaemper
committed
76463df

Commits on Aug 7, 2026

Size work units for the allocation, not just for memory
bbuschkaemper
committed
f0298fc
Correct the work-unit sizing change
bbuschkaemper
committed
f28e4b2