borg2 initial chunk merge is very slow

[pennae]

borg2.0.0b14 seems to very slow during the initial chunk index processing:

2025-02-02 15:08:18.631320500 borgstore.backends.errors.ObjectNotFound: cache/chunks
2025-02-02 15:08:18.631320500 
2025-02-02 15:08:18.648099500 cached chunk indexes: ['39267a11a8ea6a4c', 'da243a8927c10035']
2025-02-02 15:08:18.648261500 trying to load cache/chunks.39267a11a8ea6a4c from the repo...
2025-02-02 15:08:18.668600500 cache/chunks.39267a11a8ea6a4c is valid.
2025-02-02 15:08:18.669315500 cached chunk index 39267a11a8ea6a4c gets merged...
2025-02-02 15:08:18.671682500 trying to load cache/chunks.da243a8927c10035 from the repo...
2025-02-02 15:08:21.239523500 cache/chunks.da243a8927c10035 is valid.
2025-02-02 15:08:21.954073500 cached chunk index da243a8927c10035 gets merged...
2025-02-02 15:08:51.544026500 caching 1476533 new chunks.
2025-02-02 15:08:51.610601500 cached chunk indexes: ['39267a11a8ea6a4c', 'da243a8927c10035']
2025-02-02 15:08:51.610614500 caching chunks index as cache/chunks.7e288068a1b796fa in repository...
2025-02-02 15:09:01.386515500 cached chunk indexes deleted: {'39267a11a8ea6a4c', 'da243a8927c10035'}

the repo used here has 120 archives with ~1.4M objects, and the initial chunk index merge takes 30 seconds. on a large repo with 430 archives and over 2M objects it takes over a minute. during this time the progress bar of a non---debug run is stuck at zero and no indication what's happening, but one core pegged at 100%. total backup time of an unchanged source more than doubled compared to borg1 (but the .cache space it needs is two orders of magnitude smaller, which is greatly appreciated!)

is this a tuning problem that we haven't found the documentation for, or is the borg2 setup just legitimately that high?

[ThomasWaldmann]

The log looks like it works as intended. As we are talking about the chunks index (which is for all the chunks in the repo), the time does not depend on the archives count, but on the chunks count.

In 2.0.0b14 I added some partial/incremental chunks index tuning, so borg only saves the NEW chunks it created in a borg create run instead of the full chunks index (as in previous betas). It also does that from time to time within a backup to save the progress, so that we have a somehow current index in case of the backup being interrupted. On the next startup, it will first merge all the chunks indexes it finds (usually 1 full + 1 incremental) to create an updated full chunks index.

Compared to borg 1.x, these chunk indexes are stored in the repo, so depending on the repo speed, that might be slower than the chunks.archives.d/ local cache of borg 1.x (which grew rather big for many archives). But the advantage with the new method is that for multiple clients, the current full chunks index can be computed rather easily, without having to iterate over all archives or over all cached per-archive chunk indexes.

That's a lot of new code, so maybe some more tuning is possible after analyzing the issue.

[pennae]

that much we understood, and it's working really well. all index sizes are much smaller, which is really awesome to us who've been bit by our .cache/borg exploding in size a few times. i'm wondering whether a more dynamic hot/cold split of the index would help, doubling the number of index lookups but reducing the per-operation merge cost. it seems at least plausible given how much pruning helped, and a post-operation index merge of hot into cold once the hot index is large enough is probably still lots better than the segment compaction of borg1?

[ThomasWaldmann]

Not sure how hot/cold split of the index could work, so this is currently not actionable from my perspective.