control/crawler: examine sidecar files in parallel and in the background

[hennikul]

Fixes #21849.

The startup crawl asks the filesystem about every image in the library before the user
interface appears — an existence check, a stat() of the xmp, and four probes for
.txt/.TXT/.wav/.WAV — serially, and startup blocks until it has finished. It is
dominated by filesystem latency rather than cpu time, so on a library held on network
storage it dominates startup. On mine (84,924 images across 546 film rolls, on an NFS4 mount
of a NAS) that is 506,449 filesystem calls taking ~58 s from cold and ~12 s with warm caches,
of which only ~3 s is cpu. It reports nothing, every time.

This changes both what the crawl asks the filesystem and when it runs. There are three
commits and they are independent of each other.

What the crawl does

A directory at a time, instead of a probe per image. Suggested by @ralfbrown. Each
directory is listed once and the questions are answered from that listing; the only call
that still has to reach the filesystem per image is the stat() of an xmp that is really
there, for its timestamp. Everything else becomes a hash lookup. That takes the library from
506,449 filesystem calls to 84,851.

This fits because a film roll is exactly one directory — images.filename never contains a
path separator, and on my library it is 546 film rolls to 546 distinct folders — so the
images of a film roll are collected as one contiguous run and examined against a single
listing of it.

In parallel. dt_control_crawler_run() is split into three phases: collect the rows from
the database, examine the filesystem from a pool of worker threads, then apply the results.
The workers touch neither the database nor the gui and each writes only its own array slot,
so no locking is needed. The result list is assembled single threaded afterwards, walking the
items in collection order, so the reported list and the resulting database state are
identical regardless of how many threads are used. That is what makes this checkable
against the old behaviour rather than merely plausible.

Because the work is latency bound, the pool deliberately oversubscribes rather than scaling
with the core count. crawler_threads defaults to 16; setting it to 1 restores a serial
examination.

The crawl's existing semantics are kept faithfully, including the slightly surprising one
where an image whose xmp is absent skips the .txt/.wav check entirely (the original
continue).

When the crawl runs

In the background, newest film roll first. The crawl runs from a DT_JOB_QUEUE_SYSTEM_BG
job, one film roll at a time, ordered by film_rolls.access_timestamp. Startup does not wait
for it. run_crawler_in_background defaults to true; false restores the blocking startup
crawl.

But it waits for the film roll you open. @wpferguson pointed out that the blocking crawl
also protects against editing an image whose xmp was updated on another machine, and that is
right. The consequence is worse than a stale thumbnail: darktable does not re-read an xmp
when an image is opened, and dt_image_write_sidecar_file() overwrites whatever is on disk
without comparing timestamps. Editing an image before its film roll had been examined would
discard the other machine's changes — and because that write also updates write_timestamp,
the crawler would no longer report the file afterwards, so the loss would be silent.

So a film roll is examined on demand when it is opened, and its findings are reported
straight away rather than with the rest at the end of the crawl, so the user is told before
they can edit anything. Waiting for one film roll costs a median of 0.000 s and 0.09 s for
the worst of my 546, against ~58 s for the whole library.

The collect module does not open film rolls through dt_film_open(), so the collection is
followed as well, and every film roll a collection covers is waited for rather than a sample
of them, since the user can scroll to any image in it. Worst case — a collection spanning the
whole library, opened immediately at startup — that costs what the blocking crawl costs
today, and far less in every other case.

Only one film roll is examined at a time, by either the background job or a caller waiting on
a specific roll, which also keeps the crawler's database transactions from interleaving.

The conflict dialog is shown non-modally when it comes from the background crawl, since it
can appear long after startup; the synchronous path still shows it modally as before.

The queue is deliberately not persisted between sessions. An xmp can change while
darktable is not running, so every session still examines every image — this takes the work
off the critical path rather than skipping it.

Results

On the library above, cold:

	before	after
filesystem calls	506,449	84,851
startup blocked by the crawl	~58 s	none
crawl begins	before the gui appears	~1.8 s after launch, gui already up
time the crawl itself takes	~58 s, blocking	~2.8 s, in the background
whole library in one pass, 16 threads	—	~3.0 s
waiting for one film roll when opened	—	median 0.000 s, worst of 546 0.09 s

Testing

Verified against the original per-image implementation on the same library:

• the crawler's output for the whole library is identical line for line and in the same
  order, both with no updated sidecars (619 lines) and with 617 seeded ones (1,236 lines)
• the resulting images.flags column is identical across all 84,924 rows, in serial,
  parallel and background modes, and for both the per-image and per-directory examination
• the background crawl covers all 546 film rolls exactly once, 84,924 images, with no gaps
  and no duplicates
• with conflicts seeded in the film roll that is open at startup, they are reported 1.6 s in,
  on demand, rather than at the end of the crawl
• opening an old film roll moves it to the head: with a 2022 roll selected at startup it is
  examined before the newest ones

My library contains no genuine conflicts, so the conflict path was exercised by backdating
write_timestamp in a copy of the database rather than by touching any file on disk.

Quitting during the crawl behaves as it does for any other background job: if it is still
running you get the existing "darktable will be locked until background work has been done"
notice and the existing drain wait in dt_control_quit(), then it shuts down. The abort flag
is checked per image, and dt_control_crawler_stop(TRUE) in dt_cleanup() mirrors the
existing dt_stop_backthumbs_crawler(TRUE) placement.

Deliberate differences in behaviour

• An image counts as present when its directory lists it, whereas g_file_test() follows
  symlinks — so a broken symlink now counts as present rather than missing.
• The .txt/.wav lookups match the names the directory actually holds, so on a case
  insensitive filesystem a spelling other than the four checked is no longer found by
  accident. On Linux nothing changes.
• Listing a directory reads every entry, so a directory holding a great many unrelated files
  beside very few images is slower than probing would have been. I did not think that worth
  special casing.

I have not tried this on Windows or macOS. The Windows _wstati64 path is carried over
unchanged, but it now runs on a worker thread, so it is worth a look from someone who can
test there.

---

Disclosure: designed, written and tested with AI assistance (Claude Code with Claude Opus 5).
Verified against current master and builds clean with no new warnings.

[wpferguson]

The current crawler blocks until the check is done to make sure you don't load an "out of date" image.

For example

• I have my images on a NAS
• I have 2 different computers that run darktable and mount the images from the NAS
• I edit an image on computer 1 and save it
• I go to computer 2 and open the same image while the crawler is running and make another change to it, which is a change to the image state before the computer 1 change since I haven't read the updated xmp, and then I save the change which overwrites the xmp being scanned.

Blocking prevents this scenario.

I think you still need to block until the scan is complete, or at least complete for the film roll that is open.

[hennikul]

@wpferguson you are right, and my answer above was wrong — sorry. I checked what the code
actually does rather than what I assumed it did, and it is worse than a stale thumbnail.

Three things, all verifiable in current master:

• Opening an image does not re-read its xmp. dt_exif_xmp_read() is only called on import
  and by this dialog's own "reload" action (src/common/history.c:190); the darkroom works
  from database state.
• dt_image_write_sidecar_file() (src/common/image.c:2989) calls dt_exif_xmp_write()
  unconditionally — it never compares against the on-disk mtime.
• That same function then sets write_timestamp = STRFTIME('%s','now'), so after the
  overwrite write_timestamp + skew < xmp_mtime no longer holds.

So in your scenario the edit on computer 2 does not merely win — it overwrites computer 1's
sidecar and erases the evidence, because the crawler will no longer flag the file
afterwards. Nothing tells the user. My branch also had a second window I had missed: even
for a film roll that had already been examined, the findings were only reported once the
whole crawl finished, so the file could be clobbered before the dialog appeared.

Nothing in the background crawl ever applied or refreshed anything — it only collected
findings for the dialog. My "it will update it live" was simply not what the code did.

What I changed

Pushed as a second commit: wait for the film roll being opened, rather than for the whole
library.

A film roll is now examined on demand when it is opened, and its findings are reported
immediately rather than at the end of the crawl, so the user is told before they can edit
anything. Because the collect module does not go through dt_film_open(), the collection is
followed as well, and every film roll a collection covers is waited for rather than a
sample, since the user can scroll to any image in it.

The reason this is worth doing rather than just restoring the full block is the cost. From
546 film rolls on my library (84,924 images on an NFS-mounted NAS):

mean per film roll	0.021 s
slowest of the 546	0.12 s
largest roll (2,692 images)	0.06 s
whole library	~58 s

So the guarantee you are asking for costs a few tens of milliseconds at the point of opening
a film roll, instead of ~58 s before the user sees anything. The worst case — a collection
spanning the whole library, opened immediately at startup — costs what the blocking crawl
costs today, and every other case costs far less.

Only one film roll is examined at a time now, by either the background job or a caller
waiting on a specific roll, which also stops the crawler's database transactions from
interleaving.

Testing

With 2,692 conflicts seeded in the film roll that is open at startup, they are reported
1.6 s in, on demand, before the crawl has got anywhere near the rest of the library — rather
than at 13 s when it finishes. Every film roll is still examined exactly once, 84,924 images
in total with no gaps or duplicates, and the resulting images.flags column is identical to
the serial blocking crawl across all 84,924 rows.

---

Disclosure: as before, written and tested with AI assistance (Claude Code with Claude Opus 5).

[ralfbrown]

The description sounds good, though I haven't had a chance to look at the code. It's similar to what I've had in mind to try for a while now to speed up the crawler, except I wasn't thinking of making it run in the background:

• collect all directories from the library
• do a single readdir()-style pass over all files in a directory, collecting all sidecars (including .txt and.wav) and doing a stat on the xmps - store all that info in a hashmap or similar
• finally, iterate over all image ids and extract the relevant stored entries, comparing against current database state

The second phase can be parallelized per directory, the third per image. A blocking crawl can still update the progress bar and estimated time as each directory completes in phase 2. Phase 3 would probably be fast enough not to bother with a progress bar.

Keeping the same order in the user dialog is useful for checking correctness of the new version, but not actually necessary.

[hennikul]

@ralfbrown that is a better idea than what I had, and I have implemented it — pushed as a
third commit.

Before writing it I measured the two strategies against each other on my library, serially
and with a cold cache each time, so the comparison is only about the number of calls:

	filesystem calls	time
probe per image (what the crawler does today)	506,449	36.9 s
list each directory (yours)	84,851	13.4 s

Worth noting where the win actually comes from: it is not the directory listings, which are
nearly free, but the four .txt/.wav probes and the existence check per image becoming
hash lookups. The stat() of each xmp that really exists survives and still dominates —
84,305 of those 84,851 calls. So it is a 6× reduction in calls for a 2.8× reduction in time.

It fits the existing structure better than I expected, because a film roll is exactly one
directory — filename never contains a path separator, and on my library it is 546 film
rolls to 546 distinct folders. So the images of a film roll are already collected as one
contiguous run, and that run is now examined against a single listing of its directory. The
unit your phase 2 works in is the unit the background crawl was already scheduling in, so
the scheduling layer did not change at all; only the examination underneath it did.

Your three phases and the three the crawl was already split into line up one to one, which
made this a replacement of the middle one rather than a rewrite.

Combined with the parallelism and the background scheduling, on ~85k images across 546 film
rolls on an NFS mount:

	probe per image	list per directory
whole library in one pass, 16 threads, cold	2.92 s	2.96 s
background crawl, film roll at a time, cold	11.5 s	2.8 s
waiting for one film roll, median of 546	0.020 s	0.000 s
waiting for one film roll, worst of 546	0.12 s	0.09 s

The first row is the honest caveat: for a single parallel pass over the whole library this
changes nothing, because 16 concurrent workers were already hiding the call count behind
overlapping latency. Where it pays off is everywhere the work is chopped up — the film roll
at a time background crawl, where each roll is too small to hide much, drops by 4×.

The last two rows are the ones that matter for @wpferguson's point: the wait when you open a
film roll is now unmeasurable for a typical roll, and 0.09 s for the worst of mine.

On ordering — you are right that it is not necessary, but I kept it, because it is what let
me check this. The crawler's output for the whole library is identical line for line and in
the same order as the per-image version, both with no updated sidecars and with 617 seeded
ones, and images.flags is identical across all 84,924 rows. Phase 3 is a serial pass
anyway, so the property is free to keep and it makes the next change to this code checkable
the same way.

Two deliberate differences in what gets asked of the filesystem, both called out in the
commit message:

• an image counts as present when the directory lists it, whereas g_file_test() follows
  symlinks — so a broken symlink now counts as present rather than missing;
• the .txt/.wav lookups match the names the directory actually holds, so on a case
  insensitive filesystem a spelling other than the four checked is no longer found by
  accident. On Linux nothing changes.

The one case where this is slower than probing is a directory holding a great many files
next to very few images, since the listing reads every entry. I did not think that worth
special casing, but say if you disagree.

---

Disclosure: as before, written and tested with AI assistance (Claude Code with Claude Opus 5).