fix(aw-sync): skip duplicate device_id folders that truncate history on pull

[TimeToBuildBob]

Problem

When one physical device has written under two hostname folders in the sync directory, pull_all imported both. Provenance is the bucket hostname (or $aw.sync.origin), so both folders land in the same destination bucket. Resume-from-newest then silently drops everything older than the first folder's newest event.

Real-world case from #683: POCO F8 Ultra/ (9.4 MB, Jul 2026) vs poco_f8_ultra/ (273 MB, 2021–2026). If the small folder is walked first, ~1M events never import and the log says ✓ Already up to date!.

The duplicate folders come from ActivityWatch/aw-android#272 (hostname sanitization without migration). Any hostname change or folder rename reproduces it.

What changed

1. pull_all deduplicates by device_id. Discover {hostname}/{device_id}/*.db, group by device_id, keep the largest file, warn! the skipped folders. This is the hazard removal from the issue.
2. find_remotes_nonlocal applies the same collapse. So a later 3-level walker (aw-sync: daemon (the default subcommand) never pulls — two incompatible sync-folder layouts #682) does not reintroduce the bug on the daemon path.
3. Loud warning in sync_one. If a resume-from-newest pull reports up-to-date but the source has more events than the destination, log that older history was not imported and that recovering it requires deleting the destination bucket and re-pulling.

Not in this PR

• Making device_id the provenance key instead of hostname (suggested fix 2) — that's the aw-sync: move sync provenance out of the bucket ID into trustworthy bucket metadata #649 provenance work, breaking for existing synced-from bucket IDs.
• Backfilling older-than-resume events automatically (suggested fix 3, the recovery half). The warning makes that case visible; the actual backfill is a separate change.

Already-truncated destination buckets are not repaired by this PR. Delete the dest bucket and re-pull after this lands.

Testing

• cargo test -p aw-sync --offline: 17 lib tests + 10 tests/sync.rs + 3 tests/sync_roundtrip.rs
• Unit tests cover listing, largest-per-device_id selection independent of discovery order, and unique device_ids kept

[TimeToBuildBob]

Heads-up on merge order: this PR and #685 conflict in aw-sync/src/util.rs. I checked by merging the two heads locally.

The overlap is in the design, not just the text:

• fix(aw-sync): make default daemon use host-layout pull/push #685 (for aw-sync: daemon (the default subcommand) never pulls — two incompatible sync-folder layouts #682) folds find_remotes/get_remotes into one shared walker that sees both the 2-level and 3-level layouts.
• This PR (for aw-sync: duplicate folders for one device_id silently truncate history on pull #683) adds a separate walker, list_remote_dbs (it returns hostname, device_id and size), and marks get_remotes as dead_code.

Suggested order: merge #685 first, since it is the smaller layout fix and CI is green. Then rebase this PR so list_remote_dbs and the device_id dedupe sit on top of #685's single walker. That avoids ending up with two directory walkers. I'll do the rebase once #685 lands.

[codecov]

Codecov Report

❌ Patch coverage is 76.82927% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.31%. Comparing base (656f3c9) to head (f191c5f).
⚠️ Report is 113 commits behind head on master.

Files with missing lines	Patch %	Lines
aw-sync/src/sync_wrapper.rs	0.00%	15 Missing ⚠️
aw-sync/src/util.rs	95.23%	3 Missing ⚠️
aw-sync/src/sync.rs	75.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master     #686       +/-   ##
===========================================
+ Coverage   70.81%   81.31%   +10.49%     
===========================================
  Files          51       71       +20     
  Lines        2916     6460     +3544     
===========================================
+ Hits         2065     5253     +3188     
- Misses        851     1207      +356     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[TimeToBuildBob]

CI-green and mergeable — waiting only on a maintainer click.

This PR is ready to merge, but the bot has pull-only access to this repo and can't self-merge — surfacing it here so it isn't lost. The monitoring loop will stop re-flagging it now that this note is posted.

[ErikBjare]

@TimeToBuildBob This is the strongest of the three, and I think it should merge first rather than after #685.

RemoteDb { hostname, device_id, path, size } is exactly the typed-peer abstraction #684 item 5 asked for — one walk producing structured peers instead of Vec<PathBuf> that every caller re-derives meaning from. #685 and #687 both add their own walkers (collect_db_files/hostname_from_db, and scan_sync_dir/classify_top_dir respectively); if this lands first, both can build on RemoteDb instead, which removes the duplication and the util.rs merge conflict that #685 currently has with each of them.

select_remote_dbs_keeps_largest_per_device_id reversing the discovery order before selecting is the right test for this bug specifically — the trigger in #683 was fs::read_dir ordering, so order-independence is the property that actually needs proving, not just "picks the big one".

The sync_one warning is well placed. Worth noting for anyone reading later: it fires on an already-damaged destination, so it is a detection mechanism, not a repair one — as your PR description says, recovery still means deleting the bucket and re-pulling. On the setup in #683 that is a 273 MB / ~1M event re-import, which is fine but not instant.

Two smaller notes:

1. list_remote_dbs is 3-level-only, so once pull_all stops going through get_remotes(), legacy 2-level root dbs are no longer pull candidates. That is the correct outcome — the 1.19 GB root orphan in aw-sync: daemon (the default subcommand) never pulls — two incompatible sync-folder layouts #682 should never be pulled by a peer — but it is a real behaviour change relative to fix(aw-sync): make default daemon use host-layout pull/push #685's broadened walker, and the two PRs disagree about it. Please make whichever wins explicit in a comment.

2. This PR removes the only caller of get_remotes() (sync_wrapper.rs:8). fix(aw-sync): make default daemon use host-layout pull/push #685 concurrently rewrites get_remotes() fairly extensively. One of those two is wasted work — another reason to sequence this one first and rebase fix(aw-sync): make default daemon use host-layout pull/push #685 on top.

[ErikBjare]

Targeting correction to my note on #685: the dead get_remotes trio should be removed here, not there.

get_remotes becomes unreachable in 4c9fac6 itself — the moment pull_all switches to list_remote_dbs, its last caller is gone. So it is dead in #686 standalone, and since #687 is also based on 4c9fac6 it inherits the same three dead functions (util.rs:274, 290, 402 at that PR's head). Deleting in #685 would leave whichever of the other two merges first carrying unreachable code.

Also worth dropping the // TODO: share logic with find_remotes and find_remotes_nonlocal that sits just above it — that is what this stack just did.

Separately, the #687 rework looks right: remote_db_to_entry means status now classifies the same RemoteDb list pull_all consumes rather than re-walking, which is what makes its output trustworthy as a preview of what a pull would actually do. All three now merge clean in any pairing.

[TimeToBuildBob]

@ErikBjare Dropped the trio here: get_remotes, contains_db_file, contains_subdir_with_db_file, plus the stale TODO.

Two walkers left, with a comment saying why they differ:

• list_remote_dbs + select_remote_dbs_by_device_id — pull_all, 3-level
• find_remotes + find_remotes_nonlocal — advanced sync_run, relative to a given root

f191c5f on this PR. #685 and #687 restacked onto it so they do not reintroduce the dead functions.