borg2: prune dies on non-existing archive

[tve]

When I run borg prune (borg v2.0.0 b20) I get an error:

borg prune --show-rc -v --list -a daily-full -a host:tekke --keep-within 2d -w 4 -m 6 -y 50
Connected (version 2.0, client OpenSSH_10.0p2)
Authentication (publickey) successful!
[chan 0] Opened sftp connection (server version 3)
Keeping archive (rule: within #1):           tekke         daily-full 2026-02-26 17:27:16 
Keeping archive (rule: within #2):           tekke         daily-full 2026-02-25 17:45:14 
[chan 0] sftp session closed.
Archive b'\xc0\xad\xef\xd8Z\xeauV?\xca5\xa4\xc6(\xf7\x8f\x827"\r\\c\x90\xfe/*\xff(D\xfe\xd9`' does not exist
terminating with error status, rc 31

If I run a borg check there is no problem reported:

borg check --show-rc --archives-only -v
Connected (version 2.0, client OpenSSH_10.0p2)
Authentication (publickey) successful!
[chan 0] Opened sftp connection (server version 3)
Starting archive consistency check...
Analyzing archive daily-full 2026-02-22 17:01:01+00:00 25fb4049931884edfba23f31fb95eaf595987f2c521d3fe11207c818adc6ce38 (1/4)
Analyzing archive daily-full 2026-02-22 17:01:01.359832+00:00 2ca9d22f35cca98972a1cd12909a9e8482bb08833706324133077319521d9eb0 (2/4)
Analyzing archive daily-full 2026-02-25 17:45:14+00:00 f5fc4e4c5b583bcbc4790e864980b80771db1a562d58f741501d9e5ead227862 (3/4)
Analyzing archive daily-full 2026-02-26 17:27:16+00:00 5559c1305af68081fa0e542ecb9440cf6c5e7b7150e61052267234b14097dbb5 (4/4)
Archive consistency check complete, no problems found.
[chan 0] sftp session closed.
terminating with success status, rc 0

Suggestions?

[ThomasWaldmann]

Strange, I'll have a look.

That check run does not show an archive with ID c0adefd8.... (see error msg).

Can you do a ls -lR on the repository directory while this problem can still be reproduced?

Please shorten the part where it dives into the data/ subdirectory, except the c0/ad/ subdir in there.

[tve]

ls -lR: ls.txt

I am also running a full borg check but it will take a few more hours to complete.

[ThomasWaldmann]

The important parts:

./archives:
...
-rw-rw-r-- 1 admin admin 0 Feb 23 09:32 25fb4049931884edfba23f31fb95eaf595987f2c521d3fe11207c818adc6ce38
-rw-rw-r-- 1 admin admin 0 Feb 23 15:55 2ca9d22f35cca98972a1cd12909a9e8482bb08833706324133077319521d9eb0
-rw-rw-r-- 1 admin admin 0 Feb 27 09:10 5559c1305af68081fa0e542ecb9440cf6c5e7b7150e61052267234b14097dbb5
-rw-rw-r-- 1 admin admin 0 Feb 25 10:46 c0adefd85aea75563fca35a4c628f78f8237220d5c6390fe2f2aff2844fed960.del  <---
-rw-rw-r-- 1 admin admin 0 Feb 26 10:11 f5fc4e4c5b583bcbc4790e864980b80771db1a562d58f741501d9e5ead227862
...

./data/c0/ad:
total 508
-rw-rw-r-- 1 admin admin    686 Feb 25 10:46 c0adefd85aea75563fca35a4c628f78f8237220d5c6390fe2f2aff2844fed960 <---

So, the entry from archives/ was soft-deleted (as borg usually does when "deleting" an archive).

The chunk in data/ is still there, also normal behaviour until borg compact cleans up.

Not sure why borg prune errors...

[tve]

borg check completed without errors:

Finished repository check.
Checked 1214464 repository objects, 0 errors.
Finished full repository check, no problems found.

Is there any additional info I can gather to help troubleshooting?

[ThomasWaldmann]

2 things could maybe give more insights:

• reproduce the issue with --debug
• write a script that produces a small, local repo that has this issue

[tve]

Log file of borg prune with --debug:
prune.txt

[ThomasWaldmann]

It didn't give that error msg in that borg prune --debug run.

But I found:

• it always calls list_start('archives', deleted=False), so it should not even know about the ".del" archive
• Trying discovered key b'0000000000000000000000000f3409ab' in /root/.ssh/id_ed25519 - guess that comes from paramiko. Just wondered about the many zeros in there, is that ok?

[tve]

LOL! I didn't even notice. So I ran borg prune once with --debug in the terminal and it crashed like before. Then I realized that there were too many lines for me to easily copy&paste into the ticket here, so I ran it again with output redirection but spent my time obfuscating hostname & key ID (that's the zeros) and not noticing that it didn't crash.

Here is the end of the run that did crash with --debug:

[chan 0] stat(b'/home/admin/mnt/big/@borg200b20/data/dd/56/dd568fb99fa2c26cbaeda1d17421fe9c9cb9
548780fd17787eefa31716117d27')                                                                 
[chan 0] open(b'/home/admin/mnt/big/@borg200b20/data/dd/56/dd568fb99fa2c26cbaeda1d17421fe9c9cb9
548780fd17787eefa31716117d27', 'r')                                                            
[chan 0] open(b'/home/admin/mnt/big/@borg200b20/data/dd/56/dd568fb99fa2c26cbaeda1d17421fe9c9cb9
548780fd17787eefa31716117d27', 'r') -> 00000000                                                
[chan 0] close(00000000)                                                                       
borgstore: load('data/dd568fb99fa2c26cbaeda1d17421fe9c9cb9548780fd17787eefa31716117d27', offset
=0, size=None, deleted=False) -> 121232B in 73.3ms                                             
[chan 0] stat(b'/home/admin/mnt/big/@borg200b20/archives/f5fc4e4c5b583bcbc4790e864980b80771db1a
562d58f741501d9e5ead227862')                                                                   
[chan 0] posix_rename(b'/home/admin/mnt/big/@borg200b20/archives/f5fc4e4c5b583bcbc4790e864980b8
0771db1a562d58f741501d9e5ead227862', b'/home/admin/mnt/big/@borg200b20/archives/f5fc4e4c5b583bc
bc4790e864980b80771db1a562d58f741501d9e5ead227862.del')                                        
borgstore: soft_delete('archives/f5fc4e4c5b583bcbc4790e864980b80771db1a562d58f741501d9e5ead2278
62', deleted=False) ['archives/f5fc4e4c5b583bcbc4790e864980b80771db1a562d58f741501d9e5ead227862
', 'archives/f5fc4e4c5b583bcbc4790e864980b80771db1a562d58f741501d9e5ead227862.del'] -> 0B in 3.
4ms                                                                                            
borgstore: list_start('archives', deleted=False)                                               
[chan 0] listdir(b'/home/admin/mnt/big/@borg200b20/archives')                                  
borgstore: list_end('archives', deleted=False) -> 3                                            
security: saving state for 39ff8572b3a502afd8312e1034ac95ded503a1db638162f0f3ee12f88e2e0d9d to 
/root/.local/share/borg/security/39ff8572b3a502afd8312e1034ac95ded503a1db638162f0f3ee12f88e2e0d
9d                                                                                             
security: current location   sftp://admin@xxx/mnt/big/@borg200b20             
security: key type           65                                                                
security: manifest timestamp 2026-02-27T08:10:25.348986+00:00                                  
borgstore: list_start('locks', deleted=False)                                                  
[chan 0] listdir(b'/home/admin/mnt/big/@borg200b20/locks')                                     
borgstore: list_end('locks', deleted=False) -> 1                                               
[chan 0] stat(b'/home/admin/mnt/big/@borg200b20/locks/de58e45c83f4c689')                       
[chan 0] open(b'/home/admin/mnt/big/@borg200b20/locks/de58e45c83f4c689', 'r')                  
[chan 0] open(b'/home/admin/mnt/big/@borg200b20/locks/de58e45c83f4c689', 'r') -> 00000000      
[chan 0] close(00000000)                                                                       
borgstore: load('locks/de58e45c83f4c689', offset=0, size=None, deleted=False) -> 126B in 20.1ms
LOCK-RELEASE: releasing lock: {'exclusive': False, 'hostid': 'tekke@86509736491056', 'processid
': 6, 'threadid': 0, 'time': '2026-02-28T08:34:34.614+00:00', 'key': 'de58e45c83f4c689', 'dt': 
datetime.datetime(2026, 2, 28, 8, 34, 34, 614000, tzinfo=datetime.timezone.utc)}.              
LOCK-DELETE: deleting lock from store. key: de58e45c83f4c689.                                  
[chan 0] stat(b'/home/admin/mnt/big/@borg200b20/locks/de58e45c83f4c689')                       
[chan 0] remove(b'/home/admin/mnt/big/@borg200b20/locks/de58e45c83f4c689')                     
borgstore: delete('locks/de58e45c83f4c689', deleted=False) -> 0B in 7.3ms                      
[chan 0] sftp session closed.                                                                  
[chan 0] EOF sent (0)                                                                          
Archive b'\xf5\xfcNL[X;\xcb\xc4y\x0e\x86I\x80\xb8\x07q\xdb\x1aV-X\xf7AP\x1d\x9e^\xad"xb' does n
ot exist                                                                                       
                                                                                               
Error:                                                                                         
                                                                                               
Archive.DoesNotExist: Archive b'\xf5\xfcNL[X;\xcb\xc4y\x0e\x86I\x80\xb8\x07q\xdb\x1aV-X\xf7AP\x
1d\x9e^\xad"xb' does not exist                                                                 
                                                                                               
If reporting bugs, please include the following:

Traceback (most recent call last):
  File "/root/borg/src/borg/archiver/__init__.py", line 669, in main
    exit_code = archiver.run(args)
  File "/root/borg/src/borg/archiver/__init__.py", line 546, in run
    rc = func(args)
  File "/root/borg/src/borg/archiver/_common.py", line 145, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/root/borg/src/borg/archiver/prune_cmd.py", line 202, in do_prune
    list_logger.info(f"{log_message:<44} {formatter.format_item(archive, jsonline=False)}")
                                          ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/borg/src/borg/helpers/parseformat.py", line 728, in format_item
    data = self.get_item_data(item, jsonline)
  File "/root/borg/src/borg/helpers/parseformat.py", line 813, in get_item_data
    item_data[key] = self.call_keys[key]()
                     ~~~~~~~~~~~~~~~~~~~^^
  File "/root/borg/src/borg/helpers/parseformat.py", line 832, in get_meta
    return self.archive.metadata.get(key, default)
           ^^^^^^^^^^^^
  File "/root/borg/src/borg/helpers/parseformat.py", line 828, in archive
    self._archive = Archive(self.manifest, self.id, iec=self.iec, deleted=self.deleted)
                    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/borg/src/borg/archive.py", line 557, in __init__
    raise self.DoesNotExist(name)
borg.archive.Archive.DoesNotExist: Archive b'\xf5\xfcNL[X;\xcb\xc4y\x0e\x86I\x80\xb8\x07q\xdb\x
1aV-X\xf7AP\x1d\x9e^\xad"xb' does not exist

Platform: Linux tekke 6.18.2-zen2-1-zen #1 ZEN SMP PREEMPT_DYNAMIC Thu, 18 Dec 2025 18:00:58 +0
000 x86_64
Linux: Unknown Linux  
Borg: 2.0.0b21.dev0+ged6e5db16.d20260222  Python: CPython 3.13.5 msgpack: 1.1.2 fuse: pyfuse3 3
.4.2 [mfusepy,pyfuse3,llfuse]
PID: 6  CWD: /data
sys.argv: ['/root/borg-env/bin/borg', 'prune', '--show-rc', '-v', '--debug', '--list', '-a', 'd
aily-full', '-a', 'host:tekke', '--keep-within', '2d', '-w', '4', '-m', '6', '-y', '50']
SSH_ORIGINAL_COMMAND: None


terminating with error status, rc 31

I just reran a non-debug prune and the error is gone:

== running: borg prune --show-rc -v --list -a daily-full -a host:tekke --keep-within 2d -w 4 -m 6 -y 50
Connected (version 2.0, client OpenSSH_10.0p2)
Authentication (publickey) successful!
[chan 0] Opened sftp connection (server version 3)
Keeping archive (rule: weekly #1):           tekke         daily-full 2026-02-26 17:27:16 
Keeping archive (rule: weekly #2):           tekke         daily-full 2026-02-22 17:01:01 
Keeping archive (rule: weekly[oldest] #3):   tekke         daily-full 2026-02-22 17:01:01 
[chan 0] sftp session closed.
terminating with success status, rc 0