Data Corrupion - ARM

[dylanetaft]

Testing a backup process of writing LVM snapshot disk images to img files on ltfs.

I attempted this exact chain on local disk first, confirmed the image came back clean with fsck.
When writing to ltfs the block device image comes back corrupt.

blocksync-fast -s /dev/mapper/datavg01-backuplv_snap_28 --progress -f /tmp/backup_08282026.digest -d backup_082
82026.img -b 128K

mbuffer -m 8G -s 1M -i /mnt/tape/backup_08282026.img | dd conv=sparse bs=1M of=restore.img

arm-server /mnt/coldstorage # e2fsck -f restore.img
e2fsck 1.47.4 (6-Mar-2025)
Resize inode not valid. Recreate? yes
Pass 1: Checking inodes, blocks, and sizes
Inode 5767438 has an invalid extent node (blk 23109631, lblk 0)
Clear? yes
Inode 5767438 extent tree (at level 1) could be shorter. Optimize? yes
Inode 5767438, i_blocks is 246072, should be 0. Fix? yes
HTREE directory inode 5906147 has an invalid root node.
Clear HTree index? yes
Inode 6032966 has an invalid extent node (blk 24160254, lblk 0)
Clear? yes
Inode 6032966 extent tree (at level 1) could be shorter. Optimize? yes
Inode 6032966, i_blocks is 279928, should be 0. Fix? yes
HTREE directory inode 6041467 has an invalid root node.
Clear HTree index? yes
HTREE directory inode 6046868 has an invalid root node.
Clear HTree index ('a' enables 'yes' to all) ? yes
HTREE directory inode 6291827 has an invalid root node.
Clear HTree index ('a' enables 'yes' to all) ? yes
Inode 6312820 has an invalid extent node (blk 25207803, lblk 0)

I am backing up a thin LVM snapshot that is online but unmounted.

I will investigate and try to submit a PR if I am able, it might be above my paygrade.

First I am testing the tape drive with a more basic tool though, to rule out tape drive or HBA issues.
mbuffer -m 8G -s 1M -i /dev/mapper/datavg01-backuplv_snap_28 -o /dev/nst0

I am thinking unfortunately something is off with fuse3 support but TBD.

[dylanetaft]

This may relate to #603
I guess I'm testing something not pulled into main yet.

[dylanetaft]

mbuffer -m 8G -s 1M -i /dev/mapper/datavg01-backuplv_snap_28 -o /dev/nst0
mt -f /dev/nst0 rewind
truncate -s 644245094400 restore.img
mbuffer -m 8G -L -s 1M -i /dev/nst0 | dd bs=1M iflag=fullblock conv=sparse of=restore.img
This worked, so the hardware is good.

Going to try a smaller file with ltfs, and also with fullblock on the ltfs route - I'll just calculate sha256 checksums both directions that way as LVM and other issues are ruled out.

[dylanetaft]

Yeah, it does seem to be broken

End to end test:
dd if=/dev/urandom of=/mnt/coldstorage/testfile bs=1G count=25
sha256sum testfile > before.txt
blocksync-fast -s /mnt/coldstorage/testfile --progress -f /tmp/testfile.digest -d /mnt/tape/testfile
mbuffer -m 8G -s 1M -i /mnt/tape/testfile | dd conv=sparse bs=1M of=restore_testfile
sha256sum restore_testfile > after.txt
hash is different
mbuffer -m 8G -s 1M -i /mnt/tape/testfile | dd conv=sparse bs=1M of=restore_testfile2 iflag=fullblock
hash is the same as restore_testfile - neither case works

I cherry picked @matejk 's three commits
329f036
923c6db
ad4b3bb

and my own for the quantum vendor\drive workaround
ae27503

And the last commit from the main branch is 7c01479 in my local repo so I am up to date

[dylanetaft]

I may have misunderstood the structure of git for this project.
It looks like patches that fix this issue are in branch v2.4.9.0, not main.
The branch is ahead of main. I will pull this branch and cherry pick the commits.

[madjesc]

Hello @dylanetaft.
Right now branch release/v2.4.9.0 is not merged into main because its a prerelease one and its going through the testing phase.
Feel free to try it out, and let us know if you have any issue.

[dylanetaft]

Yes, works now, no corruption.
I am using your FUSE 3 commits and my PR commit for a quantum issue, and the last commit of 2a13b53 from upstream.