HTML message rejected: btrfs checksum error
A message that is not in the archive (sent off-list or not yet received): [email protected]
I have a RAID1 btrfs root/home on Fedora 42 that developed what appears to be a single data checksum error. RAM tests fine, but it's a DELL system that had memory problems early on (years ago), that were fixed by Dell BIOS memory tests (which changed the mem controller settings).
The errors seem to have started right after a scrub (see btrfs messages from journal below)
btrfs check --readonly --force --check-data-csum -p /dev/nvme0n1p2
shows a cascade of errors (which seem to be increasing in number) ..
[4/7] checking fs roots (0:00:04 elapsed, 60923
items checked)
mirror 1 bytenr 299511672832 csum 0x125beb3c expected csum 0xc8374bb569 items checked)
mirror 1 bytenr 299511676928 csum 0x4c6adf72 expected csum 0xd82f54b8 mirror 2 bytenr 299511672832 csum 0x125beb3c expected csum 0xc8374bb5 mirror 2 bytenr 299511676928 csum 0x4c6adf72 expected csum 0xd82f54b8 mirror 1 bytenr 306513821696 csum 0x8941f998 expected csum 0xa5fe1bfd94 items checked)
mirror 1 bytenr 306513825792 csum 0x8941f998 expected csum 0x77c755d4 .. and many more
I can recover the file with only 1 4kB block zeroed out.
Is there a way to read the bad sector? I thought that
mount -o ro,degraded,rescue=ignoredatacsums/dev/sda5 /mnt would read data ignoring the bad checksum? as it is, it replicates the I/O error that is raised when reading the original file.
Do you think that deleting the file with the bad checksum will solve this? or should I move to rebuilding and restoring from backups?
Apr 26 22:41:04 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 1 Apr 26 22:41:04 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 2 Apr 26 22:41:36 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/nvme0n1p2 physical 74303995904 Apr 26 22:41:36 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/nvme0n1p2, physical 74303995904, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 22:42:52 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 1 with status: 0 Apr 26 22:45:46 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/sda5 physical 147297468416 Apr 26 22:45:46 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/sda5, physical 147297468416, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 22:48:45 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 2 with status: 0 Apr 26 22:53:23 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 2 Apr 26 22:53:23 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 1 Apr 26 22:53:52 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/nvme0n1p2 physical 74303995904 Apr 26 22:53:52 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/nvme0n1p2, physical 74303995904, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 22:55:07 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 1 with status: 0 Apr 26 22:58:04 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/sda5 physical 147297468416 Apr 26 22:58:04 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/sda5, physical 147297468416, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 23:01:01 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 2 with status: 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0 Apr 27 07:35:55 fedora kernel: btrfs_print_data_csum_error: 2 callbacks suppressed Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: btrfs_dev_stat_inc_and_print: 2 callbacks suppressed Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 11, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev
/dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 11, gen 0
Quoted 25 lines · Przemek Klosowski
I have a RAID1 btrfs root/home on Fedora 42 that developed what appears to be a single data checksum error. RAM tests fine, but it's a DELL system that had memory problems early on (years ago), that were fixed by Dell BIOS memory tests (which changed the mem controller settings).
The errors seem to have started right after a scrub (see btrfs messages from journal below)
btrfs check --readonly --force --check-data-csum -p
/dev/nvme0n1p2shows a cascade of errors (which seem to be increasing in number) ..
[4/7] checking fs roots (0:00:04 elapsed, 60923
items checked)
mirror 1 bytenr 299511672832 csum0x125beb3cexpected csum0xc8374bb569items checked)
mirror 1 bytenr 299511676928 csum0x4c6adf72expected csum0xd82f54b8mirror 2 bytenr 299511672832 csum0x125beb3cexpected csum0xc8374bb5mirror 2 bytenr 299511676928 csum0x4c6adf72expected csum0xd82f54b8mirror 1 bytenr 306513821696 csum0x8941f998expected csum0xa5fe1bfd94items checked)
mirror 1 bytenr 306513825792 csum0x8941f998expected csum0x77c755d4.. and many more
Quoted 6 lines
I can recover the file with only 1 4kB block zeroed out.
Is there a way to read the bad sector? I thought that
mount -o ro,degraded,rescue=ignoredatacsums/dev/sda5 /mnt would read data ignoring the bad checksum? as it is, it replicates the I/O error that is raised when reading the original file.
It turns out to be a bug in the implementation, we expect to ignore bad data csum error and return the data directly, but it's not implemented if the csum tree is still valid...
I'll send out a patch for that, but that will also mean with rescue=idatacsums mount option, the data will only be the first one btrfs read out.
It'll be fine for your case, as both mirrors have the same csum.
Quoted 3 lines
Do you think that deleting the file with the bad checksum will solve this?
Yes.
Quoted 1 line
or should I move to rebuilding and restoring from backups?
No need, "btrfs check --check-data-csum" is the most comprehensive check we have and it only reports error of data checksum so far (better than scrub because of the comprehensive metadata checks).
Although you will need to find out all involved files, scrub is doing a good job resolving the path, but the output may be ratelimited.
I'd recommend to craft a small script, parsing all involved unique bytenr into btrfs ins logical to get a full path to the affected files.
Thanks,
Qu
Quoted 151 lines
Apr 26 22:41:04 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 1 Apr 26 22:41:04 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 2 Apr 26 22:41:36 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/nvme0n1p2 physical 74303995904 Apr 26 22:41:36 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/nvme0n1p2, physical 74303995904, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 22:42:52 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 1 with status: 0 Apr 26 22:45:46 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/sda5 physical 147297468416 Apr 26 22:45:46 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/sda5, physical 147297468416, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 22:48:45 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 2 with status: 0 Apr 26 22:53:23 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 2 Apr 26 22:53:23 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 1 Apr 26 22:53:52 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/nvme0n1p2 physical 74303995904 Apr 26 22:53:52 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/nvme0n1p2, physical 74303995904, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 22:55:07 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 1 with status: 0 Apr 26 22:58:04 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/sda5 physical 147297468416 Apr 26 22:58:04 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/sda5, physical 147297468416, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 23:01:01 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 2 with status: 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0 Apr 27 07:35:55 fedora kernel: btrfs_print_data_csum_error: 2 callbacks suppressed Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: btrfs_dev_stat_inc_and_print: 2 callbacks suppressed Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 11, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev
/dev/nvme0n1p2errs: wr 0, rd 0, flush 0, corrupt 11, gen 0
Quoted 30 lines · Qu Wenruo
Quoted 27 lines · Przemek Klosowski
I have a RAID1 btrfs root/home on Fedora 42 that developed what appears to be a single data checksum error. RAM tests fine, but it's a DELL system that had memory problems early on (years ago), that were fixed by Dell BIOS memory tests (which changed the mem controller settings).
The errors seem to have started right after a scrub (see btrfs messages from journal below)
btrfs check --readonly --force --check-data-csum -p
/dev/nvme0n1p2shows a cascade of errors (which seem to be increasing in number) ..
[4/7] checking fs roots (0:00:04 elapsed, 60923 items checked)
mirror 1 bytenr 299511672832 csum0x125beb3cexpected csum0xc8374bb569items checked)
mirror 1 bytenr 299511676928 csum0x4c6adf72expected csum0xd82f54b8mirror 2 bytenr 299511672832 csum0x125beb3cexpected csum0xc8374bb5mirror 2 bytenr 299511676928 csum0x4c6adf72expected csum0xd82f54b8mirror 1 bytenr 306513821696 csum0x8941f998expected csum0xa5fe1bfd94items checked)
mirror 1 bytenr 306513825792 csum0x8941f998expected csum0x77c755d4.. and many moreI can recover the file with only 1 4kB block zeroed out.
Quoted 8 lines
Quoted 4 lines
Is there a way to read the bad sector? I thought that
mount -o ro,degraded,rescue=ignoredatacsums/dev/sda5 /mnt would read data ignoring the bad checksum? as it is, it replicates the I/O error that is raised when reading the original file.It turns out to be a bug in the implementation, we expect to ignore bad data csum error and return the data directly, but it's not implemented if the csum tree is still valid...
I'll send out a patch for that, but that will also mean with rescue=idatacsums mount option, the data will only be the first one btrfs read out.
It is not a bug, it is already handled properly by completely ignoring the data csum tree.
I guess the problem is you're still trying to scrub with that rescue=idatacsums mount option?
That mount option is to be used with regular read, which will not do any verification now.
Please verify if regular read on those files work.
Thanks,
Qu
Quoted 173 lines
It'll be fine for your case, as both mirrors have the same csum.
Quoted 3 lines
Do you think that deleting the file with the bad checksum will solve this?
Yes.
Quoted 1 line
or should I move to rebuilding and restoring from backups?
No need, "btrfs check --check-data-csum" is the most comprehensive check we have and it only reports error of data checksum so far (better than scrub because of the comprehensive metadata checks).
Although you will need to find out all involved files, scrub is doing a good job resolving the path, but the output may be ratelimited.
I'd recommend to craft a small script, parsing all involved unique bytenr into
btrfs ins logicalto get a full path to the affected files.Thanks,
QuQuoted 151 lines
Apr 26 22:41:04 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 1 Apr 26 22:41:04 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 2 Apr 26 22:41:36 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/nvme0n1p2 physical 74303995904 Apr 26 22:41:36 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/nvme0n1p2, physical 74303995904, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 22:42:52 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 1 with status: 0 Apr 26 22:45:46 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/sda5 physical 147297468416 Apr 26 22:45:46 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/sda5, physical 147297468416, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 22:48:45 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 2 with status: 0 Apr 26 22:53:23 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 2 Apr 26 22:53:23 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 1 Apr 26 22:53:52 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/nvme0n1p2 physical 74303995904 Apr 26 22:53:52 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/nvme0n1p2, physical 74303995904, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 22:55:07 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 1 with status: 0 Apr 26 22:58:04 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/sda5 physical 147297468416 Apr 26 22:58:04 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/sda5, physical 147297468416, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 23:01:01 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 2 with status: 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0 Apr 27 07:35:55 fedora kernel: btrfs_print_data_csum_error: 2 callbacks suppressed Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: btrfs_dev_stat_inc_and_print: 2 callbacks suppressed Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 11, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev
/dev/nvme0n1p2errs: wr 0, rd 0, flush 0, corrupt 11, gen 0
A message that is not in the archive (sent off-list or not yet received): CAC=1GgEaY80tHuA1av-u8y43o_U-yF6-7b8kaDNLi=i5X-fGqw@mail.gmail.com
I guess the problem is you're still trying to scrub with that
rescue=idatacsums mount option?
No--it's my root fs so I mounted the individual device as readonly/degraded, kind-of recursively in the same live fs. I just wanted to see if I can recover more data, and thought it'd be safe because of its readonly status.
Thank you for the suggestions---after deleting the problem file, all is well (btrfs scrub shows no errors)
By the way, I was perplexed by the tooling, because it seems that I need to run
btrfs check --readonly --force --check-data-csum -p /dev/nvme0n1p2
on a live filesystem (it's my root fs) to get the error locations, and it reports additional checksum errors due to the activity on the fs (mozilla cache files, etc).
As you suggested, I extracted the offsets and piped to btrfs ins logical, which gave multiple filenames. I knew which one was the real one. I wonder if there's a way to avoid the fake ones? Of course I could have booted from flash and mounted my real fs readonly under it, got the filename, remounted rw and deleted the file.
Is there a simpler, recommended workflow for this case? Maybe making a snapshot?
BTW, btrfs dev stat keeps showing corruption_errs on both raid1 devices. Do you recommend 'btrfs dev stats -z' to zero them out?
Quoted 239 lines · Qu Wenruo
Quoted 35 lines · Qu Wenruo
Quoted 25 lines · Przemek Klosowski
I have a RAID1 btrfs root/home on Fedora 42 that developed what appears to be a single data checksum error. RAM tests fine, but it's a DELL system that had memory problems early on (years ago), that were fixed by Dell BIOS memory tests (which changed the mem controller settings).
The errors seem to have started right after a scrub (see btrfs messages from journal below)
btrfs check --readonly --force --check-data-csum -p
/dev/nvme0n1p2shows a cascade of errors (which seem to be increasing in number) ..
[4/7] checking fs roots (0:00:04 elapsed, 60923
items checked)
mirror 1 bytenr 299511672832 csum0x125beb3cexpected csum0xc8374bb569items checked)
mirror 1 bytenr 299511676928 csum0x4c6adf72expected csum0xd82f54b8mirror 2 bytenr 299511672832 csum0x125beb3cexpected csum0xc8374bb5mirror 2 bytenr 299511676928 csum0x4c6adf72expected csum0xd82f54b8mirror 1 bytenr 306513821696 csum0x8941f998expected csum0xa5fe1bfd94items checked)
mirror 1 bytenr 306513825792 csum0x8941f998expected csum0x77c755d4.. and many moreQuoted 6 lines
I can recover the file with only 1 4kB block zeroed out.
Is there a way to read the bad sector? I thought that
mount -o ro,degraded,rescue=ignoredatacsums/dev/sda5 /mnt would read data ignoring the bad checksum? as it is, it replicates the I/O error that is raised when reading the original file.Quoted 7 lines
It turns out to be a bug in the implementation, we expect to ignore bad data csum error and return the data directly, but it's not implemented if the csum tree is still valid...
I'll send out a patch for that, but that will also mean with rescue=idatacsums mount option, the data will only be the first one btrfs read out.
It is not a bug, it is already handled properly by completely ignoring the data csum tree.
I guess the problem is you're still trying to scrub with that rescue=idatacsums mount option?
That mount option is to be used with regular read, which will not do any verification now.
Please verify if regular read on those files work.
Thanks,
QuQuoted 177 lines
It'll be fine for your case, as both mirrors have the same csum.
Quoted 3 lines
Do you think that deleting the file with the bad checksum will solve this?
Yes.
Quoted 1 line
or should I move to rebuilding and restoring from backups?
No need, "btrfs check --check-data-csum" is the most comprehensive check we have and it only reports error of data checksum so far (better than scrub because of the comprehensive metadata checks).
Although you will need to find out all involved files, scrub is doing a good job resolving the path, but the output may be ratelimited.
I'd recommend to craft a small script, parsing all involved unique bytenr into
btrfs ins logicalto get a full path to the affected files.Thanks,
QuQuoted 151 lines
Apr 26 22:41:04 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 1 Apr 26 22:41:04 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 2 Apr 26 22:41:36 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/nvme0n1p2 physical 74303995904 Apr 26 22:41:36 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/nvme0n1p2, physical 74303995904, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 22:42:52 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 1 with status: 0 Apr 26 22:45:46 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/sda5 physical 147297468416 Apr 26 22:45:46 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/sda5, physical 147297468416, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 22:48:45 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 2 with status: 0 Apr 26 22:53:23 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 2 Apr 26 22:53:23 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 1 Apr 26 22:53:52 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/nvme0n1p2 physical 74303995904 Apr 26 22:53:52 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/nvme0n1p2, physical 74303995904, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 22:55:07 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 1 with status: 0 Apr 26 22:58:04 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/sda5 physical 147297468416 Apr 26 22:58:04 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/sda5, physical 147297468416, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 23:01:01 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 2 with status: 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0 Apr 27 07:35:55 fedora kernel: btrfs_print_data_csum_error: 2 callbacks suppressed Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: btrfs_dev_stat_inc_and_print: 2 callbacks suppressed Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 11, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev
/dev/nvme0n1p2errs: wr 0, rd 0, flush 0, corrupt 11, gen 0
Quoted 7 lines · Przemek Klosowski
I guess the problem is you're still trying to scrub with that
rescue=idatacsums mount option?No--it's my root fs so I mounted the individual device as readonly/degraded, kind-of recursively in the same live fs. I just wanted to see if I can recover more data, and thought it'd be safe because of its readonly status.
In that case, unless fully RO mounted with "rescue=idatacsums" mount option, btrfs will still do the data checksum verification thus leading to the error messages, and reject reads of involved blocks.
Quoted 10 lines
Thank you for the suggestions---after deleting the problem file, all is well (btrfs scrub shows no errors)
By the way, I was perplexed by the tooling, because it seems that I need to run
btrfs check --readonly --force --check-data-csum -p/dev/nvme0n1p2
on a live filesystem (it's my root fs) to get the error locations, and it reports additional checksum errors due to the activity on the fs (mozilla cache files, etc).
Quoted 3 lines
As you suggested, I extracted the offsets and piped to btrfs ins logical, which gave multiple filenames. I knew which one was the real one. I wonder if there's a way to avoid the fake ones?
Mind to give some examples?
It can be a bug in the logical resolve implementation, which is part of the kernel code.
Quoted 4 lines
Of course I
could have booted from flash and mounted my real fs readonly under it, got the filename, remounted rw and deleted the file.
Is there a simpler, recommended workflow for this case? Maybe making a snapshot?
Making snapshots will make it worse, as now you have to delete both the original file and the file in the snapshots.
Unfortunately we do not have a good workflow for that, and maybe we can add something like "btrfs rescue delete-bad-files" to do that automatically.
So far our priority is still detecting the problem, the repair is always done by the extra copies, thus no manual data corruption fix.
But there is a recent kernel fix that will force direct-IO to be buffered IO if the file has data checksum, to avoid any badly programmed direct IO users to cause false data checksum mismatch.
If you're sure the corruption is not caused by hardware, then upgrading the kernel may prevent such false alerts from happening in the future.
Quoted 3 lines
BTW, btrfs dev stat keeps showing
corruption_errson both raid1 devices. Do you recommend 'btrfs dev stats -z' to zero them out?
Since scrub and btrfs check reports no more error, it's completely fine to zero them out.
Thanks,
Qu
Quoted 244 lines · Qu Wenruo
Quoted 239 lines · Qu Wenruo
Quoted 35 lines · Przemek Klosowski
Quoted 25 lines
I have a RAID1 btrfs root/home on Fedora 42 that developed what appears to be a single data checksum error. RAM tests fine, but it's a DELL system that had memory problems early on (years ago), that were fixed by Dell BIOS memory tests (which changed the mem controller settings).
The errors seem to have started right after a scrub (see btrfs messages from journal below)
btrfs check --readonly --force --check-data-csum -p
/dev/nvme0n1p2shows a cascade of errors (which seem to be increasing in number) ..
[4/7] checking fs roots (0:00:04 elapsed, 60923
items checked)
mirror 1 bytenr 299511672832 csum0x125beb3cexpected csum0xc8374bb569items checked)
mirror 1 bytenr 299511676928 csum0x4c6adf72expected csum0xd82f54b8mirror 2 bytenr 299511672832 csum0x125beb3cexpected csum0xc8374bb5mirror 2 bytenr 299511676928 csum0x4c6adf72expected csum0xd82f54b8mirror 1 bytenr 306513821696 csum0x8941f998expected csum0xa5fe1bfd94items checked)
mirror 1 bytenr 306513825792 csum0x8941f998expected csum0x77c755d4.. and many moreQuoted 6 lines
I can recover the file with only 1 4kB block zeroed out.
Is there a way to read the bad sector? I thought that
mount -o ro,degraded,rescue=ignoredatacsums/dev/sda5 /mnt would read data ignoring the bad checksum? as it is, it replicates the I/O error that is raised when reading the original file.Quoted 7 lines
It turns out to be a bug in the implementation, we expect to ignore bad data csum error and return the data directly, but it's not implemented if the csum tree is still valid...
I'll send out a patch for that, but that will also mean with rescue=idatacsums mount option, the data will only be the first one btrfs read out.
It is not a bug, it is already handled properly by completely ignoring the data csum tree.
I guess the problem is you're still trying to scrub with that rescue=idatacsums mount option?
That mount option is to be used with regular read, which will not do any verification now.
Please verify if regular read on those files work.
Thanks,
QuQuoted 177 lines
It'll be fine for your case, as both mirrors have the same csum.
Quoted 3 lines
Do you think that deleting the file with the bad checksum will solve this?
Yes.
Quoted 1 line
or should I move to rebuilding and restoring from backups?
No need, "btrfs check --check-data-csum" is the most comprehensive check we have and it only reports error of data checksum so far (better than scrub because of the comprehensive metadata checks).
Although you will need to find out all involved files, scrub is doing a good job resolving the path, but the output may be ratelimited.
I'd recommend to craft a small script, parsing all involved unique bytenr into
btrfs ins logicalto get a full path to the affected files.Thanks,
QuQuoted 151 lines
Apr 26 22:41:04 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 1 Apr 26 22:41:04 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 2 Apr 26 22:41:36 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/nvme0n1p2 physical 74303995904 Apr 26 22:41:36 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/nvme0n1p2, physical 74303995904, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 22:42:52 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 1 with status: 0 Apr 26 22:45:46 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/sda5 physical 147297468416 Apr 26 22:45:46 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/sda5, physical 147297468416, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 22:48:45 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 2 with status: 0 Apr 26 22:53:23 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 2 Apr 26 22:53:23 fedora kernel: BTRFS info (device nvme0n1p2): scrub: started on devid 1 Apr 26 22:53:52 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/nvme0n1p2 physical 74303995904 Apr 26 22:53:52 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/nvme0n1p2, physical 74303995904, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 22:55:07 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 1 with status: 0 Apr 26 22:58:04 fedora kernel: BTRFS error (device nvme0n1p2): unable to fixup (regular) error at logical 452965761024 on dev /dev/sda5 physical 147297468416 Apr 26 22:58:04 fedora kernel: BTRFS warning (device nvme0n1p2): checksum error at logical 452965761024 on dev /dev/sda5, physical 147297468416, root 257, inode 35328, offset 26034176, length 4096, links 1 (path: usr/lib/sysimage/rpm/rpmdb.sqlite-wal) Apr 26 23:01:01 fedora kernel: BTRFS info (device nvme0n1p2): scrub: finished on devid 2 with status: 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 Apr 27 07:35:32 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:32 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0 Apr 27 07:35:33 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:33 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0 Apr 27 07:35:55 fedora kernel: btrfs_print_data_csum_error: 2 callbacks suppressed Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: btrfs_dev_stat_inc_and_print: 2 callbacks suppressed Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/nvme0n1p2 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x127b77ee expected csum 0xcf4a5572 mirror 2 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev /dev/sda5 errs: wr 0, rd 0, flush 0, corrupt 11, gen 0 Apr 27 07:35:55 fedora kernel: BTRFS warning (device nvme0n1p2): csum failed root 257 ino 35328 off 26079232 csum 0x862b6025 expected csum 0xcf4a5572 mirror 1 Apr 27 07:35:55 fedora kernel: BTRFS error (device nvme0n1p2): bdev
/dev/nvme0n1p2errs: wr 0, rd 0, flush 0, corrupt 11, gen 0