Skip to contentSkip to conversation

5.6 pretty massive unexplained btrfs corruption: parent transid verify failed + open_ctree failed

DISCUSSION Marc MERLIN 4 participants 15 messages Jul 7, 2020 – Aug 16, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw

I'd love to know what went wrong so that it doesn't happen again, but let me know if you'd like data off this before I wipe it (which I assume is the only way out at this point)

myth:~# btrfs check --mode=lowmem /dev/mapper/crypt_bcache0
Opening filesystem to check...
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
Ignoring transid failure
leaf parent key incorrect 7325633544192
ERROR: failed to read block groups: Operation not permitted
ERROR: cannot open file system

I did run bees on that filesystem, but I also just did a full btrfs check on it, and it came back clean:

Opening filesystem to check...
Checking filesystem on /dev/mapper/crypt_bcache4
UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d
[1/7] checking root items
Fixed 0 roots.
[2/7] checking extents
No device size related problem found
[3/7] checking free space cache
cache and super generation don't match, space cache will be invalidated [4/7] checking fs roots
[5/7] checking only csums items (without verifying data) [6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS) found 18089211043840 bytes used, no error found
total csum bytes: 17580412652
total tree bytes: 82326192128
total fs tree bytes: 56795086848
total extent tree bytes: 5154258944
btree space waste bytes: 13682108904
file data blocks allocated: 24050542804992

I then moved it to the target machine, started a btrfs send to it, and it failed quickly (due to a mistake I had an old btrfs binary on that server, but I'm hoping most of the work is done in kernel space and that the user space btrfs should not corrupt the disk if it's a bit old)

myth:/mnt# uname -r
5.6.5-amd64-preempt-sysrq-20190817

Soon after, the copy failed:

[ 2575.931316] BTRFS info (device dm-0): use zlib compression, level 3
[ 2575.931329] BTRFS info (device dm-0): disk space caching is enabled
[ 2575.931343] BTRFS info (device dm-0): has skinny extents
[ 2577.286749] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[ 2607.943516] BTRFS info (device dm-0): enabling ssd optimizations
[ 2708.835200] BTRFS warning (device dm-0): block group 13002170433536 has wrong amount of free space
[ 2708.835209] BTRFS warning (device dm-0): failed to load free space cache for block group 13002170433536, rebuilding it now
[ 2740.589580] BTRFS warning (device dm-0): block group 17151175950336 has wrong amount of free space
[ 2740.589593] BTRFS warning (device dm-0): failed to load free space cache for block group 17151175950336, rebuilding it now
[ 2797.204169] perf: interrupt took too long (3146 > 3138), lowering kernel.perf_event_max_sample_rate to 63500
[ 2882.545242] BTRFS info (device dm-0): the free space cache file (26234763345920) is invalid, skip it
[ 3071.631905] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
[ 3071.643430] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
[ 3071.661985] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
[ 3071.661995] BTRFS: error (device dm-0) in btrfs_run_delayed_refs:2210: errno=-5 IO failure
[ 3071.661999] BTRFS info (device dm-0): forced readonly
[ 3071.662567] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
[ 3071.663076] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
[ 3071.663083] BTRFS: error (device dm-0) in btrfs_run_delayed_refs:2210: errno=-5 IO failure

Ok, maybe there was an IO failure, although none was shown by the kernel:

however, now the FS is mostly dead:

[ 3649.106084] BTRFS info (device dm-0): use zlib compression, level 3
[ 3649.106095] BTRFS info (device dm-0): disk space caching is enabled
[ 3649.106100] BTRFS info (device dm-0): has skinny extents
[ 3650.445828] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[ 3652.952110] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 3652.959199] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 3652.959208] BTRFS error (device dm-0): failed to read block groups: -5
[ 3653.002227] BTRFS error (device dm-0): open_ctree failed
[ 3876.808183] BTRFS info (device dm-0): use zlib compression, level 3
[ 3876.808192] BTRFS info (device dm-0): disk space caching is enabled
[ 3876.808195] BTRFS info (device dm-0): has skinny extents
[ 3878.140763] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[ 3880.623113] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 3880.633290] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 3880.633298] BTRFS error (device dm-0): failed to read block groups: -5
[ 3880.669435] BTRFS error (device dm-0): open_ctree failed
[ 4057.606879] BTRFS info (device dm-0): use zlib compression, level 3
[ 4057.606890] BTRFS info (device dm-0): disk space caching is enabled
[ 4057.606894] BTRFS info (device dm-0): has skinny extents
[ 4058.886212] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[ 4061.501589] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 4061.503790] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 4061.503799] BTRFS error (device dm-0): failed to read block g
myth:/mnt# btrfs-zero-log /dev/mapper/crypt_bcache0
WARNING: this utility is deprecated, please use 'btrfs rescue zero-log'
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
Ignoring transid failure
leaf parent key incorrect 7325633544192
Clearing log on /dev/mapper/crypt_bcache0, previous log_root 0, level 0
myth:/mnt# mount -t btrfs -o recovery,nospace_cache,clear_cache /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
mount: wrong fs type, bad option, bad superblock on /dev/mapper/crypt_bcache0,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
myth:/mnt# dmtail
[ 6665.975324] BTRFS info (device dm-0): unrecognized mount option 'rootflags=recovery'
[ 6665.975357] BTRFS error (device dm-0): open_ctree failed
[ 6686.664202] BTRFS warning (device dm-0): 'recovery' is deprecated, use 'usebackuproot' instead
[ 6686.664213] BTRFS info (device dm-0): trying to use backup root at mount time
[ 6686.664219] BTRFS info (device dm-0): disabling disk space caching
[ 6686.664224] BTRFS info (device dm-0): force clearing of disk cache
[ 6686.664232] BTRFS info (device dm-0): has skinny extents
[ 6687.911926] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[ 6690.522785] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 6690.529495] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 6690.529504] BTRFS error (device dm-0): failed to read block groups: -5
[ 6690.556227] BTRFS error (device dm-0): open_ctree failed
myth:/mnt# btrfs restore /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
Skipping snapshot win_ro.20200615_02:49:02
Skipping snapshot 0Notmachines_ro.20200626_16:09:30
Skipping snapshot 1Appliances_ro.20200626_17:08:56
Skipping snapshot debian32_ro.20200626_17:11:57
Skipping snapshot debian64_ro.20200626_17:18:15
Skipping snapshot ubuntu_ro.20200626_17:18:44
Skipping snapshot win_ro.20200626_18:39:13
Skipping snapshot 0Notmachines_ro.20200629_00:48:39
Skipping snapshot 1Appliances_ro.20200629_01:11:44
Skipping snapshot debian32_ro.20200629_01:11:50
Skipping snapshot debian64_ro.20200629_01:19:52
Skipping snapshot ubuntu_ro.20200629_01:20:01
Error copying data for /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
myth:/mnt# l /mnt/btrfs_bigbackup/
is missing a lot of stuff

It's a backup server, I can recreate the data, but it will probably take 2 weeks to copy everything again, and I'd like to know what on earth happened so that I can avoid having this again.

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#1 lore raw
Quoted 41 lines · Marc MERLIN

I'd love to know what went wrong so that it doesn't happen again, but let me know if you'd like data off this before I wipe it (which I assume is the only way out at this point)

myth:~# btrfs check --mode=lowmem /dev/mapper/crypt_bcache0
Opening filesystem to check...
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
Ignoring transid failure
leaf parent key incorrect 7325633544192
ERROR: failed to read block groups: Operation not permitted
ERROR: cannot open file system

I did run bees on that filesystem, but I also just did a full btrfs check on it, and it came back clean:

Opening filesystem to check...
Checking filesystem on /dev/mapper/crypt_bcache4
UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d
[1/7] checking root items
Fixed 0 roots.
[2/7] checking extents
No device size related problem found
[3/7] checking free space cache
cache and super generation don't match, space cache will be invalidated [4/7] checking fs roots
[5/7] checking only csums items (without verifying data) [6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS) found 18089211043840 bytes used, no error found
total csum bytes: 17580412652
total tree bytes: 82326192128
total fs tree bytes: 56795086848
total extent tree bytes: 5154258944
btree space waste bytes: 13682108904
file data blocks allocated: 24050542804992

I then moved it to the target machine, started a btrfs send to it, and it failed quickly (due to a mistake I had an old btrfs binary on that server, but I'm hoping most of the work is done in kernel space and that the user space btrfs should not corrupt the disk if it's a bit old)

Quoted 8 lines
myth:/mnt# uname -r
5.6.5-amd64-preempt-sysrq-20190817

Soon after, the copy failed:

[ 2575.931316] BTRFS info (device dm-0): use zlib compression, level 3
[ 2575.931329] BTRFS info (device dm-0): disk space caching is enabled
[ 2575.931343] BTRFS info (device dm-0): has skinny extents
[ 2577.286749] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0

You have a corrupt counter here at mount time, does your logs go back far enough to see where those came in? Thanks,

Josef

#2 lore raw
Quoted 8 lines · Josef Bacik
Quoted 5 lines

Soon after, the copy failed:

[ 2575.931316] BTRFS info (device dm-0): use zlib compression, level 3
[ 2575.931329] BTRFS info (device dm-0): disk space caching is enabled
[ 2575.931343] BTRFS info (device dm-0): has skinny extents
[ 2577.286749] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0

You have a corrupt counter here at mount time, does your logs go back far enough to see where those came in? Thanks,

I copied all the logs in the original Email.

Here it is all in one swoop:
https://pastebin.com/GURufP8w

The devices going down and back up is normal, my backup script saw the backup was over (although in that case that's because it failed) and automatically unmounted the filesystem and turned off the drives.

I'm not sure how my FS got so corrupted, it's been a while since I've seen damage that bad, especially pretty unexplained like this.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#3 lore raw
In reply to Marc MERLIN
Quoted 35 lines · Marc MERLIN

I'd love to know what went wrong so that it doesn't happen again, but let me know if you'd like data off this before I wipe it (which I assume is the only way out at this point)

myth:~# btrfs check --mode=lowmem /dev/mapper/crypt_bcache0
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
Ignoring transid failure
leaf parent key incorrect 7325633544192
ERROR: failed to read block groups: Operation not permitted
ERROR: cannot open file system

I did run bees on that filesystem, but I also just did a full btrfs check on it, and it came back clean:
Opening filesystem to check...
Checking filesystem on /dev/mapper/crypt_bcache4
UUID: 36f5079e-ca6c-4855-8639-ccb82695c18d
[1/7] checking root items
Fixed 0 roots.
[2/7] checking extents
No device size related problem found
[3/7] checking free space cache
cache and super generation don't match, space cache will be invalidated
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 18089211043840 bytes used, no error found
total csum bytes: 17580412652
total tree bytes: 82326192128
total fs tree bytes: 56795086848
total extent tree bytes: 5154258944
btree space waste bytes: 13682108904
file data blocks allocated: 24050542804992

Quoted 3 lines

I then moved it to the target machine, started a btrfs send to it, and it failed quickly (due to a mistake I had an old btrfs binary on that server, but I'm hoping most of the work is done in kernel space and that the user space btrfs should not corrupt the disk if it's a bit old)

btrfs send has historically had bugs but not filesystem-damaging ones (just relatively harmless kernel crashes and send failures). btrfs receive is almost entirely userspace--it can't corrupt anything that can't be corrupted by normal filesystem operations.

Quoted 8 lines
myth:/mnt# uname -r
5.6.5-amd64-preempt-sysrq-20190817

Soon after, the copy failed:

[ 2575.931316] BTRFS info (device dm-0): use zlib compression, level 3
[ 2575.931329] BTRFS info (device dm-0): disk space caching is enabled
[ 2575.931343] BTRFS info (device dm-0): has skinny extents
[ 2577.286749] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0

This line does indicate an older problem with the filesystem. It doesn't tell us whether the corruption happened yesterday or a year ago. You will need to look at your older kernel logs for that.

Quoted 7 lines
[ 2607.943516] BTRFS info (device dm-0): enabling ssd optimizations
[ 2708.835200] BTRFS warning (device dm-0): block group 13002170433536 has wrong amount of free space
[ 2708.835209] BTRFS warning (device dm-0): failed to load free space cache for block group 13002170433536, rebuilding it now
[ 2740.589580] BTRFS warning (device dm-0): block group 17151175950336 has wrong amount of free space
[ 2740.589593] BTRFS warning (device dm-0): failed to load free space cache for block group 17151175950336, rebuilding it now
[ 2797.204169] perf: interrupt took too long (3146 > 3138), lowering kernel.perf_event_max_sample_rate to 63500
[ 2882.545242] BTRFS info (device dm-0): the free space cache file (26234763345920) is invalid, skip it

Use space_cache=v2, especially on a big filesystem because space_cache=v1 slows down linearly with filesystem size. There is no need for these warnings. They're probably a symptom rather than cause here.

Quoted 10 lines
[ 3071.631905] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
[ 3071.643430] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
[ 3071.661985] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
[ 3071.661995] BTRFS: error (device dm-0) in btrfs_run_delayed_refs:2210: errno=-5 IO failure
[ 3071.661999] BTRFS info (device dm-0): forced readonly
[ 3071.662567] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
[ 3071.663076] BTRFS error (device dm-0): parent transid verify failed on 6353897537536 wanted 359658 found 359661
[ 3071.663083] BTRFS: error (device dm-0) in btrfs_run_delayed_refs:2210: errno=-5 IO failure

Ok, maybe there was an IO failure, although none was shown by the kernel:

The "IO failure" mentioned here is the earlier parent transid verify failure. When the verification fails, the caller gets -EIO.

Quoted 15 lines

however, now the FS is mostly dead:

[ 3649.106084] BTRFS info (device dm-0): use zlib compression, level 3
[ 3649.106095] BTRFS info (device dm-0): disk space caching is enabled
[ 3649.106100] BTRFS info (device dm-0): has skinny extents
[ 3650.445828] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[ 3652.952110] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 3652.959199] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 3652.959208] BTRFS error (device dm-0): failed to read block groups: -5
[ 3653.002227] BTRFS error (device dm-0): open_ctree failed
[ 3876.808183] BTRFS info (device dm-0): use zlib compression, level 3
[ 3876.808192] BTRFS info (device dm-0): disk space caching is enabled
[ 3876.808195] BTRFS info (device dm-0): has skinny extents
[ 3878.140763] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[ 3880.623113] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 3880.633290] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
Quoted 8 lines
[ 3880.633298] BTRFS error (device dm-0): failed to read block groups: -5
[ 3880.669435] BTRFS error (device dm-0): open_ctree failed
[ 4057.606879] BTRFS info (device dm-0): use zlib compression, level 3
[ 4057.606890] BTRFS info (device dm-0): disk space caching is enabled
[ 4057.606894] BTRFS info (device dm-0): has skinny extents
[ 4058.886212] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[ 4061.501589] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 4061.503790] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661

The amount of damage here is small, but it looks like you lost a superblock update or two (btrfs expected an old page and found a new one) so the root of the filesystem now points at an old tree that has since been partially overwritten.

Any part of the filesystem on the other side of the missing metadata pages is no longer accessible without a brute force search of the metadata. This might take longer than mkfs+restore with the current btrfs check --repair, especially if you have to run chunk-recover as well.

Quoted 9 lines
myth:/mnt# btrfs-zero-log /dev/mapper/crypt_bcache0
WARNING: this utility is deprecated, please use 'btrfs rescue zero-log'
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
Ignoring transid failure
leaf parent key incorrect 7325633544192
Clearing log on /dev/mapper/crypt_bcache0, previous log_root 0, level 0

Clearing log tree will have no effect on parent transid verify failure. It only helps to work around bugs that occur during log tree replay, which happens at a later stage of mounting the filesystem.

Quoted 49 lines
myth:/mnt# mount -t btrfs -o recovery,nospace_cache,clear_cache /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
mount: wrong fs type, bad option, bad superblock on /dev/mapper/crypt_bcache0,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so
myth:/mnt# dmtail
[ 6665.975324] BTRFS info (device dm-0): unrecognized mount option 'rootflags=recovery'
[ 6665.975357] BTRFS error (device dm-0): open_ctree failed
[ 6686.664202] BTRFS warning (device dm-0): 'recovery' is deprecated, use 'usebackuproot' instead
[ 6686.664213] BTRFS info (device dm-0): trying to use backup root at mount time
[ 6686.664219] BTRFS info (device dm-0): disabling disk space caching
[ 6686.664224] BTRFS info (device dm-0): force clearing of disk cache
[ 6686.664232] BTRFS info (device dm-0): has skinny extents
[ 6687.911926] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[ 6690.522785] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 6690.529495] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 6690.529504] BTRFS error (device dm-0): failed to read block groups: -5
[ 6690.556227] BTRFS error (device dm-0): open_ctree failed
myth:/mnt# btrfs restore /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
Skipping snapshot win_ro.20200615_02:49:02
Skipping snapshot 0Notmachines_ro.20200626_16:09:30
Skipping snapshot 1Appliances_ro.20200626_17:08:56
Skipping snapshot debian32_ro.20200626_17:11:57
Skipping snapshot debian64_ro.20200626_17:18:15
Skipping snapshot ubuntu_ro.20200626_17:18:44
Skipping snapshot win_ro.20200626_18:39:13
Skipping snapshot 0Notmachines_ro.20200629_00:48:39
Skipping snapshot 1Appliances_ro.20200629_01:11:44
Skipping snapshot debian32_ro.20200629_01:11:50
Skipping snapshot debian64_ro.20200629_01:19:52
Skipping snapshot ubuntu_ro.20200629_01:20:01
Error copying data for /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
Error searching /mnt/btrfs_bigbackup/DS2/backup-btrfssend/gargamel/root_last_ro/lib/modules/5.1.19-amd64-preempt-sysrq-20180818/kernel/drivers/video/fbdev/arkfb.ko
myth:/mnt# l /mnt/btrfs_bigbackup/
is missing a lot of stuff
Quoted 3 lines

It's a backup server, I can recreate the data, but it will probably take 2 weeks to copy everything again, and I'd like to know what on earth happened so that I can avoid having this again.

"parent transid verify failed" is btrfs's gentle way of saying "your devices have busted write ordering, reconfigure or replace them and try again."

In a later post in this thread, you posted a log showing your drive model numbers and firmware revisions. White Label repackages drives made by other companies(*) under their own part number (singular, they all use the same part number), but they leave the firmware revision intact, so we can look up the firmware revision and see that you have two different WD models in your md6 array from families with known broken firmware write caching.

sdd is running firmware 0957, also found in circa-2014 WD Green. The others are running 01.01RA2 firmware that appears in a model family that includes some broken WD Green and Red models from a few years back (including the venerable datavore 80.00A80). I have a few of the WD branded versions of these drives. They are unusable with write cache enabled. 1 in 10 unclean shutdowns lead to filesystem corruption on btrfs; on ext4, git and postgresql database corruption. After disabling write cache, I've used them for years with no problems.

Hopefully your bcache drive is OK, you didn't post any details on that. bcache on a drive with buggy firmware write caching fails *spectacularly*.

You can work around buggy write cache firmware with a udev rule like this to disable write cache on all the drives:

        ACTION=="add|change", SUBSYSTEM=="block", DRIVERS=="sd", KERNEL=="sd*[!0-9]", RUN+="/sbin/hdparm -W 0 $devnode"

Note that in your logs, the kernel reports that 'sdd' has write cache disabled already, maybe due to lack of firmware support or a conservative default setting. That makes it probably the only drive in that array that is working properly.

bcache could be losing its mind too, but although I've heard a lot of rumors of bcache bugs, I've yet to catch it having a problem that wasn't directly caused by bad SSD firmware or host configuration. If the bcache was configured in writeback mode and it was separated from the backing device for a while then there could be a consistency issue that would result in something like this, but bcache is pretty good at preventing that.

In theory, space_cache=v1 might have consistency issues that lead to some or all of the symptoms above; however, a) btrfs has checks at multiple points to detect or prevent that, and b) the inconsistency would have to be caused by a firmware write cache bug or bcache bug anyway.

There are some other questionable things in your setup: you have a mdadm-raid5 with no journal device, so if PPL is also not enabled, and you are running btrfs on top, then this filesystem is vulnerable to instant destruction by mdadm-raid5 write hole after a disk fails. bcache in writeback mode with a single cache pool using multiple physical backing store devices is vulnerable to extra data corruption failures in the event that the SSD(s) go bad. I'm guessing that none of the backing store drives have scterc support, which will complicate error recovery with SATA bus timeouts and resets as disks fail (though most of the problems with that are conveniently also prevented by disabling write cache). None of these issues caused problems today, though, and won't cause a problem until disks start to fail.

(*) their product description text says "other companies", but maybe White Label is just a part of WD, hiding their shame as they dispose of unsalable inventory in an unsuspecting market. Don't know, don't care enough to find out.

Quoted 7 lines

Thanks,
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#4 lore raw

Hi Zygo, thanks for the very detailled answer.

A few replies:

Quoted 2 lines

Hopefully your bcache drive is OK, you didn't post any details on that. bcache on a drive with buggy firmware write caching fails *spectacularly*.

So, I created this as a bcache backing device so that I can elect to add a caching device later (which you can't do after the fact otherwise). There was no caching device here, though.

So before we go on, it seems that my FS is damaged beyond easy repair, that we know why (drives with broken write caching) and that unfortunately btrfs gets in states where recovery is hard, at best. If so, I'll just wipe, apply the fixes you recommended and start over.

Quoted 6 lines · Zygo Blaxell
Quoted 2 lines
[ 2575.931343] BTRFS info (device dm-0): has skinny extents
[ 2577.286749] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0

This line does indicate an older problem with the filesystem. It doesn't tell us whether the corruption happened yesterday or a year ago. You will need to look at your older kernel logs for that.

Yeah, this could have been long time ago, but remember that I ran a 2 day long btrfs check on it and it came back clean.

Quoted 5 lines
Quoted 1 line
[ 2882.545242] BTRFS info (device dm-0): the free space cache file (26234763345920) is invalid, skip it

Use space_cache=v2, especially on a big filesystem because space_cache=v1 slows down linearly with filesystem size. There is no need for these warnings. They're probably a symptom rather than cause here.

Thanks for that, was not aware of this.

Quoted 4 lines
Quoted 1 line

Ok, maybe there was an IO failure, although none was shown by the kernel:

The "IO failure" mentioned here is the earlier parent transid verify failure. When the verification fails, the caller gets -EIO.

Got it, so not a de ice IO failure.

Quoted 9 lines
Quoted 3 lines
[ 4058.886212] BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[ 4061.501589] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661
[ 4061.503790] BTRFS error (device dm-0): parent transid verify failed on 7325633544192 wanted 359658 found 359661

The amount of damage here is small, but it looks like you lost a superblock update or two (btrfs expected an old page and found a new one) so the root of the filesystem now points at an old tree that has since been partially overwritten.

Quoted 4 lines

Any part of the filesystem on the other side of the missing metadata pages is no longer accessible without a brute force search of the metadata. This might take longer than mkfs+restore with the current btrfs check --repair, especially if you have to run chunk-recover as well.

unfortunately btrfs check errors early on, so that's not going to work without manual options which seem not very obvious.

Quoted 10 lines
Quoted 9 lines
myth:/mnt# btrfs-zero-log /dev/mapper/crypt_bcache0
WARNING: this utility is deprecated, please use 'btrfs rescue zero-log'
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
parent transid verify failed on 7325633544192 wanted 359658 found 359661
Ignoring transid failure
leaf parent key incorrect 7325633544192
Clearing log on /dev/mapper/crypt_bcache0, previous log_root 0, level 0
Quoted 3 lines

Clearing log tree will have no effect on parent transid verify failure. It only helps to work around bugs that occur during log tree replay, which happens at a later stage of mounting the filesystem.

Right, I was just a bit desperate and tried it :)

Quoted 7 lines

In a later post in this thread, you posted a log showing your drive model numbers and firmware revisions. White Label repackages drives made by other companies(*) under their own part number (singular, they all use the same part number), but they leave the firmware revision intact, so we can look up the firmware revision and see that you have two different WD models in your md6 array from families with known broken firmware write caching.

Dann, I'm impressed by that analysis, thank you.

Quoted 4 lines

branded versions of these drives. They are unusable with write cache enabled. 1 in 10 unclean shutdowns lead to filesystem corruption on btrfs; on ext4, git and postgresql database corruption. After disabling write cache, I've used them for years with no problems.

Gotcha, I'm very glad you were able to figure that out. As you said, I can disable the write cache.
It's a bit sad however that ext4 would have given me something I could have recovered, while with btrfs, it's so much harder to recover if you're not a btrfs FS datastructure expert.

Quoted 2 lines

There are some other questionable things in your setup: you have a mdadm-raid5 with no journal device, so if PPL is also not enabled,

Sorry, PPL?

Quoted 2 lines

and you are running btrfs on top, then this filesystem is vulnerable to instant destruction by mdadm-raid5 write hole after a disk fails.

wait, if a disk fails, at worst I have a stripe that's half written and hopefully btrfs fails, goes read only and the transaction does not go through, so nothing happens except loss of the last written data?

I don't have an external journal because this is an external disk array I can move between machines. Would you suggest I do something else?

Quoted 4 lines

(*) their product description text says "other companies", but maybe White Label is just a part of WD, hiding their shame as they dispose of unsalable inventory in an unsuspecting market. Don't know, don't care enough to find out.

:)

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#5 lore raw
Quoted 10 lines · Marc MERLIN
Quoted 4 lines

branded versions of these drives. They are unusable with write cache enabled. 1 in 10 unclean shutdowns lead to filesystem corruption on btrfs; on ext4, git and postgresql database corruption. After disabling write cache, I've used them for years with no problems.

Gotcha, I'm very glad you were able to figure that out. As you said, I can disable the write cache.
It's a bit sad however that ext4 would have given me something I could have recovered, while with btrfs, it's so much harder to recover if you're not a btrfs FS datastructure expert.

ext2 is pretty much indestructible if you ignore data integrity. If you want to destroy ext2 metadata, you have to overwrite it one bit at a time. The few cases where damage is not proportional to write size (indirect block lists, directories) are limited to single inodes. Every metadata item appears in exactly one location that can be computed, no searching required. Recovering from arbitrarily awful firmware is one of the few things ext2 is really good at.

ext4 is significantly more fragile than ext2 if you use the newer features like extent and flex_bg, which make the metadata mobile like btrfs. These can prevent e2fsck from being able to recover a large filesystem.

I switched to btrfs back in 2014 after losing a couple of big ext4 filesystems on arrays of WD Greens, _then_ discovered the problem was the disk firmware not the filesystem.

Quoted 4 lines
Quoted 2 lines

There are some other questionable things in your setup: you have a mdadm-raid5 with no journal device, so if PPL is also not enabled,

Sorry, PPL?

Partial Parity Log. It can be enabled by mdadm --grow. It's a mdadm consistency policy, like the journal, but uses reserved metadata space instead of a separate device.

Quoted 6 lines
Quoted 2 lines

and you are running btrfs on top, then this filesystem is vulnerable to instant destruction by mdadm-raid5 write hole after a disk fails.

wait, if a disk fails, at worst I have a stripe that's half written and hopefully btrfs fails, goes read only and the transaction does not go through, so nothing happens except loss of the last written data?

If the array is degraded, and stripe is partially updated, then there is a crash or power failure, parity will be out of sync with data blocks in the stripe, so the missing disk's data cannot be generated from parity.

Both old and new data can be damaged by raid5 write hole. The data that is damaged is the block on the missing disk that must be computed using the contents of all other disks. The damage affects old and new data the stripe with equal probability, as the data and parity blocks rotate from one stripe to the next. Damaged data in an uncommitted tree (new data) will be ignored if the transaction is not completed, as no reference to the root of the uncommitted tree will exist after a crash. Damaged data in a committed tree (old data) is already committed, and if it's metadata the damage will also break the filesystem. In other words, only old data can be damaged by the write hole, because any new damaged data will be filtered out by the transaction mechanism.

If you have dup metadata in the btrfs then maybe you can recover from the mirror copy in another stripe. Hopefully that's not damaged too, but since both mirrors are updated at roughly the same time on the same disks, damage to both copies is quite likely.

mdadm PPL or the journal device finishes partial stripe updates after a crash or power failure, and avoids this failure mode.

Quoted 2 lines

I don't have an external journal because this is an external disk array I can move between machines. Would you suggest I do something else?

Enable PPL on mdadm, or use btrfs raid5 data + raid1 metadata (it's barely usable and some stuff doesn't work properly, but it can run a backup server, replace a failed disk, and usually self-repair disk corruption too).

Quoted 13 lines
Quoted 4 lines

(*) their product description text says "other companies", but maybe White Label is just a part of WD, hiding their shame as they dispose of unsalable inventory in an unsuspecting market. Don't know, don't care enough to find out.

:)

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#6 lore raw
Quoted 5 lines · Zygo Blaxell
Quoted 1 line

Sorry, PPL?

Partial Parity Log. It can be enabled by mdadm --grow. It's a mdadm consistency policy, like the journal, but uses reserved metadata space instead of a separate device.

looks like it's incompatible with --bitmap which I was already using. I'm not sure if --grow was possible to convert from one to another, so I made a new one, and now I have a 4 day wait for the initial scan (external array with Sata PMP, those are slow unfortunately)

md6 : active raid5 sdx1[5] sdw1[3] sdv1[2] sdu1[1] sdq1[0]

      23441547264 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [UUUU_]
      [>....................]  recovery =  0.0% (2920544/5860386816) finish=4835.4min speed=20189K/sec

Could you confirm that what I did here is ok?

gargamel:~# mdadm --create --verbose /dev/md6 --level=5 --consistency-policy=ppl --raid-devices=5 /dev/sd[quvwx]1
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 512K
mdadm: /dev/sdq1 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
mdadm: /dev/sdu1 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
mdadm: /dev/sdv1 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
mdadm: /dev/sdw1 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
mdadm: /dev/sdx1 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
mdadm: size set to 5860386816K
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md6 started.
gargamel:~# make-bcache -B /dev/md6
UUID:                   eb898c09-debd-4e86-972f-aecdb59670e2
Set UUID:               e14408a3-8e25-414d-ba4a-a8a7a0d7bdc9
version:                1
block_size:             1
data_offset:            16
gargamel:~# cryptsetup luksFormat --align-payload=2048 -s 256 -c aes-xts-plain64 /dev/bcache4
gargamel:~# cryptsetup luksOpen /dev/bcache4 dshelf6
Enter passphrase for /dev/bcache4:
gargamel:~# /sbin/mkfs.btrfs -L DS6 -O extref -m dup /dev/mapper/dshelf6 
Quoted 7 lines
Quoted 3 lines

wait, if a disk fails, at worst I have a stripe that's half written and hopefully btrfs fails, goes read only and the transaction does not go through, so nothing happens except loss of the last written data?

If the array is degraded, and stripe is partially updated, then there is a crash or power failure, parity will be out of sync with data blocks in the stripe, so the missing disk's data cannot be generated from parity.

I thought --bitmap would know this and know to discard the last blocks partiailly written. I guess not?

Either way, it seems that losing just a few blocks in the "wrong" place is enough to lose most of a btrfs FS. That's disappointing, I thought it was a bit more fault proof than that.

Quoted 7 lines
Quoted 2 lines

I don't have an external journal because this is an external disk array I can move between machines. Would you suggest I do something else?

Enable PPL on mdadm, or use btrfs raid5 data + raid1 metadata (it's barely usable and some stuff doesn't work properly, but it can run a backup server, replace a failed disk, and usually self-repair disk corruption too).

it's been a while since I've used btrfs raid5, it must have improved since I last did, but I haven't read that it's become production quality yet :)

Thanks again for your very helpful answers.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#7 lore raw
Quoted 9 lines · On Wed, 8 Jul 2020 10:41:10 -0700 Marc MERLIN <[email protected]> wrote:
Quoted 5 lines · Zygo Blaxell
Quoted 1 line

Sorry, PPL?

Partial Parity Log. It can be enabled by mdadm --grow. It's a mdadm consistency policy, like the journal, but uses reserved metadata space instead of a separate device.

looks like it's incompatible with --bitmap which I was already using. I'm not sure if --grow was possible to convert from one to another

At least you can definitely remove the bitmap with --bitmap=none in the grow mode. Then in the second --grow can probably add PPL (unfamiliar with that).

Signature · 2 lines
With respect,
Roman
#8 lore raw
In reply to Marc MERLIN
Quoted 44 lines · Marc MERLIN
Quoted 5 lines · Zygo Blaxell
Quoted 1 line

Sorry, PPL?

Partial Parity Log. It can be enabled by mdadm --grow. It's a mdadm consistency policy, like the journal, but uses reserved metadata space instead of a separate device.

looks like it's incompatible with --bitmap which I was already using. I'm not sure if --grow was possible to convert from one to another, so I made a new one, and now I have a 4 day wait for the initial scan (external array with Sata PMP, those are slow unfortunately)

md6 : active raid5 sdx1[5] sdw1[3] sdv1[2] sdu1[1] sdq1[0]

      23441547264 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [UUUU_]
      [>....................]  recovery =  0.0% (2920544/5860386816) finish=4835.4min speed=20189K/sec

Could you confirm that what I did here is ok?

gargamel:~# mdadm --create --verbose /dev/md6 --level=5 --consistency-policy=ppl --raid-devices=5 /dev/sd[quvwx]1
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 512K
mdadm: /dev/sdq1 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
mdadm: /dev/sdu1 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
mdadm: /dev/sdv1 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
mdadm: /dev/sdw1 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
mdadm: /dev/sdx1 appears to be part of a raid array:
       level=raid5 devices=5 ctime=Thu Jan 28 14:38:40 2016
mdadm: size set to 5860386816K
Continue creating array? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md6 started.
gargamel:~# make-bcache -B /dev/md6
UUID:                   eb898c09-debd-4e86-972f-aecdb59670e2
Set UUID:               e14408a3-8e25-414d-ba4a-a8a7a0d7bdc9
version:                1
block_size:             1
data_offset:            16
Quoted 6 lines
gargamel:~# cryptsetup luksFormat --align-payload=2048 -s 256 -c aes-xts-plain64 /dev/bcache4
gargamel:~# cryptsetup luksOpen /dev/bcache4 dshelf6
Enter passphrase for /dev/bcache4:
gargamel:~# /sbin/mkfs.btrfs -L DS6 -O extref -m dup /dev/mapper/dshelf6 

Looks OK.

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

wait, if a disk fails, at worst I have a stripe that's half written and hopefully btrfs fails, goes read only and the transaction does not go through, so nothing happens except loss of the last written data?

If the array is degraded, and stripe is partially updated, then there is a crash or power failure, parity will be out of sync with data blocks in the stripe, so the missing disk's data cannot be generated from parity.

Quoted 6 lines

I thought --bitmap would know this and know to discard the last blocks partiailly written. I guess not?

Either way, it seems that losing just a few blocks in the "wrong" place is enough to lose most of a btrfs FS. That's disappointing, I thought it was a bit more fault proof than that.

During a write to a stripe, there's a short window of time when data and parity blocks are out of sync (each disk completes writes at its own speed). If the system is interrupted during this time (crash or power failure), the stripe on disk has data blocks that don't match the parity block.

Suppose we have 5 disks. There is always a 1:1 mapping of disks to blocks in every RAID5 stripe, so let's call the data blocks on each disk A, B, C, D, and the parity block P, and ignore the other details.

When all the disks are online, before we do a write to D, we have:

A ^ B ^ C ^ Dold = Pold (1)

After the first write, we have this on disk:

A, B, C, Dnew, Pold

After the second write, we have this on disk:

A, B, C, Dnew, Pnew

A ^ B ^ C ^ Dnew = Pnew (2)

If we lose drive C, we can get it back from either the first or last state:

A ^ B ^ Cmissing ^ Dold = Pold

so

Cmissing = Pold ^ A ^ B ^ Dold (3)

Sub (1) into (3):

Cmissing = (A ^ B ^ C ^ Dold) ^ A ^ B ^ Dold

Sub A ^ A = 0, B ^ B = 0, Dold ^ Dold = 0:

Cmissing = C

So we recover C correctly. The same can be done with Pnew and Dnew.

If there is a power failure or crash and the second write doesn't happen, then we get:

Cgarbage = Pold ^ A ^ B ^ Dnew (4)

Sub (1) into (4):

Cgarbage = (A ^ B ^ C ^ Dold) ^ A ^ B ^ Dnew

Cancel A ^ A = 0, B ^ B = 0:

Cgarbage = C ^ Dold ^ Dnew

We didn't get C back, we got C xored with some other stuff.

If the writes happen in the other order, we get:

Cgarbage2 = Pnew ^ A ^ B ^ Dold (5)

Sub (2) into (5):

Cgarbage2 = (A ^ B ^ C ^ Dnew) ^ A ^ B ^ Dold

Cancel A ^ A = 0, B ^ B = 0:

Cgarbage2 = C ^ Dnew ^ Dold

We didn't get C back here either. With only 2 disks written, we get the same garbage in both cases. If there are more than 2 disks written then we get multiple possible combinations of writes xored together.

If the array is not degraded, then after the crash, mdadm assumes the data blocks are correct, and recomputes the parity block from the data blocks during resync (i.e. it updates Pold to Pnew using Dnew). The filesystem looks only at the data blocks, and all the data blocks are present, so this works.

The mdadm bitmaps speed this up by indicating which areas of the disk contain dirty stripes that may require parity recomputation. Areas that have not been written don't need to be resynced.

If the array is degraded, some of the data blocks are missing, so we can't recompute Pnew from whichever of A, B, C, and D blocks are present. If we have Pold and Dold it's OK, we can correctly compute A, B, or C if one is missing. If we have Pnew and Dnew it's OK. If P is on the missing disk it's OK, because we have all the data blocks in the stripe intact so we don't need P. If D is missing, we will recompute either Dnew or Dold depending on whether the P block contains Pnew or Pold, but btrfs transactions can handle both results so it's OK no matter what D contains. So far so good, all these cases are OK. PPL ensures we only have these cases.

If we have Pold and Dnew or Pnew and Dold, then the computed contents of A, B, and C will be broken when any of those drives are missing. Since we weren't writing to blocks A, B, or C in this transaction, but those are the blocks that are corrupted, we may have broken something btrfs already committed. btrfs will see broken csums and reject the data. If A, B, or C are btrfs metadata blocks, the filesystem is damaged. Either a mirror copy of metadata or a practical fsck is required to repair.

btrfs transactions cover only the specific blocks that were written. They don't handle cases where a write to block D corrupts data in other blocks like A, B, or C. raid5 in degraded mode has cases like that, so it doesn't work with btrfs.

Quoted 11 lines
Quoted 7 lines
Quoted 2 lines

I don't have an external journal because this is an external disk array I can move between machines. Would you suggest I do something else?

Enable PPL on mdadm, or use btrfs raid5 data + raid1 metadata (it's barely usable and some stuff doesn't work properly, but it can run a backup server, replace a failed disk, and usually self-repair disk corruption too).

it's been a while since I've used btrfs raid5, it must have improved since I last did, but I haven't read that it's become production quality yet :)

Well, don't try to run a live customer orders database on btrfs raid5. That won't end well. It can hold a backup copy of the customer orders database though, provided you follow the restrictions in

	https://lore.kernel.org/linux-btrfs/[email protected]/

and avoid (or work around) drives with write-caching bugs.

Quoted 7 lines

Thanks again for your very helpful answers.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#9 lore raw
In reply to Zygo Blaxell
Quoted 15 lines · Zygo Blaxell

sdd is running firmware 0957, also found in circa-2014 WD Green. The others are running 01.01RA2 firmware that appears in a model family that includes some broken WD Green and Red models from a few years back (including the venerable datavore 80.00A80). I have a few of the WD branded versions of these drives. They are unusable with write cache enabled. 1 in 10 unclean shutdowns lead to filesystem corruption on btrfs; on ext4, git and postgresql database corruption. After disabling write cache, I've used them for years with no problems.

Hopefully your bcache drive is OK, you didn't post any details on that. bcache on a drive with buggy firmware write caching fails *spectacularly*.

You can work around buggy write cache firmware with a udev rule like this to disable write cache on all the drives:

Quoted 6 lines
        ACTION=="add|change", SUBSYSTEM=="block", DRIVERS=="sd", KERNEL=="sd*[!0-9]", RUN+="/sbin/hdparm -W 0 $devnode"

Note that in your logs, the kernel reports that 'sdd' has write cache disabled already, maybe due to lack of firmware support or a conservative default setting. That makes it probably the only drive in that array that is working properly.

Hi Zygo, took a while to rebuild the array, but it's back up, thanks for your tips.

To avoid such pain in the future, is there a way for me to find out if my other drives have such problems so that I disable write caching on them now?

WDC WD10EADS-00L 01.0 PQ: 0 ANSI: 6
SAMSUNG HD102UJ 1AA0 PQ: 0 ANSI: 6
ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5
WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5
WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5
WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5
WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5

As for the failure you helped me with (thanks):
In hindsight, I think I know what went wrong. I was running bees on the array to reclaim duplicate space (1.2TB saved, so it was worth it). bees does lots of metadata operations, so that would explain why the last operations that didn't get saved/got corrupted caused the fatal corruption I ended up with.

Because it's an external array, I do stop it to save power and do this:
umount /dev/mapper/crypt_bcache0
sync
dmsetup remove crypt_bcache0
echo 1 > /sys/block/md6/bcache/stop
mdadm --stop /dev/md6
/etc/init.d/smartmontools stop
sleep 5
pdu disk3 off <= cuts power

Note that I umount, then sync just in case, then a bunch of stuff to free up block devices, but by then the btrfs FS has been unmounted and synced.
Yet, I added sleep 5 for good measure before turning the drives off. Do you think the cache was so broken that even with all the steps I took, it failed to flush?

Either way, I now do this as per your recommenation:
grep md6 /proc/mdstat | sed "s/.*raid5 //" | tr ' ' '\012' | sed "s/1.*//" | while read d; do /sbin/hdparm -v -W 0 /dev/$d; done

Thanks
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08
#10 lore raw
Quoted 10 lines · On Wed, 12 Aug 2020 15:34:33 -0700 Marc MERLIN <[email protected]> wrote:

Because it's an external array, I do stop it to save power and do this:
umount /dev/mapper/crypt_bcache0
sync
dmsetup remove crypt_bcache0
echo 1 > /sys/block/md6/bcache/stop
mdadm --stop /dev/md6
/etc/init.d/smartmontools stop
sleep 5
pdu disk3 off <= cuts power

Quoted 3 lines

Note that I umount, then sync just in case, then a bunch of stuff to free up block devices, but by then the btrfs FS has been unmounted and synced.

Or should have been. If it's a copy-paste from a shell script (unless you do the whole process manually every time), it doesn't appear like you check the success status of each line before continuing to the next. Either add "|| exit 1" to each of these that can fail, or just use "#!/bin/bash -e" in the script's first line.

As is, imagine umount fails ("target busy"), then dmsetup fails ("device in use"), then mdadm fails to stop the array, but then you cut power to the entire thing anyways.

Signature · 2 lines
With respect,
Roman
#11 lore raw
Quoted 5 lines · Roman Mamedov

Or should have been. If it's a copy-paste from a shell script (unless you do the whole process manually every time), it doesn't appear like you check the success status of each line before continuing to the next. Either add "|| exit 1" to each of these that can fail, or just use "#!/bin/bash -e" in the script's first line.

Sorry, yeah, it's a partial copy paste of a script, but the script has set -e
so if any command fails, the script stops.

Quoted 3 lines

As is, imagine umount fails ("target busy"), then dmsetup fails ("device in use"), then mdadm fails to stop the array, but then you cut power to the entire thing anyways.

Correct, and set -e makes sure that does not happen.

I have been using this script for years, and it's been fine, but if the drives have broken write caching and can fail to write data for more than 5 seconds, then my script will have cut the power on them before data was committed.
It feels far reaching a conclusion, but given that we already know that I did get said corruption and I'm pretty darn confident that it happened after the last time I shut down the array that way, it means that Zygo is very likely right on the write caching problem.

Which also also why I'd like to check if I need to turn it off on any on my other drives, since indeed I often bought cheaper versions of drives, which maybe had firmware bugs.

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#12 lore raw
Quoted 26 lines · Marc MERLIN
Quoted 5 lines · Roman Mamedov

Or should have been. If it's a copy-paste from a shell script (unless you do the whole process manually every time), it doesn't appear like you check the success status of each line before continuing to the next. Either add "|| exit 1" to each of these that can fail, or just use "#!/bin/bash -e" in the script's first line.

Sorry, yeah, it's a partial copy paste of a script, but the script has set -e
so if any command fails, the script stops.

Quoted 3 lines

As is, imagine umount fails ("target busy"), then dmsetup fails ("device in use"), then mdadm fails to stop the array, but then you cut power to the entire thing anyways.

Correct, and set -e makes sure that does not happen.

I have been using this script for years, and it's been fine, but if the drives have broken write caching and can fail to write data for more than 5 seconds, then my script will have cut the power on them before data was committed.
It feels far reaching a conclusion, but given that we already know that I did get said corruption and I'm pretty darn confident that it happened after the last time I shut down the array that way, it means that Zygo is very likely right on the write caching problem.

Quoted 3 lines

Which also also why I'd like to check if I need to turn it off on any on my other drives, since indeed I often bought cheaper versions of drives, which maybe had firmware bugs.

It would be nice if price correlated to firmware quality, but that's not how disk vendors make drives. There's a firmware team that produces firmware for a disk controller SoC family. It's a lot of expensive work, so it only happens once unless a major defect is found. That firmware ends up in all the drive models built around that SoC family. So you end up with White Label, WD Green, and WD Red drive models that all run the same handful of firmware builds even though there are dozens of distinct model numbers sold to different market segments (i.e. prices) over a period of many years.

WD didn't think it was important to avoid SMR in NAS drives--a requirement that was so obvious that customers ultimately sued them over it. WD certainly wouldn't think it was important for their NAS drives to have more correct firmware than the low-end discount model, and nobody sued them over it, so they all got shipped with the same bugs.

Quoted 5 lines

Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#13 lore raw
Quoted 2 lines · Zygo Blaxell

That "0A80" is the last 4 bytes of "80.00A80" which I mentioned above. They all need write cache turned off.

Cool. Those were different drives, not sure if they had the same bug.

Quoted 4 lines

I'm told that SC61 firmware has a bug fix for SC60's write cache issues; however, I have some SC60 drives and haven't had write-cache problems with them. They're probably fine. Same for WD 01.01A01--some WD Green drives are OK. I have no data for Samsung firmware.

So, as incredibly thankful as I am for you having this information, is that stuff you just google?
Is there a DB of bad firmwares, and does linux maybe have a partial one to turn off write caching on known bad drives, or am I talking crazy and in happy happy land? :)

Quoted 3 lines

bees typically only spends a single-digit percentage of its time manipulating metadata. Most of the bees IO time is waiting for btrfs to do slow metadata reads while holding a transaction lock.

Gotcha.

Quoted 8 lines

Anything after bcache stop is write cache flush theatre. If the drive's write cache firmware is broken, then you can put all the flushes you want, but the flushes don't do anything that wouldn't also be achieved by just waiting for the disks to be idle for a while.

5 seconds seems short to me. I'd keep the drive powered for a minute or two before turning it off. Ideally I'd set an idle timeout on the drive (hdparm -S) and wait until the disk spins itself down (hdparm -C).

I see. I didn't realize it could take that long for things to flush if the firmware isn't doing the job of flushing when it said it did.

Quoted 2 lines

Note that after a bus reset while the disks are online, the write caching flag might revert to default.

oh boy, thanks for the warning.

Quoted 6 lines

Write cache corruption doesn't always occur at shutdown. It can also happen if there is a UNC sector or even noise on the SATA cables that triggers a bus reset. This wouldn't be detected by the host or btrfs--as far as the storage stack is concerned, the drive ACKed the flush, so the data's on the disk, and any losses after that point are equivalent to disk-level corruption.

I see, that makes sense.

Quoted 9 lines

It would be nice if price correlated to firmware quality, but that's not how disk vendors make drives. There's a firmware team that produces firmware for a disk controller SoC family. It's a lot of expensive work, so it only happens once unless a major defect is found. That firmware ends up in all the drive models built around that SoC family. So you end up with White Label, WD Green, and WD Red drive models that all run the same handful of firmware builds even though there are dozens of distinct model numbers sold to different market segments (i.e. prices) over a period of many years.

I see, so I guess I don't feel as bad buying the cheaper white label versions :)

Quoted 5 lines

WD didn't think it was important to avoid SMR in NAS drives--a requirement that was so obvious that customers ultimately sued them over it. WD certainly wouldn't think it was important for their NAS drives to have more correct firmware than the low-end discount model, and nobody sued them over it, so they all got shipped with the same bugs.

Yeah, I saw that, most BS move ever. They definitely deserve the backlash coming with it.

Thanks for your answers.
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/  
#14 lore raw
In reply to Marc MERLIN
Quoted 30 lines · Marc MERLIN
Quoted 15 lines · Zygo Blaxell

sdd is running firmware 0957, also found in circa-2014 WD Green. The others are running 01.01RA2 firmware that appears in a model family that includes some broken WD Green and Red models from a few years back (including the venerable datavore 80.00A80). I have a few of the WD branded versions of these drives. They are unusable with write cache enabled. 1 in 10 unclean shutdowns lead to filesystem corruption on btrfs; on ext4, git and postgresql database corruption. After disabling write cache, I've used them for years with no problems.

Hopefully your bcache drive is OK, you didn't post any details on that. bcache on a drive with buggy firmware write caching fails *spectacularly*.

You can work around buggy write cache firmware with a udev rule like this to disable write cache on all the drives:

Quoted 6 lines
        ACTION=="add|change", SUBSYSTEM=="block", DRIVERS=="sd", KERNEL=="sd*[!0-9]", RUN+="/sbin/hdparm -W 0 $devnode"

Note that in your logs, the kernel reports that 'sdd' has write cache disabled already, maybe due to lack of firmware support or a conservative default setting. That makes it probably the only drive in that array that is working properly.

Hi Zygo, took a while to rebuild the array, but it's back up, thanks for your tips.

To avoid such pain in the future, is there a way for me to find out if my other drives have such problems so that I disable write caching on them now?

Quoted 12 lines

WDC WD10EADS-00L 01.0 PQ: 0 ANSI: 6
SAMSUNG HD102UJ 1AA0 PQ: 0 ANSI: 6
ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
ST6000VN0041-2EL SC61 PQ: 0 ANSI: 6
WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5
WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5
WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5
WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5
WDC WD40EFRX-68W 0A80 PQ: 0 ANSI: 5

That "0A80" is the last 4 bytes of "80.00A80" which I mentioned above. They all need write cache turned off.

I'm told that SC61 firmware has a bug fix for SC60's write cache issues; however, I have some SC60 drives and haven't had write-cache problems with them. They're probably fine. Same for WD 01.01A01--some WD Green drives are OK. I have no data for Samsung firmware.

Quoted 6 lines

As for the failure you helped me with (thanks):
In hindsight, I think I know what went wrong. I was running bees on the array to reclaim duplicate space (1.2TB saved, so it was worth it). bees does lots of metadata operations, so that would explain why the last operations that didn't get saved/got corrupted caused the fatal corruption I ended up with.

If the drive lost data writes then you'd have csum failures on data blocks. Some of my WD Reds with 0A80 firmware had those.

There's a lot of metadata writes all the time on btrfs, even with data-heavy workloads. You only need to lose any two metadata writes (one in single or raid0 profile) to break the filesystem.

bees typically only spends a single-digit percentage of its time manipulating metadata. Most of the bees IO time is waiting for btrfs to do slow metadata reads while holding a transaction lock.

Quoted 10 lines

Because it's an external array, I do stop it to save power and do this:
umount /dev/mapper/crypt_bcache0
sync
dmsetup remove crypt_bcache0
echo 1 > /sys/block/md6/bcache/stop
mdadm --stop /dev/md6
/etc/init.d/smartmontools stop
sleep 5
pdu disk3 off <= cuts power

Quoted 7 lines

Note that I umount, then sync just in case, then a bunch of stuff to free up block devices, but by then the btrfs FS has been unmounted and synced.

Yet, I added sleep 5 for good measure before turning the drives off. Do you think the cache was so broken that even with all the steps I took, it failed to flush?

Anything after bcache stop is write cache flush theatre. If the drive's write cache firmware is broken, then you can put all the flushes you want, but the flushes don't do anything that wouldn't also be achieved by just waiting for the disks to be idle for a while.

5 seconds seems short to me. I'd keep the drive powered for a minute or two before turning it off. Ideally I'd set an idle timeout on the drive (hdparm -S) and wait until the disk spins itself down (hdparm -C).

Quoted 2 lines

Either way, I now do this as per your recommenation:
grep md6 /proc/mdstat | sed "s/.*raid5 //" | tr ' ' '\012' | sed "s/1.*//" | while read d; do /sbin/hdparm -v -W 0 /dev/$d; done

Note that after a bus reset while the disks are online, the write caching flag might revert to default.

Write cache corruption doesn't always occur at shutdown. It can also happen if there is a UNC sector or even noise on the SATA cables that triggers a bus reset. This wouldn't be detected by the host or btrfs--as far as the storage stack is concerned, the drive ACKed the flush, so the data's on the disk, and any losses after that point are equivalent to disk-level corruption.

Quoted 6 lines

Thanks
Marc

Signature · 3 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08