Help requested: btrfs can't read superblock
Dear btrfs team,
I've been asked to help someone with a btrfs problem. They use CentOS 7. I've been told the host was shut down cleanly, but it won't mount a btrfs filesystem when powering on.
Below is what I've done so far. Any guidance would be appreciated!
I booted using SystemRescue 11.0. Some info about the rescue OS:
# uname -a Linux sysrescue 6.6.14-1-lts #1 SMP PREEMPT_DYNAMIC Fri, 26 Jan 2024 11:54:58 +0000 x86_64 GNU/Linux
# btrfs --version btrfs-progs v6.6.3
Here are the disks:
# btrfs fi show Label: '<redacted>' uuid: 834b4bfd-fbd4-40b9-8868-d1187c3c5a63 Total devices 4 FS bytes used 17.44TiB devid 1 size 9.10TiB used 8.94TiB path /dev/sda devid 2 size 9.10TiB used 8.94TiB path /dev/sdb devid 3 size 9.10TiB used 8.94TiB path /dev/sdc devid 4 size 9.10TiB used 8.94TiB path /dev/sdd
I tried mounting:
# mount -t btrfs -o recovery,ro /dev/sda /mnt/data
mount: /mnt/data: can't read superblock on /dev/sda.
dmesg(1) may have more information after failed mount system call.dmesg shows this:
[ 1940.962632] BTRFS info (device sda): first mount of filesystem 834b4bfd-fbd4-40b9-8868-d1187c3c5a63 [ 1940.962665] BTRFS info (device sda): using crc32c (crc32c-intel) checksum algorithm [ 1940.962686] BTRFS warning (device sda): 'recovery' is deprecated, use 'rescue=usebackuproot' instead [ 1940.962692] BTRFS info (device sda): trying to use backup root at mount time [ 1940.962698] BTRFS info (device sda): disk space caching is enabled [ 1942.115796] BTRFS critical (device sda): corrupt leaf: root=2 block=11438351450112 slot=170, invalid key objectid, have 5822947745796 expect to be aligned to 4096 [ 1942.115819] BTRFS error (device sda): read time tree block corruption detected on logical 11438351450112 mirror 2 [ 1942.116170] BTRFS critical (device sda): corrupt leaf: root=2 block=11438351450112 slot=170, invalid key objectid, have 5822947745796 expect to be aligned to 4096 [ 1942.116183] BTRFS error (device sda): read time tree block corruption detected on logical 11438351450112 mirror 1 [ 1942.116212] BTRFS error (device sda): failed to read block groups: -5 [ 1942.140369] BTRFS error (device sda): open_ctree failed
I tried to recover the superblock:
# btrfs rescue super-recover -v /dev/sda
All Devices:
Device: id = 3, name = /dev/sdc
Device: id = 2, name = /dev/sdb
Device: id = 4, name = /dev/sdd
Device: id = 1, name = /dev/sda
Before Recovering:
[All good supers]:
device name = /dev/sdc
superblock bytenr = 65536
device name = /dev/sdc
superblock bytenr = 67108864
device name = /dev/sdc
superblock bytenr = 274877906944
device name = /dev/sdb
superblock bytenr = 65536
device name = /dev/sdb
superblock bytenr = 67108864
device name = /dev/sdb
superblock bytenr = 274877906944
device name = /dev/sdd
superblock bytenr = 65536
device name = /dev/sdd
superblock bytenr = 67108864
device name = /dev/sdd
superblock bytenr = 274877906944
device name = /dev/sda
superblock bytenr = 65536
device name = /dev/sda
superblock bytenr = 67108864
device name = /dev/sda
superblock bytenr = 274877906944
[All bad supers]:
All supers are valid, no need to recover
I checked the filesystem:
# btrfs check --readonly /dev/sda Opening filesystem to check... corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608) corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608) corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608) corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608) ERROR: failed to read block groups: Operation not permitted ERROR: cannot open file system
Quoted 25 lines · 在 2024/3/19 07:52, [email protected] 写道:
Dear btrfs team,
I've been asked to help someone with a btrfs problem. They use CentOS 7. I've been told the host was shut down cleanly, but it won't mount a btrfs filesystem when powering on.
Below is what I've done so far. Any guidance would be appreciated!
I booted using SystemRescue 11.0. Some info about the rescue OS:
# uname -a Linux sysrescue 6.6.14-1-lts #1 SMP PREEMPT_DYNAMIC Fri, 26 Jan 2024 11:54:58 +0000 x86_64 GNU/Linux# btrfs --version btrfs-progs v6.6.3Here are the disks:
# btrfs fi show Label: '<redacted>' uuid: 834b4bfd-fbd4-40b9-8868-d1187c3c5a63 Total devices 4 FS bytes used 17.44TiB devid 1 size 9.10TiB used 8.94TiB path /dev/sda devid 2 size 9.10TiB used 8.94TiB path /dev/sdb devid 3 size 9.10TiB used 8.94TiB path /dev/sdc devid 4 size 9.10TiB used 8.94TiB path /dev/sdd
Quoted 5 lines
I tried mounting:
# mount -t btrfs -o recovery,ro /dev/sda /mnt/data mount: /mnt/data: can't read superblock on /dev/sda. dmesg(1) may have more information after failed mount system call.
Use "mount -o rescue=all,ro" instead.
Quoted 9 lines
dmesg shows this:
[ 1940.962632] BTRFS info (device sda): first mount of filesystem 834b4bfd-fbd4-40b9-8868-d1187c3c5a63 [ 1940.962665] BTRFS info (device sda): using crc32c (crc32c-intel) checksum algorithm [ 1940.962686] BTRFS warning (device sda): 'recovery' is deprecated, use 'rescue=usebackuproot' instead [ 1940.962692] BTRFS info (device sda): trying to use backup root at mount time [ 1940.962698] BTRFS info (device sda): disk space caching is enabled [ 1942.115796] BTRFS critical (device sda): corrupt leaf: root=2 block=11438351450112 slot=170, invalid key objectid, have 5822947745796 expect to be aligned to 4096
Bad extent item start bytenr, 5822947745796 = 0x54bc2bb6004, the last 0x004 looks like a bitflip.
Please run memtest to make sure your hardware memory is fine, or such problem would just happen again and again.
Thankfully extent tree corruption is not a big deal for "rescue=all,ro" mount.
Thanks,
Qu
Quoted 70 lines
[ 1942.115819] BTRFS error (device sda): read time tree block corruption detected on logical 11438351450112 mirror 2 [ 1942.116170] BTRFS critical (device sda): corrupt leaf: root=2 block=11438351450112 slot=170, invalid key objectid, have 5822947745796 expect to be aligned to 4096 [ 1942.116183] BTRFS error (device sda): read time tree block corruption detected on logical 11438351450112 mirror 1 [ 1942.116212] BTRFS error (device sda): failed to read block groups: -5 [ 1942.140369] BTRFS error (device sda): open_ctree failedI tried to recover the superblock:
# btrfs rescue super-recover -v /dev/sdaAll Devices:
Device: id = 3, name =/dev/sdc
Device: id = 2, name =/dev/sdb
Device: id = 4, name =/dev/sdd
Device: id = 1, name =/dev/sdaBefore Recovering:
[All good supers]:
device name =/dev/sdc
superblock bytenr = 65536device name =
/dev/sdc
superblock bytenr = 67108864device name =
/dev/sdc
superblock bytenr = 274877906944device name =
/dev/sdb
superblock bytenr = 65536device name =
/dev/sdb
superblock bytenr = 67108864device name =
/dev/sdb
superblock bytenr = 274877906944device name =
/dev/sdd
superblock bytenr = 65536device name =
/dev/sdd
superblock bytenr = 67108864device name =
/dev/sdd
superblock bytenr = 274877906944device name =
/dev/sda
superblock bytenr = 65536device name =
/dev/sda
superblock bytenr = 67108864device name =
/dev/sda
superblock bytenr = 274877906944[All bad supers]:
All supers are valid, no need to recover
I checked the filesystem:
# btrfs check --readonly /dev/sda Opening filesystem to check... corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608) corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608) corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608) corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608) ERROR: failed to read block groups: Operation not permitted ERROR: cannot open file system
Quoted 1 line · Qu Wenruo
Use "mount -o rescue=all,ro" instead.
Thank you for your suggestion. Should I run it under CentOS 7 (the original OS) or under the newer kernel in the SystemRescue distro?
Quoted 5 lines · 在 2024/3/19 11:03, [email protected] 写道:
Quoted 1 line · Qu Wenruo
Use "mount -o rescue=all,ro" instead.
Thank you for your suggestion. Should I run it under CentOS 7 (the original OS) or under the newer kernel in the SystemRescue distro?
Newer kernel please.
And still, please run memtest before doing any rescue.
As faulty hardware can always lead to weird problems no matter what.
And CentOS 7 is not recommended for btrfs usage (no proper btrfs related backport and old kernels, IIRC it's already EOL)
Thanks,
Qu
Quoted 2 lines · Qu Wenruo
And still, please run memtest before doing any rescue.
As faulty hardware can always lead to weird problems no matter what.
memtest86+ found bad RAM. I will replace that before proceeding. Thanks again.
Quoted 6 lines · 在 2024/3/20 07:14, [email protected] 写道:
Quoted 2 lines · Qu Wenruo
And still, please run memtest before doing any rescue.
As faulty hardware can always lead to weird problems no matter what.memtest86+ found bad RAM. I will replace that before proceeding. Thanks again.
Btrfs tree-checker, the second best memory tester. :)