Skip to contentSkip to conversation

Help requested: btrfs can't read superblock

DISCUSSION [email protected] 2 participants 6 messages Mar 18, 2024 – Mar 19, 2024 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Dear btrfs team,

I've been asked to help someone with a btrfs problem. They use CentOS 7. I've been told the host was shut down cleanly, but it won't mount a btrfs filesystem when powering on.

Below is what I've done so far. Any guidance would be appreciated!

I booted using SystemRescue 11.0. Some info about the rescue OS:

# uname -a
Linux sysrescue 6.6.14-1-lts #1 SMP PREEMPT_DYNAMIC Fri, 26 Jan 2024 11:54:58 +0000 x86_64 GNU/Linux
# btrfs --version
btrfs-progs v6.6.3

Here are the disks:

# btrfs fi show
Label: '<redacted>'  uuid: 834b4bfd-fbd4-40b9-8868-d1187c3c5a63
	Total devices 4 FS bytes used 17.44TiB
	devid    1 size 9.10TiB used 8.94TiB path /dev/sda
	devid    2 size 9.10TiB used 8.94TiB path /dev/sdb
	devid    3 size 9.10TiB used 8.94TiB path /dev/sdc
	devid    4 size 9.10TiB used 8.94TiB path /dev/sdd

I tried mounting:

# mount -t btrfs -o recovery,ro /dev/sda /mnt/data
mount: /mnt/data: can't read superblock on /dev/sda.
       dmesg(1) may have more information after failed mount system call.

dmesg shows this:

[ 1940.962632] BTRFS info (device sda): first mount of filesystem 834b4bfd-fbd4-40b9-8868-d1187c3c5a63
[ 1940.962665] BTRFS info (device sda): using crc32c (crc32c-intel) checksum algorithm
[ 1940.962686] BTRFS warning (device sda): 'recovery' is deprecated, use 'rescue=usebackuproot' instead
[ 1940.962692] BTRFS info (device sda): trying to use backup root at mount time
[ 1940.962698] BTRFS info (device sda): disk space caching is enabled
[ 1942.115796] BTRFS critical (device sda): corrupt leaf: root=2 block=11438351450112 slot=170, invalid key objectid, have 5822947745796 expect to be aligned to 4096
[ 1942.115819] BTRFS error (device sda): read time tree block corruption detected on logical 11438351450112 mirror 2
[ 1942.116170] BTRFS critical (device sda): corrupt leaf: root=2 block=11438351450112 slot=170, invalid key objectid, have 5822947745796 expect to be aligned to 4096
[ 1942.116183] BTRFS error (device sda): read time tree block corruption detected on logical 11438351450112 mirror 1
[ 1942.116212] BTRFS error (device sda): failed to read block groups: -5
[ 1942.140369] BTRFS error (device sda): open_ctree failed

I tried to recover the superblock:

# btrfs rescue super-recover -v /dev/sda

All Devices:
Device: id = 3, name = /dev/sdc
Device: id = 2, name = /dev/sdb
Device: id = 4, name = /dev/sdd
Device: id = 1, name = /dev/sda

Before Recovering:
[All good supers]:
device name = /dev/sdc
superblock bytenr = 65536

device name = /dev/sdc
superblock bytenr = 67108864

device name = /dev/sdc
superblock bytenr = 274877906944

device name = /dev/sdb
superblock bytenr = 65536

device name = /dev/sdb
superblock bytenr = 67108864

device name = /dev/sdb
superblock bytenr = 274877906944

device name = /dev/sdd
superblock bytenr = 65536

device name = /dev/sdd
superblock bytenr = 67108864

device name = /dev/sdd
superblock bytenr = 274877906944

device name = /dev/sda
superblock bytenr = 65536

device name = /dev/sda
superblock bytenr = 67108864

device name = /dev/sda
superblock bytenr = 274877906944

[All bad supers]:

All supers are valid, no need to recover

I checked the filesystem:

# btrfs check --readonly /dev/sda
Opening filesystem to check...
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
ERROR: failed to read block groups: Operation not permitted
ERROR: cannot open file system
#1 lore raw
Quoted 25 lines · 在 2024/3/19 07:52, [email protected] 写道:

Dear btrfs team,

I've been asked to help someone with a btrfs problem. They use CentOS 7. I've been told the host was shut down cleanly, but it won't mount a btrfs filesystem when powering on.

Below is what I've done so far. Any guidance would be appreciated!

I booted using SystemRescue 11.0. Some info about the rescue OS:

# uname -a
Linux sysrescue 6.6.14-1-lts #1 SMP PREEMPT_DYNAMIC Fri, 26 Jan 2024 11:54:58 +0000 x86_64 GNU/Linux
# btrfs --version
btrfs-progs v6.6.3

Here are the disks:

# btrfs fi show
Label: '<redacted>'  uuid: 834b4bfd-fbd4-40b9-8868-d1187c3c5a63
	Total devices 4 FS bytes used 17.44TiB
	devid    1 size 9.10TiB used 8.94TiB path /dev/sda
	devid    2 size 9.10TiB used 8.94TiB path /dev/sdb
	devid    3 size 9.10TiB used 8.94TiB path /dev/sdc
	devid    4 size 9.10TiB used 8.94TiB path /dev/sdd
Quoted 5 lines

I tried mounting:

# mount -t btrfs -o recovery,ro /dev/sda /mnt/data
mount: /mnt/data: can't read superblock on /dev/sda.
        dmesg(1) may have more information after failed mount system call.

Use "mount -o rescue=all,ro" instead.

Quoted 9 lines

dmesg shows this:

[ 1940.962632] BTRFS info (device sda): first mount of filesystem 834b4bfd-fbd4-40b9-8868-d1187c3c5a63
[ 1940.962665] BTRFS info (device sda): using crc32c (crc32c-intel) checksum algorithm
[ 1940.962686] BTRFS warning (device sda): 'recovery' is deprecated, use 'rescue=usebackuproot' instead
[ 1940.962692] BTRFS info (device sda): trying to use backup root at mount time
[ 1940.962698] BTRFS info (device sda): disk space caching is enabled
[ 1942.115796] BTRFS critical (device sda): corrupt leaf: root=2 block=11438351450112 slot=170, invalid key objectid, have 5822947745796 expect to be aligned to 4096

Bad extent item start bytenr, 5822947745796 = 0x54bc2bb6004, the last 0x004 looks like a bitflip.

Please run memtest to make sure your hardware memory is fine, or such problem would just happen again and again.

Thankfully extent tree corruption is not a big deal for "rescue=all,ro" mount.

Thanks,
Qu

Quoted 70 lines
[ 1942.115819] BTRFS error (device sda): read time tree block corruption detected on logical 11438351450112 mirror 2
[ 1942.116170] BTRFS critical (device sda): corrupt leaf: root=2 block=11438351450112 slot=170, invalid key objectid, have 5822947745796 expect to be aligned to 4096
[ 1942.116183] BTRFS error (device sda): read time tree block corruption detected on logical 11438351450112 mirror 1
[ 1942.116212] BTRFS error (device sda): failed to read block groups: -5
[ 1942.140369] BTRFS error (device sda): open_ctree failed

I tried to recover the superblock:

# btrfs rescue super-recover -v /dev/sda

All Devices:
Device: id = 3, name = /dev/sdc
Device: id = 2, name = /dev/sdb
Device: id = 4, name = /dev/sdd
Device: id = 1, name = /dev/sda

Before Recovering:
[All good supers]:
device name = /dev/sdc
superblock bytenr = 65536

device name = /dev/sdc
superblock bytenr = 67108864

device name = /dev/sdc
superblock bytenr = 274877906944

device name = /dev/sdb
superblock bytenr = 65536

device name = /dev/sdb
superblock bytenr = 67108864

device name = /dev/sdb
superblock bytenr = 274877906944

device name = /dev/sdd
superblock bytenr = 65536

device name = /dev/sdd
superblock bytenr = 67108864

device name = /dev/sdd
superblock bytenr = 274877906944

device name = /dev/sda
superblock bytenr = 65536

device name = /dev/sda
superblock bytenr = 67108864

device name = /dev/sda
superblock bytenr = 274877906944

[All bad supers]:

All supers are valid, no need to recover

I checked the filesystem:

# btrfs check --readonly /dev/sda
Opening filesystem to check...
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
corrupt leaf: root=2 block=62421692628992 slot=7, bad key order, prev (7275590123520 168 11227136) current (7258421481472 168 10948608)
ERROR: failed to read block groups: Operation not permitted
ERROR: cannot open file system
#2 lore raw
Quoted 1 line · Qu Wenruo

Use "mount -o rescue=all,ro" instead.

Thank you for your suggestion. Should I run it under CentOS 7 (the original OS) or under the newer kernel in the SystemRescue distro?

#3 lore raw
Quoted 5 lines · 在 2024/3/19 11:03, [email protected] 写道:
Quoted 1 line · Qu Wenruo

Use "mount -o rescue=all,ro" instead.

Thank you for your suggestion. Should I run it under CentOS 7 (the original OS) or under the newer kernel in the SystemRescue distro?

Newer kernel please.

And still, please run memtest before doing any rescue.
As faulty hardware can always lead to weird problems no matter what.

And CentOS 7 is not recommended for btrfs usage (no proper btrfs related backport and old kernels, IIRC it's already EOL)

Thanks,
Qu

#4 lore raw
Quoted 2 lines · Qu Wenruo

And still, please run memtest before doing any rescue.
As faulty hardware can always lead to weird problems no matter what.

memtest86+ found bad RAM. I will replace that before proceeding. Thanks again.

#5 lore raw
Quoted 6 lines · 在 2024/3/20 07:14, [email protected] 写道:
Quoted 2 lines · Qu Wenruo

And still, please run memtest before doing any rescue.
As faulty hardware can always lead to weird problems no matter what.

memtest86+ found bad RAM. I will replace that before proceeding. Thanks again.

Btrfs tree-checker, the second best memory tester. :)