Skip to contentSkip to conversation

failed to read block groups: Operation not permitted

DISCUSSION Johannes Hirte 2 participants 4 messages Oct 6, 2020 – Oct 8, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw

I recently encountered filesystem damage on a VM. During normal operation, the filesystem was remounted ro suddenly. Dmesg showed me some errors about parent transid verify failed. I've forced of the VM and tried to mount the image on the host, but failed with:

[  340.702391] BTRFS info (device loop0p1): disk space caching is enabled
[  340.702393] BTRFS info (device loop0p1): has skinny extents
[  341.815890] BTRFS error (device loop0p1): parent transid verify failed on 152064327680 wanted 323984 found 323888
[  341.831183] BTRFS error (device loop0p1): parent transid verify failed on 152064327680 wanted 323984 found 323888
[  341.831194] BTRFS error (device loop0p1): failed to read block groups: -5
[  341.851954] BTRFS error (device loop0p1): open_ctree failed

A btrfs check resulted in:

btrfs check /dev/loop0p1
Opening filesystem to check...
parent transid verify failed on 152064327680 wanted 323984 found 323888 parent transid verify failed on 152064327680 wanted 323984 found 323888 parent transid verify failed on 152064327680 wanted 323984 found 323888 Ignoring transid failure
leaf parent key incorrect 152064327680
ERROR: failed to read block groups: Operation not permitted
ERROR: cannot open file system

The host is running libvirt with kvm, btrfs with RAID1. The VMs are raw images, with btrfs too. I've switche this VM from io=native to io=io_uring, and suspect that this caused the damage. All machines are running kernel 5.8.13.

I was able to recover most of the damaged filesystem with btrfs recover. Is there anything I can do for repair it? And why does the damage happen at all with io_uring?

Signature · 2 lines
Regards,
  Johannes Hirte
#1 lore raw
Quoted 9 lines · Johannes Hirte

I recently encountered filesystem damage on a VM. During normal operation, the filesystem was remounted ro suddenly. Dmesg showed me some errors about parent transid verify failed. I've forced of the VM and tried to mount the image on the host, but failed with:

[  340.702391] BTRFS info (device loop0p1): disk space caching is enabled
[  340.702393] BTRFS info (device loop0p1): has skinny extents
[  341.815890] BTRFS error (device loop0p1): parent transid verify failed on 152064327680 wanted 323984 found 323888
[  341.831183] BTRFS error (device loop0p1): parent transid verify failed on 152064327680 wanted 323984 found 323888

Your extent tree is corrupted. Metadata CoW is broken.

I don't believe only extent tree get corrupted, other part of your fs can also be corrupted.

Quoted 15 lines
[  341.831194] BTRFS error (device loop0p1): failed to read block groups: -5
[  341.851954] BTRFS error (device loop0p1): open_ctree failed

A btrfs check resulted in:

btrfs check /dev/loop0p1
Opening filesystem to check...
parent transid verify failed on 152064327680 wanted 323984 found 323888 parent transid verify failed on 152064327680 wanted 323984 found 323888 parent transid verify failed on 152064327680 wanted 323984 found 323888 Ignoring transid failure
leaf parent key incorrect 152064327680
ERROR: failed to read block groups: Operation not permitted
ERROR: cannot open file system

Quoted 4 lines

The host is running libvirt with kvm, btrfs with RAID1. The VMs are raw images, with btrfs too. I've switche this VM from io=native to io=io_uring, and suspect that this caused the damage. All machines are running kernel 5.8.13.

I'm not sure about the io_uring setup. IIRC as long as you're not using cache=unsafe, it should be safe.

Does the io_uring ignores the flush?

Thanks,
Qu

Quoted 5 lines

I was able to recover most of the damaged filesystem with btrfs recover. Is there anything I can do for repair it? And why does the damage happen at all with io_uring?

#2 lore raw
Quoted 38 lines · Qu Wenruo
Quoted 9 lines · Johannes Hirte

I recently encountered filesystem damage on a VM. During normal operation, the filesystem was remounted ro suddenly. Dmesg showed me some errors about parent transid verify failed. I've forced of the VM and tried to mount the image on the host, but failed with:

[  340.702391] BTRFS info (device loop0p1): disk space caching is enabled
[  340.702393] BTRFS info (device loop0p1): has skinny extents
[  341.815890] BTRFS error (device loop0p1): parent transid verify failed on 152064327680 wanted 323984 found 323888
[  341.831183] BTRFS error (device loop0p1): parent transid verify failed on 152064327680 wanted 323984 found 323888

Your extent tree is corrupted. Metadata CoW is broken.

I don't believe only extent tree get corrupted, other part of your fs can also be corrupted.

Quoted 15 lines
[  341.831194] BTRFS error (device loop0p1): failed to read block groups: -5
[  341.851954] BTRFS error (device loop0p1): open_ctree failed

A btrfs check resulted in:

btrfs check /dev/loop0p1
Opening filesystem to check...
parent transid verify failed on 152064327680 wanted 323984 found 323888 parent transid verify failed on 152064327680 wanted 323984 found 323888 parent transid verify failed on 152064327680 wanted 323984 found 323888 Ignoring transid failure
leaf parent key incorrect 152064327680
ERROR: failed to read block groups: Operation not permitted
ERROR: cannot open file system

Quoted 4 lines

The host is running libvirt with kvm, btrfs with RAID1. The VMs are raw images, with btrfs too. I've switche this VM from io=native to io=io_uring, and suspect that this caused the damage. All machines are running kernel 5.8.13.

Quoted 4 lines

I'm not sure about the io_uring setup. IIRC as long as you're not using cache=unsafe, it should be safe.

Does the io_uring ignores the flush?

Putting someone with more knowledge into cc.

For another VM, I've found several errors in the log of the host machine:

BTRFS warning (device sda1): direct IO failed ino 5988432 rw 1,2131969 sector 0x123ab840 len 32768 err no 10

The VM was remounted ro too, like the first one. But in this case the filesystem was ok after a check. For the first VM with the heavily damaged filesystem there aren't any log entries.

Signature · 2 lines
Regards,
  Johannes Hirte
#3 lore raw
Quoted 47 lines · Johannes Hirte
Quoted 38 lines · Qu Wenruo
Quoted 9 lines · Johannes Hirte

I recently encountered filesystem damage on a VM. During normal operation, the filesystem was remounted ro suddenly. Dmesg showed me some errors about parent transid verify failed. I've forced of the VM and tried to mount the image on the host, but failed with:

[  340.702391] BTRFS info (device loop0p1): disk space caching is enabled
[  340.702393] BTRFS info (device loop0p1): has skinny extents
[  341.815890] BTRFS error (device loop0p1): parent transid verify failed on 152064327680 wanted 323984 found 323888
[  341.831183] BTRFS error (device loop0p1): parent transid verify failed on 152064327680 wanted 323984 found 323888

Your extent tree is corrupted. Metadata CoW is broken.

I don't believe only extent tree get corrupted, other part of your fs can also be corrupted.

Quoted 15 lines
[  341.831194] BTRFS error (device loop0p1): failed to read block groups: -5
[  341.851954] BTRFS error (device loop0p1): open_ctree failed

A btrfs check resulted in:

btrfs check /dev/loop0p1
Opening filesystem to check...
parent transid verify failed on 152064327680 wanted 323984 found 323888 parent transid verify failed on 152064327680 wanted 323984 found 323888 parent transid verify failed on 152064327680 wanted 323984 found 323888 Ignoring transid failure
leaf parent key incorrect 152064327680
ERROR: failed to read block groups: Operation not permitted
ERROR: cannot open file system

Quoted 4 lines

The host is running libvirt with kvm, btrfs with RAID1. The VMs are raw images, with btrfs too. I've switche this VM from io=native to io=io_uring, and suspect that this caused the damage. All machines are running kernel 5.8.13.

Quoted 4 lines

I'm not sure about the io_uring setup. IIRC as long as you're not using cache=unsafe, it should be safe.

Does the io_uring ignores the flush?

Putting someone with more knowledge into cc.

Quoted 7 lines

For another VM, I've found several errors in the log of the host machine:

BTRFS warning (device sda1): direct IO failed ino 5988432 rw 1,2131969 sector 0x123ab840 len 32768 err no 10

The VM was remounted ro too, like the first one. But in this case the filesystem was ok after a check.

I have to correct this. There are several csum errors on this VM like this:

BTRFS warning (device vda1): checksum/header error at logical 9660727296 on dev /dev/vda1, physical 6488064: metadata leaf (level 0) in tree 257

Signature · 2 lines
Regards,
  Johannes Hirte