Error during balancing '/': Input/output error
Hi,
I recently did a btrfs-convert from ext4 to btrfs and it went well, so I deleted the ext2 'home-backup' or whatever subvolume it created and I proceeded to balance the disk. However, I keep getting this error when I try to run this command (see bottom of message for debug info.) This might be a simple question, but I am worried that my root drive is corrupt and if I restart it might not boot so I haven't tried rebooting and mounting the disk from a live usb. Any suggestions pointing me in the right direction are most appreciated.
Thanks,
Michael
``` BEGIN DEBUG INFO ``` [michael@linux /]$ sudo btrfs balance / ERROR: error during balancing '/': Input/output error There may be more info in syslog - try dmesg | tail [michael@linux /]$ dmesg | tail [ 1235.433416] BTRFS info (device dm-1): relocating block group 265709158400 flags metadata [ 1375.701515] BTRFS info (device dm-1): found 65034 extents [ 1381.270271] BTRFS info (device dm-1): relocating block group 264635416576 flags metadata [ 1483.628290] BTRFS info (device dm-1): found 64995 extents [ 1485.160174] BTRFS info (device dm-1): relocating block group 253562454016 flags data [ 1485.352336] BTRFS warning (device dm-1): csum failed root -9 ino 265 off 1048576 csum 0x9a7afaa8 expected csum 0x63e15594 mirror 1 [ 1485.353008] BTRFS warning (device dm-1): csum failed root -9 ino 265 off 1048576 csum 0x9a7afaa8 expected csum 0x63e15594 mirror 1 [ 1485.375271] BTRFS info (device dm-1): balance: ended with status: -5 [ 1485.544838] audit: type=1106 audit(1577448716.970:152): pid=4178 uid=0 auid=1000 ses=1 msg='op=PAM:session_close grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success' [ 1485.545220] audit: type=1104 audit(1577448716.970:153): pid=4178 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_unix,pam_permit,pam_env acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success' ``` END DEBUG INFO ```
Quoted 10 lines · Michael Ruiz
Hi,
I recently did a btrfs-convert from ext4 to btrfs and it went well, so I deleted the ext2 'home-backup' or whatever subvolume it created and I proceeded to balance the disk. However, I keep getting this error when I try to run this command (see bottom of message for debug info.) This might be a simple question, but I am worried that my root drive is corrupt and if I restart it might not boot so I haven't tried rebooting and mounting the disk from a live usb. Any suggestions pointing me in the right direction are most appreciated.
There are quite some reports about this problem. So far no real corrupted confirmed.
You can confirm it by scrub.
I'm afraid there is some bug in data balance code which leads to this. We will dig further to pin down the cause.
BTW, does that problem always show up at block group 253562454016? Or it randomly shows up at different block groups?
Thanks,
Qu
Quoted 34 lines
Thanks,
Michael``` BEGIN DEBUG INFO ``` [michael@linux /]$ sudo btrfs balance / ERROR: error during balancing '/': Input/output error There may be more info in syslog - try dmesg | tail [michael@linux /]$ dmesg | tail [ 1235.433416] BTRFS info (device dm-1): relocating block group 265709158400 flags metadata [ 1375.701515] BTRFS info (device dm-1): found 65034 extents [ 1381.270271] BTRFS info (device dm-1): relocating block group 264635416576 flags metadata [ 1483.628290] BTRFS info (device dm-1): found 64995 extents [ 1485.160174] BTRFS info (device dm-1): relocating block group 253562454016 flags data [ 1485.352336] BTRFS warning (device dm-1): csum failed root -9 ino 265 off 1048576 csum 0x9a7afaa8 expected csum 0x63e15594 mirror 1 [ 1485.353008] BTRFS warning (device dm-1): csum failed root -9 ino 265 off 1048576 csum 0x9a7afaa8 expected csum 0x63e15594 mirror 1 [ 1485.375271] BTRFS info (device dm-1): balance: ended with status: -5 [ 1485.544838] audit: type=1106 audit(1577448716.970:152): pid=4178 uid=0 auid=1000 ses=1 msg='op=PAM:session_close grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success' [ 1485.545220] audit: type=1104 audit(1577448716.970:153): pid=4178 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_unix,pam_permit,pam_env acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success' ``` END DEBUG INFO ```
A message that is not in the archive (sent off-list or not yet received): 5555870.lOV4Wx5bFT@archlinux
Quoted 11 lines · Michael Ruiz
Quoted 11 lines
There are quite some reports about this problem. So far no real corrupted confirmed.
You can confirm it by scrub.
I'm afraid there is some bug in data balance code which leads to this. We will dig further to pin down the cause.
BTW, does that problem always show up at block group 253562454016? Or it randomly shows up at different block groups?
Quoted 7 lines
Quoted 2 lines
Thanks,
QuHi Qu,
It appears to be the same data blocks giving me this issue. I have tried it while mounting from a bootable usb and in my arch linux install both with similar results.
So it looks like you will hit the same -EIO even just relocating that specific block group using vrange filter.
Quoted 3 lines
I should also mention I have a btrfs root partition within a luks lvm partition becuase btrfs provides no encryption. I'm not sure this makes a difference.
I don't believe it would cause any difference.
Quoted 2 lines
Would you suggest running somekind of disk check or by running scrub? Please let me know what I should try to resolve this.
Running a scrub is always preferred.
An btrfs check run would be even better.
If all above reports no error, we need more info to dig. (See below).
Quoted 10 lines
谢谢
``` BEGIN DEBUG INFO ```
ERROR: error during balancing '/': Input/output error[ 465.880807] BTRFS info (device dm-1): found 25 extents [ 470.501319] BTRFS info (device dm-1): found 25 extents [ 471.713073] BTRFS info (device dm-1): relocating block group 253562454016 flags data [ 472.251356] BTRFS warning (device dm-1): csum failed root -9 ino 262 off1048576 csum
0x9a7afaa8expected csum0x63e15594mirror 1
Another question, is the reported csum (both have and expected) always the same?
Quoted 2 lines
[ 472.254390] BTRFS warning (device dm-1): csum failed root -9 ino 262 off1048576 csum
0x9a7afaa8expected csum0x63e15594mirror 1
This line means two things:
- We have extra inodes in data reloc tree
Normally data reloc only has one inode, 256.
It's possible we have some extra orphan inodes.
Would you like to do the following dump?
# btrfs ins dump-tree -t data_reloc /dev/dm-1
Although the result is not something representing the content when we hit the bug, it may still provide some clue.
- The original data is from logical 253563502592 Would you like to check which file(s) owns that logical? It can be done by the following command:
# btrfs ins logical-resolve 253563502592 <mnt>
Unlike previous command, it needs to be executed on a mounted btrfs other than the block device.
And the info about that inode is also helpful, like its attr (nodatacow?).
Thanks,
Qu
Quoted 2 lines
``` END DEBUG INFO ```
Hi Qu,
Thanks for the quick reply. This is what I could find.
[michael@linux /]$ sudo btrfs ins dump-tree -t data_reloc /dev/dm-1 btrfs-progs v5.4
data reloc tree key (DATA_RELOC_TREE ROOT_ITEM 2)
leaf 286941609984 items 2 free space 16061 generation 1186381 owner DATA_RELOC_TREE
leaf 286941609984 flags 0x1(WRITTEN) backref revision 1 fs uuid <redacted>
chunk uuid <redacted>
item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160
generation 2 transid 0 size 0 nbytes 16384
block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0
sequence 0 flags 0x0(none)
atime 1577186710.0 (2019-12-24 03:25:10)
ctime 1577186710.0 (2019-12-24 03:25:10)
mtime 1577186710.0 (2019-12-24 03:25:10)
otime 1577186710.0 (2019-12-24 03:25:10)
item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12
index 0 namelen 2 name: ..total bytes 255307284480
bytes used 94295339008
uuid <redacted>
[michael@archlinux /]$ sudo btrfs ins logical-resolve 253563502592 / //home/michael/.mozilla/firefox/default/storage/default/https++ +www.pinterest.com/cache/morgue/16/{b696bf53-d26a-48eb-9688- ab3c5fd49010}.final
``` BTRFS SCRUB ``` sudo btrfs scrub status / UUID: <redacted> Scrub started: Sat Dec 28 03:17:49 2019 Status: finished Duration: 0:05:09 Total to scrub: 87.82GiB Rate: 291.04MiB/s Error summary: csum=16 Corrected: 0 Uncorrectable: 16 Unverified: 0 ```
Quoted 32 lines · Michael Ruiz
Hi Qu,
Thanks for the quick reply. This is what I could find.[michael@linux /]$ sudo btrfs ins dump-tree -t
data_reloc/dev/dm-1btrfs-progs v5.4
data reloc tree key (DATA_RELOC_TREEROOT_ITEM2)
leaf 286941609984 items 2 free space 16061 generation 1186381 ownerDATA_RELOC_TREE
leaf 286941609984 flags 0x1(WRITTEN) backref revision 1 fs uuid <redacted>
chunk uuid <redacted>item 0 key (256 INODE_ITEM 0) itemoff 16123 itemsize 160 generation 2 transid 0 size 0 nbytes 16384 block group 0 mode 40755 links 1 uid 0 gid 0 rdev 0 sequence 0 flags 0x0(none) atime 1577186710.0 (2019-12-24 03:25:10) ctime 1577186710.0 (2019-12-24 03:25:10) mtime 1577186710.0 (2019-12-24 03:25:10) otime 1577186710.0 (2019-12-24 03:25:10) item 1 key (256 INODE_REF 256) itemoff 16111 itemsize 12 index 0 namelen 2 name: ..total bytes 255307284480
bytes used 94295339008
uuid <redacted>[michael@archlinux /]$ sudo btrfs ins logical-resolve 253563502592 / //home/michael/.mozilla/firefox/default/storage/default/https++ +www.pinterest.com/cache/morgue/16/{b696bf53-d26a-48eb-9688- ab3c5fd49010}.final
Quoted 13 lines
``` BTRFS SCRUB ``` sudo btrfs scrub status / UUID: <redacted> Scrub started: Sat Dec 28 03:17:49 2019 Status: finished Duration: 0:05:09 Total to scrub: 87.82GiB Rate: 291.04MiB/s Error summary: csum=16 Corrected: 0 Uncorrectable: 16 Unverified: 0 ```
Dmesg please, as that shows which file(s) is involved.
Thanks,
Qu
Quoted 5 lines · PST you
Dmesg please, as that shows which file(s) is involved.
Thanks,
Qu
Sure thing, it appears they are all from firefox cache.. Perhaps they are failing checksum because they were modified since I converted the partition earlier this week? I'm just guessing. Would you reccomend deleting these files manually or running btrfs check?
Thanks
``` BEGIN DMESG OUTPUT ```
[ 5638.996798] BTRFS warning (device dm-1): checksum error at logical
253563510784 on dev /dev/mapper/volgroup0-lv_root, physical 85319491584, root
5, inode 6477931, offset 8192, length 4096, links 1 (path: home/
michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.996805] BTRFS warning (device dm-1): checksum error at logical
253563502592 on dev /dev/mapper/volgroup0-lv_root, physical 85319483392, root
5, inode 6477931, offset 0, length 4096, links 1 (path: home/michael/.mozilla/
firefox/default/storage/default/https+++www.pinterest.com/cache/morgue/16/
{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.996808] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root
errs: wr 0, rd 0, flush 0, corrupt 17, gen 0
[ 5638.996812] BTRFS error (device dm-1): unable to fixup (regular) error at
logical 253563510784 on dev /dev/mapper/volgroup0-lv_root
[ 5638.996815] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root
errs: wr 0, rd 0, flush 0, corrupt 18, gen 0
[ 5638.996818] BTRFS error (device dm-1): unable to fixup (regular) error at
logical 253563502592 on dev /dev/mapper/volgroup0-lv_root
[ 5638.997338] BTRFS warning (device dm-1): checksum error at logical
253563506688 on dev /dev/mapper/volgroup0-lv_root, physical 85319487488, root
5, inode 6477931, offset 4096, length 4096, links 1 (path: home/
michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.997344] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root
errs: wr 0, rd 0, flush 0, corrupt 19, gen 0
[ 5638.997348] BTRFS error (device dm-1): unable to fixup (regular) error at
logical 253563506688 on dev /dev/mapper/volgroup0-lv_root
[ 5638.997526] BTRFS warning (device dm-1): checksum error at logical
253563514880 on dev /dev/mapper/volgroup0-lv_root, physical 85319495680, root
5, inode 6477931, offset 12288, length 4096, links 1 (path: home/
michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.997531] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root
errs: wr 0, rd 0, flush 0, corrupt 20, gen 0
[ 5638.997533] BTRFS error (device dm-1): unable to fixup (regular) error at
logical 253563514880 on dev /dev/mapper/volgroup0-lv_root
[ 5638.998086] BTRFS warning (device dm-1): checksum error at logical
253563518976 on dev /dev/mapper/volgroup0-lv_root, physical 85319499776, root
5, inode 6477931, offset 16384, length 4096, links 1 (path: home/
michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.998097] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root
errs: wr 0, rd 0, flush 0, corrupt 21, gen 0
[ 5638.998101] BTRFS error (device dm-1): unable to fixup (regular) error at
logical 253563518976 on dev /dev/mapper/volgroup0-lv_root
[ 5638.998542] BTRFS warning (device dm-1): checksum error at logical
253563523072 on dev /dev/mapper/volgroup0-lv_root, physical 85319503872, root
5, inode 6477931, offset 20480, length 4096, links 1 (path: home/
michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.998551] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root
errs: wr 0, rd 0, flush 0, corrupt 22, gen 0
[ 5638.998554] BTRFS error (device dm-1): unable to fixup (regular) error at
logical 253563523072 on dev /dev/mapper/volgroup0-lv_root
[ 5638.998975] BTRFS warning (device dm-1): checksum error at logical
253563527168 on dev /dev/mapper/volgroup0-lv_root, physical 85319507968, root
5, inode 6477931, offset 24576, length 4096, links 1 (path: home/
michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.998981] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root
errs: wr 0, rd 0, flush 0, corrupt 23, gen 0
[ 5638.998984] BTRFS error (device dm-1): unable to fixup (regular) error at
logical 253563527168 on dev /dev/mapper/volgroup0-lv_root
[ 5638.999356] BTRFS warning (device dm-1): checksum error at logical
253563531264 on dev /dev/mapper/volgroup0-lv_root, physical 85319512064, root
5, inode 6477931, offset 28672, length 4096, links 1 (path: home/
michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.999362] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root
errs: wr 0, rd 0, flush 0, corrupt 24, gen 0
[ 5638.999366] BTRFS error (device dm-1): unable to fixup (regular) error at
logical 253563531264 on dev /dev/mapper/volgroup0-lv_root
[ 5638.999592] BTRFS warning (device dm-1): checksum error at logical
253563535360 on dev /dev/mapper/volgroup0-lv_root, physical 85319516160, root
5, inode 6477931, offset 32768, length 4096, links 1 (path: home/
michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.999596] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root
errs: wr 0, rd 0, flush 0, corrupt 25, gen 0
[ 5638.999598] BTRFS error (device dm-1): unable to fixup (regular) error at
logical 253563535360 on dev /dev/mapper/volgroup0-lv_root
[ 5638.999861] BTRFS warning (device dm-1): checksum error at logical
253563539456 on dev /dev/mapper/volgroup0-lv_root, physical 85319520256, root
5, inode 6477931, offset 36864, length 4096, links 1 (path: home/
michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/
cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
[ 5638.999866] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root
errs: wr 0, rd 0, flush 0, corrupt 26, gen 0
[ 5638.999869] BTRFS error (device dm-1): unable to fixup (regular) error at
logical 253563539456 on dev /dev/mapper/volgroup0-lv_root
[ 5692.239308] BTRFS info (device dm-1): scrub: finished on devid 1 with
status: 0
``` END DMESG OUTPUT ```Quoted 13 lines · Michael Ruiz
Quoted 5 lines · PST you
Dmesg please, as that shows which file(s) is involved.
Thanks,
QuSure thing, it appears they are all from firefox cache.. Perhaps they are failing checksum because they were modified since I converted the partition earlier this week? I'm just guessing. Would you reccomend deleting these files manually or running btrfs check?
Thanks
Quoted 6 lines
``` BEGIN DMESG OUTPUT ```
[ 5638.996798] BTRFS warning (device dm-1): checksum error at logical253563510784 on dev
/dev/mapper/volgroup0-lv_root, physical 85319491584, root 5, inode 6477931, offset 8192, length 4096, links 1 (path: home/ michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/ cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)
Doesn't this filename look familiar with previous logical-resolve output? :)
[michael@archlinux /]$ sudo btrfs ins logical-resolve 253563502592 / //home/michael/.mozilla/firefox/default/storage/default/https++ +www.pinterest.com/cache/morgue/16/{b696bf53-d26a-48eb-9688- ab3c5fd49010}.final
The URL, the UUID, they all matches!
So yep, the file itself doesn't match its csum in the first place.
It looks like balance doesn't go regular file csum checking, but copy all data and all csum without verifying them, then check the csum of data reloc tree instead.
Thus this causes this hard-to-locate corrupted files. We should make it more user-friendly I guess.
And obviously, just remove the offending files should allow you continue.
Thanks,
Qu
Quoted 89 lines
[ 5638.996805] BTRFS warning (device dm-1): checksum error at logical253563502592 on dev
/dev/mapper/volgroup0-lv_root, physical 85319483392, root 5, inode 6477931, offset 0, length 4096, links 1 (path: home/michael/.mozilla/ firefox/default/storage/default/https+++www.pinterest.com/cache/morgue/16/ {b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)[ 5638.996808] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root errs: wr 0, rd 0, flush 0, corrupt 17, gen 0 [ 5638.996812] BTRFS error (device dm-1): unable to fixup (regular) error at logical 253563510784 on dev /dev/mapper/volgroup0-lv_root [ 5638.996815] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root errs: wr 0, rd 0, flush 0, corrupt 18, gen 0 [ 5638.996818] BTRFS error (device dm-1): unable to fixup (regular) error at logical 253563502592 on dev /dev/mapper/volgroup0-lv_root [ 5638.997338] BTRFS warning (device dm-1): checksum error at logical253563506688 on dev
/dev/mapper/volgroup0-lv_root, physical 85319487488, root 5, inode 6477931, offset 4096, length 4096, links 1 (path: home/ michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/ cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)[ 5638.997344] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root errs: wr 0, rd 0, flush 0, corrupt 19, gen 0 [ 5638.997348] BTRFS error (device dm-1): unable to fixup (regular) error at logical 253563506688 on dev /dev/mapper/volgroup0-lv_root [ 5638.997526] BTRFS warning (device dm-1): checksum error at logical253563514880 on dev
/dev/mapper/volgroup0-lv_root, physical 85319495680, root 5, inode 6477931, offset 12288, length 4096, links 1 (path: home/ michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/ cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)[ 5638.997531] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root errs: wr 0, rd 0, flush 0, corrupt 20, gen 0 [ 5638.997533] BTRFS error (device dm-1): unable to fixup (regular) error at logical 253563514880 on dev /dev/mapper/volgroup0-lv_root [ 5638.998086] BTRFS warning (device dm-1): checksum error at logical253563518976 on dev
/dev/mapper/volgroup0-lv_root, physical 85319499776, root 5, inode 6477931, offset 16384, length 4096, links 1 (path: home/ michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/ cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)[ 5638.998097] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root errs: wr 0, rd 0, flush 0, corrupt 21, gen 0 [ 5638.998101] BTRFS error (device dm-1): unable to fixup (regular) error at logical 253563518976 on dev /dev/mapper/volgroup0-lv_root [ 5638.998542] BTRFS warning (device dm-1): checksum error at logical253563523072 on dev
/dev/mapper/volgroup0-lv_root, physical 85319503872, root 5, inode 6477931, offset 20480, length 4096, links 1 (path: home/ michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/ cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)[ 5638.998551] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root errs: wr 0, rd 0, flush 0, corrupt 22, gen 0 [ 5638.998554] BTRFS error (device dm-1): unable to fixup (regular) error at logical 253563523072 on dev /dev/mapper/volgroup0-lv_root [ 5638.998975] BTRFS warning (device dm-1): checksum error at logical253563527168 on dev
/dev/mapper/volgroup0-lv_root, physical 85319507968, root 5, inode 6477931, offset 24576, length 4096, links 1 (path: home/ michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/ cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)[ 5638.998981] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root errs: wr 0, rd 0, flush 0, corrupt 23, gen 0 [ 5638.998984] BTRFS error (device dm-1): unable to fixup (regular) error at logical 253563527168 on dev /dev/mapper/volgroup0-lv_root [ 5638.999356] BTRFS warning (device dm-1): checksum error at logical253563531264 on dev
/dev/mapper/volgroup0-lv_root, physical 85319512064, root 5, inode 6477931, offset 28672, length 4096, links 1 (path: home/ michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/ cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)[ 5638.999362] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root errs: wr 0, rd 0, flush 0, corrupt 24, gen 0 [ 5638.999366] BTRFS error (device dm-1): unable to fixup (regular) error at logical 253563531264 on dev /dev/mapper/volgroup0-lv_root [ 5638.999592] BTRFS warning (device dm-1): checksum error at logical253563535360 on dev
/dev/mapper/volgroup0-lv_root, physical 85319516160, root 5, inode 6477931, offset 32768, length 4096, links 1 (path: home/ michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/ cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)[ 5638.999596] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root errs: wr 0, rd 0, flush 0, corrupt 25, gen 0 [ 5638.999598] BTRFS error (device dm-1): unable to fixup (regular) error at logical 253563535360 on dev /dev/mapper/volgroup0-lv_root [ 5638.999861] BTRFS warning (device dm-1): checksum error at logical253563539456 on dev
/dev/mapper/volgroup0-lv_root, physical 85319520256, root 5, inode 6477931, offset 36864, length 4096, links 1 (path: home/ michael/.mozilla/firefox/default/storage/default/https+++www.pinterest.com/ cache/morgue/16/{b696bf53-d26a-48eb-9688-ab3c5fd49010}.final)[ 5638.999866] BTRFS error (device dm-1): bdev /dev/mapper/volgroup0-lv_root errs: wr 0, rd 0, flush 0, corrupt 26, gen 0 [ 5638.999869] BTRFS error (device dm-1): unable to fixup (regular) error at logical 253563539456 on dev /dev/mapper/volgroup0-lv_root [ 5692.239308] BTRFS info (device dm-1): scrub: finished on devid 1 withstatus: 0
``` END DMESG OUTPUT ```
Quoted 15 lines · PST you
Doesn't this filename look familiar with previous logical-resolve output? :)
[michael@archlinux /]$ sudo btrfs ins logical-resolve 253563502592 / //home/michael/.mozilla/firefox/default/storage/default/https++ +www.pinterest.com/cache/morgue/16/{b696bf53-d26a-48eb-9688- ab3c5fd49010}.final
The URL, the UUID, they all matches!
So yep, the file itself doesn't match its csum in the first place.
It looks like balance doesn't go regular file csum checking, but copy all data and all csum without verifying them, then check the csum of data reloc tree instead.
Quoted 8 lines
Thus this causes this hard-to-locate corrupted files. We should make it more user-friendly I guess.
And obviously, just remove the offending files should allow you continue.
Thanks,
Qu
Wow, you are right. I was worried my file system was corrupted and I'm not sure the exact cause leading up to this error. If there is any more debug info I can give you so that you can get a better idea of why this is happening, please let me know.
Currently I am running balance, but I predict it will finally be able to complete sucessfully. However, if there are any issues I will send a reply here. (If I don't figure it out myself using what you just taught me) Thanks a lot for your help. I am new to using mailing lists such as these, so hopefully I did everything by the rules here.
Thanks again Qu!
Quoted 5 lines
There are quite some reports about this problem. So far no real corrupted confirmed.
You can confirm it by scrub.
Quoted 8 lines
I'm afraid there is some bug in data balance code which leads to this. We will dig further to pin down the cause.
BTW, does that problem always show up at block group 253562454016? Or it randomly shows up at different block groups?
Thanks,
Qu
Hi Qu,
It appears to be the same data blocks giving me this issue. I have tried it while mounting from a bootable usb and in my arch linux install both with similar results. I should also mention I have a btrfs root partition within a luks lvm partition becuase btrfs provides no encryption. I'm not sure this makes a difference. Would you suggest running somekind of disk check or by running scrub? Please let me know what I should try to resolve this. 谢谢
``` BEGIN DEBUG INFO ``` ERROR: error during balancing '/': Input/output error [michael@linux /]$ dmesg | tail [ 462.460626] audit: type=1104 audit(1577525576.905:124): pid=3415 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_unix,pam_permit,pam_env acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/2 res=success' [ 465.880807] BTRFS info (device dm-1): found 25 extents [ 470.501319] BTRFS info (device dm-1): found 25 extents [ 471.713073] BTRFS info (device dm-1): relocating block group 253562454016 flags data [ 472.251356] BTRFS warning (device dm-1): csum failed root -9 ino 262 off 1048576 csum 0x9a7afaa8 expected csum 0x63e15594 mirror 1 [ 472.254390] BTRFS warning (device dm-1): csum failed root -9 ino 262 off 1048576 csum 0x9a7afaa8 expected csum 0x63e15594 mirror 1 [ 472.716215] BTRFS info (device dm-1): balance: ended with status: -5 [ 473.122721] audit: type=1106 audit(1577525587.565:125): pid=2775 uid=0 auid=1000 ses=1 msg='op=PAM:session_close grantors=pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success' [ 473.122840] audit: type=1104 audit(1577525587.565:126): pid=2775 uid=0 auid=1000 ses=1 msg='op=PAM:setcred grantors=pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/3 res=success' [ 629.160619] audit: type=1131 audit(1577525743.603:127): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=packagekit comm="systemd" exe="/usr/ lib/systemd/systemd" hostname=? addr=? terminal=? res=success' ``` END DEBUG INFO ```