3.14.0rc3: did not find backref in send_root
I got this during a btrfs send:
BTRFS error (device dm-2): did not find backref in send_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560
I'll try a scrub when I've finished my backup, but is there anything I can run on the file I've found from the inode?
gargamel:/mnt/dshelf1/Sound# btrfs inspect-internal inode-resolve -v 22672 file.mp3 ioctl ret=0, bytes_left=3998, bytes_missing=0, cnt=1, missed=0 file.mp3
Anything else?
Thanks,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901Hi Marc,
This seems a regression which has been fixed by the following commit(only pushed into btrfs-next):
Thanks,
Wang
Quoted 14 lines · Marc MERLIN
I got this during a btrfs send:
BTRFS error (device dm-2): did not find backref insend_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560I'll try a scrub when I've finished my backup, but is there anything I can run on the file I've found from the inode?
gargamel:/mnt/dshelf1/Sound# btrfs inspect-internal inode-resolve -v 22672 file.mp3 ioctl ret=0, bytes_left=3998, bytes_missing=0, cnt=1, missed=0 file.mp3Anything else?
Thanks,
Marc
Re: 3.14.0rc3: btrfs send ioctl failed with -5: Input/output error
Quoted 6 lines · Wang Shilong
Hi Marc,
This seems a regression which has been fixed by the following commit(only pushed into btrfs-next):
I'll revert this, thanks.
Mmmh, but I just found another problem I didn't have before upgrading to 3.14.0-rc3 (on my laptop this time):
+ btrfs send -p /mnt/btrfs_pool1/var_ro.20140224_14:01:54 var_ro.20140225_09:27:57
+ btrfs receive /mnt/btrfs_pool2//
At subvol var_ro.20140225_09:27:57
At snapshot var_ro.20140225_09:27:57
ERROR: send ioctl failed with -5: Input/output error
ERROR: unexpected EOF in stream.
Is it another different regression?
Currently it's only happening on one of my subvolumes. The other ones are still syncing ok.
Thanks,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901Re: 3.14.0rc3: btrfs send ioctl failed with -5: Input/output error
Hi Marc,
Quoted 11 lines · Marc MERLIN
Quoted 6 lines · Wang Shilong
Hi Marc,
This seems a regression which has been fixed by the following commit(only pushed into btrfs-next):
I'll revert this, thanks.
Mmmh, but I just found another problem I didn't have before upgrading to 3.14.0-rc3 (on my laptop this time):
What is your previous version, give a double check whether the following commit exists:
Btrfs: remove transaction from btrfs send(commitid: 41ce9970)
This regression should exist in 3.14 Sine chris firstly pull big btrfs request for Linux 3.14.
Quoted 7 lines
+ btrfs send -p
/mnt/btrfs_pool1/var_ro.20140224_14:01:54var_ro.20140225_09:27:57
+ btrfs receive/mnt/btrfs_pool2//
At subvol var_ro.20140225_09:27:57
At snapshot var_ro.20140225_09:27:57
ERROR: send ioctl failed with -5: Input/output error
ERROR: unexpected EOF in stream.
Did dmesg output the same things like before?
Hm, we can esaily trigger that regression if there is snapshot and send running concurrently. (balance/scrub ...device operations can also trigger send failure.)
Thanks,
Wang
Quoted 8 lines
Is it another different regression?
Currently it's only happening on one of my subvolumes. The other ones are still syncing ok.
Thanks,
Marc
Quoted 28 lines · Wang Shilong
Hi Marc,
Quoted 11 lines · Marc MERLIN
Quoted 6 lines · Wang Shilong
Hi Marc,
This seems a regression which has been fixed by the following commit(only pushed into btrfs-next):
I'll revert this, thanks.
Mmmh, but I just found another problem I didn't have before upgrading to 3.14.0-rc3 (on my laptop this time):
What is your previous version, give a double check whether the following commit exists:
Btrfs: remove transaction from btrfs send(commitid:
41ce9970)This regression should exist in 3.14 Sine chris firstly pull big btrfs request for Linux 3.14.
Quoted 7 lines
+ btrfs send -p
/mnt/btrfs_pool1/var_ro.20140224_14:01:54var_ro.20140225_09:27:57
+ btrfs receive/mnt/btrfs_pool2//
At subvol var_ro.20140225_09:27:57
At snapshot var_ro.20140225_09:27:57
ERROR: send ioctl failed with -5: Input/output error
ERROR: unexpected EOF in stream.
Quoted 5 lines
Did dmesg output the same things like before?
Hm, we can esaily trigger that regression if there is snapshot and send running concurrently.
(balance/scrub ...device operations can also trigger send failure.)
So I was confused because I got this on my server:
Quoted 2 lines · Marc MERLIN
I got this during a btrfs send:
BTRFS error (device dm-2): did not find backref insend_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560
and that on my laptop:
[82841.563339] BTRFS info (device dm-0): csum failed ino 2590 extent 2397949952 csum 3100992190 wanted 2786811954 mirror 0
I've applied your patch from
https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8
legolas:/mnt/btrfs_pool1# bash -vx /var/local/scr/btrfs-subvolume-backup -k 5 var /mnt/btrfs_pool2/
still failed on my laptop
+ btrfs send -p /mnt/btrfs_pool1/var_ro.20140224_14:01:54 var_ro.20140225_23:28:47
+ btrfs receive /mnt/btrfs_pool2//
At subvol var_ro.20140225_23:28:47
At snapshot var_ro.20140225_23:28:47
ERROR: send ioctl failed with -5: Input/output error
ERROR: unexpected EOF in stream.
[ 216.534313] BTRFS info (device dm-0): csum failed ino 2326136 off 192512 csum 3851586574 expected csum 1402824092
Then again, this seems to be my problem on the laptop:
legolas:/mnt/btrfs_pool1# btrfs scrub status /mnt/btrfs_pool1 scrub status for 4850ee22-bf32-4131-a841-02abdb4a5ba6 scrub started at Tue Feb 25 07:35:07 2014 and finished after 1945 seconds total bytes scrubbed: 451.64GiB with 2 errors error details: csum=2 corrected errors: 0, uncorrectable errors: 2, unverified errors: 0
Ok, so that's not a btrfs send problem.
Just out of curiosity, how do I find out which inodes are compromized so that I can delete/restore them?
Back to my server, I installed the new kernel, and I'm testing send again. So far it hasn't failed, so there is a chance the patch fixed the problem, but I'll only know more tomorrow after it's run longer.
Thanks,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901Quoted 62 lines · Marc MERLIN
Quoted 27 lines · Wang Shilong
Hi Marc,
Quoted 11 lines · Marc MERLIN
Quoted 6 lines · Wang Shilong
Hi Marc,
This seems a regression which has been fixed by the following commit(only pushed into btrfs-next):
I'll revert this, thanks.
Mmmh, but I just found another problem I didn't have before upgrading to 3.14.0-rc3 (on my laptop this time):
What is your previous version, give a double check whether the following commit exists:
Btrfs: remove transaction from btrfs send(commitid:
41ce9970)This regression should exist in 3.14 Sine chris firstly pull big btrfs request for Linux 3.14.
Quoted 6 lines
+ btrfs send -p
/mnt/btrfs_pool1/var_ro.20140224_14:01:54var_ro.20140225_09:27:57
+ btrfs receive/mnt/btrfs_pool2//
At subvol var_ro.20140225_09:27:57
At snapshot var_ro.20140225_09:27:57
ERROR: send ioctl failed with -5: Input/output error
ERROR: unexpected EOF in stream.Quoted 5 lines
Did dmesg output the same things like before?
Hm, we can esaily trigger that regression if there is snapshot and send running concurrently.
(balance/scrub ...device operations can also trigger send failure.)So I was confused because I got this on my server:
Quoted 2 lines · Marc MERLIN
I got this during a btrfs send:
BTRFS error (device dm-2): did not find backref insend_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560and that on my laptop:
[82841.563339] BTRFS info (device dm-0): csum failed ino 2590 extent 2397949952 csum 3100992190 wanted 2786811954 mirror 0I've applied your patch from
https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8legolas:/mnt/btrfs_pool1# bash -vx /var/local/scr/btrfs-subvolume-backup -k 5 var /mnt/btrfs_pool2/still failed on my laptop
+ btrfs send -p/mnt/btrfs_pool1/var_ro.20140224_14:01:54var_ro.20140225_23:28:47
+ btrfs receive/mnt/btrfs_pool2//
At subvol var_ro.20140225_23:28:47
At snapshot var_ro.20140225_23:28:47
ERROR: send ioctl failed with -5: Input/output error
ERROR: unexpected EOF in stream.[ 216.534313] BTRFS info (device dm-0): csum failed ino 2326136 off 192512 csum 3851586574 expected csum 1402824092Then again, this seems to be my problem on the laptop:
legolas:/mnt/btrfs_pool1# btrfs scrub status /mnt/btrfs_pool1 scrub status for 4850ee22-bf32-4131-a841-02abdb4a5ba6 scrub started at Tue Feb 25 07:35:07 2014 and finished after 1945 seconds total bytes scrubbed: 451.64GiB with 2 errors error details: csum=2 corrected errors: 0, uncorrectable errors: 2, unverified errors: 0
Quoted 3 lines
Ok, so that's not a btrfs send problem.
Just out of curiosity, how do I find out which inodes are compromized so that I can delete/restore them?
You can use command "btrfs inspect-internal inode-resolve" which will print inode's corresponding path with it's inode id.
For your case, something like this:
btrfs inspect-internal inode-resolve -v 2326135 /mnt/btrfs_pool1
Thanks,
Wang
Quoted 7 lines
Back to my server, I installed the new kernel, and I'm testing send again. So far it hasn't failed, so there is a chance the patch fixed the problem, but I'll only know more tomorrow after it's run longer.
Thanks,
Marc
Quoted 2 lines · Wang Shilong
Quoted 2 lines
I've applied your patch from
https://git.kernel.org/cgit/linux/kernel/git/josef/btrfs-next.git/commit/?id=1334bebe71bebbca47b3b92f25511ea980fdeab8
I can confirm this fixed the btrfs send error on my server, thank you.
Quoted 15 lines
Quoted 15 lines
At snapshot var_ro.20140225_23:28:47
ERROR: send ioctl failed with -5: Input/output error
ERROR: unexpected EOF in stream.[ 216.534313] BTRFS info (device dm-0): csum failed ino 2326136 off192512 csum 3851586574 expected csum 1402824092
Then again, this seems to be my problem on the laptop:
legolas:/mnt/btrfs_pool1# btrfs scrub status /mnt/btrfs_pool1 scrub status for 4850ee22-bf32-4131-a841-02abdb4a5ba6 scrub started at Tue Feb 25 07:35:07 2014 and finished after 1945 seconds total bytes scrubbed: 451.64GiB with 2 errors error details: csum=2 corrected errors: 0, uncorrectable errors: 2, unverified errors: 0
Quoted 6 lines
Quoted 3 lines
Ok, so that's not a btrfs send problem.
Just out of curiosity, how do I find out which inodes are compromized so that I can delete/restore them?You can use command "btrfs inspect-internal inode-resolve" which will print inode's
corresponding path with it's inode id.
Yes, sorry, I wasn't clear. I know how to do this.
What I meant is that btrfs scrub tells me there are 2 errors and does not tell me which inode/subvolume the error are in.
Aah, but they were in syslog, I just hadn't found them until now :)
Ok, I'm all good then.
Thanks,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 9 lines · Marc MERLIN
I got this during a btrfs send:
BTRFS error (device dm-2): did not find backref insend_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560I'll try a scrub when I've finished my backup, but is there anything I can run on the file I've found from the inode?
gargamel:/mnt/dshelf1/Sound# btrfs inspect-internal inode-resolve -v 22672 file.mp3 ioctl ret=0, bytes_left=3998, bytes_missing=0, cnt=1, missed=0 file.mp3
I've just seen this error:
BTRFS error (device sda4): did not find backref in send_root. inode=411890, offset=307200, disk_byte=48100618240 found extent=48100618240
during a send between two snapshots I have.
after moving to 3.14.2. I've seen it on two filesystems now since moving to 3.14. I have the two readonly snapshots if there is anything helpful I can figure out from them.
Scrub reports no errors, but I don't seem to be able to back up anything now.
David
Quoted 18 lines · David Brown
Quoted 9 lines · Marc MERLIN
I got this during a btrfs send:
BTRFS error (device dm-2): did not find backref insend_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560I'll try a scrub when I've finished my backup, but is there anything I can run on the file I've found from the inode?
gargamel:/mnt/dshelf1/Sound# btrfs inspect-internal inode-resolve -v 22672 file.mp3 ioctl ret=0, bytes_left=3998, bytes_missing=0, cnt=1, missed=0 file.mp3I've just seen this error:
BTRFS error (device sda4): did not find backref in
send_root.
inode=411890, offset=307200, disk_byte=48100618240 found extent=48100618240during a send between two snapshots I have.
Quoted 9 lines
after moving to 3.14.2. I've seen it on two filesystems now since moving to 3.14. I have the two readonly snapshots if there is anything helpful I can figure out from them.
Scrub reports no errors, but I don't seem to be able to back up anything now.
David
I am also seeing this on 3.14.1 (on ArchLinux). Scrub also reports no errors.
I could also not do a ful send. Balancing made it better for a while (I was able to send a full snapshot of one subvolume, but not another), but it did not help. Offline repairing the fs with btrfsck --repair also did not affect it.
Blaz
Quoted 13 lines · David Brown
Quoted 9 lines · Marc MERLIN
I got this during a btrfs send:
BTRFS error (device dm-2): did not find backref insend_root. inode=22672, offset=524288, disk_byte=1490517954560 found extent=1490517954560I'll try a scrub when I've finished my backup, but is there anything I can run on the file I've found from the inode?
gargamel:/mnt/dshelf1/Sound# btrfs inspect-internal inode-resolve -v 22672 file.mp3 ioctl ret=0, bytes_left=3998, bytes_missing=0, cnt=1, missed=0 file.mp3I've just seen this error:
Quoted 7 lines
BTRFS error (device sda4): did not find backref in
send_root. inode=411890, offset=307200, disk_byte=48100618240 found extent=48100618240during a send between two snapshots I have.
after moving to 3.14.2. I've seen it on two filesystems now since moving to 3.14. I have the two readonly snapshots if there is anything helpful I can figure out from them.
After bisecting it seems to be caused by
commit 7ef81ac86c8a44ab9f4e6e04e1f4c9ea53615b8a
Author: Josef Bacik <[email protected]>
Date: Fri Jan 24 14:05:42 2014 -0500
Btrfs: only process as many file extents as there are refs
The backref walking code will search down to the key it is looking for and then proceed to walk _all_ of the extents on the file until it hits the end. This is suboptimal with large files, we only need to look for as many extents as we have references for that inode. I have a testcase that creates a randomly written 4 gig file and before this patch it took 6min 30sec to do the initial send, with this patch it takes 2min 30sec to do the intial send. Thanks,
This appears to be fixed in 3.15-rc5, but I wonder if either the extra fixes, or a revert of the above should be applied to 3.14 stable?
David