Regression in Linux 5.5.0-rc[1-5]: btrfs send/receive out of memory
If I perform a btrfs send receive like so:
sh -c btrfs send -p /mnt/everything/.snapshots/root.20191230 /mnt/everything/.snapshots/root.20191231 | btrfs receive /mnt/backup/.snapshots/
On Linux 5.4.0, the process completes successfully.
Starting with Linux 5.5.0-rc1 up to the current 5.5 rc, 5.5.0-rc5, the result is the OOM killer being invoked which (among other process carnage) kills the btrfs processes stopping the backup.
I'm using the same kernel config, same hardware, etc in the two tests. The system has 16 GB of RAM, CPU is an i5-6500, arch is amd64./mnt/everything is a btrfs fs on luks (/dev/mapper/sda4) on a 1 TB SATA SSD (/dev/sda4). /mnt/backup is btfs fs on luks (/dev/mapper/backup) on a 2 TB external spinning rust HDD (/dev/sdb).
# df -H /mnt/everything/ /mnt/backup Filesystem Size Used Avail Use% Mounted on /dev/mapper/sda4 949G 698G 247G 74% /mnt/everything /dev/mapper/backup 2.1T 1.1T 948G 53% /mnt/backup
I've attached logs and my kernel config to https://bugzilla.kernel.org/show_bug.cgi?id=206031
Thank you,
~Craig
Quoted 11 lines · Craig Andrews
If I perform a btrfs send receive like so:
sh -c btrfs send -p
/mnt/everything/.snapshots/root.20191230/mnt/everything/.snapshots/root.20191231| btrfs receive/mnt/backup/.snapshots/On Linux 5.4.0, the process completes successfully.
Starting with Linux 5.5.0-rc1 up to the current 5.5 rc, 5.5.0-rc5, the result is the OOM killer being invoked which (among other process carnage) kills the btrfs processes stopping the backup.
As this is on the -rc1, it's possible that changes done in the MM subsystem could change the logic of OOM and that send now is able to trigger the OOM.
There are 2 btrfs patches in send.c but they reduce amount of work, namely for heavily reflinked extents so the effects should be opposite.
To find out where's the cause, is it possible that you build a kernel from the 5.4-based branch and run the send again? It's the same set of btrfs patches that gets merged to 5.5.
From repository: git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git
branch: for-5.5-rc4
If this can't be done we'll find another way to debug it.
Quoted 20 lines · David Sterba
Quoted 11 lines · Craig Andrews
If I perform a btrfs send receive like so:
sh -c btrfs send -p
/mnt/everything/.snapshots/root.20191230/mnt/everything/.snapshots/root.20191231| btrfs receive/mnt/backup/.snapshots/On Linux 5.4.0, the process completes successfully.
Starting with Linux 5.5.0-rc1 up to the current 5.5 rc, 5.5.0-rc5, the result is the OOM killer being invoked which (among other process carnage) kills the btrfs processes stopping the backup.
As this is on the -rc1, it's possible that changes done in the MM subsystem could change the logic of OOM and that send now is able to trigger the OOM.
There are 2 btrfs patches in
send.cbut they reduce amount of work, namely for heavily reflinked extents so the effects should be opposite.
Quoted 8 lines
To find out where's the cause, is it possible that you build a kernel from the 5.4-based branch and run the send again? It's the same set of btrfs patches that gets merged to 5.5.
From repository:
git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git
branch: for-5.5-rc4If this can't be done we'll find another way to debug it.
I've grabbed, compiled, and installed the kernel. I'll be able to tell you the results of running the test tomorrow at about this time.
Thank you for help,
~Craig
Quoted 25 lines · Craig Andrews
Quoted 24 lines · David Sterba
Quoted 11 lines · Craig Andrews
If I perform a btrfs send receive like so:
sh -c btrfs send -p
/mnt/everything/.snapshots/root.20191230/mnt/everything/.snapshots/root.20191231| btrfs receive/mnt/backup/.snapshots/On Linux 5.4.0, the process completes successfully.
Starting with Linux 5.5.0-rc1 up to the current 5.5 rc, 5.5.0-rc5, the result is the OOM killer being invoked which (among other process carnage) kills the btrfs processes stopping the backup.
As this is on the -rc1, it's possible that changes done in the MM subsystem could change the logic of OOM and that send now is able to trigger the OOM.
There are 2 btrfs patches in
send.cbut they reduce amount of work, namely for heavily reflinked extents so the effects should be opposite.To find out where's the cause, is it possible that you build a kernel from the 5.4-based branch and run the send again? It's the same set of btrfs patches that gets merged to 5.5.
Quoted 8 lines
Quoted 4 lines
From repository:
git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git
branch: for-5.5-rc4If this can't be done we'll find another way to debug it.
I've grabbed, compiled, and installed the kernel. I'll be able to tell you the results of running the test tomorrow at about this time.
I couldn't get the kernel to boot so I need to spend more time on that.
Is there any info I can get that would be helpful using my current kernels (either 5.4 or 5.5.0-rc5) while I work out why I can't boot the kernel you linked me to?
Thanks,
~Craig
Quoted 20 lines · David Sterba
Quoted 11 lines · Craig Andrews
If I perform a btrfs send receive like so:
sh -c btrfs send -p
/mnt/everything/.snapshots/root.20191230/mnt/everything/.snapshots/root.20191231| btrfs receive/mnt/backup/.snapshots/On Linux 5.4.0, the process completes successfully.
Starting with Linux 5.5.0-rc1 up to the current 5.5 rc, 5.5.0-rc5, the result is the OOM killer being invoked which (among other process carnage) kills the btrfs processes stopping the backup.
As this is on the -rc1, it's possible that changes done in the MM subsystem could change the logic of OOM and that send now is able to trigger the OOM.
There are 2 btrfs patches in
send.cbut they reduce amount of work, namely for heavily reflinked extents so the effects should be opposite.
Quoted 8 lines
To find out where's the cause, is it possible that you build a kernel from the 5.4-based branch and run the send again? It's the same set of btrfs patches that gets merged to 5.5.
From repository:
git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git
branch: for-5.5-rc4If this can't be done we'll find another way to debug it.
I finally was able to test with this kernel.
With the kernel compiled from the branch "for-5.5-rc4" I do not get the out of memory error. However, I do this error:
btrfs send -p /mnt/everything/.snapshots/root.20200119 /mnt/everything/.snapshots/root.20200120 | mbuffer -v 1 | btrfs receive /mnt/backup/.snapshots/
ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument
I rebooted then immediately ran the same command with the 5.4 kernel and it completed successfully.
So I guess it's nice that I didn't get an out of memory error...
Thanks,
~Craig
Quoted 1 line
ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument
If I may add another data point here, I'm also encountering this issue on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as btrfs is concerned, this is an rc7).
On the first time, it happened after sending ~90 Gb worth of data, and aborted (as I didn't specify the -E option to btrfs send). Then, I retried with btrfs send -E 0, and it encountered the exact same error on the same file.
# btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv 2>/dev/pts/23 | btrfs rec -E 0 /newfs/ At subvol /tank/backups/.snaps/incoming/sendme/ At subvol sendme ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
The send/receive is still going on for now, currently around the ~200 Gb mark.
Bees is running on this FS (but I stopped it before doing the send/receive).
I can test patches if needed.
Signature · 1 line
Stéphane.
Quoted 5 lines · Stéphane Lesimple
Quoted 2 lines
ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument
If I may add another data point here, I'm also encountering this issue on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as btrfs is concerned, this is an rc7).
Quoted 7 lines
On the first time, it happened after sending ~90 Gb worth of data, and aborted (as I didn't specify the -E option to btrfs send). Then, I retried with btrfs send -E 0, and it encountered the exact same error on the same file.
# btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv 2>/dev/pts/23 | btrfs rec -E 0 /newfs/ At subvol /tank/backups/.snaps/incoming/sendme/ At subvol sendme ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
This is probably the same case for which I sent a fix last week:
https://patchwork.kernel.org/patch/11350129/
Thanks.
Quoted 9 lines
The send/receive is still going on for now, currently around the ~200 Gb mark.
Bees is running on this FS (but I stopped it before doing the send/receive).
I can test patches if needed.
--
Stéphane.
Signature · 3 lines
Filipe David Manana, “Whether you think you can, or you think you can't — you're right.”
Quoted 11 lines
Quoted 7 lines
# btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv 2>/dev/pts/23 | btrfs rec -E 0 /newfs/ At subvol /tank/backups/.snaps/incoming/sendme/ At subvol sendme ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argumentThis is probably the same case for which I sent a fix last week:
This seems very likely, as there is indeed a hole in the middle of the file:
0: ram 80000 disk(0) 25871fb42000:00000000-00080000 disk_sz 80000
80000: ram 80000 disk(0) 258720475000:00000000-00080000 disk_sz 80000
100000: ram 80000 disk(0) 2587215d4000:00000000-00080000 disk_sz 80000
180000: ram 80000 disk(0) 258721654000:00000000-00080000 disk_sz 80000
200000: ram 80000 disk(0) 25872177c000:00000000-00080000 disk_sz 80000
280000: ram 80000 disk(0) 258722383000:00000000-00080000 disk_sz 80000
300000: ram 80000 disk(0) 258722544000:00000000-00080000 disk_sz 80000
380000: ram 80000 disk(0) 25871c3a8000:00000000-00080000 disk_sz 80000
400000: ram 80000 disk(0) 25871c428000:00000000-00080000 disk_sz 80000
480000: ram 80000 disk(0) 25871c548000:00000000-00080000 disk_sz 80000
500000: ram 80000 disk(0) 25871cf02000:00000000-00080000 disk_sz 80000
580000: ram 80000 disk(0) 25871cf82000:00000000-00080000 disk_sz 80000
600000: ram 80000 disk(0) 25871d41f000:00000000-00080000 disk_sz 80000
680000: ram 80000 disk(0) 25871d833000:00000000-00080000 disk_sz 80000
700000: ram b000 disk(0) b9e86aeb000:00000000-0000b000 disk_sz b000
70b000: ram 11000 disk(0) 0:00000000-00011000 disk_sz 0 -- hole
71c000: ram 80000 disk(0) 25871eb75000:00000000-00080000 disk_sz 80000
79c000: ram 1000 disk(0) a00b6523000:00000000-00001000 disk_sz 1000
79d000: ram 1000 disk(0) a00b6cf5000:00000000-00001000 disk_sz 1000
79e000: ram 1000 disk(0) a00b6cf5000:00000000-00001000 disk_sz 1000
79f000: ram 1000 disk(0) a00b6cf5000:00000000-00001000 disk_sz 1000
7a0000: ram 1000 disk(0) a00b6d42000:00000000-00001000 disk_sz 1000
7a1000: ram 1000 disk(0) a9573bfa000:00000000-00001000 disk_sz 6000
7a2000: ram 1000 disk(0) a9573bfa000:00000000-00001000 disk_sz 6000
7a3000: ram 6000 disk(0) a9573bfa000:00000000-00006000 disk_sz 6000
7a9000: ram 1000 disk(0) af4f6240000:00000000-00001000 disk_sz 2d000
7aa000: ram 1000 disk(0) af4f6240000:00000000-00001000 disk_sz 2d000
7ab000: ram 2d000 disk(0) af4f6240000:00000000-0002d000 disk_sz 2d000
7d8000: ram 1000 disk(0) 1c8d313ba000:0001d000-0001e000 disk_sz 80000
7d9000: ram 43000 disk(0) a983253d000:00000000-00043000 disk_sz 43000
81c000: ram 80000 disk(0) 258721f18000:00000000-00080000 disk_sz 80000
89c000: ram 80000 disk(0) 258721f98000:00000000-00080000 disk_sz 80000
91c000: ram 80000 disk(0) 258722018000:00000000-00080000 disk_sz 80000
99c000: ram 80000 disk(0) 2587229a2000:00000000-00080000 disk_sz 80000
a1c000: ram d000 disk(0) bfa6df44000:00000000-0000d000 disk_sz e000
a29000: ram 1000 disk(0) 130963b99000:00a29000-00a2a000 disk_sz a2a000file: mednafen_saturn_libretro.dll extents 29 disk size 21741568 logical size 10653836 ratio 0.49
Thanks!
Signature · 1 line
Stéphane.
Quoted 15 lines · Filipe Manana
Quoted 5 lines · Stéphane Lesimple
Quoted 2 lines
ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument
If I may add another data point here, I'm also encountering this issue on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as btrfs is concerned, this is an rc7).
Quoted 7 lines
On the first time, it happened after sending ~90 Gb worth of data, and aborted (as I didn't specify the -E option to btrfs send). Then, I retried with btrfs send -E 0, and it encountered the exact same error on the same file.
# btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv 2>/dev/pts/23 | btrfs rec -E 0 /newfs/ At subvol /tank/backups/.snaps/incoming/sendme/ At subvol sendme ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
Quoted 5 lines
This is probably the same case for which I sent a fix last week:
https://patchwork.kernel.org/patch/11350129/
Thanks.
I applied that patch to 5.5.0-rc7 and that solved the problem. I can now do backups (which is a send/receive) successfully.
Quoted 10 lines
Quoted 10 lines
The send/receive is still going on for now, currently around the ~200 Gb mark.
Bees is running on this FS (but I stopped it before doing the send/receive).
I can test patches if needed.
--
Stéphane.
The patch appears to have fixed my problems - thank you! ~Craig
Filipe Manana
Tested-by Your NameQuoted 42 lines · Craig Andrews
Quoted 23 lines · Filipe Manana
Quoted 12 lines · Stéphane Lesimple
Quoted 2 lines
ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument
If I may add another data point here, I'm also encountering this issue on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as btrfs is concerned, this is an rc7).
On the first time, it happened after sending ~90 Gb worth of data, and aborted (as I didn't specify the -E option to btrfs send). Then, I retried with btrfs send -E 0, and it encountered the exact same error on the same file.
Quoted 8 lines
# btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv 2>/dev/pts/23 | btrfs rec -E 0 /newfs/ At subvol /tank/backups/.snaps/incoming/sendme/ At subvol sendme ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argumentQuoted 5 lines
This is probably the same case for which I sent a fix last week:
https://patchwork.kernel.org/patch/11350129/
Thanks.
I applied that patch to 5.5.0-rc7 and that solved the problem. I can now do backups (which is a send/receive) successfully.
Quoted 8 lines
Quoted 8 lines
The send/receive is still going on for now, currently around the ~200 Gb mark.
Bees is running on this FS (but I stopped it before doing the send/receive).
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
I can test patches if needed.
--
Stéphane.The patch appears to have fixed my problems - thank you!
Great!
Can you reply to the patch's thread with a:
?
Or I can reply to it myself with that if you agree.
Thanks!
Quoted 1 line
~Craig
Signature · 3 lines
Filipe David Manana, “Whether you think you can, or you think you can't — you're right.”
Craig Andrews
Tested-byQuoted 54 lines · Filipe Manana
Quoted 42 lines · Craig Andrews
Quoted 23 lines · Filipe Manana
Quoted 12 lines · Stéphane Lesimple
Quoted 2 lines
ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument
If I may add another data point here, I'm also encountering this issue on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as btrfs is concerned, this is an rc7).
On the first time, it happened after sending ~90 Gb worth of data, and aborted (as I didn't specify the -E option to btrfs send). Then, I retried with btrfs send -E 0, and it encountered the exact same error on the same file.
Quoted 8 lines
# btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv 2>/dev/pts/23 | btrfs rec -E 0 /newfs/ At subvol /tank/backups/.snaps/incoming/sendme/ At subvol sendme ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argumentQuoted 5 lines
This is probably the same case for which I sent a fix last week:
https://patchwork.kernel.org/patch/11350129/
Thanks.
I applied that patch to 5.5.0-rc7 and that solved the problem. I can now do backups (which is a send/receive) successfully.
Quoted 8 lines
Quoted 8 lines
The send/receive is still going on for now, currently around the ~200 Gb mark.
Bees is running on this FS (but I stopped it before doing the send/receive).
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
I can test patches if needed.
--
Stéphane.The patch appears to have fixed my problems - thank you!
Great!
Can you reply to the patch's thread with a:
Quoted 9 lines
Tested-by Your Name <[email protected]>?
Or I can reply to it myself with that if you agree.
Thanks!
Quoted 1 line
~Craig
Can you please send the reply?
And again, thank you!
~Craig
Quoted 64 lines · Craig Andrews
Quoted 62 lines · Filipe Manana
Quoted 43 lines · Craig Andrews
Quoted 23 lines · Filipe Manana
Quoted 12 lines · Stéphane Lesimple
Quoted 2 lines
ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument
If I may add another data point here, I'm also encountering this issue on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as btrfs is concerned, this is an rc7).
On the first time, it happened after sending ~90 Gb worth of data, and aborted (as I didn't specify the -E option to btrfs send). Then, I retried with btrfs send -E 0, and it encountered the exact same error on the same file.
Quoted 8 lines
# btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv 2>/dev/pts/23 | btrfs rec -E 0 /newfs/ At subvol /tank/backups/.snaps/incoming/sendme/ At subvol sendme ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argumentQuoted 5 lines
This is probably the same case for which I sent a fix last week:
https://patchwork.kernel.org/patch/11350129/
Thanks.
I applied that patch to 5.5.0-rc7 and that solved the problem. I can now
do backups (which is a send/receive) successfully.Quoted 8 lines
Quoted 8 lines
The send/receive is still going on for now, currently around the ~200 Gb mark.
Bees is running on this FS (but I stopped it before doing the send/receive).
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
I can test patches if needed.
--
Stéphane.The patch appears to have fixed my problems - thank you!
Great!
Can you reply to the patch's thread with a:
Tested-by Your Name <[email protected]>?
Or I can reply to it myself with that if you agree.
Quoted 7 lines
Quoted 3 lines
Thanks!
Quoted 1 line
~Craig
Can you please send the reply?
Tested-by Craig Andrews <[email protected]>
No problem, done. Thanks.
Quoted 3 lines
And again, thank you!
~Craig
Signature · 3 lines
Filipe David Manana, “Whether you think you can, or you think you can't — you're right.”
Quoted 68 lines · Filipe Manana
Quoted 66 lines · Craig Andrews
Quoted 64 lines · Filipe Manana
Quoted 43 lines · Craig Andrews
Quoted 23 lines · Filipe Manana
Quoted 12 lines · Stéphane Lesimple
Quoted 2 lines
ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument
If I may add another data point here, I'm also encountering this issue on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as btrfs is concerned, this is an rc7).
On the first time, it happened after sending ~90 Gb worth of data, and aborted (as I didn't specify the -E option to btrfs send). Then, I retried with btrfs send -E 0, and it encountered the exact same error on the same file.
Quoted 8 lines
# btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv 2>/dev/pts/23 | btrfs rec -E 0 /newfs/ At subvol /tank/backups/.snaps/incoming/sendme/ At subvol sendme ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argumentQuoted 5 lines
This is probably the same case for which I sent a fix last week:
https://patchwork.kernel.org/patch/11350129/
Thanks.
I applied that patch to 5.5.0-rc7 and that solved the problem. I can now
do backups (which is a send/receive) successfully.Quoted 8 lines
Quoted 8 lines
The send/receive is still going on for now, currently around the ~200 Gb mark.
Bees is running on this FS (but I stopped it before doing the send/receive).
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
I can test patches if needed.
--
Stéphane.The patch appears to have fixed my problems - thank you!
Great!
Can you reply to the patch's thread with a:
Tested-by Your Name <[email protected]>?
Or I can reply to it myself with that if you agree.
Thanks!
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
~Craig
Can you please send the reply?
Tested-by Craig Andrews <[email protected]>No problem, done. Thanks.
Actually I just realized one chunk of the patch can cause problems with fallocated extents beyond a file's size and a hole that ends right at the file's size (unlikely to be a common scenario anyway). I also don't think it's needed. For my tests at least, removing it doesn't cause any new problems and solves the problem with the invalid clone operations.
Can you try it for your use case too?
To be clear, this is the new version of the patch:
https://pastebin.com/raw/vewZiG9J
(A shorter version of the original)
Thanks!
Quoted 11 lines
Quoted 4 lines
And again, thank you!
~Craig--
Filipe David Manana,“Whether you think you can, or you think you can't — you're right.”
Signature · 3 lines
Filipe David Manana, “Whether you think you can, or you think you can't — you're right.”
Quoted 2 lines · Craig Andrews
Can you please send the reply?
Tested-by Craig Andrews <[email protected]>
Thanks for the report and testing! As the patch is a fix it will go to mainline soon and then it'll propagate to stable trees, 5.4 and 5.5.