Skip to contentSkip to conversation

Regression in Linux 5.5.0-rc[1-5]: btrfs send/receive out of memory

DISCUSSION Craig Andrews 4 participants 14 messages Jan 13, 2020 – Jan 28, 2020 Btrfs file system

lore
Share & tools
lore
#0 lore raw

If I perform a btrfs send receive like so:

sh -c btrfs send -p /mnt/everything/.snapshots/root.20191230 /mnt/everything/.snapshots/root.20191231 | btrfs receive /mnt/backup/.snapshots/

On Linux 5.4.0, the process completes successfully.

Starting with Linux 5.5.0-rc1 up to the current 5.5 rc, 5.5.0-rc5, the result is the OOM killer being invoked which (among other process carnage) kills the btrfs processes stopping the backup.

I'm using the same kernel config, same hardware, etc in the two tests. The system has 16 GB of RAM, CPU is an i5-6500, arch is amd64.
/mnt/everything is a btrfs fs on luks (/dev/mapper/sda4) on a 1 TB SATA SSD (/dev/sda4). /mnt/backup is btfs fs on luks (/dev/mapper/backup) on a 2 TB external spinning rust HDD (/dev/sdb).

# df -H /mnt/everything/ /mnt/backup
Filesystem          Size  Used Avail Use% Mounted on
/dev/mapper/sda4    949G  698G  247G  74% /mnt/everything
/dev/mapper/backup  2.1T  1.1T  948G  53% /mnt/backup

I've attached logs and my kernel config to https://bugzilla.kernel.org/show_bug.cgi?id=206031

Thank you,
~Craig

#1 lore raw
Quoted 11 lines · Craig Andrews

If I perform a btrfs send receive like so:

sh -c btrfs send -p /mnt/everything/.snapshots/root.20191230 /mnt/everything/.snapshots/root.20191231 | btrfs receive /mnt/backup/.snapshots/

On Linux 5.4.0, the process completes successfully.

Starting with Linux 5.5.0-rc1 up to the current 5.5 rc, 5.5.0-rc5, the result is the OOM killer being invoked which (among other process carnage) kills the btrfs processes stopping the backup.

As this is on the -rc1, it's possible that changes done in the MM subsystem could change the logic of OOM and that send now is able to trigger the OOM.

There are 2 btrfs patches in send.c but they reduce amount of work, namely for heavily reflinked extents so the effects should be opposite.

To find out where's the cause, is it possible that you build a kernel from the 5.4-based branch and run the send again? It's the same set of btrfs patches that gets merged to 5.5.

From repository: git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git
branch: for-5.5-rc4

If this can't be done we'll find another way to debug it.

#2 lore raw
Quoted 20 lines · David Sterba
Quoted 11 lines · Craig Andrews

If I perform a btrfs send receive like so:

sh -c btrfs send -p /mnt/everything/.snapshots/root.20191230 /mnt/everything/.snapshots/root.20191231 | btrfs receive /mnt/backup/.snapshots/

On Linux 5.4.0, the process completes successfully.

Starting with Linux 5.5.0-rc1 up to the current 5.5 rc, 5.5.0-rc5, the result is the OOM killer being invoked which (among other process carnage) kills the btrfs processes stopping the backup.

As this is on the -rc1, it's possible that changes done in the MM subsystem could change the logic of OOM and that send now is able to trigger the OOM.

There are 2 btrfs patches in send.c but they reduce amount of work, namely for heavily reflinked extents so the effects should be opposite.

Quoted 8 lines

To find out where's the cause, is it possible that you build a kernel from the 5.4-based branch and run the send again? It's the same set of btrfs patches that gets merged to 5.5.

From repository: git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git
branch: for-5.5-rc4

If this can't be done we'll find another way to debug it.

I've grabbed, compiled, and installed the kernel. I'll be able to tell you the results of running the test tomorrow at about this time.

Thank you for help,
~Craig

#3 lore raw
Quoted 25 lines · Craig Andrews
Quoted 24 lines · David Sterba
Quoted 11 lines · Craig Andrews

If I perform a btrfs send receive like so:

sh -c btrfs send -p /mnt/everything/.snapshots/root.20191230 /mnt/everything/.snapshots/root.20191231 | btrfs receive /mnt/backup/.snapshots/

On Linux 5.4.0, the process completes successfully.

Starting with Linux 5.5.0-rc1 up to the current 5.5 rc, 5.5.0-rc5, the result is the OOM killer being invoked which (among other process carnage) kills the btrfs processes stopping the backup.

As this is on the -rc1, it's possible that changes done in the MM subsystem could change the logic of OOM and that send now is able to trigger the OOM.

There are 2 btrfs patches in send.c but they reduce amount of work, namely for heavily reflinked extents so the effects should be opposite.

To find out where's the cause, is it possible that you build a kernel from the 5.4-based branch and run the send again? It's the same set of btrfs patches that gets merged to 5.5.

Quoted 8 lines
Quoted 4 lines

From repository: git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git
branch: for-5.5-rc4

If this can't be done we'll find another way to debug it.

I've grabbed, compiled, and installed the kernel. I'll be able to tell you the results of running the test tomorrow at about this time.

I couldn't get the kernel to boot so I need to spend more time on that.

Is there any info I can get that would be helpful using my current kernels (either 5.4 or 5.5.0-rc5) while I work out why I can't boot the kernel you linked me to?

Thanks,
~Craig

#4 lore raw
In reply to David Sterba
Quoted 20 lines · David Sterba
Quoted 11 lines · Craig Andrews

If I perform a btrfs send receive like so:

sh -c btrfs send -p /mnt/everything/.snapshots/root.20191230 /mnt/everything/.snapshots/root.20191231 | btrfs receive /mnt/backup/.snapshots/

On Linux 5.4.0, the process completes successfully.

Starting with Linux 5.5.0-rc1 up to the current 5.5 rc, 5.5.0-rc5, the result is the OOM killer being invoked which (among other process carnage) kills the btrfs processes stopping the backup.

As this is on the -rc1, it's possible that changes done in the MM subsystem could change the logic of OOM and that send now is able to trigger the OOM.

There are 2 btrfs patches in send.c but they reduce amount of work, namely for heavily reflinked extents so the effects should be opposite.

Quoted 8 lines

To find out where's the cause, is it possible that you build a kernel from the 5.4-based branch and run the send again? It's the same set of btrfs patches that gets merged to 5.5.

From repository: git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux.git
branch: for-5.5-rc4

If this can't be done we'll find another way to debug it.

I finally was able to test with this kernel.

With the kernel compiled from the branch "for-5.5-rc4" I do not get the out of memory error. However, I do this error:
btrfs send -p /mnt/everything/.snapshots/root.20200119 /mnt/everything/.snapshots/root.20200120 | mbuffer -v 1 | btrfs receive /mnt/backup/.snapshots/
ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument

I rebooted then immediately ran the same command with the 5.4 kernel and it completed successfully.

So I guess it's nice that I didn't get an out of memory error...

Thanks,
~Craig

#5 lore raw
Quoted 1 line

ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument

If I may add another data point here, I'm also encountering this issue on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as btrfs is concerned, this is an rc7).

On the first time, it happened after sending ~90 Gb worth of data, and aborted (as I didn't specify the -E option to btrfs send). Then, I retried with btrfs send -E 0, and it encountered the exact same error on the same file.

# btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv 2>/dev/pts/23 | btrfs rec -E 0 /newfs/
At subvol /tank/backups/.snaps/incoming/sendme/
At subvol sendme
ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument

The send/receive is still going on for now, currently around the ~200 Gb mark.

Bees is running on this FS (but I stopped it before doing the send/receive).

I can test patches if needed.

Signature · 1 line
Stéphane.
#6 lore raw
Quoted 5 lines · Stéphane Lesimple
Quoted 2 lines

ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument

If I may add another data point here, I'm also encountering this issue on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as btrfs is concerned, this is an rc7).

Quoted 7 lines

On the first time, it happened after sending ~90 Gb worth of data, and aborted (as I didn't specify the -E option to btrfs send). Then, I retried with btrfs send -E 0, and it encountered the exact same error on the same file.

# btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv 2>/dev/pts/23 | btrfs rec -E 0 /newfs/
At subvol /tank/backups/.snaps/incoming/sendme/
At subvol sendme
ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument

This is probably the same case for which I sent a fix last week:

https://patchwork.kernel.org/patch/11350129/

Thanks.

Quoted 9 lines

The send/receive is still going on for now, currently around the ~200 Gb mark.

Bees is running on this FS (but I stopped it before doing the send/receive).

I can test patches if needed.

--
Stéphane.

Signature · 3 lines
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”
#7 lore raw
Quoted 11 lines
Quoted 7 lines
# btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv 2>/dev/pts/23 | btrfs rec -E 0 /newfs/
At subvol /tank/backups/.snaps/incoming/sendme/
At subvol sendme
ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid
argument
ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid
argument

This is probably the same case for which I sent a fix last week:

https://patchwork.kernel.org/patch/11350129

This seems very likely, as there is indeed a hole in the middle of the file:

         0: ram    80000 disk(0) 25871fb42000:00000000-00080000 disk_sz    80000
     80000: ram    80000 disk(0) 258720475000:00000000-00080000 disk_sz    80000
    100000: ram    80000 disk(0) 2587215d4000:00000000-00080000 disk_sz    80000
    180000: ram    80000 disk(0) 258721654000:00000000-00080000 disk_sz    80000
    200000: ram    80000 disk(0) 25872177c000:00000000-00080000 disk_sz    80000
    280000: ram    80000 disk(0) 258722383000:00000000-00080000 disk_sz    80000
    300000: ram    80000 disk(0) 258722544000:00000000-00080000 disk_sz    80000
    380000: ram    80000 disk(0) 25871c3a8000:00000000-00080000 disk_sz    80000
    400000: ram    80000 disk(0) 25871c428000:00000000-00080000 disk_sz    80000
    480000: ram    80000 disk(0) 25871c548000:00000000-00080000 disk_sz    80000
    500000: ram    80000 disk(0) 25871cf02000:00000000-00080000 disk_sz    80000
    580000: ram    80000 disk(0) 25871cf82000:00000000-00080000 disk_sz    80000
    600000: ram    80000 disk(0) 25871d41f000:00000000-00080000 disk_sz    80000
    680000: ram    80000 disk(0) 25871d833000:00000000-00080000 disk_sz    80000
    700000: ram     b000 disk(0)  b9e86aeb000:00000000-0000b000 disk_sz     b000
    70b000: ram    11000 disk(0)            0:00000000-00011000 disk_sz        0 -- hole
    71c000: ram    80000 disk(0) 25871eb75000:00000000-00080000 disk_sz    80000
    79c000: ram     1000 disk(0)  a00b6523000:00000000-00001000 disk_sz     1000
    79d000: ram     1000 disk(0)  a00b6cf5000:00000000-00001000 disk_sz     1000
    79e000: ram     1000 disk(0)  a00b6cf5000:00000000-00001000 disk_sz     1000
    79f000: ram     1000 disk(0)  a00b6cf5000:00000000-00001000 disk_sz     1000
    7a0000: ram     1000 disk(0)  a00b6d42000:00000000-00001000 disk_sz     1000
    7a1000: ram     1000 disk(0)  a9573bfa000:00000000-00001000 disk_sz     6000
    7a2000: ram     1000 disk(0)  a9573bfa000:00000000-00001000 disk_sz     6000
    7a3000: ram     6000 disk(0)  a9573bfa000:00000000-00006000 disk_sz     6000
    7a9000: ram     1000 disk(0)  af4f6240000:00000000-00001000 disk_sz    2d000
    7aa000: ram     1000 disk(0)  af4f6240000:00000000-00001000 disk_sz    2d000
    7ab000: ram    2d000 disk(0)  af4f6240000:00000000-0002d000 disk_sz    2d000
    7d8000: ram     1000 disk(0) 1c8d313ba000:0001d000-0001e000 disk_sz    80000
    7d9000: ram    43000 disk(0)  a983253d000:00000000-00043000 disk_sz    43000
    81c000: ram    80000 disk(0) 258721f18000:00000000-00080000 disk_sz    80000
    89c000: ram    80000 disk(0) 258721f98000:00000000-00080000 disk_sz    80000
    91c000: ram    80000 disk(0) 258722018000:00000000-00080000 disk_sz    80000
    99c000: ram    80000 disk(0) 2587229a2000:00000000-00080000 disk_sz    80000
    a1c000: ram     d000 disk(0)  bfa6df44000:00000000-0000d000 disk_sz     e000
    a29000: ram     1000 disk(0) 130963b99000:00a29000-00a2a000 disk_sz   a2a000

file: mednafen_saturn_libretro.dll extents 29 disk size 21741568 logical size 10653836 ratio 0.49

Thanks!

Signature · 1 line
Stéphane.
#8 lore raw
In reply to Filipe Manana
Quoted 15 lines · Filipe Manana
Quoted 5 lines · Stéphane Lesimple
Quoted 2 lines

ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument

If I may add another data point here, I'm also encountering this issue on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as btrfs is concerned, this is an rc7).

Quoted 7 lines

On the first time, it happened after sending ~90 Gb worth of data, and aborted (as I didn't specify the -E option to btrfs send). Then, I retried with btrfs send -E 0, and it encountered the exact same error on the same file.

# btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv 2>/dev/pts/23 | btrfs rec -E 0 /newfs/
At subvol /tank/backups/.snaps/incoming/sendme/
At subvol sendme
ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
ERROR: failed to clone extents to retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
Quoted 5 lines

This is probably the same case for which I sent a fix last week:

https://patchwork.kernel.org/patch/11350129/

Thanks.

I applied that patch to 5.5.0-rc7 and that solved the problem. I can now do backups (which is a send/receive) successfully.

Quoted 10 lines
Quoted 10 lines

The send/receive is still going on for now, currently around the ~200 Gb mark.

Bees is running on this FS (but I stopped it before doing the send/receive).

I can test patches if needed.

--
Stéphane.

The patch appears to have fixed my problems - thank you! ~Craig

Tested-by Your Name
#9 lore raw
Quoted 42 lines · Craig Andrews
Quoted 23 lines · Filipe Manana
Quoted 12 lines · Stéphane Lesimple
Quoted 2 lines

ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument

If I may add another data point here, I'm also encountering this issue on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as btrfs is concerned, this is an rc7).

On the first time, it happened after sending ~90 Gb worth of data, and aborted (as I didn't specify the -E option to btrfs send). Then, I retried with btrfs send -E 0, and it encountered the exact same error on the same file.

Quoted 8 lines
# btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv
2>/dev/pts/23 | btrfs rec -E 0 /newfs/
At subvol /tank/backups/.snaps/incoming/sendme/
At subvol sendme
ERROR: failed to clone extents to
retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
ERROR: failed to clone extents to
retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
Quoted 5 lines

This is probably the same case for which I sent a fix last week:

https://patchwork.kernel.org/patch/11350129/

Thanks.

I applied that patch to 5.5.0-rc7 and that solved the problem. I can now do backups (which is a send/receive) successfully.

Quoted 8 lines
Quoted 8 lines

The send/receive is still going on for now, currently around the ~200 Gb mark.

Bees is running on this FS (but I stopped it before doing the send/receive).

Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

I can test patches if needed.

--
Stéphane.

The patch appears to have fixed my problems - thank you!

Great!

Can you reply to the patch's thread with a:

?

Or I can reply to it myself with that if you agree.

Thanks!

Quoted 1 line

~Craig

Signature · 3 lines
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”
author Tested-by
#10 lore raw
Quoted 54 lines · Filipe Manana
Quoted 42 lines · Craig Andrews
Quoted 23 lines · Filipe Manana
Quoted 12 lines · Stéphane Lesimple
Quoted 2 lines

ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument

If I may add another data point here, I'm also encountering this issue on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as btrfs is concerned, this is an rc7).

On the first time, it happened after sending ~90 Gb worth of data, and aborted (as I didn't specify the -E option to btrfs send). Then, I retried with btrfs send -E 0, and it encountered the exact same error on the same file.

Quoted 8 lines
# btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv
2>/dev/pts/23 | btrfs rec -E 0 /newfs/
At subvol /tank/backups/.snaps/incoming/sendme/
At subvol sendme
ERROR: failed to clone extents to
retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
ERROR: failed to clone extents to
retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
Quoted 5 lines

This is probably the same case for which I sent a fix last week:

https://patchwork.kernel.org/patch/11350129/

Thanks.

I applied that patch to 5.5.0-rc7 and that solved the problem. I can now do backups (which is a send/receive) successfully.

Quoted 8 lines
Quoted 8 lines

The send/receive is still going on for now, currently around the ~200 Gb mark.

Bees is running on this FS (but I stopped it before doing the send/receive).

Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

I can test patches if needed.

--
Stéphane.

The patch appears to have fixed my problems - thank you!

Great!

Can you reply to the patch's thread with a:

Quoted 9 lines

?

Or I can reply to it myself with that if you agree.

Thanks!

Quoted 1 line

~Craig

Can you please send the reply?

And again, thank you!
~Craig

#11 lore raw
Quoted 64 lines · Craig Andrews
Quoted 62 lines · Filipe Manana
Quoted 43 lines · Craig Andrews
Quoted 23 lines · Filipe Manana
Quoted 12 lines · Stéphane Lesimple
Quoted 2 lines

ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument

If I may add another data point here, I'm also encountering this issue on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as btrfs is concerned, this is an rc7).

On the first time, it happened after sending ~90 Gb worth of data, and aborted (as I didn't specify the -E option to btrfs send). Then, I retried with btrfs send -E 0, and it encountered the exact same error on the same file.

Quoted 8 lines
# btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv
2>/dev/pts/23 | btrfs rec -E 0 /newfs/
At subvol /tank/backups/.snaps/incoming/sendme/
At subvol sendme
ERROR: failed to clone extents to
retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
ERROR: failed to clone extents to
retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
Quoted 5 lines

This is probably the same case for which I sent a fix last week:

https://patchwork.kernel.org/patch/11350129/

Thanks.

I applied that patch to 5.5.0-rc7 and that solved the problem. I can now
do backups (which is a send/receive) successfully.

Quoted 8 lines
Quoted 8 lines

The send/receive is still going on for now, currently around the ~200 Gb mark.

Bees is running on this FS (but I stopped it before doing the send/receive).

Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

I can test patches if needed.

--
Stéphane.

The patch appears to have fixed my problems - thank you!

Great!

Can you reply to the patch's thread with a:

?

Or I can reply to it myself with that if you agree.

Quoted 7 lines
Quoted 3 lines

Thanks!

Quoted 1 line

~Craig

Can you please send the reply?

No problem, done. Thanks.

Quoted 3 lines

And again, thank you!
~Craig

Signature · 3 lines
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”
#12 lore raw
Quoted 68 lines · Filipe Manana
Quoted 66 lines · Craig Andrews
Quoted 64 lines · Filipe Manana
Quoted 43 lines · Craig Andrews
Quoted 23 lines · Filipe Manana
Quoted 12 lines · Stéphane Lesimple
Quoted 2 lines

ERROR: failed to clone extents to var/amavis/db/__db.001: Invalid argument

If I may add another data point here, I'm also encountering this issue on a 5.5.0-rc6, with the btrfs-rc7 patches applied to it (so as far as btrfs is concerned, this is an rc7).

On the first time, it happened after sending ~90 Gb worth of data, and aborted (as I didn't specify the -E option to btrfs send). Then, I retried with btrfs send -E 0, and it encountered the exact same error on the same file.

Quoted 8 lines
# btrfs send -v /tank/backups/.snaps/incoming/sendme/ | pv
2>/dev/pts/23 | btrfs rec -E 0 /newfs/
At subvol /tank/backups/.snaps/incoming/sendme/
At subvol sendme
ERROR: failed to clone extents to
retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
ERROR: failed to clone extents to
retroarch/x86_64/cores/mednafen_saturn_libretro.dll: Invalid argument
Quoted 5 lines

This is probably the same case for which I sent a fix last week:

https://patchwork.kernel.org/patch/11350129/

Thanks.

I applied that patch to 5.5.0-rc7 and that solved the problem. I can now
do backups (which is a send/receive) successfully.

Quoted 8 lines
Quoted 8 lines

The send/receive is still going on for now, currently around the ~200 Gb mark.

Bees is running on this FS (but I stopped it before doing the send/receive).

Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

I can test patches if needed.

--
Stéphane.

The patch appears to have fixed my problems - thank you!

Great!

Can you reply to the patch's thread with a:

?

Or I can reply to it myself with that if you agree.

Thanks!

Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line

~Craig

Can you please send the reply?

No problem, done. Thanks.

Actually I just realized one chunk of the patch can cause problems with fallocated extents beyond a file's size and a hole that ends right at the file's size (unlikely to be a common scenario anyway). I also don't think it's needed. For my tests at least, removing it doesn't cause any new problems and solves the problem with the invalid clone operations.

Can you try it for your use case too?

To be clear, this is the new version of the patch:

https://pastebin.com/raw/vewZiG9J

(A shorter version of the original)

Thanks!

Quoted 11 lines
Quoted 4 lines

And again, thank you!
~Craig

--
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”

Signature · 3 lines
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”