Skip to contentSkip to conversation

strange No space left on device issues

DISCUSSION Christoph Anton Mitterer 3 participants 16 messages Jul 20, 2017 1 message not in the archive Btrfs file system

lore
Share & tools
lore
#0 lore raw

Hey.

The following happens on Debian stretch systems:

# uname -a
Linux lcg-lrz-admin 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux

What I have are VMs, which run with root fs as ext4 and which I want to migrate to btrfs.
So I've added further disk images and then something like this:
- mkfs.btrfs --nodiscard --label system /dev/sdc2 (i.e. the new image)
- mounted that at /mnt
- created a subvol "root" in it
- stopped all services on the node
- remount,ro /
- mount --bind / /media
- cp -a /media/ /mnt/subvol/
- and then I'd go on move everything in place, install bootloader etc.

That used to always work, and does when I try the same with ext4 instead of btrfs on the new images.

But with btrfs I get spurious No space error like:
cp: cannot create regular file '/mnt/root/X/media/usr/share/doc/openjdk-8-jre- headless/api/java/security/PrivilegedExceptionAction.html': No space left on device
cp: cannot create regular file '/mnt/root/X/media/usr/share/doc/openjdk-8-jre- headless/api/java/security/Provider.Service.html': No space left on device
cp: cannot create regular file '/mnt/root/X/media/usr/share/doc/openjdk-8-jre- headless/api/javax/script/AbstractScriptEngine.html': No space left on device

or:
cp: preserving permissions for ‘/mnt/root/X/usr/include/c++/6/gnu/javax/crypto/keyring/BaseKeyring.h’:
No space left on device
cp: preserving permissions for ‘/mnt/root/X/usr/share/doc/cmake- data/html/variable/CMAKE_CXX_STANDARD_REQUIRED.html’: No space left on device

All these happen always (when I create a fresh btrfs on the volume and start over) with different files... and btrfs filesystem df shows plenty of space left like in terms of >15GB left.

Any ideas?

Cheers,
Chris.

#1 lore raw

Oh and I should add:
After such error, cp goes on copying (with other files)...

Same issue occurs when I do something like tar -cf - /media | tar -xf

Cheers,
Chris.

A message that is not in the archive (sent off-list or not yet received): 0102015d6083ffde-e6d15fd5-6e66-41a0-9ddf-af163c96ebd8-000000@eu-west-1.amazonses.com

#3 lore raw
Quoted 5 lines · Martin Raiber

there are patches on this list/upstream which could fix this ( e.g. "fix
delalloc accounting leak caused by u32 overflow"/"fix early ENOSPC due
to delalloc").

mhh... it's a bit problematic to test these on that nodes...

Quoted 1 line

Do you use compression?

nope...

Quoted 3 lines

It would be interesting if lowering the dirty ratio is a viable work-around (sysctl vm.dirty_background_bytes=314572800 && sysctl vm.dirty_bytes=1258291200).

doesn't seem to change anything.

#4 lore raw
In reply to a message not in the archive
Quoted 6 lines · Martin Raiber

It would be interesting if lowering the dirty ratio is a viable work-around (sysctl vm.dirty_background_bytes=314572800 && sysctl vm.dirty_bytes=1258291200).

Regards,
Martin

I took away a trailing 0 for each of them... and then it goes through without error....

sysctl vm.dirty_bytes=125829120
vm.dirty_bytes = 125829120
sysctl vm.dirty_background_bytes=31457280
vm.dirty_background_bytes = 31457280

But what does that mean now... could there be still any corruptions? And do you need to permanently set the value (until this is fixed in stable), or is this just necessary when I had this large copying operation?

Cheers,
Chris.

#5 lore raw
Quoted 17 lines · Christoph Anton Mitterer
Quoted 6 lines · Martin Raiber

It would be interesting if lowering the dirty ratio is a viable work-around (sysctl vm.dirty_background_bytes=314572800 && sysctl vm.dirty_bytes=1258291200).

Regards,
Martin

I took away a trailing 0 for each of them... and then it goes through without error....

sysctl vm.dirty_bytes=125829120
vm.dirty_bytes = 125829120
sysctl vm.dirty_background_bytes=31457280
vm.dirty_background_bytes = 31457280

Quoted 8 lines

But what does that mean now... could there be still any corruptions? And do you need to permanently set the value (until this is fixed in stable), or is this just necessary when I had this large copying operation?

Cheers,
Chris.

Could you try 4.12? If that doesn't work, could you please also try
https://patchwork.kernel.org/patch/9829593/?

#6 lore raw
Quoted 1 line · Omar Sandoval

Could you try 4.12?

Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18) x86_64 GNU/Linux
from Debian experimental, doesn't fix the issue...

Quoted 2 lines

If that doesn't work, could you please also try
https://patchwork.kernel.org/patch/9829593/?

Against 4.9?

Cheers,
Chris.

#7 lore raw
Quoted 5 lines · Christoph Anton Mitterer
Quoted 1 line · Omar Sandoval

Could you try 4.12?

Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18) x86_64 GNU/Linux
from Debian experimental, doesn't fix the issue...

Okay, didn't think it would :)

Quoted 3 lines
Quoted 2 lines

If that doesn't work, could you please also try
https://patchwork.kernel.org/patch/9829593/?

Against 4.9?

Against 4.12 would be best, thanks!

#8 lore raw
Quoted 1 line · Omar Sandoval

Against 4.12 would be best, thanks!

okay,.. but that will take a while to compile...

in the meantime... do you know whether it's more or less safe to use the 4.9 kernel without any fix, when I change the parameters mentioned before, during the massive copying?

Cheers,
Chris.

#9 lore raw
Quoted 8 lines · Christoph Anton Mitterer
Quoted 1 line · Omar Sandoval

Against 4.12 would be best, thanks!

okay,.. but that will take a while to compile...

in the meantime... do you know whether it's more or less safe to use the 4.9 kernel without any fix, when I change the parameters mentioned before, during the massive copying?

Yes, that's a safe enough workaround. It's a good idea to change the parameters back after the copy.

Quoted 2 lines

Cheers,
Chris.

#10 lore raw
Quoted 2 lines · Omar Sandoval

Yes, that's a safe enough workaround. It's a good idea to change the parameters back after the copy.

you mean even without having the fix, right?

So AFAIU, the bug doesn't really cause FS corruption, but just "false" ENOSPC and these happen during having meta-data creating (e.g. during operations like mine) only?

#11 lore raw
Quoted 4 lines · Christoph Anton Mitterer
Quoted 2 lines · Omar Sandoval

Yes, that's a safe enough workaround. It's a good idea to change the parameters back after the copy.

you mean even without having the fix, right?

Yes, even without the fix you should be okay.

Quoted 3 lines

So AFAIU, the bug doesn't really cause FS corruption, but just "false" ENOSPC and these happen during having meta-data creating (e.g. during operations like mine) only?

Right.

#12 lore raw
In reply to Omar Sandoval
Quoted 2 lines · Omar Sandoval

If that doesn't work, could you please also try
https://patchwork.kernel.org/patch/9829593/?

Okay, tried the patch now, applied upon:
Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18) x86_64 GNU/Linux (that is the Debian source package, with all their further patches and their kernel config).

with the parameters at their defaults:

# sysctl vm.dirty_bytes
vm.dirty_bytes = 0
# sysctl vm.dirty_background_bytes
vm.dirty_background_bytes = 0

Tried copying the whole image three times (before I haven had a single copy of the whole image with at least one error, so that should be "proof" enough that it fixes the isse) upon the btrfs fs,... no errors this time...

Looks good :-)

I assume you'll take care to get that patch into stable kernels? Is this patch alone enough to recommend the Debian maintainers to include it into their 4.9 long term stable kernels?

And would you recommend this as an "urgent" fix?

Cheers,
Chris.

#13 lore raw
Quoted 9 lines · Christoph Anton Mitterer
Quoted 2 lines · Omar Sandoval

If that doesn't work, could you please also try
https://patchwork.kernel.org/patch/9829593/?

Okay, tried the patch now, applied upon:
Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18) x86_64 GNU/Linux (that is the Debian source package, with all their further patches and their kernel config).

Quoted 5 lines

with the parameters at their defaults:

# sysctl vm.dirty_bytes
vm.dirty_bytes = 0
# sysctl vm.dirty_background_bytes
vm.dirty_background_bytes = 0

Just to be sure, did you explicitly write 0 to these? These sysctls are really confusing, see https://www.kernel.org/doc/Documentation/sysctl/vm.txt.
Basically, there are two ways to specify these, either as a ratio of system memory (vm.dirty_ratio and vm.dirty_background_ratio) or a static number of bytes (vm.dirty_bytes and vm.dirty_background_bytes). If you set one, the other appears as 0, and the kernel sets the ratios by default. But if you explicitly set them to 0, the kernel is going to flush stuff extremely aggressively.

Quoted 6 lines

Tried copying the whole image three times (before I haven had a single copy of the whole image with at least one error, so that should be "proof" enough that it fixes the isse) upon the btrfs fs,... no errors this time...

Looks good :-)

Awesome, glad to hear it! I hadn't been able to reproduce the issue outside of Facebook. Can I add your tested-by?

Quoted 3 lines

I assume you'll take care to get that patch into stable kernels? Is this patch alone enough to recommend the Debian maintainers to include it into their 4.9 long term stable kernels?

I'll mark it for stable, assuming Debian tracks the upstream LTS releases it should get in.

Quoted 1 line

And would you recommend this as an "urgent" fix?

This bug has been around since 4.8, so it's not _that_ urgent, but it sucks.

Thanks!

Quoted 2 lines

Cheers,
Chris.

#14 lore raw
Quoted 1 line · Omar Sandoval

Just to be sure, did you explicitly write 0 to these?

Nope... that seemed to have been the default value, i.e. I used sysctl(8) in read (and not set) mode here.

Quoted 11 lines

These sysctls are
really confusing, see https://www.kernel.org/doc/Documentation/sysctl
/vm.txt.
Basically, there are two ways to specify these, either as a ratio of system memory (vm.dirty_ratio and vm.dirty_background_ratio) or a static
number of bytes (vm.dirty_bytes and vm.dirty_background_bytes). If you
set one, the other appears as 0, and the kernel sets the ratios by default. But if you explicitly set them to 0, the kernel is going to flush stuff extremely aggressively.

I see,... not sure why both are 0 here... at least I didn't change it myself - must be something from the distro?

Quoted 2 lines

Awesome, glad to hear it! I hadn't been able to reproduce the issue outside of Facebook. Can I add your tested-by?

Sure, but better use my other mail address for it, if you don't mind:
Christoph Anton Mitterer <[email protected]>

Quoted 6 lines
Quoted 3 lines

I assume you'll take care to get that patch into stable kernels? Is this patch alone enough to recommend the Debian maintainers to include it into their 4.9 long term stable kernels?

I'll mark it for stable, assuming Debian tracks the upstream LTS releases it should get in.

Okay :-)

Nevertheless I'll open a bug at their BTS, just to be safe.

Thanks :)

Chris.

#15 lore raw
Quoted 4 lines · Christoph Anton Mitterer
Quoted 1 line · Omar Sandoval

Just to be sure, did you explicitly write 0 to these?

Nope... that seemed to have been the default value, i.e. I used sysctl(8) in read (and not set) mode here.

Okay, all good then.

Quoted 11 lines
Quoted 11 lines

These sysctls are
really confusing, see https://www.kernel.org/doc/Documentation/sysctl
/vm.txt.
Basically, there are two ways to specify these, either as a ratio of system memory (vm.dirty_ratio and vm.dirty_background_ratio) or a static
number of bytes (vm.dirty_bytes and vm.dirty_background_bytes). If you
set one, the other appears as 0, and the kernel sets the ratios by default. But if you explicitly set them to 0, the kernel is going to flush stuff extremely aggressively.

Quoted 2 lines

I see,... not sure why both are 0 here... at least I didn't change it myself - must be something from the distro?

That's normal, the default is to have the ratio set instead:

$ sysctl vm.dirty_{,background_}{bytes,ratio}
vm.dirty_bytes = 0
vm.dirty_ratio = 20
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
Quoted 4 lines
Quoted 2 lines

Awesome, glad to hear it! I hadn't been able to reproduce the issue outside of Facebook. Can I add your tested-by?

Sure, but better use my other mail address for it, if you don't mind:
Christoph Anton Mitterer <[email protected]>

No problem. I'll resend the patch with that shortly.

Quoted 14 lines
Quoted 6 lines
Quoted 3 lines

I assume you'll take care to get that patch into stable kernels? Is this patch alone enough to recommend the Debian maintainers to include it into their 4.9 long term stable kernels?

I'll mark it for stable, assuming Debian tracks the upstream LTS releases it should get in.

Okay :-)

Nevertheless I'll open a bug at their BTS, just to be safe.

Thanks :)

Chris.

#16 lore raw
In reply to Christoph Anton Mitterer
Quoted 9 lines · Christoph Anton Mitterer

On Thu, 2017-07-20 at 14:48 -0700, Omar Sandoval wrote:
[...]

Quoted 6 lines
Quoted 3 lines

I assume you'll take care to get that patch into stable kernels? Is this patch alone enough to recommend the Debian maintainers to include it into their 4.9 long term stable kernels?

I'll mark it for stable, assuming Debian tracks the upstream LTS releases it should get in.

+1 \o/

Quoted 3 lines

Okay :-)

Nevertheless I'll open a bug at their BTS, just to be safe.

The Debian kernel teams tries to minimize the amount of patches on top of the original kernel code. Every patch added comes with a maintenance burden.

So, the answer will probably be: Please get it in 4.9 itself.

Signature · 1 line
Hans van Kranenburg