strange No space left on device issues
Hey.
The following happens on Debian stretch systems:
# uname -a Linux lcg-lrz-admin 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux
What I have are VMs, which run with root fs as ext4 and which I want to migrate to btrfs.
So I've added further disk images and then something like this:
- mkfs.btrfs --nodiscard --label system /dev/sdc2 (i.e. the new image)
- mounted that at /mnt
- created a subvol "root" in it
- stopped all services on the node
- remount,ro /
- mount --bind / /media
- cp -a /media/ /mnt/subvol/
- and then I'd go on move everything in place, install bootloader etc.
That used to always work, and does when I try the same with ext4 instead of btrfs on the new images.
But with btrfs I get spurious No space error like:
cp: cannot create regular file '/mnt/root/X/media/usr/share/doc/openjdk-8-jre- headless/api/java/security/PrivilegedExceptionAction.html': No space left on device
cp: cannot create regular file '/mnt/root/X/media/usr/share/doc/openjdk-8-jre- headless/api/java/security/Provider.Service.html': No space left on device
cp: cannot create regular file '/mnt/root/X/media/usr/share/doc/openjdk-8-jre- headless/api/javax/script/AbstractScriptEngine.html': No space left on device
or:
cp: preserving permissions for ‘/mnt/root/X/usr/include/c++/6/gnu/javax/crypto/keyring/BaseKeyring.h’:
No space left on device
cp: preserving permissions for ‘/mnt/root/X/usr/share/doc/cmake- data/html/variable/CMAKE_CXX_STANDARD_REQUIRED.html’: No space left on device
All these happen always (when I create a fresh btrfs on the volume and start over) with different files... and btrfs filesystem df shows plenty of space left like in terms of >15GB left.
Any ideas?
Cheers,
Chris.
Oh and I should add:
After such error, cp goes on copying (with other files)...
Same issue occurs when I do something like tar -cf - /media | tar -xf
Cheers,
Chris.
A message that is not in the archive (sent off-list or not yet received): 0102015d6083ffde-e6d15fd5-6e66-41a0-9ddf-af163c96ebd8-000000@eu-west-1.amazonses.com
Quoted 5 lines · Martin Raiber
there are patches on this list/upstream which could fix this ( e.g. "fix
delalloc accounting leak caused by u32 overflow"/"fix early ENOSPC due
to delalloc").
mhh... it's a bit problematic to test these on that nodes...
Quoted 1 line
Do you use compression?
nope...
Quoted 3 lines
It would be interesting if lowering the dirty ratio is a viable work-around (sysctl vm.dirty_background_bytes=314572800 && sysctl vm.dirty_bytes=1258291200).
doesn't seem to change anything.
Quoted 6 lines · Martin Raiber
It would be interesting if lowering the dirty ratio is a viable work-around (sysctl vm.dirty_background_bytes=314572800 && sysctl vm.dirty_bytes=1258291200).
Regards,
Martin
I took away a trailing 0 for each of them... and then it goes through without error....
sysctl vm.dirty_bytes=125829120
vm.dirty_bytes = 125829120
sysctl vm.dirty_background_bytes=31457280
vm.dirty_background_bytes = 31457280
But what does that mean now... could there be still any corruptions? And do you need to permanently set the value (until this is fixed in stable), or is this just necessary when I had this large copying operation?
Cheers,
Chris.
Quoted 17 lines · Christoph Anton Mitterer
Quoted 6 lines · Martin Raiber
It would be interesting if lowering the dirty ratio is a viable work-around (sysctl vm.dirty_background_bytes=314572800 && sysctl vm.dirty_bytes=1258291200).
Regards,
MartinI took away a trailing 0 for each of them... and then it goes through without error....
sysctl vm.dirty_bytes=125829120
vm.dirty_bytes = 125829120
sysctl vm.dirty_background_bytes=31457280
vm.dirty_background_bytes = 31457280
Quoted 8 lines
But what does that mean now... could there be still any corruptions? And do you need to permanently set the value (until this is fixed in stable), or is this just necessary when I had this large copying operation?
Cheers,
Chris.
Could you try 4.12? If that doesn't work, could you please also try
https://patchwork.kernel.org/patch/9829593/?
Quoted 1 line · Omar Sandoval
Could you try 4.12?
Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18) x86_64 GNU/Linux
from Debian experimental, doesn't fix the issue...
Quoted 2 lines
If that doesn't work, could you please also try
https://patchwork.kernel.org/patch/9829593/?
Against 4.9?
Cheers,
Chris.
Quoted 5 lines · Christoph Anton Mitterer
Quoted 1 line · Omar Sandoval
Could you try 4.12?
Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18)
x86_64GNU/Linux
from Debian experimental, doesn't fix the issue...
Okay, didn't think it would :)
Quoted 3 lines
Quoted 2 lines
If that doesn't work, could you please also try
https://patchwork.kernel.org/patch/9829593/?Against 4.9?
Against 4.12 would be best, thanks!
Quoted 1 line · Omar Sandoval
Against 4.12 would be best, thanks!
okay,.. but that will take a while to compile...
in the meantime... do you know whether it's more or less safe to use the 4.9 kernel without any fix, when I change the parameters mentioned before, during the massive copying?
Cheers,
Chris.
Quoted 8 lines · Christoph Anton Mitterer
Quoted 1 line · Omar Sandoval
Against 4.12 would be best, thanks!
okay,.. but that will take a while to compile...
in the meantime... do you know whether it's more or less safe to use the 4.9 kernel without any fix, when I change the parameters mentioned before, during the massive copying?
Yes, that's a safe enough workaround. It's a good idea to change the parameters back after the copy.
Quoted 2 lines
Cheers,
Chris.
Quoted 2 lines · Omar Sandoval
Yes, that's a safe enough workaround. It's a good idea to change the parameters back after the copy.
you mean even without having the fix, right?
So AFAIU, the bug doesn't really cause FS corruption, but just "false" ENOSPC and these happen during having meta-data creating (e.g. during operations like mine) only?
Quoted 4 lines · Christoph Anton Mitterer
Quoted 2 lines · Omar Sandoval
Yes, that's a safe enough workaround. It's a good idea to change the parameters back after the copy.
you mean even without having the fix, right?
Yes, even without the fix you should be okay.
Quoted 3 lines
So AFAIU, the bug doesn't really cause FS corruption, but just "false" ENOSPC and these happen during having meta-data creating (e.g. during operations like mine) only?
Right.
Quoted 2 lines · Omar Sandoval
If that doesn't work, could you please also try
https://patchwork.kernel.org/patch/9829593/?
Okay, tried the patch now, applied upon:
Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18) x86_64 GNU/Linux (that is the Debian source package, with all their further patches and their kernel config).
with the parameters at their defaults:
# sysctl vm.dirty_bytes vm.dirty_bytes = 0 # sysctl vm.dirty_background_bytes vm.dirty_background_bytes = 0
Tried copying the whole image three times (before I haven had a single copy of the whole image with at least one error, so that should be "proof" enough that it fixes the isse) upon the btrfs fs,... no errors this time...
Looks good :-)
I assume you'll take care to get that patch into stable kernels? Is this patch alone enough to recommend the Debian maintainers to include it into their 4.9 long term stable kernels?
And would you recommend this as an "urgent" fix?
Cheers,
Chris.
Quoted 9 lines · Christoph Anton Mitterer
Quoted 2 lines · Omar Sandoval
If that doesn't work, could you please also try
https://patchwork.kernel.org/patch/9829593/?Okay, tried the patch now, applied upon:
Linux 4.12.0-trunk-amd64 #1 SMP Debian 4.12.2-1~exp1 (2017-07-18)x86_64GNU/Linux (that is the Debian source package, with all their further patches and their kernel config).
Quoted 5 lines
with the parameters at their defaults:
# sysctl vm.dirty_bytes vm.dirty_bytes = 0 # sysctl vm.dirty_background_bytes vm.dirty_background_bytes = 0
Just to be sure, did you explicitly write 0 to these? These sysctls are really confusing, see https://www.kernel.org/doc/Documentation/sysctl/vm.txt.
Basically, there are two ways to specify these, either as a ratio of system memory (vm.dirty_ratio and vm.dirty_background_ratio) or a static number of bytes (vm.dirty_bytes and vm.dirty_background_bytes). If you set one, the other appears as 0, and the kernel sets the ratios by default. But if you explicitly set them to 0, the kernel is going to flush stuff extremely aggressively.
Quoted 6 lines
Tried copying the whole image three times (before I haven had a single copy of the whole image with at least one error, so that should be "proof" enough that it fixes the isse) upon the btrfs fs,... no errors this time...
Looks good :-)
Awesome, glad to hear it! I hadn't been able to reproduce the issue outside of Facebook. Can I add your tested-by?
Quoted 3 lines
I assume you'll take care to get that patch into stable kernels? Is this patch alone enough to recommend the Debian maintainers to include it into their 4.9 long term stable kernels?
I'll mark it for stable, assuming Debian tracks the upstream LTS releases it should get in.
Quoted 1 line
And would you recommend this as an "urgent" fix?
This bug has been around since 4.8, so it's not _that_ urgent, but it sucks.
Thanks!
Quoted 2 lines
Cheers,
Chris.
Quoted 1 line · Omar Sandoval
Just to be sure, did you explicitly write 0 to these?
Nope... that seemed to have been the default value, i.e. I used sysctl(8) in read (and not set) mode here.
Quoted 11 lines
These sysctls are
really confusing, see https://www.kernel.org/doc/Documentation/sysctl
/vm.txt.
Basically, there are two ways to specify these, either as a ratio of system memory (vm.dirty_ratio and vm.dirty_background_ratio) or a static
number of bytes (vm.dirty_bytes and vm.dirty_background_bytes). If you
set one, the other appears as 0, and the kernel sets the ratios by default. But if you explicitly set them to 0, the kernel is going to flush stuff extremely aggressively.
I see,... not sure why both are 0 here... at least I didn't change it myself - must be something from the distro?
Quoted 2 lines
Awesome, glad to hear it! I hadn't been able to reproduce the issue outside of Facebook. Can I add your tested-by?
Sure, but better use my other mail address for it, if you don't mind:
Christoph Anton Mitterer <[email protected]>
Quoted 6 lines
Quoted 3 lines
I assume you'll take care to get that patch into stable kernels? Is this patch alone enough to recommend the Debian maintainers to include it into their 4.9 long term stable kernels?
I'll mark it for stable, assuming Debian tracks the upstream LTS releases it should get in.
Okay :-)
Nevertheless I'll open a bug at their BTS, just to be safe.
Thanks :)
Chris.
Quoted 4 lines · Christoph Anton Mitterer
Quoted 1 line · Omar Sandoval
Just to be sure, did you explicitly write 0 to these?
Nope... that seemed to have been the default value, i.e. I used sysctl(8) in read (and not set) mode here.
Okay, all good then.
Quoted 11 lines
Quoted 11 lines
These sysctls are
really confusing, see https://www.kernel.org/doc/Documentation/sysctl
/vm.txt.
Basically, there are two ways to specify these, either as a ratio of system memory (vm.dirty_ratio and vm.dirty_background_ratio) or a static
number of bytes (vm.dirty_bytes and vm.dirty_background_bytes). If you
set one, the other appears as 0, and the kernel sets the ratios by default. But if you explicitly set them to 0, the kernel is going to flush stuff extremely aggressively.
Quoted 2 lines
I see,... not sure why both are 0 here... at least I didn't change it myself - must be something from the distro?
That's normal, the default is to have the ratio set instead:
$ sysctl vm.dirty_{,background_}{bytes,ratio}
vm.dirty_bytes = 0
vm.dirty_ratio = 20
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10Quoted 4 lines
Quoted 2 lines
Awesome, glad to hear it! I hadn't been able to reproduce the issue outside of Facebook. Can I add your tested-by?
Sure, but better use my other mail address for it, if you don't mind:
Christoph Anton Mitterer <[email protected]>
No problem. I'll resend the patch with that shortly.
Quoted 14 lines
Quoted 6 lines
Quoted 3 lines
I assume you'll take care to get that patch into stable kernels? Is this patch alone enough to recommend the Debian maintainers to include it into their 4.9 long term stable kernels?
I'll mark it for stable, assuming Debian tracks the upstream LTS releases it should get in.
Okay :-)
Nevertheless I'll open a bug at their BTS, just to be safe.
Thanks :)
Chris.
Quoted 9 lines · Christoph Anton Mitterer
On Thu, 2017-07-20 at 14:48 -0700, Omar Sandoval wrote:
[...]Quoted 6 lines
Quoted 3 lines
I assume you'll take care to get that patch into stable kernels? Is this patch alone enough to recommend the Debian maintainers to include it into their 4.9 long term stable kernels?
I'll mark it for stable, assuming Debian tracks the upstream LTS releases it should get in.
+1 \o/
Quoted 3 lines
Okay :-)
Nevertheless I'll open a bug at their BTS, just to be safe.
The Debian kernel teams tries to minimize the amount of patches on top of the original kernel code. Every patch added comes with a maintenance burden.
So, the answer will probably be: Please get it in 4.9 itself.
Signature · 1 line
Hans van Kranenburg