btrfs blocks root from writing even when there is plenty of statfs->f_bfree
I've been doing some testing with btrfs exported via nfsd and hit this bug. root is unable to write to the filesystem, even though statfs reports that there should be 400k+ blocks available.
The reproducer is pretty simple. First I fill up the filesystem as an unprivileged user (let this run until I hit ENOSPC):
[vagrant@kdevops-nfs-default btrfs]$ dd if=/dev/urandom of=/media/btrfs/bigfile bs=1M
...and then try to write to a file on the fs as root:
[root@kdevops-nfs-default btrfs]# strace -f -e statfs df .
statfs(".", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096, f_blocks=26214400, f_bfree=452204, f_bavail=0, f_files=0, f_ffree=0, f_fsid={val=[0xa9d43863, 0xb08b34cc]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
Filesystem 1K-blocks Used Available Use% Mounted on/dev/nvme2n1 104857600 103048784 0 100% /media/btrfs
+++ exited with 0 +++
[root@kdevops-nfs-default btrfs]# xfs_io -f -c "pwrite -b 4096 4096 4096" ./file1
pwrite: No space left on device
This works on ext4 and xfs. The kernel is Linus' master branch as of a couple of days ago:
6.5.0-rc7-g081b0d4bef5d
Let me know if you need more info.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 7 lines · Jeff Layton
I've been doing some testing with btrfs exported via nfsd and hit this bug. root is unable to write to the filesystem, even though statfs reports that there should be 400k+ blocks available.
The reproducer is pretty simple. First I fill up the filesystem as an unprivileged user (let this run until I hit ENOSPC):
Quoted 6 lines
[vagrant@kdevops-nfs-default btrfs]$ dd if=/dev/urandom of=/media/btrfs/bigfile bs=1M
...and then try to write to a file on the fs as root:
[root@kdevops-nfs-default btrfs]# strace -f -e statfs df .
statfs(".", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096, f_blocks=26214400, f_bfree=452204, f_bavail=0, f_files=0, f_ffree=0, f_fsid={val=[0xa9d43863,0xb08b34cc]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
Note that, unlike XFS/EXT4, btrfs goes strict limits between data and metadata.
Metadata space can only be used to store tree blocks, never data. And both metadata and data space are allocated dynamically, thus we have have problems when the workload is unbalanced.
(This is a little like the inodes limits on XFS/EXT4, but more dynamic)
In your particular case, the data space is all used up, but metadata space still has some free space.
And f_btree returned by btrfs is including the metadata space, as although that's inaccurate, it's the best what we can do to report.
Quoted 6 lines
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/nvme2n1104857600 103048784 0 100% /media/btrfs
+++ exited with 0 +++
[root@kdevops-nfs-default btrfs]#xfs_io-f -c "pwrite -b 4096 4096 4096" ./file1
pwrite: No space left on device
Quoted 6 lines
This works on ext4 and xfs. The kernel is Linus' master branch as of a couple of days ago:
6.5.0-rc7-g081b0d4bef5dLet me know if you need more info.
That's why for btrfs we recommend to go "btrfs fi usage" to get a more comprehensive understand on the space usage.
In your case, Data usage should be 100%, without any spare ones.
Thanks,
Qu
Quoted 46 lines · Qu Wenruo
Quoted 10 lines · Jeff Layton
I've been doing some testing with btrfs exported via nfsd and hit this bug. root is unable to write to the filesystem, even though statfs reports that there should be 400k+ blocks available.
The reproducer is pretty simple. First I fill up the filesystem as an unprivileged user (let this run until I hit ENOSPC):
[vagrant@kdevops-nfs-default btrfs]$ dd if=/dev/urandom of=/media/btrfs/bigfile bs=1M
Quoted 7 lines
...and then try to write to a file on the fs as root:
[root@kdevops-nfs-default btrfs]# strace -f -e statfs df . statfs(".", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096, f_blocks=26214400, f_bfree=452204, f_bavail=0, f_files=0, f_ffree=0, f_fsid={val=[0xa9d43863,
0xb08b34cc]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0Note that, unlike XFS/EXT4, btrfs goes strict limits between data and metadata.
Metadata space can only be used to store tree blocks, never data. And both metadata and data space are allocated dynamically, thus we have have problems when the workload is unbalanced.
(This is a little like the inodes limits on XFS/EXT4, but more dynamic)In your particular case, the data space is all used up, but metadata space still has some free space.
And
f_btreereturned by btrfs is including the metadata space, as although that's inaccurate, it's the best what we can do to report.Quoted 10 lines
Filesystem 1K-blocks Used Available Use% Mounted on /dev/nvme2n1 104857600 103048784 0 100% /media/btrfs +++ exited with 0 +++
[root@kdevops-nfs-default btrfs]#xfs_io-f -c "pwrite -b 4096 4096 4096" ./file1
pwrite: No space left on deviceThis works on ext4 and xfs. The kernel is Linus' master branch as of a couple of days ago:
Quoted 8 lines
Quoted 3 lines
6.5.0-rc7-g081b0d4bef5d
Let me know if you need more info.
That's why for btrfs we recommend to go "btrfs fi usage" to get a more comprehensive understand on the space usage.
In your case, Data usage should be 100%, without any spare ones.
Another thing is, btrfs doesn't have any extra data space reserved for root user, instead btrfs has extra space reserved for metadata, as metadata COW will cost extra space, even doing things like unlinking a file.
Thus this would be a behavior difference here.
Thanks,
Qu
Quoted 3 lines
Thanks,
Qu
Quoted 55 lines · Qu Wenruo
Quoted 53 lines · Qu Wenruo
Quoted 10 lines · Jeff Layton
I've been doing some testing with btrfs exported via nfsd and hit this bug. root is unable to write to the filesystem, even though statfs reports that there should be 400k+ blocks available.
The reproducer is pretty simple. First I fill up the filesystem as an unprivileged user (let this run until I hit ENOSPC):
[vagrant@kdevops-nfs-default btrfs]$ dd if=/dev/urandom of=/media/btrfs/bigfile bs=1M
Quoted 7 lines
...and then try to write to a file on the fs as root:
[root@kdevops-nfs-default btrfs]# strace -f -e statfs df . statfs(".", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096, f_blocks=26214400, f_bfree=452204, f_bavail=0, f_files=0, f_ffree=0, f_fsid={val=[0xa9d43863,
0xb08b34cc]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0Note that, unlike XFS/EXT4, btrfs goes strict limits between data and metadata.
Metadata space can only be used to store tree blocks, never data. And both metadata and data space are allocated dynamically, thus we have have problems when the workload is unbalanced.
(This is a little like the inodes limits on XFS/EXT4, but more dynamic)In your particular case, the data space is all used up, but metadata space still has some free space.
And
f_btreereturned by btrfs is including the metadata space, as although that's inaccurate, it's the best what we can do to report.Quoted 7 lines
Filesystem 1K-blocks Used Available Use% Mounted on /dev/nvme2n1 104857600 103048784 0 100% /media/btrfs +++ exited with 0 +++
[root@kdevops-nfs-default btrfs]#xfs_io-f -c "pwrite -b 4096 4096 4096" ./file1
pwrite: No space left on deviceQuoted 6 lines
This works on ext4 and xfs. The kernel is Linus' master branch as of a couple of days ago:
6.5.0-rc7-g081b0d4bef5d
Let me know if you need more info.
That's why for btrfs we recommend to go "btrfs fi usage" to get a more comprehensive understand on the space usage.
Quoted 8 lines
Quoted 1 line
In your case, Data usage should be 100%, without any spare ones.
Another thing is, btrfs doesn't have any extra data space reserved for root user, instead btrfs has extra space reserved for metadata, as metadata COW will cost extra space, even doing things like unlinking a file.
Thus this would be a behavior difference here.
Got it, thanks for the explanation.
FWIW, I noticed this because generic/186 and generic/187 were failing on NFS exported btrfs, but was passing with XFS, they get _notrun:
generic/186 [not run] Can't fragment free space on btrfs.
generic/187 [not run] Can't fragment free space on btrfs.
I'll probably just skip this test on NFS since it only reliably passes on certain exported filesystems.
Thanks again!
Signature · 1 line
Jeff Layton <[email protected]>