Weird behaviour with project quotas
Hello,
As far as I understand, and from my tests, on folders on which a project quota is applied, either the available quota or the actually avialable space should be reported when using "df". However on a running system (Debian 12, kernel 6.1 Debian) I have incoherent results:
The volume /mnt/raid is 100 TB and has 500GB free.
There are several folders like /mnt/raid/project1, /mnt/raid/project2 etc with various quotas (20TB, 30TB, etc).
Though there are only 500 GB free on the whole filesystem, "df /mnt/raid/project1" etc reports several TB available, which is the remaining quota for the folder. Of course the users are led to believe there is more space available than what's actually the case (in fact that's why the volume is down to so little space) which causes all sort of problems.
What's strange is that I can't reproduce this behaviour on another system. Every time I try to reproduce it, available space reported by df on a project quota folder matches actual available space.
Any idea on what could be going wrong ?
Signature · 6 lines
------------------------------------------------------------------------ Emmanuel Florac | Direction technique ------------------------------------------------------------------------ https://intellique.com +33 6 16 30 15 95 ------------------------------------------------------------------------
Quoted 6 lines · Emmanuel Florac
Hello,
As far as I understand, and from my tests, on folders on which a project quota is applied, either the available quota or the actually avialable space should be reported when using "df".
Only if you are using project quotas as directories quotas. i.e. the directory you are querying with df needs to have the XFS_DIFLAG_PROJINHERIT flag set on it for df to behave this way.
Quoted 2 lines
However on a
running system (Debian 12, kernel 6.1 Debian) I have incoherent results:
32 bit or 64 bit architecture?
Quoted 4 lines
The volume
/mnt/raidis 100 TB and has 500GB free.There are several folders like
/mnt/raid/project1,/mnt/raid/project2etc with various quotas (20TB, 30TB, etc).
Output of df and a project quota report showing usage and limits would be useful here.
Then, for each of the top level project directories you are querying with df, also run xfs_io -rxc "stat" <dir> and post the output. This will tell us if the project quota is set up correctly for df to report quota limits for them.
It would also be useful to know if the actual quota usage is correct
- having the output of du -s /mnt/raid/project1 to count the blocks and find /mnt/raid/project1 -print |wc -l to count the files in quota controlled directories. That'll give us some idea if there's a quota accounting issue.
Quoted 1 line
Any idea on what could be going wrong ?
No. There's nothing obvious that stands out in the current TOT code...
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Le Fri, 29 Nov 2024 08:07:09 +1100
Dave Chinner <[email protected]> écrivait:
Quoted 7 lines
Quoted 3 lines
As far as I understand, and from my tests, on folders on which a project quota is applied, either the available quota or the actually avialable space should be reported when using "df".
Only if you are using project quotas as directories quotas. i.e. the directory you are querying with df needs to have the
XFS_DIFLAG_PROJINHERITflag set on it for df to behave this way.
Interesting, and how is this set ? I basically set up quotas using something like
xfs_quota -x -c 'project -s -p /mnt/raid/project1 10' /mnt/raid
xfs_quota -x -c "limit -p bhard=30000g 10" /mnt/raid
Quoted 5 lines
Quoted 3 lines
However on a
running system (Debian 12, kernel 6.1 Debian) I have incoherent results:32 bit or 64 bit architecture?
AMD64, the most common one.
Quoted 13 lines
Quoted 4 lines
The volume
/mnt/raidis 100 TB and has 500GB free.There are several folders like
/mnt/raid/project1,/mnt/raid/project2etc with various quotas (20TB, 30TB, etc).Output of df and a project quota report showing usage and limits would be useful here.
Then, for each of the top level project directories you are querying with df, also run
xfs_io -rxc "stat" <dir>and post the output. This will tell us if the project quota is set up correctly for df to report quota limits for them.
Quoted 6 lines
It would also be useful to know if the actual quota usage is correct
- having the output ofdu -s /mnt/raid/project1to count the blocks andfind /mnt/raid/project1 -print |wc -lto count the files in quota controlled directories. That'll give us some idea if there's a quota accounting issue.
OK, I'll run these as soon as I have a connection to the system. There seemed to be no error with the reported used space though, only available remaining space in quota'ed directories; however df reports actually available space for directories without quota set.
Signature · 5 lines
------------------------------------------------------------------------ Emmanuel Florac | Direction technique ------------------------------------------------------------------------ https://intellique.com ------------------------------------------------------------------------
Quoted 11 lines · Emmanuel Florac
Le Fri, 29 Nov 2024 08:07:09 +1100
Dave Chinner <[email protected]> écrivait:Quoted 7 lines
Quoted 3 lines
As far as I understand, and from my tests, on folders on which a project quota is applied, either the available quota or the actually avialable space should be reported when using "df".
Only if you are using project quotas as directories quotas. i.e. the directory you are querying with df needs to have the
XFS_DIFLAG_PROJINHERITflag set on it for df to behave this way.
Quoted 6 lines
Interesting, and how is this set ? I basically set up quotas using something like
xfs_quota-x -c 'project -s -p/mnt/raid/project110'/mnt/raid
xfs_quota-x -c "limit -p bhard=30000g 10"/mnt/raid
That should set it up appropriately, hence the need to check if it has actually been set up correctly on disk.
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
However on a
running system (Debian 12, kernel 6.1 Debian) I have incoherent results:
Quoted 22 lines
Quoted 1 line
32 bit or 64 bit architecture?
AMD64, the most common one.
Quoted 18 lines
Quoted 4 lines
The volume
/mnt/raidis 100 TB and has 500GB free.There are several folders like
/mnt/raid/project1,/mnt/raid/project2etc with various quotas (20TB, 30TB, etc).Output of df and a project quota report showing usage and limits would be useful here.
Then, for each of the top level project directories you are querying with df, also run
xfs_io -rxc "stat" <dir>and post the output. This will tell us if the project quota is set up correctly for df to report quota limits for them.It would also be useful to know if the actual quota usage is correct
- having the output ofdu -s /mnt/raid/project1to count the blocks andfind /mnt/raid/project1 -print |wc -lto count the files in quota controlled directories. That'll give us some idea if there's a quota accounting issue.
iAnother thought occurred to me - can you also check that /etc/projid and /etc/projects is similar on all machines, and post the contents of them from the bad machine?
Quoted 4 lines
OK, I'll run these as soon as I have a connection to the system. There seemed to be no error with the reported used space though, only available remaining space in quota'ed directories; however df reports actually available space for directories without quota set.
Ok - it would be helpful to post the commands you run and the output so that we both know exactly what behaviour you are seeing.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Le Sat, 30 Nov 2024 09:14:20 +1100
Dave Chinner <[email protected]> écrivait:
Quoted 5 lines
Quoted 1 line
xfs_quota-x -c "limit -p bhard=30000g 10"/mnt/raidThat should set it up appropriately, hence the need to check if it has actually been set up correctly on disk.
Unfortunately in the meantime the users did some cleanup, therefore the displayed information is coherent again (as there is more free space on the filesystem as a whole as any remaining allocated quota).
xfs_quota -x -c "report -p"
Project quota on /mnt/raid (/dev/mapper/vg0-raid)
Blocks
Project ID Used Soft Hard Warn/Grace
----------- ------------------------------------------------------
<snip>
#40 10795758244 0 16106127360 00 [--------]
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Output of df and a project quota report showing usage and limits would be useful here.
looking at the corresponding folder :
/dev/mapper/vg0-raid 15T 11T 5,0T 68% /mnt/raid/pad
du -s /mnt/raid/pad
10795758244 /mnt/raid/pad
# find /mnt/raid/pad -print | wc -l 39086
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Then, for each of the top level project directories you are querying with df, also run
xfs_io -rxc "stat" <dir>and post the output. This will tell us if the project quota is set up correctly for df to report quota limits for them.
Starting with "pad" :
# xfs_io -rxc "stat" pad fd.path = "." fd.flags = non-sync,non-direct,read-only stat.ino = 6442662464 stat.type = directory stat.size = 4096 stat.blocks = 16 fsxattr.xflags = 0x200 \[--------P--------\] fsxattr.projid = 40 fsxattr.extsize = 0 fsxattr.cowextsize = 0 fsxattr.nextents = 2 fsxattr.naextents = 0 dioattr.mem = 0x200 dioattr.miniosz = 512 dioattr.maxiosz = 2147483136
# xfs_io -rxc "stat" rush fd.path = "." fd.flags = non-sync,non-direct,read-only stat.ino = 142 stat.type = directory stat.size = 283 stat.blocks = 0 fsxattr.xflags = 0x200 \[--------P--------\] fsxattr.projid = 10 fsxattr.extsize = 0 fsxattr.cowextsize = 0 fsxattr.nextents = 0 fsxattr.naextents = 0 dioattr.mem = 0x200 dioattr.miniosz = 512 dioattr.maxiosz = 2147483136
# xfs_io -rxc "stat" labo fd.path = "." fd.flags = non-sync,non-direct,read-only stat.ino = 2147695168 stat.type = directory stat.size = 310 stat.blocks = 0 fsxattr.xflags = 0x200 \[--------P--------\] fsxattr.projid = 20 fsxattr.extsize = 0 fsxattr.cowextsize = 0 fsxattr.nextents = 0 fsxattr.naextents = 0 dioattr.mem = 0x200 dioattr.miniosz = 512 dioattr.maxiosz = 2147483136
# xfs_io -rxc "stat" prods fd.path = "." fd.flags = non-sync,non-direct,read-only stat.ino = 4295178816 stat.type = directory stat.size = 319 stat.blocks = 0 fsxattr.xflags = 0x200 \[--------P--------\] fsxattr.projid = 30 fsxattr.extsize = 0 fsxattr.cowextsize = 0 fsxattr.nextents = 0 fsxattr.naextents = 0 dioattr.mem = 0x200 dioattr.miniosz = 512 dioattr.maxiosz = 2147483136
Quoted 11 lines
Quoted 6 lines
Quoted 6 lines
It would also be useful to know if the actual quota usage is correct
- having the output ofdu -s /mnt/raid/project1to count the blocks andfind /mnt/raid/project1 -print |wc -lto count the files in quota controlled directories. That'll give us some idea if there's a quota accounting issue.iAnother thought occurred to me - can you also check that
/etc/projidand/etc/projectsis similar on all machines, and post the contents of them from the bad machine?
Hum, actually they didn't set up neither projid nor projects. Of course I did create these during my tests, but could this be the culprit ?
Signature · 6 lines
------------------------------------------------------------------------ Emmanuel Florac | Direction technique ------------------------------------------------------------------------ https://intellique.com +33 6 16 30 15 95 ------------------------------------------------------------------------
Quoted 121 lines · Emmanuel Florac
Le Sat, 30 Nov 2024 09:14:20 +1100
Dave Chinner <[email protected]> écrivait:Quoted 5 lines
Quoted 1 line
xfs_quota-x -c "limit -p bhard=30000g 10"/mnt/raidThat should set it up appropriately, hence the need to check if it has actually been set up correctly on disk.
Unfortunately in the meantime the users did some cleanup, therefore the displayed information is coherent again (as there is more free space on the filesystem as a whole as any remaining allocated quota).
xfs_quota-x -c "report -p"
Project quota on/mnt/raid(/dev/mapper/vg0-raid)
Blocks
Project ID Used Soft Hard Warn/Grace----------- ------------------------------------------------------
<snip>
#40 10795758244 0 16106127360 00 [--------]Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Output of df and a project quota report showing usage and limits would be useful here.
looking at the corresponding folder :
/dev/mapper/vg0-raid15T 11T 5,0T 68%/mnt/raid/paddu -s
/mnt/raid/pad
10795758244/mnt/raid/pad# find /mnt/raid/pad -print | wc -l 39086Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Then, for each of the top level project directories you are querying with df, also run
xfs_io -rxc "stat" <dir>and post the output. This will tell us if the project quota is set up correctly for df to report quota limits for them.Starting with "pad" :
# xfs_io -rxc "stat" pad fd.path = "." fd.flags = non-sync,non-direct,read-only stat.ino = 6442662464 stat.type = directory stat.size = 4096 stat.blocks = 16 fsxattr.xflags = 0x200 \[--------P--------\] fsxattr.projid = 40 fsxattr.extsize = 0 fsxattr.cowextsize = 0 fsxattr.nextents = 2 fsxattr.naextents = 0 dioattr.mem = 0x200 dioattr.miniosz = 512 dioattr.maxiosz = 2147483136# xfs_io -rxc "stat" rush fd.path = "." fd.flags = non-sync,non-direct,read-only stat.ino = 142 stat.type = directory stat.size = 283 stat.blocks = 0 fsxattr.xflags = 0x200 \[--------P--------\] fsxattr.projid = 10 fsxattr.extsize = 0 fsxattr.cowextsize = 0 fsxattr.nextents = 0 fsxattr.naextents = 0 dioattr.mem = 0x200 dioattr.miniosz = 512 dioattr.maxiosz = 2147483136# xfs_io -rxc "stat" labo fd.path = "." fd.flags = non-sync,non-direct,read-only stat.ino = 2147695168 stat.type = directory stat.size = 310 stat.blocks = 0 fsxattr.xflags = 0x200 \[--------P--------\] fsxattr.projid = 20 fsxattr.extsize = 0 fsxattr.cowextsize = 0 fsxattr.nextents = 0 fsxattr.naextents = 0 dioattr.mem = 0x200 dioattr.miniosz = 512 dioattr.maxiosz = 2147483136# xfs_io -rxc "stat" prods fd.path = "." fd.flags = non-sync,non-direct,read-only stat.ino = 4295178816 stat.type = directory stat.size = 319 stat.blocks = 0 fsxattr.xflags = 0x200 \[--------P--------\] fsxattr.projid = 30 fsxattr.extsize = 0 fsxattr.cowextsize = 0 fsxattr.nextents = 0 fsxattr.naextents = 0 dioattr.mem = 0x200 dioattr.miniosz = 512 dioattr.maxiosz = 2147483136Quoted 7 lines
Quoted 6 lines
Quoted 6 lines
It would also be useful to know if the actual quota usage is correct
- having the output ofdu -s /mnt/raid/project1to count the blocks andfind /mnt/raid/project1 -print |wc -lto count the files in quota controlled directories. That'll give us some idea if there's a quota accounting issue.
Quoted 7 lines
Quoted 4 lines
iAnother thought occurred to me - can you also check that
/etc/projidand/etc/projectsis similar on all machines, and post the contents of them from the bad machine?Hum, actually they didn't set up neither projid nor projects. Of course I did create these during my tests, but could this be the culprit ?
Does this recreate the symptoms?
# mkfs.xfs -f /dev/sda
meta-data=/dev/sda isize=512 agcount=4, agsize=1298176 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=1
= reflink=1 bigtime=1 inobtcount=1 nrext64=1
= exchange=0 metadir=0
data = bsize=4096 blocks=5192704, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1, parent=0
log =internal log bsize=4096 blocks=16384, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
= rgcount=0 rgsize=0 extents
Discarding blocks...Done.
# mount /dev/sda /mnt -o prjquota
# xfs_quota -x -c 'limit -p bhard=2G 55' /mnt
# mkdir /mnt/dir
# xfs_io -c 'chproj 55' -c 'chattr +P' -c 'stat -vvvv' /mnt/dir
fd.path = "/mnt/dir"
fd.flags = non-sync,non-direct,read-write
stat.ino = 132
stat.type = directory
stat.size = 6
stat.blocks = 0
stat.atime = Thu Dec 12 12:07:53 2024
stat.mtime = Thu Dec 12 12:07:53 2024
stat.ctime = Thu Dec 12 12:08:12 2024
fsxattr.xflags = 0x200 [proj-inherit]
fsxattr.projid = 55
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 0
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136
# df /mnt /mnt/dir
Filesystem Size Used Avail Use% Mounted on
/dev/sda 20G 420M 20G 3% /mnt
/dev/sda 2.0G 0 2.0G 0% /mnt
# fallocate -l 19g /mnt/a
# df /mnt /mnt/dir
Filesystem Size Used Avail Use% Mounted on
/dev/sda 20G 20G 345M 99% /mnt
/dev/sda 2.0G 0 2.0G 0% /mntClearly, df should be reporting 345M available for both cases, since we haven't actually used any of project 55's blocks.
# xfs_io -f -c 'pwrite -S 0x59 0 1m' -c fsync -c 'stat -vvvv' /mnt/dir/fork wrote 1048576/1048576 bytes at offset 0 1 MiB, 256 ops; 0.0008 sec (1.121 GiB/sec and 293915.0402 ops/sec) fd.path = "/mnt/dir/fork" fd.flags = non-sync,non-direct,read-write stat.ino = 134 stat.type = regular file stat.size = 1048576 stat.blocks = 2048 stat.atime = Thu Dec 12 12:11:06 2024 stat.mtime = Thu Dec 12 12:11:06 2024 stat.ctime = Thu Dec 12 12:11:06 2024 fsxattr.xflags = 0x0 [] fsxattr.projid = 55 fsxattr.extsize = 0 fsxattr.cowextsize = 0 fsxattr.nextents = 1 fsxattr.naextents = 0 dioattr.mem = 0x200 dioattr.miniosz = 512 dioattr.maxiosz = 2147483136 # df /mnt /mnt/dir Filesystem Size Used Avail Use% Mounted on /dev/sda 20G 20G 344M 99% /mnt /dev/sda 2.0G 1.0M 2.0G 1% /mnt
I think this behavior comes from xfs_fill_statvfs_from_dquot, which does this:
limit = blkres->softlimit ?
blkres->softlimit :
blkres->hardlimit;
if (limit && statp->f_blocks > limit) {
statp->f_blocks = limit;
statp->f_bfree = statp->f_bavail =
(statp->f_blocks > blkres->reserved) ?
(statp->f_blocks - blkres->reserved) : 0;
}I think the f_bfree/f_bavail assignment is wrong because it doesn't handle the case where f_bfree was less than (limit - reserved).
if (limit) {
uint64_t remaining = 0;
if (statp->f_blocks > limit)
statp->f_blocks = limit;
if (limit > blkres->reserved)
remaining = limit - blkres->reserved;
statp->f_bfree = min(statp->f_bfree, remaining);
statp->f_bavail = min(statp->f_bavail, remaining);
}This fixes the df output a bit:
# df /mnt /mnt/dir Filesystem Size Used Avail Use% Mounted on /dev/sda 20G 20G 344M 99% /mnt /dev/sda 2.0G 1.7G 344M 84% /mnt
Though the "used" column is nonsense now. But I guess that's why statfs only defines total blocks and free/available blocks.
--D
Quoted 8 lines
Signature · 6 lines
------------------------------------------------------------------------ Emmanuel Florac | Direction technique ------------------------------------------------------------------------ https://intellique.com +33 6 16 30 15 95 ------------------------------------------------------------------------
Le Thu, 12 Dec 2024 12:25:47 -0800
"Darrick J. Wong" <[email protected]> écrivait:
Quoted 2 lines
Does this recreate the symptoms?
<snip>
Quoted 66 lines
# df /mnt /mnt/dir Filesystem Size Used Avail Use% Mounted on /dev/sda 20G 420M 20G 3% /mnt /dev/sda 2.0G 0 2.0G 0% /mnt # fallocate -l 19g /mnt/a # df /mnt /mnt/dir Filesystem Size Used Avail Use% Mounted on /dev/sda 20G 20G 345M 99% /mnt /dev/sda 2.0G 0 2.0G 0% /mntClearly, df should be reporting 345M available for both cases, since we haven't actually used any of project 55's blocks.
# xfs_io -f -c 'pwrite -S 0x59 0 1m' -c fsync -c 'stat -vvvv' /mnt/dir/fork wrote 1048576/1048576 bytes at offset 0 1 MiB, 256 ops; 0.0008 sec (1.121 GiB/sec and 293915.0402 ops/sec) fd.path = "/mnt/dir/fork" fd.flags = non-sync,non-direct,read-write stat.ino = 134 stat.type = regular file stat.size = 1048576 stat.blocks = 2048 stat.atime = Thu Dec 12 12:11:06 2024 stat.mtime = Thu Dec 12 12:11:06 2024 stat.ctime = Thu Dec 12 12:11:06 2024 fsxattr.xflags = 0x0 [] fsxattr.projid = 55 fsxattr.extsize = 0 fsxattr.cowextsize = 0 fsxattr.nextents = 1 fsxattr.naextents = 0 dioattr.mem = 0x200 dioattr.miniosz = 512 dioattr.maxiosz = 2147483136 # df /mnt /mnt/dir Filesystem Size Used Avail Use% Mounted on /dev/sda 20G 20G 344M 99% /mnt /dev/sda 2.0G 1.0M 2.0G 1% /mntI think this behavior comes from
xfs_fill_statvfs_from_dquot, which does this:limit = blkres->softlimit ? blkres->softlimit : blkres->hardlimit; if (limit && statp->f_blocks > limit) { statp->f_blocks = limit; statp->f_bfree = statp->f_bavail = (statp->f_blocks > blkres->reserved) ? (statp->f_blocks - blkres->reserved) : 0; }I think the f_bfree/f_bavail assignment is wrong because it doesn't handle the case where
f_bfreewas less than (limit - reserved).if (limit) { uint64_t remaining = 0; if (statp->f_blocks > limit) statp->f_blocks = limit; if (limit > blkres->reserved) remaining = limit - blkres->reserved; statp->f_bfree = min(statp->f_bfree, remaining); statp->f_bavail = min(statp->f_bavail, remaining); }
Quoted 8 lines
This fixes the df output a bit:
# df /mnt /mnt/dir Filesystem Size Used Avail Use% Mounted on /dev/sda 20G 20G 344M 99% /mnt /dev/sda 2.0G 1.7G 344M 84% /mntThough the "used" column is nonsense now. But I guess that's why statfs only defines total blocks and free/available blocks.
Yep, that looks exactly like the problem we've met. Does the fact that not all folders have project quota change something in that case ?
Signature · 6 lines
------------------------------------------------------------------------ Emmanuel Florac | Direction technique ------------------------------------------------------------------------ https://intellique.com +33 6 16 30 15 95 ------------------------------------------------------------------------
Quoted 79 lines · Emmanuel Florac
Le Thu, 12 Dec 2024 12:25:47 -0800
"Darrick J. Wong" <[email protected]> écrivait:Quoted 2 lines
Does this recreate the symptoms?
<snip>
Quoted 72 lines
# df /mnt /mnt/dir Filesystem Size Used Avail Use% Mounted on /dev/sda 20G 420M 20G 3% /mnt /dev/sda 2.0G 0 2.0G 0% /mnt # fallocate -l 19g /mnt/a # df /mnt /mnt/dir Filesystem Size Used Avail Use% Mounted on /dev/sda 20G 20G 345M 99% /mnt /dev/sda 2.0G 0 2.0G 0% /mntClearly, df should be reporting 345M available for both cases, since we haven't actually used any of project 55's blocks.
# xfs_io -f -c 'pwrite -S 0x59 0 1m' -c fsync -c 'stat -vvvv' /mnt/dir/fork wrote 1048576/1048576 bytes at offset 0 1 MiB, 256 ops; 0.0008 sec (1.121 GiB/sec and 293915.0402 ops/sec) fd.path = "/mnt/dir/fork" fd.flags = non-sync,non-direct,read-write stat.ino = 134 stat.type = regular file stat.size = 1048576 stat.blocks = 2048 stat.atime = Thu Dec 12 12:11:06 2024 stat.mtime = Thu Dec 12 12:11:06 2024 stat.ctime = Thu Dec 12 12:11:06 2024 fsxattr.xflags = 0x0 [] fsxattr.projid = 55 fsxattr.extsize = 0 fsxattr.cowextsize = 0 fsxattr.nextents = 1 fsxattr.naextents = 0 dioattr.mem = 0x200 dioattr.miniosz = 512 dioattr.maxiosz = 2147483136 # df /mnt /mnt/dir Filesystem Size Used Avail Use% Mounted on /dev/sda 20G 20G 344M 99% /mnt /dev/sda 2.0G 1.0M 2.0G 1% /mntI think this behavior comes from
xfs_fill_statvfs_from_dquot, which does this:limit = blkres->softlimit ? blkres->softlimit : blkres->hardlimit; if (limit && statp->f_blocks > limit) { statp->f_blocks = limit; statp->f_bfree = statp->f_bavail = (statp->f_blocks > blkres->reserved) ? (statp->f_blocks - blkres->reserved) : 0; }I think the f_bfree/f_bavail assignment is wrong because it doesn't handle the case where
f_bfreewas less than (limit - reserved).if (limit) { uint64_t remaining = 0; if (statp->f_blocks > limit) statp->f_blocks = limit; if (limit > blkres->reserved) remaining = limit - blkres->reserved; statp->f_bfree = min(statp->f_bfree, remaining); statp->f_bavail = min(statp->f_bavail, remaining); }This fixes the df output a bit:
# df /mnt /mnt/dir Filesystem Size Used Avail Use% Mounted on /dev/sda 20G 20G 344M 99% /mnt /dev/sda 2.0G 1.7G 344M 84% /mnt
Quoted 5 lines
Quoted 2 lines
Though the "used" column is nonsense now. But I guess that's why statfs only defines total blocks and free/available blocks.
Yep, that looks exactly like the problem we've met. Does the fact that not all folders have project quota change something in that case ?
No, I don't think that changes anything. If you can build your own kernel, can you try this out?
--D
xfs: don't over-report free space or inodes in statvfs
Emmanual Florac reports a strange occurrence when project quota limits are enabled, free space is lower than the remaining quota, and someone runs statvfs:
# mkfs.xfs -f /dev/sda # mount /dev/sda /mnt -o prjquota # xfs_quota -x -c 'limit -p bhard=2G 55' /mnt # mkdir /mnt/dir # xfs_io -c 'chproj 55' -c 'chattr +P' -c 'stat -vvvv' /mnt/dir # fallocate -l 19g /mnt/a # df /mnt /mnt/dir Filesystem Size Used Avail Use% Mounted on /dev/sda 20G 20G 345M 99% /mnt /dev/sda 2.0G 0 2.0G 0% /mnt
I think the bug here is that xfs_fill_statvfs_from_dquot unconditionally assigns to f_bfree without checking that the filesystem has enough free space to fill the remaining project quota. However, this is a longstanding behavior of xfs so it's unclear what to do here.
--- fs/xfs/xfs_qm_bhv.c | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-)
fs/xfs/xfs_qm_bhv.c +17 -10
@@ -32,21 +32,28 @@ xfs_fill_statvfs_from_dquot( limit = blkres->softlimit ? blkres->softlimit : blkres->hardlimit; -if (limit && statp->f_blocks > limit) { - statp->f_blocks = limit; - statp->f_bfree = statp->f_bavail = - (statp->f_blocks > blkres->reserved) ? - (statp->f_blocks - blkres->reserved) :0 ; +if (limit) { +uint64_t remaining =0 ; + +if (limit > blkres->reserved) + remaining = limit - blkres->reserved; + + statp->f_blocks =min (statp->f_blocks, limit); + statp->f_bfree =min (statp->f_bfree, remaining); + statp->f_bavail =min (statp->f_bavail, remaining); } limit = dqp->q_ino.softlimit ? dqp->q_ino.softlimit : dqp->q_ino.hardlimit; -if (limit && statp->f_files > limit) { - statp->f_files = limit; - statp->f_ffree = - (statp->f_files > dqp->q_ino.reserved) ? - (statp->f_files - dqp->q_ino.reserved) :0 ; +if (limit) { +uint64_t remaining =0 ; + +if (limit > dqp->q_ino.reserved) + remaining = limit - dqp->q_ino.reserved; + + statp->f_files =min (statp->f_files, limit); + statp->f_ffree =min (statp->f_ffree, remaining); } }
Le Fri, 13 Dec 2024 09:15:37 -0800
"Darrick J. Wong" <[email protected]> écrivait:
Quoted 6 lines
No, I don't think that changes anything. If you can build your own kernel, can you try this out?
--D
xfs: don't over-report free space or inodes in statvfs
I'll give it a try, but that looks like a patch for old weird RedHat kernel, I'm running plain vanilla generally, and much higher versions, I'll see how it applies :)
Signature · 6 lines
------------------------------------------------------------------------ Emmanuel Florac | Direction technique ------------------------------------------------------------------------ https://intellique.com +33 6 16 30 15 95 ------------------------------------------------------------------------
Quoted 6 lines · Emmanuel Florac
Le Fri, 13 Dec 2024 09:15:37 -0800
"Darrick J. Wong" <[email protected]> écrivait:Quoted 3 lines
No, I don't think that changes anything. If you can build your own kernel, can you try this out?
Quoted 7 lines
Quoted 3 lines
--D
xfs: don't over-report free space or inodes in statvfs
I'll give it a try, but that looks like a patch for old weird RedHat kernel, I'm running plain vanilla generally, and much higher versions, I'll see how it applies :)
That's from 6.13-rc3; I don't do RH kernels.
--D
Quoted 8 lines
Signature · 6 lines
------------------------------------------------------------------------ Emmanuel Florac | Direction technique ------------------------------------------------------------------------ https://intellique.com +33 6 16 30 15 95 ------------------------------------------------------------------------
Le Tue, 17 Dec 2024 08:50:42 -0800
"Darrick J. Wong" <[email protected]> écrivait:
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
xfs: don't over-report free space or inodes in statvfs
I'll give it a try, but that looks like a patch for old weird RedHat kernel, I'm running plain vanilla generally, and much higher versions, I'll see how it applies :)
That's from 6.13-rc3; I don't do RH kernels.
Sorry, I've seen the line
Quoted 1 line
Cc <[email protected]> # v2.6.18
And thought it was about some RH7.x kernel or something :)
Signature · 5 lines
------------------------------------------------------------------------ Emmanuel Florac | Direction technique ------------------------------------------------------------------------ https://intellique.com ------------------------------------------------------------------------
Quoted 9 lines · Emmanuel Florac
Le Tue, 17 Dec 2024 08:50:42 -0800
"Darrick J. Wong" <[email protected]> écrivait:Quoted 6 lines
Quoted 5 lines
Quoted 1 line
xfs: don't over-report free space or inodes in statvfs
I'll give it a try, but that looks like a patch for old weird RedHat kernel, I'm running plain vanilla generally, and much higher versions, I'll see how it applies :)
Quoted 7 lines
Quoted 1 line
That's from 6.13-rc3; I don't do RH kernels.
Sorry, I've seen the line
Quoted 1 line
Cc <[email protected]> # v2.6.18And thought it was about some RH7.x kernel or something :)
Ah. Nope, cc:stable and RHEL kernels are not related.
--D
Quoted 7 lines
Signature · 5 lines
------------------------------------------------------------------------ Emmanuel Florac | Direction technique ------------------------------------------------------------------------ https://intellique.com ------------------------------------------------------------------------