Skip to contentSkip to conversation

Weird behaviour with project quotas

DISCUSSION Emmanuel Florac 3 participants 12 messages Nov 28, 2024 – Dec 18, 2024 XFS filesystem

lore
Share & tools
lore
#0 lore raw

Hello,

As far as I understand, and from my tests, on folders on which a project quota is applied, either the available quota or the actually avialable space should be reported when using "df". However on a running system (Debian 12, kernel 6.1 Debian) I have incoherent results:

The volume /mnt/raid is 100 TB and has 500GB free.

There are several folders like /mnt/raid/project1, /mnt/raid/project2 etc with various quotas (20TB, 30TB, etc).

Though there are only 500 GB free on the whole filesystem, "df /mnt/raid/project1" etc reports several TB available, which is the remaining quota for the folder. Of course the users are led to believe there is more space available than what's actually the case (in fact that's why the volume is down to so little space) which causes all sort of problems.

What's strange is that I can't reproduce this behaviour on another system. Every time I try to reproduce it, available space reported by df on a project quota folder matches actual available space.

Any idea on what could be going wrong ?

Signature · 6 lines
------------------------------------------------------------------------
   Emmanuel Florac     |   Direction technique
------------------------------------------------------------------------
   https://intellique.com
   +33 6 16 30 15 95
------------------------------------------------------------------------
#1 lore raw
Quoted 6 lines · Emmanuel Florac

Hello,

As far as I understand, and from my tests, on folders on which a project quota is applied, either the available quota or the actually avialable space should be reported when using "df".

Only if you are using project quotas as directories quotas. i.e. the directory you are querying with df needs to have the XFS_DIFLAG_PROJINHERIT flag set on it for df to behave this way.

Quoted 2 lines

However on a
running system (Debian 12, kernel 6.1 Debian) I have incoherent results:

32 bit or 64 bit architecture?

Quoted 4 lines

The volume /mnt/raid is 100 TB and has 500GB free.

There are several folders like /mnt/raid/project1, /mnt/raid/project2 etc with various quotas (20TB, 30TB, etc).

Output of df and a project quota report showing usage and limits would be useful here.

Then, for each of the top level project directories you are querying with df, also run xfs_io -rxc "stat" <dir> and post the output. This will tell us if the project quota is set up correctly for df to report quota limits for them.

It would also be useful to know if the actual quota usage is correct
- having the output of du -s /mnt/raid/project1 to count the blocks and find /mnt/raid/project1 -print |wc -l to count the files in quota controlled directories. That'll give us some idea if there's a quota accounting issue.

Quoted 1 line

Any idea on what could be going wrong ?

No. There's nothing obvious that stands out in the current TOT code...

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#2 lore raw

Le Fri, 29 Nov 2024 08:07:09 +1100
Dave Chinner <[email protected]> écrivait:

Quoted 7 lines
Quoted 3 lines

As far as I understand, and from my tests, on folders on which a project quota is applied, either the available quota or the actually avialable space should be reported when using "df".

Only if you are using project quotas as directories quotas. i.e. the directory you are querying with df needs to have the XFS_DIFLAG_PROJINHERIT flag set on it for df to behave this way.

Interesting, and how is this set ? I basically set up quotas using something like

xfs_quota -x -c 'project -s -p /mnt/raid/project1 10' /mnt/raid

xfs_quota -x -c "limit -p bhard=30000g 10" /mnt/raid

Quoted 5 lines
Quoted 3 lines

However on a
running system (Debian 12, kernel 6.1 Debian) I have incoherent results:

32 bit or 64 bit architecture?

AMD64, the most common one.

Quoted 13 lines
Quoted 4 lines

The volume /mnt/raid is 100 TB and has 500GB free.

There are several folders like /mnt/raid/project1, /mnt/raid/project2 etc with various quotas (20TB, 30TB, etc).

Output of df and a project quota report showing usage and limits would be useful here.

Then, for each of the top level project directories you are querying with df, also run xfs_io -rxc "stat" <dir> and post the output. This will tell us if the project quota is set up correctly for df to report quota limits for them.

Quoted 6 lines

It would also be useful to know if the actual quota usage is correct
- having the output of du -s /mnt/raid/project1 to count the blocks and find /mnt/raid/project1 -print |wc -l to count the files in quota controlled directories. That'll give us some idea if there's a quota accounting issue.

OK, I'll run these as soon as I have a connection to the system. There seemed to be no error with the reported used space though, only available remaining space in quota'ed directories; however df reports actually available space for directories without quota set.

Signature · 5 lines
------------------------------------------------------------------------
   Emmanuel Florac     |   Direction technique
------------------------------------------------------------------------
   https://intellique.com
------------------------------------------------------------------------
#3 lore raw
Quoted 11 lines · Emmanuel Florac

Le Fri, 29 Nov 2024 08:07:09 +1100
Dave Chinner <[email protected]> écrivait:

Quoted 7 lines
Quoted 3 lines

As far as I understand, and from my tests, on folders on which a project quota is applied, either the available quota or the actually avialable space should be reported when using "df".

Only if you are using project quotas as directories quotas. i.e. the directory you are querying with df needs to have the XFS_DIFLAG_PROJINHERIT flag set on it for df to behave this way.

Quoted 6 lines

Interesting, and how is this set ? I basically set up quotas using something like

xfs_quota -x -c 'project -s -p /mnt/raid/project1 10' /mnt/raid

xfs_quota -x -c "limit -p bhard=30000g 10" /mnt/raid

That should set it up appropriately, hence the need to check if it has actually been set up correctly on disk.

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

However on a
running system (Debian 12, kernel 6.1 Debian) I have incoherent results:

Quoted 22 lines
Quoted 1 line

32 bit or 64 bit architecture?

AMD64, the most common one.

Quoted 18 lines
Quoted 4 lines

The volume /mnt/raid is 100 TB and has 500GB free.

There are several folders like /mnt/raid/project1, /mnt/raid/project2 etc with various quotas (20TB, 30TB, etc).

Output of df and a project quota report showing usage and limits would be useful here.

Then, for each of the top level project directories you are querying with df, also run xfs_io -rxc "stat" <dir> and post the output. This will tell us if the project quota is set up correctly for df to report quota limits for them.

It would also be useful to know if the actual quota usage is correct
- having the output of du -s /mnt/raid/project1 to count the blocks and find /mnt/raid/project1 -print |wc -l to count the files in quota controlled directories. That'll give us some idea if there's a quota accounting issue.

iAnother thought occurred to me - can you also check that /etc/projid and /etc/projects is similar on all machines, and post the contents of them from the bad machine?

Quoted 4 lines

OK, I'll run these as soon as I have a connection to the system. There seemed to be no error with the reported used space though, only available remaining space in quota'ed directories; however df reports actually available space for directories without quota set.

Ok - it would be helpful to post the commands you run and the output so that we both know exactly what behaviour you are seeing.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#4 lore raw

Le Sat, 30 Nov 2024 09:14:20 +1100
Dave Chinner <[email protected]> écrivait:

Quoted 5 lines
Quoted 1 line

xfs_quota -x -c "limit -p bhard=30000g 10" /mnt/raid

That should set it up appropriately, hence the need to check if it has actually been set up correctly on disk.

Unfortunately in the meantime the users did some cleanup, therefore the displayed information is coherent again (as there is more free space on the filesystem as a whole as any remaining allocated quota).

xfs_quota -x -c "report -p"
Project quota on /mnt/raid (/dev/mapper/vg0-raid)
Blocks
Project ID Used Soft Hard Warn/Grace

----------- ------------------------------------------------------

<snip>
#40 10795758244 0 16106127360 00 [--------]

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Output of df and a project quota report showing usage and limits would be useful here.

looking at the corresponding folder :

/dev/mapper/vg0-raid 15T 11T 5,0T 68% /mnt/raid/pad

du -s /mnt/raid/pad
10795758244 /mnt/raid/pad

# find /mnt/raid/pad -print | wc -l
39086
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Then, for each of the top level project directories you are querying with df, also run xfs_io -rxc "stat" <dir> and post the output. This will tell us if the project quota is set up correctly for df to report quota limits for them.

Starting with "pad" :

# xfs_io -rxc "stat" pad
fd.path = "."
fd.flags = non-sync,non-direct,read-only
stat.ino = 6442662464
stat.type = directory
stat.size = 4096
stat.blocks = 16
fsxattr.xflags = 0x200 \[--------P--------\]
fsxattr.projid = 40
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 2
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136
# xfs_io -rxc "stat" rush
fd.path = "."
fd.flags = non-sync,non-direct,read-only
stat.ino = 142
stat.type = directory
stat.size = 283
stat.blocks = 0
fsxattr.xflags = 0x200 \[--------P--------\]
fsxattr.projid = 10
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 0
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136
# xfs_io -rxc "stat" labo
fd.path = "."
fd.flags = non-sync,non-direct,read-only
stat.ino = 2147695168
stat.type = directory
stat.size = 310
stat.blocks = 0
fsxattr.xflags = 0x200 \[--------P--------\]
fsxattr.projid = 20
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 0
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136
# xfs_io -rxc "stat" prods
fd.path = "."
fd.flags = non-sync,non-direct,read-only
stat.ino = 4295178816
stat.type = directory
stat.size = 319
stat.blocks = 0
fsxattr.xflags = 0x200 \[--------P--------\]
fsxattr.projid = 30
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 0
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136
Quoted 11 lines
Quoted 6 lines
Quoted 6 lines

It would also be useful to know if the actual quota usage is correct
- having the output of du -s /mnt/raid/project1 to count the blocks and find /mnt/raid/project1 -print |wc -l to count the files in quota controlled directories. That'll give us some idea if there's a quota accounting issue.

iAnother thought occurred to me - can you also check that /etc/projid and /etc/projects is similar on all machines, and post the contents of them from the bad machine?

Hum, actually they didn't set up neither projid nor projects. Of course I did create these during my tests, but could this be the culprit ?

Signature · 6 lines
------------------------------------------------------------------------
   Emmanuel Florac     |   Direction technique
------------------------------------------------------------------------
   https://intellique.com
   +33 6 16 30 15 95
------------------------------------------------------------------------
#5 lore raw
Quoted 121 lines · Emmanuel Florac

Le Sat, 30 Nov 2024 09:14:20 +1100
Dave Chinner <[email protected]> écrivait:

Quoted 5 lines
Quoted 1 line

xfs_quota -x -c "limit -p bhard=30000g 10" /mnt/raid

That should set it up appropriately, hence the need to check if it has actually been set up correctly on disk.

Unfortunately in the meantime the users did some cleanup, therefore the displayed information is coherent again (as there is more free space on the filesystem as a whole as any remaining allocated quota).

xfs_quota -x -c "report -p"
Project quota on /mnt/raid (/dev/mapper/vg0-raid)
Blocks
Project ID Used Soft Hard Warn/Grace

----------- ------------------------------------------------------

<snip>
#40 10795758244 0 16106127360 00 [--------]

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Output of df and a project quota report showing usage and limits would be useful here.

looking at the corresponding folder :

/dev/mapper/vg0-raid 15T 11T 5,0T 68% /mnt/raid/pad

du -s /mnt/raid/pad
10795758244 /mnt/raid/pad

# find /mnt/raid/pad -print | wc -l
39086
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Then, for each of the top level project directories you are querying with df, also run xfs_io -rxc "stat" <dir> and post the output. This will tell us if the project quota is set up correctly for df to report quota limits for them.

Starting with "pad" :

# xfs_io -rxc "stat" pad
fd.path = "."
fd.flags = non-sync,non-direct,read-only
stat.ino = 6442662464
stat.type = directory
stat.size = 4096
stat.blocks = 16
fsxattr.xflags = 0x200 \[--------P--------\]
fsxattr.projid = 40
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 2
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136
# xfs_io -rxc "stat" rush
fd.path = "."
fd.flags = non-sync,non-direct,read-only
stat.ino = 142
stat.type = directory
stat.size = 283
stat.blocks = 0
fsxattr.xflags = 0x200 \[--------P--------\]
fsxattr.projid = 10
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 0
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136
# xfs_io -rxc "stat" labo
fd.path = "."
fd.flags = non-sync,non-direct,read-only
stat.ino = 2147695168
stat.type = directory
stat.size = 310
stat.blocks = 0
fsxattr.xflags = 0x200 \[--------P--------\]
fsxattr.projid = 20
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 0
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136
# xfs_io -rxc "stat" prods
fd.path = "."
fd.flags = non-sync,non-direct,read-only
stat.ino = 4295178816
stat.type = directory
stat.size = 319
stat.blocks = 0
fsxattr.xflags = 0x200 \[--------P--------\]
fsxattr.projid = 30
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 0
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136
Quoted 7 lines
Quoted 6 lines
Quoted 6 lines

It would also be useful to know if the actual quota usage is correct
- having the output of du -s /mnt/raid/project1 to count the blocks and find /mnt/raid/project1 -print |wc -l to count the files in quota controlled directories. That'll give us some idea if there's a quota accounting issue.

Quoted 7 lines
Quoted 4 lines

iAnother thought occurred to me - can you also check that /etc/projid and /etc/projects is similar on all machines, and post the contents of them from the bad machine?

Hum, actually they didn't set up neither projid nor projects. Of course I did create these during my tests, but could this be the culprit ?

Does this recreate the symptoms?

# mkfs.xfs -f /dev/sda
meta-data=/dev/sda               isize=512    agcount=4, agsize=1298176 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=1
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=1
         =                       exchange=0   metadir=0
data     =                       bsize=4096   blocks=5192704, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1, parent=0
log      =internal log           bsize=4096   blocks=16384, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
         =                       rgcount=0    rgsize=0 extents
Discarding blocks...Done.
# mount /dev/sda /mnt -o prjquota
# xfs_quota  -x -c 'limit -p bhard=2G 55' /mnt
# mkdir /mnt/dir
# xfs_io -c 'chproj 55' -c 'chattr +P' -c 'stat -vvvv' /mnt/dir
fd.path = "/mnt/dir"
fd.flags = non-sync,non-direct,read-write
stat.ino = 132
stat.type = directory
stat.size = 6
stat.blocks = 0
stat.atime = Thu Dec 12 12:07:53 2024
stat.mtime = Thu Dec 12 12:07:53 2024
stat.ctime = Thu Dec 12 12:08:12 2024
fsxattr.xflags = 0x200 [proj-inherit]
fsxattr.projid = 55
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 0
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136
# df /mnt /mnt/dir
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda         20G  420M   20G   3% /mnt
/dev/sda        2.0G     0  2.0G   0% /mnt
# fallocate -l 19g /mnt/a
# df /mnt /mnt/dir
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda         20G   20G  345M  99% /mnt
/dev/sda        2.0G     0  2.0G   0% /mnt

Clearly, df should be reporting 345M available for both cases, since we haven't actually used any of project 55's blocks.

# xfs_io -f -c 'pwrite -S 0x59 0 1m' -c fsync -c 'stat -vvvv' /mnt/dir/fork
wrote 1048576/1048576 bytes at offset 0
1 MiB, 256 ops; 0.0008 sec (1.121 GiB/sec and 293915.0402 ops/sec)
fd.path = "/mnt/dir/fork"
fd.flags = non-sync,non-direct,read-write
stat.ino = 134
stat.type = regular file
stat.size = 1048576
stat.blocks = 2048
stat.atime = Thu Dec 12 12:11:06 2024
stat.mtime = Thu Dec 12 12:11:06 2024
stat.ctime = Thu Dec 12 12:11:06 2024
fsxattr.xflags = 0x0 []
fsxattr.projid = 55
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 1
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136
# df /mnt /mnt/dir
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda         20G   20G  344M  99% /mnt
/dev/sda        2.0G  1.0M  2.0G   1% /mnt

I think this behavior comes from xfs_fill_statvfs_from_dquot, which does this:

	limit = blkres->softlimit ?
		blkres->softlimit :
		blkres->hardlimit;
	if (limit && statp->f_blocks > limit) {
		statp->f_blocks = limit;
		statp->f_bfree = statp->f_bavail =
			(statp->f_blocks > blkres->reserved) ?
			 (statp->f_blocks - blkres->reserved) : 0;
	}

I think the f_bfree/f_bavail assignment is wrong because it doesn't handle the case where f_bfree was less than (limit - reserved).

	if (limit) {
		uint64_t	remaining = 0;

		if (statp->f_blocks > limit)
			statp->f_blocks = limit;
		if (limit > blkres->reserved)
			remaining = limit - blkres->reserved;
		statp->f_bfree = min(statp->f_bfree, remaining);
		statp->f_bavail = min(statp->f_bavail, remaining);
	}

This fixes the df output a bit:

# df /mnt /mnt/dir
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda         20G   20G  344M  99% /mnt
/dev/sda        2.0G  1.7G  344M  84% /mnt

Though the "used" column is nonsense now. But I guess that's why statfs only defines total blocks and free/available blocks.

--D

Quoted 8 lines
Signature · 6 lines
------------------------------------------------------------------------
   Emmanuel Florac     |   Direction technique
------------------------------------------------------------------------
   https://intellique.com
   +33 6 16 30 15 95
------------------------------------------------------------------------
#6 lore raw

Le Thu, 12 Dec 2024 12:25:47 -0800
"Darrick J. Wong" <[email protected]> écrivait:

Quoted 2 lines

Does this recreate the symptoms?

<snip>

Quoted 66 lines
# df /mnt /mnt/dir
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda         20G  420M   20G   3% /mnt
/dev/sda        2.0G     0  2.0G   0% /mnt
# fallocate -l 19g /mnt/a
# df /mnt /mnt/dir
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda         20G   20G  345M  99% /mnt
/dev/sda        2.0G     0  2.0G   0% /mnt

Clearly, df should be reporting 345M available for both cases, since we haven't actually used any of project 55's blocks.

# xfs_io -f -c 'pwrite -S 0x59 0 1m' -c fsync -c 'stat -vvvv'
/mnt/dir/fork wrote 1048576/1048576 bytes at offset 0
1 MiB, 256 ops; 0.0008 sec (1.121 GiB/sec and 293915.0402 ops/sec)
fd.path = "/mnt/dir/fork"
fd.flags = non-sync,non-direct,read-write
stat.ino = 134
stat.type = regular file
stat.size = 1048576
stat.blocks = 2048
stat.atime = Thu Dec 12 12:11:06 2024
stat.mtime = Thu Dec 12 12:11:06 2024
stat.ctime = Thu Dec 12 12:11:06 2024
fsxattr.xflags = 0x0 []
fsxattr.projid = 55
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 1
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136
# df /mnt /mnt/dir
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda         20G   20G  344M  99% /mnt
/dev/sda        2.0G  1.0M  2.0G   1% /mnt

I think this behavior comes from xfs_fill_statvfs_from_dquot, which does this:

	limit = blkres->softlimit ?
		blkres->softlimit :
		blkres->hardlimit;
	if (limit && statp->f_blocks > limit) {
		statp->f_blocks = limit;
		statp->f_bfree = statp->f_bavail =
			(statp->f_blocks > blkres->reserved) ?
			 (statp->f_blocks - blkres->reserved) : 0;
	}

I think the f_bfree/f_bavail assignment is wrong because it doesn't handle the case where f_bfree was less than (limit - reserved).

	if (limit) {
		uint64_t	remaining = 0;

		if (statp->f_blocks > limit)
			statp->f_blocks = limit;
		if (limit > blkres->reserved)
			remaining = limit - blkres->reserved;
		statp->f_bfree = min(statp->f_bfree, remaining);
		statp->f_bavail = min(statp->f_bavail, remaining);
	}
Quoted 8 lines

This fixes the df output a bit:

# df /mnt /mnt/dir
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda         20G   20G  344M  99% /mnt
/dev/sda        2.0G  1.7G  344M  84% /mnt

Though the "used" column is nonsense now. But I guess that's why statfs only defines total blocks and free/available blocks.

Yep, that looks exactly like the problem we've met. Does the fact that not all folders have project quota change something in that case ?

Signature · 6 lines
------------------------------------------------------------------------
   Emmanuel Florac     |   Direction technique
------------------------------------------------------------------------
   https://intellique.com
   +33 6 16 30 15 95
------------------------------------------------------------------------
#7 lore raw
Quoted 79 lines · Emmanuel Florac

Le Thu, 12 Dec 2024 12:25:47 -0800
"Darrick J. Wong" <[email protected]> écrivait:

Quoted 2 lines

Does this recreate the symptoms?

<snip>

Quoted 72 lines
# df /mnt /mnt/dir
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda         20G  420M   20G   3% /mnt
/dev/sda        2.0G     0  2.0G   0% /mnt
# fallocate -l 19g /mnt/a
# df /mnt /mnt/dir
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda         20G   20G  345M  99% /mnt
/dev/sda        2.0G     0  2.0G   0% /mnt

Clearly, df should be reporting 345M available for both cases, since we haven't actually used any of project 55's blocks.

# xfs_io -f -c 'pwrite -S 0x59 0 1m' -c fsync -c 'stat -vvvv'
/mnt/dir/fork wrote 1048576/1048576 bytes at offset 0
1 MiB, 256 ops; 0.0008 sec (1.121 GiB/sec and 293915.0402 ops/sec)
fd.path = "/mnt/dir/fork"
fd.flags = non-sync,non-direct,read-write
stat.ino = 134
stat.type = regular file
stat.size = 1048576
stat.blocks = 2048
stat.atime = Thu Dec 12 12:11:06 2024
stat.mtime = Thu Dec 12 12:11:06 2024
stat.ctime = Thu Dec 12 12:11:06 2024
fsxattr.xflags = 0x0 []
fsxattr.projid = 55
fsxattr.extsize = 0
fsxattr.cowextsize = 0
fsxattr.nextents = 1
fsxattr.naextents = 0
dioattr.mem = 0x200
dioattr.miniosz = 512
dioattr.maxiosz = 2147483136
# df /mnt /mnt/dir
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda         20G   20G  344M  99% /mnt
/dev/sda        2.0G  1.0M  2.0G   1% /mnt

I think this behavior comes from xfs_fill_statvfs_from_dquot, which does this:

	limit = blkres->softlimit ?
		blkres->softlimit :
		blkres->hardlimit;
	if (limit && statp->f_blocks > limit) {
		statp->f_blocks = limit;
		statp->f_bfree = statp->f_bavail =
			(statp->f_blocks > blkres->reserved) ?
			 (statp->f_blocks - blkres->reserved) : 0;
	}

I think the f_bfree/f_bavail assignment is wrong because it doesn't handle the case where f_bfree was less than (limit - reserved).

	if (limit) {
		uint64_t	remaining = 0;

		if (statp->f_blocks > limit)
			statp->f_blocks = limit;
		if (limit > blkres->reserved)
			remaining = limit - blkres->reserved;
		statp->f_bfree = min(statp->f_bfree, remaining);
		statp->f_bavail = min(statp->f_bavail, remaining);
	}

This fixes the df output a bit:

# df /mnt /mnt/dir
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda         20G   20G  344M  99% /mnt
/dev/sda        2.0G  1.7G  344M  84% /mnt
Quoted 5 lines
Quoted 2 lines

Though the "used" column is nonsense now. But I guess that's why statfs only defines total blocks and free/available blocks.

Yep, that looks exactly like the problem we've met. Does the fact that not all folders have project quota change something in that case ?

No, I don't think that changes anything. If you can build your own kernel, can you try this out?

--D

xfs: don't over-report free space or inodes in statvfs

Emmanual Florac reports a strange occurrence when project quota limits are enabled, free space is lower than the remaining quota, and someone runs statvfs:

  # mkfs.xfs -f /dev/sda
  # mount /dev/sda /mnt -o prjquota
  # xfs_quota  -x -c 'limit -p bhard=2G 55' /mnt
  # mkdir /mnt/dir
  # xfs_io -c 'chproj 55' -c 'chattr +P' -c 'stat -vvvv' /mnt/dir
  # fallocate -l 19g /mnt/a
  # df /mnt /mnt/dir
  Filesystem      Size  Used Avail Use% Mounted on
  /dev/sda         20G   20G  345M  99% /mnt
  /dev/sda        2.0G     0  2.0G   0% /mnt

I think the bug here is that xfs_fill_statvfs_from_dquot unconditionally assigns to f_bfree without checking that the filesystem has enough free space to fill the remaining project quota. However, this is a longstanding behavior of xfs so it's unclear what to do here.

Cc <[email protected]> # v2.6.18
Fixes 932f2c323196c2 ("[XFS] statvfs component of directory/project quota support, code originally by Glen.")
---
 fs/xfs/xfs_qm_bhv.c |   27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)
fs/xfs/xfs_qm_bhv.c +17 -10
index 847ba29630e9d8..db5b8afd9d1b97 100644
@@ -32,21 +32,28 @@ xfs_fill_statvfs_from_dquot(
 	limit = blkres->softlimit ?
 		blkres->softlimit :
 		blkres->hardlimit;
-	if (limit && statp->f_blocks > limit) {
-		statp->f_blocks = limit;
-		statp->f_bfree = statp->f_bavail =
-			(statp->f_blocks > blkres->reserved) ?
-			 (statp->f_blocks - blkres->reserved) : 0;
+	if (limit) {
+		uint64_t	remaining = 0;
+
+		if (limit > blkres->reserved)
+			remaining = limit - blkres->reserved;
+
+		statp->f_blocks = min(statp->f_blocks, limit);
+		statp->f_bfree = min(statp->f_bfree, remaining);
+		statp->f_bavail = min(statp->f_bavail, remaining);
 	}
 
 	limit = dqp->q_ino.softlimit ?
 		dqp->q_ino.softlimit :
 		dqp->q_ino.hardlimit;
-	if (limit && statp->f_files > limit) {
-		statp->f_files = limit;
-		statp->f_ffree =
-			(statp->f_files > dqp->q_ino.reserved) ?
-			 (statp->f_files - dqp->q_ino.reserved) : 0;
+	if (limit) {
+		uint64_t	remaining = 0;
+
+		if (limit > dqp->q_ino.reserved)
+			remaining = limit - dqp->q_ino.reserved;
+
+		statp->f_files = min(statp->f_files, limit);
+		statp->f_ffree = min(statp->f_ffree, remaining);
 	}
 }
 
#8 lore raw

Le Fri, 13 Dec 2024 09:15:37 -0800
"Darrick J. Wong" <[email protected]> écrivait:

Quoted 6 lines

No, I don't think that changes anything. If you can build your own kernel, can you try this out?

--D

xfs: don't over-report free space or inodes in statvfs

I'll give it a try, but that looks like a patch for old weird RedHat kernel, I'm running plain vanilla generally, and much higher versions, I'll see how it applies :)

Signature · 6 lines
------------------------------------------------------------------------
   Emmanuel Florac     |   Direction technique
------------------------------------------------------------------------
   https://intellique.com
   +33 6 16 30 15 95
------------------------------------------------------------------------
#9 lore raw
Quoted 6 lines · Emmanuel Florac

Le Fri, 13 Dec 2024 09:15:37 -0800
"Darrick J. Wong" <[email protected]> écrivait:

Quoted 3 lines

No, I don't think that changes anything. If you can build your own kernel, can you try this out?

Quoted 7 lines
Quoted 3 lines

--D

xfs: don't over-report free space or inodes in statvfs

I'll give it a try, but that looks like a patch for old weird RedHat kernel, I'm running plain vanilla generally, and much higher versions, I'll see how it applies :)

That's from 6.13-rc3; I don't do RH kernels.

--D

Quoted 8 lines
Signature · 6 lines
------------------------------------------------------------------------
   Emmanuel Florac     |   Direction technique
------------------------------------------------------------------------
   https://intellique.com
   +33 6 16 30 15 95
------------------------------------------------------------------------
#10 lore raw

Le Tue, 17 Dec 2024 08:50:42 -0800
"Darrick J. Wong" <[email protected]> écrivait:

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

xfs: don't over-report free space or inodes in statvfs

I'll give it a try, but that looks like a patch for old weird RedHat kernel, I'm running plain vanilla generally, and much higher versions, I'll see how it applies :)

That's from 6.13-rc3; I don't do RH kernels.

Sorry, I've seen the line

Quoted 1 line
Cc <[email protected]> # v2.6.18

And thought it was about some RH7.x kernel or something :)

Signature · 5 lines
------------------------------------------------------------------------
   Emmanuel Florac     |   Direction technique
------------------------------------------------------------------------
   https://intellique.com
------------------------------------------------------------------------
#11 lore raw
Quoted 9 lines · Emmanuel Florac

Le Tue, 17 Dec 2024 08:50:42 -0800
"Darrick J. Wong" <[email protected]> écrivait:

Quoted 6 lines
Quoted 5 lines
Quoted 1 line

xfs: don't over-report free space or inodes in statvfs

I'll give it a try, but that looks like a patch for old weird RedHat kernel, I'm running plain vanilla generally, and much higher versions, I'll see how it applies :)

Quoted 7 lines
Quoted 1 line

That's from 6.13-rc3; I don't do RH kernels.

Sorry, I've seen the line

Quoted 1 line
Cc <[email protected]> # v2.6.18

And thought it was about some RH7.x kernel or something :)

Ah. Nope, cc:stable and RHEL kernels are not related.

--D

Quoted 7 lines
Signature · 5 lines
------------------------------------------------------------------------
   Emmanuel Florac     |   Direction technique
------------------------------------------------------------------------
   https://intellique.com
------------------------------------------------------------------------