common/quota: fix to wait for all inodes been evicted in _check_quota_usage()
PATCH 1/1 common/quota: fix to wait for all inodes been evicted in _check_quota_usage()
generic/233 3s ... - output mismatch (see /share/git/fstests/results//generic/233.out.bad)
--- tests/generic/233.out 2025-01-12 21:57:40.259440359 +0800
+++ /share/git/fstests/results//generic/233.out.bad 2025-12-04 03:02:26.000000000 +0800
@@ -4,4 +4,12 @@
seed = S
Comparing user usage
+4c4
+< #1000 -- 31476 32000 32000 994 1000 1000
+---
+> #1000 -- 31476 32000 32000 944 1000 1000
...
(Run 'diff -u /share/git/fstests/tests/generic/233.out /share/git/fstests/results//generic/233.out.bad' to see the entire diff)Ran: generic/233
Failures: generic/233
Failed 1 of 1 tests
Sometimes, generic/233 will fail due to it founds inode count is mismatched in between in-memory one and on-disk one.
The reason is cgroup v2 implementation requires to increase inode reference, and then attach it to thread related cgroup writeback structure, once it needs to switch once write owner changes of target inode, a kernel thread will process it and then release inode reference via evict_inode().
So, sync & drop_cache may not guarantee all inodes being evicted, as cgroup has one more refernece on inodes during the time.
If inode has not been evicted, dquot inode reference will not be release, it will lead to inode quota mismatch.
Let's add a delay to wait for cgroup switching completion before quota check.
--- common/quota | 3 +++ 1 file changed, 3 insertions(+)
common/quota +3 -0
@@ -333,6 +333,9 @@ _check_quota_usage() # XXX: really need an ioctl instead of this big hammer echo 3 > /proc/sys/vm/drop_caches + # wait for inode_switch_wbs_wor_fn() to release inodes + sleep 3 + VFS_QUOTA=0 case $FSTYP in ext2|ext3|ext4|f2fs|gfs2|bcachefs)
Signature · 1 line
2.49.0
Quoted 44 lines · Chao Yu
generic/233 3s ... - output mismatch (see /share/git/fstests/results//generic/233.out.bad)
--- tests/generic/233.out 2025-01-12 21:57:40.259440359 +0800 +++ /share/git/fstests/results//generic/233.out.bad 2025-12-04 03:02:26.000000000 +0800 @@ -4,4 +4,12 @@ seed = S Comparing user usage +4c4 +< #1000 -- 31476 32000 32000 994 1000 1000 +--- +> #1000 -- 31476 32000 32000 944 1000 1000 ... (Run 'diff -u /share/git/fstests/tests/generic/233.out /share/git/fstests/results//generic/233.out.bad' to see the entire diff)Ran: generic/233
Failures: generic/233
Failed 1 of 1 testsSometimes, generic/233 will fail due to it founds inode count is mismatched in between in-memory one and on-disk one.
The reason is cgroup v2 implementation requires to increase inode reference, and then attach it to thread related cgroup writeback structure, once it needs to switch once write owner changes of target inode, a kernel thread will process it and then release inode reference via
evict_inode().So, sync &
drop_cachemay not guarantee all inodes being evicted, as cgroup has one more refernece on inodes during the time.If inode has not been evicted, dquot inode reference will not be release, it will lead to inode quota mismatch.
Let's add a delay to wait for cgroup switching completion before quota check.
Cc Jaegeuk Kim <[email protected]>Signed-off-by Chao Yu <[email protected]>--- common/quota | 3 +++ 1 file changed, 3 insertions(+)common/quota +0
-0
common/quota _check_quota_usage() · line 337
Quoted 5 lines
common/quota +2
-0@@ -333,6 +333,9 @@ _check_quota_usage() # XXX: really need an ioctl instead of this big hammer echo 3 > /proc/sys/vm/drop_caches + # wait for inode_switch_wbs_wor_fn() to release inodes + sleep 3
Odd -- I've never seen this problem crop up on ext4 or XFS. What's different about f2fs such that writeback is still in progress?
Oh, XFS cycles the mount and ext4 does ... quotaoff???
Perhaps the sleep 3 thing should only be done for f2fs? Or do you see it on other filesystems as well?
--D
Quoted 8 lines
common/quota +1
-0+ VFS_QUOTA=0 case $FSTYP in ext2|ext3|ext4|f2fs|gfs2|bcachefs)Signature · 1 line
2.49.0
Quoted 48 lines · Darrick J. Wong
Quoted 47 lines · Chao Yu
generic/233 3s ... - output mismatch (see /share/git/fstests/results//generic/233.out.bad)
--- tests/generic/233.out 2025-01-12 21:57:40.259440359 +0800 +++ /share/git/fstests/results//generic/233.out.bad 2025-12-04 03:02:26.000000000 +0800 @@ -4,4 +4,12 @@ seed = S Comparing user usage +4c4 +< #1000 -- 31476 32000 32000 994 1000 1000 +--- +> #1000 -- 31476 32000 32000 944 1000 1000 ... (Run 'diff -u /share/git/fstests/tests/generic/233.out /share/git/fstests/results//generic/233.out.bad' to see the entire diff)Ran: generic/233
Failures: generic/233
Failed 1 of 1 testsSometimes, generic/233 will fail due to it founds inode count is mismatched in between in-memory one and on-disk one.
The reason is cgroup v2 implementation requires to increase inode reference, and then attach it to thread related cgroup writeback structure, once it needs to switch once write owner changes of target inode, a kernel thread will process it and then release inode reference via
evict_inode().So, sync &
drop_cachemay not guarantee all inodes being evicted, as cgroup has one more refernece on inodes during the time.If inode has not been evicted, dquot inode reference will not be release, it will lead to inode quota mismatch.
Let's add a delay to wait for cgroup switching completion before quota check.
Cc Jaegeuk Kim <[email protected]>Signed-off-by Chao Yu <[email protected]>--- common/quota | 3 +++ 1 file changed, 3 insertions(+)common/quota +0
-0@@ -333,6 +333,9 @@ _check_quota_usage() # XXX: really need an ioctl instead of this big hammer echo 3 > /proc/sys/vm/drop_caches
Quoted 7 lines
common/quota _check_quota_usage() · line 337
Quoted 2 lines
common/quota +2
-0+ # wait for inode_switch_wbs_wor_fn() to release inodes + sleep 3Odd -- I've never seen this problem crop up on ext4 or XFS. What's different about f2fs such that writeback is still in progress?
Oh, XFS cycles the mount and ext4 does ... quotaoff???
f2fs does quotaoff like ext4 as well, but the difference ext4_quota_off() will call sync_filesystem() before turning off quota, f2fs doesn't, not sure this is the key to wait/speedup inode_switch_wbs_wor_fn() or not.
Quoted 1 line
Quoted 1 line
Perhaps the sleep 3 thing should only be done for f2fs?
Yes, I think so, let me revise the patch.
Quoted 1 line
Or do you see it on other filesystems as well?
I tested ext4, it doesn't fail.
Thanks,
Quoted 11 lines
--D
Quoted 8 lines
+
VFS_QUOTA=0
case $FSTYP in
ext2|ext3|ext4|f2fs|gfs2|bcachefs)Signature · 1 line
2.49.0