ceph/005: verify correct statfs behaviour with quotas
PATCH 1/1 ceph/005: verify correct statfs behaviour with quotas
When using a directory with 'max_bytes' quota as a base for a mount, statfs shall use that 'max_bytes' value as the total disk size. That value shall be used even when using subdirectory as base for the mount.
A bug was found where, when this subdirectory also had a 'max_files' quota, the real filesystem size would be returned instead of the parent 'max_bytes' quota value. This test case verifies this bug is fixed.
--- tests/ceph/005 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/ceph/005.out | 2 ++ 2 files changed, 42 insertions(+) create mode 100755 tests/ceph/005 create mode 100644 tests/ceph/005.out
tests/ceph/005 new file +40 -0
@@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 005 +# +# Make sure statfs reports correct total size when: +# 1. using a directory with 'max_byte' quota as base for a mount +# 2. using a subdirectory of the above directory with 'max_files' quota +# +. ./common/preamble +_begin_fstest auto quick quota + +_supported_fs generic +_require_scratch + +_scratch_mount +mkdir -p $SCRATCH_MNT/quota-dir/subdir + +# set quotas +quota=$((1024*10000)) +$SETFATTR_PROG -n ceph.quota.max_bytes -v $quota $SCRATCH_MNT/quota-dir +$SETFATTR_PROG -n ceph.quota.max_files -v $quota $SCRATCH_MNT/quota-dir/subdir +_scratch_unmount + +SCRATCH_DEV=$SCRATCH_DEV/quota-dir _scratch_mount +total=`df -kP $SCRATCH_MNT | grep -v Filesystem | awk '{print $2}'` +SCRATCH_DEV=$SCRATCH_DEV/quota-dir _scratch_unmount +[ $total -eq 8192 ] || _fail "Incorrect statfs for quota-dir: $total" + +SCRATCH_DEV=$SCRATCH_DEV/quota-dir/subdir _scratch_mount +total=`df -kP $SCRATCH_MNT | grep -v Filesystem | awk '{print $2}'` +SCRATCH_DEV=$SCRATCH_DEV/quota-dir/subdir _scratch_unmount +[ $total -eq 8192 ] || _fail "Incorrect statfs for quota-dir/subdir: $total" + +echo "Silence is golden" + +# success, all done +status=0 +exit
tests/ceph/005.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 005 +Silence is golden
Hi Luís,
It looks like this one is still in need of review...
Quoted 38 lines · Luís Henriques
When using a directory with '
max_bytes' quota as a base for a mount, statfs shall use that 'max_bytes' value as the total disk size. That value shall be used even when using subdirectory as base for the mount.A bug was found where, when this subdirectory also had a '
max_files' quota, the real filesystem size would be returned instead of the parent 'max_bytes' quota value. This test case verifies this bug is fixed.Signed-off-by Luís Henriques <[email protected]>--- tests/ceph/005 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/ceph/005.out | 2 ++ 2 files changed, 42 insertions(+) create mode 100755 tests/ceph/005 create mode 100644 tests/ceph/005.outtests/ceph/005 new file +16
-0@@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 005 +# +# Make sure statfs reports correct total size when: +# 1. using a directory with 'max_byte' quota as base for a mount +# 2. using a subdirectory of the above directory with 'max_files' quota +# +. ./common/preamble +_begin_fstest auto quick quota + +_supported_fs generic +_require_scratch +
tests/ceph/005 line 26
Quoted 10 lines
tests/ceph/005 new file +10
-0+_scratch_mount +mkdir -p $SCRATCH_MNT/quota-dir/subdir + +# set quotas +quota=$((1024*10000)) +$SETFATTR_PROG -n ceph.quota.max_bytes -v $quota $SCRATCH_MNT/quota-dir +$SETFATTR_PROG -n ceph.quota.max_files -v $quota $SCRATCH_MNT/quota-dir/subdir +_scratch_unmount + +SCRATCH_DEV=$SCRATCH_DEV/quota-dir _scratch_mount
Aside from the standard please-quote-your-variables gripe, I'm a little confused with the use of SCRATCH_DEV for this test. Network FSes where mkfs isn't provided don't generally use it. Is there any way that this could be run against TEST_DEV, or does the umount / mount complicate things too much?
Cheers, David
Quoted 3 lines · David Disseldorp
Hi Luís,
It looks like this one is still in need of review...
Ah! Thanks for reminding me about it, David!
Quoted 45 lines · Luís Henriques
Quoted 43 lines
When using a directory with '
max_bytes' quota as a base for a mount, statfs shall use that 'max_bytes' value as the total disk size. That value shall be used even when using subdirectory as base for the mount.A bug was found where, when this subdirectory also had a '
max_files' quota, the real filesystem size would be returned instead of the parent 'max_bytes' quota value. This test case verifies this bug is fixed.Signed-off-by Luís Henriques <[email protected]>--- tests/ceph/005 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/ceph/005.out | 2 ++ 2 files changed, 42 insertions(+) create mode 100755 tests/ceph/005 create mode 100644 tests/ceph/005.outtests/ceph/005 new file +20
-0@@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 005 +# +# Make sure statfs reports correct total size when: +# 1. using a directory with 'max_byte' quota as base for a mount +# 2. using a subdirectory of the above directory with 'max_files' quota +# +. ./common/preamble +_begin_fstest auto quick quota + +_supported_fs generic +_require_scratch + +_scratch_mount +mkdir -p $SCRATCH_MNT/quota-dir/subdir + +# set quotas
Quoted 8 lines
tests/ceph/005 line 26
Quoted 6 lines
tests/ceph/005 new file +6
-0+quota=$((1024*10000)) +$SETFATTR_PROG -n ceph.quota.max_bytes -v $quota $SCRATCH_MNT/quota-dir +$SETFATTR_PROG -n ceph.quota.max_files -v $quota $SCRATCH_MNT/quota-dir/subdir +_scratch_unmount + +SCRATCH_DEV=$SCRATCH_DEV/quota-dir _scratch_mountAside from the standard please-quote-your-variables gripe, I'm a little
Sure, I'll fix those in next iteration.
Quoted 4 lines
confused with the use of
SCRATCH_DEVfor this test. Network FSes where mkfs isn't provided don't generally use it. Is there any way that this could be run againstTEST_DEV, or does the umount / mount complicate things too much?
When I looked at other tests doing similar things (i.e. changing the mount device during the test), they all seemed to be using SCRATCH_DEV. I guess that I could change TEST_DEV instead. I'll revisit this and see if that works.
Anyway, regarding the usage of SCRATCH_DEV in cephfs, I've used several different approaches:
- Use 2 different filesystems created on the same cluster,
- Use 2 volumes on the same filesystem, or
- Simply use 2 directories in the same filesystem.
I tend to use the later most of the times, as it's easier to setup :-)
Cheers,
Signature · 1 line
Luís
Quoted 74 lines · Luís Henriques
Quoted 3 lines · David Disseldorp
Hi Luís,
It looks like this one is still in need of review...
Ah! Thanks for reminding me about it, David!
Quoted 45 lines · Luís Henriques
Quoted 43 lines
When using a directory with '
max_bytes' quota as a base for a mount, statfs shall use that 'max_bytes' value as the total disk size. That value shall be used even when using subdirectory as base for the mount.A bug was found where, when this subdirectory also had a '
max_files' quota, the real filesystem size would be returned instead of the parent 'max_bytes' quota value. This test case verifies this bug is fixed.Signed-off-by Luís Henriques <[email protected]>--- tests/ceph/005 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/ceph/005.out | 2 ++ 2 files changed, 42 insertions(+) create mode 100755 tests/ceph/005 create mode 100644 tests/ceph/005.outtests/ceph/005 new file +20
-0@@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 005 +# +# Make sure statfs reports correct total size when: +# 1. using a directory with 'max_byte' quota as base for a mount +# 2. using a subdirectory of the above directory with 'max_files' quota +# +. ./common/preamble +_begin_fstest auto quick quota + +_supported_fs generic +_require_scratch + +_scratch_mount +mkdir -p $SCRATCH_MNT/quota-dir/subdir + +# set quotasQuoted 8 lines
tests/ceph/005 line 26
Quoted 6 lines
tests/ceph/005 new file +6
-0+quota=$((1024*10000)) +$SETFATTR_PROG -n ceph.quota.max_bytes -v $quota $SCRATCH_MNT/quota-dir +$SETFATTR_PROG -n ceph.quota.max_files -v $quota $SCRATCH_MNT/quota-dir/subdir +_scratch_unmount + +SCRATCH_DEV=$SCRATCH_DEV/quota-dir _scratch_mountAside from the standard please-quote-your-variables gripe, I'm a little
Sure, I'll fix those in next iteration.
Quoted 4 lines
confused with the use of
SCRATCH_DEVfor this test. Network FSes where mkfs isn't provided don't generally use it. Is there any way that this could be run againstTEST_DEV, or does the umount / mount complicate things too much?When I looked at other tests doing similar things (i.e. changing the mount device during the test), they all seemed to be using
SCRATCH_DEV. I guess that I could changeTEST_DEVinstead. I'll revisit this and see if that works.
Quoted 6 lines
Anyway, regarding the usage of
SCRATCH_DEVin cephfs, I've used several different approaches:- Use 2 different filesystems created on the same cluster,
- Use 2 volumes on the same filesystem, or
- Simply use 2 directories in the same filesystem.
Looking at _scratch_mkfs($FSTYP=ceph) there is support for scratch filesystem reinitialization, so I suppose this should be okay. With cephfs we could actually go one step further and call "ceph fs rm/new", but that's something for another day :-).
Cheers, David
Quoted 79 lines · David Disseldorp
Quoted 77 lines · Luís Henriques
Quoted 3 lines · David Disseldorp
Hi Luís,
It looks like this one is still in need of review...
Ah! Thanks for reminding me about it, David!
Quoted 45 lines · Luís Henriques
Quoted 43 lines
When using a directory with '
max_bytes' quota as a base for a mount, statfs shall use that 'max_bytes' value as the total disk size. That value shall be used even when using subdirectory as base for the mount.A bug was found where, when this subdirectory also had a '
max_files' quota, the real filesystem size would be returned instead of the parent 'max_bytes' quota value. This test case verifies this bug is fixed.Signed-off-by Luís Henriques <[email protected]>--- tests/ceph/005 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/ceph/005.out | 2 ++ 2 files changed, 42 insertions(+) create mode 100755 tests/ceph/005 create mode 100644 tests/ceph/005.outtests/ceph/005 new file +20
-0@@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 005 +# +# Make sure statfs reports correct total size when: +# 1. using a directory with 'max_byte' quota as base for a mount +# 2. using a subdirectory of the above directory with 'max_files' quota +# +. ./common/preamble +_begin_fstest auto quick quota + +_supported_fs generic +_require_scratch + +_scratch_mount +mkdir -p $SCRATCH_MNT/quota-dir/subdir + +# set quotasQuoted 8 lines
tests/ceph/005 line 26
Quoted 6 lines
tests/ceph/005 new file +6
-0+quota=$((1024*10000)) +$SETFATTR_PROG -n ceph.quota.max_bytes -v $quota $SCRATCH_MNT/quota-dir +$SETFATTR_PROG -n ceph.quota.max_files -v $quota $SCRATCH_MNT/quota-dir/subdir +_scratch_unmount + +SCRATCH_DEV=$SCRATCH_DEV/quota-dir _scratch_mountAside from the standard please-quote-your-variables gripe, I'm a little
Sure, I'll fix those in next iteration.
Quoted 4 lines
confused with the use of
SCRATCH_DEVfor this test. Network FSes where mkfs isn't provided don't generally use it. Is there any way that this could be run againstTEST_DEV, or does the umount / mount complicate things too much?When I looked at other tests doing similar things (i.e. changing the mount device during the test), they all seemed to be using
SCRATCH_DEV. I guess that I could changeTEST_DEVinstead. I'll revisit this and see if that works.Anyway, regarding the usage of
SCRATCH_DEVin cephfs, I've used several different approaches:
Quoted 8 lines
Quoted 3 lines
- Use 2 different filesystems created on the same cluster,
- Use 2 volumes on the same filesystem, or
- Simply use 2 directories in the same filesystem.Looking at
_scratch_mkfs($FSTYP=ceph)there is support for scratch filesystem reinitialization, so I suppose this should be okay. With cephfs we could actually go one step further and call "ceph fs rm/new", but that's something for another day :-).
Right, that would require a more complex test setup, with user-space tools on the test box/VM. It definitely would be desirable to have such an option for example in teuthology (the ceph testing framework), but for the simple runs I usually do on VMs, I'd rather keep the default as-is.
Cheers,
Signature · 1 line
Luís
Quoted 26 lines · Luís Henriques
When using a directory with '
max_bytes' quota as a base for a mount, statfs shall use that 'max_bytes' value as the total disk size. That value shall be used even when using subdirectory as base for the mount.A bug was found where, when this subdirectory also had a '
max_files' quota, the real filesystem size would be returned instead of the parent 'max_bytes' quota value. This test case verifies this bug is fixed.Signed-off-by Luís Henriques <[email protected]>--- tests/ceph/005 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/ceph/005.out | 2 ++ 2 files changed, 42 insertions(+) create mode 100755 tests/ceph/005 create mode 100644 tests/ceph/005.outtests/ceph/005 new file +4
-0@@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved. +#
tests/ceph/005 line 14
Quoted 10 lines
tests/ceph/005 new file +10
-0+# FS QA Test 005 +# +# Make sure statfs reports correct total size when: +# 1. using a directory with 'max_byte' quota as base for a mount +# 2. using a subdirectory of the above directory with 'max_files' quota +# +. ./common/preamble +_begin_fstest auto quick quota + +_supported_fs generic
As this case name is ceph/005, so I suppose you'd like to support 'ceph' only.
tests/ceph/005 line 26
Quoted 12 lines
tests/ceph/005 new file +12
-0+_require_scratch + +_scratch_mount +mkdir -p $SCRATCH_MNT/quota-dir/subdir + +# set quotas +quota=$((1024*10000)) +$SETFATTR_PROG -n ceph.quota.max_bytes -v $quota $SCRATCH_MNT/quota-dir +$SETFATTR_PROG -n ceph.quota.max_files -v $quota $SCRATCH_MNT/quota-dir/subdir +_scratch_unmount + +SCRATCH_DEV=$SCRATCH_DEV/quota-dir _scratch_mount
Try to not use SCRATCH_DEV like this.
tests/ceph/005 line 27
Quoted 1 line
tests/ceph/005 new file +1
-0+total=`df -kP $SCRATCH_MNT | grep -v Filesystem | awk '{print $2}'`
^^ $DF_PROG
As we have _get_total_inode(), _get_used_inode(), _get_used_inode_percent(), _get_free_inode() and _get_available_space() in common/rc, I don't mind add one more:
_get_total_space()
{
if [ -z "$1" ]; then
echo "Usage: _get_total_space <mnt>"
exit 1
fi
local total_kb;
total_kb=`$DF_PROG $1 | tail -n1 | awk '{ print $2 }'`
echo $((total_kb * 1024))
}tests/ceph/005 line 29
Quoted 2 lines
tests/ceph/005 new file +2
-0+SCRATCH_DEV=$SCRATCH_DEV/quota-dir _scratch_unmount +[ $total -eq 8192 ] || _fail "Incorrect statfs for quota-dir: $total"
^^^^
I'm not familar with ceph, I just found "quota=$((1024*10000))" in this case, didn't find any place metioned 8192. So may you help to demystify why we expect "8192" at here?
And if "8192" is a fixed expected number at here, then we can print it directly, as golden image, see below ...
tests/ceph/005 line 34
Quoted 5 lines
tests/ceph/005 new file +5
-0+ +SCRATCH_DEV=$SCRATCH_DEV/quota-dir/subdir _scratch_mount +total=`df -kP $SCRATCH_MNT | grep -v Filesystem | awk '{print $2}'` +SCRATCH_DEV=$SCRATCH_DEV/quota-dir/subdir _scratch_unmount +[ $total -eq 8192 ] || _fail "Incorrect statfs for quota-dir/subdir: $total"
May below code helps?
_require_test
localdir=$TEST_DIR/ceph-quota-dir-$seq
rm -rf $localdir
mkdir -p ${localdir}/subdir
...
$SETFATTR_PROG -n ceph.quota.max_bytes -v $quota $localdir $SETFATTR_PROG -n ceph.quota.max_bytes -v $quota $localdir/subdir
...
SCRATCH_DEV=$localdir _scratch_mount
echo ceph quota size is $(_get_total_space $SCRATCH_MNT) SCRATCH_DEV=$localdir _scratch_unmount
SCRATCH_DEV=$localdir/subdir _scratch_mount
echo subdir ceph quota size is $(_get_total_space $SCRATCH_MNT) SCRATCH_DEV=$localdir/subdir _scratch_unmount
Thanks,
Zorro
Quoted 15 lines
tests/ceph/005 new file +6
-0+ +echo "Silence is golden" + +# success, all done +status=0 +exittests/ceph/005.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 005 +Silence is golden
Quoted 31 lines · Zorro Lang
Quoted 30 lines · Luís Henriques
When using a directory with '
max_bytes' quota as a base for a mount, statfs shall use that 'max_bytes' value as the total disk size. That value shall be used even when using subdirectory as base for the mount.A bug was found where, when this subdirectory also had a '
max_files' quota, the real filesystem size would be returned instead of the parent 'max_bytes' quota value. This test case verifies this bug is fixed.Signed-off-by Luís Henriques <[email protected]>--- tests/ceph/005 | 40 ++++++++++++++++++++++++++++++++++++++++ tests/ceph/005.out | 2 ++ 2 files changed, 42 insertions(+) create mode 100755 tests/ceph/005 create mode 100644 tests/ceph/005.outtests/ceph/005 new file +8
-0@@ -0,0 +1,40 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved. +# +# FS QA Test 005 +# +# Make sure statfs reports correct total size when: +# 1. using a directory with 'max_byte' quota as base for a mount
Quoted 8 lines
tests/ceph/005 line 14
Quoted 6 lines
tests/ceph/005 new file +6
-0+# 2. using a subdirectory of the above directory with 'max_files' quota +# +. ./common/preamble +_begin_fstest auto quick quota + +_supported_fs genericAs this case name is ceph/005, so I suppose you'd like to support 'ceph' only.
Yep, my mistake, sorry. I'll fix it in next rev.
Quoted 6 lines
Quoted 6 lines
tests/ceph/005 new file +6
-0+_require_scratch + +_scratch_mount +mkdir -p $SCRATCH_MNT/quota-dir/subdir + +# set quotas
Quoted 8 lines
tests/ceph/005 line 26
Quoted 6 lines
tests/ceph/005 new file +6
-0+quota=$((1024*10000)) +$SETFATTR_PROG -n ceph.quota.max_bytes -v $quota $SCRATCH_MNT/quota-dir +$SETFATTR_PROG -n ceph.quota.max_files -v $quota $SCRATCH_MNT/quota-dir/subdir +_scratch_unmount + +SCRATCH_DEV=$SCRATCH_DEV/quota-dir _scratch_mountTry to not use
SCRATCH_DEVlike this.
I used this because I saw other tests doing something similar. Basically, I need to remount a filesystem with a different base directory. Changing the SCRATCH_DEV looked like a simple solution.
Quoted 8 lines
tests/ceph/005 line 27
Quoted 1 line
tests/ceph/005 new file +1
-0+total=`df -kP $SCRATCH_MNT | grep -v Filesystem | awk '{print $2}'`^^ $DF_PROGAs we have
_get_total_inode(),_get_used_inode(),_get_used_inode_percent(),_get_free_inode()and_get_available_space()in common/rc, I don't mind add one more:
_get_total_space()
Quoted 9 lines
{
if [ -z "$1" ]; then
echo "Usage: _get_total_space <mnt>"
exit 1
fi
local total_kb;
total_kb=`$DF_PROG $1 | tail -n1 | awk '{ print $2 }'`
echo $((total_kb * 1024))
}Right, this makes sense.
Quoted 9 lines
tests/ceph/005 line 29
Quoted 2 lines
tests/ceph/005 new file +2
-0+SCRATCH_DEV=$SCRATCH_DEV/quota-dir _scratch_unmount +[ $total -eq 8192 ] || _fail "Incorrect statfs for quota-dir: $total"^^^^I'm not familar with ceph, I just found "quota=$((1024*10000))" in this case, didn't find any place metioned 8192. So may you help to demystify why we expect "8192" at here?
And if "8192" is a fixed expected number at here, then we can print it directly, as golden image, see below ...
Hmm... OK, I'm struggling to remember the details about this, and it was only a month ago I wrote this test! Which is a sign that I should have, at least, added a comment explaining this value.
I'll need to dig into the statfs code again to explain why we're setting quotas to 10M and 'df' shows 8M (which is the default size for 2 ceph objects btw). I'll revisit this test in the next few days and sort this mystery.
Thanks a lot for your review.
Cheers
Signature · 1 line
Luís
Quoted 45 lines
tests/ceph/005 line 34
Quoted 5 lines
tests/ceph/005 new file +5
-0+ +SCRATCH_DEV=$SCRATCH_DEV/quota-dir/subdir _scratch_mount +total=`df -kP $SCRATCH_MNT | grep -v Filesystem | awk '{print $2}'` +SCRATCH_DEV=$SCRATCH_DEV/quota-dir/subdir _scratch_unmount +[ $total -eq 8192 ] || _fail "Incorrect statfs for quota-dir/subdir: $total"May below code helps?
_require_testlocaldir=$TEST_DIR/ceph-quota-dir-$seq
rm -rf $localdir
mkdir -p ${localdir}/subdir
...
$SETFATTR_PROG -n ceph.quota.max_bytes -v $quota $localdir $SETFATTR_PROG -n ceph.quota.max_bytes -v $quota $localdir/subdir
...SCRATCH_DEV=$localdir
_scratch_mount
echo ceph quota size is $(_get_total_space $SCRATCH_MNT) SCRATCH_DEV=$localdir_scratch_unmountSCRATCH_DEV=$localdir/subdir
_scratch_mount
echo subdir ceph quota size is $(_get_total_space $SCRATCH_MNT) SCRATCH_DEV=$localdir/subdir_scratch_unmountThanks,
ZorroQuoted 15 lines
tests/ceph/005 new file +6
-0+ +echo "Silence is golden" + +# success, all done +status=0 +exittests/ceph/005.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 005 +Silence is golden