Skip to contentSkip to conversation

ceph/005: verify correct statfs behaviour with quotas

PATCH Superseded Luís Henriques 3 participants 6 messages Jun 15, 2022 – Jun 27, 2022

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 ceph/005: verify correct statfs behaviour with quotas

When using a directory with 'max_bytes' quota as a base for a mount, statfs shall use that 'max_bytes' value as the total disk size. That value shall be used even when using subdirectory as base for the mount.

A bug was found where, when this subdirectory also had a 'max_files' quota, the real filesystem size would be returned instead of the parent 'max_bytes' quota value. This test case verifies this bug is fixed.

---
Finally, I've managed to come back to this test. The major changes since v1 are:
- creation of an helper for getting total mount space using 'df'
- now the test sends quota size to stdout

 common/rc          | 13 +++++++++++++
 tests/ceph/005     | 38 ++++++++++++++++++++++++++++++++++++++
 tests/ceph/005.out |  4 ++++
 3 files changed, 55 insertions(+)
 create mode 100755 tests/ceph/005
 create mode 100644 tests/ceph/005.out
common/rc +13 -0
index 2f31ca464621..72eabb7a428c 100644
@@ -4254,6 +4254,19 @@ _get_available_space()
 	echo $((avail_kb * 1024))
 }
 
+# get the total space in bytes
+#
+_get_total_space()
+{
+	if [ -z "$1" ]; then
+		echo "Usage: _get_total_space <mnt>"
+		exit 1
+	fi
+	local total_kb;
+	total_kb=`$DF_PROG $1 | tail -n1 | awk '{ print $3 }'`
+	echo $(($total_kb * 1024))
+}
+
 # return device size in kb
 _get_device_size()
 {
tests/ceph/005 new file +38 -0
new file mode 100755
index 000000000000..7eb687e8a092
@@ -0,0 +1,38 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test 005
+#
+# Make sure statfs reports correct total size when:
+# 1. using a directory with 'max_byte' quota as base for a mount
+# 2. using a subdirectory of the above directory with 'max_files' quota
+#
+. ./common/preamble
+_begin_fstest auto quick quota
+
+_supported_fs ceph
+_require_scratch
+
+_scratch_mount
+mkdir -p "$SCRATCH_MNT/quota-dir/subdir"
+
+# set quota
+quota=$((2 ** 30)) # 1G 
+$SETFATTR_PROG -n ceph.quota.max_bytes -v "$quota" "$SCRATCH_MNT/quota-dir"
+$SETFATTR_PROG -n ceph.quota.max_files -v "$quota" "$SCRATCH_MNT/quota-dir/subdir"
+_scratch_unmount
+
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir" _scratch_mount
+echo ceph quota size is $(_get_total_space "$SCRATCH_MNT") bytes
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir" _scratch_unmount
+
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir/subdir" _scratch_mount
+echo subdir ceph quota size is $(_get_total_space "$SCRATCH_MNT") bytes
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir/subdir" _scratch_unmount
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/ceph/005.out new file +4 -0
new file mode 100644
index 000000000000..47798b1fcd6f
@@ -0,0 +1,4 @@
+QA output created by 005
+ceph quota size is 1073741824 bytes
+subdir ceph quota size is 1073741824 bytes
+Silence is golden
#1 lore raw
Quoted 70 lines · Luís Henriques

When using a directory with 'max_bytes' quota as a base for a mount, statfs shall use that 'max_bytes' value as the total disk size. That value shall be used even when using subdirectory as base for the mount.

A bug was found where, when this subdirectory also had a 'max_files' quota, the real filesystem size would be returned instead of the parent 'max_bytes' quota value. This test case verifies this bug is fixed.

---
Finally, I've managed to come back to this test. The major changes since v1 are:
- creation of an helper for getting total mount space using 'df'
- now the test sends quota size to stdout

 common/rc          | 13 +++++++++++++
 tests/ceph/005     | 38 ++++++++++++++++++++++++++++++++++++++
 tests/ceph/005.out |  4 ++++
 3 files changed, 55 insertions(+)
 create mode 100755 tests/ceph/005
 create mode 100644 tests/ceph/005.out
common/rc +13 -0
index 2f31ca464621..72eabb7a428c 100644
@@ -4254,6 +4254,19 @@ _get_available_space()
 	echo $((avail_kb * 1024))
 }
 
+# get the total space in bytes
+#
+_get_total_space()
+{
+	if [ -z "$1" ]; then
+		echo "Usage: _get_total_space <mnt>"
+		exit 1
+	fi
+	local total_kb;
+	total_kb=`$DF_PROG $1 | tail -n1 | awk '{ print $3 }'`
+	echo $(($total_kb * 1024))
+}
+
 # return device size in kb
 _get_device_size()
 {
tests/ceph/005 new file +18 -0
new file mode 100755
index 000000000000..7eb687e8a092
@@ -0,0 +1,38 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test 005
+#
+# Make sure statfs reports correct total size when:
+# 1. using a directory with 'max_byte' quota as base for a mount
+# 2. using a subdirectory of the above directory with 'max_files' quota
+#
+. ./common/preamble
+_begin_fstest auto quick quota
+
+_supported_fs ceph
+_require_scratch
+
+_scratch_mount
+mkdir -p "$SCRATCH_MNT/quota-dir/subdir"

tests/ceph/005 line 28

Quoted 10 lines
tests/ceph/005 new file +10 -0
+
+# set quota
+quota=$((2 ** 30)) # 1G 
+$SETFATTR_PROG -n ceph.quota.max_bytes -v "$quota" "$SCRATCH_MNT/quota-dir"
+$SETFATTR_PROG -n ceph.quota.max_files -v "$quota" "$SCRATCH_MNT/quota-dir/subdir"
+_scratch_unmount
+
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir" _scratch_mount
+echo ceph quota size is $(_get_total_space "$SCRATCH_MNT") bytes
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir" _scratch_unmount

One week past, I hope to get some reviewing points from ceph mail list, due to I am really not familiar with cephfs, especially about the SCRATCH_DEV change as above.

I tried to ask if we can use a directory under TEST_DIR to do that, but Luís said he need the "SCRATCH_DEV=$SCRATCH_DEV/xxxx" things. Is it always a good usage for cephfs?

Hope to get review from ceph forks, Thanks!

Thanks,
Zorro

Quoted 21 lines
tests/ceph/005 new file +10 -0
+
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir/subdir" _scratch_mount
+echo subdir ceph quota size is $(_get_total_space "$SCRATCH_MNT") bytes
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir/subdir" _scratch_unmount
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/ceph/005.out new file +4 -0
new file mode 100644
index 000000000000..47798b1fcd6f
@@ -0,0 +1,4 @@
+QA output created by 005
+ceph quota size is 1073741824 bytes
+subdir ceph quota size is 1073741824 bytes
+Silence is golden
#2 lore raw
In reply to Luís Henriques

Hi Luis,

Sorry for late.

Quoted 71 lines · Luís Henriques

When using a directory with 'max_bytes' quota as a base for a mount, statfs shall use that 'max_bytes' value as the total disk size. That value shall be used even when using subdirectory as base for the mount.

A bug was found where, when this subdirectory also had a 'max_files' quota, the real filesystem size would be returned instead of the parent 'max_bytes' quota value. This test case verifies this bug is fixed.

---
Finally, I've managed to come back to this test. The major changes since v1 are:
- creation of an helper for getting total mount space using 'df'
- now the test sends quota size to stdout

  common/rc          | 13 +++++++++++++
  tests/ceph/005     | 38 ++++++++++++++++++++++++++++++++++++++
  tests/ceph/005.out |  4 ++++
  3 files changed, 55 insertions(+)

create mode 100755 tests/ceph/005
create mode 100644 tests/ceph/005.out

common/rc +13 -0
index 2f31ca464621..72eabb7a428c 100644
@@ -4254,6 +4254,19 @@ _get_available_space()
  	echo $((avail_kb * 1024))
  }
 
+# get the total space in bytes
+#
+_get_total_space()
+{
+	if [ -z "$1" ]; then
+		echo "Usage: _get_total_space <mnt>"
+		exit 1
+	fi
+	local total_kb;
+	total_kb=`$DF_PROG $1 | tail -n1 | awk '{ print $3 }'`
+	echo $(($total_kb * 1024))
+}
+
  # return device size in kb
  _get_device_size()
  {
tests/ceph/005 new file +19 -0
new file mode 100755
index 000000000000..7eb687e8a092
@@ -0,0 +1,38 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test 005
+#
+# Make sure statfs reports correct total size when:
+# 1. using a directory with 'max_byte' quota as base for a mount
+# 2. using a subdirectory of the above directory with 'max_files' quota
+#
+. ./common/preamble
+_begin_fstest auto quick quota
+
+_supported_fs ceph
+_require_scratch
+
+_scratch_mount
+mkdir -p "$SCRATCH_MNT/quota-dir/subdir"
+

tests/ceph/005 line 29

Quoted 10 lines
tests/ceph/005 new file +10 -0
+# set quota
+quota=$((2 ** 30)) # 1G
+$SETFATTR_PROG -n ceph.quota.max_bytes -v "$quota" "$SCRATCH_MNT/quota-dir"
+$SETFATTR_PROG -n ceph.quota.max_files -v "$quota" "$SCRATCH_MNT/quota-dir/subdir"
+_scratch_unmount
+
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir" _scratch_mount
+echo ceph quota size is $(_get_total_space "$SCRATCH_MNT") bytes
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir" _scratch_unmount
+

For the 'SCRATCH_DEV' here, if you do:

SCRATCH_DEV="$SCRATCH_DEV/quota-dir"

twice, won't it be "${ceph_scratch_dev}/quota-dir/quota-dir" at last ?

Shouldn't it be:

SCRATCH_DEV="$TEST_DIR/quota-dir" ?

-- Xiubo

Quoted 20 lines
tests/ceph/005 new file +9 -0
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir/subdir" _scratch_mount
+echo subdir ceph quota size is $(_get_total_space "$SCRATCH_MNT") bytes
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir/subdir" _scratch_unmount
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/ceph/005.out new file +4 -0
new file mode 100644
index 000000000000..47798b1fcd6f
@@ -0,0 +1,4 @@
+QA output created by 005
+ceph quota size is 1073741824 bytes
+subdir ceph quota size is 1073741824 bytes
+Silence is golden
#3 lore raw
Quoted 88 lines · Xiubo Li

Hi Luis,

Sorry for late.

Quoted 70 lines · Luís Henriques

When using a directory with 'max_bytes' quota as a base for a mount, statfs shall use that 'max_bytes' value as the total disk size. That value shall be used even when using subdirectory as base for the mount.

A bug was found where, when this subdirectory also had a 'max_files' quota, the real filesystem size would be returned instead of the parent 'max_bytes' quota value. This test case verifies this bug is fixed.

---
Finally, I've managed to come back to this test. The major changes since v1 are:
- creation of an helper for getting total mount space using 'df'
- now the test sends quota size to stdout

  common/rc          | 13 +++++++++++++
  tests/ceph/005     | 38 ++++++++++++++++++++++++++++++++++++++
  tests/ceph/005.out |  4 ++++
  3 files changed, 55 insertions(+)

create mode 100755 tests/ceph/005
create mode 100644 tests/ceph/005.out

common/rc +13 -0
index 2f31ca464621..72eabb7a428c 100644
@@ -4254,6 +4254,19 @@ _get_available_space()
  	echo $((avail_kb * 1024))
  }
+# get the total space in bytes
+#
+_get_total_space()
+{
+	if [ -z "$1" ]; then
+		echo "Usage: _get_total_space <mnt>"
+		exit 1
+	fi
+	local total_kb;
+	total_kb=`$DF_PROG $1 | tail -n1 | awk '{ print $3 }'`
+	echo $(($total_kb * 1024))
+}
+
  # return device size in kb
  _get_device_size()
  {
tests/ceph/005 new file +19 -0
new file mode 100755
index 000000000000..7eb687e8a092
@@ -0,0 +1,38 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test 005
+#
+# Make sure statfs reports correct total size when:
+# 1. using a directory with 'max_byte' quota as base for a mount
+# 2. using a subdirectory of the above directory with 'max_files' quota
+#
+. ./common/preamble
+_begin_fstest auto quick quota
+
+_supported_fs ceph
+_require_scratch
+
+_scratch_mount
+mkdir -p "$SCRATCH_MNT/quota-dir/subdir"
+

tests/ceph/005 line 29

Quoted 10 lines
tests/ceph/005 new file +10 -0
+# set quota
+quota=$((2 ** 30)) # 1G
+$SETFATTR_PROG -n ceph.quota.max_bytes -v "$quota" "$SCRATCH_MNT/quota-dir"
+$SETFATTR_PROG -n ceph.quota.max_files -v "$quota" "$SCRATCH_MNT/quota-dir/subdir"
+_scratch_unmount
+
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir" _scratch_mount
+echo ceph quota size is $(_get_total_space "$SCRATCH_MNT") bytes
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir" _scratch_unmount
+

For the 'SCRATCH_DEV' here, if you do:

Quoted 7 lines

SCRATCH_DEV="$SCRATCH_DEV/quota-dir"

twice, won't it be "${ceph_scratch_dev}/quota-dir/quota-dir" at last ?

Shouldn't it be:

SCRATCH_DEV="$TEST_DIR/quota-dir" ?

No, actually we do really need to have $SCRATCH_DEV and not $TEST_DIR here, because we want to have SCRATCH_DEV set to something like:

	<mon-ip-addr>:<port>:/quota-dir

so that we will use that directory (with quotas) as base for the mount.

Regarding the second attribution using the already modifed $SCRATCH_DEV variable, that's not really what is happening (and I had to go double-check myself, as you got me confused too :-).

So, if you do:

SCRATCH_DEV="$SCRATCH_DEV/quota-dir" _scratch_mount SCRATCH_DEV="$SCRATCH_DEV/quota-dir" _scratch_unmount

the SCRATCH_DEV value is changed *only* for the _scratch_[un]mount functions, but SCRATCH_DEV value isn't really changed after these attributions. This is probably a bashism (I'm not really sure), but you can see a similar pattern in other places (see, for example, test xfs/234).

Anyway, if you prefer, I'm fine sending v3 of this test doing something like:

SCRATCH_DEV_ORIG="$SCRATCH_DEV" SCRATCH_DEV="$SCRATCH_DEV_ORIG/quota-dir" _scratch_mount
...

Cheers,
--
Luís

Quoted 24 lines

-- Xiubo

Quoted 20 lines
tests/ceph/005 new file +9 -0
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir/subdir" _scratch_mount
+echo subdir ceph quota size is $(_get_total_space "$SCRATCH_MNT") bytes
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir/subdir" _scratch_unmount
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/ceph/005.out new file +4 -0
new file mode 100644
index 000000000000..47798b1fcd6f
@@ -0,0 +1,4 @@
+QA output created by 005
+ceph quota size is 1073741824 bytes
+subdir ceph quota size is 1073741824 bytes
+Silence is golden
#4 lore raw
Quoted 108 lines · Luís Henriques
Quoted 87 lines · Xiubo Li

Hi Luis,

Sorry for late.

Quoted 70 lines · Luís Henriques

When using a directory with 'max_bytes' quota as a base for a mount, statfs shall use that 'max_bytes' value as the total disk size. That value shall be used even when using subdirectory as base for the mount.

A bug was found where, when this subdirectory also had a 'max_files' quota, the real filesystem size would be returned instead of the parent 'max_bytes' quota value. This test case verifies this bug is fixed.

---
Finally, I've managed to come back to this test. The major changes since v1 are:

   - creation of an helper for getting total mount space using 'df'
   - now the test sends quota size to stdout

   common/rc          | 13 +++++++++++++
   tests/ceph/005     | 38 ++++++++++++++++++++++++++++++++++++++
   tests/ceph/005.out |  4 ++++
   3 files changed, 55 insertions(+)
   create mode 100755 tests/ceph/005
   create mode 100644 tests/ceph/005.out
common/rc +13 -0
index 2f31ca464621..72eabb7a428c 100644
@@ -4254,6 +4254,19 @@ _get_available_space()
   	echo $((avail_kb * 1024))
   }
+# get the total space in bytes
+#
+_get_total_space()
+{
+	if [ -z "$1" ]; then
+		echo "Usage: _get_total_space <mnt>"
+		exit 1
+	fi
+	local total_kb;
+	total_kb=`$DF_PROG $1 | tail -n1 | awk '{ print $3 }'`
+	echo $(($total_kb * 1024))
+}
+
   # return device size in kb
   _get_device_size()
   {
tests/ceph/005 new file +19 -0
new file mode 100755
index 000000000000..7eb687e8a092
@@ -0,0 +1,38 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test 005
+#
+# Make sure statfs reports correct total size when:
+# 1. using a directory with 'max_byte' quota as base for a mount
+# 2. using a subdirectory of the above directory with 'max_files' quota
+#
+. ./common/preamble
+_begin_fstest auto quick quota
+
+_supported_fs ceph
+_require_scratch
+
+_scratch_mount
+mkdir -p "$SCRATCH_MNT/quota-dir/subdir"
+

tests/ceph/005 line 29

Quoted 10 lines
tests/ceph/005 new file +10 -0
+# set quota
+quota=$((2 ** 30)) # 1G
+$SETFATTR_PROG -n ceph.quota.max_bytes -v "$quota" "$SCRATCH_MNT/quota-dir"
+$SETFATTR_PROG -n ceph.quota.max_files -v "$quota" "$SCRATCH_MNT/quota-dir/subdir"
+_scratch_unmount
+
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir" _scratch_mount
+echo ceph quota size is $(_get_total_space "$SCRATCH_MNT") bytes
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir" _scratch_unmount
+

For the 'SCRATCH_DEV' here, if you do:

Quoted 7 lines

SCRATCH_DEV="$SCRATCH_DEV/quota-dir"

twice, won't it be "${ceph_scratch_dev}/quota-dir/quota-dir" at last ?

Shouldn't it be:

SCRATCH_DEV="$TEST_DIR/quota-dir" ?

No, actually we do really need to have $SCRATCH_DEV and not $TEST_DIR here, because we want to have SCRATCH_DEV set to something like:

	<mon-ip-addr>:<port>:/quota-dir

so that we will use that directory (with quotas) as base for the mount.

Regarding the second attribution using the already modifed $SCRATCH_DEV variable, that's not really what is happening (and I had to go double-check myself, as you got me confused too :-).

So, if you do:

Quoted 8 lines

SCRATCH_DEV="$SCRATCH_DEV/quota-dir" _scratch_mount SCRATCH_DEV="$SCRATCH_DEV/quota-dir" _scratch_unmount

the SCRATCH_DEV value is changed *only* for the _scratch_[un]mount functions, but SCRATCH_DEV value isn't really changed after these attributions. This is probably a bashism (I'm not really sure), but you can see a similar pattern in other places (see, for example, test xfs/234).

Sorry, could you give the link about this ?

Checked the xfs/234, I didn't find any place is using the similar pattern.

This is what I see:

 53 # Now restore the obfuscated one back and take a look around 54 echo "Restore metadump"
 55 xfs_mdrestore $metadump_file $TEST_DIR/image
 56 SCRATCH_DEV=$TEST_DIR/image _scratch_mount
 57 SCRATCH_DEV=$TEST_DIR/image _scratch_unmount
 58
 59 echo "Check restored fs"
 60 _check_generic_filesystem $metadump_file

Quoted 2 lines

Anyway, if you prefer, I'm fine sending v3 of this test doing something like:

Locally I just test this use case, it seems working as my guess.

-- Xiubo

Quoted 31 lines

SCRATCH_DEV_ORIG="$SCRATCH_DEV" SCRATCH_DEV="$SCRATCH_DEV_ORIG/quota-dir" _scratch_mount
...

Cheers,
--
Luís

Quoted 22 lines

-- Xiubo

Quoted 20 lines
tests/ceph/005 new file +9 -0
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir/subdir" _scratch_mount
+echo subdir ceph quota size is $(_get_total_space "$SCRATCH_MNT") bytes
+SCRATCH_DEV="$SCRATCH_DEV/quota-dir/subdir" _scratch_unmount
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/ceph/005.out new file +4 -0
new file mode 100644
index 000000000000..47798b1fcd6f
@@ -0,0 +1,4 @@
+QA output created by 005
+ceph quota size is 1073741824 bytes
+subdir ceph quota size is 1073741824 bytes
+Silence is golden
#5 lore raw

On Mon, Jun 27, 2022 at 05:35:32PM +0800, Xiubo Li wrote:
<...>

Quoted 6 lines

Sorry, could you give the link about this ?

Checked the xfs/234, I didn't find any place is using the similar pattern.

This is what I see:

Quoted 9 lines

 53 # Now restore the obfuscated one back and take a look around 54 echo "Restore metadump"
 55 xfs_mdrestore $metadump_file $TEST_DIR/image
 56 SCRATCH_DEV=$TEST_DIR/image _scratch_mount
 57 SCRATCH_DEV=$TEST_DIR/image _scratch_unmount
 58
 59 echo "Check restored fs"
 60 _check_generic_filesystem $metadump_file

Doh! OK, you're right, looks like I'll need to change my glasses graduation again.

Quoted 5 lines
Quoted 3 lines

Anyway, if you prefer, I'm fine sending v3 of this test doing something like:

Locally I just test this use case, it seems working as my guess.

That's odd. Here's the test I've used locally to confirm it works:

8<---------------------------------------------------------------- #!/bin/bash

MYVAR="HELLO"

myfunc()
{
echo "in function: $MYVAR"
}

MYVAR="$MYVAR WORLD" myfunc
MYVAR="$MYVAR WORLD" myfunc
myfunc
echo "$MYVAR" 8<----------------------------------------------------------------

When I run this in my laptop I see:

in function: HELLO WORLD
in function: HELLO WORLD
in function: HELLO
HELLO

Anyway, I think it's better to make this behaviour explicit in the test. I'll send out v3 shortly. Thanks Xiubo.

Cheers,
--
Luís