Skip to contentSkip to conversation

Initial CephFS tests

RFC PATCH Superseded Luis Henriques 5 participants 17 messages Apr 2, 2019 – Apr 16, 2019

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover Initial CephFS tests

Hi all,

This is the first attempt to (finally!) add CephFS specific tests to fstests. Hopefully, more will follow once an initial set is accepted -- I do have a few more that could easily be added, and I'm sure that in the future more CephFS-specific tests can continue to be developed. This test suite has already proved to be quite useful in finding bugs in this filesystem; adding specific tests for it is just the next logic step.

For now, just two simple tests to do some basic checks on quotas features.

Cheers,
--
Luis

Luis Henriques (2):
ceph: test basic ceph.quota.max_files quota
ceph: test basic ceph.quota.max_bytes quota

 tests/ceph/001      | 108 ++++++++++++++++++++++++++++++++
 tests/ceph/001.out  |   1 +
 tests/ceph/002      | 147 ++++++++++++++++++++++++++++++++++++++++++++
 tests/ceph/002.out  |   1 +
 tests/ceph/Makefile |  20 ++++++
 tests/ceph/group    |   2 +
 6 files changed, 279 insertions(+)
 create mode 100755 tests/ceph/001
 create mode 100644 tests/ceph/001.out
 create mode 100755 tests/ceph/002
 create mode 100644 tests/ceph/002.out
 create mode 100644 tests/ceph/Makefile
 create mode 100644 tests/ceph/group
#1 lore raw

RFC PATCH 1/2 ceph: test basic ceph.quota.max_files quota

Simple set of checks for CephFS max_files directory quota implementation.

---
 tests/ceph/001      | 108 ++++++++++++++++++++++++++++++++++++++++++++
 tests/ceph/001.out  |   1 +
 tests/ceph/Makefile |  20 ++++++++
 tests/ceph/group    |   1 +
 4 files changed, 130 insertions(+)
 create mode 100755 tests/ceph/001
 create mode 100644 tests/ceph/001.out
 create mode 100644 tests/ceph/Makefile
 create mode 100644 tests/ceph/group
tests/ceph/001 new file +108 -0
new file mode 100755
index 000000000000..c4fe310d5135
@@ -0,0 +1,108 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 SUSE LLC. All Rights Reserved.
+#
+# FS QA Test No. 001
+#
+# Test basic ceph.quota.max_files quota features.
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+testdir=$TEST_DIR/quota-test
+
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+	rm -rf $testdir
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/attr
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ceph
+
+_require_attrs
+
+set_quota()
+{
+	val=$1
+	dir=$2
+	$SETFATTR_PROG -n ceph.quota.max_files -v $val $dir >/dev/null 2>&1
+}
+get_quota()
+{
+	dir=$1
+	$GETFATTR_PROG --only-values -n ceph.quota.max_files $dir 2> /dev/null
+}
+
+mkdir $testdir
+
+# test setting quota
+set_quota 10 $testdir
+ret=$(get_quota $testdir)
+if [ "$ret" -ne 10 ]; then
+	_fail "expected max_files quota to be 10, got '$ret' instead"
+fi
+# set quota to largest acceptable value (0x7FFFFFFFFFFFFFFF)
+set_quota 9223372036854775807 $testdir
+ret=$(get_quota $testdir)
+if [ "$ret" -ne 9223372036854775807 ]; then
+	_fail "expected max_files quota to be 9223372036854775807, got '$ret' instead"
+fi
+# test resetting quota
+set_quota 0 $testdir
+ret=$(get_quota $testdir)
+if [ -n "$ret" ]; then
+	_fail "expected 0 max_files quota, got '$ret' instead"
+fi
+# set quota to invalid values (0x8000000000000000 and -1)
+set_quota 9223372036854775808 $testdir
+ret=$(get_quota $testdir)
+if [ -n "$ret" ]; then
+	_fail "expected max_files quota to be 0, got '$ret' instead"
+fi
+set_quota -1 $testdir
+ret=$(get_quota $testdir)
+if [ -n "$ret" ]; then
+	_fail "expected max_files quota to be 0, got '$ret' instead"
+fi
+
+set_quota 5 $testdir
+mkdir $testdir/0
+touch $testdir/0/1
+touch $testdir/0/2
+touch $testdir/3
+touch $testdir/4 2> /dev/null
+if [ $? -ne 1 ]; then
+	_fail "Should have failed to create file"
+fi
+mkdir $testdir/5 2> /dev/null
+if [ $? -ne 1 ]; then
+	_fail "Should have failed to create directory"
+fi
+
+set_quota 0 $testdir
+touch $testdir/4 2> /dev/null
+if [ $? -ne 0 ]; then
+	_fail "Should have succeeded to create file"
+fi
+mkdir $testdir/5 2> /dev/null
+if [ $? -ne 0 ]; then
+	_fail "Should have succeeded to create directory"
+fi
+
+# success, all done
+status=0
+exit
tests/ceph/001.out new file +1 -0
new file mode 100644
index 000000000000..097d046a6fd4
@@ -0,0 +1 @@
+QA output created by 001
tests/ceph/Makefile new file +20 -0
new file mode 100644
index 000000000000..f93862af4f31
@@ -0,0 +1,20 @@
+#
+# Copyright (c) 2019 SUSE LLC. All Rights Reserved.
+#
+
+TOPDIR = ../..
+include $(TOPDIR)/include/builddefs
+
+CEPHFS_DIR = cephfs
+TARGET_DIR = $(PKG_LIB_DIR)/$(TESTS_DIR)/$(CEPHFS_DIR)
+
+include $(BUILDRULES)
+
+install:
+	$(INSTALL) -m 755 -d $(TARGET_DIR)
+	$(INSTALL) -m 755 $(TESTS) $(TARGET_DIR)
+	$(INSTALL) -m 644 group $(TARGET_DIR)
+	$(INSTALL) -m 644 $(OUTFILES) $(TARGET_DIR)
+
+# Nothing.
+install-dev install-lib:
tests/ceph/group new file +1 -0
new file mode 100644
index 000000000000..e389bc6ec7ee
@@ -0,0 +1 @@
+001 auto quick quota
#2 lore raw

RFC PATCH 2/2 ceph: test basic ceph.quota.max_bytes quota

In reply to Luis Henriques

Simple set of checks for CephFS max_bytes directory quota implementation.

---
 tests/ceph/002     | 147 +++++++++++++++++++++++++++++++++++++++++++++
 tests/ceph/002.out |   1 +
 tests/ceph/group   |   1 +
 3 files changed, 149 insertions(+)
 create mode 100755 tests/ceph/002
 create mode 100644 tests/ceph/002.out
tests/ceph/002 new file +147 -0
new file mode 100755
index 000000000000..313865dc639e
@@ -0,0 +1,147 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 SUSE LLC. All Rights Reserved.
+#
+# FS QA Test No. 002
+#
+# This tests basic ceph.quota.max_bytes quota features.
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+testdir=$TEST_DIR/quota-test
+
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+	rm -rf $testdir
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/attr
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ceph
+
+_require_attrs
+
+set_quota()
+{
+	val=$1
+	dir=$2
+	$SETFATTR_PROG -n ceph.quota.max_bytes -v $val $dir >/dev/null 2>&1
+}
+
+get_quota()
+{
+	dir=$1
+	$GETFATTR_PROG --only-values -n ceph.quota.max_bytes $dir 2> /dev/null
+}
+
+# function to write a file.  We use a loop because quotas in CephFS is a
+# "best-effort" implementation, i.e. a write may actually be allowed even if the
+# quota is being exceeded.  Using a loop reduces the chances of this to happen.
+#
+# NOTE: 'size' parameter is in M
+write_file()
+{
+	file=$1
+	size=$2 # size in M
+	for (( i = 1; i < $size; i++ )); do
+		$XFS_IO_PROG -f -c "pwrite -W $((i * 1048576)) 1048576" \
+			     $file >/dev/null 2>&1
+	done
+}
+
+# Check a file size
+#
+# NOTE: 'expected' (size) parameter is in M
+check_file_size()
+{
+	file=$1
+	expected=$(($2 * 1048576))
+	size=$(stat -c %s $file)
+	if [ "$size" -ne "$expected" ]; then
+		_fail "Expecting file with $expected got $size"
+	fi
+}
+
+mkdir $testdir
+
+# test setting quota
+set_quota 1000000 $testdir
+ret=$(get_quota $testdir)
+if [ "$ret" -ne 1000000 ]; then
+	_fail "expected max_bytes quota to be 1000000, got '$ret' instead"
+fi
+# set quota to largest acceptable value (0x7FFFFFFFFFFFFFFF)
+set_quota 9223372036854775807 $testdir
+ret=$(get_quota $testdir)
+if [ "$ret" -ne 9223372036854775807 ]; then
+	_fail "expected max_bytes quota to be 9223372036854775807, got '$ret' instead"
+fi
+# test resetting quota
+set_quota 0 $testdir
+ret=$(get_quota $testdir)
+if [ -n "$ret" ]; then
+	_fail "expected 0 max_bytes quota, got '$ret' instead"
+fi
+# set quota to invalid values (0x8000000000000000 and -1)
+set_quota 9223372036854775808 $testdir
+ret=$(get_quota $testdir)
+if [ -n "$ret" ]; then
+	_fail "expected max_bytes quota to be 0, got '$ret' instead"
+fi
+set_quota -1 $testdir
+ret=$(get_quota $testdir)
+if [ -n "$ret" ]; then
+	_fail "expected max_bytes quota to be 0, got '$ret' instead"
+fi
+
+bigfile="$testdir/bigfile"
+
+# set quota to 10M
+set_quota $((10 * 1048576)) $testdir
+
+# write 9M file
+write_file $bigfile 9
+check_file_size $bigfile 9
+rm $bigfile
+
+# try to write 11M file
+write_file $bigfile 11 # 11M
+check_file_size $bigfile 10
+rm $bigfile
+
+# write 5 x 2M files
+for (( j = 1; j < 6; j++ )); do
+	smallfile="$testdir/smallfile_$j"
+	write_file $smallfile 2 # 2M
+	check_file_size $smallfile 2
+done
+
+# try write another 2M file
+smallfile="$testdir/smallfile_fail"
+write_file $smallfile 2
+check_file_size $smallfile 0
+
+# reset quota
+set_quota 0 $testdir
+
+# write 2M file
+write_file $smallfile 2
+check_file_size $smallfile 2
+
+# success, all done
+status=0
+exit
tests/ceph/002.out new file +1 -0
new file mode 100644
index 000000000000..c57ca23e5cbe
@@ -0,0 +1 @@
+QA output created by 002
tests/ceph/group +1 -0
index e389bc6ec7ee..02da95169c67 100644
@@ -1 +1,2 @@
 001 auto quick quota
+002 auto quick quota
#3 lore raw

Re: [RFC PATCH 2/2] ceph: test basic ceph.quota.max_bytes quota

Quoted 21 lines · Luis Henriques

Simple set of checks for CephFS max_bytes directory quota implementation.

---
 tests/ceph/002     | 147 +++++++++++++++++++++++++++++++++++++++++++++
 tests/ceph/002.out |   1 +
 tests/ceph/group   |   1 +
 3 files changed, 149 insertions(+)
 create mode 100755 tests/ceph/002
 create mode 100644 tests/ceph/002.out
tests/ceph/002 new file +4 -0
new file mode 100755
index 000000000000..313865dc639e
@@ -0,0 +1,147 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 SUSE LLC. All Rights Reserved.
+#

tests/ceph/002 line 14

Quoted 10 lines
tests/ceph/002 new file +10 -0
+# FS QA Test No. 002
+#
+# This tests basic ceph.quota.max_bytes quota features.
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+testdir=$TEST_DIR/quota-test

Try not to name local variables the same as when known global variables. When we talk about "test dir", we mean the mount point for the test device, not the local, tests specific work directory.
i.e. this is a "work dir", not a "test dir".

And, often, we just name them after the test that is running, so we can identify what test left them behind. i.e.

workdir=$TEST_DIR/$seq

tests/ceph/002 line 24

Quoted 10 lines
tests/ceph/002 new file +10 -0
+
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+	rm -rf $testdir

Leave it behind for post-mortem analysis if necessary, remove it before starting this test execution....

Quoted 21 lines
tests/ceph/002 new file +21 -0
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/attr
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ceph
+
+_require_attrs
+
+set_quota()
+{
+	val=$1
+	dir=$2
+	$SETFATTR_PROG -n ceph.quota.max_bytes -v $val $dir >/dev/null 2>&1
+}
+
+get_quota()

tests/ceph/002 line 55

Quoted 10 lines
tests/ceph/002 new file +10 -0
+{
+	dir=$1
+	$GETFATTR_PROG --only-values -n ceph.quota.max_bytes $dir 2> /dev/null
+}
+
+# function to write a file.  We use a loop because quotas in CephFS is a
+# "best-effort" implementation, i.e. a write may actually be allowed even if the
+# quota is being exceeded.  Using a loop reduces the chances of this to happen.
+#
+# NOTE: 'size' parameter is in M

xfs_io accepts "1m" as one megabyte.

tests/ceph/002 line 64

Quoted 9 lines
tests/ceph/002 new file +9 -0
+write_file()
+{
+	file=$1
+	size=$2 # size in M
+	for (( i = 1; i < $size; i++ )); do
+		$XFS_IO_PROG -f -c "pwrite -W $((i * 1048576)) 1048576" \
+			     $file >/dev/null 2>&1
+	done
+}

Makes no sense to me. xfs_io does a write() loop internally with this pwrite command of 4kB writes - the default buffer size. If you want xfs_io to loop doing 1MB sized pwrite() calls, then all you need is this:

	$XFS_IO_PROG -f -c "pwrite -w -B 1m 0 ${size}m" $file | _filter_xfs_io

tests/ceph/002 line 76

Quoted 12 lines
tests/ceph/002 new file +12 -0
+
+# Check a file size
+#
+# NOTE: 'expected' (size) parameter is in M
+check_file_size()
+{
+	file=$1
+	expected=$(($2 * 1048576))
+	size=$(stat -c %s $file)
+	if [ "$size" -ne "$expected" ]; then
+		_fail "Expecting file with $expected got $size"
+	fi

Just emit the size into the output, and if it's not the correct size the test will fail because of a golden output mismatch.

i.e. just do:

stat -c %s $file

and nothing else.

tests/ceph/002 line 86

Quoted 10 lines
tests/ceph/002 new file +10 -0
+}
+
+mkdir $testdir
+
+# test setting quota
+set_quota 1000000 $testdir
+ret=$(get_quota $testdir)
+if [ "$ret" -ne 1000000 ]; then
+	_fail "expected max_bytes quota to be 1000000, got '$ret' instead"
+fi

Ditto. You should almost never need to use "if [comparison] then _fail ...." in fstests, because the golden output matching is precisely for this purpose.

Quoted 13 lines
tests/ceph/002 new file +13 -0
+# set quota to largest acceptable value (0x7FFFFFFFFFFFFFFF)
+set_quota 9223372036854775807 $testdir
+ret=$(get_quota $testdir)
+if [ "$ret" -ne 9223372036854775807 ]; then
+	_fail "expected max_bytes quota to be 9223372036854775807, got '$ret' instead"
+fi
+# test resetting quota
+set_quota 0 $testdir
+ret=$(get_quota $testdir)
+if [ -n "$ret" ]; then
+	_fail "expected 0 max_bytes quota, got '$ret' instead"
+fi
+# set quota to invalid values (0x8000000000000000 and -1)

tests/ceph/002 line 109

Quoted 10 lines
tests/ceph/002 new file +10 -0
+set_quota 9223372036854775808 $testdir
+ret=$(get_quota $testdir)
+if [ -n "$ret" ]; then
+	_fail "expected max_bytes quota to be 0, got '$ret' instead"
+fi
+set_quota -1 $testdir
+ret=$(get_quota $testdir)
+if [ -n "$ret" ]; then
+	_fail "expected max_bytes quota to be 0, got '$ret' instead"
+fi

But didn't you test these boundary conditions in the last test? Why repeat them here?

tests/ceph/002 line 119

Quoted 9 lines
tests/ceph/002 new file +9 -0
+bigfile="$testdir/bigfile"
+
+# set quota to 10M
+set_quota $((10 * 1048576)) $testdir
+
+# write 9M file
+write_file $bigfile 9
+check_file_size $bigfile 9
+rm $bigfile

which is:

$XFS_IO_PROG -f -c "pwrite -W -B 1m 0 9m" $bigfile | _filter_xfs_io stat -c %s $bigfile
rm -f $bigfile

So no need for functions here...

tests/ceph/002.out line 1

Quoted 7 lines
tests/ceph/002.out new file +1 -0
new file mode 100644
index 000000000000..c57ca23e5cbe
@@ -0,0 +1 @@
+QA output created by 002

An empty golden image file and lots of _fail calls in the test is an indication you are probably not quite doing things the right way :P

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#4 lore raw

Re: [RFC PATCH 2/2] ceph: test basic ceph.quota.max_bytes quota

Quoted 106 lines · Dave Chinner
Quoted 21 lines · Luis Henriques

Simple set of checks for CephFS max_bytes directory quota implementation.

---
 tests/ceph/002     | 147 +++++++++++++++++++++++++++++++++++++++++++++
 tests/ceph/002.out |   1 +
 tests/ceph/group   |   1 +
 3 files changed, 149 insertions(+)
 create mode 100755 tests/ceph/002
 create mode 100644 tests/ceph/002.out
tests/ceph/002 new file +4 -0
new file mode 100755
index 000000000000..313865dc639e
@@ -0,0 +1,147 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 SUSE LLC. All Rights Reserved.
+#

tests/ceph/002 line 14

Quoted 10 lines
tests/ceph/002 new file +10 -0
+# FS QA Test No. 002
+#
+# This tests basic ceph.quota.max_bytes quota features.
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+testdir=$TEST_DIR/quota-test

Try not to name local variables the same as when known global variables. When we talk about "test dir", we mean the mount point for the test device, not the local, tests specific work directory.
i.e. this is a "work dir", not a "test dir".

And, often, we just name them after the test that is running, so we can identify what test left them behind. i.e.

workdir=$TEST_DIR/$seq

tests/ceph/002 line 24

Quoted 10 lines
tests/ceph/002 new file +10 -0
+
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+	rm -rf $testdir

Leave it behind for post-mortem analysis if necessary, remove it before starting this test execution....

Quoted 21 lines
tests/ceph/002 new file +21 -0
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/attr
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ceph
+
+_require_attrs
+
+set_quota()
+{
+	val=$1
+	dir=$2
+	$SETFATTR_PROG -n ceph.quota.max_bytes -v $val $dir >/dev/null 2>&1
+}
+
+get_quota()

tests/ceph/002 line 55

Quoted 10 lines
tests/ceph/002 new file +10 -0
+{
+	dir=$1
+	$GETFATTR_PROG --only-values -n ceph.quota.max_bytes $dir 2> /dev/null
+}
+
+# function to write a file.  We use a loop because quotas in CephFS is a
+# "best-effort" implementation, i.e. a write may actually be allowed even if the
+# quota is being exceeded.  Using a loop reduces the chances of this to happen.
+#
+# NOTE: 'size' parameter is in M

xfs_io accepts "1m" as one megabyte.

tests/ceph/002 line 64

Quoted 9 lines
tests/ceph/002 new file +9 -0
+write_file()
+{
+	file=$1
+	size=$2 # size in M
+	for (( i = 1; i < $size; i++ )); do
+		$XFS_IO_PROG -f -c "pwrite -W $((i * 1048576)) 1048576" \
+			     $file >/dev/null 2>&1
+	done
+}

Makes no sense to me. xfs_io does a write() loop internally with this pwrite command of 4kB writes - the default buffer size. If you want xfs_io to loop doing 1MB sized pwrite() calls, then all you need is this:

Quoted 2 lines
	$XFS_IO_PROG -f -c "pwrite -w -B 1m 0 ${size}m" $file | _filter_xfs_io

Thank you for your review, Dave. I'll make sure the next revision of these tests will include all your comments implemented... except for this one.

The reason I'm using a loop for writing a file is due to the nature of the (very!) loose definition of quotas in CephFS. Basically, clients will likely write some amount of data over the configured limit because the servers they are communicating with to write the data (the OSDs) have no idea about the concept of quotas (or files even); the filesystem view in the cluster is managed at a different level, with the help of the MDS and the client itself.

So, the loop in this function is simply to allow the metadata associated with the file to be updated while we're writing the file. If I use a single pwrite, the whole file will be written before we get a -EDQUOT.

If an admin wants to really enforce some hard quotas in the filesystem, there are other means to do that, but not at the filesystem level. There are some more details on the quota implementation in Ceph here:

http://docs.ceph.com/docs/master/cephfs/quota/

I hope this makes sense and helps understanding why I need a loop to be used in this test.

Cheers

Signature · 1 line
Luis
Quoted 96 lines

tests/ceph/002 line 76

Quoted 12 lines
tests/ceph/002 new file +12 -0
+
+# Check a file size
+#
+# NOTE: 'expected' (size) parameter is in M
+check_file_size()
+{
+	file=$1
+	expected=$(($2 * 1048576))
+	size=$(stat -c %s $file)
+	if [ "$size" -ne "$expected" ]; then
+		_fail "Expecting file with $expected got $size"
+	fi

Just emit the size into the output, and if it's not the correct size the test will fail because of a golden output mismatch.

i.e. just do:

stat -c %s $file

and nothing else.

tests/ceph/002 line 86

Quoted 10 lines
tests/ceph/002 new file +10 -0
+}
+
+mkdir $testdir
+
+# test setting quota
+set_quota 1000000 $testdir
+ret=$(get_quota $testdir)
+if [ "$ret" -ne 1000000 ]; then
+	_fail "expected max_bytes quota to be 1000000, got '$ret' instead"
+fi

Ditto. You should almost never need to use "if [comparison] then _fail ...." in fstests, because the golden output matching is precisely for this purpose.

Quoted 13 lines
tests/ceph/002 new file +13 -0
+# set quota to largest acceptable value (0x7FFFFFFFFFFFFFFF)
+set_quota 9223372036854775807 $testdir
+ret=$(get_quota $testdir)
+if [ "$ret" -ne 9223372036854775807 ]; then
+	_fail "expected max_bytes quota to be 9223372036854775807, got '$ret' instead"
+fi
+# test resetting quota
+set_quota 0 $testdir
+ret=$(get_quota $testdir)
+if [ -n "$ret" ]; then
+	_fail "expected 0 max_bytes quota, got '$ret' instead"
+fi
+# set quota to invalid values (0x8000000000000000 and -1)

tests/ceph/002 line 109

Quoted 10 lines
tests/ceph/002 new file +10 -0
+set_quota 9223372036854775808 $testdir
+ret=$(get_quota $testdir)
+if [ -n "$ret" ]; then
+	_fail "expected max_bytes quota to be 0, got '$ret' instead"
+fi
+set_quota -1 $testdir
+ret=$(get_quota $testdir)
+if [ -n "$ret" ]; then
+	_fail "expected max_bytes quota to be 0, got '$ret' instead"
+fi

But didn't you test these boundary conditions in the last test? Why repeat them here?

tests/ceph/002 line 119

Quoted 9 lines
tests/ceph/002 new file +9 -0
+bigfile="$testdir/bigfile"
+
+# set quota to 10M
+set_quota $((10 * 1048576)) $testdir
+
+# write 9M file
+write_file $bigfile 9
+check_file_size $bigfile 9
+rm $bigfile

which is:

$XFS_IO_PROG -f -c "pwrite -W -B 1m 0 9m" $bigfile | _filter_xfs_io stat -c %s $bigfile
rm -f $bigfile

So no need for functions here...

tests/ceph/002.out line 1

Quoted 7 lines
tests/ceph/002.out new file +1 -0
new file mode 100644
index 000000000000..c57ca23e5cbe
@@ -0,0 +1 @@
+QA output created by 002

An empty golden image file and lots of _fail calls in the test is an indication you are probably not quite doing things the right way :P

Cheers,

Dave.

#5 lore raw

Re: [RFC PATCH 2/2] ceph: test basic ceph.quota.max_bytes quota

Quoted 123 lines · Luis Henriques
Quoted 106 lines · Dave Chinner
Quoted 21 lines · Luis Henriques

Simple set of checks for CephFS max_bytes directory quota implementation.

---
 tests/ceph/002     | 147 +++++++++++++++++++++++++++++++++++++++++++++
 tests/ceph/002.out |   1 +
 tests/ceph/group   |   1 +
 3 files changed, 149 insertions(+)
 create mode 100755 tests/ceph/002
 create mode 100644 tests/ceph/002.out
tests/ceph/002 new file +4 -0
new file mode 100755
index 000000000000..313865dc639e
@@ -0,0 +1,147 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 SUSE LLC. All Rights Reserved.
+#

tests/ceph/002 line 14

Quoted 10 lines
tests/ceph/002 new file +10 -0
+# FS QA Test No. 002
+#
+# This tests basic ceph.quota.max_bytes quota features.
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+testdir=$TEST_DIR/quota-test

Try not to name local variables the same as when known global variables. When we talk about "test dir", we mean the mount point for the test device, not the local, tests specific work directory.
i.e. this is a "work dir", not a "test dir".

And, often, we just name them after the test that is running, so we can identify what test left them behind. i.e.

workdir=$TEST_DIR/$seq

tests/ceph/002 line 24

Quoted 10 lines
tests/ceph/002 new file +10 -0
+
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+	rm -rf $testdir

Leave it behind for post-mortem analysis if necessary, remove it before starting this test execution....

Quoted 21 lines
tests/ceph/002 new file +21 -0
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/attr
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ceph
+
+_require_attrs
+
+set_quota()
+{
+	val=$1
+	dir=$2
+	$SETFATTR_PROG -n ceph.quota.max_bytes -v $val $dir >/dev/null 2>&1
+}
+
+get_quota()

tests/ceph/002 line 55

Quoted 10 lines
tests/ceph/002 new file +10 -0
+{
+	dir=$1
+	$GETFATTR_PROG --only-values -n ceph.quota.max_bytes $dir 2> /dev/null
+}
+
+# function to write a file.  We use a loop because quotas in CephFS is a
+# "best-effort" implementation, i.e. a write may actually be allowed even if the
+# quota is being exceeded.  Using a loop reduces the chances of this to happen.
+#
+# NOTE: 'size' parameter is in M

xfs_io accepts "1m" as one megabyte.

tests/ceph/002 line 64

Quoted 9 lines
tests/ceph/002 new file +9 -0
+write_file()
+{
+	file=$1
+	size=$2 # size in M
+	for (( i = 1; i < $size; i++ )); do
+		$XFS_IO_PROG -f -c "pwrite -W $((i * 1048576)) 1048576" \
+			     $file >/dev/null 2>&1
+	done
+}

Makes no sense to me. xfs_io does a write() loop internally with this pwrite command of 4kB writes - the default buffer size. If you want xfs_io to loop doing 1MB sized pwrite() calls, then all you need is this:

Quoted 2 lines
	$XFS_IO_PROG -f -c "pwrite -w -B 1m 0 ${size}m" $file | _filter_xfs_io

Thank you for your review, Dave. I'll make sure the next revision of these tests will include all your comments implemented... except for this one.

The reason I'm using a loop for writing a file is due to the nature of the (very!) loose definition of quotas in CephFS. Basically, clients will likely write some amount of data over the configured limit because the servers they are communicating with to write the data (the OSDs) have no idea about the concept of quotas (or files even); the filesystem view in the cluster is managed at a different level, with the help of the MDS and the client itself.

Quoted 2 lines

So, the loop in this function is simply to allow the metadata associated with the file to be updated while we're writing the file. If I use a

But the metadata will be modified while writing the file even with a single invocation of xfs_io. It's just that you are moving the loop inside xfs_io rather than having to invoke xfs_io a lot of time. Also, just because you are using a single -c "pwrite" command doesn't mean this will translate to a single call to pwrite. As Dave mentioned, the default block size is 4k meaning :

"pwrite -w -B 1m 0 ${size}m"

will result in 'size / 1m' writes of size 1m, each being a distinct call to pwrite.

Quoted 13 lines

single pwrite, the whole file will be written before we get a -EDQUOT.

If an admin wants to really enforce some hard quotas in the filesystem, there are other means to do that, but not at the filesystem level. There are some more details on the quota implementation in Ceph here:

http://docs.ceph.com/docs/master/cephfs/quota/

I hope this makes sense and helps understanding why I need a loop to be used in this test.

Cheers

#6 lore raw

Re: [RFC PATCH 2/2] ceph: test basic ceph.quota.max_bytes quota

Quoted 124 lines · Nikolay Borisov
Quoted 123 lines · Luis Henriques
Quoted 106 lines · Dave Chinner
Quoted 21 lines · Luis Henriques

Simple set of checks for CephFS max_bytes directory quota implementation.

---
 tests/ceph/002     | 147 +++++++++++++++++++++++++++++++++++++++++++++
 tests/ceph/002.out |   1 +
 tests/ceph/group   |   1 +
 3 files changed, 149 insertions(+)
 create mode 100755 tests/ceph/002
 create mode 100644 tests/ceph/002.out
tests/ceph/002 new file +4 -0
new file mode 100755
index 000000000000..313865dc639e
@@ -0,0 +1,147 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2019 SUSE LLC. All Rights Reserved.
+#

tests/ceph/002 line 14

Quoted 10 lines
tests/ceph/002 new file +10 -0
+# FS QA Test No. 002
+#
+# This tests basic ceph.quota.max_bytes quota features.
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+testdir=$TEST_DIR/quota-test

Try not to name local variables the same as when known global variables. When we talk about "test dir", we mean the mount point for the test device, not the local, tests specific work directory.
i.e. this is a "work dir", not a "test dir".

And, often, we just name them after the test that is running, so we can identify what test left them behind. i.e.

workdir=$TEST_DIR/$seq

tests/ceph/002 line 24

Quoted 10 lines
tests/ceph/002 new file +10 -0
+
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+	rm -rf $testdir

Leave it behind for post-mortem analysis if necessary, remove it before starting this test execution....

Quoted 21 lines
tests/ceph/002 new file +21 -0
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/attr
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs ceph
+
+_require_attrs
+
+set_quota()
+{
+	val=$1
+	dir=$2
+	$SETFATTR_PROG -n ceph.quota.max_bytes -v $val $dir >/dev/null 2>&1
+}
+
+get_quota()

tests/ceph/002 line 55

Quoted 10 lines
tests/ceph/002 new file +10 -0
+{
+	dir=$1
+	$GETFATTR_PROG --only-values -n ceph.quota.max_bytes $dir 2> /dev/null
+}
+
+# function to write a file.  We use a loop because quotas in CephFS is a
+# "best-effort" implementation, i.e. a write may actually be allowed even if the
+# quota is being exceeded.  Using a loop reduces the chances of this to happen.
+#
+# NOTE: 'size' parameter is in M

xfs_io accepts "1m" as one megabyte.

tests/ceph/002 line 64

Quoted 9 lines
tests/ceph/002 new file +9 -0
+write_file()
+{
+	file=$1
+	size=$2 # size in M
+	for (( i = 1; i < $size; i++ )); do
+		$XFS_IO_PROG -f -c "pwrite -W $((i * 1048576)) 1048576" \
+			     $file >/dev/null 2>&1
+	done
+}

Makes no sense to me. xfs_io does a write() loop internally with this pwrite command of 4kB writes - the default buffer size. If you want xfs_io to loop doing 1MB sized pwrite() calls, then all you need is this:

Quoted 2 lines
	$XFS_IO_PROG -f -c "pwrite -w -B 1m 0 ${size}m" $file | _filter_xfs_io

Thank you for your review, Dave. I'll make sure the next revision of these tests will include all your comments implemented... except for this one.

The reason I'm using a loop for writing a file is due to the nature of the (very!) loose definition of quotas in CephFS. Basically, clients will likely write some amount of data over the configured limit because the servers they are communicating with to write the data (the OSDs) have no idea about the concept of quotas (or files even); the filesystem view in the cluster is managed at a different level, with the help of the MDS and the client itself.

Quoted 5 lines
Quoted 2 lines

So, the loop in this function is simply to allow the metadata associated with the file to be updated while we're writing the file. If I use a

But the metadata will be modified while writing the file even with a single invocation of xfs_io.

No, that's not true. It would be too expensive to keep the metadata server updated while writing to a file. So, making sure there's actually an open/close to the file (plus the fsync in pwrite) helps making sure the metadata is flushed into the MDS.

(And yes I _did_ tried to use xfs_io with the 1m loop and the test fails because it simply writes all the data and gets no EDQUOT error.)

Cheers,

Signature · 1 line
Luis
Quoted 24 lines

It's just that you are moving the loop inside xfs_io rather than having to invoke xfs_io a lot of time. Also, just because you are using a single -c "pwrite" command doesn't mean this will translate to a single call to pwrite. As Dave mentioned, the default block size is 4k meaning :

"pwrite -w -B 1m 0 ${size}m"

will result in 'size / 1m' writes of size 1m, each being a distinct call to pwrite.

Quoted 13 lines

single pwrite, the whole file will be written before we get a -EDQUOT.

If an admin wants to really enforce some hard quotas in the filesystem, there are other means to do that, but not at the filesystem level. There are some more details on the quota implementation in Ceph here:

http://docs.ceph.com/docs/master/cephfs/quota/

I hope this makes sense and helps understanding why I need a loop to be used in this test.

Cheers

#7 lore raw

Re: [RFC PATCH 2/2] ceph: test basic ceph.quota.max_bytes quota

Quoted 26 lines · Luis Henriques
Quoted 25 lines · Nikolay Borisov
Quoted 23 lines · Luis Henriques
Quoted 7 lines · Dave Chinner

Makes no sense to me. xfs_io does a write() loop internally with this pwrite command of 4kB writes - the default buffer size. If you want xfs_io to loop doing 1MB sized pwrite() calls, then all you need is this:

	$XFS_IO_PROG -f -c "pwrite -w -B 1m 0 ${size}m" $file | _filter_xfs_io

Thank you for your review, Dave. I'll make sure the next revision of these tests will include all your comments implemented... except for this one.

The reason I'm using a loop for writing a file is due to the nature of the (very!) loose definition of quotas in CephFS. Basically, clients will likely write some amount of data over the configured limit because the servers they are communicating with to write the data (the OSDs) have no idea about the concept of quotas (or files even); the filesystem view in the cluster is managed at a different level, with the help of the MDS and the client itself.

So, the loop in this function is simply to allow the metadata associated with the file to be updated while we're writing the file. If I use a

Quoted 7 lines
Quoted 2 lines

But the metadata will be modified while writing the file even with a single invocation of xfs_io.

No, that's not true. It would be too expensive to keep the metadata server updated while writing to a file. So, making sure there's actually an open/close to the file (plus the fsync in pwrite) helps making sure the metadata is flushed into the MDS.

/me sighs.

So you want:

	loop until ${size}MB written:
		write 1MB
		fsync
		  -> flush data to server
		  -> flush metadata to server

i.e. this one liner:

xfs_io -f -c "pwrite -D -B 1m 0 ${size}m" /path/to/file

Fundamentally, if you find yourself writing a loop around xfs_io to break up a sequential IO stream into individual chunks, then you are most likely doing something xfs_io can already do. And if xfs_io cannot do it, then the right thing to do is to modify xfs_io to be able to do it and then use xfs_io....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#8 lore raw

Re: [RFC PATCH 2/2] ceph: test basic ceph.quota.max_bytes quota

Quoted 45 lines · Dave Chinner
Quoted 26 lines · Luis Henriques
Quoted 25 lines · Nikolay Borisov
Quoted 23 lines · Luis Henriques
Quoted 7 lines · Dave Chinner

Makes no sense to me. xfs_io does a write() loop internally with this pwrite command of 4kB writes - the default buffer size. If you want xfs_io to loop doing 1MB sized pwrite() calls, then all you need is this:

	$XFS_IO_PROG -f -c "pwrite -w -B 1m 0 ${size}m" $file | _filter_xfs_io

Thank you for your review, Dave. I'll make sure the next revision of these tests will include all your comments implemented... except for this one.

The reason I'm using a loop for writing a file is due to the nature of the (very!) loose definition of quotas in CephFS. Basically, clients will likely write some amount of data over the configured limit because the servers they are communicating with to write the data (the OSDs) have no idea about the concept of quotas (or files even); the filesystem view in the cluster is managed at a different level, with the help of the MDS and the client itself.

So, the loop in this function is simply to allow the metadata associated with the file to be updated while we're writing the file. If I use a

Quoted 7 lines
Quoted 2 lines

But the metadata will be modified while writing the file even with a single invocation of xfs_io.

No, that's not true. It would be too expensive to keep the metadata server updated while writing to a file. So, making sure there's actually an open/close to the file (plus the fsync in pwrite) helps making sure the metadata is flushed into the MDS.

/me sighs.

So you want:

	loop until ${size}MB written:
		write 1MB
		fsync
		  -> flush data to server
		  -> flush metadata to server
Quoted 3 lines

i.e. this one liner:

xfs_io -f -c "pwrite -D -B 1m 0 ${size}m" /path/to/file

Unfortunately, that doesn't do what I want either :-/ (and I guess you meant '-b 1m', not '-B 1m', right?)

[ Zheng: please feel free to correct me if I'm saying something really
stupid below. ]

So, one of the key things in my loop is the open/close operations. When a file is closed in cephfs the capabilities (that's ceph jargon for what sort of operations a client is allowed to perform on an inode) will likely be released and that's when the metadata server will get the updated file size. Before that, the client is allowed to modify the file size if it has acquired the capabilities for doing so.

OTOH, a pwrite operation will eventually get the -EDQUOT even with the one-liner above because the client itself will realize it has exceeded a certain threshold set by the MDS and will eventually update the server with the new file size. However that won't happen at a deterministic file size. For example, if quota is 10m and we're writing 20m, we may get the error after writing 15m.

Does this make sense?

So, I guess I *could* use your one-liner in the test, but I would need to slightly change the test logic -- I would need to write enough data to the file to make sure I would get the -EDQUOT but I wouldn't be able to actually check the file size as it will not be constant.

Quoted 5 lines

Fundamentally, if you find yourself writing a loop around xfs_io to break up a sequential IO stream into individual chunks, then you are most likely doing something xfs_io can already do. And if xfs_io cannot do it, then the right thing to do is to modify xfs_io to be able to do it and then use xfs_io....

Got it! But I guess it wouldn't make sense to change xfs_io for this specific scenario where I want several open-write-close cycles.

Cheers,

Signature · 1 line
Luis
#9 lore raw

Re: [RFC PATCH 2/2] ceph: test basic ceph.quota.max_bytes quota

Quoted 47 lines · Luis Henriques
Quoted 45 lines · Dave Chinner
Quoted 26 lines · Luis Henriques
Quoted 25 lines · Nikolay Borisov
Quoted 23 lines · Luis Henriques
Quoted 7 lines · Dave Chinner

Makes no sense to me. xfs_io does a write() loop internally with this pwrite command of 4kB writes - the default buffer size. If you want xfs_io to loop doing 1MB sized pwrite() calls, then all you need is this:

	$XFS_IO_PROG -f -c "pwrite -w -B 1m 0 ${size}m" $file | _filter_xfs_io

Thank you for your review, Dave. I'll make sure the next revision of these tests will include all your comments implemented... except for this one.

The reason I'm using a loop for writing a file is due to the nature of the (very!) loose definition of quotas in CephFS. Basically, clients will likely write some amount of data over the configured limit because the servers they are communicating with to write the data (the OSDs) have no idea about the concept of quotas (or files even); the filesystem view in the cluster is managed at a different level, with the help of the MDS and the client itself.

So, the loop in this function is simply to allow the metadata associated with the file to be updated while we're writing the file. If I use a

Quoted 7 lines
Quoted 2 lines

But the metadata will be modified while writing the file even with a single invocation of xfs_io.

No, that's not true. It would be too expensive to keep the metadata server updated while writing to a file. So, making sure there's actually an open/close to the file (plus the fsync in pwrite) helps making sure the metadata is flushed into the MDS.

/me sighs.

So you want:

	loop until ${size}MB written:
		write 1MB
		fsync
		  -> flush data to server
		  -> flush metadata to server
Quoted 6 lines
Quoted 3 lines

i.e. this one liner:

xfs_io -f -c "pwrite -D -B 1m 0 ${size}m" /path/to/file

Unfortunately, that doesn't do what I want either :-/ (and I guess you meant '-b 1m', not '-B 1m', right?)

Yes. But I definitely did mean "-D" so that RWF_DSYNC was used with each 1MB write.

Quoted 9 lines

[ Zheng: please feel free to correct me if I'm saying something really
stupid below. ]

So, one of the key things in my loop is the open/close operations. When a file is closed in cephfs the capabilities (that's ceph jargon for what sort of operations a client is allowed to perform on an inode) will likely be released and that's when the metadata server will get the updated file size. Before that, the client is allowed to modify the file size if it has acquired the capabilities for doing so.

So you are saying that O_DSYNC writes on ceph do not force file size metadata changes to the metadata server to be made stable?

Quoted 4 lines

OTOH, a pwrite operation will eventually get the -EDQUOT even with the one-liner above because the client itself will realize it has exceeded a certain threshold set by the MDS and will eventually update the server with the new file size.

Sure, but if the client crashes without having sent the updated file size to the server as part of an extending O_DSYNC write, then how is it recovered when the client reconnects to the server and accesses the file again?

Quoted 5 lines

However that won't happen at a deterministic
file size. For example, if quota is 10m and we're writing 20m, we may get the error after writing 15m.

Does this make sense?

Only makes sense to me if O_DSYNC is ignored by the ceph client...

Quoted 5 lines

So, I guess I *could* use your one-liner in the test, but I would need to slightly change the test logic -- I would need to write enough data to the file to make sure I would get the -EDQUOT but I wouldn't be able to actually check the file size as it will not be constant.

Quoted 8 lines
Quoted 5 lines

Fundamentally, if you find yourself writing a loop around xfs_io to break up a sequential IO stream into individual chunks, then you are most likely doing something xfs_io can already do. And if xfs_io cannot do it, then the right thing to do is to modify xfs_io to be able to do it and then use xfs_io....

Got it! But I guess it wouldn't make sense to change xfs_io for this specific scenario where I want several open-write-close cycles.

That's how individual NFS client writes appear to filesystem under the NFS server. I've previously considered adding an option in xfs_io to mimic this open-write-close loop per buffer so it's easy to exercise such behaviours, but never actually required it to reproduce the problems I was chasing. So it's definitely something that xfs_io /could/ do if necessary.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#10 lore raw

Re: [RFC PATCH 2/2] ceph: test basic ceph.quota.max_bytes quota

Quoted 76 lines · Dave Chinner
Quoted 47 lines · Luis Henriques
Quoted 45 lines · Dave Chinner
Quoted 26 lines · Luis Henriques
Quoted 25 lines · Nikolay Borisov
Quoted 23 lines · Luis Henriques
Quoted 7 lines · Dave Chinner

Makes no sense to me. xfs_io does a write() loop internally with this pwrite command of 4kB writes - the default buffer size. If you want xfs_io to loop doing 1MB sized pwrite() calls, then all you need is this:

	$XFS_IO_PROG -f -c "pwrite -w -B 1m 0 ${size}m" $file | _filter_xfs_io

Thank you for your review, Dave. I'll make sure the next revision of these tests will include all your comments implemented... except for this one.

The reason I'm using a loop for writing a file is due to the nature of the (very!) loose definition of quotas in CephFS. Basically, clients will likely write some amount of data over the configured limit because the servers they are communicating with to write the data (the OSDs) have no idea about the concept of quotas (or files even); the filesystem view in the cluster is managed at a different level, with the help of the MDS and the client itself.

So, the loop in this function is simply to allow the metadata associated with the file to be updated while we're writing the file. If I use a

Quoted 7 lines
Quoted 2 lines

But the metadata will be modified while writing the file even with a single invocation of xfs_io.

No, that's not true. It would be too expensive to keep the metadata server updated while writing to a file. So, making sure there's actually an open/close to the file (plus the fsync in pwrite) helps making sure the metadata is flushed into the MDS.

/me sighs.

So you want:

	loop until ${size}MB written:
		write 1MB
		fsync
		  -> flush data to server
		  -> flush metadata to server
Quoted 6 lines
Quoted 3 lines

i.e. this one liner:

xfs_io -f -c "pwrite -D -B 1m 0 ${size}m" /path/to/file

Unfortunately, that doesn't do what I want either :-/ (and I guess you meant '-b 1m', not '-B 1m', right?)

Yes. But I definitely did mean "-D" so that RWF_DSYNC was used with each 1MB write.

Quoted 9 lines

[ Zheng: please feel free to correct me if I'm saying something really
stupid below. ]

So, one of the key things in my loop is the open/close operations. When a file is closed in cephfs the capabilities (that's ceph jargon for what sort of operations a client is allowed to perform on an inode) will likely be released and that's when the metadata server will get the updated file size. Before that, the client is allowed to modify the file size if it has acquired the capabilities for doing so.

So you are saying that O_DSYNC writes on ceph do not force file size metadata changes to the metadata server to be made stable?

Quoted 4 lines

OTOH, a pwrite operation will eventually get the -EDQUOT even with the one-liner above because the client itself will realize it has exceeded a certain threshold set by the MDS and will eventually update the server with the new file size.

Quoted 4 lines

Sure, but if the client crashes without having sent the updated file size to the server as part of an extending O_DSYNC write, then how is it recovered when the client reconnects to the server and accesses the file again?

For DSYNC write, client has already written data to object store. If client crashes, MDS will set file to 'recovering' state and probe file size by checking object store. Accessing the file is blocked during recovery.

Regards
Yan, Zheng

Quoted 34 lines
Quoted 6 lines

However that won't happen at a deterministic
file size. For example, if quota is 10m and we're writing 20m, we may get the error after writing 15m.

Does this make sense?

Only makes sense to me if O_DSYNC is ignored by the ceph client...

Quoted 5 lines

So, I guess I *could* use your one-liner in the test, but I would need to slightly change the test logic -- I would need to write enough data to the file to make sure I would get the -EDQUOT but I wouldn't be able to actually check the file size as it will not be constant.

Quoted 8 lines
Quoted 5 lines

Fundamentally, if you find yourself writing a loop around xfs_io to break up a sequential IO stream into individual chunks, then you are most likely doing something xfs_io can already do. And if xfs_io cannot do it, then the right thing to do is to modify xfs_io to be able to do it and then use xfs_io....

Got it! But I guess it wouldn't make sense to change xfs_io for this specific scenario where I want several open-write-close cycles.

That's how individual NFS client writes appear to filesystem under the NFS server. I've previously considered adding an option in xfs_io to mimic this open-write-close loop per buffer so it's easy to exercise such behaviours, but never actually required it to reproduce the problems I was chasing. So it's definitely something that xfs_io /could/ do if necessary.

Cheers,

Dave.

#11 lore raw

Re: [RFC PATCH 2/2] ceph: test basic ceph.quota.max_bytes quota

Quoted 83 lines · Zheng
Quoted 76 lines · Dave Chinner
Quoted 47 lines · Luis Henriques
Quoted 45 lines · Dave Chinner
Quoted 26 lines · Luis Henriques
Quoted 25 lines · Nikolay Borisov
Quoted 23 lines · Luis Henriques
Quoted 7 lines · Dave Chinner

Makes no sense to me. xfs_io does a write() loop internally with this pwrite command of 4kB writes - the default buffer size. If you want xfs_io to loop doing 1MB sized pwrite() calls, then all you need is this:

	$XFS_IO_PROG -f -c "pwrite -w -B 1m 0 ${size}m" $file | _filter_xfs_io

Thank you for your review, Dave. I'll make sure the next revision of these tests will include all your comments implemented... except for this one.

The reason I'm using a loop for writing a file is due to the nature of the (very!) loose definition of quotas in CephFS. Basically, clients will likely write some amount of data over the configured limit because the servers they are communicating with to write the data (the OSDs) have no idea about the concept of quotas (or files even); the filesystem view in the cluster is managed at a different level, with the help of the MDS and the client itself.

So, the loop in this function is simply to allow the metadata associated with the file to be updated while we're writing the file. If I use a

Quoted 7 lines
Quoted 2 lines

But the metadata will be modified while writing the file even with a single invocation of xfs_io.

No, that's not true. It would be too expensive to keep the metadata server updated while writing to a file. So, making sure there's actually an open/close to the file (plus the fsync in pwrite) helps making sure the metadata is flushed into the MDS.

/me sighs.

So you want:

	loop until ${size}MB written:
		write 1MB
		fsync
		  -> flush data to server
		  -> flush metadata to server
Quoted 6 lines
Quoted 3 lines

i.e. this one liner:

xfs_io -f -c "pwrite -D -B 1m 0 ${size}m" /path/to/file

Unfortunately, that doesn't do what I want either :-/ (and I guess you meant '-b 1m', not '-B 1m', right?)

Yes. But I definitely did mean "-D" so that RWF_DSYNC was used with each 1MB write.

Quoted 9 lines

[ Zheng: please feel free to correct me if I'm saying something really
stupid below. ]

So, one of the key things in my loop is the open/close operations. When a file is closed in cephfs the capabilities (that's ceph jargon for what sort of operations a client is allowed to perform on an inode) will likely be released and that's when the metadata server will get the updated file size. Before that, the client is allowed to modify the file size if it has acquired the capabilities for doing so.

So you are saying that O_DSYNC writes on ceph do not force file size metadata changes to the metadata server to be made stable?

Quoted 4 lines

OTOH, a pwrite operation will eventually get the -EDQUOT even with the one-liner above because the client itself will realize it has exceeded a certain threshold set by the MDS and will eventually update the server with the new file size.

Quoted 4 lines

Sure, but if the client crashes without having sent the updated file size to the server as part of an extending O_DSYNC write, then how is it recovered when the client reconnects to the server and accesses the file again?

Quoted 3 lines

For DSYNC write, client has already written data to object store. If client crashes, MDS will set file to 'recovering' state and probe file size by checking object store. Accessing the file is blocked during recovery.

Thank you for chiming in, Zheng.

Quoted 36 lines

Regards
Yan, Zheng

Quoted 29 lines
Quoted 6 lines

However that won't happen at a deterministic
file size. For example, if quota is 10m and we're writing 20m, we may get the error after writing 15m.

Does this make sense?

Only makes sense to me if O_DSYNC is ignored by the ceph client...

Quoted 5 lines

So, I guess I *could* use your one-liner in the test, but I would need to slightly change the test logic -- I would need to write enough data to the file to make sure I would get the -EDQUOT but I wouldn't be able to actually check the file size as it will not be constant.

Quoted 8 lines
Quoted 5 lines

Fundamentally, if you find yourself writing a loop around xfs_io to break up a sequential IO stream into individual chunks, then you are most likely doing something xfs_io can already do. And if xfs_io cannot do it, then the right thing to do is to modify xfs_io to be able to do it and then use xfs_io....

Got it! But I guess it wouldn't make sense to change xfs_io for this specific scenario where I want several open-write-close cycles.

That's how individual NFS client writes appear to filesystem under the NFS server. I've previously considered adding an option in xfs_io to mimic this open-write-close loop per buffer so it's easy to exercise such behaviours, but never actually required it to reproduce the problems I was chasing. So it's definitely something that xfs_io /could/ do if necessary.

Ok, since there seems to be other use-cases for this, I agree it may be worth adding that option then. I'll see if I can come up with a patch for that.

Cheers,

Signature · 1 line
Luis
#12 lore raw

Re: [RFC PATCH 2/2] ceph: test basic ceph.quota.max_bytes quota

In reply to Yan, Zheng
Quoted 83 lines · Zheng
Quoted 76 lines · Dave Chinner
Quoted 47 lines · Luis Henriques
Quoted 45 lines · Dave Chinner
Quoted 26 lines · Luis Henriques
Quoted 25 lines · Nikolay Borisov
Quoted 23 lines · Luis Henriques
Quoted 7 lines · Dave Chinner

Makes no sense to me. xfs_io does a write() loop internally with this pwrite command of 4kB writes - the default buffer size. If you want xfs_io to loop doing 1MB sized pwrite() calls, then all you need is this:

	$XFS_IO_PROG -f -c "pwrite -w -B 1m 0 ${size}m" $file | _filter_xfs_io

Thank you for your review, Dave. I'll make sure the next revision of these tests will include all your comments implemented... except for this one.

The reason I'm using a loop for writing a file is due to the nature of the (very!) loose definition of quotas in CephFS. Basically, clients will likely write some amount of data over the configured limit because the servers they are communicating with to write the data (the OSDs) have no idea about the concept of quotas (or files even); the filesystem view in the cluster is managed at a different level, with the help of the MDS and the client itself.

So, the loop in this function is simply to allow the metadata associated with the file to be updated while we're writing the file. If I use a

Quoted 7 lines
Quoted 2 lines

But the metadata will be modified while writing the file even with a single invocation of xfs_io.

No, that's not true. It would be too expensive to keep the metadata server updated while writing to a file. So, making sure there's actually an open/close to the file (plus the fsync in pwrite) helps making sure the metadata is flushed into the MDS.

/me sighs.

So you want:

	loop until ${size}MB written:
		write 1MB
		fsync
		  -> flush data to server
		  -> flush metadata to server
Quoted 6 lines
Quoted 3 lines

i.e. this one liner:

xfs_io -f -c "pwrite -D -B 1m 0 ${size}m" /path/to/file

Unfortunately, that doesn't do what I want either :-/ (and I guess you meant '-b 1m', not '-B 1m', right?)

Yes. But I definitely did mean "-D" so that RWF_DSYNC was used with each 1MB write.

Quoted 9 lines

[ Zheng: please feel free to correct me if I'm saying something really
stupid below. ]

So, one of the key things in my loop is the open/close operations. When a file is closed in cephfs the capabilities (that's ceph jargon for what sort of operations a client is allowed to perform on an inode) will likely be released and that's when the metadata server will get the updated file size. Before that, the client is allowed to modify the file size if it has acquired the capabilities for doing so.

So you are saying that O_DSYNC writes on ceph do not force file size metadata changes to the metadata server to be made stable?

Quoted 4 lines

OTOH, a pwrite operation will eventually get the -EDQUOT even with the one-liner above because the client itself will realize it has exceeded a certain threshold set by the MDS and will eventually update the server with the new file size.

Quoted 4 lines

Sure, but if the client crashes without having sent the updated file size to the server as part of an extending O_DSYNC write, then how is it recovered when the client reconnects to the server and accesses the file again?

Quoted 3 lines

For DSYNC write, client has already written data to object store. If client crashes, MDS will set file to 'recovering' state and probe file size by checking object store. Accessing the file is blocked during recovery.

IOWs, ceph allows data integrity writes to the object store even though those writes breach quota limits on that object store? i.e. ceph quota essentially ignores O_SYNC/O_DSYNC metadata requirements?

FWIW, quotas normally have soft and hard limits - soft limits can be breached with a warning and a time limit to return under the soft limit, but the quota hard limit should /never/ be breached by users.

I guess that's the way of the world these days - fast and loose because everyone demands fast before correct....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#13 lore raw

Re: [RFC PATCH 2/2] ceph: test basic ceph.quota.max_bytes quota

Quoted 84 lines · Dave Chinner
Quoted 83 lines · Zheng
Quoted 80 lines · Dave Chinner
Quoted 47 lines · Luis Henriques
Quoted 45 lines · Dave Chinner
Quoted 26 lines · Luis Henriques
Quoted 25 lines · Nikolay Borisov
Quoted 23 lines · Luis Henriques
Quoted 7 lines · Dave Chinner

Makes no sense to me. xfs_io does a write() loop internally with this pwrite command of 4kB writes - the default buffer size. If you want xfs_io to loop doing 1MB sized pwrite() calls, then all you need is this:

	$XFS_IO_PROG -f -c "pwrite -w -B 1m 0 ${size}m" $file | _filter_xfs_io

Thank you for your review, Dave. I'll make sure the next revision of these tests will include all your comments implemented... except for this one.

The reason I'm using a loop for writing a file is due to the nature of the (very!) loose definition of quotas in CephFS. Basically, clients will likely write some amount of data over the configured limit because the servers they are communicating with to write the data (the OSDs) have no idea about the concept of quotas (or files even); the filesystem view in the cluster is managed at a different level, with the help of the MDS and the client itself.

So, the loop in this function is simply to allow the metadata associated with the file to be updated while we're writing the file. If I use a

Quoted 7 lines
Quoted 2 lines

But the metadata will be modified while writing the file even with a single invocation of xfs_io.

No, that's not true. It would be too expensive to keep the metadata server updated while writing to a file. So, making sure there's actually an open/close to the file (plus the fsync in pwrite) helps making sure the metadata is flushed into the MDS.

/me sighs.

So you want:

	loop until ${size}MB written:
		write 1MB
		fsync
		  -> flush data to server
		  -> flush metadata to server
Quoted 6 lines
Quoted 3 lines

i.e. this one liner:

xfs_io -f -c "pwrite -D -B 1m 0 ${size}m" /path/to/file

Unfortunately, that doesn't do what I want either :-/ (and I guess you meant '-b 1m', not '-B 1m', right?)

Yes. But I definitely did mean "-D" so that RWF_DSYNC was used with each 1MB write.

Quoted 9 lines

[ Zheng: please feel free to correct me if I'm saying something really
stupid below. ]

So, one of the key things in my loop is the open/close operations. When a file is closed in cephfs the capabilities (that's ceph jargon for what sort of operations a client is allowed to perform on an inode) will likely be released and that's when the metadata server will get the updated file size. Before that, the client is allowed to modify the file size if it has acquired the capabilities for doing so.

So you are saying that O_DSYNC writes on ceph do not force file size metadata changes to the metadata server to be made stable?

Quoted 4 lines

OTOH, a pwrite operation will eventually get the -EDQUOT even with the one-liner above because the client itself will realize it has exceeded a certain threshold set by the MDS and will eventually update the server with the new file size.

Sure, but if the client crashes without having sent the updated file size to the server as part of an extending O_DSYNC write, then how is it recovered when the client reconnects to the server and accesses the file again?

Quoted 8 lines
Quoted 3 lines

For DSYNC write, client has already written data to object store. If client crashes, MDS will set file to 'recovering' state and probe file size by checking object store. Accessing the file is blocked during recovery.

IOWs, ceph allows data integrity writes to the object store even though those writes breach limits on that object store? i.e. ceph quota essentially ignores O_SYNC/O_DSYNC metadata requirements?

Current cephfs quota implementation checks quota (compare i_size and quota setting) at very beginning of ceph_write_iter(). Nothing do with O_SYNC and O_DSYNC.

Regards
Yan, Zheng

Quoted 11 lines

FWIW, quotas normally have soft and hard limits - soft limits can be breached with a warning and a time limit to return under the soft limit, but the quota hard limit should /never/ be breached by users.

I guess that's the way of the world these days - fast and loose because everyone demands fast before correct....

Cheers,

Dave.

#14 lore raw

Re: [RFC PATCH 2/2] ceph: test basic ceph.quota.max_bytes quota

Quoted 9 lines · Zheng
Quoted 8 lines · Dave Chinner
Quoted 6 lines · Zheng
Quoted 2 lines · Dave Chinner
Quoted 1 line · Luis Henriques

Dave Chinner <[email protected]> writes:

For DSYNC write, client has already written data to object store. If client crashes, MDS will set file to 'recovering' state and probe file size by checking object store. Accessing the file is blocked during recovery.

Quoted 8 lines
Quoted 4 lines

IOWs, ceph allows data integrity writes to the object store even though those writes breach limits on that object store? i.e. ceph quota essentially ignores O_SYNC/O_DSYNC metadata requirements?

Current cephfs quota implementation checks quota (compare i_size and quota setting) at very beginning of ceph_write_iter(). Nothing do with O_SYNC and O_DSYNC.

Hold on, if the quota is checked on the client at the start of every write, then why is it not enforced /exactly/? Where does this "we didn't notice we'd run out of quota" overrun come from then?

i.e. the test changes are implying that quota is not accurately checked and enforced on every write, and that there is something less that exact about quotas on the ceph client. Yet you say they are checked on every write.

Where does the need to open/close files and force flushing client state to the MDS come from if quota is actually being checked on every write as you say it is?

i.e. I'm trying to work out if this change is just working around bugs in ceph quota accounting and I'm being told conflicting things about how the ceph client accounts and enforces quota limits. Can you please clearly explain how the quota enforcedment works and why close/open between writes is necessary for accurate quota enforcement so that we have some clue as to why these rubbery limit hacks are necessary?

If we don't understand why a test does something and it's not adequately documented, we can't really be expected to maintain it in working order....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#15 lore raw

Re: [RFC PATCH 2/2] ceph: test basic ceph.quota.max_bytes quota

Quoted 15 lines · Dave Chinner
Quoted 14 lines · Zheng
Quoted 12 lines · Dave Chinner
Quoted 6 lines · Zheng
Quoted 2 lines · Dave Chinner
Quoted 1 line · Luis Henriques

Dave Chinner <[email protected]> writes:

For DSYNC write, client has already written data to object store. If client crashes, MDS will set file to 'recovering' state and probe file size by checking object store. Accessing the file is blocked during recovery.

IOWs, ceph allows data integrity writes to the object store even though those writes breach limits on that object store? i.e. ceph quota essentially ignores O_SYNC/O_DSYNC metadata requirements?

Quoted 7 lines
Quoted 3 lines

Current cephfs quota implementation checks quota (compare i_size and quota setting) at very beginning of ceph_write_iter(). Nothing do with O_SYNC and O_DSYNC.

Hold on, if the quota is checked on the client at the start of every write, then why is it not enforced /exactly/? Where does this "we didn't notice we'd run out of quota" overrun come from then?

Ok, there's an extra piece of information that I don't think it was mentioned in the thread yet, which is the (recursive) directory statistics. These stats are maintained by the MDS and it's against these stats that the client actually checks if there's an overrun. The checks can be done against the amount of bytes ('max_bytes' quota) and/or number of files ('max_files' quota), depending on which quotas are enabled.

_Maybe_ there's space for some optimization on the client-side by locally updating the stats received from the MDS with local writes, files creation/deletion, etc. But I'm not sure it's worth the effort, because:

- for each operation (write, truncate, create, ...) we would also have the extra overhead of walking up the directory tree (or at least the quota realms tree) to update the stats for each directory; and

- we may have more clients writing to the same directories, and thus messing up with the dir quotas anyway. I.e. the quotas overrun problem would still be there, and we would still require to open/close files in the test.

Hopefully this helps clarifying why the open/close loop is a needed hack with the current quotas design.

Cheers,

Signature · 1 line
Luis
Quoted 25 lines

i.e. the test changes are implying that quota is not accurately checked and enforced on every write, and that there is something less that exact about quotas on the ceph client. Yet you say they are checked on every write.

Where does the need to open/close files and force flushing client state to the MDS come from if quota is actually being checked on every write as you say it is?

i.e. I'm trying to work out if this change is just working around bugs in ceph quota accounting and I'm being told conflicting things about how the ceph client accounts and enforces quota limits. Can you please clearly explain how the quota enforcedment works and why close/open between writes is necessary for accurate quota enforcement so that we have some clue as to why these rubbery limit hacks are necessary?

If we don't understand why a test does something and it's not adequately documented, we can't really be expected to maintain it in working order....

Cheers,

Dave.

#16 lore raw

Re: [RFC PATCH 2/2] ceph: test basic ceph.quota.max_bytes quota

Quoted 43 lines · Luis Henriques
Quoted 15 lines · Dave Chinner
Quoted 14 lines · Zheng
Quoted 12 lines · Dave Chinner
Quoted 6 lines · Zheng
Quoted 2 lines · Dave Chinner
Quoted 1 line · Luis Henriques

Dave Chinner <[email protected]> writes:

For DSYNC write, client has already written data to object store. If client crashes, MDS will set file to 'recovering' state and probe file size by checking object store. Accessing the file is blocked during recovery.

IOWs, ceph allows data integrity writes to the object store even though those writes breach limits on that object store? i.e. ceph quota essentially ignores O_SYNC/O_DSYNC metadata requirements?

Quoted 7 lines
Quoted 3 lines

Current cephfs quota implementation checks quota (compare i_size and quota setting) at very beginning of ceph_write_iter(). Nothing do with O_SYNC and O_DSYNC.

Hold on, if the quota is checked on the client at the start of every write, then why is it not enforced /exactly/? Where does this "we didn't notice we'd run out of quota" overrun come from then?

Ok, there's an extra piece of information that I don't think it was mentioned in the thread yet, which is the (recursive) directory statistics. These stats are maintained by the MDS and it's against these stats that the client actually checks if there's an overrun. The checks can be done against the amount of bytes ('max_bytes' quota) and/or number of files ('max_files' quota), depending on which quotas are enabled.

_Maybe_ there's space for some optimization on the client-side by locally updating the stats received from the MDS with local writes, files creation/deletion, etc. But I'm not sure it's worth the effort, because:

- for each operation (write, truncate, create, ...) we would also have the extra overhead of walking up the directory tree (or at least the quota realms tree) to update the stats for each directory; and

Quoted 7 lines

- we may have more clients writing to the same directories, and thus messing up with the dir quotas anyway. I.e. the quotas overrun problem would still be there, and we would still require to open/close files in the test.

Hopefully this helps clarifying why the open/close loop is a needed hack with the current quotas design.

More broadly, and without reference to any specific design, Ceph is a distributed storage system that allows multiple active clients to independently do activity against multiple servers, and which allows directory quotas.

Strict enforcement of quotas requires a single centralized authority to know the data within each directory and decide on every individual write operation if there is enough space to allow it; this is easy on a local filesystem but fundamentally cripples a multi-computer one since it implies serializing every IO on a single server. (Yes, there are LOTS of tricks you can pull so that "normally" it's not a problem and only impacts IO when you are actually approaching the quota limit; yes, there are ways to try and proactively reserve quota for a given writer; they all break down into the "contact a single server and serialize on it" when you actually approach the quota and nobody has yet expressed any interest in Ceph's quotas being that precise anyway.)

Quoted 30 lines

Cheers,
--
Luis

Quoted 25 lines

i.e. the test changes are implying that quota is not accurately checked and enforced on every write, and that there is something less that exact about quotas on the ceph client. Yet you say they are checked on every write.

Where does the need to open/close files and force flushing client state to the MDS come from if quota is actually being checked on every write as you say it is?

i.e. I'm trying to work out if this change is just working around bugs in ceph quota accounting and I'm being told conflicting things about how the ceph client accounts and enforces quota limits. Can you please clearly explain how the quota enforcedment works and why close/open between writes is necessary for accurate quota enforcement so that we have some clue as to why these rubbery limit hacks are necessary?

If we don't understand why a test does something and it's not adequately documented, we can't really be expected to maintain it in working order....

Cheers,

Dave.