Skip to contentSkip to conversation

btrfs: simple quotas fstests

PATCH Superseded Boris Burkov 3 participants 10 messages Sep 27, 2023 – Sep 28, 2023 Btrfs file system

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v3 cover btrfs: simple quotas fstests

Add a new test for specific squota scenarios (btrfs/301)

Also made modifications for passing existing qgroups tests when possible and for passing all tests with simple quota enabled via mkfs and with squota-aware btrfs check. Since this required reading sysfs files of scratch fses, did a bit of refactoring to make those checks target a device rather than assuming TEST_DEV.

btrfs/301 depends on the kernel patchset:
https://lore.kernel.org/linux-btrfs/[email protected]/
and the btrfs-progs patchset:
https://lore.kernel.org/linux-btrfs/[email protected]/
(and config appropriate binaries to use squota-aware versions)
---
Changelog:
v3:
- change btrfs/400 to btrfs/301
v2:
- new sysfs helpers in common
- better gating for the new squota test
- fix various formatting issues
- get rid of noisy dmesg logging

Boris Burkov (6):
common: refactor sysfs_attr functions
btrfs: quota mode helpers
btrfs/301: new test for simple quotas
btrfs: quota rescan helpers
btrfs: use new rescan wrapper
btrfs: skip squota incompatible tests

 common/btrfs        |  56 ++++++
 common/rc           | 127 ++++++++-----
 tests/btrfs/017     |   1 +
 tests/btrfs/022     |   1 +
 tests/btrfs/028     |   2 +-
 tests/btrfs/057     |   1 +
 tests/btrfs/091     |   3 +-
 tests/btrfs/104     |   2 +-
 tests/btrfs/123     |   2 +-
 tests/btrfs/126     |   2 +-
 tests/btrfs/139     |   2 +-
 tests/btrfs/153     |   2 +-
 tests/btrfs/171     |   6 +-
 tests/btrfs/179     |   2 +-
 tests/btrfs/180     |   2 +-
 tests/btrfs/190     |   2 +-
 tests/btrfs/193     |   2 +-
 tests/btrfs/210     |   2 +-
 tests/btrfs/224     |   6 +-
 tests/btrfs/230     |   2 +-
 tests/btrfs/232     |   2 +-
 tests/btrfs/301     | 423 ++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/301.out |   2 +
 23 files changed, 587 insertions(+), 65 deletions(-)
 create mode 100755 tests/btrfs/301
 create mode 100644 tests/btrfs/301.out
Signature · 1 line
2.42.0
#1 lore raw

PATCH v3 1/6 common: refactor sysfs_attr functions

Expand the has/get/require functions to allow passing a dev by parameter, and implement the test_dev specific one in terms of the new generic one.

---
 common/rc | 127 +++++++++++++++++++++++++++++++++++-------------------
 1 file changed, 82 insertions(+), 45 deletions(-)
common/rc +82 -45
index 1618ded54..c3cd53546 100644
@@ -4689,47 +4689,33 @@ run_fsx()
 	rm -f $tmp.fsx
 }
 
-# Test for the existence of a sysfs entry at /sys/fs/$FSTYP/DEV/$ATTR
+_require_statx()
+{
+	$here/src/stat_test --check-statx ||
+	_notrun "This test requires the statx system call"
+}
+
+# Get the path to the sysfs directory for the fs on a device
 #
 # Only one argument is needed:
-#  - attr: path name under /sys/fs/$FSTYP/DEV
+#  - dev: mounted block device for the fs
 #
 # Usage example:
-#   _require_fs_sysfs error/fail_at_unmount
-_has_fs_sysfs()
+#   _fs_sysfs_dname /dev/mapper/scratch-dev
+_fs_sysfs_dname()
 {
-	local attr=$1
-	local dname
+	local dev=$1
+
+	if [ ! -b "$dev" ]; then
+		_fail "Usage: _fs_sysfs_dname <mounted_device>"
+	fi
 
 	case "$FSTYP" in
 	btrfs)
-		dname=$(findmnt -n -o UUID $TEST_DEV) ;;
+		findmnt -n -o UUID ${dev} ;;
 	*)
-		dname=$(_short_dev $TEST_DEV) ;;
+		_short_dev $dev ;;
 	esac
-
-	if [ -z "$attr" -o -z "$dname" ];then
-		_fail "Usage: _require_fs_sysfs <sysfs_attr_path>"
-	fi
-
-	test -e /sys/fs/${FSTYP}/${dname}/${attr}
-}
-
-# Require the existence of a sysfs entry at /sys/fs/$FSTYP/DEV/$ATTR
-_require_fs_sysfs()
-{
-	_has_fs_sysfs "$@" && return
-
-	local attr=$1
-	local dname=$(_short_dev $TEST_DEV)
-
-	_notrun "This test requires /sys/fs/${FSTYP}/${dname}/${attr}"
-}
-
-_require_statx()
-{
-	$here/src/stat_test --check-statx ||
-	_notrun "This test requires the statx system call"
 }
 
 # Write "content" into /sys/fs/$FSTYP/$DEV/$ATTR
@@ -4753,13 +4739,7 @@ _set_fs_sysfs_attr()
 		_fail "Usage: _set_fs_sysfs_attr <mounted_device> <attr> <content>"
 	fi
 
-	local dname
-	case "$FSTYP" in
-	btrfs)
-		dname=$(findmnt -n -o UUID ${dev}) ;;
-	*)
-		dname=$(_short_dev $dev) ;;
-	esac
+	local dname=$(_fs_sysfs_dname $dev)
 
 	echo "$content" > /sys/fs/${FSTYP}/${dname}/${attr}
 }
@@ -4781,17 +4761,74 @@ _get_fs_sysfs_attr()
 		_fail "Usage: _get_fs_sysfs_attr <mounted_device> <attr>"
 	fi
 
-	local dname
-	case "$FSTYP" in
-	btrfs)
-		dname=$(findmnt -n -o UUID ${dev}) ;;
-	*)
-		dname=$(_short_dev $dev) ;;
-	esac
+	local dname=$(_fs_sysfs_dname $dev)
 
 	cat /sys/fs/${FSTYP}/${dname}/${attr}
 }
 
+# Test for the existence of a sysfs entry at /sys/fs/$FSTYP/$DEV/$ATTR
+#
+# All arguments are necessary, and in this order:
+#  - dev: device name, e.g. $SCRATCH_DEV
+#  - attr: path name under /sys/fs/$FSTYP/$dev
+#
+# Usage example:
+#   _has_fs_sysfs_attr /dev/mapper/scratch-dev error/fail_at_unmount
+_has_fs_sysfs_attr()
+{
+	local dev=$1
+	local attr=$2
+
+	if [ ! -b "$dev" -o -z "$attr" ];then
+		_fail "Usage: _get_fs_sysfs_attr <mounted_device> <attr>"
+	fi
+
+	local dname=$(_fs_sysfs_dname $dev)
+
+	test -e /sys/fs/${FSTYP}/${dname}/${attr}
+}
+
+# Require the existence of a sysfs entry at /sys/fs/$FSTYP/$DEV/$ATTR
+# All arguments are necessary, and in this order:
+#  - dev: device name, e.g. $SCRATCH_DEV
+#  - attr: path name under /sys/fs/$FSTYP/$dev
+#
+# Usage example:
+#   _require_fs_sysfs_attr /dev/mapper/scratch-dev error/fail_at_unmount
+_require_fs_sysfs_attr()
+{
+	_has_fs_sysfs_attr "$@" && return
+
+	local dev=$1
+	local attr=$2
+	local dname=$(_fs_sysfs_dname $dev)
+
+	_notrun "This test requires /sys/fs/${FSTYP}/${dname}/${attr}"
+}
+
+# Test for the existence of a sysfs entry at /sys/fs/$FSTYP/DEV/$ATTR
+#
+# Only one argument is needed:
+#  - attr: path name under /sys/fs/$FSTYP/DEV
+#
+# Usage example:
+#   _has_fs_sysfs error/fail_at_unmount
+_has_fs_sysfs()
+{
+	_has_fs_sysfs_attr $TEST_DEV "$@"
+}
+
+# Require the existence of a sysfs entry at /sys/fs/$FSTYP/DEV/$ATTR
+_require_fs_sysfs()
+{
+	_has_fs_sysfs "$@" && return
+
+	local attr=$1
+	local dname=$(_short_dev $TEST_DEV)
+
+	_notrun "This test requires /sys/fs/${FSTYP}/${dname}/${attr}"
+}
+
 # Generic test for specific filesystem feature.
 # Currently only implemented to test overlayfs features.
 _require_scratch_feature()
Signature · 1 line
2.42.0
#2 lore raw

PATCH v3 2/6 btrfs: quota mode helpers

In reply to Boris Burkov

To facilitate skipping tests depending on the qgroup mode after mkfs, add support for figuring out the mode. This cannot just rely on the new sysfs file, since it might not be present on older kernels.

---
 common/btrfs | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
common/btrfs +38 -0
index c9903a413..37796cc6e 100644
@@ -689,3 +689,41 @@ _require_btrfs_scratch_logical_resolve_v2()
 	fi
 	_scratch_unmount
 }
+
+_qgroup_mode()
+{
+	local dev=$1
+
+	if [ ! -b "$dev" ]; then
+		_fail "Usage: _qgroup_mode <mounted_device>"
+	fi
+
+	if _has_fs_sysfs_attr $dev /qgroups/mode; then
+		_get_fs_sysfs_attr $dev qgroups/mode
+	else
+		echo "disabled"
+	fi
+}
+
+_check_regular_qgroup()
+{
+	_qgroup_mode "$@" | grep -q 'qgroup'
+}
+
+_require_scratch_qgroup()
+{
+	_scratch_mkfs >>$seqres.full 2>&1
+	_scratch_mount
+	_run_btrfs_util_prog quota enable $SCRATCH_MNT
+	_check_regular_qgroup $SCRATCH_DEV || _notrun "not running normal qgroups"
+	_scratch_unmount
+}
+
+_require_scratch_enable_simple_quota()
+{
+    _scratch_mkfs >>$seqres.full 2>&1
+    _scratch_mount
+	_qgroup_mode $SCRATCH_DEV | grep 'squota' && _notrun "cannot enable simple quota; on by default"
+    $BTRFS_UTIL_PROG quota enable --simple $SCRATCH_MNT || _notrun "simple quotas not available"
+	_scratch_unmount
+}
Signature · 1 line
2.42.0
#3 lore raw

Re: [PATCH v3 2/6] btrfs: quota mode helpers

Quoted 47 lines · Boris Burkov

To facilitate skipping tests depending on the qgroup mode after mkfs, add support for figuring out the mode. This cannot just rely on the new sysfs file, since it might not be present on older kernels.

---
  common/btrfs | 38 ++++++++++++++++++++++++++++++++++++++
  1 file changed, 38 insertions(+)
common/btrfs +30 -0
index c9903a413..37796cc6e 100644
@@ -689,3 +689,41 @@ _require_btrfs_scratch_logical_resolve_v2()
  	fi
  	_scratch_unmount
  }
+
+_qgroup_mode()
+{
+	local dev=$1
+
+	if [ ! -b "$dev" ]; then
+		_fail "Usage: _qgroup_mode <mounted_device>"
+	fi
+
+	if _has_fs_sysfs_attr $dev /qgroups/mode; then
+		_get_fs_sysfs_attr $dev qgroups/mode
+	else
+		echo "disabled"
+	fi
+}
+
+_check_regular_qgroup()
+{
+	_qgroup_mode "$@" | grep -q 'qgroup'
+}
+
+_require_scratch_qgroup()
+{
+	_scratch_mkfs >>$seqres.full 2>&1
+	_scratch_mount
+	_run_btrfs_util_prog quota enable $SCRATCH_MNT
+	_check_regular_qgroup $SCRATCH_DEV || _notrun "not running normal qgroups"
+	_scratch_unmount
+}
+

common/btrfs _require_btrfs_scratch_logical_resolve_v2() · line 729

Quoted 10 lines
common/btrfs +8 -0
 
 
+_require_scratch_enable_simple_quota()
+{
+    _scratch_mkfs >>$seqres.full 2>&1
+    _scratch_mount
+	_qgroup_mode $SCRATCH_DEV | grep 'squota' && _notrun "cannot enable simple quota; on by default"
+    $BTRFS_UTIL_PROG quota enable --simple $SCRATCH_MNT || _notrun "simple quotas not available"
+	_scratch_unmount
+}

The rest of the patches look good so far. However, the indentation is mixed above; lines have a combination of spaces and tabs. Please help fix it if you are resending the whole series, or else I will fix it while merging.

Thanks, Anand

#4 lore raw

PATCH v3 3/6 btrfs/301: new test for simple quotas

In reply to Boris Burkov

Test some interesting basic and edge cases of simple quotas.

To some extent, this is redundant with the alternate testing strategy of using MKFS_OPTIONS to enable simple quotas, running the full suite and relying on kernel warnings and fsck to surface issues.

---
 tests/btrfs/301     | 418 ++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/301.out |   2 +
 2 files changed, 420 insertions(+)
 create mode 100755 tests/btrfs/301
 create mode 100644 tests/btrfs/301.out
tests/btrfs/301 new file +418 -0
new file mode 100755
index 000000000..527c25230
@@ -0,0 +1,418 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2023 Meta Platforms, Inc.  All Rights Reserved.
+#
+# FS QA Test 301
+#
+# Test common btrfs simple quotas scenarios involving sharing extents and
+# removing them in various orders.
+#
+. ./common/preamble
+_begin_fstest auto quick qgroup copy_range snapshot
+
+# Import common functions.
+# . ./common/filter
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_require_scratch
+_require_scratch_enable_simple_quota
+
+subv=$SCRATCH_MNT/subv
+nested=$SCRATCH_MNT/subv/nested
+snap=$SCRATCH_MNT/snap
+k=1024
+m=$(($k * $k))
+nr_fill=1024
+fill_sz=$((8 * $k))
+total_fill=$(($nr_fill * $fill_sz))
+eb_sz=$((16 * $k))
+ext_sz=$((128 * m))
+limit_nr=8
+limit=$(($ext_sz * $limit_nr))
+
+get_qgroup_usage()
+{
+	local qgroupid=$1
+
+	$BTRFS_UTIL_PROG qgroup show --sync --raw $SCRATCH_MNT | grep "$qgroupid" | $AWK_PROG '{print $3}'
+}
+
+get_subvol_usage()
+{
+	local subvolid=$1
+	get_qgroup_usage "0/$subvolid"
+}
+
+count_subvol_owned_metadata()
+{
+	local subvolid=$1
+	# find nodes and leaves owned by the subvol, then get unique offsets
+	# to account for snapshots sharing metadata.
+	count=$($BTRFS_UTIL_PROG inspect-internal dump-tree $SCRATCH_DEV | \
+		grep "owner $subvolid" | awk '{print $2}' | sort | uniq | wc -l)
+	# output bytes rather than number of metadata blocks
+	echo $(($count * $eb_sz))
+}
+
+check_qgroup_usage()
+{
+	local qgroupid=$1
+	local expected=$2
+	local actual=$(get_qgroup_usage $qgroupid)
+
+	[ $expected -eq $actual ] || _fail "qgroup $qgroupid mismatched usage $actual vs $expected"
+}
+
+check_subvol_usage()
+{
+	local subvolid=$1
+	local expected_data=$2
+	# need to sync to see updated usage numbers.
+	# could probably improve by placing syncs only where they are strictly
+	# needed after actual operations, but it is more error prone.
+	sync
+
+	local expected_meta=$(count_subvol_owned_metadata $subvolid)
+	local actual=$(get_subvol_usage $subvolid)
+	local expected=$(($expected_data + $expected_meta))
+
+	[ $expected -eq $actual ] || _fail "subvol $subvolid mismatched usage $actual vs $expected (expected data $expected_data expected meta $expected_meta diff $(($actual - $expected)))"
+	echo "OK $subvolid $expected_data $expected_meta $actual" >> $seqres.full
+}
+
+set_subvol_limit()
+{
+	local subvolid=$1
+	local limit=$2
+
+	$BTRFS_UTIL_PROG qgroup limit $2 0/$1 $SCRATCH_MNT
+}
+
+sync_check_subvol_usage()
+{
+	sync
+	check_subvol_usage $@
+}
+
+trigger_cleaner()
+{
+	$BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT
+	sleep 1
+	$BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT
+}
+
+cycle_mount_check_subvol_usage()
+{
+	_scratch_cycle_mount
+	check_subvol_usage $@
+}
+
+
+do_write()
+{
+	local file=$1
+	local sz=$2
+
+	$XFS_IO_PROG -fc "pwrite -q 0 $sz" $file
+	local ret=$?
+	return $ret
+}
+
+do_enospc_write()
+{
+	local file=$1
+	local sz=$2
+
+	do_write $file $sz 2>/dev/null && _fail "write expected enospc"
+}
+
+do_falloc()
+{
+	local file=$1
+	local sz=$2
+
+	$XFS_IO_PROG -fc "falloc 0 $sz" $file
+}
+
+do_enospc_falloc()
+{
+	local file=$1
+	local sz=$2
+
+	do_falloc $file $sz 2>/dev/null && _fail "falloc expected enospc"
+}
+
+enable_quota()
+{
+	local mode=$1
+
+	[ $mode == "n" ] && return
+	arg=$([ $mode == "s" ] && echo "--simple")
+
+	$BTRFS_UTIL_PROG quota enable $arg $SCRATCH_MNT
+}
+
+prepare()
+{
+	_scratch_mkfs >> $seqres.full
+	_scratch_mount
+	enable_quota "s"
+	$BTRFS_UTIL_PROG subvolume create $subv >> $seqres.full
+	set_subvol_limit 256 $limit
+	check_subvol_usage 256 0
+
+	# Create a bunch of little filler files to generate several levels in
+	# the btree, to make snapshotting sharing scenarios complex enough.
+	$FIO_PROG --name=filler --directory=$subv --rw=randwrite --nrfiles=$nr_fill --filesize=$fill_sz >/dev/null 2>&1
+	check_subvol_usage 256 $total_fill
+
+	# Create a single file whose extents we will explicitly share/unshare.
+	do_write $subv/f $ext_sz
+	check_subvol_usage 256 $(($total_fill + $ext_sz))
+}
+
+prepare_snapshotted()
+{
+	prepare
+	$BTRFS_UTIL_PROG subvolume snapshot $subv $snap >> $seqres.full
+	echo "snapshot" >> $seqres.full
+	check_subvol_usage 256 $(($total_fill + $ext_sz))
+	check_subvol_usage 257 0
+}
+
+prepare_nested()
+{
+	prepare
+	$BTRFS_UTIL_PROG qgroup create 1/100 $SCRATCH_MNT
+	$BTRFS_UTIL_PROG qgroup assign 0/256 1/100 $SCRATCH_MNT >> $seqres.full
+	$BTRFS_UTIL_PROG subvolume create $nested >> $seqres.full
+	do_write $nested/f $ext_sz
+	check_subvol_usage 257 $ext_sz
+	check_subvol_usage 256 $(($total_fill + $ext_sz))
+	local subv_usage=$(get_subvol_usage 256)
+	local nested_usage=$(get_subvol_usage 257)
+	check_qgroup_usage 1/100 $(($subv_usage + $nested_usage))
+}
+
+basic_accounting()
+{
+	prepare
+	echo "basic" >> $seqres.full
+	echo "delete file" >> $seqres.full
+	rm $subv/f
+	check_subvol_usage 256 $total_fill
+	cycle_mount_check_subvol_usage 256 $total_fill
+	do_write $subv/tmp 512M
+	rm $subv/tmp
+	do_write $subv/tmp 512M
+	rm $subv/tmp
+	do_enospc_falloc $subv/large_falloc 2G
+	do_enospc_write $subv/large 2G
+	_scratch_unmount
+}
+
+reservation_accounting()
+{
+	prepare
+	for i in $(seq 10); do
+		do_write $subv/tmp 512M
+		rm $subv/tmp
+	done
+	do_enospc_write $subv/large 2G
+	_scratch_unmount
+}
+
+snapshot_accounting()
+{
+	prepare_snapshotted
+	echo "unshare snapshot metadata" >> $seqres.full
+	touch $snap/f
+	check_subvol_usage 256 $(($total_fill + $ext_sz))
+	check_subvol_usage 257 0
+	echo "unshare snapshot data" >> $seqres.full
+	do_write $snap/f $ext_sz
+	check_subvol_usage 256 $(($total_fill + $ext_sz))
+	check_subvol_usage 257 $ext_sz
+	echo "delete snapshot file" >> $seqres.full
+	rm $snap/f
+	check_subvol_usage 256 $(($total_fill + $ext_sz))
+	check_subvol_usage 257 0
+	echo "delete original file" >> $seqres.full
+	rm $subv/f
+	check_subvol_usage 256 $total_fill
+	check_subvol_usage 257 0
+	cycle_mount_check_subvol_usage 256 $total_fill
+	check_subvol_usage 257 0
+	_scratch_unmount
+}
+
+delete_subvol_file()
+{
+	prepare_snapshotted
+	echo "delete subvol file first" >> $seqres.full
+	rm $subv/f
+	check_subvol_usage 256 $(($total_fill + $ext_sz))
+	check_subvol_usage 257 0
+	rm $snap/f
+	trigger_cleaner
+	check_subvol_usage 256 $total_fill
+	check_subvol_usage 257 0
+	cycle_mount_check_subvol_usage 256 $total_fill
+	check_subvol_usage 257 0
+	_scratch_unmount
+}
+
+delete_snapshot_file()
+{
+	prepare_snapshotted
+	echo "delete snapshot file first" >> $seqres.full
+	rm $snap/f
+	check_subvol_usage 256 $(($total_fill + $ext_sz))
+	check_subvol_usage 257 0
+	rm $subv/f
+	check_subvol_usage 256 $total_fill
+	check_subvol_usage 257 0
+	cycle_mount_check_subvol_usage 256 $total_fill
+	check_subvol_usage 257 0
+	_scratch_unmount
+}
+
+delete_subvol()
+{
+	echo "del sv" > /dev/ksmg
+	prepare_snapshotted
+	echo "delete subvol first" >> $seqres.full
+	$BTRFS_UTIL_PROG subvolume delete $subv >> $seqres.full
+	check_subvol_usage 256 $(($total_fill + $ext_sz))
+	check_subvol_usage 257 0
+	rm $snap/f
+	trigger_cleaner
+	check_subvol_usage 256 $total_fill
+	check_subvol_usage 257 0
+	$BTRFS_UTIL_PROG subvolume delete $snap >> $seqres.full
+	trigger_cleaner
+	check_subvol_usage 256 0
+	check_subvol_usage 257 0
+	cycle_mount_check_subvol_usage 256 0
+	check_subvol_usage 257 0
+	_scratch_unmount
+}
+
+delete_snapshot()
+{
+	echo "del snap" > /dev/ksmg
+	prepare_snapshotted
+	echo "delete snapshot first" >> $seqres.full
+	$BTRFS_UTIL_PROG subvolume delete $snap >> $seqres.full
+	check_subvol_usage 256 $(($total_fill + $ext_sz))
+	check_subvol_usage 257 0
+	$BTRFS_UTIL_PROG subvolume delete $subv >> $seqres.full
+	trigger_cleaner
+	check_subvol_usage 256 0
+	check_subvol_usage 257 0
+	_scratch_unmount
+}
+
+nested_accounting()
+{
+	echo "nested" > /dev/ksmg
+	prepare_nested
+	echo "nested" >> $seqres.full
+	echo "delete file" >> $seqres.full
+	rm $subv/f
+	check_subvol_usage 256 $total_fill
+	check_subvol_usage 257 $ext_sz
+	local subv_usage=$(get_subvol_usage 256)
+	local nested_usage=$(get_subvol_usage 257)
+	check_qgroup_usage 1/100 $(($subv_usage + $nested_usage))
+	rm $nested/f
+	check_subvol_usage 256 $total_fill
+	check_subvol_usage 257 0
+	subv_usage=$(get_subvol_usage 256)
+	nested_usage=$(get_subvol_usage 257)
+	check_qgroup_usage 1/100 $(($subv_usage + $nested_usage))
+	_scratch_unmount
+}
+
+enable_mature()
+{
+	echo "mature" > /dev/ksmg
+	_scratch_mkfs >> $seqres.full
+	_scratch_mount
+	$BTRFS_UTIL_PROG subvolume create $subv >> $seqres.full
+	do_write $subv/f $ext_sz
+	sync
+	enable_quota "s"
+	set_subvol_limit 256 $limit
+	_scratch_cycle_mount
+	usage=$(get_subvol_usage 256)
+	[ $usage -lt $ext_sz ] || _fail "captured usage from before enable $usage"
+	do_write $subv/g $ext_sz
+	usage=$(get_subvol_usage 256)
+	[ $usage -lt $ext_sz ] && "failed to capture usage after enable $usage"
+	check_subvol_usage 256 $ext_sz
+	rm $subv/f
+	check_subvol_usage 256 $ext_sz
+	_scratch_cycle_mount
+	rm $subv/g
+	check_subvol_usage 256 0
+	_scratch_unmount
+}
+
+reflink_accounting()
+{
+	prepare
+	# do more reflinks than would fit
+	for i in $(seq $(($limit_nr * 2))); do
+		cp --reflink=always $subv/f $subv/f.i
+	done
+	# no additional data usage from reflinks
+	check_subvol_usage 256 $(($total_fill + $ext_sz))
+	_scratch_unmount
+}
+
+delete_link()
+{
+	prepare
+	cp --reflink=always $subv/f $subv/f.link
+	check_subvol_usage 256 $(($total_fill + $ext_sz))
+	rm $subv/f.link
+	check_subvol_usage 256 $(($total_fill + $ext_sz))
+	rm $subv/f
+	check_subvol_usage 256 $(($total_fill))
+	_scratch_unmount
+}
+
+delete_linked()
+{
+	prepare
+	cp --reflink=always $subv/f $subv/f.link
+	check_subvol_usage 256 $(($total_fill + $ext_sz))
+	rm $subv/f
+	check_subvol_usage 256 $(($total_fill + $ext_sz))
+	rm $subv/f.link
+	check_subvol_usage 256 $(($total_fill))
+	_scratch_unmount
+}
+
+basic_accounting
+reservation_accounting
+snapshot_accounting
+delete_subvol_file
+delete_snapshot_file
+delete_subvol
+delete_snapshot
+nested_accounting
+enable_mature
+reflink_accounting
+delete_link
+delete_linked
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/btrfs/301.out new file +2 -0
new file mode 100644
index 000000000..4025504e4
@@ -0,0 +1,2 @@
+QA output created by 301
+Silence is golden
Signature · 1 line
2.42.0
#5 lore raw

Re: [PATCH v3 3/6] btrfs/301: new test for simple quotas

Quoted 22 lines · Boris Burkov

Test some interesting basic and edge cases of simple quotas.

To some extent, this is redundant with the alternate testing strategy of using MKFS_OPTIONS to enable simple quotas, running the full suite and relying on kernel warnings and fsck to surface issues.

---
 tests/btrfs/301     | 418 ++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/301.out |   2 +
 2 files changed, 420 insertions(+)
 create mode 100755 tests/btrfs/301
 create mode 100644 tests/btrfs/301.out
tests/btrfs/301 new file +1 -0
new file mode 100755
index 000000000..527c25230
@@ -0,0 +1,418 @@
+#! /bin/bash

tests/btrfs/301 line 11

Quoted 10 lines
tests/btrfs/301 new file +10 -0
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2023 Meta Platforms, Inc.  All Rights Reserved.
+#
+# FS QA Test 301
+#
+# Test common btrfs simple quotas scenarios involving sharing extents and
+# removing them in various orders.
+#
+. ./common/preamble
+_begin_fstest auto quick qgroup copy_range snapshot

Why the copy_range group? The test doesn't exercise copy_range.

It does exercise reflinking, so it should be in the "clone" group. Also the following groups are missing: subvol prealloc

tests/btrfs/301 line 14

Quoted 3 lines
tests/btrfs/301 new file +3 -0
+
+# Import common functions.
+# . ./common/filter

Here a:

. ./common/reflink

tests/btrfs/301 line 20

Quoted 6 lines
tests/btrfs/301 new file +6 -0
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_require_scratch

Should be: _require_scratch_reflink

tests/btrfs/301 line 21

Quoted 1 line
tests/btrfs/301 new file +1 -0
+_require_scratch_enable_simple_quota

Here missing:

_require_cp_reflink
_require_btrfs_command inspect-internal dump-tree
_require_xfs_io_command "falloc"

tests/btrfs/301 line 27

Quoted 6 lines
tests/btrfs/301 new file +6 -0
+
+subv=$SCRATCH_MNT/subv
+nested=$SCRATCH_MNT/subv/nested
+snap=$SCRATCH_MNT/snap
+k=1024
+m=$(($k * $k))

Do we really need to define variables with the size of a KiB and a MiB? Are there any programmers who don't know what those are? :)

tests/btrfs/301 line 29

Quoted 2 lines
tests/btrfs/301 new file +2 -0
+nr_fill=1024
+fill_sz=$((8 * $k))

Just doing ...=$((8 * 1024 ))
Everyone would be aware it's 8 KiB...

tests/btrfs/301 line 31

Quoted 2 lines
tests/btrfs/301 new file +2 -0
+total_fill=$(($nr_fill * $fill_sz))
+eb_sz=$((16 * $k))

Same here.

tests/btrfs/301 line 32

Quoted 1 line
tests/btrfs/301 new file +1 -0
+ext_sz=$((128 * m))

And here 128 * 1024 * 1024

Quoted 13 lines
tests/btrfs/301 new file +13 -0
+limit_nr=8
+limit=$(($ext_sz * $limit_nr))
+
+get_qgroup_usage()
+{
+       local qgroupid=$1
+
+       $BTRFS_UTIL_PROG qgroup show --sync --raw $SCRATCH_MNT | grep "$qgroupid" | $AWK_PROG '{print $3}'
+}
+
+get_subvol_usage()
+{
+       local subvolid=$1

tests/btrfs/301 line 55

Quoted 10 lines
tests/btrfs/301 new file +10 -0
+       get_qgroup_usage "0/$subvolid"
+}
+
+count_subvol_owned_metadata()
+{
+       local subvolid=$1
+       # find nodes and leaves owned by the subvol, then get unique offsets
+       # to account for snapshots sharing metadata.
+       count=$($BTRFS_UTIL_PROG inspect-internal dump-tree $SCRATCH_DEV | \
+               grep "owner $subvolid" | awk '{print $2}' | sort | uniq | wc -l)

awk -> $AWK_PROG

tests/btrfs/301 line 57

Quoted 2 lines
tests/btrfs/301 new file +2 -0
+       # output bytes rather than number of metadata blocks
+       echo $(($count * $eb_sz))

So $eb_sz is the extent buffer, leaf/node, size. This is expecting the default of 16K on x86 (4K pages systems).
Will the test fail if one runs with MKFS_OPTIONS="--nodesize 4K" for example, or some other different node size?

What about on a 64K pages system, where by default the node size is 64K? Will the test fail?

I would rather make the test flexible to run with any node size, or if not doable or impractical,
make it require a 4K page size and explicitly pass --nodesize 16K to mkfs. We have tests where we do this.

Quoted 8 lines
tests/btrfs/301 new file +8 -0
+}
+
+check_qgroup_usage()
+{
+       local qgroupid=$1
+       local expected=$2
+       local actual=$(get_qgroup_usage $qgroupid)
+

tests/btrfs/301 line 75

Quoted 10 lines
tests/btrfs/301 new file +10 -0
+       [ $expected -eq $actual ] || _fail "qgroup $qgroupid mismatched usage $actual vs $expected"
+}
+
+check_subvol_usage()
+{
+       local subvolid=$1
+       local expected_data=$2
+       # need to sync to see updated usage numbers.
+       # could probably improve by placing syncs only where they are strictly
+       # needed after actual operations, but it is more error prone.

Can we get proper sentences?
Some are ending with punctuation, others are not. First word should also be capitalized.
Same applies to everywhere else in the test.

tests/btrfs/301 line 76

Quoted 1 line
tests/btrfs/301 new file +1 -0
+       sync

I would rather move the sync to the helpers called below, and comment that it's because they use
the btrfs inspect-internal command, which reads from disk and therefore everything needs to be
flushed.

Quoted 12 lines
tests/btrfs/301 new file +12 -0
+
+       local expected_meta=$(count_subvol_owned_metadata $subvolid)
+       local actual=$(get_subvol_usage $subvolid)
+       local expected=$(($expected_data + $expected_meta))
+
+       [ $expected -eq $actual ] || _fail "subvol $subvolid mismatched usage $actual vs $expected (expected data $expected_data expected meta $expected_meta diff $(($actual - $expected)))"
+       echo "OK $subvolid $expected_data $expected_meta $actual" >> $seqres.full
+}
+
+set_subvol_limit()
+{
+       local subvolid=$1

tests/btrfs/301 line 98

Quoted 10 lines
tests/btrfs/301 new file +10 -0
+       local limit=$2
+
+       $BTRFS_UTIL_PROG qgroup limit $2 0/$1 $SCRATCH_MNT
+}
+
+sync_check_subvol_usage()
+{
+       sync
+       check_subvol_usage $@
+}

Does this helper make any sense? Because check_subvol_usage() is doing a sync call already.
And it's not called anywhere... so it can be deleted.

tests/btrfs/301 line 104

Quoted 6 lines
tests/btrfs/301 new file +6 -0
+
+trigger_cleaner()
+{
+       $BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT
+       sleep 1
+       $BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT

This may be too timing sensitive.
We have other tests that do something like this:

_scratch_remount commit=1
sleep 1.5

At the very least add comments explaining why that guarantees the cleaner is triggered and
it completes before the function returns, just like we have in btrfs/276 for example.

Quoted 7 lines
tests/btrfs/301 new file +7 -0
+}
+
+cycle_mount_check_subvol_usage()
+{
+       _scratch_cycle_mount
+       check_subvol_usage $@
+}

tests/btrfs/301 line 121

Quoted 10 lines
tests/btrfs/301 new file +10 -0
+
+
+do_write()
+{
+       local file=$1
+       local sz=$2
+
+       $XFS_IO_PROG -fc "pwrite -q 0 $sz" $file
+       local ret=$?
+       return $ret

Do we need this? Can't the function end just with the $XFS_IO_PROG call? Wouldn't that be the same?

tests/btrfs/301 line 129

Quoted 8 lines
tests/btrfs/301 new file +8 -0
+}
+
+do_enospc_write()
+{
+       local file=$1
+       local sz=$2
+
+       do_write $file $sz 2>/dev/null && _fail "write expected enospc"

So this pattern of calling _fail everywhere, is a bit against what is generally done in fstests.
Just don't redirect stderr - make it part of the golden output - this is even better because we can confirm the write failed with -ENOSPC and not some other error.

Quoted 6 lines
tests/btrfs/301 new file +6 -0
+}
+
+do_falloc()
+{
+       local file=$1
+       local sz=$2

tests/btrfs/301 line 145

Quoted 10 lines
tests/btrfs/301 new file +10 -0
+
+       $XFS_IO_PROG -fc "falloc 0 $sz" $file
+}
+
+do_enospc_falloc()
+{
+       local file=$1
+       local sz=$2
+
+       do_falloc $file $sz 2>/dev/null && _fail "falloc expected enospc"

Same here as before.

Quoted 14 lines
tests/btrfs/301 new file +14 -0
+}
+
+enable_quota()
+{
+       local mode=$1
+
+       [ $mode == "n" ] && return
+       arg=$([ $mode == "s" ] && echo "--simple")
+
+       $BTRFS_UTIL_PROG quota enable $arg $SCRATCH_MNT
+}
+
+prepare()
+{

tests/btrfs/301 line 169

Quoted 10 lines
tests/btrfs/301 new file +10 -0
+       _scratch_mkfs >> $seqres.full
+       _scratch_mount
+       enable_quota "s"
+       $BTRFS_UTIL_PROG subvolume create $subv >> $seqres.full
+       set_subvol_limit 256 $limit
+       check_subvol_usage 256 0
+
+       # Create a bunch of little filler files to generate several levels in
+       # the btree, to make snapshotting sharing scenarios complex enough.
+       $FIO_PROG --name=filler --directory=$subv --rw=randwrite --nrfiles=$nr_fill --filesize=$fill_sz >/dev/null 2>&1

Using fio, we need a _require_fio somewhere. See other tests like generic/299 for example.

Quoted 106 lines
tests/btrfs/301 new file +106 -0
+       check_subvol_usage 256 $total_fill
+
+       # Create a single file whose extents we will explicitly share/unshare.
+       do_write $subv/f $ext_sz
+       check_subvol_usage 256 $(($total_fill + $ext_sz))
+}
+
+prepare_snapshotted()
+{
+       prepare
+       $BTRFS_UTIL_PROG subvolume snapshot $subv $snap >> $seqres.full
+       echo "snapshot" >> $seqres.full
+       check_subvol_usage 256 $(($total_fill + $ext_sz))
+       check_subvol_usage 257 0
+}
+
+prepare_nested()
+{
+       prepare
+       $BTRFS_UTIL_PROG qgroup create 1/100 $SCRATCH_MNT
+       $BTRFS_UTIL_PROG qgroup assign 0/256 1/100 $SCRATCH_MNT >> $seqres.full
+       $BTRFS_UTIL_PROG subvolume create $nested >> $seqres.full
+       do_write $nested/f $ext_sz
+       check_subvol_usage 257 $ext_sz
+       check_subvol_usage 256 $(($total_fill + $ext_sz))
+       local subv_usage=$(get_subvol_usage 256)
+       local nested_usage=$(get_subvol_usage 257)
+       check_qgroup_usage 1/100 $(($subv_usage + $nested_usage))
+}
+
+basic_accounting()
+{
+       prepare
+       echo "basic" >> $seqres.full
+       echo "delete file" >> $seqres.full
+       rm $subv/f
+       check_subvol_usage 256 $total_fill
+       cycle_mount_check_subvol_usage 256 $total_fill
+       do_write $subv/tmp 512M
+       rm $subv/tmp
+       do_write $subv/tmp 512M
+       rm $subv/tmp
+       do_enospc_falloc $subv/large_falloc 2G
+       do_enospc_write $subv/large 2G
+       _scratch_unmount
+}
+
+reservation_accounting()
+{
+       prepare
+       for i in $(seq 10); do
+               do_write $subv/tmp 512M
+               rm $subv/tmp
+       done
+       do_enospc_write $subv/large 2G
+       _scratch_unmount
+}
+
+snapshot_accounting()
+{
+       prepare_snapshotted
+       echo "unshare snapshot metadata" >> $seqres.full
+       touch $snap/f
+       check_subvol_usage 256 $(($total_fill + $ext_sz))
+       check_subvol_usage 257 0
+       echo "unshare snapshot data" >> $seqres.full
+       do_write $snap/f $ext_sz
+       check_subvol_usage 256 $(($total_fill + $ext_sz))
+       check_subvol_usage 257 $ext_sz
+       echo "delete snapshot file" >> $seqres.full
+       rm $snap/f
+       check_subvol_usage 256 $(($total_fill + $ext_sz))
+       check_subvol_usage 257 0
+       echo "delete original file" >> $seqres.full
+       rm $subv/f
+       check_subvol_usage 256 $total_fill
+       check_subvol_usage 257 0
+       cycle_mount_check_subvol_usage 256 $total_fill
+       check_subvol_usage 257 0
+       _scratch_unmount
+}
+
+delete_subvol_file()
+{
+       prepare_snapshotted
+       echo "delete subvol file first" >> $seqres.full
+       rm $subv/f
+       check_subvol_usage 256 $(($total_fill + $ext_sz))
+       check_subvol_usage 257 0
+       rm $snap/f
+       trigger_cleaner
+       check_subvol_usage 256 $total_fill
+       check_subvol_usage 257 0
+       cycle_mount_check_subvol_usage 256 $total_fill
+       check_subvol_usage 257 0
+       _scratch_unmount
+}
+
+delete_snapshot_file()
+{
+       prepare_snapshotted
+       echo "delete snapshot file first" >> $seqres.full
+       rm $snap/f
+       check_subvol_usage 256 $(($total_fill + $ext_sz))
+       check_subvol_usage 257 0
+       rm $subv/f

tests/btrfs/301 line 285

Quoted 10 lines
tests/btrfs/301 new file +10 -0
+       check_subvol_usage 256 $total_fill
+       check_subvol_usage 257 0
+       cycle_mount_check_subvol_usage 256 $total_fill
+       check_subvol_usage 257 0
+       _scratch_unmount
+}
+
+delete_subvol()
+{
+       echo "del sv" > /dev/ksmg

Missing a >> $seqres.full here instead of > /dev/ksmg - which is also misspelled, should be kmsg.

Quoted 26 lines
tests/btrfs/301 new file +26 -0
+       prepare_snapshotted
+       echo "delete subvol first" >> $seqres.full
+       $BTRFS_UTIL_PROG subvolume delete $subv >> $seqres.full
+       check_subvol_usage 256 $(($total_fill + $ext_sz))
+       check_subvol_usage 257 0
+       rm $snap/f
+       trigger_cleaner
+       check_subvol_usage 256 $total_fill
+       check_subvol_usage 257 0
+       $BTRFS_UTIL_PROG subvolume delete $snap >> $seqres.full
+       trigger_cleaner
+       check_subvol_usage 256 0
+       check_subvol_usage 257 0
+       cycle_mount_check_subvol_usage 256 0
+       check_subvol_usage 257 0
+       _scratch_unmount
+}
+
+delete_snapshot()
+{
+       echo "del snap" > /dev/ksmg
+       prepare_snapshotted
+       echo "delete snapshot first" >> $seqres.full
+       $BTRFS_UTIL_PROG subvolume delete $snap >> $seqres.full
+       check_subvol_usage 256 $(($total_fill + $ext_sz))
+       check_subvol_usage 257 0

tests/btrfs/301 line 321

Quoted 10 lines
tests/btrfs/301 new file +10 -0
+       $BTRFS_UTIL_PROG subvolume delete $subv >> $seqres.full
+       trigger_cleaner
+       check_subvol_usage 256 0
+       check_subvol_usage 257 0
+       _scratch_unmount
+}
+
+nested_accounting()
+{
+       echo "nested" > /dev/ksmg

Same here.

Quoted 11 lines
tests/btrfs/301 new file +11 -0
+       prepare_nested
+       echo "nested" >> $seqres.full
+       echo "delete file" >> $seqres.full
+       rm $subv/f
+       check_subvol_usage 256 $total_fill
+       check_subvol_usage 257 $ext_sz
+       local subv_usage=$(get_subvol_usage 256)
+       local nested_usage=$(get_subvol_usage 257)
+       check_qgroup_usage 1/100 $(($subv_usage + $nested_usage))
+       rm $nested/f
+       check_subvol_usage 256 $total_fill

tests/btrfs/301 line 342

Quoted 10 lines
tests/btrfs/301 new file +10 -0
+       check_subvol_usage 257 0
+       subv_usage=$(get_subvol_usage 256)
+       nested_usage=$(get_subvol_usage 257)
+       check_qgroup_usage 1/100 $(($subv_usage + $nested_usage))
+       _scratch_unmount
+}
+
+enable_mature()
+{
+       echo "mature" > /dev/ksmg

Same here.

tests/btrfs/301 line 347

Quoted 5 lines
tests/btrfs/301 new file +5 -0
+       _scratch_mkfs >> $seqres.full
+       _scratch_mount
+       $BTRFS_UTIL_PROG subvolume create $subv >> $seqres.full
+       do_write $subv/f $ext_sz
+       sync

Why is the sync needed? At least add a comment.

tests/btrfs/301 line 348

Quoted 1 line
tests/btrfs/301 new file +1 -0
+       enable_quota "s"

If enabling quotas requires flushing all delalloc, the kernel should do it, no need for user space to call sync or anything equivalent. Would be odd to get incorrect quotas when enabling because there's pending delalloc.

Quoted 12 lines
tests/btrfs/301 new file +12 -0
+       set_subvol_limit 256 $limit
+       _scratch_cycle_mount
+       usage=$(get_subvol_usage 256)
+       [ $usage -lt $ext_sz ] || _fail "captured usage from before enable $usage"
+       do_write $subv/g $ext_sz
+       usage=$(get_subvol_usage 256)
+       [ $usage -lt $ext_sz ] && "failed to capture usage after enable $usage"
+       check_subvol_usage 256 $ext_sz
+       rm $subv/f
+       check_subvol_usage 256 $ext_sz
+       _scratch_cycle_mount
+       rm $subv/g

tests/btrfs/301 line 370

Quoted 10 lines
tests/btrfs/301 new file +10 -0
+       check_subvol_usage 256 0
+       _scratch_unmount
+}
+
+reflink_accounting()
+{
+       prepare
+       # do more reflinks than would fit
+       for i in $(seq $(($limit_nr * 2))); do
+               cp --reflink=always $subv/f $subv/f.i

Use the helper _cp_reflink instead.

tests/btrfs/301 line 380

Quoted 10 lines
tests/btrfs/301 new file +10 -0
+       done
+       # no additional data usage from reflinks
+       check_subvol_usage 256 $(($total_fill + $ext_sz))
+       _scratch_unmount
+}
+
+delete_link()
+{
+       prepare
+       cp --reflink=always $subv/f $subv/f.link

Same here.

tests/btrfs/301 line 392

Quoted 12 lines
tests/btrfs/301 new file +12 -0
+       check_subvol_usage 256 $(($total_fill + $ext_sz))
+       rm $subv/f.link
+       check_subvol_usage 256 $(($total_fill + $ext_sz))
+       rm $subv/f
+       check_subvol_usage 256 $(($total_fill))
+       _scratch_unmount
+}
+
+delete_linked()
+{
+       prepare
+       cp --reflink=always $subv/f $subv/f.link

Same here.

Thanks.

Quoted 37 lines
tests/btrfs/301 new file +26 -0
+       check_subvol_usage 256 $(($total_fill + $ext_sz))
+       rm $subv/f
+       check_subvol_usage 256 $(($total_fill + $ext_sz))
+       rm $subv/f.link
+       check_subvol_usage 256 $(($total_fill))
+       _scratch_unmount
+}
+
+basic_accounting
+reservation_accounting
+snapshot_accounting
+delete_subvol_file
+delete_snapshot_file
+delete_subvol
+delete_snapshot
+nested_accounting
+enable_mature
+reflink_accounting
+delete_link
+delete_linked
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/btrfs/301.out new file +2 -1
new file mode 100644
index 000000000..4025504e4
@@ -0,0 +1,2 @@
+QA output created by 301
+Silence is golden
--

2.42.0

#6 lore raw

PATCH v3 4/6 btrfs: quota rescan helpers

In reply to Boris Burkov

Many btrfs tests explicitly trigger quota rescan. This is not a meaningful operation for simple quotas, so we wrap it in a helper that doesn't blow up quite so badly and lets us run those tests where the rescan is a qgroup detail.

---
 common/btrfs | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
common/btrfs +18 -0
index 37796cc6e..0053fec48 100644
@@ -710,6 +710,24 @@ _check_regular_qgroup()
 	_qgroup_mode "$@" | grep -q 'qgroup'
 }
 
+_qgroup_rescan()
+{
+	local mnt=$1
+
+	_check_regular_qgroup $dev || return 1
+	_run_btrfs_util_prog quota rescan -w $mnt
+}
+
+_require_qgroup_rescan()
+{
+	_scratch_mkfs >>$seqres.full 2>&1
+	_scratch_mount
+	_run_btrfs_util_prog quota enable $SCRATCH_MNT
+	$BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT || \
+		_notrun "not able to run quota rescan"
+	_scratch_unmount
+}
+
 _require_scratch_qgroup()
 {
 	_scratch_mkfs >>$seqres.full 2>&1
Signature · 1 line
2.42.0
#7 lore raw

Re: [PATCH v3 4/6] btrfs: quota rescan helpers

Quoted 15 lines · Boris Burkov

Many btrfs tests explicitly trigger quota rescan. This is not a meaningful operation for simple quotas, so we wrap it in a helper that doesn't blow up quite so badly and lets us run those tests where the rescan is a qgroup detail.

---
  common/btrfs | 18 ++++++++++++++++++
  1 file changed, 18 insertions(+)
common/btrfs +0 -0
index 37796cc6e..0053fec48 100644

common/btrfs _check_regular_qgroup() · line 717

Quoted 8 lines
common/btrfs +5 -0
@@ -710,6 +710,24 @@ _check_regular_qgroup()
  	_qgroup_mode "$@" | grep -q 'qgroup'
  }
 
+_qgroup_rescan()
+{
+	local mnt=$1
+
+	_check_regular_qgroup $dev || return 1

$dev is not initialized.

Test case such as btrfs/028 fails with MKFS_OPTIONS="-O squota" config option.

btrfs/028 30s ... [failed, exit status 1]- output mismatch (see /xfstests-dev/results//btrfs/028.out.bad)

    --- tests/btrfs/028.out	2023-02-20 12:32:31.399005973 +0800
    +++ /xfstests-dev/results//btrfs/028.out.bad	2023-09-28 

20:54:52.744848575 +0800
@@ -1,2 +1,3 @@
QA output created by 028
-Silence is golden
+failed: '/usr/local/bin/btrfs quota rescan -w /mnt/scratch' +(see /xfstests-dev/results//btrfs/028.full for details)
...
(Run 'diff -u /xfstests-dev/tests/btrfs/028.out
/xfstests-dev/results//btrfs/028.out.bad' to see the entire diff)

Thanks, Anand

Quoted 16 lines
common/btrfs +13 -0
+	_run_btrfs_util_prog quota rescan -w $mnt
+}
+
+_require_qgroup_rescan()
+{
+	_scratch_mkfs >>$seqres.full 2>&1
+	_scratch_mount
+	_run_btrfs_util_prog quota enable $SCRATCH_MNT
+	$BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT || \
+		_notrun "not able to run quota rescan"
+	_scratch_unmount
+}
+
  _require_scratch_qgroup()
  {
  	_scratch_mkfs >>$seqres.full 2>&1
#8 lore raw

PATCH v3 5/6 btrfs: use new rescan wrapper

In reply to Boris Burkov

These tests can pass in simple quota mode if we skip the rescans via the wrapper.

---
 tests/btrfs/022 | 1 +
 tests/btrfs/028 | 2 +-
 tests/btrfs/104 | 2 +-
 tests/btrfs/123 | 2 +-
 tests/btrfs/126 | 2 +-
 tests/btrfs/139 | 2 +-
 tests/btrfs/153 | 2 +-
 tests/btrfs/171 | 6 +++---
 tests/btrfs/179 | 2 +-
 tests/btrfs/180 | 2 +-
 tests/btrfs/190 | 2 +-
 tests/btrfs/193 | 2 +-
 tests/btrfs/210 | 2 +-
 tests/btrfs/224 | 6 +++---
 tests/btrfs/230 | 2 +-
 tests/btrfs/232 | 2 +-
 16 files changed, 20 insertions(+), 19 deletions(-)
tests/btrfs/022 +1 -0
index e2d37b094..b1ef2fdf7 100755
@@ -14,6 +14,7 @@ _begin_fstest auto qgroup limit
 
 _supported_fs btrfs
 _require_scratch
+_require_qgroup_rescan
 _require_btrfs_qgroup_report
 
 # Test to make sure we can actually turn it on and it makes sense
tests/btrfs/028 +1 -1
index fe0678f86..c4080c950 100755
@@ -25,7 +25,7 @@ _scratch_mkfs >/dev/null
 _scratch_mount
 
 _run_btrfs_util_prog quota enable $SCRATCH_MNT
-_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT
+_qgroup_rescan $SCRATCH_MNT
 
 # Increase the probability of generating de-refer extent, and decrease
 # other.
tests/btrfs/104 +1 -1
index 499de6bfb..c9528eb13 100755
@@ -94,7 +94,7 @@ _explode_fs_tree 1 $SCRATCH_MNT/snap2/files-snap2
 # Enable qgroups now that we have our filesystem prepared. This
 # will kick off a scan which we will have to wait for.
 _run_btrfs_util_prog quota enable $SCRATCH_MNT
-_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT
+_qgroup_rescan $SCRATCH_MNT
 
 # Remount to clear cache, force everything to disk
 _scratch_cycle_mount
tests/btrfs/123 +1 -1
index c179eeec9..4c5b7e116 100755
@@ -39,7 +39,7 @@ sync
 
 # enable quota and rescan to get correct number
 _run_btrfs_util_prog quota enable $SCRATCH_MNT
-_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT
+_qgroup_rescan $SCRATCH_MNT
 
 # now balance data block groups to corrupt qgroup
 _run_btrfs_balance_start -d $SCRATCH_MNT >> $seqres.full
tests/btrfs/126 +1 -1
index 2b0edb65b..1bb24e00f 100755
@@ -28,7 +28,7 @@ _scratch_mkfs >/dev/null
 _scratch_mount "-o enospc_debug"
 
 _run_btrfs_util_prog quota enable $SCRATCH_MNT
-_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT
+_qgroup_rescan $SCRATCH_MNT
 _run_btrfs_util_prog qgroup limit 512K 0/5 $SCRATCH_MNT
 
 # The amount of written data may change due to different nodesize at mkfs time,
tests/btrfs/139 +1 -1
index c4b09f9fc..f3d92ba46 100755
@@ -30,7 +30,7 @@ SUBVOL=$SCRATCH_MNT/subvol
 
 _run_btrfs_util_prog subvolume create $SUBVOL
 _run_btrfs_util_prog quota enable $SCRATCH_MNT
-_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT
+_qgroup_rescan $SCRATCH_MNT
 _run_btrfs_util_prog qgroup limit -e 1G $SUBVOL
 
 # Write and delete files within 1G limits, multiple times
tests/btrfs/153 +1 -1
index 99fab1018..4a5fe2b8c 100755
@@ -24,7 +24,7 @@ _scratch_mkfs >/dev/null
 _scratch_mount
 
 _run_btrfs_util_prog quota enable $SCRATCH_MNT
-_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT
+_qgroup_rescan $SCRATCH_MNT
 _run_btrfs_util_prog qgroup limit 100M 0/5 $SCRATCH_MNT
 
 testfile1=$SCRATCH_MNT/testfile1
tests/btrfs/171 +3 -3
index 461cdd0fa..6a1ed1c1a 100755
@@ -35,7 +35,7 @@ $BTRFS_UTIL_PROG subvolume snapshot "$SCRATCH_MNT/subvol" \
 	"$SCRATCH_MNT/snapshot" > /dev/null
 
 $BTRFS_UTIL_PROG quota enable "$SCRATCH_MNT" > /dev/null
-$BTRFS_UTIL_PROG quota rescan -w "$SCRATCH_MNT" > /dev/null
+_qgroup_rescan $SCRATCH_MNT > /dev/null
 
 # Create high level qgroup
 $BTRFS_UTIL_PROG qgroup create 1/0 "$SCRATCH_MNT"
@@ -45,7 +45,7 @@ $BTRFS_UTIL_PROG qgroup assign "$SCRATCH_MNT/snapshot" 1/0 "$SCRATCH_MNT" \
 
 # Above assignment will mark qgroup inconsistent due to the shared extents
 # between subvol/snapshot/high level qgroup, do rescan here.
-$BTRFS_UTIL_PROG quota rescan -w "$SCRATCH_MNT" > /dev/null
+_qgroup_rescan $SCRATCH_MNT > /dev/null
 
 # Now remove the qgroup relationship and make 1/0 childless
 # Due to the shared extent outside of 1/0, we will mark qgroup inconsistent
@@ -54,7 +54,7 @@ $BTRFS_UTIL_PROG qgroup remove "$SCRATCH_MNT/snapshot" 1/0 "$SCRATCH_MNT" \
 	2>&1 | _filter_btrfs_qgroup_assign_warnings
 
 # Above removal also marks qgroup inconsistent, rescan again
-$BTRFS_UTIL_PROG quota rescan -w "$SCRATCH_MNT" > /dev/null
+_qgroup_rescan $SCRATCH_MNT > /dev/null
 
 # After the test, btrfs check will verify qgroup numbers to catch any
 # corruption.
tests/btrfs/179 +1 -1
index 479667f05..00013af0a 100755
@@ -33,7 +33,7 @@ _scratch_mount
 mkdir -p "$SCRATCH_MNT/snapshots"
 $BTRFS_UTIL_PROG subvolume create "$SCRATCH_MNT/src" > /dev/null
 $BTRFS_UTIL_PROG quota enable "$SCRATCH_MNT" > /dev/null
-$BTRFS_UTIL_PROG quota rescan -w "$SCRATCH_MNT" > /dev/null
+_qgroup_rescan "$SCRATCH_MNT" > /dev/null
 
 fill_workload()
 {
tests/btrfs/180 +1 -1
index b7c8dac96..aa195f7b4 100755
@@ -27,7 +27,7 @@ _scratch_mkfs > /dev/null
 _scratch_mount
 
 $BTRFS_UTIL_PROG quota enable "$SCRATCH_MNT" > /dev/null
-$BTRFS_UTIL_PROG quota rescan -w "$SCRATCH_MNT" > /dev/null
+_qgroup_rescan "$SCRATCH_MNT" > /dev/null
 $BTRFS_UTIL_PROG qgroup limit -e 1G "$SCRATCH_MNT"
 
 $XFS_IO_PROG -f -c "falloc 0 900M" "$SCRATCH_MNT/padding"
tests/btrfs/190 +1 -1
index 974438c15..f78c14fe4 100755
@@ -30,7 +30,7 @@ _log_writes_mkfs >> $seqres.full 2>&1
 
 _log_writes_mount
 $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT >> $seqres.full
-$BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT >> $seqres.full
+_qgroup_rescan $SCRATCH_MNT >> $seqres.full
 
 # Create enough metadata for later balance
 for ((i = 0; i < $nr_files; i++)); do
tests/btrfs/193 +1 -1
index b4632ab0a..67220c7a5 100755
@@ -26,7 +26,7 @@ _scratch_mkfs > /dev/null
 _scratch_mount
 
 $BTRFS_UTIL_PROG quota enable "$SCRATCH_MNT" > /dev/null
-$BTRFS_UTIL_PROG quota rescan -w "$SCRATCH_MNT" > /dev/null
+_qgroup_rescan "$SCRATCH_MNT" > /dev/null
 $BTRFS_UTIL_PROG qgroup limit -e 256M "$SCRATCH_MNT"
 
 # Create a file with the following layout:
tests/btrfs/210 +1 -1
index 383a307ff..f3d769fa0 100755
@@ -29,7 +29,7 @@ _pwrite_byte 0xcd 0 16M "$SCRATCH_MNT/src/file" > /dev/null
 # by qgroup
 sync
 $BTRFS_UTIL_PROG quota enable "$SCRATCH_MNT"
-$BTRFS_UTIL_PROG quota rescan -w "$SCRATCH_MNT" > /dev/null
+_qgroup_rescan "$SCRATCH_MNT" > /dev/null
 $BTRFS_UTIL_PROG qgroup create 1/0 "$SCRATCH_MNT"
 
 # Create a snapshot with qgroup inherit
tests/btrfs/224 +3 -3
index d7ec57360..de10942fe 100755
@@ -30,7 +30,7 @@ assign_shared_test()
 
 	echo "=== qgroup assign shared test ===" >> $seqres.full
 	$BTRFS_UTIL_PROG quota enable $SCRATCH_MNT
-	$BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT >> $seqres.full
+	_qgroup_rescan $SCRATCH_MNT >> $seqres.full
 
 	$BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/a >> $seqres.full
 	$BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/b >> $seqres.full
@@ -54,7 +54,7 @@ assign_no_shared_test()
 
 	echo "=== qgroup assign no shared test ===" >> $seqres.full
 	$BTRFS_UTIL_PROG quota enable $SCRATCH_MNT
-	$BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT >> $seqres.full
+	_qgroup_rescan $SCRATCH_MNT >> $seqres.full
 
 	$BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/a >> $seqres.full
 	$BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/b >> $seqres.full
@@ -75,7 +75,7 @@ snapshot_test()
 
 	echo "=== qgroup snapshot test ===" >> $seqres.full
 	$BTRFS_UTIL_PROG quota enable $SCRATCH_MNT
-	$BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT >> $seqres.full
+	_qgroup_rescan $SCRATCH_MNT >> $seqres.full
 
 	$BTRFS_UTIL_PROG subvolume create $SCRATCH_MNT/a >> $seqres.full
 	_ddt of="$SCRATCH_MNT"/a/file1 bs=1M count=1 >> $seqres.full 2>&1
tests/btrfs/230 +1 -1
index 46b0c6369..7a4cd18e9 100755
@@ -31,7 +31,7 @@ _pwrite_byte 0xcd 0 1G $SCRATCH_MNT/file >> $seqres.full
 sync
 
 $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT
-$BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT >> $seqres.full
+_qgroup_rescan $SCRATCH_MNT >> $seqres.full
 
 # Set the limit to just 512MiB, which is way below the existing usage
 $BTRFS_UTIL_PROG qgroup limit  512M 0/5 $SCRATCH_MNT
tests/btrfs/232 +1 -1
index 02c7e49de..debe070bb 100755
@@ -46,7 +46,7 @@ _pwrite_byte 0xcd 0 900m $SCRATCH_MNT/file >> $seqres.full
 sync
 
 $BTRFS_UTIL_PROG quota enable $SCRATCH_MNT
-$BTRFS_UTIL_PROG quota rescan -w $SCRATCH_MNT >> $seqres.full
+_qgroup_rescan $SCRATCH_MNT >> $seqres.full
 # set the limit to 1 g, leaving us just 100mb of slack space
 $BTRFS_UTIL_PROG qgroup limit 1G 0/5 $SCRATCH_MNT
 
Signature · 1 line
2.42.0
#9 lore raw

PATCH v3 6/6 btrfs: skip squota incompatible tests

In reply to Boris Burkov

These tests cannot succeed if mkfs enable squotas, as they either test the specifics of qgroups behavior or they test *enabling* squotas. Skip these in squota mode.

---
 tests/btrfs/017 | 1 +
 tests/btrfs/057 | 1 +
 tests/btrfs/091 | 3 ++-
 tests/btrfs/301 | 5 +++++
 4 files changed, 9 insertions(+), 1 deletion(-)
tests/btrfs/017 +1 -0
index 622071018..496cc7df1 100755
@@ -22,6 +22,7 @@ _begin_fstest auto quick qgroup
 
 _supported_fs btrfs
 _require_scratch
+_require_scratch_qgroup
 _require_cloner
 
 # Currently in btrfs the node/leaf size can not be smaller than the page
tests/btrfs/057 +1 -0
index 782d854a0..e932a6572 100755
@@ -15,6 +15,7 @@ _begin_fstest auto quick
 # real QA test starts here
 _supported_fs btrfs
 _require_scratch
+_require_qgroup_rescan
 
 _scratch_mkfs_sized $((1024 * 1024 * 1024)) >> $seqres.full 2>&1
 
tests/btrfs/091 +2 -1
index f2cd00b2e..a71e03406 100755
@@ -19,6 +19,7 @@ _begin_fstest auto quick qgroup
 _supported_fs btrfs
 _require_scratch
 _require_cp_reflink
+_require_scratch_qgroup
 
 # use largest node/leaf size (64K) to allow the test to be run on arch with
 # page size > 4k.
@@ -35,7 +36,7 @@ _run_btrfs_util_prog subvolume create $SCRATCH_MNT/subv2
 _run_btrfs_util_prog subvolume create $SCRATCH_MNT/subv3
 
 _run_btrfs_util_prog quota enable $SCRATCH_MNT
-_run_btrfs_util_prog quota rescan -w $SCRATCH_MNT
+_qgroup_rescan $SCRATCH_MNT
 
 $XFS_IO_PROG -f -c "pwrite 0 256K" $SCRATCH_MNT/subv1/file1 | _filter_xfs_io
 cp --reflink $SCRATCH_MNT/subv1/file1 $SCRATCH_MNT/subv2/file1
tests/btrfs/301 +5 -0
index 527c25230..f6b55c83f 100755
@@ -33,6 +33,11 @@ ext_sz=$((128 * m))
 limit_nr=8
 limit=$(($ext_sz * $limit_nr))
 
+_scratch_mkfs >> $seqres.full
+_scratch_mount
+_qgroup_mode $SCRATCH_MNT | grep 'squota' && _notrun "test relies on starting without simple quotas enabled"
+_scratch_unmount
+
 get_qgroup_usage()
 {
 	local qgroupid=$1
Signature · 1 line
2.42.0