Skip to contentSkip to conversation

fstests: catch us up to xfsprogs 6.10

PATCHBOMB v2024.08.25 New Darrick J. Wong 2 participants 20 messages Aug 27, 2024 – Aug 30, 2024

lore
Share & tools
lore
#0 lore raw

Hi Zorro,

Now that Carlos has released xfsprogs 6.10, here's some new functional tests for the improvements that landed this cycle. This includes the new realtime volume FITRIM support, filesystem properties, deceptive name detection in xfs_scrub, and the (hopefully final) structure of the background xfs_scrub systemd services.

--D

#1 lore raw

PATCH 1/5 fstests: detect deceptive filename extensions

Hi all,

In early 2023, malware researchers disclosed a phishing attack that was targeted at people running Linux workstations. The attack vector involved the use of filenames containing what looked like a file extension but instead contained a lookalike for the full stop (".") and a common extension ("pdf"). Enhance xfs_scrub phase 5 to detect these types of attacks and warn the system administrator. Add functional testing for this code.

If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.

This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.

--D

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-detect-deceptive-extensions

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=scrub-detect-deceptive-extensions
---
Commits in this patchset:
* generic/453: test confusable name detection with 32-bit unicode codepoints
* generic/453: check xfs_scrub detection of confusing job offers

---
 tests/generic/453 |  111 +++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 99 insertions(+), 12 deletions(-)
#2 lore raw

PATCH 1/2 generic/453: test confusable name detection with 32-bit unicode codepoints

From: Darrick J. Wong <[email protected]>

Test the confusable name detection when there are 32-bit unicode sequences in use. In other words, emoji. Change the xfs_scrub test to dump the output to a file instead of passing huge echo commands around.

---
 tests/generic/453 |   32 +++++++++++++++++++++-----------
 1 file changed, 21 insertions(+), 11 deletions(-)
tests/generic/453 +21 -11
index a0fb802e9b..930e6408ff 100755
@@ -148,6 +148,10 @@ setf "combmark_\xe1\x80\x9c\xe1\x80\xaf\xe1\x80\xad.txt" "combining marks"
 setd ".\xe2\x80\x8d" "zero width joiners in dot entry"
 setd "..\xe2\x80\x8d" "zero width joiners in dotdot entry"
 
+# utf8 sequence mapping to a u32 unicode codepoint that can be confused
+setf "toilet_bowl.\xf0\x9f\x9a\xbd" "toilet emoji"
+setf "toilet_bow\xe2\x80\x8dl.\xf0\x9f\x9a\xbd" "toilet emoji with zero width joiner"
+
 ls -la $testdir >> $seqres.full
 
 echo "Test files"
@@ -198,6 +202,9 @@ testf "combmark_\xe1\x80\x9c\xe1\x80\xaf\xe1\x80\xad.txt" "combining marks"
 testd ".\xe2\x80\x8d" "zero width joiners in dot entry"
 testd "..\xe2\x80\x8d" "zero width joiners in dotdot entry"
 
+testf "toilet_bowl.\xf0\x9f\x9a\xbd" "toilet emoji"
+testf "toilet_bow\xe2\x80\x8dl.\xf0\x9f\x9a\xbd" "toilet emoji with zero width joiner"
+
 echo "Uniqueness of inodes?"
 stat -c '%i' "${testdir}/"* | sort | uniq -c | while read nr inum; do
 	if [ "${nr}" -gt 1 ]; then
@@ -208,18 +215,21 @@ done
 echo "Test XFS online scrub, if applicable"
 
 if _check_xfs_scrub_does_unicode "$SCRATCH_MNT" "$SCRATCH_DEV"; then
-	output="$(LC_ALL="C.UTF-8" ${XFS_SCRUB_PROG} -v -n "${SCRATCH_MNT}" 2>&1 | filter_scrub)"
-	echo "${output}" | grep -q "french_" || echo "No complaints about french e accent?"
-	echo "${output}" | grep -q "greek_" || echo "No complaints about greek letter mess?"
-	echo "${output}" | grep -q "arabic_" || echo "No complaints about arabic expanded string?"
-	echo "${output}" | grep -q "mixed_" || echo "No complaints about mixed script confusables?"
-	echo "${output}" | grep -q "hyphens_" || echo "No complaints about hyphenation confusables?"
-	echo "${output}" | grep -q "dz_digraph_" || echo "No complaints about single script confusables?"
-	echo "${output}" | grep -q "inadequate_" || echo "No complaints about inadequate rendering confusables?"
-	echo "${output}" | grep -q "prohibition_" || echo "No complaints about prohibited sequence confusables?"
-	echo "${output}" | grep -q "zerojoin_" || echo "No complaints about zero-width join confusables?"
+	LC_ALL="C.UTF-8" ${XFS_SCRUB_PROG} -v -n "${SCRATCH_MNT}" 2>&1 | filter_scrub > $tmp.scrub
+
+	grep -q "french_" $tmp.scrub || echo "No complaints about french e accent?"
+	grep -q "greek_" $tmp.scrub || echo "No complaints about greek letter mess?"
+	grep -q "arabic_" $tmp.scrub || echo "No complaints about arabic expanded string?"
+	grep -q "mixed_" $tmp.scrub || echo "No complaints about mixed script confusables?"
+	grep -q "hyphens_" $tmp.scrub || echo "No complaints about hyphenation confusables?"
+	grep -q "dz_digraph_" $tmp.scrub || echo "No complaints about single script confusables?"
+	grep -q "inadequate_" $tmp.scrub || echo "No complaints about inadequate rendering confusables?"
+	grep -q "prohibition_" $tmp.scrub || echo "No complaints about prohibited sequence confusables?"
+	grep -q "zerojoin_" $tmp.scrub || echo "No complaints about zero-width join confusables?"
+	grep -q "toilet_" $tmp.scrub || echo "No complaints about zero-width join confusables with emoji?"
+
 	echo "Actual xfs_scrub output:" >> $seqres.full
-	echo "${output}" >> $seqres.full
+	cat $tmp.scrub >> $seqres.full
 fi
 
 # success, all done
#4 lore raw

PATCH 2/2 generic/453: check xfs_scrub detection of confusing job offers

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Earlier this year, ESET revealed that Linux users had been tricked into opening executables containing malware payloads. The trickery came in the form of a malicious zip file containing a filename with the string "job offer․pdf". Note that the filename does *not* denote a real pdf file, since the last four codepoints in the file name are "ONE DOT LEADER", p, d, and f. Not period (ok, FULL STOP), p, d, f like you'd normally expect.

Now that xfs_scrub can look for codepoints that could be confused with a period followed by alphanumerics, let's make sure it actually works.

---
 tests/generic/453 |   79 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 78 insertions(+), 1 deletion(-)
tests/generic/453 +78 -1
index 930e6408ff..855243a860 100755
@@ -36,6 +36,15 @@ setf() {
 	echo "Storing ${key} ($(hexbytes "${key}")) -> ${value}" >> $seqres.full
 }
 
+setchild() {
+	subdir="$1"
+	key="$(echo -e "$2")"
+
+	mkdir -p "${testdir}/${subdir}"
+	echo "$subdir" > "${testdir}/${subdir}/${key}"
+	echo "Storing ${subdir}/${key} ($(hexbytes "${key}")) -> ${subdir}" >> $seqres.full
+}
+
 setd() {
 	key="$(echo -e "$1")"
 	value="$2"
@@ -63,6 +72,24 @@ testf() {
 	fi
 }
 
+testchild() {
+	subdir="$1"
+	key="$(echo -e "$2")"
+	fname="${testdir}/${subdir}/${key}"
+
+	echo "Testing ${subdir}/${key} ($(hexbytes "${key}")) -> ${subdir}" >> $seqres.full
+
+	if [ ! -e "${fname}" ]; then
+		echo "Key ${key} does not exist for ${subdir} test??"
+		return
+	fi
+
+	actual_value="$(cat "${fname}")"
+	if [ "${actual_value}" != "${subdir}" ]; then
+		echo "Key ${key} has value ${subdir}, expected ${actual_value}."
+	fi
+}
+
 testd() {
 	key="$(echo -e "$1")"
 	value="$2"
@@ -152,7 +179,27 @@ setd "..\xe2\x80\x8d" "zero width joiners in dotdot entry"
 setf "toilet_bowl.\xf0\x9f\x9a\xbd" "toilet emoji"
 setf "toilet_bow\xe2\x80\x8dl.\xf0\x9f\x9a\xbd" "toilet emoji with zero width joiner"
 
-ls -la $testdir >> $seqres.full
+# decoy file extensions used in 3cx malware attack, and similar ones
+setchild "one_dot_leader" "job offer\xe2\x80\xa4pdf"
+setchild "small_full_stop" "job offer\xef\xb9\x92pdf"
+setchild "fullwidth_full_stop" "job offer\xef\xbc\x8epdf"
+setchild "syriac_supralinear" "job offer\xdc\x81pdf"
+setchild "syriac_sublinear" "job offer\xdc\x82pdf"
+setchild "lisu_letter_tone" "job offer\xea\x93\xb8pdf"
+setchild "actual_period" "job offer.pdf"
+setchild "one_dot_leader_zero_width_space" "job offer\xe2\x80\xa4\xe2\x80\x8dpdf"
+
+# again, but this time all in the same directory to trip the confusable
+# detector
+setf "job offer\xe2\x80\xa4pdf" "one dot leader"
+setf "job offer\xef\xb9\x92pdf" "small full stop"
+setf "job offer\xef\xbc\x8epdf" "fullwidth full stop"
+setf "job offer\xdc\x81pdf" "syriac supralinear full stop"
+setf "job offer\xdc\x82pdf" "syriac sublinear full stop"
+setf "job offer\xea\x93\xb8pdf" "lisu letter tone mya ti"
+setf "job offer.pdf" "actual period"
+
+ls -laR $testdir >> $seqres.full
 
 echo "Test files"
 testf "french_caf\xc3\xa9.txt" "NFC"
@@ -205,6 +252,23 @@ testd "..\xe2\x80\x8d" "zero width joiners in dotdot entry"
 testf "toilet_bowl.\xf0\x9f\x9a\xbd" "toilet emoji"
 testf "toilet_bow\xe2\x80\x8dl.\xf0\x9f\x9a\xbd" "toilet emoji with zero width joiner"
 
+testchild "one_dot_leader" "job offer\xe2\x80\xa4pdf"
+testchild "small_full_stop" "job offer\xef\xb9\x92pdf"
+testchild "fullwidth_full_stop" "job offer\xef\xbc\x8epdf"
+testchild "syriac_supralinear" "job offer\xdc\x81pdf"
+testchild "syriac_sublinear" "job offer\xdc\x82pdf"
+testchild "lisu_letter_tone" "job offer\xea\x93\xb8pdf"
+testchild "actual_period" "job offer.pdf"
+testchild "one_dot_leader_zero_width_space" "job offer\xe2\x80\xa4\xe2\x80\x8dpdf"
+
+testf "job offer\xe2\x80\xa4pdf" "one dot leader"
+testf "job offer\xef\xb9\x92pdf" "small full stop"
+testf "job offer\xef\xbc\x8epdf" "fullwidth full stop"
+testf "job offer\xdc\x81pdf" "syriac supralinear full stop"
+testf "job offer\xdc\x82pdf" "syriac sublinear full stop"
+testf "job offer\xea\x93\xb8pdf" "lisu letter tone mya ti"
+testf "job offer.pdf" "actual period"
+
 echo "Uniqueness of inodes?"
 stat -c '%i' "${testdir}/"* | sort | uniq -c | while read nr inum; do
 	if [ "${nr}" -gt 1 ]; then
@@ -228,6 +292,19 @@ if _check_xfs_scrub_does_unicode "$SCRATCH_MNT" "$SCRATCH_DEV"; then
 	grep -q "zerojoin_" $tmp.scrub || echo "No complaints about zero-width join confusables?"
 	grep -q "toilet_" $tmp.scrub || echo "No complaints about zero-width join confusables with emoji?"
 
+	# Does xfs_scrub complain at all about the job offer files?  Pre-2023
+	# versions did not know to screen for that.
+	if grep -q "job offer" $tmp.scrub; then
+		grep -q 'job offer.xe2.x80.xa4pdf' $tmp.scrub || echo "No complaints about one dot leader?"
+		grep -q "job offer.xef.xb9.x92pdf" $tmp.scrub || echo "No complaints about small full stop?"
+		grep -q "job offer.xef.xbc.x8epdf" $tmp.scrub || echo "No complaints about fullwidth full stop?"
+		grep -q "job offer.xdc.x81pdf" $tmp.scrub || echo "No complaints about syriac supralinear full stop?"
+		grep -q "job offer.xdc.x82pdf" $tmp.scrub || echo "No complaints about syriac sublinear full stop?"
+		grep -q "job offer.xea.x93.xb8pdf" $tmp.scrub || echo "No complaints about lisu letter tone mya ti?"
+		grep -q "job offer.*could be confused with" $tmp.scrub || echo "No complaints about confusing job offers?"
+		grep -q "job offer.xe2.x80.xa4.xe2.x80.x8dpdf" $tmp.scrub || echo "No complaints about one dot leader with invisible space?"
+	fi
+
 	echo "Actual xfs_scrub output:" >> $seqres.full
 	cat $tmp.scrub >> $seqres.full
 fi
#6 lore raw

PATCH 2/5 fstests: test systemd background services

In reply to Darrick J. Wong

Hi all,

Add a couple of new tests to check that the systemd services for xfs_scrub are at least minimally functional.

If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.

This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=scrub-improvements
---
Commits in this patchset:
* xfs: test xfs_scrub services

---
 common/rc          |   14 +++++
 common/systemd     |   73 +++++++++++++++++++++++++++
 tests/xfs/1863     |  140 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1863.out |    5 ++
 4 files changed, 232 insertions(+)
 create mode 100644 common/systemd
 create mode 100755 tests/xfs/1863
 create mode 100644 tests/xfs/1863.out
#7 lore raw

PATCH 1/1 xfs: test xfs_scrub services

From: Darrick J. Wong <[email protected]>

Create a pair of new tests that check that xfs_scrub and xfs_scrub_all will find and test mounted filesystems.

---
 common/rc          |   14 +++++
 common/systemd     |   73 +++++++++++++++++++++++++++
 tests/xfs/1863     |  140 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1863.out |    5 ++
 4 files changed, 232 insertions(+)
 create mode 100644 common/systemd
 create mode 100755 tests/xfs/1863
 create mode 100644 tests/xfs/1863.out
common/rc +14 -0
index 9da9fe1882..5a26669925 100644
@@ -5480,6 +5480,20 @@ _require_duplicate_fsid()
 	esac
 }
 
+# Can we find a program in the $PATH?
+_have_program() {
+	command -v "$1" &>/dev/null
+}
+
+# Require that a program can be found via the $PATH, or complain otherwise
+_require_program() {
+	local cmd="$1"
+	local tag="$2"
+
+	test -z "$tag" && tag="$(basename "$cmd")"
+	_have_program "$1" || _notrun "$tag required"
+}
+
 init_rc
 
 ################################################################################
common/systemd new file +73 -0

73 lines not on this page · raw message

tests/xfs/1863 new file +140 -0

140 lines not on this page · raw message

tests/xfs/1863.out new file +5 -0
new file mode 100644
index 0000000000..27c2d79a63
@@ -0,0 +1,5 @@
+QA output created by 1863
+Format and populate
+Scrub Scratch FS
+Scrub Everything
+Scrub Done
#9 lore raw

PATCH 3/5 fstests: use free space histograms to reduce fstrim runtime

In reply to Darrick J. Wong

Hi all,

This patchset dramatically reduces the runtime of the FITRIM calls made during phase 8 of xfs_scrub. It turns out that phase 8 can really get bogged down if the free space contains a large number of very small extents. In these cases, the runtime can increase by an order of magnitude to free less than 1% of the free space. This is not worth the time, since we're spending a lot of time to do very little work. The FITRIM ioctl allows us to specify a minimum extent length, so we can use statistical methods to compute a minlen parameter.

It turns out xfs_db/spaceman already have the code needed to create histograms of free space extent lengths. We add the ability to compute a CDF of the extent lengths, which make it easy to pick a minimum length corresponding to 99% of the free space. In most cases, this results in dramatic reductions in phase 8 runtime. Hence, move the histogram code to libfrog, and wire up xfs_scrub, since phase 7 already walks the fsmap.

We also add a new -o suboption to xfs_scrub so that people who /do/ want to examine every free extent can do so.

If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.

This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.

--D

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-fstrim-minlen-freesp-histogram

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=scrub-fstrim-minlen-freesp-histogram
---
Commits in this patchset:
* xfs/004: fix column extraction code

---
 tests/xfs/004 |   19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)
#10 lore raw

PATCH 1/1 xfs/004: fix column extraction code

From: Darrick J. Wong <[email protected]>

Now that the xfs_db freesp command prints a CDF of the free space histograms, fix the pct column extraction code to handle the two new columns by <cough> using awk.

---
 tests/xfs/004 |   19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)
tests/xfs/004 +11 -8
index 8b2fd9b320..473b82c945 100755
@@ -82,14 +82,17 @@ then
 fi
 
 # check the 'pct' field from freesp command is good
-perl -ne '
-	    BEGIN	{ $percent = 0; }
-	    /free/	&& next;	# skip over free extent size number
-	    if (/\s+(\d+\.\d+)$/) {
-		$percent += $1;
-	    }
-	    END	{ $percent += 0.5; print int($percent), "\n" }	# round up
-' <$tmp.xfs_db >$tmp.ans
+awk '
+{
+	if ($0 ~ /free/) {
+		next;
+	}
+
+	percent += $5;
+}
+END {
+	printf("%d\n", int(percent + 0.5));
+}' < $tmp.xfs_db > $tmp.ans
 ans="`cat $tmp.ans`"
 echo "Checking percent column yields 100: $ans"
 if [ "$ans" != 100 ]
Reviewed-by on v1 1/1
#11 lore raw

Re: [PATCH 1/1] xfs/004: fix column extraction code

Quoted 5 lines · Darrick J. Wong

From: Darrick J. Wong <[email protected]>

Now that the xfs_db freesp command prints a CDF of the free space histograms, fix the pct column extraction code to handle the two new columns by <cough> using awk.

Looks good:

.. and we really need this ASAP to not fail tests with the latest xfsprogs.

#12 lore raw

PATCH 4/5 fstests: enable FITRIM for the realtime section

In reply to Darrick J. Wong

Hi all,

One thing that's been missing for a long time is the ability to tell underlying storage that it can unmap the unused space on the realtime device. This short series exposes this functionality through FITRIM. Callers that want ranged FITRIM should be aware that the realtime space exists in the offset range after the data device. However, it is anticipated that most callers pass in offset=0 len=-1ULL and will not notice or care.

If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.

This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=realtime-discard

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=realtime-discard
---
Commits in this patchset:
* xfs: refactor statfs field extraction
* common/xfs: FITRIM now supports realtime volumes

---
 common/xfs    |   46 ++++++++++++++++++++++++++++++++++++++++++++--
 tests/xfs/176 |    4 ++--
 tests/xfs/187 |    6 +++---
 tests/xfs/541 |    6 ++----
 4 files changed, 51 insertions(+), 11 deletions(-)
#13 lore raw

PATCH 1/2 xfs: refactor statfs field extraction

From: Darrick J. Wong <[email protected]>

Prepare for the next patch by refactoring the open-coded bits that call statfs on a mounted xfs filesystem to extract a status field.

---
 common/xfs    |    6 ++++++
 tests/xfs/176 |    4 ++--
 tests/xfs/187 |    6 +++---
 tests/xfs/541 |    6 ++----
 4 files changed, 13 insertions(+), 9 deletions(-)
common/xfs +6 -0
index 6370f17523..7ee6fbec84 100644
@@ -1818,3 +1818,9 @@ _require_xfs_parent()
 		|| _notrun "kernel does not support parent pointers"
 	_scratch_unmount
 }
+
+# Extract a statfs attribute of the given mounted XFS filesystem.
+_xfs_statfs_field()
+{
+	$XFS_IO_PROG -c 'statfs' "$1" | grep -E "$2" | cut -d ' ' -f 3
+}
tests/xfs/176 +2 -2
index db7001a5b9..8d58590fd9 100755
@@ -47,7 +47,7 @@ fi
 
 _scratch_mount
 _xfs_force_bdev data $SCRATCH_MNT
-old_dblocks=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | grep geom.datablocks)
+old_dblocks=$(_xfs_statfs_field "$SCRATCH_MNT" geom.datablocks)
 
 mkdir $SCRATCH_MNT/save/ $SCRATCH_MNT/urk/
 sino=$(stat -c '%i' $SCRATCH_MNT/save)
@@ -170,7 +170,7 @@ for ((ino = target_ino; ino >= icluster_ino; ino--)); do
 	res=$?
 
 	# Make sure shrink did not work
-	new_dblocks=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | grep geom.datablocks)
+	new_dblocks=$(_xfs_statfs_field "$SCRATCH_MNT" geom.datablocks)
 	if [ "$new_dblocks" != "$old_dblocks" ]; then
 		echo "should not have shrank $old_dblocks -> $new_dblocks"
 		break
tests/xfs/187 +3 -3
index 04ff9a81b6..56a9adc164 100755
@@ -77,8 +77,8 @@ _xfs_force_bdev realtime $SCRATCH_MNT
 
 # Set the extent size hint larger than the realtime extent size.  This is
 # necessary to exercise the minlen constraints on the realtime allocator.
-fsbsize=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | grep geom.bsize | awk '{print $3}')
-rtextsize_blks=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | grep geom.rtextsize | awk '{print $3}')
+fsbsize=$(_xfs_statfs_field "$SCRATCH_MNT" geom.bsize)
+rtextsize_blks=$(_xfs_statfs_field "$SCRATCH_MNT" geom.rtextsize)
 extsize=$((2 * rtextsize_blks * fsbsize))
 
 echo "rtextsize_blks=$rtextsize_blks extsize=$extsize" >> $seqres.full
@@ -133,7 +133,7 @@ punch_off=$((bigfile_sz - frag_sz))
 $here/src/punch-alternating $SCRATCH_MNT/bigfile -o $((punch_off / fsbsize)) -i $((rtextsize_blks * 2)) -s $rtextsize_blks
 
 # Make sure we have some free rtextents.
-free_rtx=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | grep statfs.f_bavail | awk '{print $3}')
+free_rtx=$(_xfs_statfs_field "$SCRATCH_MNT" statfs.f_bavail)
 if [ $free_rtx -eq 0 ]; then
 	echo "Expected fragmented free rt space, found none."
 fi
tests/xfs/541 +2 -4
index f18b801cfe..518373fa89 100755
@@ -81,13 +81,11 @@ test $grow_extszhint -eq 0 || \
 	echo "expected post-grow extszhint 0, got $grow_extszhint"
 
 # Check that we now have rt extents.
-rtextents=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | \
-	grep 'geom.rtextents' | cut -d ' ' -f 3)
+rtextents=$(_xfs_statfs_field "$SCRATCH_MNT" geom.rtextents)
 test $rtextents -gt 0 || echo "expected rtextents > 0"
 
 # Check the new rt extent size.
-after_rtextsz_blocks=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | \
-	grep 'geom.rtextsize' | cut -d ' ' -f 3)
+after_rtextsz_blocks=$(_xfs_statfs_field "$SCRATCH_MNT" geom.rtextsize)
 test $after_rtextsz_blocks -eq $new_rtextsz_blocks || \
 	echo "expected rtextsize $new_rtextsz_blocks, got $after_rtextsz_blocks"
 
#15 lore raw

PATCH 2/2 common/xfs: FITRIM now supports realtime volumes

In reply to Darrick J. Wong

From: Darrick J. Wong <[email protected]>

XFS now supports FITRIM to the realtime volume. Detect this support and enable it.

---
 common/xfs |   40 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 38 insertions(+), 2 deletions(-)
common/xfs +38 -2
index 7ee6fbec84..de557ebd90 100644
@@ -1777,8 +1777,44 @@ _require_xfs_scratch_atomicswap()
 # of 1024 byte blocks.
 _xfs_discard_max_offset_kb()
 {
-	$XFS_IO_PROG -c 'statfs' "$1" | \
-		awk '{g[$1] = $3} END {print (g["geom.bsize"] * g["geom.datablocks"] / 1024)}'
+	local statfs
+
+	# Use awk to read the statfs output for the XFS filesystem, compute
+	# the two possible FITRIM offset maximums, and then use some horrid
+	# bash magic to import the five numbers as an indexed array.  There's
+	# no better way to do this in bash since you can't readarray to build
+	# an associative array.  Elements are as follows:
+	#
+	# 0: fsblock size in bytes
+	# 1: Data volume size in fsblocks.
+	# 2: Realtime volume size in fsblocks.
+	# 3: Max FITRIM offset if we can only trim the data volume
+	# 4: Max FITRIM offset if we can trim the data and rt volumes
+	readarray -t statfs < <($XFS_IO_PROG -c 'statfs' "$1" | \
+		awk '{g[$1] = $3} END {printf("%d\n%d\n%d\n%d\n%d\n",
+			g["geom.bsize"],
+			g["geom.datablocks"],
+			g["geom.rtblocks"],
+			g["geom.bsize"] * g["geom.datablocks"] / 1024,
+			g["geom.bsize"] * (g["geom.datablocks"] + g["geom.rtblocks"]) / 1024);}')
+
+	# If the kernel supports discarding the realtime volume, then it will
+	# not reject a FITRIM for fsblock dblks+1, even if the len/minlen
+	# arguments are absurd.
+	if [ "${statfs[2]}" -gt 0 ]; then
+		if $FSTRIM_PROG -o "$((statfs[0] * statfs[1]))" \
+				-l "${statfs[0]}" \
+				-m "$((statfs[0] * 2))" "$1" &>/dev/null; then
+			# The kernel supports discarding the rt volume, so
+			# print out the second answer from above.
+			echo "${statfs[4]}"
+			return
+		fi
+	fi
+
+	# The kernel does not support discarding the rt volume or there is no
+	# rt volume.  Print out the first answer from above.
+	echo "${statfs[3]}"
 }
 
 # check if mkfs and the kernel support nocrc (v4) file systems
#17 lore raw

PATCH 5/5 fstests: xfs filesystem properties

In reply to Darrick J. Wong

Hi all,

It would be very useful if system administrators could set properties for a given xfs filesystem to control its behavior. This we can do easily and extensibly by setting ATTR_ROOT (aka "trusted") extended attributes on the root directory. To prevent this from becoming a weird free for all, let's add some library and tooling support so that sysadmins simply run the xfs_property program to administer these properties.

If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.

This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.

--D

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=filesystem-properties

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=filesystem-properties
---
Commits in this patchset:
* xfs: functional testing for filesystem properties

---
 common/config       |    1 
 common/xfs          |   14 ++++-
 doc/group-names.txt |    1 
 tests/generic/062   |    4 +
 tests/xfs/1886      |  137 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1886.out  |   53 ++++++++++++++++++++
 tests/xfs/1887      |  122 +++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1887.out  |   46 +++++++++++++++++
 tests/xfs/1888      |   66 +++++++++++++++++++++++++
 tests/xfs/1888.out  |    9 +++
 tests/xfs/1889      |   63 +++++++++++++++++++++++
 tests/xfs/1889.out  |    8 +++
 12 files changed, 522 insertions(+), 2 deletions(-)
 create mode 100755 tests/xfs/1886
 create mode 100644 tests/xfs/1886.out
 create mode 100755 tests/xfs/1887
 create mode 100644 tests/xfs/1887.out
 create mode 100755 tests/xfs/1888
 create mode 100644 tests/xfs/1888.out
 create mode 100755 tests/xfs/1889
 create mode 100644 tests/xfs/1889.out
#18 lore raw

PATCH 1/1 xfs: functional testing for filesystem properties

From: Darrick J. Wong <[email protected]>

Make sure that fs property storage and retrieval actually work.

---
 common/config       |    1 
 common/xfs          |   14 ++++-
 doc/group-names.txt |    1 
 tests/generic/062   |    4 +
 tests/xfs/1886      |  137 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1886.out  |   53 ++++++++++++++++++++
 tests/xfs/1887      |  122 +++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1887.out  |   46 +++++++++++++++++
 tests/xfs/1888      |   66 +++++++++++++++++++++++++
 tests/xfs/1888.out  |    9 +++
 tests/xfs/1889      |   63 +++++++++++++++++++++++
 tests/xfs/1889.out  |    8 +++
 12 files changed, 522 insertions(+), 2 deletions(-)
 create mode 100755 tests/xfs/1886
 create mode 100644 tests/xfs/1886.out
 create mode 100755 tests/xfs/1887
 create mode 100644 tests/xfs/1887.out
 create mode 100755 tests/xfs/1888
 create mode 100644 tests/xfs/1888.out
 create mode 100755 tests/xfs/1889
 create mode 100644 tests/xfs/1889.out
common/config +1 -0
index 307f93fbce..668b185ff0 100644
@@ -234,6 +234,7 @@ export GZIP_PROG="$(type -P gzip)"
 export BTRFS_IMAGE_PROG="$(type -P btrfs-image)"
 export BTRFS_MAP_LOGICAL_PROG=$(type -P btrfs-map-logical)
 export PARTED_PROG="$(type -P parted)"
+export XFS_PROPERTY_PROG="$(type -P xfs_property)"
 
 # use 'udevadm settle' or 'udevsettle' to wait for lv to be settled.
 # newer systems have udevadm command but older systems like RHEL5 don't.
common/xfs +12 -2
index de557ebd90..62e3100ee1 100644
@@ -1323,8 +1323,8 @@ _require_xfs_spaceman_command()
 	testfile=$TEST_DIR/$$.xfs_spaceman
 	touch $testfile
 	case $command in
-	"health")
-		testio=`$XFS_SPACEMAN_PROG -c "health $param" $TEST_DIR 2>&1`
+	"health"|"listfsprops")
+		testio=`$XFS_SPACEMAN_PROG -c "$command $param" $TEST_DIR 2>&1`
 		param_checked=1
 		;;
 	*)
@@ -1860,3 +1860,13 @@ _xfs_statfs_field()
 {
 	$XFS_IO_PROG -c 'statfs' "$1" | grep -E "$2" | cut -d ' ' -f 3
 }
+
+# Wipe all filesystem properties from an xfs filesystem.  The sole argument
+# must be the root directory of a filesystem.
+_wipe_xfs_properties()
+{
+	getfattr --match="^trusted.xfs:" --absolute-names --dump --encoding=hex "$1" | \
+			grep '=' | sed -e 's/=.*$//g' | while read name; do
+		setfattr --remove="$name" "$1"
+	done
+}
doc/group-names.txt +1 -0
index 6cf717969d..ed886caac0 100644
@@ -56,6 +56,7 @@ fiexchange		XFS_IOC_EXCHANGE_RANGE ioctl
 freeze			filesystem freeze tests
 fsck			general fsck tests
 fsmap			FS_IOC_GETFSMAP ioctl
+fsproperties		Filesystem properties
 fsr			XFS free space reorganizer
 fuzzers			filesystem fuzz tests
 growfs			increasing the size of a filesystem
tests/generic/062 +4 -0
index 8f4dfcbf55..f0904992d1 100755
@@ -75,6 +75,10 @@ fi
 
 _require_attrs $ATTR_MODES
 
+# Wipe all xfs filesystem properties (which are rootdir xattrs) before we dump
+# them all later.
+test $FSTYP = "xfs" && _wipe_xfs_properties $SCRATCH_MNT
+
 for nsp in $ATTR_MODES; do
 	for inode in reg dir lnk dev/b dev/c dev/p; do
 
tests/xfs/1886 new file +137 -0

137 lines not on this page · raw message

tests/xfs/1886.out new file +53 -0

53 lines not on this page · raw message

tests/xfs/1887 new file +122 -0

122 lines not on this page · raw message

tests/xfs/1887.out new file +46 -0

46 lines not on this page · raw message

tests/xfs/1888 new file +66 -0

66 lines not on this page · raw message

tests/xfs/1888.out new file +9 -0
new file mode 100644
index 0000000000..73857cabd4
@@ -0,0 +1,9 @@
+QA output created by 1888
+autofsck=repair
+failed to get attr xfs:autofsck on inode XXX: No data available
+autofsck=none
+autofsck=check
+autofsck=optimize
+autofsck=repair
+autofsck=none
+autofsck=repair
tests/xfs/1889 new file +63 -0

63 lines not on this page · raw message

tests/xfs/1889.out new file +8 -0
new file mode 100644
index 0000000000..fd8123b53a
@@ -0,0 +1,8 @@
+QA output created by 1889
+Info: TEST_DIR/1889.mount: Checking and repairing per autofsck directive.
+Info: TEST_DIR/1889.mount: Disabling scrub per autofsck directive.
+Info: TEST_DIR/1889.mount: Checking per autofsck directive.
+Info: TEST_DIR/1889.mount: Optimizing per autofsck directive.
+Info: TEST_DIR/1889.mount: Checking and repairing per autofsck directive.
+Info: TEST_DIR/1889.mount: Disabling scrub per autofsck directive.
+Info: TEST_DIR/1889.mount: Checking and repairing per autofsck directive.