fstests: catch us up to xfsprogs 6.10
Hi Zorro,
Now that Carlos has released xfsprogs 6.10, here's some new functional tests for the improvements that landed this cycle. This includes the new realtime volume FITRIM support, filesystem properties, deceptive name detection in xfs_scrub, and the (hopefully final) structure of the background xfs_scrub systemd services.
--D
PATCH 1/5 fstests: detect deceptive filename extensions
Hi all,
In early 2023, malware researchers disclosed a phishing attack that was targeted at people running Linux workstations. The attack vector involved the use of filenames containing what looked like a file extension but instead contained a lookalike for the full stop (".") and a common extension ("pdf"). Enhance xfs_scrub phase 5 to detect these types of attacks and warn the system administrator. Add functional testing for this code.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-detect-deceptive-extensions
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=scrub-detect-deceptive-extensions
---
Commits in this patchset:
* generic/453: test confusable name detection with 32-bit unicode codepoints
* generic/453: check xfs_scrub detection of confusing job offers
--- tests/generic/453 | 111 +++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 99 insertions(+), 12 deletions(-)
PATCH 1/2 generic/453: test confusable name detection with 32-bit unicode codepoints
From: Darrick J. Wong <[email protected]>
Test the confusable name detection when there are 32-bit unicode sequences in use. In other words, emoji. Change the xfs_scrub test to dump the output to a file instead of passing huge echo commands around.
--- tests/generic/453 | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-)
tests/generic/453 +21 -11
@@ -148,6 +148,10 @@ setf "combmark_\xe1\x80\x9c\xe1\x80\xaf\xe1\x80\xad.txt" "combining marks" setd ".\xe2\x80\x8d" "zero width joiners in dot entry" setd "..\xe2\x80\x8d" "zero width joiners in dotdot entry" +# utf8 sequence mapping to a u32 unicode codepoint that can be confused +setf "toilet_bowl.\xf0\x9f\x9a\xbd" "toilet emoji" +setf "toilet_bow\xe2\x80\x8dl.\xf0\x9f\x9a\xbd" "toilet emoji with zero width joiner" + ls -la $testdir >> $seqres.full echo "Test files"
@@ -198,6 +202,9 @@ testf "combmark_\xe1\x80\x9c\xe1\x80\xaf\xe1\x80\xad.txt" "combining marks" testd ".\xe2\x80\x8d" "zero width joiners in dot entry" testd "..\xe2\x80\x8d" "zero width joiners in dotdot entry" +testf "toilet_bowl.\xf0\x9f\x9a\xbd" "toilet emoji" +testf "toilet_bow\xe2\x80\x8dl.\xf0\x9f\x9a\xbd" "toilet emoji with zero width joiner" + echo "Uniqueness of inodes?" stat -c '%i' "${testdir}/"* | sort | uniq -c | while read nr inum; do if [ "${nr}" -gt 1 ]; then
@@ -208,18 +215,21 @@ done echo "Test XFS online scrub, if applicable" if _check_xfs_scrub_does_unicode "$SCRATCH_MNT" "$SCRATCH_DEV"; then - output="$(LC_ALL="C.UTF-8" ${XFS_SCRUB_PROG} -v -n "${SCRATCH_MNT}" 2>&1 | filter_scrub)" - echo "${output}" | grep -q "french_" || echo "No complaints about french e accent?" - echo "${output}" | grep -q "greek_" || echo "No complaints about greek letter mess?" - echo "${output}" | grep -q "arabic_" || echo "No complaints about arabic expanded string?" - echo "${output}" | grep -q "mixed_" || echo "No complaints about mixed script confusables?" - echo "${output}" | grep -q "hyphens_" || echo "No complaints about hyphenation confusables?" - echo "${output}" | grep -q "dz_digraph_" || echo "No complaints about single script confusables?" - echo "${output}" | grep -q "inadequate_" || echo "No complaints about inadequate rendering confusables?" - echo "${output}" | grep -q "prohibition_" || echo "No complaints about prohibited sequence confusables?" - echo "${output}" | grep -q "zerojoin_" || echo "No complaints about zero-width join confusables?" + LC_ALL="C.UTF-8" ${XFS_SCRUB_PROG} -v -n "${SCRATCH_MNT}" 2>&1 | filter_scrub > $tmp.scrub + + grep -q "french_" $tmp.scrub || echo "No complaints about french e accent?" + grep -q "greek_" $tmp.scrub || echo "No complaints about greek letter mess?" + grep -q "arabic_" $tmp.scrub || echo "No complaints about arabic expanded string?" + grep -q "mixed_" $tmp.scrub || echo "No complaints about mixed script confusables?" + grep -q "hyphens_" $tmp.scrub || echo "No complaints about hyphenation confusables?" + grep -q "dz_digraph_" $tmp.scrub || echo "No complaints about single script confusables?" + grep -q "inadequate_" $tmp.scrub || echo "No complaints about inadequate rendering confusables?" + grep -q "prohibition_" $tmp.scrub || echo "No complaints about prohibited sequence confusables?" + grep -q "zerojoin_" $tmp.scrub || echo "No complaints about zero-width join confusables?" + grep -q "toilet_" $tmp.scrub || echo "No complaints about zero-width join confusables with emoji?" + echo "Actual xfs_scrub output:" >> $seqres.full - echo "${output}" >> $seqres.full + cat $tmp.scrub >> $seqres.full fi # success, all done
Re: [PATCH 1/2] generic/453: test confusable name detection with 32-bit unicode codepoints
Looks good:
PATCH 2/2 generic/453: check xfs_scrub detection of confusing job offers
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
Earlier this year, ESET revealed that Linux users had been tricked into opening executables containing malware payloads. The trickery came in the form of a malicious zip file containing a filename with the string "job offer․pdf". Note that the filename does *not* denote a real pdf file, since the last four codepoints in the file name are "ONE DOT LEADER", p, d, and f. Not period (ok, FULL STOP), p, d, f like you'd normally expect.
Now that xfs_scrub can look for codepoints that could be confused with a period followed by alphanumerics, let's make sure it actually works.
--- tests/generic/453 | 79 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 78 insertions(+), 1 deletion(-)
tests/generic/453 +78 -1
@@ -36,6 +36,15 @@ setf() { echo "Storing ${key} ($(hexbytes "${key}")) -> ${value}" >> $seqres.full } +setchild() { + subdir="$1" + key="$(echo -e "$2")" + + mkdir -p "${testdir}/${subdir}" + echo "$subdir" > "${testdir}/${subdir}/${key}" + echo "Storing ${subdir}/${key} ($(hexbytes "${key}")) -> ${subdir}" >> $seqres.full +} + setd() { key="$(echo -e "$1")" value="$2"
@@ -63,6 +72,24 @@ testf() { fi } +testchild() { + subdir="$1" + key="$(echo -e "$2")" + fname="${testdir}/${subdir}/${key}" + + echo "Testing ${subdir}/${key} ($(hexbytes "${key}")) -> ${subdir}" >> $seqres.full + + if [ ! -e "${fname}" ]; then + echo "Key ${key} does not exist for ${subdir} test??" + return + fi + + actual_value="$(cat "${fname}")" + if [ "${actual_value}" != "${subdir}" ]; then + echo "Key ${key} has value ${subdir}, expected ${actual_value}." + fi +} + testd() { key="$(echo -e "$1")" value="$2"
@@ -152,7 +179,27 @@ setd "..\xe2\x80\x8d" "zero width joiners in dotdot entry" setf "toilet_bowl.\xf0\x9f\x9a\xbd" "toilet emoji" setf "toilet_bow\xe2\x80\x8dl.\xf0\x9f\x9a\xbd" "toilet emoji with zero width joiner" -ls -la $testdir >> $seqres.full +# decoy file extensions used in 3cx malware attack, and similar ones +setchild "one_dot_leader" "job offer\xe2\x80\xa4pdf" +setchild "small_full_stop" "job offer\xef\xb9\x92pdf" +setchild "fullwidth_full_stop" "job offer\xef\xbc\x8epdf" +setchild "syriac_supralinear" "job offer\xdc\x81pdf" +setchild "syriac_sublinear" "job offer\xdc\x82pdf" +setchild "lisu_letter_tone" "job offer\xea\x93\xb8pdf" +setchild "actual_period" "job offer.pdf" +setchild "one_dot_leader_zero_width_space" "job offer\xe2\x80\xa4\xe2\x80\x8dpdf" + +# again, but this time all in the same directory to trip the confusable +# detector +setf "job offer\xe2\x80\xa4pdf" "one dot leader" +setf "job offer\xef\xb9\x92pdf" "small full stop" +setf "job offer\xef\xbc\x8epdf" "fullwidth full stop" +setf "job offer\xdc\x81pdf" "syriac supralinear full stop" +setf "job offer\xdc\x82pdf" "syriac sublinear full stop" +setf "job offer\xea\x93\xb8pdf" "lisu letter tone mya ti" +setf "job offer.pdf" "actual period" + +ls -laR $testdir >> $seqres.full echo "Test files" testf "french_caf\xc3\xa9.txt" "NFC"
@@ -205,6 +252,23 @@ testd "..\xe2\x80\x8d" "zero width joiners in dotdot entry" testf "toilet_bowl.\xf0\x9f\x9a\xbd" "toilet emoji" testf "toilet_bow\xe2\x80\x8dl.\xf0\x9f\x9a\xbd" "toilet emoji with zero width joiner" +testchild "one_dot_leader" "job offer\xe2\x80\xa4pdf" +testchild "small_full_stop" "job offer\xef\xb9\x92pdf" +testchild "fullwidth_full_stop" "job offer\xef\xbc\x8epdf" +testchild "syriac_supralinear" "job offer\xdc\x81pdf" +testchild "syriac_sublinear" "job offer\xdc\x82pdf" +testchild "lisu_letter_tone" "job offer\xea\x93\xb8pdf" +testchild "actual_period" "job offer.pdf" +testchild "one_dot_leader_zero_width_space" "job offer\xe2\x80\xa4\xe2\x80\x8dpdf" + +testf "job offer\xe2\x80\xa4pdf" "one dot leader" +testf "job offer\xef\xb9\x92pdf" "small full stop" +testf "job offer\xef\xbc\x8epdf" "fullwidth full stop" +testf "job offer\xdc\x81pdf" "syriac supralinear full stop" +testf "job offer\xdc\x82pdf" "syriac sublinear full stop" +testf "job offer\xea\x93\xb8pdf" "lisu letter tone mya ti" +testf "job offer.pdf" "actual period" + echo "Uniqueness of inodes?" stat -c '%i' "${testdir}/"* | sort | uniq -c | while read nr inum; do if [ "${nr}" -gt 1 ]; then
@@ -228,6 +292,19 @@ if _check_xfs_scrub_does_unicode "$SCRATCH_MNT" "$SCRATCH_DEV"; then grep -q "zerojoin_" $tmp.scrub || echo "No complaints about zero-width join confusables?" grep -q "toilet_" $tmp.scrub || echo "No complaints about zero-width join confusables with emoji?" + # Does xfs_scrub complain at all about the job offer files? Pre-2023 + # versions did not know to screen for that. + if grep -q "job offer" $tmp.scrub; then + grep -q 'job offer.xe2.x80.xa4pdf' $tmp.scrub || echo "No complaints about one dot leader?" + grep -q "job offer.xef.xb9.x92pdf" $tmp.scrub || echo "No complaints about small full stop?" + grep -q "job offer.xef.xbc.x8epdf" $tmp.scrub || echo "No complaints about fullwidth full stop?" + grep -q "job offer.xdc.x81pdf" $tmp.scrub || echo "No complaints about syriac supralinear full stop?" + grep -q "job offer.xdc.x82pdf" $tmp.scrub || echo "No complaints about syriac sublinear full stop?" + grep -q "job offer.xea.x93.xb8pdf" $tmp.scrub || echo "No complaints about lisu letter tone mya ti?" + grep -q "job offer.*could be confused with" $tmp.scrub || echo "No complaints about confusing job offers?" + grep -q "job offer.xe2.x80.xa4.xe2.x80.x8dpdf" $tmp.scrub || echo "No complaints about one dot leader with invisible space?" + fi + echo "Actual xfs_scrub output:" >> $seqres.full cat $tmp.scrub >> $seqres.full fi
Re: [PATCH 2/2] generic/453: check xfs_scrub detection of confusing job offers
Looks good:
PATCH 2/5 fstests: test systemd background services
In reply to Darrick J. WongHi all,
Add a couple of new tests to check that the systemd services for xfs_scrub are at least minimally functional.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=scrub-improvements
---
Commits in this patchset:
* xfs: test xfs_scrub services
--- common/rc | 14 +++++ common/systemd | 73 +++++++++++++++++++++++++++ tests/xfs/1863 | 140 ++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1863.out | 5 ++ 4 files changed, 232 insertions(+) create mode 100644 common/systemd create mode 100755 tests/xfs/1863 create mode 100644 tests/xfs/1863.out
PATCH 1/1 xfs: test xfs_scrub services
From: Darrick J. Wong <[email protected]>
Create a pair of new tests that check that xfs_scrub and xfs_scrub_all will find and test mounted filesystems.
--- common/rc | 14 +++++ common/systemd | 73 +++++++++++++++++++++++++++ tests/xfs/1863 | 140 ++++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1863.out | 5 ++ 4 files changed, 232 insertions(+) create mode 100644 common/systemd create mode 100755 tests/xfs/1863 create mode 100644 tests/xfs/1863.out
common/rc +14 -0
@@ -5480,6 +5480,20 @@ _require_duplicate_fsid() esac } +# Can we find a program in the $PATH? +_have_program() { + command -v "$1" &>/dev/null +} + +# Require that a program can be found via the $PATH, or complain otherwise +_require_program() { + local cmd="$1" + local tag="$2" + + test -z "$tag" && tag="$(basename "$cmd")" + _have_program "$1" || _notrun "$tag required" +} + init_rc ################################################################################
common/systemd new file +73 -0
73 lines not on this page · raw message
tests/xfs/1863 new file +140 -0
140 lines not on this page · raw message
tests/xfs/1863.out new file +5 -0
@@ -0,0 +1,5 @@ +QA output created by 1863 +Format and populate +Scrub Scratch FS +Scrub Everything +Scrub Done
Re: [PATCH 1/1] xfs: test xfs_scrub services
Looks good:
PATCH 3/5 fstests: use free space histograms to reduce fstrim runtime
In reply to Darrick J. WongHi all,
This patchset dramatically reduces the runtime of the FITRIM calls made during phase 8 of xfs_scrub. It turns out that phase 8 can really get bogged down if the free space contains a large number of very small extents. In these cases, the runtime can increase by an order of magnitude to free less than 1% of the free space. This is not worth the time, since we're spending a lot of time to do very little work. The FITRIM ioctl allows us to specify a minimum extent length, so we can use statistical methods to compute a minlen parameter.
It turns out xfs_db/spaceman already have the code needed to create histograms of free space extent lengths. We add the ability to compute a CDF of the extent lengths, which make it easy to pick a minimum length corresponding to 99% of the free space. In most cases, this results in dramatic reductions in phase 8 runtime. Hence, move the histogram code to libfrog, and wire up xfs_scrub, since phase 7 already walks the fsmap.
We also add a new -o suboption to xfs_scrub so that people who /do/ want to examine every free extent can do so.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=scrub-fstrim-minlen-freesp-histogram
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=scrub-fstrim-minlen-freesp-histogram
---
Commits in this patchset:
* xfs/004: fix column extraction code
--- tests/xfs/004 | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-)
PATCH 1/1 xfs/004: fix column extraction code
From: Darrick J. Wong <[email protected]>
Now that the xfs_db freesp command prints a CDF of the free space histograms, fix the pct column extraction code to handle the two new columns by <cough> using awk.
--- tests/xfs/004 | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-)
tests/xfs/004 +11 -8
@@ -82,14 +82,17 @@ then fi # check the 'pct' field from freesp command is good -perl -ne ' - BEGIN { $percent = 0; } - /free/ && next; # skip over free extent size number - if (/\s+(\d+\.\d+)$/) { - $percent += $1; - } - END { $percent += 0.5; print int($percent), "\n" } # round up -' <$tmp.xfs_db >$tmp.ans +awk ' +{ + if ($0 ~ /free/) { + next; + } + + percent += $5; +} +END { + printf("%d\n", int(percent + 0.5)); +}' < $tmp.xfs_db > $tmp.ans ans="`cat $tmp.ans`" echo "Checking percent column yields 100: $ans" if [ "$ans" != 100 ]
Re: [PATCH 1/1] xfs/004: fix column extraction code
Quoted 5 lines · Darrick J. Wong
From: Darrick J. Wong <[email protected]>
Now that the
xfs_dbfreesp command prints a CDF of the free space histograms, fix the pct column extraction code to handle the two new columns by <cough> using awk.
Looks good:
.. and we really need this ASAP to not fail tests with the latest xfsprogs.
PATCH 4/5 fstests: enable FITRIM for the realtime section
In reply to Darrick J. WongHi all,
One thing that's been missing for a long time is the ability to tell underlying storage that it can unmap the unused space on the realtime device. This short series exposes this functionality through FITRIM. Callers that want ranged FITRIM should be aware that the realtime space exists in the offset range after the data device. However, it is anticipated that most callers pass in offset=0 len=-1ULL and will not notice or care.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=realtime-discard
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=realtime-discard
---
Commits in this patchset:
* xfs: refactor statfs field extraction
* common/xfs: FITRIM now supports realtime volumes
--- common/xfs | 46 ++++++++++++++++++++++++++++++++++++++++++++-- tests/xfs/176 | 4 ++-- tests/xfs/187 | 6 +++--- tests/xfs/541 | 6 ++---- 4 files changed, 51 insertions(+), 11 deletions(-)
PATCH 1/2 xfs: refactor statfs field extraction
From: Darrick J. Wong <[email protected]>
Prepare for the next patch by refactoring the open-coded bits that call statfs on a mounted xfs filesystem to extract a status field.
--- common/xfs | 6 ++++++ tests/xfs/176 | 4 ++-- tests/xfs/187 | 6 +++--- tests/xfs/541 | 6 ++---- 4 files changed, 13 insertions(+), 9 deletions(-)
common/xfs +6 -0
@@ -1818,3 +1818,9 @@ _require_xfs_parent() || _notrun "kernel does not support parent pointers" _scratch_unmount } + +# Extract a statfs attribute of the given mounted XFS filesystem. +_xfs_statfs_field() +{ + $XFS_IO_PROG -c 'statfs' "$1" | grep -E "$2" | cut -d ' ' -f 3 +}
tests/xfs/176 +2 -2
@@ -47,7 +47,7 @@ fi _scratch_mount _xfs_force_bdev data $SCRATCH_MNT -old_dblocks=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | grep geom.datablocks) +old_dblocks=$(_xfs_statfs_field "$SCRATCH_MNT" geom.datablocks) mkdir $SCRATCH_MNT/save/ $SCRATCH_MNT/urk/ sino=$(stat -c '%i' $SCRATCH_MNT/save)
@@ -170,7 +170,7 @@ for ((ino = target_ino; ino >= icluster_ino; ino--)); do res=$? # Make sure shrink did not work - new_dblocks=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | grep geom.datablocks) + new_dblocks=$(_xfs_statfs_field "$SCRATCH_MNT" geom.datablocks) if [ "$new_dblocks" != "$old_dblocks" ]; then echo "should not have shrank $old_dblocks -> $new_dblocks" break
tests/xfs/187 +3 -3
@@ -77,8 +77,8 @@ _xfs_force_bdev realtime $SCRATCH_MNT # Set the extent size hint larger than the realtime extent size. This is # necessary to exercise the minlen constraints on the realtime allocator. -fsbsize=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | grep geom.bsize | awk '{print $3}') -rtextsize_blks=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | grep geom.rtextsize | awk '{print $3}') +fsbsize=$(_xfs_statfs_field "$SCRATCH_MNT" geom.bsize) +rtextsize_blks=$(_xfs_statfs_field "$SCRATCH_MNT" geom.rtextsize) extsize=$((2 * rtextsize_blks * fsbsize)) echo "rtextsize_blks=$rtextsize_blks extsize=$extsize" >> $seqres.full
@@ -133,7 +133,7 @@ punch_off=$((bigfile_sz - frag_sz)) $here/src/punch-alternating $SCRATCH_MNT/bigfile -o $((punch_off / fsbsize)) -i $((rtextsize_blks * 2)) -s $rtextsize_blks # Make sure we have some free rtextents. -free_rtx=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | grep statfs.f_bavail | awk '{print $3}') +free_rtx=$(_xfs_statfs_field "$SCRATCH_MNT" statfs.f_bavail) if [ $free_rtx -eq 0 ]; then echo "Expected fragmented free rt space, found none." fi
tests/xfs/541 +2 -4
@@ -81,13 +81,11 @@ test $grow_extszhint -eq 0 || \ echo "expected post-grow extszhint 0, got $grow_extszhint" # Check that we now have rt extents. -rtextents=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | \ - grep 'geom.rtextents' | cut -d ' ' -f 3) +rtextents=$(_xfs_statfs_field "$SCRATCH_MNT" geom.rtextents) test $rtextents -gt 0 || echo "expected rtextents > 0" # Check the new rt extent size. -after_rtextsz_blocks=$($XFS_IO_PROG -c 'statfs' $SCRATCH_MNT | \ - grep 'geom.rtextsize' | cut -d ' ' -f 3) +after_rtextsz_blocks=$(_xfs_statfs_field "$SCRATCH_MNT" geom.rtextsize) test $after_rtextsz_blocks -eq $new_rtextsz_blocks || \ echo "expected rtextsize $new_rtextsz_blocks, got $after_rtextsz_blocks"
Re: [PATCH 1/2] xfs: refactor statfs field extraction
Looks good:
PATCH 2/2 common/xfs: FITRIM now supports realtime volumes
In reply to Darrick J. WongFrom: Darrick J. Wong <[email protected]>
XFS now supports FITRIM to the realtime volume. Detect this support and enable it.
--- common/xfs | 40 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2 deletions(-)
common/xfs +38 -2
@@ -1777,8 +1777,44 @@ _require_xfs_scratch_atomicswap() # of 1024 byte blocks. _xfs_discard_max_offset_kb() { - $XFS_IO_PROG -c 'statfs' "$1" | \ - awk '{g[$1] = $3} END {print (g["geom.bsize"] * g["geom.datablocks"] / 1024)}' + local statfs + + # Use awk to read the statfs output for the XFS filesystem, compute + # the two possible FITRIM offset maximums, and then use some horrid + # bash magic to import the five numbers as an indexed array. There's + # no better way to do this in bash since you can't readarray to build + # an associative array. Elements are as follows: + # + # 0: fsblock size in bytes + # 1: Data volume size in fsblocks. + # 2: Realtime volume size in fsblocks. + # 3: Max FITRIM offset if we can only trim the data volume + # 4: Max FITRIM offset if we can trim the data and rt volumes + readarray -t statfs < <($XFS_IO_PROG -c 'statfs' "$1" | \ + awk '{g[$1] = $3} END {printf("%d\n%d\n%d\n%d\n%d\n", + g["geom.bsize"], + g["geom.datablocks"], + g["geom.rtblocks"], + g["geom.bsize"] * g["geom.datablocks"] / 1024, + g["geom.bsize"] * (g["geom.datablocks"] + g["geom.rtblocks"]) / 1024);}') + + # If the kernel supports discarding the realtime volume, then it will + # not reject a FITRIM for fsblock dblks+1, even if the len/minlen + # arguments are absurd. + if [ "${statfs[2]}" -gt 0 ]; then + if $FSTRIM_PROG -o "$((statfs[0] * statfs[1]))" \ + -l "${statfs[0]}" \ + -m "$((statfs[0] * 2))" "$1" &>/dev/null; then + # The kernel supports discarding the rt volume, so + # print out the second answer from above. + echo "${statfs[4]}" + return + fi + fi + + # The kernel does not support discarding the rt volume or there is no + # rt volume. Print out the first answer from above. + echo "${statfs[3]}" } # check if mkfs and the kernel support nocrc (v4) file systems
Re: [PATCH 2/2] common/xfs: FITRIM now supports realtime volumes
Looks good:
PATCH 5/5 fstests: xfs filesystem properties
In reply to Darrick J. WongHi all,
It would be very useful if system administrators could set properties for a given xfs filesystem to control its behavior. This we can do easily and extensibly by setting ATTR_ROOT (aka "trusted") extended attributes on the root directory. To prevent this from becoming a weird free for all, let's add some library and tooling support so that sysadmins simply run the xfs_property program to administer these properties.
If you're going to start using this code, I strongly recommend pulling from my git trees, which are linked below.
This has been running on the djcloud for months with no problems. Enjoy! Comments and questions are, as always, welcome.
--D
xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=filesystem-properties
fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=filesystem-properties
---
Commits in this patchset:
* xfs: functional testing for filesystem properties
--- common/config | 1 common/xfs | 14 ++++- doc/group-names.txt | 1 tests/generic/062 | 4 + tests/xfs/1886 | 137 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1886.out | 53 ++++++++++++++++++++ tests/xfs/1887 | 122 +++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1887.out | 46 +++++++++++++++++ tests/xfs/1888 | 66 +++++++++++++++++++++++++ tests/xfs/1888.out | 9 +++ tests/xfs/1889 | 63 +++++++++++++++++++++++ tests/xfs/1889.out | 8 +++ 12 files changed, 522 insertions(+), 2 deletions(-) create mode 100755 tests/xfs/1886 create mode 100644 tests/xfs/1886.out create mode 100755 tests/xfs/1887 create mode 100644 tests/xfs/1887.out create mode 100755 tests/xfs/1888 create mode 100644 tests/xfs/1888.out create mode 100755 tests/xfs/1889 create mode 100644 tests/xfs/1889.out
PATCH 1/1 xfs: functional testing for filesystem properties
From: Darrick J. Wong <[email protected]>
Make sure that fs property storage and retrieval actually work.
--- common/config | 1 common/xfs | 14 ++++- doc/group-names.txt | 1 tests/generic/062 | 4 + tests/xfs/1886 | 137 +++++++++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1886.out | 53 ++++++++++++++++++++ tests/xfs/1887 | 122 +++++++++++++++++++++++++++++++++++++++++++++ tests/xfs/1887.out | 46 +++++++++++++++++ tests/xfs/1888 | 66 +++++++++++++++++++++++++ tests/xfs/1888.out | 9 +++ tests/xfs/1889 | 63 +++++++++++++++++++++++ tests/xfs/1889.out | 8 +++ 12 files changed, 522 insertions(+), 2 deletions(-) create mode 100755 tests/xfs/1886 create mode 100644 tests/xfs/1886.out create mode 100755 tests/xfs/1887 create mode 100644 tests/xfs/1887.out create mode 100755 tests/xfs/1888 create mode 100644 tests/xfs/1888.out create mode 100755 tests/xfs/1889 create mode 100644 tests/xfs/1889.out
common/config +1 -0
@@ -234,6 +234,7 @@ export GZIP_PROG="$(type -P gzip)" export BTRFS_IMAGE_PROG="$(type -P btrfs-image)" export BTRFS_MAP_LOGICAL_PROG=$(type -P btrfs-map-logical) export PARTED_PROG="$(type -P parted)" +export XFS_PROPERTY_PROG="$(type -P xfs_property)" # use 'udevadm settle' or 'udevsettle' to wait for lv to be settled. # newer systems have udevadm command but older systems like RHEL5 don't.
common/xfs +12 -2
@@ -1323,8 +1323,8 @@ _require_xfs_spaceman_command() testfile=$TEST_DIR/$$.xfs_spaceman touch $testfile case $command in - "health") - testio=`$XFS_SPACEMAN_PROG -c "health $param" $TEST_DIR 2>&1` + "health"|"listfsprops") + testio=`$XFS_SPACEMAN_PROG -c "$command $param" $TEST_DIR 2>&1` param_checked=1 ;; *)
@@ -1860,3 +1860,13 @@ _xfs_statfs_field() { $XFS_IO_PROG -c 'statfs' "$1" | grep -E "$2" | cut -d ' ' -f 3 } + +# Wipe all filesystem properties from an xfs filesystem. The sole argument +# must be the root directory of a filesystem. +_wipe_xfs_properties() +{ + getfattr --match="^trusted.xfs:" --absolute-names --dump --encoding=hex "$1" | \ + grep '=' | sed -e 's/=.*$//g' | while read name; do + setfattr --remove="$name" "$1" + done +}
doc/group-names.txt +1 -0
@@ -56,6 +56,7 @@ fiexchange XFS_IOC_EXCHANGE_RANGE ioctl freeze filesystem freeze tests fsck general fsck tests fsmap FS_IOC_GETFSMAP ioctl +fsproperties Filesystem properties fsr XFS free space reorganizer fuzzers filesystem fuzz tests growfs increasing the size of a filesystem
tests/generic/062 +4 -0
@@ -75,6 +75,10 @@ fi _require_attrs $ATTR_MODES +# Wipe all xfs filesystem properties (which are rootdir xattrs) before we dump +# them all later. +test $FSTYP = "xfs" && _wipe_xfs_properties $SCRATCH_MNT + for nsp in $ATTR_MODES; do for inode in reg dir lnk dev/b dev/c dev/p; do
tests/xfs/1886 new file +137 -0
137 lines not on this page · raw message
tests/xfs/1886.out new file +53 -0
53 lines not on this page · raw message
tests/xfs/1887 new file +122 -0
122 lines not on this page · raw message
tests/xfs/1887.out new file +46 -0
46 lines not on this page · raw message
tests/xfs/1888 new file +66 -0
66 lines not on this page · raw message
tests/xfs/1888.out new file +9 -0
@@ -0,0 +1,9 @@ +QA output created by 1888 +autofsck=repair +failed to get attr xfs:autofsck on inode XXX: No data available +autofsck=none +autofsck=check +autofsck=optimize +autofsck=repair +autofsck=none +autofsck=repair
tests/xfs/1889 new file +63 -0
63 lines not on this page · raw message
tests/xfs/1889.out new file +8 -0
@@ -0,0 +1,8 @@ +QA output created by 1889 +Info: TEST_DIR/1889.mount: Checking and repairing per autofsck directive. +Info: TEST_DIR/1889.mount: Disabling scrub per autofsck directive. +Info: TEST_DIR/1889.mount: Checking per autofsck directive. +Info: TEST_DIR/1889.mount: Optimizing per autofsck directive. +Info: TEST_DIR/1889.mount: Checking and repairing per autofsck directive. +Info: TEST_DIR/1889.mount: Disabling scrub per autofsck directive. +Info: TEST_DIR/1889.mount: Checking and repairing per autofsck directive.
Re: [PATCH 1/1] xfs: functional testing for filesystem properties
Looks good: