Skip to contentSkip to conversation

fstest: Crashmonkey rename tests ported to xfstest

PATCH In review Arvind Raghavan 2 participants 5 messages Apr 24, 2020 – Apr 28, 2020

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fstest: Crashmonkey rename tests ported to xfstest

This patch aims to add tests to the xfstest suite to check whether renamed files recover after a crash. These test cases were generated by CrashMonkey, a crash-consistency testing framework built at the SASLab at UT Austin.

This patch batches 37 tests into a single test, each of which creates a file, renames it, syncs one of the files/parent directories, and ensures that the synced file/dir is equivalent before and after a crash.

This test takes around 12-15 seconds to run locally and is thus placed in the 'quick' group.

---
 tests/generic/484     | 179 ++++++++++++++++++++++++++++++++++++++++++
 tests/generic/484.out |   2 +
 tests/generic/group   |   1 +
 3 files changed, 182 insertions(+)
 create mode 100755 tests/generic/484
 create mode 100644 tests/generic/484.out
tests/generic/484 new file +179 -0
new file mode 100755
index 00000000..b27d828d
@@ -0,0 +1,179 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2020 The University of Texas at Austin.  All Rights Reserved.
+#
+# FS QA Test 484
+#
+# Test case created by CrashMonkey
+#
+# Test if we create a rename a file and persist it that the
+# appropriate files exist.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	_cleanup_flakey
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmflakey
+
+# 256MB in byte
+fssize=$((2**20 * 256))
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_scratch_nocheck
+_require_dm_target flakey
+
+# initialize scratch device
+_scratch_mkfs_sized $fssize >> $seqres.full 2>&1
+_require_metadata_journaling $SCRATCH_DEV
+_init_flakey
+
+stat_opt='-c  %n - blocks: %b size: %s inode: %i links: %h'
+
+# Usage: general_stat [--data-only] file1 [--data-only] [file2] ..
+# If the --data-only flag precedes a file, then only that file's
+# data will be printed. If a file is synced, then general_stat
+# should provide identical output before and after a crash.
+general_stat() {
+	local data_only="false"
+	while (( "$#" )); do
+		case $1 in
+			--data-only)
+				data_only="true"
+				;;
+			*)
+				local path="$1"
+				echo "-- $path --"
+				if [ ! -e "$path" ]; then
+					echo "Doesn't exist!"
+				elif [ "$data_only" = "true" ] && [ -d "$path" ]; then
+					echo "Directory Data"
+					[ -z "$(ls -A $path)" ] || ls -1 "$path" | sort
+				elif [ "$data_only" = "true" ]; then
+					echo "File Data"
+					od "$path"
+				elif [ -d "$path" ]; then
+					echo "Directory Metadata"
+					stat "$stat_opt" "$path"
+					echo "Directory Data"
+					[ -z "$(ls -A $path)" ] || ls -1 "$path" | sort
+				else
+					echo "File Metadata"
+					stat "$stat_opt" "$path"
+					echo "File Data"
+					od "$path"
+				fi
+				data_only="false"
+				;;
+		esac
+		shift
+	done
+}
+
+check_consistency()
+{
+	before=$(general_stat "$@")
+	_flakey_drop_and_remount
+	after=$(general_stat "$@")
+
+	if [ "$before" != "$after" ]; then
+		echo -e "Before:\n$before"
+		echo -e "After:\n$after"
+	fi
+
+	# Using _unmount_flakey nondeterministically fails here with
+	# "target is busy". Calling 'sync' doesn't fix the issue. Lazy
+	# unmount waits for lingering processes to exit.
+	$UMOUNT_PROG -l $SCRATCH_MNT
+	_check_scratch_fs $FLAKEY_DEV
+}
+
+clean_dir()
+{
+	_mount_flakey
+	rm -rf $(find $SCRATCH_MNT/* | grep -v "lost+found")
+	$UMOUNT_PROG -l $SCRATCH_MNT
+}
+
+do_fsync_check() {
+	local file="$1"
+	local sync_op="$2"
+
+	if [ $sync_op == "fdatasync" ]; then
+		$XFS_IO_PROG -c "fdatasync" $file
+		check_consistency --data-only $file
+	else
+		$XFS_IO_PROG -c "fsync" $file
+		check_consistency $file
+	fi
+}
+
+# Wraps mv, deletes dest dir if exists.
+rename() {
+	[ -d $1 ] && [ -d $2 ] && rm -rf $2
+	mv $1 $2
+}
+
+function rename_template() {
+	local file1="$1"
+	local file2="$2"
+	local fsync_file="$3"
+	local fsync_command="$4"
+
+	_mount_flakey
+	mkdir -p $(dirname $file1)
+	touch $file1
+	mkdir -p $(dirname $file2)
+	rename $file1 $file2
+	do_fsync_check $fsync_file $fsync_command
+	clean_dir
+}
+
+file1_options=(
+	"$SCRATCH_MNT/A/C/bar"
+	"$SCRATCH_MNT/A/bar"
+	"$SCRATCH_MNT/A/foo"
+	"$SCRATCH_MNT/foo"
+)
+file2_options=(
+	"$SCRATCH_MNT/A/C/bar"
+	"$SCRATCH_MNT/A/bar"
+	"$SCRATCH_MNT/bar"
+)
+
+for file1 in ${file1_options[@]}; do
+	for file2 in ${file2_options[@]}; do
+		[ "$file1" == "$file2" ] && continue
+		to_sync=("$(dirname $file1)" "$(dirname $file2)")
+		uniques=($(for v in ${to_sync[@]}; do echo $v; done | sort -u))
+		for fsync_file in ${uniques[@]}; do
+			for fsync_command in fsync fdatasync; do
+				rename_template $file1 $file2 $fsync_file $fsync_command
+			done
+		done
+	done
+done
+
+# Success, all done
+echo "Silence is golden"
+status=0
+exit
tests/generic/484.out new file +2 -0
new file mode 100644
index 00000000..94f2f0bd
@@ -0,0 +1,2 @@
+QA output created by 484
+Silence is golden
tests/generic/group +1 -0
index 718575ba..987ad2a0 100644
@@ -486,6 +486,7 @@
 481 auto quick log metadata
 482 auto metadata replay thin
 483 auto quick log metadata
+484 auto quick log metadata
 485 auto quick insert
 486 auto quick attr
 487 auto quick eio
Signature · 1 line
2.20.1
#1 lore raw
Quoted 6 lines · Arvind Raghavan

This patch aims to add tests to the xfstest suite to check whether renamed files recover after a crash. These test cases were generated by CrashMonkey, a crash-consistency testing framework built at the SASLab at UT Austin.

Quoted 8 lines

This patch batches 37 tests into a single test, each of which creates a file, renames it, syncs one of the files/parent directories, and ensures that the synced file/dir is equivalent before and after a crash.

This test takes around 12-15 seconds to run locally and is thus placed in the 'quick' group.

Nice work. Some things to improve...

Quoted 50 lines
---
 tests/generic/484     | 179 ++++++++++++++++++++++++++++++++++++++++++
 tests/generic/484.out |   2 +
 tests/generic/group   |   1 +
 3 files changed, 182 insertions(+)
 create mode 100755 tests/generic/484
 create mode 100644 tests/generic/484.out
tests/generic/484 new file +36 -0
new file mode 100755
index 00000000..b27d828d
@@ -0,0 +1,179 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2020 The University of Texas at Austin.  All Rights Reserved.
+#
+# FS QA Test 484
+#
+# Test case created by CrashMonkey
+#
+# Test if we create a rename a file and persist it that the
+# appropriate files exist.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1       # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+       _cleanup_flakey
+       cd /
+       rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmflakey
+
+# 256MB in byte
+fssize=$((2**20 * 256))
+
+# remove previous $seqres.full before test

tests/generic/484 line 46

Quoted 10 lines
tests/generic/484 new file +10 -0
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_scratch_nocheck
+_require_dm_target flakey
+
+# initialize scratch device
+_scratch_mkfs_sized $fssize >> $seqres.full 2>&1

Why limit the fs size?
In general this results is test coverage for non real life filesystems, so sub-optimal IMO.

tests/generic/484 line 55

Quoted 9 lines
tests/generic/484 new file +9 -0
+_require_metadata_journaling $SCRATCH_DEV
+_init_flakey
+
+stat_opt='-c  %n - blocks: %b size: %s inode: %i links: %h'
+
+# Usage: general_stat [--data-only] file1 [--data-only] [file2] ..
+# If the --data-only flag precedes a file, then only that file's
+# data will be printed. If a file is synced, then general_stat
+# should provide identical output before and after a crash.

This helper seems overly complicated for the use case in this test, but it is local to the test. I understand it is auto generated for other tests but that's not the way to go.
I suggest you look into using existing FSSUM_PROG tool.

Quoted 5 lines
tests/generic/484 new file +5 -0
+general_stat() {
+       local data_only="false"
+       while (( "$#" )); do
+               case $1 in
+                       --data-only)

tests/generic/484 line 70

Quoted 10 lines
tests/generic/484 new file +10 -0
+                               data_only="true"
+                               ;;
+                       *)
+                               local path="$1"
+                               echo "-- $path --"
+                               if [ ! -e "$path" ]; then
+                                       echo "Doesn't exist!"
+                               elif [ "$data_only" = "true" ] && [ -d "$path" ]; then
+                                       echo "Directory Data"
+                                       [ -z "$(ls -A $path)" ] || ls -1 "$path" | sort

[ -z "$(ls -A $path)" ] || seems to add nothing.

Quoted 25 lines
tests/generic/484 new file +25 -0
+                               elif [ "$data_only" = "true" ]; then
+                                       echo "File Data"
+                                       od "$path"
+                               elif [ -d "$path" ]; then
+                                       echo "Directory Metadata"
+                                       stat "$stat_opt" "$path"
+                                       echo "Directory Data"
+                                       [ -z "$(ls -A $path)" ] || ls -1 "$path" | sort
+                               else
+                                       echo "File Metadata"
+                                       stat "$stat_opt" "$path"
+                                       echo "File Data"
+                                       od "$path"
+                               fi
+                               data_only="false"
+                               ;;
+               esac
+               shift
+       done
+}
+
+check_consistency()
+{
+       before=$(general_stat "$@")
+       _flakey_drop_and_remount

tests/generic/484 line 105

Quoted 10 lines
tests/generic/484 new file +10 -0
+       after=$(general_stat "$@")
+
+       if [ "$before" != "$after" ]; then
+               echo -e "Before:\n$before"
+               echo -e "After:\n$after"
+       fi
+
+       # Using _unmount_flakey nondeterministically fails here with
+       # "target is busy". Calling 'sync' doesn't fix the issue. Lazy
+       # unmount waits for lingering processes to exit.

No way. None of the other flaky tests have this kludge.
You must be doing something wrong, or there is another bug
in the infrastructure that needs to be fixed instead of being papered over.

tests/generic/484 line 113

Quoted 8 lines
tests/generic/484 new file +8 -0
+       $UMOUNT_PROG -l $SCRATCH_MNT
+       _check_scratch_fs $FLAKEY_DEV
+}
+
+clean_dir()
+{
+       _mount_flakey
+       rm -rf $(find $SCRATCH_MNT/* | grep -v "lost+found")

Use testdir=$SCRATCH_MNT/testdir for base of your test and you won't need this blacklisting of lost+found.

tests/generic/484 line 114

Quoted 1 line
tests/generic/484 new file +1 -0
+       $UMOUNT_PROG -l $SCRATCH_MNT

And maybe this is what you are doing wrong??
Why is this unmount lazy?

tests/generic/484 line 122

Quoted 8 lines
tests/generic/484 new file +8 -0
+}
+
+do_fsync_check() {
+       local file="$1"
+       local sync_op="$2"
+
+       if [ $sync_op == "fdatasync" ]; then
+               $XFS_IO_PROG -c "fdatasync" $file

You don't really need to put this statement inside if/else

tests/generic/484 line 123

Quoted 1 line
tests/generic/484 new file +1 -0
+               check_consistency --data-only $file

And you could probably also pass sync_op to check_consistency() instead of converting it here. In other words, this helper does not do much, so you could get rid of it. up to you.

Thanks,
Amir.

#2 lore raw

Thanks for the feedback!

Quoted 76 lines · Amir Goldstein
Quoted 6 lines · Arvind Raghavan

This patch aims to add tests to the xfstest suite to check whether renamed files recover after a crash. These test cases were generated by CrashMonkey, a crash-consistency testing framework built at the SASLab at UT Austin.

Quoted 8 lines

This patch batches 37 tests into a single test, each of which creates a file, renames it, syncs one of the files/parent directories, and ensures that the synced file/dir is equivalent before and after a crash.

This test takes around 12-15 seconds to run locally and is thus placed in the 'quick' group.

Nice work. Some things to improve...

Quoted 56 lines
---
 tests/generic/484     | 179 ++++++++++++++++++++++++++++++++++++++++++
 tests/generic/484.out |   2 +
 tests/generic/group   |   1 +
 3 files changed, 182 insertions(+)
 create mode 100755 tests/generic/484
 create mode 100644 tests/generic/484.out
tests/generic/484 new file +42 -0
new file mode 100755
index 00000000..b27d828d
@@ -0,0 +1,179 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2020 The University of Texas at Austin.  All Rights Reserved.
+#
+# FS QA Test 484
+#
+# Test case created by CrashMonkey
+#
+# Test if we create a rename a file and persist it that the
+# appropriate files exist.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1       # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+       _cleanup_flakey
+       cd /
+       rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmflakey
+
+# 256MB in byte
+fssize=$((2**20 * 256))
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_scratch_nocheck
Quoted 8 lines

tests/generic/484 line 46

Quoted 4 lines
tests/generic/484 new file +4 -0
+_require_dm_target flakey
+
+# initialize scratch device
+_scratch_mkfs_sized $fssize >> $seqres.full 2>&1

Why limit the fs size?
In general this results is test coverage for non real life filesystems, so sub-optimal IMO.

It was an artifact from the originally Crashmonkey test harness, but we can go ahead and remove it now.

Quoted 6 lines
Quoted 6 lines
tests/generic/484 new file +6 -0
+_require_metadata_journaling $SCRATCH_DEV
+_init_flakey
+
+stat_opt='-c  %n - blocks: %b size: %s inode: %i links: %h'
+
+# Usage: general_stat [--data-only] file1 [--data-only] [file2] ..
Quoted 8 lines

tests/generic/484 line 55

Quoted 3 lines
tests/generic/484 new file +3 -0
+# If the --data-only flag precedes a file, then only that file's
+# data will be printed. If a file is synced, then general_stat
+# should provide identical output before and after a crash.

This helper seems overly complicated for the use case in this test, but it is local to the test. I understand it is auto generated for other tests but that's not the way to go.
I suggest you look into using existing FSSUM_PROG tool.

I agree that seems like a more elegant solution. We essentially have four cases that we need to check:

(1) fsync file (meta + data)
(2) fdatasync file
(3) fsync dir
(4) fdatasync dir

FSSUM_PROG works great for (3) and (4), since it looks like its intended to be used only on directories. md5sum works well for (2), but it doesn't capture the metadata needed for (1). FSSUM_PROG has a way to print a full "manifest", which contains the metadata/data hashes for each file, so I believe we can grep/cut it out of the manifest output to test for (1). I'll send out a V2 patch if you think this seems like a reasonable approach.

Also, see below for a note on FSSUM_PROG.

Quoted 49 lines
Quoted 5 lines
tests/generic/484 new file +5 -0
+general_stat() {
+       local data_only="false"
+       while (( "$#" )); do
+               case $1 in
+                       --data-only)

tests/generic/484 line 70

Quoted 10 lines
tests/generic/484 new file +10 -0
+                               data_only="true"
+                               ;;
+                       *)
+                               local path="$1"
+                               echo "-- $path --"
+                               if [ ! -e "$path" ]; then
+                                       echo "Doesn't exist!"
+                               elif [ "$data_only" = "true" ] && [ -d "$path" ]; then
+                                       echo "Directory Data"
+                                       [ -z "$(ls -A $path)" ] || ls -1 "$path" | sort

[ -z "$(ls -A $path)" ] || seems to add nothing.

Quoted 31 lines
tests/generic/484 new file +31 -0
+                               elif [ "$data_only" = "true" ]; then
+                                       echo "File Data"
+                                       od "$path"
+                               elif [ -d "$path" ]; then
+                                       echo "Directory Metadata"
+                                       stat "$stat_opt" "$path"
+                                       echo "Directory Data"
+                                       [ -z "$(ls -A $path)" ] || ls -1 "$path" | sort
+                               else
+                                       echo "File Metadata"
+                                       stat "$stat_opt" "$path"
+                                       echo "File Data"
+                                       od "$path"
+                               fi
+                               data_only="false"
+                               ;;
+               esac
+               shift
+       done
+}
+
+check_consistency()
+{
+       before=$(general_stat "$@")
+       _flakey_drop_and_remount
+       after=$(general_stat "$@")
+
+       if [ "$before" != "$after" ]; then
+               echo -e "Before:\n$before"
+               echo -e "After:\n$after"
+       fi
Quoted 8 lines

tests/generic/484 line 105

Quoted 4 lines
tests/generic/484 new file +4 -0
+
+       # Using _unmount_flakey nondeterministically fails here with
+       # "target is busy". Calling 'sync' doesn't fix the issue. Lazy
+       # unmount waits for lingering processes to exit.

No way. None of the other flaky tests have this kludge.
You must be doing something wrong, or there is another bug
in the infrastructure that needs to be fixed instead of being papered over.

Yeah, this was a bug on my end. I've fixed it and removed all of the lazy unmounts.

Quoted 28 lines

tests/generic/484 line 113

Quoted 8 lines
tests/generic/484 new file +8 -0
+       $UMOUNT_PROG -l $SCRATCH_MNT
+       _check_scratch_fs $FLAKEY_DEV
+}
+
+clean_dir()
+{
+       _mount_flakey
+       rm -rf $(find $SCRATCH_MNT/* | grep -v "lost+found")

Use testdir=$SCRATCH_MNT/testdir for base of your test and you won't need this blacklisting of lost+found.

tests/generic/484 line 114

Quoted 1 line
tests/generic/484 new file +1 -0
+       $UMOUNT_PROG -l $SCRATCH_MNT

And maybe this is what you are doing wrong??
Why is this unmount lazy?

tests/generic/484 line 122

Quoted 8 lines
tests/generic/484 new file +8 -0
+}
+
+do_fsync_check() {
+       local file="$1"
+       local sync_op="$2"
+
+       if [ $sync_op == "fdatasync" ]; then
+               $XFS_IO_PROG -c "fdatasync" $file

You don't really need to put this statement inside if/else

Quoted 8 lines

tests/generic/484 line 123

Quoted 1 line
tests/generic/484 new file +1 -0
+               check_consistency --data-only $file

And you could probably also pass sync_op to check_consistency() instead of converting it here. In other words, this helper does not do much, so you could get rid of it. up to you.

Thanks,
Amir.

I believe there is a minor bug in the fssum program. Below are the relevant lines from the help output.

usage: fssum <options> <path>

    options:
    ...
    -[ugoamcdxe]: specify which fields to include in checksum
                  calculation.
         ...
         x      : include xattrs
         ...
    ...
    -x path     : exclude path when building checksum
                  (multiple ok)
    ...

There seems to be a duplicate flag which prevents the user from passing in -x to include the xattrs field in checksum calculation. This can be fixed with a simple patch that changes the latter flag to "-i path", and updates the documentation to say "ignores path when building checksum". I've got such a commit built locally than I can send out in a patch if you think it makes sense. None of the tests in the repo use the flag as AFAIK, but we will end up using it for the xattr tests.

#3 lore raw
Quoted 124 lines · Arvind Raghavan

Thanks for the feedback!

Quoted 76 lines · Amir Goldstein
Quoted 6 lines · Arvind Raghavan

This patch aims to add tests to the xfstest suite to check whether renamed files recover after a crash. These test cases were generated by CrashMonkey, a crash-consistency testing framework built at the SASLab at UT Austin.

Quoted 8 lines

This patch batches 37 tests into a single test, each of which creates a file, renames it, syncs one of the files/parent directories, and ensures that the synced file/dir is equivalent before and after a crash.

This test takes around 12-15 seconds to run locally and is thus placed in the 'quick' group.

Nice work. Some things to improve...

Quoted 56 lines
---
 tests/generic/484     | 179 ++++++++++++++++++++++++++++++++++++++++++
 tests/generic/484.out |   2 +
 tests/generic/group   |   1 +
 3 files changed, 182 insertions(+)
 create mode 100755 tests/generic/484
 create mode 100644 tests/generic/484.out
tests/generic/484 new file +42 -0
new file mode 100755
index 00000000..b27d828d
@@ -0,0 +1,179 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2020 The University of Texas at Austin.  All Rights Reserved.
+#
+# FS QA Test 484
+#
+# Test case created by CrashMonkey
+#
+# Test if we create a rename a file and persist it that the
+# appropriate files exist.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1       # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+       _cleanup_flakey
+       cd /
+       rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/dmflakey
+
+# 256MB in byte
+fssize=$((2**20 * 256))
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+_supported_fs generic
+_supported_os Linux
+_require_scratch_nocheck
Quoted 8 lines

tests/generic/484 line 46

Quoted 4 lines
tests/generic/484 new file +4 -0
+_require_dm_target flakey
+
+# initialize scratch device
+_scratch_mkfs_sized $fssize >> $seqres.full 2>&1

Why limit the fs size?
In general this results is test coverage for non real life filesystems, so sub-optimal IMO.

It was an artifact from the originally Crashmonkey test harness, but we can go ahead and remove it now.

Quoted 6 lines
Quoted 6 lines
tests/generic/484 new file +6 -0
+_require_metadata_journaling $SCRATCH_DEV
+_init_flakey
+
+stat_opt='-c  %n - blocks: %b size: %s inode: %i links: %h'
+
+# Usage: general_stat [--data-only] file1 [--data-only] [file2] ..
Quoted 8 lines

tests/generic/484 line 55

Quoted 3 lines
tests/generic/484 new file +3 -0
+# If the --data-only flag precedes a file, then only that file's
+# data will be printed. If a file is synced, then general_stat
+# should provide identical output before and after a crash.

This helper seems overly complicated for the use case in this test, but it is local to the test. I understand it is auto generated for other tests but that's not the way to go.
I suggest you look into using existing FSSUM_PROG tool.

I agree that seems like a more elegant solution. We essentially have four cases that we need to check:

(1) fsync file (meta + data)
(2) fdatasync file
(3) fsync dir
(4) fdatasync dir

FSSUM_PROG works great for (3) and (4), since it looks like its intended to be used only on directories. md5sum works well for (2), but it doesn't capture the metadata needed for (1). FSSUM_PROG has a way to print a full "manifest", which contains the metadata/data hashes for each file, so I believe we can grep/cut it out of the manifest output to test for (1). I'll send out a V2 patch if you think this seems like a reasonable approach.

Quoted 1 line

Also, see below for a note on FSSUM_PROG.

Upon looking more closely into fssum, I've changed my mind. It looks like fssum walks a directory recursively, adding all file names, metadata, and data into a checksum. This is useful for a filesytem with stronger guarantees, but under POSIX standards, the only guarantees for fsycing a dir are that its immediate dirents are correct and that its metadata is synced, not necessarily the metadata of its children.

Because of this, I believe it would be more readable and concise to rewrite this function instead of grep/cutting the output of fssum. I can clean it up by removing the echos and case/shifting and turn it into something that concisely expresses the following:

fsync file -> stat file, md5sum file
fdatasync file -> md5sum file
fsync dir -> stat dir, ls dir
fdatasync dir -> ls dir

Would those changes make more sense?

Quoted 122 lines
Quoted 49 lines
Quoted 5 lines
tests/generic/484 new file +5 -0
+general_stat() {
+       local data_only="false"
+       while (( "$#" )); do
+               case $1 in
+                       --data-only)

tests/generic/484 line 70

Quoted 10 lines
tests/generic/484 new file +10 -0
+                               data_only="true"
+                               ;;
+                       *)
+                               local path="$1"
+                               echo "-- $path --"
+                               if [ ! -e "$path" ]; then
+                                       echo "Doesn't exist!"
+                               elif [ "$data_only" = "true" ] && [ -d "$path" ]; then
+                                       echo "Directory Data"
+                                       [ -z "$(ls -A $path)" ] || ls -1 "$path" | sort

[ -z "$(ls -A $path)" ] || seems to add nothing.

Quoted 31 lines
tests/generic/484 new file +31 -0
+                               elif [ "$data_only" = "true" ]; then
+                                       echo "File Data"
+                                       od "$path"
+                               elif [ -d "$path" ]; then
+                                       echo "Directory Metadata"
+                                       stat "$stat_opt" "$path"
+                                       echo "Directory Data"
+                                       [ -z "$(ls -A $path)" ] || ls -1 "$path" | sort
+                               else
+                                       echo "File Metadata"
+                                       stat "$stat_opt" "$path"
+                                       echo "File Data"
+                                       od "$path"
+                               fi
+                               data_only="false"
+                               ;;
+               esac
+               shift
+       done
+}
+
+check_consistency()
+{
+       before=$(general_stat "$@")
+       _flakey_drop_and_remount
+       after=$(general_stat "$@")
+
+       if [ "$before" != "$after" ]; then
+               echo -e "Before:\n$before"
+               echo -e "After:\n$after"
+       fi
Quoted 8 lines

tests/generic/484 line 105

Quoted 4 lines
tests/generic/484 new file +4 -0
+
+       # Using _unmount_flakey nondeterministically fails here with
+       # "target is busy". Calling 'sync' doesn't fix the issue. Lazy
+       # unmount waits for lingering processes to exit.

No way. None of the other flaky tests have this kludge.
You must be doing something wrong, or there is another bug
in the infrastructure that needs to be fixed instead of being papered over.

Yeah, this was a bug on my end. I've fixed it and removed all of the lazy unmounts.

Quoted 28 lines

tests/generic/484 line 113

Quoted 8 lines
tests/generic/484 new file +8 -0
+       $UMOUNT_PROG -l $SCRATCH_MNT
+       _check_scratch_fs $FLAKEY_DEV
+}
+
+clean_dir()
+{
+       _mount_flakey
+       rm -rf $(find $SCRATCH_MNT/* | grep -v "lost+found")

Use testdir=$SCRATCH_MNT/testdir for base of your test and you won't need this blacklisting of lost+found.

tests/generic/484 line 114

Quoted 1 line
tests/generic/484 new file +1 -0
+       $UMOUNT_PROG -l $SCRATCH_MNT

And maybe this is what you are doing wrong??
Why is this unmount lazy?

tests/generic/484 line 122

Quoted 8 lines
tests/generic/484 new file +8 -0
+}
+
+do_fsync_check() {
+       local file="$1"
+       local sync_op="$2"
+
+       if [ $sync_op == "fdatasync" ]; then
+               $XFS_IO_PROG -c "fdatasync" $file

You don't really need to put this statement inside if/else

Quoted 8 lines

tests/generic/484 line 123

Quoted 1 line
tests/generic/484 new file +1 -0
+               check_consistency --data-only $file

And you could probably also pass sync_op to check_consistency() instead of converting it here. In other words, this helper does not do much, so you could get rid of it. up to you.

Thanks,
Amir.

I believe there is a minor bug in the fssum program. Below are the relevant lines from the help output.

usage: fssum <options> <path>

    options:
    ...
    -[ugoamcdxe]: specify which fields to include in checksum
                  calculation.
         ...
         x      : include xattrs
         ...
    ...
    -x path     : exclude path when building checksum
                  (multiple ok)
    ...

There seems to be a duplicate flag which prevents the user from passing in -x to include the xattrs field in checksum calculation. This can be fixed with a simple patch that changes the latter flag to "-i path", and updates the documentation to say "ignores path when building checksum". I've got such a commit built locally than I can send out in a patch if you think it makes sense. None of the tests in the repo use the flag as AFAIK, but we will end up using it for the xattr tests.

#4 lore raw
Quoted 38 lines
Quoted 22 lines
Quoted 4 lines

This helper seems overly complicated for the use case in this test, but it is local to the test. I understand it is auto generated for other tests but that's not the way to go.
I suggest you look into using existing FSSUM_PROG tool.

I agree that seems like a more elegant solution. We essentially have four cases that we need to check:

(1) fsync file (meta + data)
(2) fdatasync file
(3) fsync dir
(4) fdatasync dir

FSSUM_PROG works great for (3) and (4), since it looks like its intended to be used only on directories. md5sum works well for (2), but it doesn't capture the metadata needed for (1). FSSUM_PROG has a way to print a full "manifest", which contains the metadata/data hashes for each file, so I believe we can grep/cut it out of the manifest output to test for (1). I'll send out a V2 patch if you think this seems like a reasonable approach.

Quoted 1 line

Also, see below for a note on FSSUM_PROG.

Upon looking more closely into fssum, I've changed my mind. It looks like fssum walks a directory recursively, adding all file names, metadata, and data into a checksum. This is useful for a filesytem with stronger guarantees, but under POSIX standards, the only guarantees for fsycing a dir are that its immediate dirents are correct and that its metadata is synced, not necessarily the metadata of its children.

Because of this, I believe it would be more readable and concise to rewrite this function instead of grep/cutting the output of fssum. I can clean it up by removing the echos and case/shifting and turn it into something that concisely expresses the following:

Quoted 7 lines

fsync file -> stat file, md5sum file
fdatasync file -> md5sum file
fsync dir -> stat dir, ls dir
fdatasync dir -> ls dir

Would those changes make more sense?

Either this or add support for single file and non recursive mode to fssum. Either would be fine by me, but latter would make your tests overall better.

[...]

Quoted 17 lines
Quoted 17 lines

I believe there is a minor bug in the fssum program. Below are the relevant lines from the help output.

usage: fssum <options> <path>

    options:
    ...
    -[ugoamcdxe]: specify which fields to include in checksum
                  calculation.
         ...
         x      : include xattrs
         ...
    ...
    -x path     : exclude path when building checksum
                  (multiple ok)
    ...
Quoted 8 lines
Quoted 8 lines

There seems to be a duplicate flag which prevents the user from passing in -x to include the xattrs field in checksum calculation. This can be fixed with a simple patch that changes the latter flag to "-i path", and updates the documentation to say "ignores path when building checksum". I've got such a commit built locally than I can send out in a patch if you think it makes sense. None of the tests in the repo use the flag as AFAIK, but we will end up using it for the xattr tests.

Sure. please send the patch.
Also need to fix the documentation of default flags:
"...The default field mask is ugoamCdES" it omits x.

Thanks,
Amir.