Skip to contentSkip to conversation

generic: add a test that stresses metadata eviction

PATCH Applied Christoph Hellwig 5 participants 8 messages May 18 – May 19

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 generic: add a test that stresses metadata eviction

The fsverity test generic/759 could be used to reproduce a regression in the xfs buffer hashing. Carlos Maiolino figured out that even removing the fsverity-specific bits would reproduce the issue. This test is thus a copy of generic/759 with the fsverity bits striped out, and run 5 times as that seems to be the threshold at which the issue reproduces reliably.

---

Not sure how to best credit Eric for the original patch and Carlos for the extraction, any ideas?

 tests/generic/4201     | 108 +++++++++++++++++++++++++++++++++++++++++
 tests/generic/4201.out |   2 +
 2 files changed, 110 insertions(+)
 create mode 100755 tests/generic/4201
 create mode 100644 tests/generic/4201.out
tests/generic/4201 new file +108 -0
new file mode 100755
index 000000000000..fc367beeadaf
@@ -0,0 +1,108 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright 2019 Google LLC
+#
+# FS QA Test generic/4201
+#
+# Race dropping file systems caches vs fsstress an repeated reading of files.
+# Based on generic/579 with the fsverity bits stripped out, and an increased
+# number of runs.
+#
+. ./common/preamble
+_begin_fstest auto stress
+
+stop_run()
+{
+	# Stop all subprocesses.
+	_kill_fsstress
+	touch $tmp.done
+	wait
+}
+
+_cleanup()
+{
+	stop_run
+	rm -f $tmp.*
+}
+
+. ./common/filter
+
+fsv_file_size=10000000
+nproc_enabler=$((4 * LOAD_FACTOR))
+nproc_reader=$((6 * LOAD_FACTOR))
+nproc_stress=$((3 * LOAD_FACTOR))
+runtime=$((20 * TIME_FACTOR))
+
+run_test()
+{
+	rm -rf $tmp.done
+
+	# Create the test files and start the fs-verity enabler processes.
+	for ((proc = 0; proc < nproc_enabler; proc++)); do
+		orig_file=$SCRATCH_MNT/orig$proc
+		fsv_file=$SCRATCH_MNT/fsv$proc
+		head -c $fsv_file_size /dev/urandom > $orig_file
+		(
+			while [ ! -e $tmp.done ]; do
+				rm -f $fsv_file
+				cp $orig_file $fsv_file
+				# Give the readers some time to read from the file.
+				sleep 0.$((RANDOM % 100))
+			done
+		) &
+	done
+
+	# Start the reader processes.
+	for ((proc = 0; proc < nproc_reader; proc++)); do
+		(
+			while [ ! -e $tmp.done ]; do
+				# Choose a random file for each iteration, so that
+				# sometimes multiple processes read from the same file.
+				i=$((RANDOM % nproc_enabler))
+				orig_file=$SCRATCH_MNT/orig$i
+				fsv_file=$SCRATCH_MNT/fsv$i
+
+				# After the copy from $orig_file to $fsv_file has
+				# completed, the contents of these two files should
+				# match, regardless of whether verity has been enabled
+				# or not yet (or is currently being enabled).
+				cmp $orig_file $fsv_file |& _filter_scratch | \
+					sed -e "s/'//g" | \
+					grep -v "SCRATCH_MNT/fsv$i: No such file or directory" | \
+					grep -v "EOF on SCRATCH_MNT/fsv$i"
+				done
+		) &
+	done
+
+	# Start a process that occasionally runs 'sync && drop_caches'.  This makes more
+	# reads go through fs-verity for real, rather than just returning pagecache.
+	(
+		while [ ! -e $tmp.done ]; do
+			sleep 2.$((RANDOM % 100))
+			_scratch_sync && echo 3 > /proc/sys/vm/drop_caches
+		done
+	) &
+
+	# Start the fsstress processes.
+	_run_fsstress_bg -p $nproc_stress -l 0 -d $SCRATCH_MNT/stressdir
+
+	# Run for a while.
+	sleep $runtime
+
+	stop_run
+}
+
+for i in $(seq 1 5); do
+	echo "Starting run $i" >> $seqres.full
+
+	_scratch_mkfs &>> $seqres.full
+	_scratch_mount
+	run_test
+	_scratch_unmount
+done
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/generic/4201.out new file +2 -0
new file mode 100644
index 000000000000..2c3f3d42ff3f
@@ -0,0 +1,2 @@
+QA output created by 4201
+Silence is golden
Signature · 1 line
2.53.0
#1 lore raw
Quoted 7 lines · Christoph Hellwig

The fsverity test generic/759 could be used to reproduce a regression in the xfs buffer hashing. Carlos Maiolino figured out that even removing the fsverity-specific bits would reproduce the issue. This test is thus a copy of generic/759 with the fsverity bits striped out, and run 5 times as that seems to be the threshold at which the issue reproduces reliably.

Quoted 5 lines

---

Not sure how to best credit Eric for the original patch and Carlos for the extraction, any ideas?

To be fair, Andrey initially suggested stripping off the fs-verity parts, IIRC because he wanted to isolate his own series from this crash. My role was mostly reproduce it and hunt the bug down. Risking to come with yet another tag, perhaps something like:

Based-on-generic-579-by: <Eric...>

It doesn't look ugly to me, but who knows...

Quoted 133 lines
 tests/generic/4201     | 108 +++++++++++++++++++++++++++++++++++++++++
 tests/generic/4201.out |   2 +
 2 files changed, 110 insertions(+)
 create mode 100755 tests/generic/4201
 create mode 100644 tests/generic/4201.out
tests/generic/4201 new file +108 -0
new file mode 100755
index 000000000000..fc367beeadaf
@@ -0,0 +1,108 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright 2019 Google LLC
+#
+# FS QA Test generic/4201
+#
+# Race dropping file systems caches vs fsstress an repeated reading of files.
+# Based on generic/579 with the fsverity bits stripped out, and an increased
+# number of runs.
+#
+. ./common/preamble
+_begin_fstest auto stress
+
+stop_run()
+{
+	# Stop all subprocesses.
+	_kill_fsstress
+	touch $tmp.done
+	wait
+}
+
+_cleanup()
+{
+	stop_run
+	rm -f $tmp.*
+}
+
+. ./common/filter
+
+fsv_file_size=10000000
+nproc_enabler=$((4 * LOAD_FACTOR))
+nproc_reader=$((6 * LOAD_FACTOR))
+nproc_stress=$((3 * LOAD_FACTOR))
+runtime=$((20 * TIME_FACTOR))
+
+run_test()
+{
+	rm -rf $tmp.done
+
+	# Create the test files and start the fs-verity enabler processes.
+	for ((proc = 0; proc < nproc_enabler; proc++)); do
+		orig_file=$SCRATCH_MNT/orig$proc
+		fsv_file=$SCRATCH_MNT/fsv$proc
+		head -c $fsv_file_size /dev/urandom > $orig_file
+		(
+			while [ ! -e $tmp.done ]; do
+				rm -f $fsv_file
+				cp $orig_file $fsv_file
+				# Give the readers some time to read from the file.
+				sleep 0.$((RANDOM % 100))
+			done
+		) &
+	done
+
+	# Start the reader processes.
+	for ((proc = 0; proc < nproc_reader; proc++)); do
+		(
+			while [ ! -e $tmp.done ]; do
+				# Choose a random file for each iteration, so that
+				# sometimes multiple processes read from the same file.
+				i=$((RANDOM % nproc_enabler))
+				orig_file=$SCRATCH_MNT/orig$i
+				fsv_file=$SCRATCH_MNT/fsv$i
+
+				# After the copy from $orig_file to $fsv_file has
+				# completed, the contents of these two files should
+				# match, regardless of whether verity has been enabled
+				# or not yet (or is currently being enabled).
+				cmp $orig_file $fsv_file |& _filter_scratch | \
+					sed -e "s/'//g" | \
+					grep -v "SCRATCH_MNT/fsv$i: No such file or directory" | \
+					grep -v "EOF on SCRATCH_MNT/fsv$i"
+				done
+		) &
+	done
+
+	# Start a process that occasionally runs 'sync && drop_caches'.  This makes more
+	# reads go through fs-verity for real, rather than just returning pagecache.
+	(
+		while [ ! -e $tmp.done ]; do
+			sleep 2.$((RANDOM % 100))
+			_scratch_sync && echo 3 > /proc/sys/vm/drop_caches
+		done
+	) &
+
+	# Start the fsstress processes.
+	_run_fsstress_bg -p $nproc_stress -l 0 -d $SCRATCH_MNT/stressdir
+
+	# Run for a while.
+	sleep $runtime
+
+	stop_run
+}
+
+for i in $(seq 1 5); do
+	echo "Starting run $i" >> $seqres.full
+
+	_scratch_mkfs &>> $seqres.full
+	_scratch_mount
+	run_test
+	_scratch_unmount
+done
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/generic/4201.out new file +2 -0
new file mode 100644
index 000000000000..2c3f3d42ff3f
@@ -0,0 +1,2 @@
+QA output created by 4201
+Silence is golden
Signature · 1 line
2.53.0
#2 lore raw
Quoted 2 lines · Carlos Maiolino
Quoted 1 line · Christoph Hellwig

The fsverity test generic/759 could be used to reproduce a regression in

                              ^^^
                              579
Quoted 12 lines
Quoted 11 lines

the xfs buffer hashing. Carlos Maiolino figured out that even removing the fsverity-specific bits would reproduce the issue. This test is thus a copy of generic/759 with the fsverity bits striped out, and run 5 times as that seems to be the threshold at which the issue reproduces reliably.

---

Not sure how to best credit Eric for the original patch and Carlos for the extraction, any ideas?

Quoted 8 lines

To be fair, Andrey initially suggested stripping off the fs-verity parts, IIRC because he wanted to isolate his own series from this crash. My role was mostly reproduce it and hunt the bug down. Risking to come with yet another tag, perhaps something like:

Based-on-generic-579-by: <Eric...>

It doesn't look ugly to me, but who knows...

Hmm... Generally, we state which case was copied/forked from in the introductory comments of the case and in the commit log. If you want to specifically mention the original author, I think we can add a dedicated line in the commit log, for example:

'This case is copied from generic/579, which was written by Eric Biggers.'

Currently, we don’t have a formal commit trailer to define this specific scenario, but perhaps this is a good opportunity for us to discuss and agree on one. After all, copying and modifying existing cases happens quite often in xfstests. For example, we could specify the source case in the commit body, and then add a commit trailer likes "Based-on-code-by", the whole things like:

This case is copied by generic/579, I removed the fsverity-specific part to reproduce ....
...

What do you and others think?

Quoted 8 lines
Quoted 8 lines

 tests/generic/4201     | 108 +++++++++++++++++++++++++++++++++++++++++
 tests/generic/4201.out |   2 +
 2 files changed, 110 insertions(+)
 create mode 100755 tests/generic/4201
 create mode 100644 tests/generic/4201.out
Quoted 9 lines

tests/generic/4201 line 3

Quoted 9 lines
tests/generic/4201 new file +3 -0
new file mode 100755
index 000000000000..fc367beeadaf
@@ -0,0 +1,108 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright 2019 Google LLC

I saw you tried to keep the original copyright of generic/579. I'm not sure if we can have two Copyright lines at here, one is yours, the other is the original one?

Thanks,
Zorro

Quoted 117 lines
Quoted 117 lines
tests/generic/4201 new file +105 -0
+#
+# FS QA Test generic/4201
+#
+# Race dropping file systems caches vs fsstress an repeated reading of files.
+# Based on generic/579 with the fsverity bits stripped out, and an increased
+# number of runs.
+#
+. ./common/preamble
+_begin_fstest auto stress
+
+stop_run()
+{
+	# Stop all subprocesses.
+	_kill_fsstress
+	touch $tmp.done
+	wait
+}
+
+_cleanup()
+{
+	stop_run
+	rm -f $tmp.*
+}
+
+. ./common/filter
+
+fsv_file_size=10000000
+nproc_enabler=$((4 * LOAD_FACTOR))
+nproc_reader=$((6 * LOAD_FACTOR))
+nproc_stress=$((3 * LOAD_FACTOR))
+runtime=$((20 * TIME_FACTOR))
+
+run_test()
+{
+	rm -rf $tmp.done
+
+	# Create the test files and start the fs-verity enabler processes.
+	for ((proc = 0; proc < nproc_enabler; proc++)); do
+		orig_file=$SCRATCH_MNT/orig$proc
+		fsv_file=$SCRATCH_MNT/fsv$proc
+		head -c $fsv_file_size /dev/urandom > $orig_file
+		(
+			while [ ! -e $tmp.done ]; do
+				rm -f $fsv_file
+				cp $orig_file $fsv_file
+				# Give the readers some time to read from the file.
+				sleep 0.$((RANDOM % 100))
+			done
+		) &
+	done
+
+	# Start the reader processes.
+	for ((proc = 0; proc < nproc_reader; proc++)); do
+		(
+			while [ ! -e $tmp.done ]; do
+				# Choose a random file for each iteration, so that
+				# sometimes multiple processes read from the same file.
+				i=$((RANDOM % nproc_enabler))
+				orig_file=$SCRATCH_MNT/orig$i
+				fsv_file=$SCRATCH_MNT/fsv$i
+
+				# After the copy from $orig_file to $fsv_file has
+				# completed, the contents of these two files should
+				# match, regardless of whether verity has been enabled
+				# or not yet (or is currently being enabled).
+				cmp $orig_file $fsv_file |& _filter_scratch | \
+					sed -e "s/'//g" | \
+					grep -v "SCRATCH_MNT/fsv$i: No such file or directory" | \
+					grep -v "EOF on SCRATCH_MNT/fsv$i"
+				done
+		) &
+	done
+
+	# Start a process that occasionally runs 'sync && drop_caches'.  This makes more
+	# reads go through fs-verity for real, rather than just returning pagecache.
+	(
+		while [ ! -e $tmp.done ]; do
+			sleep 2.$((RANDOM % 100))
+			_scratch_sync && echo 3 > /proc/sys/vm/drop_caches
+		done
+	) &
+
+	# Start the fsstress processes.
+	_run_fsstress_bg -p $nproc_stress -l 0 -d $SCRATCH_MNT/stressdir
+
+	# Run for a while.
+	sleep $runtime
+
+	stop_run
+}
+
+for i in $(seq 1 5); do
+	echo "Starting run $i" >> $seqres.full
+
+	_scratch_mkfs &>> $seqres.full
+	_scratch_mount
+	run_test
+	_scratch_unmount
+done
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
tests/generic/4201.out new file +2 -0
new file mode 100644
index 000000000000..2c3f3d42ff3f
@@ -0,0 +1,2 @@
+QA output created by 4201
+Silence is golden
Signature · 1 line
2.53.0
#3 lore raw
Quoted 5 lines · Zorro Lang
Quoted 2 lines · Carlos Maiolino
Quoted 1 line · Christoph Hellwig

The fsverity test generic/759 could be used to reproduce a regression in

                              ^^^
                              579

Yeah.

Quoted 4 lines

Hmm... Generally, we state which case was copied/forked from in the introductory comments of the case and in the commit log. If you want to specifically mention the original author, I think we can add a dedicated line in the commit log, for example:

If Eric is fine with that I can stick to the normal rule.

Quoted 1 line

What do you and others think?

I don't really care at all. I just want to make sure I don't smub everyone involved with this the wrong way :)

#4 lore raw
Quoted 14 lines · Christoph Hellwig
Quoted 5 lines · Zorro Lang
Quoted 2 lines · Carlos Maiolino
Quoted 1 line · Christoph Hellwig

The fsverity test generic/759 could be used to reproduce a regression in

                              ^^^
                              579

Yeah.

Quoted 4 lines

Hmm... Generally, we state which case was copied/forked from in the introductory comments of the case and in the commit log. If you want to specifically mention the original author, I think we can add a dedicated line in the commit log, for example:

Quoted 6 lines

If Eric is fine with that I can stick to the normal rule.

Quoted 1 line

What do you and others think?

I don't really care at all. I just want to make sure I don't smub everyone involved with this the wrong way :)

The patch looks fine as-is (other than the typo already pointed out). It already preserves the original copyright line and has a comment "# Based on generic/579 with the fsverity bits stripped out". I don't need any additional credit. Thanks,

- Eric

Applied
#5 lore raw
In reply to Christoph Hellwig
Quoted 14 lines · Christoph Hellwig
Quoted 5 lines · Zorro Lang
Quoted 2 lines · Carlos Maiolino
Quoted 1 line · Christoph Hellwig

The fsverity test generic/759 could be used to reproduce a regression in

                              ^^^
                              579

Yeah.

Quoted 4 lines

Hmm... Generally, we state which case was copied/forked from in the introductory comments of the case and in the commit log. If you want to specifically mention the original author, I think we can add a dedicated line in the commit log, for example:

Quoted 6 lines

If Eric is fine with that I can stick to the normal rule.

Quoted 1 line

What do you and others think?

I don't really care at all. I just want to make sure I don't smub everyone involved with this the wrong way :)

I've merged this patch with above typo fixes. Please feel free to check patches-in-queue branch.

Thanks,
Zorro

Quoted 2 lines
#6 lore raw
In reply to Zorro Lang
Quoted 3 lines · Zorro Lang

Currently, we don’t have a formal commit trailer to define this specific scenario, but perhaps this is a good opportunity for us to discuss and agree on one.

My suggestion is that this should go in a comment of the forked test, e.g.

# Based on generic/579 with fsverity-specific bits removed

That way, if someone fixes a problem in generic/579 in the future, a quick grep might turn up other tests that need a similar adjustment. The commit trailer doesn't seem as useful to me since it's harder to find.

- Ted

#7 lore raw
Quoted 5 lines · Theodore Tso
Quoted 3 lines · Zorro Lang

Currently, we don’t have a formal commit trailer to define this specific scenario, but perhaps this is a good opportunity for us to discuss and agree on one.

Quoted 8 lines

My suggestion is that this should go in a comment of the forked test, e.g.

# Based on generic/579 with fsverity-specific bits removed

That way, if someone fixes a problem in generic/579 in the future, a quick grep might turn up other tests that need a similar adjustment. The commit trailer doesn't seem as useful to me since it's harder to find.

Thanks Ted, since Eric agreed with this as well and there are no other objections, I think we can hold off on creating a new git trailer. Let's just proceed as usual: mention the original case number in the commit log, and also add it to the case's initial comments would make it easier to track.

Thanks,
Zorro

Quoted 3 lines

- Ted