generic: add a test that stresses metadata eviction
PATCH 1/1 generic: add a test that stresses metadata eviction
The fsverity test generic/759 could be used to reproduce a regression in the xfs buffer hashing. Carlos Maiolino figured out that even removing the fsverity-specific bits would reproduce the issue. This test is thus a copy of generic/759 with the fsverity bits striped out, and run 5 times as that seems to be the threshold at which the issue reproduces reliably.
---
Not sure how to best credit Eric for the original patch and Carlos for the extraction, any ideas?
tests/generic/4201 | 108 +++++++++++++++++++++++++++++++++++++++++ tests/generic/4201.out | 2 + 2 files changed, 110 insertions(+) create mode 100755 tests/generic/4201 create mode 100644 tests/generic/4201.out
tests/generic/4201 new file +108 -0
@@ -0,0 +1,108 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright 2019 Google LLC +# +# FS QA Test generic/4201 +# +# Race dropping file systems caches vs fsstress an repeated reading of files. +# Based on generic/579 with the fsverity bits stripped out, and an increased +# number of runs. +# +. ./common/preamble +_begin_fstest auto stress + +stop_run() +{ + # Stop all subprocesses. + _kill_fsstress + touch $tmp.done + wait +} + +_cleanup() +{ + stop_run + rm -f $tmp.* +} + +. ./common/filter + +fsv_file_size=10000000 +nproc_enabler=$((4 * LOAD_FACTOR)) +nproc_reader=$((6 * LOAD_FACTOR)) +nproc_stress=$((3 * LOAD_FACTOR)) +runtime=$((20 * TIME_FACTOR)) + +run_test() +{ + rm -rf $tmp.done + + # Create the test files and start the fs-verity enabler processes. + for ((proc = 0; proc < nproc_enabler; proc++)); do + orig_file=$SCRATCH_MNT/orig$proc + fsv_file=$SCRATCH_MNT/fsv$proc + head -c $fsv_file_size /dev/urandom > $orig_file + ( + while [ ! -e $tmp.done ]; do + rm -f $fsv_file + cp $orig_file $fsv_file + # Give the readers some time to read from the file. + sleep 0.$((RANDOM % 100)) + done + ) & + done + + # Start the reader processes. + for ((proc = 0; proc < nproc_reader; proc++)); do + ( + while [ ! -e $tmp.done ]; do + # Choose a random file for each iteration, so that + # sometimes multiple processes read from the same file. + i=$((RANDOM % nproc_enabler)) + orig_file=$SCRATCH_MNT/orig$i + fsv_file=$SCRATCH_MNT/fsv$i + + # After the copy from $orig_file to $fsv_file has + # completed, the contents of these two files should + # match, regardless of whether verity has been enabled + # or not yet (or is currently being enabled). + cmp $orig_file $fsv_file |& _filter_scratch | \ + sed -e "s/'//g" | \ + grep -v "SCRATCH_MNT/fsv$i: No such file or directory" | \ + grep -v "EOF on SCRATCH_MNT/fsv$i" + done + ) & + done + + # Start a process that occasionally runs 'sync && drop_caches'. This makes more + # reads go through fs-verity for real, rather than just returning pagecache. + ( + while [ ! -e $tmp.done ]; do + sleep 2.$((RANDOM % 100)) + _scratch_sync && echo 3 > /proc/sys/vm/drop_caches + done + ) & + + # Start the fsstress processes. + _run_fsstress_bg -p $nproc_stress -l 0 -d $SCRATCH_MNT/stressdir + + # Run for a while. + sleep $runtime + + stop_run +} + +for i in $(seq 1 5); do + echo "Starting run $i" >> $seqres.full + + _scratch_mkfs &>> $seqres.full + _scratch_mount + run_test + _scratch_unmount +done + +echo "Silence is golden" + +# success, all done +status=0 +exit
tests/generic/4201.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 4201 +Silence is golden
Signature · 1 line
2.53.0
Quoted 7 lines · Christoph Hellwig
The fsverity test generic/759 could be used to reproduce a regression in the xfs buffer hashing. Carlos Maiolino figured out that even removing the fsverity-specific bits would reproduce the issue. This test is thus a copy of generic/759 with the fsverity bits striped out, and run 5 times as that seems to be the threshold at which the issue reproduces reliably.
Quoted 5 lines
Signed-off-by Christoph Hellwig <[email protected]>---
Not sure how to best credit Eric for the original patch and Carlos for the extraction, any ideas?
To be fair, Andrey initially suggested stripping off the fs-verity parts, IIRC because he wanted to isolate his own series from this crash. My role was mostly reproduce it and hunt the bug down. Risking to come with yet another tag, perhaps something like:
Based-on-generic-579-by: <Eric...>
It doesn't look ugly to me, but who knows...
Quoted 133 lines
tests/generic/4201 | 108 +++++++++++++++++++++++++++++++++++++++++ tests/generic/4201.out | 2 + 2 files changed, 110 insertions(+) create mode 100755 tests/generic/4201 create mode 100644 tests/generic/4201.outtests/generic/4201 new file +108
-0@@ -0,0 +1,108 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright 2019 Google LLC +# +# FS QA Test generic/4201 +# +# Race dropping file systems caches vs fsstress an repeated reading of files. +# Based on generic/579 with the fsverity bits stripped out, and an increased +# number of runs. +# +. ./common/preamble +_begin_fstest auto stress + +stop_run() +{ + # Stop all subprocesses. + _kill_fsstress + touch $tmp.done + wait +} + +_cleanup() +{ + stop_run + rm -f $tmp.* +} + +. ./common/filter + +fsv_file_size=10000000 +nproc_enabler=$((4 * LOAD_FACTOR)) +nproc_reader=$((6 * LOAD_FACTOR)) +nproc_stress=$((3 * LOAD_FACTOR)) +runtime=$((20 * TIME_FACTOR)) + +run_test() +{ + rm -rf $tmp.done + + # Create the test files and start the fs-verity enabler processes. + for ((proc = 0; proc < nproc_enabler; proc++)); do + orig_file=$SCRATCH_MNT/orig$proc + fsv_file=$SCRATCH_MNT/fsv$proc + head -c $fsv_file_size /dev/urandom > $orig_file + ( + while [ ! -e $tmp.done ]; do + rm -f $fsv_file + cp $orig_file $fsv_file + # Give the readers some time to read from the file. + sleep 0.$((RANDOM % 100)) + done + ) & + done + + # Start the reader processes. + for ((proc = 0; proc < nproc_reader; proc++)); do + ( + while [ ! -e $tmp.done ]; do + # Choose a random file for each iteration, so that + # sometimes multiple processes read from the same file. + i=$((RANDOM % nproc_enabler)) + orig_file=$SCRATCH_MNT/orig$i + fsv_file=$SCRATCH_MNT/fsv$i + + # After the copy from $orig_file to $fsv_file has + # completed, the contents of these two files should + # match, regardless of whether verity has been enabled + # or not yet (or is currently being enabled). + cmp $orig_file $fsv_file |& _filter_scratch | \ + sed -e "s/'//g" | \ + grep -v "SCRATCH_MNT/fsv$i: No such file or directory" | \ + grep -v "EOF on SCRATCH_MNT/fsv$i" + done + ) & + done + + # Start a process that occasionally runs 'sync && drop_caches'. This makes more + # reads go through fs-verity for real, rather than just returning pagecache. + ( + while [ ! -e $tmp.done ]; do + sleep 2.$((RANDOM % 100)) + _scratch_sync && echo 3 > /proc/sys/vm/drop_caches + done + ) & + + # Start the fsstress processes. + _run_fsstress_bg -p $nproc_stress -l 0 -d $SCRATCH_MNT/stressdir + + # Run for a while. + sleep $runtime + + stop_run +} + +for i in $(seq 1 5); do + echo "Starting run $i" >> $seqres.full + + _scratch_mkfs &>> $seqres.full + _scratch_mount + run_test + _scratch_unmount +done + +echo "Silence is golden" + +# success, all done +status=0 +exittests/generic/4201.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 4201 +Silence is goldenSignature · 1 line
2.53.0
Quoted 2 lines · Carlos Maiolino
Quoted 1 line · Christoph Hellwig
The fsverity test generic/759 could be used to reproduce a regression in
^^^
579Quoted 12 lines
Quoted 11 lines
the xfs buffer hashing. Carlos Maiolino figured out that even removing the fsverity-specific bits would reproduce the issue. This test is thus a copy of generic/759 with the fsverity bits striped out, and run 5 times as that seems to be the threshold at which the issue reproduces reliably.
Signed-off-by Christoph Hellwig <[email protected]>---
Not sure how to best credit Eric for the original patch and Carlos for the extraction, any ideas?
Quoted 8 lines
To be fair, Andrey initially suggested stripping off the fs-verity parts, IIRC because he wanted to isolate his own series from this crash. My role was mostly reproduce it and hunt the bug down. Risking to come with yet another tag, perhaps something like:
Based-on-generic-579-by: <Eric...>
It doesn't look ugly to me, but who knows...
Hmm... Generally, we state which case was copied/forked from in the introductory comments of the case and in the commit log. If you want to specifically mention the original author, I think we can add a dedicated line in the commit log, for example:
'This case is copied from generic/579, which was written by Eric Biggers.'
Currently, we don’t have a formal commit trailer to define this specific scenario, but perhaps this is a good opportunity for us to discuss and agree on one. After all, copying and modifying existing cases happens quite often in xfstests. For example, we could specify the source case in the commit body, and then add a commit trailer likes "Based-on-code-by", the whole things like:
This case is copied by generic/579, I removed the fsverity-specific part to reproduce ....
...
What do you and others think?
Quoted 8 lines
Quoted 8 lines
tests/generic/4201 | 108 +++++++++++++++++++++++++++++++++++++++++ tests/generic/4201.out | 2 + 2 files changed, 110 insertions(+) create mode 100755 tests/generic/4201 create mode 100644 tests/generic/4201.out
Quoted 9 lines
tests/generic/4201 line 3
Quoted 9 lines
tests/generic/4201 new file +3
-0@@ -0,0 +1,108 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright 2019 Google LLC
I saw you tried to keep the original copyright of generic/579. I'm not sure if we can have two Copyright lines at here, one is yours, the other is the original one?
Thanks,
Zorro
Quoted 117 lines
Quoted 117 lines
tests/generic/4201 new file +105
-0+# +# FS QA Test generic/4201 +# +# Race dropping file systems caches vs fsstress an repeated reading of files. +# Based on generic/579 with the fsverity bits stripped out, and an increased +# number of runs. +# +. ./common/preamble +_begin_fstest auto stress + +stop_run() +{ + # Stop all subprocesses. + _kill_fsstress + touch $tmp.done + wait +} + +_cleanup() +{ + stop_run + rm -f $tmp.* +} + +. ./common/filter + +fsv_file_size=10000000 +nproc_enabler=$((4 * LOAD_FACTOR)) +nproc_reader=$((6 * LOAD_FACTOR)) +nproc_stress=$((3 * LOAD_FACTOR)) +runtime=$((20 * TIME_FACTOR)) + +run_test() +{ + rm -rf $tmp.done + + # Create the test files and start the fs-verity enabler processes. + for ((proc = 0; proc < nproc_enabler; proc++)); do + orig_file=$SCRATCH_MNT/orig$proc + fsv_file=$SCRATCH_MNT/fsv$proc + head -c $fsv_file_size /dev/urandom > $orig_file + ( + while [ ! -e $tmp.done ]; do + rm -f $fsv_file + cp $orig_file $fsv_file + # Give the readers some time to read from the file. + sleep 0.$((RANDOM % 100)) + done + ) & + done + + # Start the reader processes. + for ((proc = 0; proc < nproc_reader; proc++)); do + ( + while [ ! -e $tmp.done ]; do + # Choose a random file for each iteration, so that + # sometimes multiple processes read from the same file. + i=$((RANDOM % nproc_enabler)) + orig_file=$SCRATCH_MNT/orig$i + fsv_file=$SCRATCH_MNT/fsv$i + + # After the copy from $orig_file to $fsv_file has + # completed, the contents of these two files should + # match, regardless of whether verity has been enabled + # or not yet (or is currently being enabled). + cmp $orig_file $fsv_file |& _filter_scratch | \ + sed -e "s/'//g" | \ + grep -v "SCRATCH_MNT/fsv$i: No such file or directory" | \ + grep -v "EOF on SCRATCH_MNT/fsv$i" + done + ) & + done + + # Start a process that occasionally runs 'sync && drop_caches'. This makes more + # reads go through fs-verity for real, rather than just returning pagecache. + ( + while [ ! -e $tmp.done ]; do + sleep 2.$((RANDOM % 100)) + _scratch_sync && echo 3 > /proc/sys/vm/drop_caches + done + ) & + + # Start the fsstress processes. + _run_fsstress_bg -p $nproc_stress -l 0 -d $SCRATCH_MNT/stressdir + + # Run for a while. + sleep $runtime + + stop_run +} + +for i in $(seq 1 5); do + echo "Starting run $i" >> $seqres.full + + _scratch_mkfs &>> $seqres.full + _scratch_mount + run_test + _scratch_unmount +done + +echo "Silence is golden" + +# success, all done +status=0 +exittests/generic/4201.out new file +2
-0@@ -0,0 +1,2 @@ +QA output created by 4201 +Silence is goldenSignature · 1 line
2.53.0
Quoted 5 lines · Zorro Lang
Quoted 2 lines · Carlos Maiolino
Quoted 1 line · Christoph Hellwig
The fsverity test generic/759 could be used to reproduce a regression in
^^^ 579
Yeah.
Quoted 4 lines
Hmm... Generally, we state which case was copied/forked from in the introductory comments of the case and in the commit log. If you want to specifically mention the original author, I think we can add a dedicated line in the commit log, for example:
If Eric is fine with that I can stick to the normal rule.
Quoted 1 line
What do you and others think?
I don't really care at all. I just want to make sure I don't smub everyone involved with this the wrong way :)
Quoted 14 lines · Christoph Hellwig
Quoted 5 lines · Zorro Lang
Quoted 2 lines · Carlos Maiolino
Quoted 1 line · Christoph Hellwig
The fsverity test generic/759 could be used to reproduce a regression in
^^^ 579Yeah.
Quoted 4 lines
Hmm... Generally, we state which case was copied/forked from in the introductory comments of the case and in the commit log. If you want to specifically mention the original author, I think we can add a dedicated line in the commit log, for example:
Quoted 6 lines
If Eric is fine with that I can stick to the normal rule.
Quoted 1 line
What do you and others think?
I don't really care at all. I just want to make sure I don't smub everyone involved with this the wrong way :)
The patch looks fine as-is (other than the typo already pointed out). It already preserves the original copyright line and has a comment "# Based on generic/579 with the fsverity bits stripped out". I don't need any additional credit. Thanks,
- Eric
Zorro Lang
AppliedQuoted 14 lines · Christoph Hellwig
Quoted 5 lines · Zorro Lang
Quoted 2 lines · Carlos Maiolino
Quoted 1 line · Christoph Hellwig
The fsverity test generic/759 could be used to reproduce a regression in
^^^ 579Yeah.
Quoted 4 lines
Hmm... Generally, we state which case was copied/forked from in the introductory comments of the case and in the commit log. If you want to specifically mention the original author, I think we can add a dedicated line in the commit log, for example:
Quoted 6 lines
If Eric is fine with that I can stick to the normal rule.
Quoted 1 line
What do you and others think?
I don't really care at all. I just want to make sure I don't smub everyone involved with this the wrong way :)
I've merged this patch with above typo fixes. Please feel free to check patches-in-queue branch.
Thanks,
Zorro
Quoted 2 lines
Quoted 3 lines · Zorro Lang
Currently, we don’t have a formal commit trailer to define this specific scenario, but perhaps this is a good opportunity for us to discuss and agree on one.
My suggestion is that this should go in a comment of the forked test, e.g.
# Based on generic/579 with fsverity-specific bits removed
That way, if someone fixes a problem in generic/579 in the future, a quick grep might turn up other tests that need a similar adjustment. The commit trailer doesn't seem as useful to me since it's harder to find.
- Ted
Quoted 5 lines · Theodore Tso
Quoted 3 lines · Zorro Lang
Currently, we don’t have a formal commit trailer to define this specific scenario, but perhaps this is a good opportunity for us to discuss and agree on one.
Quoted 8 lines
My suggestion is that this should go in a comment of the forked test, e.g.
# Based on generic/579 with fsverity-specific bits removed
That way, if someone fixes a problem in generic/579 in the future, a quick grep might turn up other tests that need a similar adjustment. The commit trailer doesn't seem as useful to me since it's harder to find.
Thanks Ted, since Eric agreed with this as well and there are no other objections, I think we can hold off on creating a new git trailer. Let's just proceed as usual: mention the original case number in the commit log, and also add it to the case's initial comments would make it easier to track.
Thanks,
Zorro
Quoted 3 lines
- Ted