Skip to contentSkip to conversation

fstests: concurrent test execution

RFC PATCH In review Dave Chinner 3 participants 65 messages Nov 27, 2024 – Dec 10, 2024 Page 1 of 2

lore
Share & tools
lore
#0 lore raw

cover fstests: concurrent test execution

Hi folks,

This patchset introduces the ability to run fstests concurrently instead of serially as the current check script does. A git branch containing this patchset can be pulled from here:

https://git.kernel.org/pub/scm/linux/kernel/git/dgc/xfstests-dev.git check-parallel

The motivation for this is the ever growing runtime of fstests as more tests are added, along with the extremely low resource usage of individual tests. This means that a typical machine running fstests is under-utilised and a single auto group test set execution takes hours.

Yes, I know that I could scale out testing by running lots of little VMs at once, and I already do that. The problem is that I don't get back a complete auto test run result for hours. Want to check that a one-line fix has not caused any regressions? That's, at minimum, an overnight wait for the test farm to crunch through a dozen configs.

On my 64p/128GB RAM VM, 'check -g auto -s xfs -x dump' typically takes around 230-240 minutes to run. With this patchset applied, it runs the same set of tests in 8-10 minutes. I can run ~25 complete auto group test sets with check-parallel in the same time it takes check to run one.

IOWs, I can have the most common config complete a full regression test run as fast as I can turn around a new kernel with a new change to test. I have CPU, memory and IO to burn in my test machines, but what I lack is instant feedback for the change I just made. check-parallel is fast enough that it gives me pretty much instant feedback....

Most of this patchset is preparing infrastructure for concurrent test execution and fixing bugs in tests that I've found whilst getting concurrent execution working reliably. The infrastructure changes center around:

- getting rid of killall - there's nothing quite like one test killing the processes of 15 other tests at the same time...
- cleaning up background process instantiation and reaping, which is a lot more important when an interrupt needs to kill hundreds of processes instead of just a couple.
- isolating error reporting (e.g. dmesg filtering) so that one test failure doesn't trigger lots of other false failure detections
- sanitising the use of loopback devices
- avoiding the use of fixed device names - multiple tests need to use dm-flakey, dm-error, etc devices at the same time, so each test needs a unique device name
- marking tests that are unreliable when outside perturbations occur. These include tests that expect to find delalloc extents, write specific offset patterns in memory then sync them to create specific layouts, etc. e.g. If another test runs sync(1), then those write patterns no longer produce the expected output.
- taming tests that weren't designed for high CPU count machines
- replacing udevadm settle calls because udev is -never- idle when there are tens to hundreds of block devices and filesystems being rapidly set up and torn down.
- converting sync(1) and sync(2) to syncfs(2) to avoid ihaving hundreds of concurrent superblock list traversals lock-stepping with multiple mount/unmounts every second.

There are lots of little other things, but those are the main test and test infrastructure changes. Some of these are big - the fsstress execution rework touches 105 files, but it now means that every single fsstress execution in fstests is controlled by 4 helper functions:

_run_fsstress() - run fsstress synchronously
_run_fsstress_bg - run fsstress in background
_wait_for_fsstress - wait for background fsstress
_kill_fsstress - kill+wait for background fsstress

The test infrastructure also automatically handles cleanup of fsstress processes when the test is interrupted, so tests using fsstress don't need a custom _cleanup() function just to call _kill_fsstress(). This is something that should have been done a long time ago, but now it is critical for being able to manage multiple independent concurrent fsstress invocations sanely.

There are some tests that just can't be run reliably in a concurrent environment - if there is outside interference in, say, page cache flushing then the tests fail. These tests have been added to the "unreliable_in_parallel" test group with a comment explaining why they are unreliable. The check-parallel script automatically excludes this test group.

The only remaining set of tests that are somewhat flakey are the tests that exercise quotas. Quota tests randomly fail for various reasons. Sometimes they don't detect EDQUOT conditions. Sometimes repquota emits weird "device not found" errors. Sometimes grace periods don't start, sometimes they don't time out, or time out and then don't trigger EDQUOT. I don't know why these weird things are happening yet, and until I properly integrate test group selection into check-parallel I can't really isolate the quota tests to focus on them alone.

There are several patches that speed up test runtime. There were several tests that were taking 12-15 minutes to run each, and these made up >95% of the entire check-parallel runtime. In general, these tests have been made to run with more concurrency to speed them up. the result is that the longest individual test runtime has dropped to around 7 minutes, and the elapsed runtime for check-parallel has dropped to 8-10 minutes.

Hence there are 39 patches that are doing prep work on tests and infrastructure to make tests run reliabily in concurrent test contexts. The last patch is the check-parallel runner script that runs the tests concurrently.

The check-parallel script is still very rudimentary. I hard coded the tests it runs (generic+xfs auto tests) and the concurrency so that I could run explicit sets of tests with './check --exact-order <list>'. It is pointed at a mounted directory, and it creates all the infrastructure it needs to run the tests within that directory.

This enabled me to break up the tests across a set of identical runner process contexts. Each runner does:

- create runner directory
- create test and scratch image files
- create loopback devices for test and scratch devices
- sets up results directories
- execute check in it's own private mount namespace so it can't see any of the mounts that other runners are using.
- tears down loopback devices
- reports test failures.

If you run with the same directory over and over again, then it reuses the same runner infrastructure and test and scratch image files. The results directories are not overwritten as they are date-stamped, hence using the same mount point automatically creates a result archive for later data mining.

A typical target directory (/mnt/xfs) looks like:

/mnt/xfs/runner-0/...
/mnt/xfs/runner-1/...
....
/mnt/xfs/runner-63/...

And each runner directory:

log
results-2024-11-19-11:22:25/...
results-2024-11-19-12:36:28/...
.....
results-2024-11-27-13:32:42/...
scratch/
scratch.img
test/
test.img

The log file is the check output for that runner and should look familiar:

SECTION       -- xfs
FSTYP         -- xfs (debug)
PLATFORM      -- Linux/x86_64 test1 6.12.0-dgc+ #297 SMP PREEMPT_DYNAMIC Wed Nov 27 08:13:06 AEDT 2024
MKFS_OPTIONS  -- -f -m rmapbt=1 /dev/loop10

MOUNT_OPTIONS -- -o context=system_u:object_r:root_t:s0 /dev/loop10 /mnt/xfs/runner-10/scratch

generic/387        309s
generic/551        49s
generic/289        2s
xfs/274        3s
generic/677        3s
generic/143        5s
generic/304        1s
generic/390        4s
generic/427        4s
xfs/103        1s
generic/252        3s
xfs/045        1s
generic/374        2s
generic/002        1s
generic/534        1s
generic/039        1s

generic/595 [not run] No encryption support for xfs
xfs/122 [not run] Could not compile test program (see end of /mnt/xfs/runner-10/results-2024-11-27-13:32:42/xfs/xfs/122.full)
xfs/556 [not run] xfs_scrub not found
Ran: generic/387 generic/551 generic/289 xfs/274 generic/677 generic/143 generic/304 generic/390 generic/427 xfs/103 generic/252 xfs/045 generic/374 generic/002 generic/534 generic/039 generic/595 xfs/122 xfs/556 Not run: generic/595 xfs/122 xfs/556
Passed all 19 tests

SECTION -- xfs
=========================
Ran: generic/387 generic/551 generic/289 xfs/274 generic/677 generic/143 generic/304 generic/390 generic/427 xfs/103 generic/252 xfs/045 generic/374 generic/002 generic/534 generic/039 generic/595 xfs/122 xfs/556 Not run: generic/595 xfs/122 xfs/556
Passed all 19 tests

Doing something with all the log files from a run can be done with "/mnt/xfs/*/log". e.g. grep, vi, etc.

The results directory contains the same check.{full,log,time} and test results directories as per a normal check invocation, so there's little difference in checking/analysing results with a parallel execution run.

Because the tests run in private mount namespaces, it's easy to see what check-parallel is running at any point in time using 'pstree -N mnt'. Here's a run that is hung on an unmount not completing:

$ pstree -N mnt
[4026531841]
bash
bash───pstree
[0]
sudo───sudo───check-parallel─┬─check-parallel───nsexec───check───311───fsync-tester
                             ├─check-parallel───nsexec───check───467───open_by_handle
                             ├─check-parallel───nsexec───check───338
                             ├─check-parallel───nsexec───check───421
                             ├─check-parallel───nsexec───check───441
                             ├─check-parallel───nsexec───check───232
                             ├─check-parallel───nsexec───check───477───open_by_handle
                             ├─check-parallel───nsexec───check───420
                             ├─check-parallel───nsexec───check───426───open_by_handle
                             ├─check-parallel───nsexec───check───756───open_by_handle
                             ├─check-parallel───nsexec───check───231
                             ├─check-parallel───nsexec───check───475───475.fsstress───475.fsstress───{475.fsstress}
                             ├─check-parallel───nsexec───check───388───388.fsstress───388.fsstress───{388.fsstress}
                             ├─check-parallel───nsexec───check───259───sync
                             ├─check-parallel───nsexec───check───622───sync
                             ├─check-parallel───nsexec───check───318───sync
                             ├─check-parallel───nsexec───check───753───umount
                             ├─check-parallel───nsexec───check───086
                             ├─check-parallel───nsexec───check───648───648.fsstress───648.fsstress───{648.fsstress}
                             ├─check-parallel───nsexec───check───391
                             ├─check-parallel───nsexec───check───315───sync
                             └─check-parallel───nsexec───check───183───bulkstat_unlink

All the other processes are in sync or dropping caches and stuck waiting for the superblock s_umount lock that the unmount holds. Finding where it is stuck:

$ pgrep [u]mount
1081885
$ sudo cat /proc/1081885/stack
[<0>] xfs_ail_push_all_sync+0x9c/0xf0
[<0>] xfs_unmount_flush_inodes+0x41/0x70
[<0>] xfs_unmountfs+0x59/0x190
[<0>] xfs_fs_put_super+0x3b/0x90
[<0>] generic_shutdown_super+0x77/0x160
[<0>] kill_block_super+0x1b/0x40
[<0>] xfs_kill_sb+0x12/0x30
[<0>] deactivate_locked_super+0x38/0x100
[<0>] deactivate_super+0x41/0x50
[<0>] cleanup_mnt+0x9f/0x160
[<0>] __cleanup_mnt+0x12/0x20
[<0>] task_work_run+0x89/0xb0
[<0>] resume_user_mode_work+0x4f/0x60
[<0>] syscall_exit_to_user_mode+0x76/0xb0
[<0>] do_syscall_64+0x74/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x76/0x7e
$

Yup, known bug - a shutdown vs XFS_ISTALE inode cluster freeing issue that leaves pinned, stale inodes in the AIL.

The point I'm making here is that running tests concurrently doesn't change anything material in how you'd go about discovering and diagnosing failures. The only difference is in what that initial failure might look like. e.g. Failures that result in dmesg output will cause a huge number of tests to all fail with "check dmesg for failure output" reports. Hence there is a bit of sifting to find which test triggered the dmesg output, but failure detection still works just fine.

The check-parallel script is really only at proof-of-concept stage. It is sufficient to run tests in parallel, but that's about it. most of the work so far has gone into making the generic and XFS tests run reliably in parallel. I have not run any other tests, and haven't done full conversions of other test directories or C code in src/. e.g. the sync -> syncfs was only done for the common dir and the generic and xfs tests dirs

I do not plan to do these conversions for ext4/btrfs/overlay/etc any time soon as these conversions are mostly about improving execution time, not test correctness. Hence they are not a priority for me - the priority is further developing the concurrent execution environment. i.e. check-parallel.

To that end, I need to factor all the test selection and exclusion code out of check, and do the same with the actual test list runner loop. That way I can reuse all the existing code from within the check-parallel context rather than having to call check to do all of that work itself. I would like to get check-parallel to the point where it is mostly just s/check/check-parallel/ on the command line to move from serial to concurrent test execution.

I also want to try to integrate the config section stuff into check-parallel. This is more a case of defining what devices the config needs to create (i.e. as loop devices) rather than what devices it should be using. I think I can do this just by defining a different set of environment variables (e.g. NEED_SCRATCHDEV, NEED_LOGDEV, etc) and triggering the loop device creation from these variables.

In a way, the fact that check-parallel bootstraps it's own runtime environment almost makes it entirely zero-config. As it stands right now, you should be able to pull this patchset, create your base test directory (make sure you have at least 100GB of free disk space), run './check-parallel <test_dir> -x dump' and it Should Just Work.

At this point, the test infrastructure problems are largely solved. My focus is now on further development of the check-parallel script and integrating it tightly into the existing check infrastructure rather than open-coding test lists and configuration information. This will probably take a bit of time, so I'd like to get the bug fixes, improvements and infrastructure changes underway so I'm not left carrying a huge patchset for months....

----------------------------------------------------------------
Dave Chinner (40):
xfs/448: get rid of assert-on-failure
fstests: cleanup fsstress process management
fuzzy: don't use killall
fstests: per-test dmflakey instances
fstests: per-test dmerror instances
fstests: per-test dmhuge instances
fstests: per-test dmthin instances
fstests: per-test dmdust instances
fstests: per-test dmdelay instances
fstests: fix DM device creation/removal vs udev races
fstests: use syncfs rather than sync
fstests: clean up mount and unmount operations
fstests: clean up loop device instantiation
fstests: xfs/227 is really slow
fstests: mark tests that are unreliable when run in parallel
fstests: use udevadm wait in preference to settle xfs/442: rescale load so it's not exponential xfs/176: fix broken setup code
xfs/177: remove unused slab object count location checks
fstests: remove uses of killall where possible generic/127: reduce runtime
quota: system project quota files need to be shared
dmesg: reduce noise from other tests
fstests: stop using /tmp directly
fstests: scale some tests for high CPU count sanity generic/310: cleanup killing background processes
filter: handle mount errors from CONFIG_BLK_DEV_WRITE_MOUNTED=y
filters: add a filter that accepts EIO instead of other errors generic/085: general cleanup for reliability and debugging
fstests: don't use directory stacks
fstests: clean up a couple of dm-flakey tests
fstests: clean up termination of various tests
vfstests: some tests require the testdir to be shared xfs/629: single extent files should be within tolerance xfs/076: fix broken mkfs filtering
fstests: capture some failures to seqres.full
fstests: always use fail-at-unmount semantics for XFS generic/062: don't leave debug files in $here on failure
fstests: quota grace periods unreliable under load
fstests: check-parallel

 check                                         |   12 -
 check-parallel                                |  205 ++++++++++++++++++
 common/btrfs                                  |    4
 common/config                                 |   36 ++-
 common/dmdelay                                |   24 +-
 common/dmdust                                 |   14 -
 common/dmerror                                |   74 +++---
 common/dmflakey                               |   60 ++---
 common/dmhugedisk                             |   21 +
 common/dmlogwrites                            |    4
 common/dmthin                                 |   12 -
 common/encrypt                                |    2
 common/filter                                 |   17 +
 common/fuzzy                                  |   37 +--
 common/log                                    |    2
 common/metadump                               |   32 +-
 common/overlay                                |   10
 common/populate                               |    8
 common/preamble                               |    1
 common/quota                                  |   37 ---
 common/rc                                     |  166 +++++++++++---
 common/repair                                 |    2
 common/report                                 |    2
 common/verity                                 |    2
 common/xfs                                    |    2
 doc/group-names.txt                           |    1
 doc/requirement-checking.txt                  |    6
 ltp/fsstress.c                                |   28 ++
 src/aio-dio-regress/aio-last-ref-held-by-io.c |    5
 src/dmerror                                   |    6
 tests/btrfs/004                               |   11
 tests/btrfs/007                               |    3
 tests/btrfs/012                               |    4
 tests/btrfs/028                               |    6
 tests/btrfs/049                               |    4
 tests/btrfs/057                               |    4
 tests/btrfs/060                               |   14 -
 tests/btrfs/061                               |   13 -
 tests/btrfs/062                               |   13 -
 tests/btrfs/063                               |   13 -
 tests/btrfs/064                               |   13 -
 tests/btrfs/065                               |   14 -
 tests/btrfs/066                               |   14 -
 tests/btrfs/067                               |   14 -
 tests/btrfs/068                               |   14 -
 tests/btrfs/069                               |   13 -
 tests/btrfs/070                               |   13 -
 tests/btrfs/071                               |   13 -
 tests/btrfs/072                               |   14 -
 tests/btrfs/073                               |   13 -
 tests/btrfs/074                               |   13 -
 tests/btrfs/078                               |   12 -
 tests/btrfs/100                               |    4
 tests/btrfs/101                               |    4
 tests/btrfs/136                               |    6
 tests/btrfs/160                               |    3
 tests/btrfs/192                               |   12 -
 tests/btrfs/195                               |    2
 tests/btrfs/212                               |   16 -
 tests/btrfs/232                               |    4
 tests/btrfs/252                               |    5
 tests/btrfs/261                               |    2
 tests/btrfs/284                               |    4
 tests/btrfs/286                               |    2
 tests/btrfs/291                               |    5
 tests/btrfs/320                               |    6
 tests/btrfs/332                               |    4
 tests/ext4/004                                |    4
 tests/ext4/057                                |   14 -
 tests/ext4/058                                |    3
 tests/ext4/307                                |    4
 tests/generic/013                             |   22 -
 tests/generic/015                             |    2
 tests/generic/019                             |   13 -
 tests/generic/029                             |    2
 tests/generic/030                             |    2
 tests/generic/032                             |    2
 tests/generic/034                             |    2
 tests/generic/039                             |    2
 tests/generic/040                             |    2
 tests/generic/041                             |    2
 tests/generic/042                             |    4
 tests/generic/048                             |    2
 tests/generic/049                             |    4
 tests/generic/050                             |    3
 tests/generic/051                             |   24 --
 tests/generic/054                             |    2
 tests/generic/055                             |    6
 tests/generic/057                             |    2
 tests/generic/059                             |    2
 tests/generic/062                             |    2
 tests/generic/065                             |    2
 tests/generic/066                             |    5
 tests/generic/067                             |   17 -
 tests/generic/068                             |    7
 tests/generic/070                             |   12 -
 tests/generic/073                             |    2
 tests/generic/076                             |   10
 tests/generic/076.out                         |    1
 tests/generic/081                             |    8
 tests/generic/083                             |   13 -
 tests/generic/083.out                         |    1
 tests/generic/084                             |   12 -
 tests/generic/085                             |   21 +
 tests/generic/090                             |    4
 tests/generic/092                             |    2
 tests/generic/098                             |    4
 tests/generic/099                             |    8
 tests/generic/101                             |    2
 tests/generic/104                             |    2
 tests/generic/106                             |    2
 tests/generic/107                             |    2
 tests/generic/108                             |    9
 tests/generic/109                             |    5
 tests/generic/117                             |    4
 tests/generic/127                             |   67 +++--
 tests/generic/127.out                         |    6
 tests/generic/135                             |   24 --
 tests/generic/150                             |    2
 tests/generic/151                             |    2
 tests/generic/152                             |    2
 tests/generic/157                             |    3
 tests/generic/158                             |    3
 tests/generic/159                             |    2
 tests/generic/160                             |    2
 tests/generic/162                             |    4
 tests/generic/163                             |    4
 tests/generic/164                             |    4
 tests/generic/165                             |    4
 tests/generic/166                             |    6
 tests/generic/167                             |    4
 tests/generic/168                             |    7
 tests/generic/170                             |    7
 tests/generic/171                             |    6
 tests/generic/172                             |    6
 tests/generic/173                             |    6
 tests/generic/174                             |    6
 tests/generic/204                             |    8
 tests/generic/232                             |    5
 tests/generic/232.out                         |    1
 tests/generic/247                             |    2
 tests/generic/250                             |    2
 tests/generic/251                             |    5
 tests/generic/252                             |    5
 tests/generic/265                             |    2
 tests/generic/266                             |    2
 tests/generic/267                             |    2
 tests/generic/268                             |    2
 tests/generic/269                             |    8
 tests/generic/270                             |   10
 tests/generic/271                             |    2
 tests/generic/272                             |    2
 tests/generic/273                             |    2
 tests/generic/274                             |    6
 tests/generic/275                             |    6
 tests/generic/276                             |    2
 tests/generic/278                             |    2
 tests/generic/279                             |    2
 tests/generic/281                             |    2
 tests/generic/282                             |    2
 tests/generic/283                             |    2
 tests/generic/306                             |    2
 tests/generic/310                             |   24 +-
 tests/generic/315                             |    2
 tests/generic/317                             |    2
 tests/generic/318                             |    2
 tests/generic/321                             |    4
 tests/generic/323                             |    7
 tests/generic/325                             |    2
 tests/generic/328                             |    4
 tests/generic/329                             |    5
 tests/generic/330                             |    2
 tests/generic/331                             |    7
 tests/generic/332                             |    4
 tests/generic/333                             |    6
 tests/generic/334                             |    6
 tests/generic/335                             |    2
 tests/generic/336                             |    9
 tests/generic/341                             |    2
 tests/generic/342                             |    2
 tests/generic/343                             |    2
 tests/generic/347                             |    2
 tests/generic/348                             |    2
 tests/generic/353                             |    2
 tests/generic/361                             |    8
 tests/generic/373                             |    4
 tests/generic/374                             |    4
 tests/generic/376                             |    2
 tests/generic/382                             |    2
 tests/generic/387                             |    2
 tests/generic/388                             |   24 --
 tests/generic/390                             |   11
 tests/generic/391                             |    2
 tests/generic/395                             |    2
 tests/generic/409                             |   11
 tests/generic/410                             |   11
 tests/generic/411                             |   11
 tests/generic/416                             |    2
 tests/generic/422                             |    4
 tests/generic/425                             |    2
 tests/generic/441                             |    2
 tests/generic/459                             |    8
 tests/generic/461                             |   17 -
 tests/generic/464                             |   10
 tests/generic/474                             |    4
 tests/generic/475                             |   17 -
 tests/generic/476                             |   15 -
 tests/generic/479                             |    2
 tests/generic/480                             |    2
 tests/generic/482                             |   12 -
 tests/generic/483                             |    2
 tests/generic/484                             |    3
 tests/generic/489                             |    2
 tests/generic/502                             |    2
 tests/generic/505                             |    2
 tests/generic/506                             |    2
 tests/generic/507                             |    2
 tests/generic/508                             |    2
 tests/generic/510                             |    2
 tests/generic/520                             |    2
 tests/generic/526                             |    2
 tests/generic/527                             |    2
 tests/generic/530                             |    2
 tests/generic/531                             |    8
 tests/generic/535                             |    2
 tests/generic/546                             |    2
 tests/generic/547                             |    5
 tests/generic/556                             |    2
 tests/generic/560                             |    7
 tests/generic/561                             |   25 +-
 tests/generic/563                             |   24 +-
 tests/generic/564                             |   12 -
 tests/generic/579                             |    8
 tests/generic/585                             |    4
 tests/generic/589                             |   11
 tests/generic/590                             |    9
 tests/generic/599                             |    2
 tests/generic/601                             |    7
 tests/generic/603                             |   14 -
 tests/generic/604                             |    2
 tests/generic/610                             |    2
 tests/generic/620                             |    1
 tests/generic/628                             |    4
 tests/generic/629                             |    4
 tests/generic/631                             |    2
 tests/generic/632                             |    2
 tests/generic/640                             |    2
 tests/generic/642                             |   10
 tests/generic/648                             |   25 --
 tests/generic/650                             |   19 -
 tests/generic/670                             |    2
 tests/generic/671                             |    2
 tests/generic/672                             |    2
 tests/generic/673                             |    2
 tests/generic/674                             |    2
 tests/generic/675                             |    2
 tests/generic/677                             |    2
 tests/generic/683                             |    2
 tests/generic/684                             |    2
 tests/generic/685                             |    2
 tests/generic/686                             |    2
 tests/generic/687                             |    2
 tests/generic/688                             |    2
 tests/generic/690                             |    2
 tests/generic/691                             |    6
 tests/generic/694                             |    2
 tests/generic/695                             |    2
 tests/generic/698                             |    4
 tests/generic/699                             |    8
 tests/generic/703                             |    2
 tests/generic/704                             |    2
 tests/generic/707                             |    7
 tests/generic/716                             |    2
 tests/generic/717                             |    2
 tests/generic/718                             |    2
 tests/generic/719                             |    2
 tests/generic/721                             |    2
 tests/generic/722                             |   15 -
 tests/generic/725                             |    2
 tests/generic/726                             |    2
 tests/generic/727                             |    2
 tests/generic/730                             |    2
 tests/generic/731                             |    2
 tests/generic/732                             |    4
 tests/generic/735                             |    2
 tests/generic/738                             |    2
 tests/generic/741                             |    2
 tests/generic/743                             |    4
 tests/generic/744                             |   10
 tests/generic/745                             |    4
 tests/generic/746                             |   18 -
 tests/generic/747                             |    4
 tests/generic/749                             |    4
 tests/generic/750                             |   10
 tests/generic/751                             |    1
 tests/generic/753                             |   17 -
 tests/overlay/019                             |   48 ++--
 tests/overlay/021                             |    8
 tests/overlay/058                             |   12 -
 tests/xfs/006                                 |    7
 tests/xfs/011                                 |   11
 tests/xfs/013                                 |   22 -
 tests/xfs/014                                 |   11
 tests/xfs/016                                 |    4
 tests/xfs/017                                 |   13 -
 tests/xfs/017.out                             |    1
 tests/xfs/032                                 |    2
 tests/xfs/049                                 |   39 ++-
 tests/xfs/050                                 |    5
 tests/xfs/051                                 |   10
 tests/xfs/052                                 |    2
 tests/xfs/057                                 |    9
 tests/xfs/070                                 |    9
 tests/xfs/073                                 |   36 +--
 tests/xfs/074                                 |   23 +-
 tests/xfs/076                                 |    8
 tests/xfs/077                                 |    2
 tests/xfs/078                                 |   20 -
 tests/xfs/079                                 |   17 -
 tests/xfs/104                                 |   14 -
 tests/xfs/110                                 |    2
 tests/xfs/118                                 |    4
 tests/xfs/119                                 |    2
 tests/xfs/128                                 |    2
 tests/xfs/133                                 |    2
 tests/xfs/134                                 |    2
 tests/xfs/137                                 |    2
 tests/xfs/141                                 |   13 -
 tests/xfs/148                                 |   29 +-
 tests/xfs/149                                 |    8
 tests/xfs/154                                 |    1
 tests/xfs/158                                 |    4
 tests/xfs/161                                 |    2
 tests/xfs/167                                 |   17 -
 tests/xfs/168                                 |    8
 tests/xfs/176                                 |   10
 tests/xfs/177                                 |   15 -
 tests/xfs/186                                 |    4
 tests/xfs/195                                 |    2
 tests/xfs/201                                 |    4
 tests/xfs/212                                 |    2
 tests/xfs/216                                 |   23 +-
 tests/xfs/217                                 |   24 +-
 tests/xfs/227                                 |   59 +++--
 tests/xfs/231                                 |    4
 tests/xfs/232                                 |   10
 tests/xfs/234                                 |    2
 tests/xfs/236                                 |    2
 tests/xfs/237                                 |   13 -
 tests/xfs/239                                 |    2
 tests/xfs/240                                 |    7
 tests/xfs/241                                 |    2
 tests/xfs/243                                 |   11
 tests/xfs/246                                 |    2
 tests/xfs/250                                 |   19 +
 tests/xfs/259                                 |   13 -
 tests/xfs/264                                 |    4
 tests/xfs/265                                 |    2
 tests/xfs/270                                 |    2
 tests/xfs/272                                 |    2
 tests/xfs/274                                 |    2
 tests/xfs/276                                 |    2
 tests/xfs/289                                 |    4
 tests/xfs/291                                 |    4
 tests/xfs/297                                 |   12 -
 tests/xfs/300                                 |    8
 tests/xfs/305                                 |    8
 tests/xfs/309                                 |    2
 tests/xfs/312                                 |    2
 tests/xfs/313                                 |    2
 tests/xfs/314                                 |    2
 tests/xfs/315                                 |    4
 tests/xfs/316                                 |    2
 tests/xfs/317                                 |    2
 tests/xfs/318                                 |    4
 tests/xfs/319                                 |    2
 tests/xfs/320                                 |    2
 tests/xfs/321                                 |    2
 tests/xfs/322                                 |    2
 tests/xfs/323                                 |    2
 tests/xfs/324                                 |    2
 tests/xfs/325                                 |    4
 tests/xfs/326                                 |    4
 tests/xfs/327                                 |    2
 tests/xfs/420                                 |    2
 tests/xfs/421                                 |    2
 tests/xfs/423                                 |    2
 tests/xfs/438                                 |    2
 tests/xfs/440                                 |    8
 tests/xfs/442                                 |   15 -
 tests/xfs/448                                 |    6
 tests/xfs/495                                 |    2
 tests/xfs/501                                 |    2
 tests/xfs/502                                 |    2
 tests/xfs/507                                 |    2
 tests/xfs/511                                 |    2
 tests/xfs/513                                 |   48 +---
 tests/xfs/519                                 |    2
 tests/xfs/520                                 |    2
 tests/xfs/521                                 |    8
 tests/xfs/527                                 |    5
 tests/xfs/528                                 |   10
 tests/xfs/530                                 |   11
 tests/xfs/538                                 |    4
 tests/xfs/541                                 |    2
 tests/xfs/544                                 |    2
 tests/xfs/553                                 |    4
 tests/xfs/558                                 |    7
 tests/xfs/601                                 |    2
 tests/xfs/606                                 |   14 -
 tests/xfs/607                                 |    6
 tests/xfs/609                                 |   20 -
 tests/xfs/610                                 |   20 -
 tests/xfs/613                                 |   44 +--
 tests/xfs/613.out                             |    1
 tests/xfs/617                                 |    2
 tests/xfs/629                                 |    6
 tests/xfs/630                                 |    2
 tests/xfs/631                                 |    9
 tests/xfs/790                                 |    2
 tests/xfs/791                                 |    2
 tests/xfs/792                                 |    2
 tests/xfs/802                                 |    7
 423 files changed, 1827 insertions(+), 1629 deletions(-)
#1 lore raw

PATCH 1/40 xfs/448: get rid of assert-on-failure

From: Dave Chinner <[email protected]>

The bug this problem exercises has been fixed for quite some time, but the test does not run on XFS debug kernels that have fatal asserts enable. There is no reason for this now that the test does not assert fail on most kernels regularly tested, so kill the check and enable the test to run on all XFS configs.

---
 tests/xfs/448 | 6 ------
 1 file changed, 6 deletions(-)
tests/xfs/448 +0 -6
index fbd1af383..032c94d39 100755
@@ -13,10 +13,6 @@
 # up the directory tree by running xfs_scrub will crash the
 # kernel in __xfs_dir3_data_check.
 #
-# Notice:
-# we should have non fatal asserts configured, because assert
-# failures triggered by the intentional corrupt would crash system.
-#
 . ./common/preamble
 _begin_fstest auto quick fuzzers
 
@@ -26,8 +22,6 @@ _begin_fstest auto quick fuzzers
 
 _require_scratch_nocheck
 _require_xfs_io_command "scrub"
-# Corrupt XFS on purpose, and skip if assert failures would crash system.
-_require_no_xfs_bug_on_assert
 
 rm -f "$seqres.full"
 
Signature · 1 line
2.45.2
#2 lore raw

PATCH 2/40 fstests: cleanup fsstress process management

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Lots of tests run fsstress in the background and then have to kill it and/or provide special cleanup functions to kill the background fsstress processes. They typically use $KILLALL_PROG for this.

Use of killall is problematic for running multiple tests in parallel in that one test can kill other tests' processes. However, because fsstress itself forks and runs children, there are very few avenues for shell scripts to ensure all the fsstress processes actually die.

With bash, it is especially nasty, because sending SIGTERM will result in bash outputting error messages ("Killed: ..." that will cause golden output mismatches and hence test failures. Hence we also need to be able to tell the main fstress process to die without triggering these messages.

To avoid the process tracking problems, we change to use pkill rather than killall (more options for process selection) and we stop using the $here/ltp/fsstress binary. Instead, we copy the $here/ltp/fsstress to $TEST_DIR/$seq.fsstress so that the test has a unique fsstress binary name. This allows the pkill filter to select just the fsstress processes the test has run. The fsstress binary name is held in _FSSTRESS_NAME, and the program to run is _FSSTRESS_PROG.

We also track the primary fsstress process ID, and store that in _FSSTRESS_PID. We do this so that we have a PID to wait against so that we don't return before the fsstress processes are dead. To this end, we add a SIGPIPE handler to the primary process so that it dying doesn't trigger bash 'killed' message output. We can send 'pkill -PIPE $_FSSTRESS_NAME' to all the fsstress processes and the primary process will then enter the "wait for children to die" processing loop before it exits. In this way, we can wait for the primary fsstress process and when it exits we know that all it's children have also finished and gone away. This makes killing fsstress invocations reliable and noise free.

This is accomplished by the helpers added to common/rc:

_run_fsstress
_run_fsstress_bg
_wait_for_fsstress
_kill_fstress

This also means that all fsstress invocations now obey FSSTRESS_AVOID environment restrictions, many of which didn't.

We add a call to _kill_fstress into the generic _cleanup() function. This means that tests using fsstress don't need to add a special local _cleanup function just to call _kill_fsstress() so that background fsstress processes are killed when the user interrupts the tests with ctrl-c.

Further, killall in the _cleanup() function is often used to attempt to expedite killing of foreground execution fsstress processes. This doesn't actually work because of the way bash processes interupt signals. That is, it waits for the currently executing process to finish execution, then runs the trap function. Hence a foreground fsstress won't ever be interrupted by ctrl-c. By implementing _run_fsstress() as a background process and a wait call, the wait() call is interrupted by the signal and the cleanup trap is run immediately. Hence the fsstress processes are killed immediately and the test exits cleanly almost immediately.

The result of all this is common, clean handling of fsstress execution and termination. There are a few exceptions for special cases, but the vast majority of tests that run fsstress use the above four wrapper functions exclusively.

---
 common/fuzzy          | 28 ++++++++++----------
 common/preamble       |  1 +
 common/rc             | 59 +++++++++++++++++++++++++++++++++++++++++++
 ltp/fsstress.c        | 16 +++++++++---
 tests/btrfs/004       | 11 ++++----
 tests/btrfs/007       |  3 ++-
 tests/btrfs/012       |  4 +--
 tests/btrfs/028       |  6 ++---
 tests/btrfs/049       |  4 +--
 tests/btrfs/057       |  4 +--
 tests/btrfs/060       | 14 +++-------
 tests/btrfs/061       | 13 +++-------
 tests/btrfs/062       | 13 +++-------
 tests/btrfs/063       | 13 +++-------
 tests/btrfs/064       | 13 +++-------
 tests/btrfs/065       | 14 +++-------
 tests/btrfs/066       | 14 +++-------
 tests/btrfs/067       | 14 +++-------
 tests/btrfs/068       | 14 +++-------
 tests/btrfs/069       | 13 +++-------
 tests/btrfs/070       | 13 +++-------
 tests/btrfs/071       | 13 +++-------
 tests/btrfs/072       | 14 +++-------
 tests/btrfs/073       | 13 +++-------
 tests/btrfs/074       | 13 +++-------
 tests/btrfs/078       | 12 ++-------
 tests/btrfs/100       |  4 +--
 tests/btrfs/101       |  4 +--
 tests/btrfs/136       |  6 ++---
 tests/btrfs/192       | 11 ++++----
 tests/btrfs/195       |  2 +-
 tests/btrfs/212       | 14 +++++-----
 tests/btrfs/232       |  4 +--
 tests/btrfs/252       |  5 ++--
 tests/btrfs/261       |  2 +-
 tests/btrfs/284       |  4 +--
 tests/btrfs/286       |  2 +-
 tests/btrfs/320       |  6 ++---
 tests/btrfs/332       |  4 +--
 tests/ext4/004        |  4 +--
 tests/ext4/057        | 14 +++-------
 tests/ext4/058        |  3 +--
 tests/ext4/307        |  4 +--
 tests/generic/013     | 22 ++++------------
 tests/generic/019     | 13 +++++-----
 tests/generic/051     | 20 +++------------
 tests/generic/055     |  6 ++---
 tests/generic/068     |  7 +++--
 tests/generic/070     | 12 ++-------
 tests/generic/076     | 10 +++-----
 tests/generic/076.out |  1 -
 tests/generic/083     | 13 ++--------
 tests/generic/083.out |  1 -
 tests/generic/117     |  4 +--
 tests/generic/232     |  5 ++--
 tests/generic/232.out |  1 -
 tests/generic/269     |  8 +++---
 tests/generic/270     |  9 ++++---
 tests/generic/388     | 24 ++----------------
 tests/generic/390     | 11 +++-----
 tests/generic/409     |  9 ++++---
 tests/generic/410     |  9 ++++---
 tests/generic/411     |  9 ++++---
 tests/generic/461     | 15 ++---------
 tests/generic/475     | 17 ++++---------
 tests/generic/476     | 15 +----------
 tests/generic/482     | 12 ++++-----
 tests/generic/547     |  5 ++--
 tests/generic/560     |  7 +++--
 tests/generic/561     | 17 ++++---------
 tests/generic/579     |  6 ++---
 tests/generic/585     |  4 +--
 tests/generic/589     |  7 ++---
 tests/generic/642     | 10 +-------
 tests/generic/648     | 19 ++++----------
 tests/generic/650     | 15 ++++++-----
 tests/generic/750     | 10 +++-----
 tests/generic/753     | 17 ++++---------
 tests/overlay/019     | 48 ++++++++++++++++++++++++-----------
 tests/overlay/021     |  8 +++---
 tests/xfs/006         |  7 ++---
 tests/xfs/011         | 10 +++-----
 tests/xfs/013         | 16 ++----------
 tests/xfs/017         | 13 ++--------
 tests/xfs/017.out     |  1 -
 tests/xfs/032         |  2 +-
 tests/xfs/049         |  4 +--
 tests/xfs/051         | 14 ++--------
 tests/xfs/057         |  8 +++---
 tests/xfs/077         |  2 +-
 tests/xfs/079         | 14 ++--------
 tests/xfs/104         | 14 +++-------
 tests/xfs/137         |  2 +-
 tests/xfs/141         | 12 +--------
 tests/xfs/158         |  4 +--
 tests/xfs/167         | 14 +++-------
 tests/xfs/168         |  8 +++---
 tests/xfs/264         |  4 +--
 tests/xfs/270         |  2 +-
 tests/xfs/297         |  8 +++---
 tests/xfs/305         |  8 +++---
 tests/xfs/442         | 10 +-------
 tests/xfs/538         |  4 +--
 tests/xfs/609         | 20 +++------------
 tests/xfs/610         | 20 +++------------
 105 files changed, 414 insertions(+), 675 deletions(-)
common/fuzzy +13 -15
index 14d6cb104..9181520a3 100644
@@ -12,13 +12,13 @@ _scratch_fuzz_modify() {
 	echo "+++ stressing filesystem"
 	mkdir -p $SCRATCH_MNT/data
 	[ "$FSTYP" == "xfs" ] && _xfs_force_bdev data $SCRATCH_MNT/data
-	$FSSTRESS_PROG "${fsstress_args[@]}" -d $SCRATCH_MNT/data
+	_run_fsstress "${fsstress_args[@]}" -d $SCRATCH_MNT/data
 
 	if [ "$FSTYP" = "xfs" ]; then
 		if _xfs_has_feature "$SCRATCH_MNT" realtime; then
 			mkdir -p $SCRATCH_MNT/rt
 			_xfs_force_bdev realtime $SCRATCH_MNT/rt
-			$FSSTRESS_PROG "${fsstress_args[@]}" -d $SCRATCH_MNT/rt
+			_run_fsstress "${fsstress_args[@]}" -d $SCRATCH_MNT/rt
 		else
 			echo "+++ xfs realtime not configured"
 		fi
@@ -965,7 +965,7 @@ __stress_scrub_fsx_loop() {
 			res=$?
 			echo "$mode fsx exits with $res at $(date)" >> $seqres.full
 			if [ "$res" -ne 0 ] && [ "$res" -ne 124 ]; then
-				# Stop if fsstress returns error.  Mask off
+				# Stop if fsx returns error.  Mask off
 				# the magic code 124 because that is how the
 				# timeout(1) program communicates that we ran
 				# out of time.
@@ -1096,7 +1096,7 @@ __stress_scrub_fsstress_loop() {
 
 	# As of March 2022, 2 million fsstress ops should be enough to keep
 	# any filesystem busy for a couple of hours.
-	local args=$(_scale_fsstress_args -p 4 -d $SCRATCH_MNT -n 2000000 "${focus[@]}" $FSSTRESS_AVOID)
+	local args=$(_scale_fsstress_args -p 4 -d $SCRATCH_MNT -n 2000000 "${focus[@]}")
 	echo "Running $FSSTRESS_PROG $args" >> $seqres.full
 
 	if [ -n "$remount_period" ]; then
@@ -1107,17 +1107,13 @@ __stress_scrub_fsstress_loop() {
 			# anything.
 			test "$mode" = "rw" && __stress_scrub_clean_scratch && continue
 
-			timeout -s TERM "$remount_period" $FSSTRESS_PROG \
-					$args $rw_arg >> $seqres.full
+			_run_fsstress_bg $args $rw_arg >> $seqres.full
+			sleep $remount_period
+			_kill_fsstress
 			res=$?
 			echo "$mode fsstress exits with $res at $(date)" >> $seqres.full
-			if [ "$res" -ne 0 ] && [ "$res" -ne 124 ]; then
-				# Stop if fsstress returns error.  Mask off
-				# the magic code 124 because that is how the
-				# timeout(1) program communicates that we ran
-				# out of time.
-				break;
-			fi
+			[ "$res" -ne 0 ] && break;
+
 			if [ "$mode" = "rw" ]; then
 				mode="ro"
 				rw_arg="-R"
@@ -1139,7 +1135,7 @@ __stress_scrub_fsstress_loop() {
 	while __stress_scrub_running "$end" "$runningfile"; do
 		# Need to recheck running conditions if we cleared anything
 		__stress_scrub_clean_scratch && continue
-		$FSSTRESS_PROG $args >> $seqres.full
+		_run_fsstress $args >> $seqres.full
 		echo "fsstress exits with $? at $(date)" >> $seqres.full
 	done
 	rm -f "$runningfile"
@@ -1181,7 +1177,8 @@ _scratch_xfs_stress_scrub_cleanup() {
 	# Send SIGINT so that bash won't print a 'Terminated' message that
 	# distorts the golden output.
 	echo "Killing stressor processes at $(date)" >> $seqres.full
-	$KILLALL_PROG -INT xfs_io fsstress fsx xfs_scrub >> $seqres.full 2>&1
+	_kill_fsstress
+	$KILLALL_PROG -INT xfs_io fsx xfs_scrub >> $seqres.full 2>&1
 
 	# Tests are not allowed to exit with the scratch fs frozen.  If we
 	# started a fs freeze/thaw background loop, wait for that loop to exit
@@ -1394,6 +1391,7 @@ _scratch_xfs_stress_scrub() {
 
 	"__stress_scrub_${exerciser}_loop" "$end" "$runningfile" \
 			"$remount_period" "$stress_tgt" &
+	_FSSTRESS_PID=$!
 
 	if [ -n "$freeze" ]; then
 		__stress_scrub_freeze_loop "$end" "$runningfile" &
common/preamble +1 -0
index e60cd949a..78e45d522 100644
@@ -7,6 +7,7 @@
 # Standard cleanup function.  Individual tests can override this.
 _cleanup()
 {
+	_kill_fsstress
 	cd /
 	rm -r -f $tmp.*
 }
common/rc +59 -0

65 lines not on this page · raw message

ltp/fsstress.c +13 -3
index 3d248ee25..a6840f28c 100644
@@ -444,6 +444,7 @@ void sg_handler(int signum)
 {
 	switch (signum) {
 	case SIGTERM:
+	case SIGPIPE:
 		should_stop = 1;
 		break;
 	case SIGBUS:
@@ -469,6 +470,9 @@ keep_looping(int i, int loops)
 {
 	int ret;
 
+	if (should_stop)
+		return false;
+
 	if (deadline.tv_nsec) {
 		struct timespec now;
 
@@ -732,14 +736,17 @@ int main(int argc, char **argv)
 		perror("sigaction failed");
 		exit(1);
 	}
+	if (sigaction(SIGPIPE, &action, 0)) {
+		perror("sigaction failed");
+		exit(1);
+	}
 
 	for (i = 0; i < nproc; i++) {
 		if (fork() == 0) {
 			sigemptyset(&action.sa_mask);
-			action.sa_handler = SIG_DFL;
-			if (sigaction(SIGTERM, &action, 0))
-				return 1;
 			action.sa_handler = sg_handler;
+			if (sigaction(SIGTERM, &action, 0))
+				return 1;
 			if (sigaction(SIGBUS, &action, 0))
 				return 1;
 #ifdef HAVE_SYS_PRCTL_H
@@ -1196,6 +1203,9 @@ keep_running(opnum_t opno, opnum_t operations)
 {
 	int ret;
 
+	if (should_stop)
+		return false;
+
 	if (deadline.tv_nsec) {
 		struct timespec now;
 
tests/btrfs/004 +5 -6
index 5a2ce9931..06eedde22 100755
@@ -17,6 +17,7 @@ noise_pid=0
 # Override the default cleanup function.
 _cleanup()
 {
+	_kill_fsstress
 	rm $tmp.running
 	wait
 	rm -f $tmp.*
@@ -159,8 +160,7 @@ workout()
 	_scratch_mkfs_sized $fsz >>$seqres.full 2>&1
 	_scratch_mount
 	# -w ensures that the only ops are ones which cause write I/O
-	run_check $FSSTRESS_PROG -d $SCRATCH_MNT -w -p $procs -n 2000 \
-		$FSSTRESS_AVOID
+	_run_fsstress -d $SCRATCH_MNT -w -p $procs -n 2000
 
 	_btrfs subvolume snapshot $SCRATCH_MNT \
 		$SCRATCH_MNT/$snap_name
@@ -170,15 +170,14 @@ workout()
 
 	# make some noise but ensure we're not touching existing data
 	# extents.
-	run_check $FSSTRESS_PROG -d $SCRATCH_MNT -p $procs -n 4000 \
+	_run_fsstress -d $SCRATCH_MNT -p $procs -n 4000 \
 		-z -f chown=3 -f link=1 -f mkdir=2 -f mknod=2 \
 		-f rename=2 -f setxattr=1 -f symlink=2
 
 	clean_dir="$SCRATCH_MNT/next"
 	mkdir $clean_dir
 	# now make more files to get a higher tree
-	run_check $FSSTRESS_PROG -d $clean_dir -w -p $procs -n 2000 \
-		$FSSTRESS_AVOID
+	_run_fsstress -d $clean_dir -w -p $procs -n 2000
 	run_check _scratch_unmount
 	_scratch_mount "-o atime"
 
@@ -186,7 +185,7 @@ workout()
 		# make background noise while backrefs are being walked
 		while [ -f "$tmp.running" ]; do
 			echo background fsstress >>$seqres.full
-			run_check $FSSTRESS_PROG -d $SCRATCH_MNT/bgnoise -n 999
+			_run_fsstress -d $SCRATCH_MNT/bgnoise -n 999
 			echo background rm >>$seqres.full
 			rm -rf $SCRATCH_MNT/bgnoise/
 		done &
tests/btrfs/007 +2 -1
index a7eb62162..d9cc32750 100755
@@ -18,6 +18,7 @@ _begin_fstest auto quick rw metadata send seek
 # Override the default cleanup function.
 _cleanup()
 {
+	_kill_fsstress
 	cd /
 	rm -f $tmp.*
 	rm -fr $send_files_dir
@@ -45,7 +46,7 @@ workout()
 	_scratch_mkfs_sized $fsz >>$seqres.full 2>&1
 	_scratch_mount "-o noatime"
 
-	run_check $FSSTRESS_PROG -d $SCRATCH_MNT -n $ops $FSSTRESS_AVOID -x \
+	_run_fsstress -d $SCRATCH_MNT -n $ops -x \
 		"$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base"
 
 	_btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incr
tests/btrfs/012 +2 -2
index 5811b3b33..f41d7e4eb 100755
@@ -45,7 +45,7 @@ mount -t ext4 $SCRATCH_DEV $SCRATCH_MNT
 
 echo "populating the initial ext fs:" >> $seqres.full
 mkdir "$SCRATCH_MNT/$BASENAME"
-$FSSTRESS_PROG -w -d "$SCRATCH_MNT/$BASENAME" -n 20 -p 500 >> $seqres.full
+_run_fsstress -w -d "$SCRATCH_MNT/$BASENAME" -n 20 -p 500
 
 # Create the checksum to verify later.
 $FSSUM_PROG -A -f -w $tmp.original "$SCRATCH_MNT/$BASENAME"
@@ -74,7 +74,7 @@ umount $SCRATCH_MNT/mnt
 
 echo "Generating new data on the converted btrfs" >> $seqres.full
 mkdir -p $SCRATCH_MNT/new 
-$FSSTRESS_PROG -w -d "$SCRATCH_MNT/new" -n 20 -p 500 >> $seqres.full
+_run_fsstress -w -d "$SCRATCH_MNT/new" -n 20 -p 500
 
 _scratch_unmount
 
tests/btrfs/028 +2 -4
index f64fc831d..85e42f31e 100755
@@ -32,8 +32,7 @@ args=`_scale_fsstress_args -z \
 	-f fsync=10 -n 100000 -p 2 \
 	-d $SCRATCH_MNT/stress_dir`
 echo "Run fsstress $args" >>$seqres.full
-$FSSTRESS_PROG $args >>$seqres.full &
-fsstress_pid=$!
+_run_fsstress_bg $args
 
 echo "Start balance" >>$seqres.full
 _btrfs_stress_balance -d $SCRATCH_MNT >/dev/null 2>&1 &
@@ -41,8 +40,7 @@ balance_pid=$!
 
 # 30s is enough to trigger bug
 sleep $((30*$TIME_FACTOR))
-kill $fsstress_pid &> /dev/null
-wait $fsstress_pid &> /dev/null
+_kill_fsstress
 _btrfs_kill_stress_balance_pid $balance_pid
 
 # The qgroups accounting will be checked by 'btrfs check' (fsck) after the
tests/btrfs/049 +2 -2
index 19eec7851..5c09942ee 100755
@@ -39,7 +39,7 @@ args=`_scale_fsstress_args -z \
 	-f write=10 -f creat=10 \
 	-n 1000 -p 2 -d $SCRATCH_MNT/stress_dir`
 echo "Run fsstress $args" >>$seqres.full
-$FSSTRESS_PROG $args >>$seqres.full
+_run_fsstress $args >>$seqres.full
 
 # Start and pause balance to ensure it will be restored on remount
 echo "Start balance" >>$seqres.full
@@ -68,7 +68,7 @@ $BTRFS_UTIL_PROG balance resume "$SCRATCH_MNT" &>/dev/null
 [ $? -eq 0 ] || _fail "Couldn't resume balance after device add"
 
 # Add more files so that new balance won't fish immediately
-$FSSTRESS_PROG $args >/dev/null 2>&1
+_run_fsstress $args
 
 # Now pause->resume balance. This ensures balance paused is properly set in
 # the kernel and won't trigger an assertion failure.
tests/btrfs/057 +2 -2
index 6c3999463..1e871dd1b 100755
@@ -19,12 +19,12 @@ _scratch_mkfs_sized $((1024 * 1024 * 1024)) >> $seqres.full 2>&1
 _scratch_mount
 
 # -w ensures that the only ops are ones which cause write I/O
-run_check $FSSTRESS_PROG -d $SCRATCH_MNT -w -p 5 -n 1000 $FSSTRESS_AVOID
+_run_fsstress -d $SCRATCH_MNT -w -p 5 -n 1000
 
 _btrfs subvolume snapshot $SCRATCH_MNT \
 	$SCRATCH_MNT/snap1
 
-run_check $FSSTRESS_PROG -d $SCRATCH_MNT/snap1 -w -p 5 -n 1000 $FSSTRESS_AVOID
+_run_fsstress -d $SCRATCH_MNT/snap1 -w -p 5 -n 1000
 
 _btrfs quota enable $SCRATCH_MNT
 _btrfs quota rescan -w $SCRATCH_MNT
tests/btrfs/060 +4 -10
index 75c10bd23..21f15ec89 100755
@@ -21,10 +21,7 @@ _cleanup()
 	if [ ! -z "$balance_pid" ]; then
 		_btrfs_kill_stress_balance_pid $balance_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -51,10 +48,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start balance worker: " >>$seqres.full
 	_btrfs_stress_balance $SCRATCH_MNT >/dev/null 2>&1 &
@@ -67,9 +63,7 @@ run_test()
 	echo "$subvol_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
-
+	_wait_for_fsstress
 	_btrfs_kill_stress_subvolume_pid $subvol_pid $stop_file $subvol_mnt
 	unset subvol_pid
 	_btrfs_kill_stress_balance_pid $balance_pid
tests/btrfs/061 +4 -9
index 2b3b76a7f..5a2bd7090 100755
@@ -20,10 +20,7 @@ _cleanup()
 	if [ ! -z "$scrub_pid" ]; then
 		_btrfs_kill_stress_scrub_pid $scrub_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -47,10 +44,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start balance worker: " >>$seqres.full
 	_btrfs_stress_balance $SCRATCH_MNT >/dev/null 2>&1 &
@@ -63,8 +59,7 @@ run_test()
 	echo "$scrub_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
+	_wait_for_fsstress
 	_btrfs_kill_stress_balance_pid $balance_pid
 	unset balance_pid
 	_btrfs_kill_stress_scrub_pid $scrub_pid
tests/btrfs/062 +4 -9
index 4ab7ca534..a25d6d117 100755
@@ -20,10 +20,7 @@ _cleanup()
 	if [ ! -z "$defrag_pid" ]; then
 		_btrfs_kill_stress_defrag_pid $defrag_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -48,10 +45,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start balance worker: " >>$seqres.full
 	_btrfs_stress_balance $SCRATCH_MNT >/dev/null 2>&1 &
@@ -64,8 +60,7 @@ run_test()
 	echo "$defrag_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
+	_wait_for_fsstress
 	_btrfs_kill_stress_balance_pid $balance_pid
 	unset balance_pid
 	_btrfs_kill_stress_defrag_pid $defrag_pid
tests/btrfs/063 +4 -9
index ea4275d29..7d51ff554 100755
@@ -20,10 +20,7 @@ _cleanup()
 	if [ ! -z "$remount_pid" ]; then
 		_btrfs_kill_stress_remount_compress_pid $remount_pid $SCRATCH_MNT
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -47,10 +44,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start balance worker: " >>$seqres.full
 	_btrfs_stress_balance $SCRATCH_MNT >/dev/null 2>&1 &
@@ -63,8 +59,7 @@ run_test()
 	echo "$remount_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
+	_wait_for_fsstress
 	_btrfs_kill_stress_balance_pid $balance_pid
 	unset balance_pid
 	_btrfs_kill_stress_remount_compress_pid $remount_pid $SCRATCH_MNT
tests/btrfs/064 +4 -9
index a8aa62513..3b98f327e 100755
@@ -22,10 +22,7 @@ _cleanup()
 	if [ ! -z "$replace_pid" ]; then
 		_btrfs_kill_stress_replace_pid $replace_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -57,10 +54,9 @@ run_test()
 	_scratch_mount >>$seqres.full 2>&1
 	SCRATCH_DEV_POOL=$saved_scratch_dev_pool
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	# Start both balance and replace in the background.
 	# Either balance or replace shall run, the other fails.
@@ -75,8 +71,7 @@ run_test()
 	echo "$replace_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
+	_wait_for_fsstress
 	_btrfs_kill_stress_balance_pid $balance_pid
 	unset balance_pid
 	_btrfs_kill_stress_replace_pid $replace_pid
tests/btrfs/065 +4 -10
index b87c66d6e..f0c9ffb04 100755
@@ -21,10 +21,7 @@ _cleanup()
 	if [ ! -z "$replace_pid" ]; then
 		_btrfs_kill_stress_replace_pid $replace_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -59,10 +56,9 @@ run_test()
 	_scratch_mount >>$seqres.full 2>&1
 	SCRATCH_DEV_POOL=$saved_scratch_dev_pool
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start subvolume worker: " >>$seqres.full
 	_btrfs_stress_subvolume $SCRATCH_DEV $SCRATCH_MNT subvol_$$ $subvol_mnt $stop_file >/dev/null 2>&1 &
@@ -75,9 +71,7 @@ run_test()
 	echo "$replace_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
-
+	_wait_for_fsstress
 	_btrfs_kill_stress_subvolume_pid $subvol_pid $stop_file $subvol_mnt
 	unset subvol_pid
 	_btrfs_kill_stress_replace_pid $replace_pid
tests/btrfs/066 +4 -10
index cc7cd9b72..e3a083b94 100755
@@ -21,10 +21,7 @@ _cleanup()
 	if [ ! -z "$scrub_pid" ]; then
 		_btrfs_kill_stress_scrub_pid $scrub_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -51,10 +48,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start subvolume worker: " >>$seqres.full
 	_btrfs_stress_subvolume $SCRATCH_DEV $SCRATCH_MNT subvol_$$ $subvol_mnt $stop_file >/dev/null 2>&1 &
@@ -67,9 +63,7 @@ run_test()
 	echo "$scrub_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
-
+	_wait_for_fsstress
 	_btrfs_kill_stress_subvolume_pid $subvol_pid $stop_file $subvol_mnt
 	unset subvol_pid
 	_btrfs_kill_stress_scrub_pid $scrub_pid
tests/btrfs/067 +4 -10
index 0b4730500..768993116 100755
@@ -21,10 +21,7 @@ _cleanup()
 	if [ ! -z "$defrag_pid" ]; then
 		_btrfs_kill_stress_defrag_pid $defrag_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -52,10 +49,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start subvolume worker: " >>$seqres.full
 	_btrfs_stress_subvolume $SCRATCH_DEV $SCRATCH_MNT subvol_$$ $subvol_mnt $stop_file >/dev/null 2>&1 &
@@ -68,9 +64,7 @@ run_test()
 	echo "$defrag_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
-
+	_wait_for_fsstress
 	_btrfs_kill_stress_subvolume_pid $subvol_pid $stop_file $subvol_mnt
 	unset subvol_pid
 	_btrfs_kill_stress_defrag_pid $defrag_pid
tests/btrfs/068 +4 -10
index 83e932e84..3d221259f 100755
@@ -22,10 +22,7 @@ _cleanup()
 	if [ ! -z "$remount_pid" ]; then
 		_btrfs_kill_stress_remount_compress_pid $remount_pid $SCRATCH_MNT
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -52,10 +49,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start subvolume worker: " >>$seqres.full
 	_btrfs_stress_subvolume $SCRATCH_DEV $SCRATCH_MNT subvol_$$ $subvol_mnt $stop_file >/dev/null 2>&1 &
@@ -68,9 +64,7 @@ run_test()
 	echo "$remount_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
-
+	_wait_for_fsstress
 	_btrfs_kill_stress_subvolume_pid $subvol_pid $stop_file $subvol_mnt
 	unset subvol_pid
 	_btrfs_kill_stress_remount_compress_pid $remount_pid $SCRATCH_MNT
tests/btrfs/069 +4 -9
index 510551760..7954e80a8 100755
@@ -20,10 +20,7 @@ _cleanup()
 	if [ ! -z "$scrub_pid" ]; then
 		_btrfs_kill_stress_scrub_pid $scrub_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -55,10 +52,9 @@ run_test()
 	_scratch_mount >>$seqres.full 2>&1
 	SCRATCH_DEV_POOL=$saved_scratch_dev_pool
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start replace worker: " >>$seqres.full
 	_btrfs_stress_replace $SCRATCH_MNT >>$seqres.full 2>&1 &
@@ -71,8 +67,7 @@ run_test()
 	echo "$scrub_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
+	_wait_for_fsstress
 	_btrfs_kill_stress_scrub_pid $scrub_pid
 	unset scrub_pid
 	_btrfs_kill_stress_replace_pid $replace_pid
tests/btrfs/070 +4 -9
index f2e9dfcb1..c18380034 100755
@@ -20,10 +20,7 @@ _cleanup()
 	if [ ! -z "$defrag_pid" ]; then
 		_btrfs_kill_stress_defrag_pid $defrag_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -56,10 +53,9 @@ run_test()
 	_scratch_mount >>$seqres.full 2>&1
 	SCRATCH_DEV_POOL=$saved_scratch_dev_pool
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start replace worker: " >>$seqres.full
 	_btrfs_stress_replace $SCRATCH_MNT >>$seqres.full 2>&1 &
@@ -72,8 +68,7 @@ run_test()
 	echo "$defrag_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
+	_wait_for_fsstress
 	_btrfs_kill_stress_replace_pid $replace_pid
 	unset replace_pid
 	_btrfs_kill_stress_defrag_pid $defrag_pid
tests/btrfs/071 +4 -9
index 5c65bcfe0..5c2b725b0 100755
@@ -20,10 +20,7 @@ _cleanup()
 	if [ ! -z "$remount_pid" ]; then
 		_btrfs_kill_stress_remount_compress_pid $remount_pid $SCRATCH_MNT
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -55,10 +52,9 @@ run_test()
 	_scratch_mount >>$seqres.full 2>&1
 	SCRATCH_DEV_POOL=$saved_scratch_dev_pool
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start replace worker: " >>$seqres.full
 	_btrfs_stress_replace $SCRATCH_MNT >>$seqres.full 2>&1 &
@@ -71,8 +67,7 @@ run_test()
 	echo "$remount_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
+	_wait_for_fsstress
 	_btrfs_kill_stress_replace_pid $replace_pid
 	unset replace_pid
 	_btrfs_kill_stress_remount_compress_pid $remount_pid $SCRATCH_MNT
tests/btrfs/072 +4 -10
index 0a3da5ffd..327508874 100755
@@ -20,10 +20,7 @@ _cleanup()
 	if [ ! -z "$scrub_pid" ]; then
 		_btrfs_kill_stress_scrub_pid $scrub_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -48,10 +45,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start scrub worker: " >>$seqres.full
 	_btrfs_stress_scrub $SCRATCH_MNT >/dev/null 2>&1 &
@@ -64,9 +60,7 @@ run_test()
 	echo "$defrag_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
-
+	_wait_for_fsstress
 	_btrfs_kill_stress_defrag_pid $defrag_pid
 	unset defrag_pid
 	_btrfs_kill_stress_scrub_pid $scrub_pid
tests/btrfs/073 +4 -9
index bf7e9ca7a..b77e14c91 100755
@@ -20,10 +20,7 @@ _cleanup()
 	if [ ! -z "$scrub_pid" ]; then
 		_btrfs_kill_stress_scrub_pid $scrub_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -47,10 +44,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start scrub worker: " >>$seqres.full
 	_btrfs_stress_scrub $SCRATCH_MNT >/dev/null 2>&1 &
@@ -63,8 +59,7 @@ run_test()
 	echo "$remount_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
+	_wait_for_fsstress
 	_btrfs_kill_stress_remount_compress_pid $remount_pid $SCRATCH_MNT
 	unset remount_pid
 	_btrfs_kill_stress_scrub_pid $scrub_pid
tests/btrfs/074 +4 -9
index f78267159..a752707db 100755
@@ -20,10 +20,7 @@ _cleanup()
 	if [ ! -z "$defrag_pid" ]; then
 		_btrfs_kill_stress_defrag_pid $defrag_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -48,10 +45,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start defrag worker: " >>$seqres.full
 	_btrfs_stress_defrag $SCRATCH_MNT $with_compress >/dev/null 2>&1 &
@@ -64,8 +60,7 @@ run_test()
 	echo "$remount_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
+	_wait_for_fsstress
 	_btrfs_kill_stress_remount_compress_pid $remount_pid $SCRATCH_MNT
 	unset remount_pid
 	_btrfs_kill_stress_defrag_pid $defrag_pid
tests/btrfs/078 +2 -10
index bbebeff39..b802f9d86 100755
@@ -16,14 +16,6 @@
 . ./common/preamble
 _begin_fstest auto snapshot
 
-tmp=`mktemp -d`
-
-# Override the default cleanup function.
-_cleanup()
-{
-	rm -fr $tmp
-}
-
 . ./common/filter
 
 _require_scratch
@@ -39,8 +31,8 @@ workout()
 
 	snapshot_cmd="$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT"
 	snapshot_cmd="$snapshot_cmd $SCRATCH_MNT/snap_\`date +'%H_%M_%S_%N'\`"
-	run_check $FSSTRESS_PROG -p $procs \
-	    -x "$snapshot_cmd" -X $num_snapshots -d $SCRATCH_MNT -n $ops
+	_run_fsstress -p $procs -x "$snapshot_cmd" -X $num_snapshots \
+			-d $SCRATCH_MNT -n $ops
 }
 
 ops=8000
tests/btrfs/100 +2 -2
index 46bfc4f74..a319c7bbb 100755
@@ -13,6 +13,7 @@ _begin_fstest auto replace volume eio
 # Override the default cleanup function.
 _cleanup()
 {
+	_kill_fsstress
 	_dmerror_cleanup
 	rm -f $tmp.*
 }
@@ -39,8 +40,7 @@ error_devid=`$BTRFS_UTIL_PROG filesystem show -m $SCRATCH_MNT |\
 
 snapshot_cmd="$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT"
 snapshot_cmd="$snapshot_cmd $SCRATCH_MNT/snap_\`date +'%H_%M_%S_%N'\`"
-run_check $FSSTRESS_PROG -d $SCRATCH_MNT -n 200 -p 8 $FSSTRESS_AVOID -x \
-							"$snapshot_cmd" -X 50
+_run_fsstress -d $SCRATCH_MNT -n 200 -p 8 -x "$snapshot_cmd" -X 50
 
 # now load the error into the DMERROR_DEV
 _dmerror_load_error_table
tests/btrfs/101 +2 -2
index c65e14ea0..cb14d6a07 100755
@@ -13,6 +13,7 @@ _begin_fstest auto replace volume eio
 # Override the default cleanup function.
 _cleanup()
 {
+	_kill_fsstress
 	_dmerror_cleanup
 	rm -f $tmp.*
 }
@@ -40,8 +41,7 @@ error_devid=`$BTRFS_UTIL_PROG filesystem show -m $SCRATCH_MNT |\
 
 snapshot_cmd="$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT"
 snapshot_cmd="$snapshot_cmd $SCRATCH_MNT/snap_\`date +'%H_%M_%S_%N'\`"
-run_check $FSSTRESS_PROG -d $SCRATCH_MNT -n 200 -p 8 $FSSTRESS_AVOID -x \
-							"$snapshot_cmd" -X 50
+_run_fsstress -d $SCRATCH_MNT -n 200 -p 8 -x "$snapshot_cmd" -X 50
 
 # now load the error into the DMERROR_DEV
 _dmerror_load_error_table
tests/btrfs/136 +2 -4
index 2a5280fb9..65bbcf516 100755
@@ -34,11 +34,9 @@ BTRFS_MD5SUM="$tmp.btrfs"
 populate_data(){
 	data_path=$1
 	mkdir -p $data_path
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $data_path`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $data_path`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
-	wait $fsstress_pid
+	_run_fsstress $args
 }
 
 # Create & populate an ext3 filesystem
tests/btrfs/192 +5 -6
index f7fb65b8d..cc8e1e003 100755
@@ -13,12 +13,12 @@ _begin_fstest auto replay snapshot stress recoveryloop
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
+	_kill_fsstress
 	kill -q $pid1 &> /dev/null
 	kill -q $pid2 &> /dev/null
-	"$KILLALL_PROG" -q $FSSTRESS_PROG &> /dev/null
 	wait
 	_log_writes_cleanup &> /dev/null
+	cd /
 	rm -f $tmp.*
 }
 
@@ -46,8 +46,7 @@ nr_cpus=$("$here/src/feature" -o)
 if [ $nr_cpus -gt 8 ]; then
 	nr_cpus=8
 fi
-fsstress_args=$(_scale_fsstress_args -w -d $SCRATCH_MNT -n 99999 -p $nr_cpus \
-		$FSSTRESS_AVOID)
+fsstress_args=$(_scale_fsstress_args -w -d $SCRATCH_MNT -n 99999 -p $nr_cpus )
 _log_writes_init $SCRATCH_DEV
 
 # Discard the whole devices so when some tree pointer is wrong, it won't point
@@ -135,10 +134,10 @@ pid1=$!
 delete_workload &
 pid2=$!
 
-"$FSSTRESS_PROG" $fsstress_args >> $seqres.full &
+_run_fsstress_bg $fsstress_args
 sleep $runtime
 
-"$KILLALL_PROG" -q "$FSSTRESS_PROG" &> /dev/null
+_kill_fsstress
 kill $pid1 &> /dev/null
 kill $pid2 &> /dev/null
 wait
tests/btrfs/195 +1 -1
index 72fc3a775..4dffddc1a 100755
@@ -49,7 +49,7 @@ run_testcase() {
 	_scratch_mount
 
 	# Create random filesystem with 20k write ops
-	$FSSTRESS_PROG -d $SCRATCH_MNT -w -n 10000 $FSSTRESS_AVOID >>$seqres.full 2>&1
+	_run_fsstress -d $SCRATCH_MNT -w -n 10000
 
 	_run_btrfs_balance_start -f -dconvert=$dst_type $SCRATCH_MNT >> $seqres.full
 	[ $? -eq 0 ] || echo "$1: Failed convert"
tests/btrfs/212 +7 -7
index f356d7d0f..745b9598a 100755
@@ -13,13 +13,13 @@ _begin_fstest auto balance dangerous
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
+	_kill_fsstress
 	kill $balance_pid &> /dev/null
 	kill $cancel_pid &> /dev/null
-	"$KILLALL_PROG" -q $FSSTRESS_PROG &> /dev/null
-	$BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
 	wait
+	$BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
+	cd /
+	rm -f $tmp.*
 }
 
 . ./common/filter
@@ -49,7 +49,7 @@ cancel_workload()
 	done
 }
 
-$FSSTRESS_PROG -d $SCRATCH_MNT -w -n 100000  >> $seqres.full 2>/dev/null &
+_run_fsstress_bg -d $SCRATCH_MNT -w -n 100000
 balance_workload &
 balance_pid=$!
 
@@ -58,11 +58,11 @@ cancel_pid=$!
 
 sleep $runtime
 
+_kill_fsstress
 kill $balance_pid
 kill $cancel_pid
-"$KILLALL_PROG" -q $FSSTRESS_PROG &> /dev/null
-$BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
 wait
+$BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
 
 echo "Silence is golden"
 # success, all done
tests/btrfs/232 +2 -2
index 4dcc39cc4..f843edbf7 100755
@@ -23,8 +23,8 @@ writer()
 	trap "wait; exit" SIGTERM
 
 	while true; do
-		args=`_scale_fsstress_args -p 20 -n 1000 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
-		$FSSTRESS_PROG $args >> $seqres.full
+		args=`_scale_fsstress_args -p 20 -n 1000 -d $SCRATCH_MNT/stressdir`
+		_run_fsstress $args
 	done
 }
 
tests/btrfs/252 +3 -2
index 2da02ffa1..bb111b909 100755
@@ -24,6 +24,7 @@ _begin_fstest auto send balance stress
 
 _cleanup()
 {
+	_kill_fsstress
 	if [ ! -z $balance_pid ]; then
 		kill $balance_pid &> /dev/null
 		wait $balance_pid
@@ -79,10 +80,10 @@ snapshot_cmd="$snapshot_cmd \"$snapshots_dir/snap_\`date +'%s%N'\`\""
 # it's pointless to have them.
 #
 echo "Running fsstress..." >> $seqres.full
-$FSSTRESS_PROG $FSSTRESS_AVOID -d "$data_subvol" -p 1 -w \
+_run_fsstress -d "$data_subvol" -p 1 -w \
 	       -f subvol_create=0 -f subvol_delete=0 -f snapshot=0 \
 	       -x "$snapshot_cmd" -X $num_snapshots \
-	       -n $total_fsstress_ops >> $seqres.full
+	       -n $total_fsstress_ops
 
 snapshots=(`IFS=$'\n' ls -1 "$snapshots_dir"`)
 
tests/btrfs/261 +1 -1
index 4052baaec..7a08b5667 100755
@@ -35,7 +35,7 @@ prepare_fs()
 	# Then use fsstress to generate some extra contents.
 	# Disable setattr related operations, as it may set NODATACOW which will
 	# not allow us to use btrfs checksum to verify the content.
-	$FSSTRESS_PROG -f setattr=0 -d $SCRATCH_MNT -w -n 3000 >> $seqres.full
+	_run_fsstress -f setattr=0 -d $SCRATCH_MNT -w -n 3000
 	sync
 
 	# Save the fssum of this fs
tests/btrfs/284 +2 -2
index 6c554f325..ec3bc2720 100755
@@ -49,8 +49,8 @@ run_send_test()
 
 	# Use a single process so that in case of failure it's easier to
 	# reproduce by using the same seed (logged in $seqres.full).
-	run_check $FSSTRESS_PROG -d $SCRATCH_MNT -p 1 -n $((LOAD_FACTOR * 200)) \
-		  -w $FSSTRESS_AVOID -x "$snapshot_cmd"
+	_run_fsstress -d $SCRATCH_MNT -p 1 -n $((LOAD_FACTOR * 200)) \
+		  -w -x "$snapshot_cmd"
 
 	$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/snap2 \
 			 >> $seqres.full
tests/btrfs/286 +1 -1
index b8fa27673..7909fa507 100755
@@ -31,7 +31,7 @@ workload()
 	# Use nodatasum mount option, so all data won't have checksum.
 	_scratch_mount -o nodatasum
 
-	$FSSTRESS_PROG -p 10 -n 200 -d $SCRATCH_MNT >> $seqres.full
+	_run_fsstress -p 10 -n 200 -d $SCRATCH_MNT
 	sync
 
 	# Generate fssum for later verification, here we only care
tests/btrfs/320 +2 -4
index 15549165e..bcfb96d1a 100755
@@ -28,8 +28,7 @@ _basic_test()
 	$BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | grep $subvolid >> \
 		$seqres.full 2>&1
 	[ $? -eq 0 ] || _fail "couldn't find our subvols quota group"
-	run_check $FSSTRESS_PROG -d $SCRATCH_MNT/a -w -p 1 -n 2000 \
-		$FSSTRESS_AVOID
+	_run_fsstress -d $SCRATCH_MNT/a -w -p 1 -n 2000
 	_btrfs subvolume snapshot $SCRATCH_MNT/a \
 		$SCRATCH_MNT/b
 
@@ -55,8 +54,7 @@ _rescan_test()
 	_btrfs subvolume create $SCRATCH_MNT/a
 	_btrfs quota enable $SCRATCH_MNT/a
 	subvolid=$(_btrfs_get_subvolid $SCRATCH_MNT a)
-	run_check $FSSTRESS_PROG -d $SCRATCH_MNT/a -w -p 1 -n 2000 \
-		$FSSTRESS_AVOID
+	_run_fsstress -d $SCRATCH_MNT/a -w -p 1 -n 2000
 	sync
 	output=$($BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | grep "0/$subvolid")
 	echo "qgroup values before rescan: $output" >> $seqres.full
tests/btrfs/332 +2 -2
index ab9860668..4fcdad4cd 100755
@@ -32,7 +32,7 @@ d1=$SCRATCH_MNT/d1
 d2=$SCRATCH_MNT/d2
 mkdir $d1
 mkdir $d2
-run_check $FSSTRESS_PROG -d $d1 -w -n 2000 $FSSTRESS_AVOID
+_run_fsstress -d $d1 -w -n 2000
 fssum_pre=$($FSSUM_PROG -A $SCRATCH_MNT)
 
 # enable squotas
@@ -45,7 +45,7 @@ fssum_post=$($FSSUM_PROG -A $SCRATCH_MNT)
 	|| echo "fssum $fssum_pre does not match $fssum_post after enabling squota"
 
 # do some more stuff
-run_check $FSSTRESS_PROG -d $d2 -w -n 2000 $FSSTRESS_AVOID
+_run_fsstress -d $d2 -w -n 2000
 fssum_pre=$($FSSUM_PROG -A $SCRATCH_MNT)
 _scratch_unmount
 _check_btrfs_filesystem $SCRATCH_DEV
tests/ext4/004 +2 -2
index 20cfdb444..ab2f838e9 100755
@@ -12,6 +12,7 @@ _begin_fstest auto dump
 # Override the default cleanup function.
 _cleanup()
 {
+	_kill_fsstress
 	cd /
 	rm -f $tmp.*
 	# remove the generated data, which is much and meaningless.
@@ -29,8 +30,7 @@ workout()
 	echo "Run fsstress" >> $seqres.full
 	args=`_scale_fsstress_args -z -f creat=5 -f write=20 -f mkdir=5 -n 100 -p 15 -d $dump_dir`
 	echo "fsstress $args" >> $seqres.full
-
-	$FSSTRESS_PROG $args >> $seqres.full 2>&1
+	_run_fsstress $args
 
 	echo "start Dump/Restore" >> $seqres.full
 	cd $TEST_DIR
tests/ext4/057 +3 -11
index 529f0c298..73cdf941a 100755
@@ -8,15 +8,6 @@
 . ./common/preamble
 _begin_fstest auto ioctl
 
-# Override the default cleanup function.
-_cleanup()
-{
-        cd /
-        rm -r -f $tmp.*
-        kill -9 $fsstress_pid 2>/dev/null;
-        wait > /dev/null 2>&1
-}
-
 # Import common functions.
 . ./common/filter
 
@@ -41,8 +32,7 @@ _scratch_mount
 
 # Begin fsstress while modifying UUID
 fsstress_args=$(_scale_fsstress_args -d $SCRATCH_MNT -p 15 -n 999999)
-$FSSTRESS_PROG $fsstress_args >> $seqres.full &
-fsstress_pid=$!
+_run_fsstress_bg $fsstress_args
 
 for n in $(seq 1 20); do
         new_uuid=$($UUIDGEN_PROG)
@@ -57,6 +47,8 @@ for n in $(seq 1 20); do
         fi
 done
 
+_kill_fsstress
+
 # success, all done
 echo "Silence is golden"
 status=0
tests/ext4/058 +1 -2
index a7fc5e6c6..f85364964 100755
@@ -13,7 +13,6 @@
 . ./common/preamble
 _begin_fstest auto quick
 
-
 _supported_fs ext4
 _fixed_by_kernel_commit a08f789d2ab5 \
 	"ext4: fix bug_on ext4_mb_use_inode_pa"
@@ -23,7 +22,7 @@ _require_scratch
 _scratch_mkfs -g 256 >> $seqres.full 2>&1 || _fail "mkfs failed"
 _scratch_mount
 
-$FSSTRESS_PROG -d $SCRATCH_MNT/stress -n 1000 >> $seqres.full 2>&1
+_run_fsstress -d $SCRATCH_MNT/stress -n 1000
 
 echo "Silence is golden"
 
tests/ext4/307 +2 -2
index f7c95c51b..8361f0431 100755
@@ -19,9 +19,9 @@ _workout()
 	echo ""
 	echo "Run fsstress"
 	out=$SCRATCH_MNT/fsstress.$$
-	args=`_scale_fsstress_args -p4 -n999 -f setattr=1 $FSSTRESS_AVOID -d $out`
+	args=`_scale_fsstress_args -p4 -n999 -f setattr=1 -d $out`
 	echo "fsstress $args" >> $seqres.full
-	$FSSTRESS_PROG $args >> $seqres.full
+	_run_fsstress $args
 	find $out -type f > $out.list
 	cat $out.list | xargs  md5sum > $out.md5sum
 	usage=`du -sch $out | tail -n1 | gawk '{ print $1 }'`
tests/generic/013 +5 -17
index 3a000b97f..0b86d8b33 100755
@@ -9,18 +9,6 @@
 . ./common/preamble
 _begin_fstest other ioctl udf auto quick
 
-status=0	# success is the default!
-
-# Override the default cleanup function.
-_cleanup()
-{
-    cd /
-    # we might get here with a RO FS
-    _test_cycle_mount
-    # now remove fsstress directory.
-    rm -rf $TEST_DIR/fsstress.$$.*
-}
-
 # Import common functions.
 . ./common/filter
 
@@ -30,7 +18,7 @@ _do_test()
     _param="$2"
     _count="$3"
 
-    out=$TEST_DIR/fsstress.$$.$_n
+    out=$TEST_DIR/fsstress.$seq.$_n
     rm -rf $out
     if ! mkdir $out
     then
@@ -44,9 +32,8 @@ _do_test()
     echo "fsstress.$_n : $_param"
     echo "-----------------------------------------------"
     # -m limits number of users/groups so check doesn't fail (malloc) later
-    dbgoutfile=$seqres.full
-    if ! $FSSTRESS_PROG $_param $FSSTRESS_AVOID -v -m 8 -n $_count -d $out >>$dbgoutfile 2>&1
-    then
+    _run_fsstress $_param -v -m 8 -n $_count -d $out
+    if [ $? -ne 0 ]; then
         echo "    fsstress (count=$_count) returned $? - see $seqres.full"
         echo "--------------------------------------"       >>$seqres.full
         echo "$_n - output from fsstress:"                  >>$seqres.full
@@ -62,6 +49,7 @@ _do_test()
 _require_test
 
 echo "brevity is wit..."
+status=0
 
 count=1000
 procs=20
@@ -80,7 +68,7 @@ _do_test 2 "-p $procs -r" $count
 
 _do_test 3 "-p 4 -z -f rmdir=10 -f link=10 -f creat=10 -f mkdir=10 -f rename=30 -f stat=30 -f unlink=30 -f truncate=20" $count
 
-# if all ok by here then probably don't need $seqres.full
+rm -rf $TEST_DIR/fsstress.$seq.*
 
 exit
 
tests/generic/019 +6 -7
index fe117ac8a..bed916b53 100755
@@ -22,7 +22,9 @@ _require_fail_make_request
 # Override the default cleanup function.
 _cleanup()
 {
-	kill $fs_pid $fio_pid &> /dev/null
+	_kill_fsstress
+	kill $fio_pid &> /dev/null
+	wait
 	_disallow_fail_make_request
 	cd /
 	rm -r -f $tmp.*
@@ -85,13 +87,12 @@ FSSTRESS_AVOID="$FSSTRESS_AVOID -ffsync=0 -fsync=0 -ffdatasync=0 -f setattr=1"
 _workout()
 {
 	out=$SCRATCH_MNT/fsstress.$$
-	args=`_scale_fsstress_args -p 1 -n999999999 -f setattr=0 $FSSTRESS_AVOID -d $out`
+	args=`_scale_fsstress_args -p 1 -n999999999 -f setattr=0 -d $out`
 	echo ""
 	echo "Start fsstress.."
 	echo ""
 	echo "fsstress $args" >> $seqres.full
-	$FSSTRESS_PROG $args >> $seqres.full 2>&1 &
-	fs_pid=$!
+	_run_fsstress_bg $args
 	echo "Start fio.."
 	cat $fio_config >>  $seqres.full
 	$FIO_PROG $fio_config >> $seqres.full 2>&1 &
@@ -107,10 +108,8 @@ _workout()
 	    >> $seqres.full 2>&1 && \
 	    _fail "failed: still able to perform integrity fsync on $SCRATCH_MNT"
 
-	kill $fs_pid &> /dev/null
-	wait $fs_pid
+	_kill_fsstress
 	wait $fio_pid
-	unset fs_pid
 	unset fio_pid
 
 	# We expect that broken FS still can be umounted
tests/generic/051 +4 -16
index 65571fdd0..69250cde6 100755
@@ -14,18 +14,8 @@ _begin_fstest shutdown auto stress log metadata repair
 # Import common functions.
 . ./common/filter
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	_scratch_unmount 2>/dev/null
-	rm -f $tmp.*
-}
-
-
 _require_scratch
 _require_scratch_shutdown
-_require_command "$KILLALL_PROG" killall
 
 _scratch_mkfs > $seqres.full 2>&1
 _require_metadata_journaling $SCRATCH_DEV
@@ -37,25 +27,23 @@ PROCS=$((2 * LOAD_FACTOR))
 load_dir=$SCRATCH_MNT/test
 
 # let this run for a while
-$FSSTRESS_PROG $FSSTRESS_AVOID -n10000000 -p $PROCS -d $load_dir >> $seqres.full 2>&1 &
+_run_fsstress_bg -n 10000000 -p $PROCS -d $load_dir
 sleep $SLEEP_TIME
-$KILLALL_PROG -q $FSSTRESS_PROG
-wait
+_kill_fsstress
 sync
 _scratch_unmount
 
 # now mount again, run the load again, this time with a shutdown.
 _scratch_mount
 $XFS_FSR_PROG -v $load_dir >> $seqres.full 2>&1
-$FSSTRESS_PROG -n10000000 -p $PROCS -d $load_dir >> $seqres.full 2>&1 &
+_run_fsstress_bg -n10000000 -p $PROCS -d $load_dir
 sleep $SLEEP_TIME
 sync
 
 # now shutdown and unmount
 sleep 5
 _scratch_shutdown
-$KILLALL_PROG -q $FSSTRESS_PROG
-wait
+_kill_fsstress
 
 # for some reason fsstress processes manage to live on beyond the wait?
 sleep 5
tests/generic/055 +3 -3
index b1126c901..e57f95faf 100755
@@ -23,9 +23,9 @@ _do_meta()
     param="-p 4 -z -f rmdir=10 -f link=10 -f creat=10 -f mkdir=10 \
            -f rename=30 -f stat=30 -f unlink=30 -f truncate=20"
     _echofull "calling fsstress $param -m8 -n $count"
-    FSSTRESS_ARGS=`_scale_fsstress_args $param $FSSTRESS_AVOID -m 8 -n $count -d $out`
-    if ! $FSSTRESS_PROG $FSSTRESS_ARGS >>$seqres.full 2>&1
-    then
+    FSSTRESS_ARGS=`_scale_fsstress_args $param -m 8 -n $count -d $out`
+    _run_fsstress $FSSTRESS_ARGS
+    if [ $? -ne 0 ]; then
 	_echofull "fsstress failed"
     fi
 }
tests/generic/068 +3 -4
index 1e8248b9d..26c5ceea6 100755
@@ -19,8 +19,7 @@ _cleanup()
 {
 	# Make sure $SCRATCH_MNT is unfreezed
 	xfs_freeze -u $SCRATCH_MNT 2>/dev/null
-	[ -n "$pid" ] && kill -9 $pid 2>/dev/null
-	wait $pid
+	_kill_fsstress
 	cd /
 	rm -f $tmp.*
 }
@@ -54,8 +53,8 @@ touch $tmp.running
       do
       # We do both read & write IO - not only is this more realistic,
       # but it also potentially tests atime updates
-      FSSTRESS_ARGS=`_scale_fsstress_args -d $STRESS_DIR -p $procs -n $nops $FSSTRESS_AVOID`
-      $FSSTRESS_PROG $FSSTRESS_ARGS >>$seqres.full
+      FSSTRESS_ARGS=`_scale_fsstress_args -d $STRESS_DIR -p $procs -n $nops`
+      _run_fsstress $FSSTRESS_ARGS >>$seqres.full
     done
 
     rm -r $STRESS_DIR/*
tests/generic/070 +2 -10
index a8e84effd..000fc0482 100755
@@ -9,19 +9,10 @@
 . ./common/preamble
 _begin_fstest attr udf auto quick stress
 
-# Override the default cleanup function.
-_cleanup()
-{
-    cd /
-    rm -rf $TEST_DIR/fsstress
-    rm -f $tmp.*
-}
-
 # Import common functions.
 . ./common/filter
 . ./common/attr
 
-
 _require_test
 _require_attrs
 
@@ -34,7 +25,8 @@ FSSTRESS_ARGS=`_scale_fsstress_args \
 	-f attr_set=100 \
 	-f attr_remove=100 \
         -p 1 -n 10000 -S c`
-$FSSTRESS_PROG $FSSTRESS_ARGS >$seqres.full 2>&1
+_run_fsstress $FSSTRESS_ARGS
+rm -rf $TEST_DIR/fsstress
 
 status=$?
 exit
tests/generic/076 +4 -6
index b50c2df5e..100f19715 100755
@@ -21,16 +21,14 @@ _lets_get_pidst()
 # Override the default cleanup function.
 _cleanup()
 {
-	echo "*** unmount"
-	_scratch_unmount 2>/dev/null
 	_lets_get_pidst
+	cd /
+	rm -f $tmp.*
 }
-_register_cleanup "_cleanup; rm -f $tmp.*"
 
 # Import common functions.
 . ./common/filter
 
-
 _require_scratch
 _require_local_device $SCRATCH_DEV
 
@@ -48,9 +46,9 @@ echo "*** test concurrent block/fs access"
 cat $SCRATCH_DEV >/dev/null &
 pid=$!
 
-FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -p 2 -n 2000 $FSSTRESS_AVOID`
+FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -p 2 -n 2000`
 echo "run fsstress with args: $FSSTRESS_ARGS" >>$seqres.full
-$FSSTRESS_PROG $FSSTRESS_ARGS >>$seqres.full
+_run_fsstress $FSSTRESS_ARGS
 _lets_get_pidst
 
 echo "*** done"
tests/generic/076.out +0 -1
index 54a8f7096..74a65bdec 100644
@@ -2,4 +2,3 @@ QA output created by 076
 *** init fs
 *** test concurrent block/fs access
 *** done
-*** unmount
tests/generic/083 +2 -11
index ff4785eee..6aa0ea06d 100755
@@ -18,18 +18,9 @@
 . ./common/preamble
 _begin_fstest rw auto enospc stress
 
-# Override the default cleanup function.
-_cleanup()
-{
-	echo "*** unmount"
-	_scratch_unmount 2>/dev/null
-	rm -f $tmp.*
-}
-
 # Import common functions.
 . ./common/filter
 
-
 _require_scratch
 _require_no_large_scratch_dev
 
@@ -52,8 +43,8 @@ workout()
 	_scratch_mount
 
 	# -w ensures that the only ops are ones which cause write I/O
-	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs -n $nops $FSSTRESS_AVOID`
-	$FSSTRESS_PROG $FSSTRESS_ARGS >>$seqres.full
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs -n $nops`
+	_run_fsstress $FSSTRESS_ARGS >>$seqres.full
 }
 
 echo "*** test out-of-space handling for random write operations"
tests/generic/083.out +0 -1
index 025afe30e..b853eadb4 100644
@@ -1,4 +1,3 @@
 QA output created by 083
 *** test out-of-space handling for random write operations
 *** done
-*** unmount
tests/generic/117 +1 -3
index f9769be94..a34f86165 100755
@@ -63,9 +63,7 @@ echo Running fsstress in serial:
 i=0
 while [ $i -lt $ITERATIONS ]; do
     echo fsstress iteration: $i | tee -a $seqres.full
-    $FSSTRESS_PROG \
-	-d $SCRATCH_MNT/fsstress \
-	$fss_ops -S c >>$seqres.full 2>&1
+    _run_fsstress -d $SCRATCH_MNT/fsstress $fss_ops -S c
 
     let i=$i+1
 done
tests/generic/232 +2 -3
index 35934cc18..c903a5619 100755
@@ -30,13 +30,12 @@ _fsstress()
 
 	out=$SCRATCH_MNT/fsstress.$$
 	count=2000
-	args=`_scale_fsstress_args -d $out -n $count -p 7 $FSSTRESS_AVOID`
+	args=`_scale_fsstress_args -d $out -n $count -p 7`
 
 	echo "fsstress $args" >> $seqres.full
-	if ! $FSSTRESS_PROG $args | tee -a $seqres.full | _filter_num
+	if ! _run_fsstress $args
 	then
 		echo "    fsstress $args returned $?"
-		cat $tmp.out | tee -a $seqres.full
 		status=1
 	fi
 }
tests/generic/232.out +0 -1
index 5da53d4bf..9214607f0 100644
@@ -2,6 +2,5 @@ QA output created by 232
 
 Testing fsstress
 
-seed = S
 Comparing user usage
 Comparing group usage
tests/generic/269 +3 -5
index 341fcd22f..979cc15cb 100755
@@ -21,10 +21,9 @@ _workout()
 	num_iterations=10
 	enospc_time=2
 	out=$SCRATCH_MNT/fsstress.$$
-	args=`_scale_fsstress_args -p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out`
+	args=`_scale_fsstress_args -p128 -n999999999 -f setattr=1 -d $out`
 	echo "fsstress $args" >> $seqres.full
-	$FSSTRESS_PROG $args &>> $seqres.full &
-	pid=$!
+	_run_fsstress_bg $args
 	echo "Run dd writers in parallel"
 	for ((i=0; i < num_iterations; i++))
 	do
@@ -34,8 +33,7 @@ _workout()
 		sleep $enospc_time
 	done
 	echo "Killing fsstress process..." >> $seqres.full
-	kill $pid >> $seqres.full 2>&1
-	wait $pid
+	_kill_fsstress
 }
 
 _require_scratch
tests/generic/270 +5 -4
index cf523f4ea..aff379ac5 100755
@@ -28,8 +28,8 @@ _workout()
 	args=`_scale_fsstress_args -p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out`
 	echo "fsstress $args" >> $seqres.full
 	# Grant chown capability 
-	cp $FSSTRESS_PROG  $tmp.fsstress.bin
-	$SETCAP_PROG cap_chown=epi  $tmp.fsstress.bin
+	cp $FSSTRESS_PROG $_FSSTRESS_PROG
+	$SETCAP_PROG cap_chown=epi $_FSSTRESS_PROG
 
 	# io_uring accounts memory it needs under the rlimit memlocked option,
 	# which can be quite low on some setups (especially 64K pagesize). root
@@ -37,7 +37,8 @@ _workout()
 	# io_uring_queue_init fail on ENOMEM, set max locked memory to unlimited
 	# temporarily.
 	ulimit -l unlimited
-	(su $qa_user -c "$tmp.fsstress.bin $args" &) > /dev/null 2>&1
+	su $qa_user -c "$_FSSTRESS_PROG $args" > /dev/null 2>&1 &
+	_FSSTRESS_PID=$!
 
 	echo "Run dd writers in parallel"
 	for ((i=0; i < num_iterations; i++))
@@ -49,7 +50,7 @@ _workout()
 		sleep $enospc_time
 	done
 
-	$KILLALL_PROG -w $tmp.fsstress.bin
+	_kill_fsstress
 }
 
 _require_quota
tests/generic/388 +2 -22
index 89ddda31d..34609a0d3 100755
@@ -17,23 +17,9 @@
 . ./common/preamble
 _begin_fstest shutdown auto log metadata recoveryloop
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-	_scratch_unmount > /dev/null 2>&1
-}
-
-# Import common functions.
-
-# Modify as appropriate.
-
 _require_scratch
 _require_local_device $SCRATCH_DEV
 _require_scratch_shutdown
-_require_command "$KILLALL_PROG" "killall"
 
 echo "Silence is golden."
 
@@ -42,20 +28,14 @@ _require_metadata_journaling $SCRATCH_DEV
 _scratch_mount
 
 while _soak_loop_running $((50 * TIME_FACTOR)); do
-	($FSSTRESS_PROG $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p 4 >> $seqres.full &) \
-		> /dev/null 2>&1
+	_run_fsstress_bg -d $SCRATCH_MNT -n 999999 -p 4
 
 	# purposely include 0 second sleeps to test shutdown immediately after
 	# recovery
 	sleep $((RANDOM % 3))
 	_scratch_shutdown
 
-	ps -e | grep fsstress > /dev/null 2>&1
-	while [ $? -eq 0 ]; do
-		$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-		wait > /dev/null 2>&1
-		ps -e | grep fsstress > /dev/null 2>&1
-	done
+	_kill_fsstress
 
 	# Toggle between rw and ro mounts for recovery. Quit if any mount
 	# attempt fails so we don't shutdown the host fs.
tests/generic/390 +4 -7
index 02f6c5eef..52270782e 100755
@@ -18,8 +18,7 @@ _cleanup()
 	[ -n "$freeze_pids" ] && kill -9 $freeze_pids 2>/dev/null
 	wait $freeze_pids
 	xfs_freeze -u $SCRATCH_MNT 2>/dev/null
-	[ -n "$fsstress_pid" ] && kill -9 $fsstress_pid 2>/dev/null
-	wait $fsstress_pid
+	_kill_fsstress
 	rm -f $tmp.*
 }
 
@@ -49,9 +48,8 @@ fi
 nops=1000
 stress_dir="$SCRATCH_MNT/fsstress_test_dir"
 mkdir "$stress_dir"
-fsstress_args=`_scale_fsstress_args -d $stress_dir -p $procs -n $nops $FSSTRESS_AVOID`
-$FSSTRESS_PROG $fsstress_args >>$seqres.full 2>&1 &
-fsstress_pid=$!
+fsstress_args=`_scale_fsstress_args -d $stress_dir -p $procs -n $nops`
+_run_fsstress_bg $fsstress_args
 
 # Start multi-threads freeze/unfreeze
 for ((i=0; i<$procs; i++)); do
@@ -62,9 +60,8 @@ for ((i=0; i<$procs; i++)); do
 	freeze_pids="$! $freeze_pids"
 done
 
-wait $fsstress_pid
+_wait_for_fsstress
 result=$?
-unset fsstress_pid
 wait $freeze_pids
 unset freeze_pids
 
tests/generic/409 +5 -4
index b7edc2ac6..ab7e7eb55 100755
@@ -25,11 +25,12 @@ _begin_fstest auto quick mount
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
+	_kill_fsstress
 	_clear_mount_stack
 	# make sure there's no bug cause dentry isn't be freed
 	rm -rf $MNTHEAD
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -43,7 +44,7 @@ fs_stress()
 {
 	local target=$1
 
-	$FSSTRESS_PROG -z -n 50 -p 3 \
+	_run_fsstress -z -n 50 -p 3 \
 		       -f creat=5 \
 		       -f mkdir=5 \
 		       -f link=2 \
@@ -56,7 +57,7 @@ fs_stress()
 		       -f chown=1 \
 		       -f getdents=1 \
 		       -f fiemap=1 \
-		       -d $target >>$seqres.full
+		       -d $target
 	sync
 }
 
tests/generic/410 +5 -4
index 902f27144..f0f0921be 100755
@@ -33,11 +33,12 @@ _begin_fstest auto quick mount
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
+	_kill_fsstress
 	_clear_mount_stack
 	# make sure there's no bug cause dentry isn't be freed
 	rm -rf $MNTHEAD
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -51,7 +52,7 @@ fs_stress()
 {
 	local target=$1
 
-	$FSSTRESS_PROG -z -n 50 -p 3 \
+	_run_fsstress -z -n 50 -p 3 \
 		       -f creat=5 \
 		       -f mkdir=5 \
 		       -f link=2 \
@@ -64,7 +65,7 @@ fs_stress()
 		       -f chown=1 \
 		       -f getdents=1 \
 		       -f fiemap=1 \
-		       -d $target >>$seqres.full
+		       -d $target
 	sync
 }
 
tests/generic/411 +5 -4
index c35436c82..3b55b4f91 100755
@@ -14,11 +14,12 @@ _begin_fstest auto quick mount
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
+	_kill_fsstress
 	_clear_mount_stack
 	# make sure there's no bug cause dentry isn't be freed
 	rm -rf $MNTHEAD
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -32,7 +33,7 @@ fs_stress()
 {
 	local target=$1
 
-	$FSSTRESS_PROG -z -n 500 -p 5 \
+	_run_fsstress -z -n 500 -p 5 \
 		       -f creat=5 \
 		       -f mkdir=5 \
 		       -f dwrite=1 \
@@ -47,7 +48,7 @@ fs_stress()
 		       -f chown=1 \
 		       -f getdents=1 \
 		       -f fiemap=1 \
-		       -d $target >>$seqres.full
+		       -d $target
 	sync
 }
 
tests/generic/461 +2 -13
index 468ce46f0..95dd6a34d 100755
@@ -14,18 +14,8 @@ _begin_fstest auto shutdown stress
 # Import common functions.
 . ./common/filter
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	_scratch_unmount 2>/dev/null
-	rm -f $tmp.*
-}
-
-
 _require_scratch_nocheck
 _require_scratch_shutdown
-_require_command "$KILLALL_PROG" killall
 
 _scratch_mkfs > $seqres.full 2>&1
 _scratch_mount
@@ -35,15 +25,14 @@ PROCS=$((4 * LOAD_FACTOR))
 
 load_dir=$SCRATCH_MNT/test
 
-$FSSTRESS_PROG $FSSTRESS_AVOID -n10000000 -p $PROCS -d $load_dir >> $seqres.full 2>&1 &
+_run_fsstress_bg -n10000000 -p $PROCS -d $load_dir
 sleep $SLEEP_TIME
 sync
 
 # now shutdown and unmount
 sleep 5
 _scratch_shutdown
-$KILLALL_PROG -q $FSSTRESS_PROG
-wait
+_kill_fsstress
 
 # for some reason fsstress processes manage to live on beyond the wait?
 sleep 5
tests/generic/475 +5 -12
index 4b854f9ab..9df2051fb 100755
@@ -17,11 +17,11 @@ _begin_fstest shutdown auto log metadata eio recoveryloop smoketest
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
+	_kill_fsstress
 	_dmerror_unmount
 	_dmerror_cleanup
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -31,7 +31,6 @@ _cleanup()
 
 _require_scratch
 _require_dm_target error
-_require_command "$KILLALL_PROG" "killall"
 
 echo "Silence is golden."
 
@@ -41,8 +40,7 @@ _dmerror_init
 _dmerror_mount
 
 while _soak_loop_running $((50 * TIME_FACTOR)); do
-	($FSSTRESS_PROG $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p $((LOAD_FACTOR * 4)) >> $seqres.full &) \
-		> /dev/null 2>&1
+	_run_fsstress_bg -d $SCRATCH_MNT -n 999999 -p $((LOAD_FACTOR * 4))
 
 	# purposely include 0 second sleeps to test shutdown immediately after
 	# recovery
@@ -55,12 +53,7 @@ while _soak_loop_running $((50 * TIME_FACTOR)); do
 	# error table helper *without* 'lockfs'.
 	_dmerror_load_error_table
 
-	ps -e | grep fsstress > /dev/null 2>&1
-	while [ $? -eq 0 ]; do
-		$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-		wait > /dev/null 2>&1
-		ps -e | grep fsstress > /dev/null 2>&1
-	done
+	_kill_fsstress
 
 	# Mount again to replay log after loading working table, so we have a
 	# consistent XFS after test.
tests/generic/476 +1 -14
index cf7402a12..769b3745f 100755
@@ -10,20 +10,7 @@
 . ./common/preamble
 _begin_fstest auto rw long_rw stress soak smoketest
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-}
-
-# Import common functions.
-
-# Modify as appropriate.
-
 _require_scratch
-_require_command "$KILLALL_PROG" "killall"
 
 echo "Silence is golden."
 
@@ -35,7 +22,7 @@ nr_ops=$((25000 * nr_cpus * TIME_FACTOR))
 fsstress_args=(-w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus)
 test -n "$SOAK_DURATION" && fsstress_args+=(--duration="$SOAK_DURATION")
 
-$FSSTRESS_PROG $FSSTRESS_AVOID "${fsstress_args[@]}" >> $seqres.full
+_run_fsstress_bg "${fsstress_args[@]}"
 
 # success, all done
 status=0
tests/generic/482 +5 -7
index 54fee07d4..8c114ee03 100755
@@ -18,10 +18,10 @@ _begin_fstest auto metadata replay thin recoveryloop
 # failure leaves the corpse intact for post-mortem failure analysis.
 _cleanup()
 {
-	cd /
-	$KILLALL_PROG -KILL -q $FSSTRESS_PROG &> /dev/null
+	_kill_fsstress
 	_log_writes_cleanup &> /dev/null
 	_dmthin_cleanup
+	cd /
 	rm -f $tmp.*
 }
 
@@ -29,7 +29,7 @@ _cleanup()
 #_cleanup()
 #{
 #	cd /
-#	$KILLALL_PROG -KILL -q $FSSTRESS_PROG &> /dev/null
+#	[ -n "$fsstress_pid" ] && kill $fsstress_pid
 #	if [ $status -eq 0 ]; then
 #		_log_writes_cleanup &> /dev/null
 #		_dmthin_cleanup
@@ -48,7 +48,6 @@ _cleanup()
 # Modify as appropriate.
 
 _require_no_logdev
-_require_command "$KILLALL_PROG" killall
 # Use thin device as replay device, which requires $SCRATCH_DEV
 _require_scratch_nocheck
 # and we need extra device as log device
@@ -60,8 +59,7 @@ nr_cpus=$("$here/src/feature" -o)
 if [ $nr_cpus -gt 8 ]; then
 	nr_cpus=8
 fi
-fsstress_args=$(_scale_fsstress_args -w -d $SCRATCH_MNT -n 512 -p $nr_cpus \
-		$FSSTRESS_AVOID)
+fsstress_args=$(_scale_fsstress_args -w -d $SCRATCH_MNT -n 512 -p $nr_cpus)
 
 size=$(_small_fs_size_mb 200)           # 200m phys/virt size
 devsize=$((1024*1024*size / 512))
@@ -77,7 +75,7 @@ _log_writes_mkfs >> $seqres.full 2>&1
 _log_writes_mark mkfs
 
 _log_writes_mount
-run_check $FSSTRESS_PROG $fsstress_args
+_run_fsstress $fsstress_args
 _log_writes_unmount
 
 _log_writes_remove
tests/generic/547 +3 -2
index 1e3881db9..14d02b4fd 100755
@@ -14,6 +14,7 @@ _begin_fstest auto quick log
 # Override the default cleanup function.
 _cleanup()
 {
+	_kill_fsstress
 	_cleanup_flakey
 	cd /
 	rm -f $tmp.*
@@ -38,10 +39,10 @@ _init_flakey
 _mount_flakey
 
 mkdir $SCRATCH_MNT/test
-args=`_scale_fsstress_args -p 4 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/test`
+args=`_scale_fsstress_args -p 4 -n 100 -d $SCRATCH_MNT/test`
 args="$args -f mknod=0 -f symlink=0"
 echo "Running fsstress with arguments: $args" >>$seqres.full
-$FSSTRESS_PROG $args >>$seqres.full
+_run_fsstress $args
 
 # Fsync every file and directory.
 find $SCRATCH_MNT/test \( -type f -o -type d \) -exec $XFS_IO_PROG -c fsync {} \;
tests/generic/560 +3 -4
index 62983d69b..067d3ec00 100755
@@ -30,8 +30,8 @@ function iterate_dedup_verify()
 		find $dest -type f -exec md5sum {} \; \
 			> $md5file$index
 		# Make some noise
-		$FSSTRESS_PROG $fsstress_opts -d $noisedir \
-			       -n 200 -p $((5 * LOAD_FACTOR)) >/dev/null 2>&1
+		_run_fsstress $fsstress_opts -d $noisedir \
+			       -n 200 -p $((5 * LOAD_FACTOR))
 		# Too many output, so only save error output
 		$DUPEREMOVE_PROG -dr --dedupe-options=same $dupdir \
 			>/dev/null 2>$seqres.full
@@ -51,8 +51,7 @@ md5file=${tmp}.md5sum
 
 fsstress_opts="-w -r"
 # Create some files to be original data
-$FSSTRESS_PROG $fsstress_opts -d $srcdir \
-	       -n 500 -p $((5 * LOAD_FACTOR)) >/dev/null 2>&1
+_run_fsstress $fsstress_opts -d $srcdir -n 500 -p $((5 * LOAD_FACTOR))
 
 # Calculate how many test cycles will be run
 src_size=`du -ks $srcdir | awk '{print $1}'`
tests/generic/561 +5 -12
index 39e5977a3..3e931b1a7 100755
@@ -13,9 +13,9 @@ _begin_fstest auto stress dedupe
 # Override the default cleanup function.
 _cleanup()
 {
+	end_test
 	cd /
 	rm -f $tmp.*
-	end_test
 }
 
 # Import common functions.
@@ -23,28 +23,20 @@ _cleanup()
 . ./common/reflink
 
 _require_scratch_duperemove
-_require_command "$KILLALL_PROG" killall
 
 _scratch_mkfs > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 
 function end_test()
 {
-	local f=1
+	_kill_fsstress
 
 	# stop duperemove running
 	if [ -e $dupe_run ]; then
 		rm -f $dupe_run
-		$KILLALL_PROG -q $DUPEREMOVE_PROG > /dev/null 2>&1
+		kill $dedup_pids
 		wait $dedup_pids
 	fi
-
-	# Make sure all fsstress get killed
-	while [ $f -ne 0 ]; do
-		$KILLALL_PROG -q $FSSTRESS_PROG > /dev/null 2>&1
-		sleep 1
-		f=`ps -eLf | grep $FSSTRESS_PROG | grep -v "grep" | wc -l`
-	done
 }
 
 sleep_time=$((50 * TIME_FACTOR))
@@ -53,7 +45,8 @@ sleep_time=$((50 * TIME_FACTOR))
 testdir="$SCRATCH_MNT/dir"
 mkdir $testdir
 fsstress_opts="-r -n 1000 -p $((5 * LOAD_FACTOR))"
-$FSSTRESS_PROG $fsstress_opts -d $testdir -l 0 >> $seqres.full 2>&1 &
+_run_fsstress_bg $fsstress_opts -d $testdir -l 0
+
 dedup_pids=""
 dupe_run=$TEST_DIR/${seq}-running
 # Start several dedupe processes on same directory
tests/generic/579 +2 -4
index 3191342b3..4187ab0f0 100755
@@ -16,7 +16,7 @@ _begin_fstest auto stress verity
 _cleanup()
 {
 	# Stop all subprocesses.
-	$KILLALL_PROG -q $FSSTRESS_PROG
+	_kill_fsstress
 	touch $tmp.done
 	wait
 
@@ -29,7 +29,6 @@ _cleanup()
 . ./common/verity
 
 _require_scratch_verity
-_require_command "$KILLALL_PROG" killall
 _disable_fsverity_signatures
 
 _scratch_mkfs_verity &>> $seqres.full
@@ -92,8 +91,7 @@ done
 ) &
 
 # Start the fsstress processes.
-$FSSTRESS_PROG $FSSTRESS_AVOID -p $nproc_stress -l 0 -d $SCRATCH_MNT/stressdir \
-	>> $seqres.full 2>&1 &
+_run_fsstress_bg -p $nproc_stress -l 0 -d $SCRATCH_MNT/stressdir
 
 # Run for a while.
 sleep $runtime
tests/generic/585 +2 -2
index fb675c8d5..b95cd9abf 100755
@@ -23,10 +23,10 @@ _scratch_mount >> $seqres.full 2>&1
 # start a create and rename(rename_whiteout) workload. These processes
 # occur simultaneously may cause the deadlock between AGI and AGF with
 # RENAME_WHITEOUT.
-$FSSTRESS_PROG -z -n 150 -p 100 \
+_run_fsstress -z -n 150 -p 100 \
 		-f mknod=5 \
 		-f rwhiteout=5 \
-		-d $SCRATCH_MNT/fsstress >> $seqres.full 2>&1
+		-d $SCRATCH_MNT/fsstress
 
 echo Silence is golden
 
tests/generic/589 +4 -3
index 0ce16556a..969a8ac61 100755
@@ -28,11 +28,12 @@ _begin_fstest auto mount
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
+	_kill_fsstress
 	_clear_mount_stack
 	# make sure there's no bug cause dentry isn't be freed
 	rm -rf $MNTHEAD
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -46,7 +47,7 @@ fs_stress()
 {
 	local target=$1
 
-	$FSSTRESS_PROG -n 50 -p 3 -d $target >>$seqres.full
+	_run_fsstress -n 50 -p 3 -d $target
 	sync
 }
 
tests/generic/642 +1 -9
index a7112a08f..4b92a9c18 100755
@@ -10,17 +10,9 @@
 . ./common/preamble
 _begin_fstest auto soak attr long_rw stress smoketest
 
-_cleanup()
-{
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-	cd /
-	rm -f $tmp.*
-}
-
 # Modify as appropriate.
 
 _require_scratch
-_require_command "$KILLALL_PROG" "killall"
 
 echo "Silence is golden."
 
@@ -50,7 +42,7 @@ args+=('-f' "setfattr=20")
 args+=('-f' "attr_set=60")	# sets larger xattrs
 test -n "$SOAK_DURATION" && args+=(--duration="$SOAK_DURATION")
 
-$FSSTRESS_PROG "${args[@]}" $FSSTRESS_AVOID -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full
+_run_fsstress "${args[@]}" -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus
 
 # success, all done
 status=0
tests/generic/648 +5 -14
index 29d1b470b..e4c9990e1 100755
@@ -16,16 +16,15 @@ _begin_fstest shutdown auto log metadata eio recoveryloop
 
 _cleanup()
 {
-	cd /
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-	wait
+	_kill_fsstress
 	if [ -n "$loopmnt" ]; then
 		$UMOUNT_PROG $loopmnt 2>/dev/null
 		rm -r -f $loopmnt
 	fi
-	rm -f $tmp.*
 	_dmerror_unmount
 	_dmerror_cleanup
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -37,7 +36,6 @@ _cleanup()
 _require_scratch_reflink
 _require_cp_reflink
 _require_dm_target error
-_require_command "$KILLALL_PROG" "killall"
 _require_loop
 
 echo "Silence is golden."
@@ -71,8 +69,6 @@ snap_loop_fs() {
 	rm -f "$snap_aliveflag"
 }
 
-fsstress=($FSSTRESS_PROG $FSSTRESS_AVOID -d "$loopmnt" -n 999999 -p "$((LOAD_FACTOR * 4))")
-
 while _soak_loop_running $((25 * TIME_FACTOR)); do
 	touch $scratch_aliveflag
 	snap_loop_fs >> $seqres.full 2>&1 &
@@ -84,7 +80,7 @@ while _soak_loop_running $((25 * TIME_FACTOR)); do
 		break
 	fi
 
-	("${fsstress[@]}" >> $seqres.full &) > /dev/null 2>&1
+	_run_fsstress_bg -d "$loopmnt" -n 999999 -p "$((LOAD_FACTOR * 4))"
 
 	# purposely include 0 second sleeps to test shutdown immediately after
 	# recovery
@@ -98,12 +94,7 @@ while _soak_loop_running $((25 * TIME_FACTOR)); do
 	# error table helper *without* 'lockfs'.
 	_dmerror_load_error_table
 
-	ps -e | grep fsstress > /dev/null 2>&1
-	while [ $? -eq 0 ]; do
-		$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-		wait > /dev/null 2>&1
-		ps -e | grep fsstress > /dev/null 2>&1
-	done
+	_kill_fsstress
 	for ((j = 0; j < 10; j++)); do
 		test -e "$snap_aliveflag" || break
 		sleep 1
tests/generic/650 +9 -6
index 5d2cb8977..36a23e48d 100755
@@ -16,14 +16,14 @@ _begin_fstest auto rw stress soak
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
+	_kill_fsstress
 	wait	# for exercise_cpu_hotplug subprocess
 	for i in "$sysfs_cpu_dir/"cpu*/online; do
 		echo 1 > "$i" 2>/dev/null
 	done
 	test -n "$stress_dir" && rm -r -f "$stress_dir"
+	cd /
+	rm -f $tmp.*
 }
 
 exercise_cpu_hotplug()
@@ -39,7 +39,6 @@ exercise_cpu_hotplug()
 }
 
 _require_test
-_require_command "$KILLALL_PROG" "killall"
 
 sysfs_cpu_dir="/sys/devices/system/cpu"
 
@@ -71,13 +70,17 @@ test "$nr_cpus" -gt 1024 && nr_cpus="$nr_hotplug_cpus"
 fsstress_args+=(-p $nr_cpus)
 if [ -n "$SOAK_DURATION" ]; then
 	test "$SOAK_DURATION" -lt 10 && SOAK_DURATION=10
-	fsstress_args+=(--duration="$((SOAK_DURATION / 10))")
+else
+	# run for 30s per iteration max
+	SOAK_DURATION=300
 fi
+fsstress_args+=(--duration="$((SOAK_DURATION / 10))")
 
 nr_ops=$((2500 * TIME_FACTOR))
 fsstress_args+=(-n $nr_ops)
 for ((i = 0; i < 10; i++)); do
-	$FSSTRESS_PROG $FSSTRESS_AVOID -w "${fsstress_args[@]}" >> $seqres.full
+	_run_fsstress_bg -w "${fsstress_args[@]}"
+	_wait_for_fsstress
 	_test_cycle_mount
 done
 
tests/generic/750 +4 -6
index dba8021d6..5c54a5c78 100755
@@ -11,13 +11,12 @@ _begin_fstest auto rw long_rw stress soak smoketest
 
 _cleanup()
 {
-	cd /
+	_kill_fsstress
 	rm -f $runfile
-	rm -f $tmp.*
 	kill -9 $trigger_compaction_pid > /dev/null 2>&1
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-
 	wait > /dev/null 2>&1
+	rm -f $tmp.*
+	cd /
 }
 
 # Import common functions.
@@ -26,7 +25,6 @@ _cleanup()
 
 _require_scratch
 _require_vm_compaction
-_require_command "$KILLALL_PROG" "killall"
 
 # We still deadlock with this test on v6.10-rc2, we need more work.
 # but the below makes things better.
@@ -52,7 +50,7 @@ while [ -e $runfile ]; do
 done &
 trigger_compaction_pid=$!
 
-$FSSTRESS_PROG $FSSTRESS_AVOID "${fsstress_args[@]}" >> $seqres.full
+_run_fsstress "${fsstress_args[@]}"
 
 rm -f $runfile
 wait > /dev/null 2>&1
tests/generic/753 +5 -12
index e427d62d1..f5665320a 100755
@@ -16,11 +16,11 @@ _begin_fstest shutdown auto log metadata eio recoveryloop attr
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
+	_kill_fsstress
 	_dmerror_unmount
 	_dmerror_cleanup
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -30,7 +30,6 @@ _cleanup()
 
 _require_scratch
 _require_dm_target error
-_require_command "$KILLALL_PROG" "killall"
 
 echo "Silence is golden."
 
@@ -59,8 +58,7 @@ args+=('-f' "setfattr=20")
 args+=('-f' "attr_set=60")	# sets larger xattrs
 
 while _soak_loop_running $((50 * TIME_FACTOR)); do
-	($FSSTRESS_PROG "${args[@]}" $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p $((LOAD_FACTOR * 4)) >> $seqres.full &) \
-		> /dev/null 2>&1
+	_run_fsstress_bg "${args[@]}" -d $SCRATCH_MNT -n 999999 -p $((LOAD_FACTOR * 4))
 
 	# purposely include 0 second sleeps to test shutdown immediately after
 	# recovery
@@ -73,12 +71,7 @@ while _soak_loop_running $((50 * TIME_FACTOR)); do
 	# error table helper *without* 'lockfs'.
 	_dmerror_load_error_table
 
-	ps -e | grep fsstress > /dev/null 2>&1
-	while [ $? -eq 0 ]; do
-		$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-		wait > /dev/null 2>&1
-		ps -e | grep fsstress > /dev/null 2>&1
-	done
+	_kill_fsstress
 
 	# Mount again to replay log after loading working table, so we have a
 	# consistent XFS after test.
tests/overlay/019 +33 -15

68 lines not on this page · raw message

tests/overlay/021 +4 -4
index 95a9ada5d..ee5a51007 100755
@@ -33,8 +33,8 @@ d_low=$lowerdir/$testdir
 mkdir -p $d_low
 
 # Create 4K empty files in 4 directories
-echo $FSSTRESS_PROG -d $d_low -p 4 -z -f creat=1 -n 1024 -v >> $seqres.full
-$FSSTRESS_PROG -d $d_low -p 4 -z -f creat=1 -n 1024 -v >> $seqres.full 2>&1
+echo fsstress -d $d_low -p 4 -z -f creat=1 -n 1024 -v >> $seqres.full
+_run_fsstress -d $d_low -p 4 -z -f creat=1 -n 1024 -v
 echo "--------------------------------------"       >> $seqres.full
 echo "Created 1K files in lower directory.  "       >> $seqres.full
 
@@ -91,9 +91,9 @@ echo "Go team truncate!!                    "       >> $seqres.full
 # Give team 'touch' a 1 second head start.
 # Team 'truncate' players should catch up after few copy up bombs.
 sleep 1
-$FSSTRESS_PROG -d $d_top -p 4 -z -f creat=1 -n 1024 -v >> $seqres.full &
+_run_fsstress -d $d_top -p 4 -z -f creat=1 -n 1024 -v
 
-wait %1 %2 %3 %4 %5
+wait %1 %2 %3 %4
 
 echo "Silence is golden"
 status=0
tests/xfs/006 +4 -3
index 50b36947d..20fd104bc 100755
@@ -13,9 +13,10 @@ _begin_fstest auto quick mount eio
 # Override the default cleanup function.
 _cleanup()
 {
+	_kill_fsstress
+	_dmerror_cleanup
 	cd /
 	rm -f $tmp.*
-	_dmerror_cleanup
 }
 
 # Import common functions.
@@ -45,7 +46,7 @@ fi
 # start a metadata-intensive workload, but no data allocation operation.
 # Because uncompleted new space allocation I/Os may cause XFS to shutdown
 # after loading error table.
-$FSSTRESS_PROG -z -n 5000 -p 10 \
+_run_fsstress -z -n 5000 -p 10 \
 	       -f creat=10 \
 	       -f resvsp=1 \
 	       -f truncate=1 \
@@ -57,7 +58,7 @@ $FSSTRESS_PROG -z -n 5000 -p 10 \
 	       -f unlink=1 \
 	       -f symlink=1 \
 	       -f rename=1 \
-	       -d $SCRATCH_MNT/fsstress >> $seqres.full 2>&1
+	       -d $SCRATCH_MNT/fsstress
 
 # Loading error table without "--nolockfs" option. Because "--nolockfs"
 # won't freeze fs, then some running I/Os may cause XFS to shutdown
tests/xfs/011 +3 -7
index df967f098..ed69879c5 100755
@@ -26,8 +26,7 @@ _cleanup()
 {
 	# Make sure $SCRATCH_MNT is unfreezed
 	xfs_freeze -u $SCRATCH_MNT 2>/dev/null
-	$KILLALL_PROG -9 fsstress 2>/dev/null
-	wait
+	_kill_fsstress
 	cd /
 	rm -f $tmp.*
 }
@@ -102,8 +101,7 @@ _scratch_mount
 
 _check_scratch_log_state
 
-$FSSTRESS_PROG -d $SCRATCH_MNT/fsstress -n 9999999 -p 2 -S t \
-	>> $seqres.full 2>&1 &
+_run_fsstress_bg -d $SCRATCH_MNT/fsstress -n 9999999 -p 2 -S t
 
 iters=5
 while [ $iters -gt 0 ]; do
@@ -112,9 +110,7 @@ while [ $iters -gt 0 ]; do
 	iters=$((iters - 1))
 done
 
-$KILLALL_PROG $FSSTRESS_PROG
-wait
-
+_kill_fsstress
 _scratch_unmount
 
 status=0
tests/xfs/013 +2 -14
index f4f406aa9..c68c6ad85 100755
@@ -16,16 +16,6 @@ _begin_fstest auto metadata stress
 # Import common functions.
 . ./common/filter
 
-# Override the default cleanup function.
-_cleanup()
-{
-	$KILLALL_PROG -9 fsstress 2>/dev/null
-	wait
-	cd /
-	_scratch_unmount 2>/dev/null
-	rm -f $tmp.*
-}
-
 filter_enospc() {
 	sed -e '/^.*No space left on device.*/d'
 }
@@ -103,8 +93,7 @@ _create $SCRATCH_MNT/dir1 $COUNT
 _cleaner $SCRATCH_MNT $LOOPS $MINDIRS &
 
 # start a background stress workload on the fs
-$FSSTRESS_PROG -d $SCRATCH_MNT/fsstress -n 9999999 -p 2 -S t \
-	>> $seqres.full 2>&1 &
+_run_fsstress_bg -d $SCRATCH_MNT/fsstress -n 9999999 -p 2 -S t
 
 # Each cycle clones the current directory and makes a random file replacement
 # pass on the new directory. The directory is copied to the next using hard
@@ -127,8 +116,7 @@ do
 	_rand_replace $SCRATCH_MNT/dir$((i+1)) $COUNT
 done
 
-$KILLALL_PROG fsstress
-wait
+_kill_fsstress
 
 # clean out the competing fsstress allocations, then everything else
 rm -rf $SCRATCH_MNT/fsstress
tests/xfs/017 +2 -11
index c40d9cf09..263ecc753 100755
@@ -9,15 +9,6 @@
 . ./common/preamble
 _begin_fstest mount auto quick stress
 
-_register_cleanup "_cleanup; rm -f $tmp.*"
-
-# Override the default cleanup function.
-_cleanup()
-{
-    echo "*** unmount"
-    _scratch_unmount 2>/dev/null
-}
-
 # Import common functions.
 . ./common/filter
 
@@ -41,8 +32,8 @@ echo "*** test"
 for l in 0 1 2 3 4
 do
         echo "    *** test $l"
-	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -n 1000 $FSSTRESS_AVOID`
-        $FSSTRESS_PROG  $FSSTRESS_ARGS >>$seqres.full
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -n 1000`
+        _run_fsstress $FSSTRESS_ARGS
 
         _try_scratch_mount -o remount,ro \
             || _fail "remount ro failed"
tests/xfs/017.out +0 -1
index 2d11c9492..e61793df1 100644
@@ -7,4 +7,3 @@ QA output created by 017
     *** test 3
     *** test 4
 *** done
-*** unmount
tests/xfs/032 +1 -1
index 75edf0e9c..1ecc02fe0 100755
@@ -50,7 +50,7 @@ while [ $SECTORSIZE -le $PAGESIZE ]; do
 		fi
 		_scratch_mount
 		# light population of the fs
-		$FSSTRESS_PROG -n 100 -d $SCRATCH_MNT >> $seqres.full 2>&1
+		_run_fsstress -n 100 -d $SCRATCH_MNT
 		_scratch_unmount
 
 		# Test "duplicate" copy at first, if $XFS_COPY_PROG won't do it.
tests/xfs/049 +2 -2
index 668ac3745..4163a144f 100755
@@ -68,7 +68,7 @@ mount -t xfs -o loop $SCRATCH_MNT/test.xfs $SCRATCH_MNT/test >> $seqres.full 2>&
     || _fail "!!! failed to loop mount xfs"
 
 _log "stress"
-$FSSTRESS_PROG -d $SCRATCH_MNT/test -n 1000 $FSSTRESS_AVOID >> $seqres.full 2>&1 \
+_run_fsstress -d $SCRATCH_MNT/test -n 1000 \
     || _fail "!!! stress failed"
 
 _log "clean"
@@ -88,7 +88,7 @@ mount -t ext2 -o loop $SCRATCH_MNT/test/test.ext2 $SCRATCH_MNT/test2 >> $seqres.
     || _fail "!!! failed to loop mount xfs"
 
 _log "stress ext2 on xfs via loop"
-$FSSTRESS_PROG -d $SCRATCH_MNT/test2 -n 1000 $FSSTRESS_AVOID >> $seqres.full 2>&1 \
+_run_fsstress -d $SCRATCH_MNT/test2 -n 1000 \
     || _fail "!!! stress ext2 failed"
 
 _log "clean"
tests/xfs/051 +2 -12
index 43fee4c45..bb9c36da8 100755
@@ -11,15 +11,6 @@
 . ./common/preamble
 _begin_fstest shutdown auto log metadata
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 $FSSTRESS_PROG > /dev/null 2>&1
-	_scratch_unmount > /dev/null 2>&1
-}
-
 # Import common functions.
 . ./common/dmflakey
 
@@ -37,11 +28,10 @@ _scratch_mount
 
 # Start a workload and shutdown the fs. The subsequent mount will require log
 # recovery.
-$FSSTRESS_PROG -n 9999 -p 2 -w -d $SCRATCH_MNT &>> $seqres.full &
+_run_fsstress_bg -n 9999 -p 2 -w -d $SCRATCH_MNT
 sleep 5
 _scratch_shutdown -f
-$KILLALL_PROG -q $FSSTRESS_PROG
-wait
+_kill_fsstress
 _scratch_unmount
 
 # Initialize a dm-flakey device that will pass I/Os for 5s and fail thereafter.
tests/xfs/057 +4 -4
index f1c947795..62eb8b93c 100755
@@ -26,12 +26,12 @@ _begin_fstest auto log recoveryloop
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
 	[ -e /sys/fs/xfs/$sdev/errortag/log_item_pin ] &&
 		echo 0 > /sys/fs/xfs/$sdev/errortag/log_item_pin
+	_kill_fsstress
 	wait > /dev/null 2>&1
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -53,7 +53,7 @@ _scratch_mkfs_sized $((1024 * 1024 * 500)) >> $seqres.full 2>&1
 _scratch_mount
 
 # populate the fs with some data and cycle the mount to reset the log head/tail
-$FSSTRESS_PROG -d $SCRATCH_MNT -z -fcreat=1 -p 4 -n 100000 >> $seqres.full
+_run_fsstress -d $SCRATCH_MNT -z -fcreat=1 -p 4 -n 100000
 _scratch_cycle_mount || _fail "cycle mount failed"
 
 # Pin the tail and start a file removal workload. File removal tends to
tests/xfs/077 +1 -1
index 4a4ac4702..3d8ecd9f6 100755
@@ -54,7 +54,7 @@ ORIG_UUID=`_scratch_xfs_db -c "uuid" | awk '{print $NF}'`
 
 _scratch_mount
 # Put some stuff on the fs
-$FSSTRESS_PROG -d $SCRATCH_MNT -n 100 -p 4 >> $seqres.full 2>&1
+_run_fsstress -d $SCRATCH_MNT -n 100 -p 4
 _scratch_unmount
 
 # Can xfs_db change it?
tests/xfs/079 +2 -12
index 46a15ed78..794d2db49 100755
@@ -17,19 +17,9 @@
 . ./common/preamble
 _begin_fstest shutdown auto log quick
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-	wait > /dev/null 2>&1
-}
-
 # Import common functions.
 . ./common/log
 
-
 # Modify as appropriate.
 _require_scratch
 _require_v2log
@@ -43,10 +33,10 @@ _scratch_mkfs >> $seqres.full 2>&1 || _fail "mkfs failed"
 _scratch_mount "-o logbsize=32k"
 
 # Run a workload to dirty the log, wait a bit and shutdown the fs.
-$FSSTRESS_PROG -d $SCRATCH_MNT -p 4 -n 99999999 >> $seqres.full 2>&1 &
+_run_fsstress_bg -d $SCRATCH_MNT -p 4 -n 99999999
 sleep 10
 _scratch_shutdown -f
-wait
+_wait_for_fsstress
 
 # Remount with a different log buffer size. Going from 32k to 64k increases the
 # log record extended header count, as the log record header can only handle 32k
tests/xfs/104 +4 -10
index 7f11f89a5..cd625d6b7 100755
@@ -40,9 +40,8 @@ _stress_scratch()
 	procs=3
 	nops=1000
 	# -w ensures that the only ops are ones which cause write I/O
-	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
-	    -n $nops $FSSTRESS_AVOID`
-	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 &
+	args=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs -n $nops`
+	_run_fsstress_bg $args
 }
 
 _require_scratch
@@ -73,14 +72,11 @@ for i in `seq 125 -1 90`; do
 	break
 done
 
-#
 # Grow the filesystem while actively stressing it...
-# Kick off more stress threads on each iteration, grow; repeat.
-#
 while [ $size -le $endsize ]; do
 	echo "*** stressing filesystem"
 	echo "*** stressing a ${sizeb} block filesystem" >> $seqres.full
-	_stress_scratch
+	_stress_scratch 
 	sleep 1
 	size=`expr $size + $incsize`
 	sizeb=`expr $size / $dbsize`	# in data blocks
@@ -92,10 +88,8 @@ while [ $size -le $endsize ]; do
 	[ `expr $size % $modsize` -eq 0 ] && wait	# every 4th iteration
 	echo AGCOUNT=$agcount | tee -a $seqres.full
 	echo && echo >> $seqres.full
+	_wait_for_fsstress
 done
-wait	# stop for any remaining stress processes
-
-_scratch_unmount
 
 status=0
 exit
tests/xfs/137 +1 -1
index dfc653573..d97942bf6 100755
@@ -29,7 +29,7 @@ _scratch_xfs_db -x -c "logformat -c 3" >> $seqres.full 2>&1
 
 # do some work on the fs to update metadata LSNs
 _scratch_mount
-$FSSTRESS_PROG -d $SCRATCH_MNT -n 999 -p 4 -w >> $seqres.full 2>&1
+_run_fsstress -d $SCRATCH_MNT -n 999 -p 4 -w
 _scratch_unmount
 
 # Reformat to the current cycle and try to mount. This fails in most cases
tests/xfs/141 +1 -11
index 5e9067e24..b630ba10d 100755
@@ -14,19 +14,9 @@
 . ./common/preamble
 _begin_fstest auto log metadata
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-	wait > /dev/null 2>&1
-}
-
 # Import common functions.
 . ./common/inject
 
-
 # Modify as appropriate.
 _require_xfs_io_error_injection "log_bad_crc"
 _require_scratch
@@ -49,7 +39,7 @@ while [ $nr_times -gt 0 ]; do
 
 	# Run fsstress until the filesystem shuts down. It will shut down
 	# automatically when error injection triggers.
-	$FSSTRESS_PROG -d $SCRATCH_MNT -p 4 -n 999999 >> $seqres.full 2>&1
+	_run_fsstress -d $SCRATCH_MNT -p 4 -n 999999
 
 	# Verify that we can remount the fs. Log recovery should handle the torn
 	# write.
tests/xfs/158 +2 -2
index 3c4e60f0e..89bf8c851 100755
@@ -31,7 +31,7 @@ _scratch_mkfs -m crc=1,inobtcount=0 >> $seqres.full
 _scratch_mount
 
 mkdir $SCRATCH_MNT/stress
-$FSSTRESS_PROG -d $SCRATCH_MNT/stress -n 1000 >> $seqres.full
+_run_fsstress -d $SCRATCH_MNT/stress -n 1000
 echo moo > $SCRATCH_MNT/urk
 
 _scratch_unmount
@@ -56,7 +56,7 @@ _check_scratch_xfs_features NEEDSREPAIR INOBTCNT
 
 echo "Filesystem should be usable again"
 _scratch_mount
-$FSSTRESS_PROG -d $SCRATCH_MNT/stress -n 1000 >> $seqres.full
+_run_fsstress -d $SCRATCH_MNT/stress -n 1000
 _scratch_unmount
 _check_scratch_fs
 _check_scratch_xfs_features INOBTCNT
tests/xfs/167 +4 -10
index f9da261db..5ef2aa2ea 100755
@@ -9,20 +9,12 @@
 . ./common/preamble
 _begin_fstest rw metadata auto stress prealloc
 
-# Override the default cleanup function.
-_cleanup()
-{
-	$KILLALL_PROG -r -q -TERM fsstress 2> /dev/null
-	wait	# ensures all fsstress processes died
-}
-
 workout()
 {
 	procs=100
 	nops=15000
-	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -p $procs -n $nops \
-	    $FSSTRESS_AVOID`
-	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full &
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -p $procs -n $nops`
+	_run_fsstress_bg $FSSTRESS_ARGS
 	sleep 2
 }
 
@@ -52,6 +44,8 @@ rm -f $TEST_FILE
 workout
 $TEST_PROG $LOOPS $TEST_FILE
 
+_kill_fsstress
+
 echo "     *** test done"
 
 status=0
tests/xfs/168 +4 -4
index f187a336f..098e0c86a 100755
@@ -38,9 +38,8 @@ stress_scratch()
 	local procs=3
 	local nops=1000
 	# -w ensures that the only ops are ones which cause write I/O
-	local FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w \
-		-p $procs -n $nops $FSSTRESS_AVOID`
-	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1
+	local args=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs -n $nops`
+	_run_fsstress_bg $args
 }
 
 _require_scratch_xfs_shrink
@@ -73,7 +72,7 @@ while [ $totalcount -gt 0 ]; do
 	decsize=`expr  41 \* 1048576 + 1 + $RANDOM \* $RANDOM % 1048576`
 
 	while [ $size -gt $endsize ]; do
-		stress_scratch &
+		stress_scratch
 		sleep 1
 
 		decb=`expr $decsize / $dbsize`    # in data blocks
@@ -95,6 +94,7 @@ while [ $totalcount -gt 0 ]; do
 		. $tmp.growfs
 
 		size=`expr $dblocks \* $dbsize`
+		_kill_fsstress
 		_scratch_unmount
 		_scratch_xfs_repair -n >> $seqres.full 2>&1 || \
 			_fail "xfs_repair failed with shrinking $sizeb"
tests/xfs/264 +2 -2
index 109fecd1c..a6e816d3c 100755
@@ -55,7 +55,7 @@ do_test()
 	# start a metadata-intensive workload, but no data allocation operation.
 	# Because uncompleted new space allocation I/Os may cause XFS to shutdown
 	# after loading error table.
-	$FSSTRESS_PROG -z -n 5000 -p 10 \
+	_run_fsstress -z -n 5000 -p 10 \
 		       -f creat=10 \
 		       -f resvsp=1 \
 		       -f truncate=1 \
@@ -67,7 +67,7 @@ do_test()
 		       -f unlink=1 \
 		       -f symlink=1 \
 		       -f rename=1 \
-		       -d $SCRATCH_MNT/fsstress >> $seqres.full 2>&1
+		       -d $SCRATCH_MNT/fsstress
 
 	# Loading error table without "--nolockfs" option. Because "--nolockfs"
 	# won't freeze fs, then some running I/Os may cause XFS to shutdown
tests/xfs/270 +1 -1
index 3744df5a9..d3bce386a 100755
@@ -80,7 +80,7 @@ if [ $? -ne 0 ]; then
 else
 	# no hang/panic is fine
 	cat $SCRATCH_MNT/testfile > /dev/null
-	$FSSTRESS_PROG -d $SCRATCH_MNT -p 4 -n 400 >>$seqres.full 2>&1
+	_run_fsstress -d $SCRATCH_MNT -p 4 -n 400
 fi
 
 # remount as rw, kernel should reject it
tests/xfs/297 +3 -5
index 2c5b03c5c..66c5d0cc7 100755
@@ -16,8 +16,7 @@ _cleanup()
 {
 	# Make sure $SCRATCH_MNT is unfreezed
 	xfs_freeze -u $SCRATCH_MNT 2>/dev/null
-	$KILLALL_PROG -q -9 $FSSTRESS_PROG
-	wait
+	_kill_fsstress
 	cd /
 	rm -f $tmp.*
 }
@@ -37,7 +36,7 @@ _scratch_mount
 STRESS_DIR="$SCRATCH_MNT/testdir"
 mkdir -p $STRESS_DIR
 
-$FSSTRESS_PROG -d $STRESS_DIR -n 100 -p 1000 $FSSTRESS_AVOID >>$seqres.full &
+_run_fsstress_bg -d $STRESS_DIR -n 1000 -p 1000 $FSSTRESS_AVOID
 
 # Freeze/unfreeze file system randomly
 echo "Start freeze/unfreeze randomly" | tee -a $seqres.full
@@ -60,8 +59,7 @@ while [ $LOOP -gt 0 ];do
 	let LOOP=$LOOP-1
 done
 echo "Test done" | tee -a $seqres.full
-$KILLALL_PROG -q $FSSTRESS_PROG
-wait
+_kill_fsstress
 
 status=0
 exit
tests/xfs/305 +3 -5
index 0ad3ef7fb..6371ed8a6 100755
@@ -4,7 +4,7 @@
 #
 # FS QA Test No. 305
 #
-# Test to verify that turn group/project quotas off while fstress and
+# Test to verify that turn group/project quotas off while fsstress and
 # user quotas are left on.
 #
 . ./common/preamble
@@ -18,7 +18,6 @@ _begin_fstest auto quota
 
 _require_scratch
 _require_xfs_quota
-_require_command "$KILLALL_PROG" killall
 
 _scratch_mkfs_xfs -m crc=1 >/dev/null 2>&1
 
@@ -33,12 +32,11 @@ _exercise()
 	_qmount
 	mkdir -p $QUOTA_DIR
 
-	$FSSTRESS_PROG -d $QUOTA_DIR -n 1000000 -p 100 $FSSTRESS_AVOID >>$seqres.full &
+	_run_fsstress_bg -d $QUOTA_DIR -n 1000000 -p 100
 	sleep 10
 	$XFS_QUOTA_PROG -x -c "disable -$type" $SCRATCH_DEV
 	sleep 5
-	$KILLALL_PROG -q $FSSTRESS_PROG
-	wait
+	_kill_fsstress
 }
 
 echo "*** turn off group quotas"
tests/xfs/442 +1 -9
index 77d08fda5..5cbd8dd19 100755
@@ -12,14 +12,6 @@
 . ./common/preamble
 _begin_fstest auto stress clone quota
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-}
-
 # Import common functions.
 . ./common/quota
 . ./common/filter
@@ -72,7 +64,7 @@ _scratch_mount >> $seqres.full 2>&1
 
 nr_cpus=$((LOAD_FACTOR * 4))
 nr_ops=$((25000 * nr_cpus * TIME_FACTOR))
-$FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full
+_run_fsstress -w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus
 
 echo "Check quota before remount"
 check_quota_du_blocks
tests/xfs/538 +2 -2
index 57113d341..f858cadc3 100755
@@ -46,7 +46,7 @@ echo "Inject bmap_alloc_minlen_extent error tag"
 _scratch_inject_error bmap_alloc_minlen_extent 1
 
 echo "Execute fsstress"
-$FSSTRESS_PROG -d $SCRATCH_MNT \
+_run_fsstress -d $SCRATCH_MNT \
 		$(_scale_fsstress_args -p 75 -n 1000) \
 		-f bulkstat=0 \
 		-f bulkstat1=0 \
@@ -61,7 +61,7 @@ $FSSTRESS_PROG -d $SCRATCH_MNT \
 		-f readv=0 \
 		-f stat=0 \
 		-f aread=0 \
-		-f dread=0 >> $seqres.full
+		-f dread=0
 
 # success, all done
 status=0
tests/xfs/609 +3 -17
index c23b6893d..88dc3c683 100755
@@ -18,21 +18,12 @@ _stress_scratch()
 	nops=999999
 	# -w ensures that the only ops are ones which cause write I/O
 	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
-	    -n $nops $FSSTRESS_AVOID`
-	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 &
+	    -n $nops`
+	_run_fsstress_bg $FSSTRESS_ARGS >> $seqres.full 2>&1
 }
 
 _require_scratch
 _require_command "$XFS_GROWFS_PROG" xfs_growfs
-_require_command "$KILLALL_PROG" killall
-
-_cleanup()
-{
-	$KILLALL_ALL fsstress > /dev/null 2>&1
-	wait
-	cd /
-	rm -f $tmp.*
-}
 
 _scratch_mkfs_xfs | _filter_mkfs >$seqres.full 2>$tmp.mkfs
 . $tmp.mkfs	# extract blocksize and data size for scratch device
@@ -63,12 +54,7 @@ while [ $size -le $endsize ]; do
 
 	sleep $((RANDOM % 3))
 	_scratch_shutdown
-	ps -e | grep fsstress > /dev/null 2>&1
-	while [ $? -eq 0 ]; do
-		$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-		wait > /dev/null 2>&1
-		ps -e | grep fsstress > /dev/null 2>&1
-	done
+	_kill_fsstress
 	_scratch_cycle_mount
 done > /dev/null 2>&1
 wait	# stop for any remaining stress processes
tests/xfs/610 +3 -17
index 23fbd8585..8610b912c 100755
@@ -18,22 +18,13 @@ _stress_scratch()
 	nops=999999
 	# -w ensures that the only ops are ones which cause write I/O
 	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
-	    -n $nops $FSSTRESS_AVOID`
-	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 &
+	    -n $nops`
+	_run_fsstress_bg $FSSTRESS_ARGS >> $seqres.full 2>&1
 }
 
 _require_scratch
 _require_realtime
 _require_command "$XFS_GROWFS_PROG" xfs_growfs
-_require_command "$KILLALL_PROG" killall
-
-_cleanup()
-{
-	$KILLALL_ALL fsstress > /dev/null 2>&1
-	wait
-	cd /
-	rm -f $tmp.*
-}
 
 _scratch_mkfs_xfs | _filter_mkfs >$seqres.full 2>$tmp.mkfs
 . $tmp.mkfs	# extract blocksize and data size for scratch device
@@ -65,12 +56,7 @@ while [ $size -le $endsize ]; do
 
 	sleep $((RANDOM % 3))
 	_scratch_shutdown
-	ps -e | grep fsstress > /dev/null 2>&1
-	while [ $? -eq 0 ]; do
-		$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-		wait > /dev/null 2>&1
-		ps -e | grep fsstress > /dev/null 2>&1
-	done
+	_kill_fsstress
 	_scratch_cycle_mount
 done > /dev/null 2>&1
 wait	# stop for any remaining stress processes
Signature · 1 line
2.45.2
#3 lore raw

Re: [PATCH 02/40] fstests: cleanup fsstress process management

Quoted 72 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Lots of tests run fsstress in the background and then have to kill it and/or provide special cleanup functions to kill the background fsstress processes. They typically use $KILLALL_PROG for this.

Use of killall is problematic for running multiple tests in parallel in that one test can kill other tests' processes. However, because fsstress itself forks and runs children, there are very few avenues for shell scripts to ensure all the fsstress processes actually die.

With bash, it is especially nasty, because sending SIGTERM will result in bash outputting error messages ("Killed: ..." that will cause golden output mismatches and hence test failures. Hence we also need to be able to tell the main fstress process to die without triggering these messages.

To avoid the process tracking problems, we change to use pkill rather than killall (more options for process selection) and we stop using the $here/ltp/fsstress binary. Instead, we copy the $here/ltp/fsstress to $TEST_DIR/$seq.fsstress so that the test has a unique fsstress binary name. This allows the pkill filter to select just the fsstress processes the test has run. The fsstress binary name is held in _FSSTRESS_NAME, and the program to run is _FSSTRESS_PROG.

We also track the primary fsstress process ID, and store that in _FSSTRESS_PID. We do this so that we have a PID to wait against so that we don't return before the fsstress processes are dead. To this end, we add a SIGPIPE handler to the primary process so that it dying doesn't trigger bash 'killed' message output. We can send 'pkill -PIPE $_FSSTRESS_NAME' to all the fsstress processes and the primary process will then enter the "wait for children to die" processing loop before it exits. In this way, we can wait for the primary fsstress process and when it exits we know that all it's children have also finished and gone away. This makes killing fsstress invocations reliable and noise free.

This is accomplished by the helpers added to common/rc:

_run_fsstress
_run_fsstress_bg
_wait_for_fsstress
_kill_fstress

This also means that all fsstress invocations now obey FSSTRESS_AVOID environment restrictions, many of which didn't.

We add a call to _kill_fstress into the generic _cleanup() function. This means that tests using fsstress don't need to add a special local _cleanup function just to call _kill_fsstress() so that background fsstress processes are killed when the user interrupts the tests with ctrl-c.

Further, killall in the _cleanup() function is often used to attempt to expedite killing of foreground execution fsstress processes. This doesn't actually work because of the way bash processes interupt signals. That is, it waits for the currently executing process to finish execution, then runs the trap function. Hence a foreground fsstress won't ever be interrupted by ctrl-c. By implementing _run_fsstress() as a background process and a wait call, the wait() call is interrupted by the signal and the cleanup trap is run immediately. Hence the fsstress processes are killed immediately and the test exits cleanly almost immediately.

The result of all this is common, clean handling of fsstress execution and termination. There are a few exceptions for special cases, but the vast majority of tests that run fsstress use the above four wrapper functions exclusively.

---

[snip]

Quoted 15 lines
tests/btrfs/028 +1 -2
index f64fc831d..85e42f31e 100755
@@ -32,8 +32,7 @@ args=`_scale_fsstress_args -z \
 	-f fsync=10 -n 100000 -p 2 \
 	-d $SCRATCH_MNT/stress_dir`
 echo "Run fsstress $args" >>$seqres.full
-$FSSTRESS_PROG $args >>$seqres.full &
-fsstress_pid=$!
+_run_fsstress_bg $args
 
 echo "Start balance" >>$seqres.full
 _btrfs_stress_balance -d $SCRATCH_MNT >/dev/null 2>&1 &

tests/btrfs/028 balance_pid=$! · line 44

Quoted 7 lines
tests/btrfs/028 +1 -2
@@ -41,8 +40,7 @@ balance_pid=$!
 
 # 30s is enough to trigger bug
 sleep $((30*$TIME_FACTOR))
-kill $fsstress_pid &> /dev/null
-wait $fsstress_pid &> /dev/null
+_kill_fsstress
 _btrfs_kill_stress_balance_pid $balance_pid

I'll fix the conflict of this patch with:
https://lore.kernel.org/fstests/20241129031846.rije5u7o5o64v6gv@dell-per750-06-vm-08.rhts.eng.pek2.redhat.com/T/#m8174317696134bdc64d27c2f3012f7f843135f91

by below change [1], if you have more suggestions, feel free to tell me.

Thanks,
Zorro

[1]

tests/btrfs/028 +2 -5
index 4c6396ddb..b41e1077a 100755
@@ -20,10 +20,7 @@ _cleanup()
        if [ ! -z "$balance_pid" ]; then
                _btrfs_kill_stress_balance_pid $balance_pid
        fi
-       if [ ! -z "$fsstress_pid" ]; then
-               kill $fsstress_pid &> /dev/null
-               wait $fsstress_pid &> /dev/null
-       fi
+       _kill_fsstress
 }
 
 . ./common/filter
@@ -55,7 +52,7 @@ balance_pid=$!
 sleep $((30*$TIME_FACTOR))
 _kill_fsstress
 _btrfs_kill_stress_balance_pid $balance_pid
-unset fsstress_pid balance_pid
+unset balance_pid
 
#4 lore raw

Re: [PATCH 02/40] fstests: cleanup fsstress process management

In reply to Dave Chinner
Quoted 72 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Lots of tests run fsstress in the background and then have to kill it and/or provide special cleanup functions to kill the background fsstress processes. They typically use $KILLALL_PROG for this.

Use of killall is problematic for running multiple tests in parallel in that one test can kill other tests' processes. However, because fsstress itself forks and runs children, there are very few avenues for shell scripts to ensure all the fsstress processes actually die.

With bash, it is especially nasty, because sending SIGTERM will result in bash outputting error messages ("Killed: ..." that will cause golden output mismatches and hence test failures. Hence we also need to be able to tell the main fstress process to die without triggering these messages.

To avoid the process tracking problems, we change to use pkill rather than killall (more options for process selection) and we stop using the $here/ltp/fsstress binary. Instead, we copy the $here/ltp/fsstress to $TEST_DIR/$seq.fsstress so that the test has a unique fsstress binary name. This allows the pkill filter to select just the fsstress processes the test has run. The fsstress binary name is held in _FSSTRESS_NAME, and the program to run is _FSSTRESS_PROG.

We also track the primary fsstress process ID, and store that in _FSSTRESS_PID. We do this so that we have a PID to wait against so that we don't return before the fsstress processes are dead. To this end, we add a SIGPIPE handler to the primary process so that it dying doesn't trigger bash 'killed' message output. We can send 'pkill -PIPE $_FSSTRESS_NAME' to all the fsstress processes and the primary process will then enter the "wait for children to die" processing loop before it exits. In this way, we can wait for the primary fsstress process and when it exits we know that all it's children have also finished and gone away. This makes killing fsstress invocations reliable and noise free.

This is accomplished by the helpers added to common/rc:

_run_fsstress
_run_fsstress_bg
_wait_for_fsstress
_kill_fstress

This also means that all fsstress invocations now obey FSSTRESS_AVOID environment restrictions, many of which didn't.

We add a call to _kill_fstress into the generic _cleanup() function. This means that tests using fsstress don't need to add a special local _cleanup function just to call _kill_fsstress() so that background fsstress processes are killed when the user interrupts the tests with ctrl-c.

Further, killall in the _cleanup() function is often used to attempt to expedite killing of foreground execution fsstress processes. This doesn't actually work because of the way bash processes interupt signals. That is, it waits for the currently executing process to finish execution, then runs the trap function. Hence a foreground fsstress won't ever be interrupted by ctrl-c. By implementing _run_fsstress() as a background process and a wait call, the wait() call is interrupted by the signal and the cleanup trap is run immediately. Hence the fsstress processes are killed immediately and the test exits cleanly almost immediately.

The result of all this is common, clean handling of fsstress execution and termination. There are a few exceptions for special cases, but the vast majority of tests that run fsstress use the above four wrapper functions exclusively.

---

[snip]

Quoted 9 lines
common/rc +9 -0
+# Common execution handling for fsstress invocation.
+#
+# We need per-test fsstress binaries because of the way fsstress forks and
+# tests run it in the background and/or nest it. Trying to kill fsstress
+# tasks is unreliable because killing parent fsstress task does not guarantee
+# that the children get killed. Hence the historic use of killall for stopping
+# execution.
+#
+# However, we can't just kill all fsstress binaries as multiple tests might be

common/rc line 27

Quoted 10 lines
common/rc +10 -0
+# running fsstress at the same time. Hence copy the fsstress binary to a test
+# specific binary on the test device and use pkill to select that only that
+# task name to kill.
+#
+# If tasks want to start fsstress themselves (e.g. under a different uid) then
+# they can set up _FSSTRESS_BIN and record _FSSTRESS_PID themselves. Then if the
+# test is killed then it will get cleaned up automatically.
+
+_FSSTRESS_BIN="$seq.fsstress"
+_FSSTRESS_PROG="$TEST_DIR/$seq.fsstress"

Hi Dave,

I'm wondering if the "$seq.fsstress" can be unique name? For example, if generic/561, xfs/561, btrfs/561, ext4/561 run fsstress in parallel, won't they have same "561.fsstress", then "pkill -PIPE $_FSSTRESS_BIN" kills all these cases' fsstress processes?

Thanks,
Zorro

Quoted 3,305 lines
common/rc +40 -0
+_FSSTRESS_PID=""
+_wait_for_fsstress()
+{
+	local ret=0
+
+	if [ -n "$_FSSTRESS_PID" ]; then
+		wait $_FSSTRESS_PID >> $seqres.full 2>&1
+		ret=$?
+		unset _FSSTRESS_PID
+	fi
+	rm -f $_FSSTRESS_PROG
+	return $ret
+}
+
+# Standard fsstress cleanup function. Individual tests can override this.
+_kill_fsstress()
+{
+	if [ -n "$_FSSTRESS_PID" ]; then
+		# use SIGPIPE to avoid "Killed" messages from bash
+		echo "killing $_FSSTRESS_BIN" >> $seqres.full
+		pkill -PIPE $_FSSTRESS_BIN >> $seqres.full 2>&1
+		_wait_for_fsstress
+		return $?
+	fi
+}
+
+_run_fsstress_bg()
+{
+	cp -f $FSSTRESS_PROG $_FSSTRESS_PROG
+	$_FSSTRESS_PROG $FSSTRESS_AVOID $* >> $seqres.full 2>&1 &
+	_FSSTRESS_PID=$!
+}
+
+_run_fsstress()
+{
+	_run_fsstress_bg $*
+	_wait_for_fsstress
+	return $?
+}
+
 _wallclock()
 {
     date "+%s"
ltp/fsstress.c +13 -3
index 3d248ee25..a6840f28c 100644
@@ -444,6 +444,7 @@ void sg_handler(int signum)
 {
 	switch (signum) {
 	case SIGTERM:
+	case SIGPIPE:
 		should_stop = 1;
 		break;
 	case SIGBUS:
@@ -469,6 +470,9 @@ keep_looping(int i, int loops)
 {
 	int ret;
 
+	if (should_stop)
+		return false;
+
 	if (deadline.tv_nsec) {
 		struct timespec now;
 
@@ -732,14 +736,17 @@ int main(int argc, char **argv)
 		perror("sigaction failed");
 		exit(1);
 	}
+	if (sigaction(SIGPIPE, &action, 0)) {
+		perror("sigaction failed");
+		exit(1);
+	}
 
 	for (i = 0; i < nproc; i++) {
 		if (fork() == 0) {
 			sigemptyset(&action.sa_mask);
-			action.sa_handler = SIG_DFL;
-			if (sigaction(SIGTERM, &action, 0))
-				return 1;
 			action.sa_handler = sg_handler;
+			if (sigaction(SIGTERM, &action, 0))
+				return 1;
 			if (sigaction(SIGBUS, &action, 0))
 				return 1;
 #ifdef HAVE_SYS_PRCTL_H
@@ -1196,6 +1203,9 @@ keep_running(opnum_t opno, opnum_t operations)
 {
 	int ret;
 
+	if (should_stop)
+		return false;
+
 	if (deadline.tv_nsec) {
 		struct timespec now;
 
tests/btrfs/004 +5 -6
index 5a2ce9931..06eedde22 100755
@@ -17,6 +17,7 @@ noise_pid=0
 # Override the default cleanup function.
 _cleanup()
 {
+	_kill_fsstress
 	rm $tmp.running
 	wait
 	rm -f $tmp.*
@@ -159,8 +160,7 @@ workout()
 	_scratch_mkfs_sized $fsz >>$seqres.full 2>&1
 	_scratch_mount
 	# -w ensures that the only ops are ones which cause write I/O
-	run_check $FSSTRESS_PROG -d $SCRATCH_MNT -w -p $procs -n 2000 \
-		$FSSTRESS_AVOID
+	_run_fsstress -d $SCRATCH_MNT -w -p $procs -n 2000
 
 	_btrfs subvolume snapshot $SCRATCH_MNT \
 		$SCRATCH_MNT/$snap_name
@@ -170,15 +170,14 @@ workout()
 
 	# make some noise but ensure we're not touching existing data
 	# extents.
-	run_check $FSSTRESS_PROG -d $SCRATCH_MNT -p $procs -n 4000 \
+	_run_fsstress -d $SCRATCH_MNT -p $procs -n 4000 \
 		-z -f chown=3 -f link=1 -f mkdir=2 -f mknod=2 \
 		-f rename=2 -f setxattr=1 -f symlink=2
 
 	clean_dir="$SCRATCH_MNT/next"
 	mkdir $clean_dir
 	# now make more files to get a higher tree
-	run_check $FSSTRESS_PROG -d $clean_dir -w -p $procs -n 2000 \
-		$FSSTRESS_AVOID
+	_run_fsstress -d $clean_dir -w -p $procs -n 2000
 	run_check _scratch_unmount
 	_scratch_mount "-o atime"
 
@@ -186,7 +185,7 @@ workout()
 		# make background noise while backrefs are being walked
 		while [ -f "$tmp.running" ]; do
 			echo background fsstress >>$seqres.full
-			run_check $FSSTRESS_PROG -d $SCRATCH_MNT/bgnoise -n 999
+			_run_fsstress -d $SCRATCH_MNT/bgnoise -n 999
 			echo background rm >>$seqres.full
 			rm -rf $SCRATCH_MNT/bgnoise/
 		done &
tests/btrfs/007 +2 -1
index a7eb62162..d9cc32750 100755
@@ -18,6 +18,7 @@ _begin_fstest auto quick rw metadata send seek
 # Override the default cleanup function.
 _cleanup()
 {
+	_kill_fsstress
 	cd /
 	rm -f $tmp.*
 	rm -fr $send_files_dir
@@ -45,7 +46,7 @@ workout()
 	_scratch_mkfs_sized $fsz >>$seqres.full 2>&1
 	_scratch_mount "-o noatime"
 
-	run_check $FSSTRESS_PROG -d $SCRATCH_MNT -n $ops $FSSTRESS_AVOID -x \
+	_run_fsstress -d $SCRATCH_MNT -n $ops -x \
 		"$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/base"
 
 	_btrfs subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/incr
tests/btrfs/012 +2 -2
index 5811b3b33..f41d7e4eb 100755
@@ -45,7 +45,7 @@ mount -t ext4 $SCRATCH_DEV $SCRATCH_MNT
 
 echo "populating the initial ext fs:" >> $seqres.full
 mkdir "$SCRATCH_MNT/$BASENAME"
-$FSSTRESS_PROG -w -d "$SCRATCH_MNT/$BASENAME" -n 20 -p 500 >> $seqres.full
+_run_fsstress -w -d "$SCRATCH_MNT/$BASENAME" -n 20 -p 500
 
 # Create the checksum to verify later.
 $FSSUM_PROG -A -f -w $tmp.original "$SCRATCH_MNT/$BASENAME"
@@ -74,7 +74,7 @@ umount $SCRATCH_MNT/mnt
 
 echo "Generating new data on the converted btrfs" >> $seqres.full
 mkdir -p $SCRATCH_MNT/new 
-$FSSTRESS_PROG -w -d "$SCRATCH_MNT/new" -n 20 -p 500 >> $seqres.full
+_run_fsstress -w -d "$SCRATCH_MNT/new" -n 20 -p 500
 
 _scratch_unmount
 
tests/btrfs/028 +2 -4
index f64fc831d..85e42f31e 100755
@@ -32,8 +32,7 @@ args=`_scale_fsstress_args -z \
 	-f fsync=10 -n 100000 -p 2 \
 	-d $SCRATCH_MNT/stress_dir`
 echo "Run fsstress $args" >>$seqres.full
-$FSSTRESS_PROG $args >>$seqres.full &
-fsstress_pid=$!
+_run_fsstress_bg $args
 
 echo "Start balance" >>$seqres.full
 _btrfs_stress_balance -d $SCRATCH_MNT >/dev/null 2>&1 &
@@ -41,8 +40,7 @@ balance_pid=$!
 
 # 30s is enough to trigger bug
 sleep $((30*$TIME_FACTOR))
-kill $fsstress_pid &> /dev/null
-wait $fsstress_pid &> /dev/null
+_kill_fsstress
 _btrfs_kill_stress_balance_pid $balance_pid
 
 # The qgroups accounting will be checked by 'btrfs check' (fsck) after the
tests/btrfs/049 +2 -2
index 19eec7851..5c09942ee 100755
@@ -39,7 +39,7 @@ args=`_scale_fsstress_args -z \
 	-f write=10 -f creat=10 \
 	-n 1000 -p 2 -d $SCRATCH_MNT/stress_dir`
 echo "Run fsstress $args" >>$seqres.full
-$FSSTRESS_PROG $args >>$seqres.full
+_run_fsstress $args >>$seqres.full
 
 # Start and pause balance to ensure it will be restored on remount
 echo "Start balance" >>$seqres.full
@@ -68,7 +68,7 @@ $BTRFS_UTIL_PROG balance resume "$SCRATCH_MNT" &>/dev/null
 [ $? -eq 0 ] || _fail "Couldn't resume balance after device add"
 
 # Add more files so that new balance won't fish immediately
-$FSSTRESS_PROG $args >/dev/null 2>&1
+_run_fsstress $args
 
 # Now pause->resume balance. This ensures balance paused is properly set in
 # the kernel and won't trigger an assertion failure.
tests/btrfs/057 +2 -2
index 6c3999463..1e871dd1b 100755
@@ -19,12 +19,12 @@ _scratch_mkfs_sized $((1024 * 1024 * 1024)) >> $seqres.full 2>&1
 _scratch_mount
 
 # -w ensures that the only ops are ones which cause write I/O
-run_check $FSSTRESS_PROG -d $SCRATCH_MNT -w -p 5 -n 1000 $FSSTRESS_AVOID
+_run_fsstress -d $SCRATCH_MNT -w -p 5 -n 1000
 
 _btrfs subvolume snapshot $SCRATCH_MNT \
 	$SCRATCH_MNT/snap1
 
-run_check $FSSTRESS_PROG -d $SCRATCH_MNT/snap1 -w -p 5 -n 1000 $FSSTRESS_AVOID
+_run_fsstress -d $SCRATCH_MNT/snap1 -w -p 5 -n 1000
 
 _btrfs quota enable $SCRATCH_MNT
 _btrfs quota rescan -w $SCRATCH_MNT
tests/btrfs/060 +4 -10
index 75c10bd23..21f15ec89 100755
@@ -21,10 +21,7 @@ _cleanup()
 	if [ ! -z "$balance_pid" ]; then
 		_btrfs_kill_stress_balance_pid $balance_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -51,10 +48,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start balance worker: " >>$seqres.full
 	_btrfs_stress_balance $SCRATCH_MNT >/dev/null 2>&1 &
@@ -67,9 +63,7 @@ run_test()
 	echo "$subvol_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
-
+	_wait_for_fsstress
 	_btrfs_kill_stress_subvolume_pid $subvol_pid $stop_file $subvol_mnt
 	unset subvol_pid
 	_btrfs_kill_stress_balance_pid $balance_pid
tests/btrfs/061 +4 -9
index 2b3b76a7f..5a2bd7090 100755
@@ -20,10 +20,7 @@ _cleanup()
 	if [ ! -z "$scrub_pid" ]; then
 		_btrfs_kill_stress_scrub_pid $scrub_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -47,10 +44,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start balance worker: " >>$seqres.full
 	_btrfs_stress_balance $SCRATCH_MNT >/dev/null 2>&1 &
@@ -63,8 +59,7 @@ run_test()
 	echo "$scrub_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
+	_wait_for_fsstress
 	_btrfs_kill_stress_balance_pid $balance_pid
 	unset balance_pid
 	_btrfs_kill_stress_scrub_pid $scrub_pid
tests/btrfs/062 +4 -9
index 4ab7ca534..a25d6d117 100755
@@ -20,10 +20,7 @@ _cleanup()
 	if [ ! -z "$defrag_pid" ]; then
 		_btrfs_kill_stress_defrag_pid $defrag_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -48,10 +45,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start balance worker: " >>$seqres.full
 	_btrfs_stress_balance $SCRATCH_MNT >/dev/null 2>&1 &
@@ -64,8 +60,7 @@ run_test()
 	echo "$defrag_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
+	_wait_for_fsstress
 	_btrfs_kill_stress_balance_pid $balance_pid
 	unset balance_pid
 	_btrfs_kill_stress_defrag_pid $defrag_pid
tests/btrfs/063 +4 -9
index ea4275d29..7d51ff554 100755
@@ -20,10 +20,7 @@ _cleanup()
 	if [ ! -z "$remount_pid" ]; then
 		_btrfs_kill_stress_remount_compress_pid $remount_pid $SCRATCH_MNT
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -47,10 +44,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start balance worker: " >>$seqres.full
 	_btrfs_stress_balance $SCRATCH_MNT >/dev/null 2>&1 &
@@ -63,8 +59,7 @@ run_test()
 	echo "$remount_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
+	_wait_for_fsstress
 	_btrfs_kill_stress_balance_pid $balance_pid
 	unset balance_pid
 	_btrfs_kill_stress_remount_compress_pid $remount_pid $SCRATCH_MNT
tests/btrfs/064 +4 -9
index a8aa62513..3b98f327e 100755
@@ -22,10 +22,7 @@ _cleanup()
 	if [ ! -z "$replace_pid" ]; then
 		_btrfs_kill_stress_replace_pid $replace_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -57,10 +54,9 @@ run_test()
 	_scratch_mount >>$seqres.full 2>&1
 	SCRATCH_DEV_POOL=$saved_scratch_dev_pool
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	# Start both balance and replace in the background.
 	# Either balance or replace shall run, the other fails.
@@ -75,8 +71,7 @@ run_test()
 	echo "$replace_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
+	_wait_for_fsstress
 	_btrfs_kill_stress_balance_pid $balance_pid
 	unset balance_pid
 	_btrfs_kill_stress_replace_pid $replace_pid
tests/btrfs/065 +4 -10
index b87c66d6e..f0c9ffb04 100755
@@ -21,10 +21,7 @@ _cleanup()
 	if [ ! -z "$replace_pid" ]; then
 		_btrfs_kill_stress_replace_pid $replace_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -59,10 +56,9 @@ run_test()
 	_scratch_mount >>$seqres.full 2>&1
 	SCRATCH_DEV_POOL=$saved_scratch_dev_pool
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start subvolume worker: " >>$seqres.full
 	_btrfs_stress_subvolume $SCRATCH_DEV $SCRATCH_MNT subvol_$$ $subvol_mnt $stop_file >/dev/null 2>&1 &
@@ -75,9 +71,7 @@ run_test()
 	echo "$replace_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
-
+	_wait_for_fsstress
 	_btrfs_kill_stress_subvolume_pid $subvol_pid $stop_file $subvol_mnt
 	unset subvol_pid
 	_btrfs_kill_stress_replace_pid $replace_pid
tests/btrfs/066 +4 -10
index cc7cd9b72..e3a083b94 100755
@@ -21,10 +21,7 @@ _cleanup()
 	if [ ! -z "$scrub_pid" ]; then
 		_btrfs_kill_stress_scrub_pid $scrub_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -51,10 +48,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start subvolume worker: " >>$seqres.full
 	_btrfs_stress_subvolume $SCRATCH_DEV $SCRATCH_MNT subvol_$$ $subvol_mnt $stop_file >/dev/null 2>&1 &
@@ -67,9 +63,7 @@ run_test()
 	echo "$scrub_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
-
+	_wait_for_fsstress
 	_btrfs_kill_stress_subvolume_pid $subvol_pid $stop_file $subvol_mnt
 	unset subvol_pid
 	_btrfs_kill_stress_scrub_pid $scrub_pid
tests/btrfs/067 +4 -10
index 0b4730500..768993116 100755
@@ -21,10 +21,7 @@ _cleanup()
 	if [ ! -z "$defrag_pid" ]; then
 		_btrfs_kill_stress_defrag_pid $defrag_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -52,10 +49,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start subvolume worker: " >>$seqres.full
 	_btrfs_stress_subvolume $SCRATCH_DEV $SCRATCH_MNT subvol_$$ $subvol_mnt $stop_file >/dev/null 2>&1 &
@@ -68,9 +64,7 @@ run_test()
 	echo "$defrag_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
-
+	_wait_for_fsstress
 	_btrfs_kill_stress_subvolume_pid $subvol_pid $stop_file $subvol_mnt
 	unset subvol_pid
 	_btrfs_kill_stress_defrag_pid $defrag_pid
tests/btrfs/068 +4 -10
index 83e932e84..3d221259f 100755
@@ -22,10 +22,7 @@ _cleanup()
 	if [ ! -z "$remount_pid" ]; then
 		_btrfs_kill_stress_remount_compress_pid $remount_pid $SCRATCH_MNT
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -52,10 +49,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start subvolume worker: " >>$seqres.full
 	_btrfs_stress_subvolume $SCRATCH_DEV $SCRATCH_MNT subvol_$$ $subvol_mnt $stop_file >/dev/null 2>&1 &
@@ -68,9 +64,7 @@ run_test()
 	echo "$remount_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
-
+	_wait_for_fsstress
 	_btrfs_kill_stress_subvolume_pid $subvol_pid $stop_file $subvol_mnt
 	unset subvol_pid
 	_btrfs_kill_stress_remount_compress_pid $remount_pid $SCRATCH_MNT
tests/btrfs/069 +4 -9
index 510551760..7954e80a8 100755
@@ -20,10 +20,7 @@ _cleanup()
 	if [ ! -z "$scrub_pid" ]; then
 		_btrfs_kill_stress_scrub_pid $scrub_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -55,10 +52,9 @@ run_test()
 	_scratch_mount >>$seqres.full 2>&1
 	SCRATCH_DEV_POOL=$saved_scratch_dev_pool
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start replace worker: " >>$seqres.full
 	_btrfs_stress_replace $SCRATCH_MNT >>$seqres.full 2>&1 &
@@ -71,8 +67,7 @@ run_test()
 	echo "$scrub_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
+	_wait_for_fsstress
 	_btrfs_kill_stress_scrub_pid $scrub_pid
 	unset scrub_pid
 	_btrfs_kill_stress_replace_pid $replace_pid
tests/btrfs/070 +4 -9
index f2e9dfcb1..c18380034 100755
@@ -20,10 +20,7 @@ _cleanup()
 	if [ ! -z "$defrag_pid" ]; then
 		_btrfs_kill_stress_defrag_pid $defrag_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -56,10 +53,9 @@ run_test()
 	_scratch_mount >>$seqres.full 2>&1
 	SCRATCH_DEV_POOL=$saved_scratch_dev_pool
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start replace worker: " >>$seqres.full
 	_btrfs_stress_replace $SCRATCH_MNT >>$seqres.full 2>&1 &
@@ -72,8 +68,7 @@ run_test()
 	echo "$defrag_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
+	_wait_for_fsstress
 	_btrfs_kill_stress_replace_pid $replace_pid
 	unset replace_pid
 	_btrfs_kill_stress_defrag_pid $defrag_pid
tests/btrfs/071 +4 -9
index 5c65bcfe0..5c2b725b0 100755
@@ -20,10 +20,7 @@ _cleanup()
 	if [ ! -z "$remount_pid" ]; then
 		_btrfs_kill_stress_remount_compress_pid $remount_pid $SCRATCH_MNT
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -55,10 +52,9 @@ run_test()
 	_scratch_mount >>$seqres.full 2>&1
 	SCRATCH_DEV_POOL=$saved_scratch_dev_pool
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start replace worker: " >>$seqres.full
 	_btrfs_stress_replace $SCRATCH_MNT >>$seqres.full 2>&1 &
@@ -71,8 +67,7 @@ run_test()
 	echo "$remount_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
+	_wait_for_fsstress
 	_btrfs_kill_stress_replace_pid $replace_pid
 	unset replace_pid
 	_btrfs_kill_stress_remount_compress_pid $remount_pid $SCRATCH_MNT
tests/btrfs/072 +4 -10
index 0a3da5ffd..327508874 100755
@@ -20,10 +20,7 @@ _cleanup()
 	if [ ! -z "$scrub_pid" ]; then
 		_btrfs_kill_stress_scrub_pid $scrub_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -48,10 +45,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start scrub worker: " >>$seqres.full
 	_btrfs_stress_scrub $SCRATCH_MNT >/dev/null 2>&1 &
@@ -64,9 +60,7 @@ run_test()
 	echo "$defrag_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
-
+	_wait_for_fsstress
 	_btrfs_kill_stress_defrag_pid $defrag_pid
 	unset defrag_pid
 	_btrfs_kill_stress_scrub_pid $scrub_pid
tests/btrfs/073 +4 -9
index bf7e9ca7a..b77e14c91 100755
@@ -20,10 +20,7 @@ _cleanup()
 	if [ ! -z "$scrub_pid" ]; then
 		_btrfs_kill_stress_scrub_pid $scrub_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -47,10 +44,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start scrub worker: " >>$seqres.full
 	_btrfs_stress_scrub $SCRATCH_MNT >/dev/null 2>&1 &
@@ -63,8 +59,7 @@ run_test()
 	echo "$remount_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
+	_wait_for_fsstress
 	_btrfs_kill_stress_remount_compress_pid $remount_pid $SCRATCH_MNT
 	unset remount_pid
 	_btrfs_kill_stress_scrub_pid $scrub_pid
tests/btrfs/074 +4 -9
index f78267159..a752707db 100755
@@ -20,10 +20,7 @@ _cleanup()
 	if [ ! -z "$defrag_pid" ]; then
 		_btrfs_kill_stress_defrag_pid $defrag_pid
 	fi
-	if [ ! -z "$fsstress_pid" ]; then
-		kill $fsstress_pid &> /dev/null
-		wait $fsstress_pid &> /dev/null
-	fi
+	_kill_fsstress
 }
 
 . ./common/filter
@@ -48,10 +45,9 @@ run_test()
 	fi
 	_scratch_mount >>$seqres.full 2>&1
 
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $SCRATCH_MNT/stressdir`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
+	_run_fsstress_bg $args
 
 	echo -n "Start defrag worker: " >>$seqres.full
 	_btrfs_stress_defrag $SCRATCH_MNT $with_compress >/dev/null 2>&1 &
@@ -64,8 +60,7 @@ run_test()
 	echo "$remount_pid" >>$seqres.full
 
 	echo "Wait for fsstress to exit and kill all background workers" >>$seqres.full
-	wait $fsstress_pid
-	unset fsstress_pid
+	_wait_for_fsstress
 	_btrfs_kill_stress_remount_compress_pid $remount_pid $SCRATCH_MNT
 	unset remount_pid
 	_btrfs_kill_stress_defrag_pid $defrag_pid
tests/btrfs/078 +2 -10
index bbebeff39..b802f9d86 100755
@@ -16,14 +16,6 @@
 . ./common/preamble
 _begin_fstest auto snapshot
 
-tmp=`mktemp -d`
-
-# Override the default cleanup function.
-_cleanup()
-{
-	rm -fr $tmp
-}
-
 . ./common/filter
 
 _require_scratch
@@ -39,8 +31,8 @@ workout()
 
 	snapshot_cmd="$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT"
 	snapshot_cmd="$snapshot_cmd $SCRATCH_MNT/snap_\`date +'%H_%M_%S_%N'\`"
-	run_check $FSSTRESS_PROG -p $procs \
-	    -x "$snapshot_cmd" -X $num_snapshots -d $SCRATCH_MNT -n $ops
+	_run_fsstress -p $procs -x "$snapshot_cmd" -X $num_snapshots \
+			-d $SCRATCH_MNT -n $ops
 }
 
 ops=8000
tests/btrfs/100 +2 -2
index 46bfc4f74..a319c7bbb 100755
@@ -13,6 +13,7 @@ _begin_fstest auto replace volume eio
 # Override the default cleanup function.
 _cleanup()
 {
+	_kill_fsstress
 	_dmerror_cleanup
 	rm -f $tmp.*
 }
@@ -39,8 +40,7 @@ error_devid=`$BTRFS_UTIL_PROG filesystem show -m $SCRATCH_MNT |\
 
 snapshot_cmd="$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT"
 snapshot_cmd="$snapshot_cmd $SCRATCH_MNT/snap_\`date +'%H_%M_%S_%N'\`"
-run_check $FSSTRESS_PROG -d $SCRATCH_MNT -n 200 -p 8 $FSSTRESS_AVOID -x \
-							"$snapshot_cmd" -X 50
+_run_fsstress -d $SCRATCH_MNT -n 200 -p 8 -x "$snapshot_cmd" -X 50
 
 # now load the error into the DMERROR_DEV
 _dmerror_load_error_table
tests/btrfs/101 +2 -2
index c65e14ea0..cb14d6a07 100755
@@ -13,6 +13,7 @@ _begin_fstest auto replace volume eio
 # Override the default cleanup function.
 _cleanup()
 {
+	_kill_fsstress
 	_dmerror_cleanup
 	rm -f $tmp.*
 }
@@ -40,8 +41,7 @@ error_devid=`$BTRFS_UTIL_PROG filesystem show -m $SCRATCH_MNT |\
 
 snapshot_cmd="$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT"
 snapshot_cmd="$snapshot_cmd $SCRATCH_MNT/snap_\`date +'%H_%M_%S_%N'\`"
-run_check $FSSTRESS_PROG -d $SCRATCH_MNT -n 200 -p 8 $FSSTRESS_AVOID -x \
-							"$snapshot_cmd" -X 50
+_run_fsstress -d $SCRATCH_MNT -n 200 -p 8 -x "$snapshot_cmd" -X 50
 
 # now load the error into the DMERROR_DEV
 _dmerror_load_error_table
tests/btrfs/136 +2 -4
index 2a5280fb9..65bbcf516 100755
@@ -34,11 +34,9 @@ BTRFS_MD5SUM="$tmp.btrfs"
 populate_data(){
 	data_path=$1
 	mkdir -p $data_path
-	args=`_scale_fsstress_args -p 20 -n 100 $FSSTRESS_AVOID -d $data_path`
+	args=`_scale_fsstress_args -p 20 -n 100 -d $data_path`
 	echo "Run fsstress $args" >>$seqres.full
-	$FSSTRESS_PROG $args >>$seqres.full &
-	fsstress_pid=$!
-	wait $fsstress_pid
+	_run_fsstress $args
 }
 
 # Create & populate an ext3 filesystem
tests/btrfs/192 +5 -6
index f7fb65b8d..cc8e1e003 100755
@@ -13,12 +13,12 @@ _begin_fstest auto replay snapshot stress recoveryloop
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
+	_kill_fsstress
 	kill -q $pid1 &> /dev/null
 	kill -q $pid2 &> /dev/null
-	"$KILLALL_PROG" -q $FSSTRESS_PROG &> /dev/null
 	wait
 	_log_writes_cleanup &> /dev/null
+	cd /
 	rm -f $tmp.*
 }
 
@@ -46,8 +46,7 @@ nr_cpus=$("$here/src/feature" -o)
 if [ $nr_cpus -gt 8 ]; then
 	nr_cpus=8
 fi
-fsstress_args=$(_scale_fsstress_args -w -d $SCRATCH_MNT -n 99999 -p $nr_cpus \
-		$FSSTRESS_AVOID)
+fsstress_args=$(_scale_fsstress_args -w -d $SCRATCH_MNT -n 99999 -p $nr_cpus )
 _log_writes_init $SCRATCH_DEV
 
 # Discard the whole devices so when some tree pointer is wrong, it won't point
@@ -135,10 +134,10 @@ pid1=$!
 delete_workload &
 pid2=$!
 
-"$FSSTRESS_PROG" $fsstress_args >> $seqres.full &
+_run_fsstress_bg $fsstress_args
 sleep $runtime
 
-"$KILLALL_PROG" -q "$FSSTRESS_PROG" &> /dev/null
+_kill_fsstress
 kill $pid1 &> /dev/null
 kill $pid2 &> /dev/null
 wait
tests/btrfs/195 +1 -1
index 72fc3a775..4dffddc1a 100755
@@ -49,7 +49,7 @@ run_testcase() {
 	_scratch_mount
 
 	# Create random filesystem with 20k write ops
-	$FSSTRESS_PROG -d $SCRATCH_MNT -w -n 10000 $FSSTRESS_AVOID >>$seqres.full 2>&1
+	_run_fsstress -d $SCRATCH_MNT -w -n 10000
 
 	_run_btrfs_balance_start -f -dconvert=$dst_type $SCRATCH_MNT >> $seqres.full
 	[ $? -eq 0 ] || echo "$1: Failed convert"
tests/btrfs/212 +7 -7
index f356d7d0f..745b9598a 100755
@@ -13,13 +13,13 @@ _begin_fstest auto balance dangerous
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
+	_kill_fsstress
 	kill $balance_pid &> /dev/null
 	kill $cancel_pid &> /dev/null
-	"$KILLALL_PROG" -q $FSSTRESS_PROG &> /dev/null
-	$BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
 	wait
+	$BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
+	cd /
+	rm -f $tmp.*
 }
 
 . ./common/filter
@@ -49,7 +49,7 @@ cancel_workload()
 	done
 }
 
-$FSSTRESS_PROG -d $SCRATCH_MNT -w -n 100000  >> $seqres.full 2>/dev/null &
+_run_fsstress_bg -d $SCRATCH_MNT -w -n 100000
 balance_workload &
 balance_pid=$!
 
@@ -58,11 +58,11 @@ cancel_pid=$!
 
 sleep $runtime
 
+_kill_fsstress
 kill $balance_pid
 kill $cancel_pid
-"$KILLALL_PROG" -q $FSSTRESS_PROG &> /dev/null
-$BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
 wait
+$BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
 
 echo "Silence is golden"
 # success, all done
tests/btrfs/232 +2 -2
index 4dcc39cc4..f843edbf7 100755
@@ -23,8 +23,8 @@ writer()
 	trap "wait; exit" SIGTERM
 
 	while true; do
-		args=`_scale_fsstress_args -p 20 -n 1000 $FSSTRESS_AVOID -d $SCRATCH_MNT/stressdir`
-		$FSSTRESS_PROG $args >> $seqres.full
+		args=`_scale_fsstress_args -p 20 -n 1000 -d $SCRATCH_MNT/stressdir`
+		_run_fsstress $args
 	done
 }
 
tests/btrfs/252 +3 -2
index 2da02ffa1..bb111b909 100755
@@ -24,6 +24,7 @@ _begin_fstest auto send balance stress
 
 _cleanup()
 {
+	_kill_fsstress
 	if [ ! -z $balance_pid ]; then
 		kill $balance_pid &> /dev/null
 		wait $balance_pid
@@ -79,10 +80,10 @@ snapshot_cmd="$snapshot_cmd \"$snapshots_dir/snap_\`date +'%s%N'\`\""
 # it's pointless to have them.
 #
 echo "Running fsstress..." >> $seqres.full
-$FSSTRESS_PROG $FSSTRESS_AVOID -d "$data_subvol" -p 1 -w \
+_run_fsstress -d "$data_subvol" -p 1 -w \
 	       -f subvol_create=0 -f subvol_delete=0 -f snapshot=0 \
 	       -x "$snapshot_cmd" -X $num_snapshots \
-	       -n $total_fsstress_ops >> $seqres.full
+	       -n $total_fsstress_ops
 
 snapshots=(`IFS=$'\n' ls -1 "$snapshots_dir"`)
 
tests/btrfs/261 +1 -1
index 4052baaec..7a08b5667 100755
@@ -35,7 +35,7 @@ prepare_fs()
 	# Then use fsstress to generate some extra contents.
 	# Disable setattr related operations, as it may set NODATACOW which will
 	# not allow us to use btrfs checksum to verify the content.
-	$FSSTRESS_PROG -f setattr=0 -d $SCRATCH_MNT -w -n 3000 >> $seqres.full
+	_run_fsstress -f setattr=0 -d $SCRATCH_MNT -w -n 3000
 	sync
 
 	# Save the fssum of this fs
tests/btrfs/284 +2 -2
index 6c554f325..ec3bc2720 100755
@@ -49,8 +49,8 @@ run_send_test()
 
 	# Use a single process so that in case of failure it's easier to
 	# reproduce by using the same seed (logged in $seqres.full).
-	run_check $FSSTRESS_PROG -d $SCRATCH_MNT -p 1 -n $((LOAD_FACTOR * 200)) \
-		  -w $FSSTRESS_AVOID -x "$snapshot_cmd"
+	_run_fsstress -d $SCRATCH_MNT -p 1 -n $((LOAD_FACTOR * 200)) \
+		  -w -x "$snapshot_cmd"
 
 	$BTRFS_UTIL_PROG subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/snap2 \
 			 >> $seqres.full
tests/btrfs/286 +1 -1
index b8fa27673..7909fa507 100755
@@ -31,7 +31,7 @@ workload()
 	# Use nodatasum mount option, so all data won't have checksum.
 	_scratch_mount -o nodatasum
 
-	$FSSTRESS_PROG -p 10 -n 200 -d $SCRATCH_MNT >> $seqres.full
+	_run_fsstress -p 10 -n 200 -d $SCRATCH_MNT
 	sync
 
 	# Generate fssum for later verification, here we only care
tests/btrfs/320 +2 -4
index 15549165e..bcfb96d1a 100755
@@ -28,8 +28,7 @@ _basic_test()
 	$BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | grep $subvolid >> \
 		$seqres.full 2>&1
 	[ $? -eq 0 ] || _fail "couldn't find our subvols quota group"
-	run_check $FSSTRESS_PROG -d $SCRATCH_MNT/a -w -p 1 -n 2000 \
-		$FSSTRESS_AVOID
+	_run_fsstress -d $SCRATCH_MNT/a -w -p 1 -n 2000
 	_btrfs subvolume snapshot $SCRATCH_MNT/a \
 		$SCRATCH_MNT/b
 
@@ -55,8 +54,7 @@ _rescan_test()
 	_btrfs subvolume create $SCRATCH_MNT/a
 	_btrfs quota enable $SCRATCH_MNT/a
 	subvolid=$(_btrfs_get_subvolid $SCRATCH_MNT a)
-	run_check $FSSTRESS_PROG -d $SCRATCH_MNT/a -w -p 1 -n 2000 \
-		$FSSTRESS_AVOID
+	_run_fsstress -d $SCRATCH_MNT/a -w -p 1 -n 2000
 	sync
 	output=$($BTRFS_UTIL_PROG qgroup show $units $SCRATCH_MNT | grep "0/$subvolid")
 	echo "qgroup values before rescan: $output" >> $seqres.full
tests/btrfs/332 +2 -2
index ab9860668..4fcdad4cd 100755
@@ -32,7 +32,7 @@ d1=$SCRATCH_MNT/d1
 d2=$SCRATCH_MNT/d2
 mkdir $d1
 mkdir $d2
-run_check $FSSTRESS_PROG -d $d1 -w -n 2000 $FSSTRESS_AVOID
+_run_fsstress -d $d1 -w -n 2000
 fssum_pre=$($FSSUM_PROG -A $SCRATCH_MNT)
 
 # enable squotas
@@ -45,7 +45,7 @@ fssum_post=$($FSSUM_PROG -A $SCRATCH_MNT)
 	|| echo "fssum $fssum_pre does not match $fssum_post after enabling squota"
 
 # do some more stuff
-run_check $FSSTRESS_PROG -d $d2 -w -n 2000 $FSSTRESS_AVOID
+_run_fsstress -d $d2 -w -n 2000
 fssum_pre=$($FSSUM_PROG -A $SCRATCH_MNT)
 _scratch_unmount
 _check_btrfs_filesystem $SCRATCH_DEV
tests/ext4/004 +2 -2
index 20cfdb444..ab2f838e9 100755
@@ -12,6 +12,7 @@ _begin_fstest auto dump
 # Override the default cleanup function.
 _cleanup()
 {
+	_kill_fsstress
 	cd /
 	rm -f $tmp.*
 	# remove the generated data, which is much and meaningless.
@@ -29,8 +30,7 @@ workout()
 	echo "Run fsstress" >> $seqres.full
 	args=`_scale_fsstress_args -z -f creat=5 -f write=20 -f mkdir=5 -n 100 -p 15 -d $dump_dir`
 	echo "fsstress $args" >> $seqres.full
-
-	$FSSTRESS_PROG $args >> $seqres.full 2>&1
+	_run_fsstress $args
 
 	echo "start Dump/Restore" >> $seqres.full
 	cd $TEST_DIR
tests/ext4/057 +3 -11
index 529f0c298..73cdf941a 100755
@@ -8,15 +8,6 @@
 . ./common/preamble
 _begin_fstest auto ioctl
 
-# Override the default cleanup function.
-_cleanup()
-{
-        cd /
-        rm -r -f $tmp.*
-        kill -9 $fsstress_pid 2>/dev/null;
-        wait > /dev/null 2>&1
-}
-
 # Import common functions.
 . ./common/filter
 
@@ -41,8 +32,7 @@ _scratch_mount
 
 # Begin fsstress while modifying UUID
 fsstress_args=$(_scale_fsstress_args -d $SCRATCH_MNT -p 15 -n 999999)
-$FSSTRESS_PROG $fsstress_args >> $seqres.full &
-fsstress_pid=$!
+_run_fsstress_bg $fsstress_args
 
 for n in $(seq 1 20); do
         new_uuid=$($UUIDGEN_PROG)
@@ -57,6 +47,8 @@ for n in $(seq 1 20); do
         fi
 done
 
+_kill_fsstress
+
 # success, all done
 echo "Silence is golden"
 status=0
tests/ext4/058 +1 -2
index a7fc5e6c6..f85364964 100755
@@ -13,7 +13,6 @@
 . ./common/preamble
 _begin_fstest auto quick
 
-
 _supported_fs ext4
 _fixed_by_kernel_commit a08f789d2ab5 \
 	"ext4: fix bug_on ext4_mb_use_inode_pa"
@@ -23,7 +22,7 @@ _require_scratch
 _scratch_mkfs -g 256 >> $seqres.full 2>&1 || _fail "mkfs failed"
 _scratch_mount
 
-$FSSTRESS_PROG -d $SCRATCH_MNT/stress -n 1000 >> $seqres.full 2>&1
+_run_fsstress -d $SCRATCH_MNT/stress -n 1000
 
 echo "Silence is golden"
 
tests/ext4/307 +2 -2
index f7c95c51b..8361f0431 100755
@@ -19,9 +19,9 @@ _workout()
 	echo ""
 	echo "Run fsstress"
 	out=$SCRATCH_MNT/fsstress.$$
-	args=`_scale_fsstress_args -p4 -n999 -f setattr=1 $FSSTRESS_AVOID -d $out`
+	args=`_scale_fsstress_args -p4 -n999 -f setattr=1 -d $out`
 	echo "fsstress $args" >> $seqres.full
-	$FSSTRESS_PROG $args >> $seqres.full
+	_run_fsstress $args
 	find $out -type f > $out.list
 	cat $out.list | xargs  md5sum > $out.md5sum
 	usage=`du -sch $out | tail -n1 | gawk '{ print $1 }'`
tests/generic/013 +5 -17
index 3a000b97f..0b86d8b33 100755
@@ -9,18 +9,6 @@
 . ./common/preamble
 _begin_fstest other ioctl udf auto quick
 
-status=0	# success is the default!
-
-# Override the default cleanup function.
-_cleanup()
-{
-    cd /
-    # we might get here with a RO FS
-    _test_cycle_mount
-    # now remove fsstress directory.
-    rm -rf $TEST_DIR/fsstress.$$.*
-}
-
 # Import common functions.
 . ./common/filter
 
@@ -30,7 +18,7 @@ _do_test()
     _param="$2"
     _count="$3"
 
-    out=$TEST_DIR/fsstress.$$.$_n
+    out=$TEST_DIR/fsstress.$seq.$_n
     rm -rf $out
     if ! mkdir $out
     then
@@ -44,9 +32,8 @@ _do_test()
     echo "fsstress.$_n : $_param"
     echo "-----------------------------------------------"
     # -m limits number of users/groups so check doesn't fail (malloc) later
-    dbgoutfile=$seqres.full
-    if ! $FSSTRESS_PROG $_param $FSSTRESS_AVOID -v -m 8 -n $_count -d $out >>$dbgoutfile 2>&1
-    then
+    _run_fsstress $_param -v -m 8 -n $_count -d $out
+    if [ $? -ne 0 ]; then
         echo "    fsstress (count=$_count) returned $? - see $seqres.full"
         echo "--------------------------------------"       >>$seqres.full
         echo "$_n - output from fsstress:"                  >>$seqres.full
@@ -62,6 +49,7 @@ _do_test()
 _require_test
 
 echo "brevity is wit..."
+status=0
 
 count=1000
 procs=20
@@ -80,7 +68,7 @@ _do_test 2 "-p $procs -r" $count
 
 _do_test 3 "-p 4 -z -f rmdir=10 -f link=10 -f creat=10 -f mkdir=10 -f rename=30 -f stat=30 -f unlink=30 -f truncate=20" $count
 
-# if all ok by here then probably don't need $seqres.full
+rm -rf $TEST_DIR/fsstress.$seq.*
 
 exit
 
tests/generic/019 +6 -7
index fe117ac8a..bed916b53 100755
@@ -22,7 +22,9 @@ _require_fail_make_request
 # Override the default cleanup function.
 _cleanup()
 {
-	kill $fs_pid $fio_pid &> /dev/null
+	_kill_fsstress
+	kill $fio_pid &> /dev/null
+	wait
 	_disallow_fail_make_request
 	cd /
 	rm -r -f $tmp.*
@@ -85,13 +87,12 @@ FSSTRESS_AVOID="$FSSTRESS_AVOID -ffsync=0 -fsync=0 -ffdatasync=0 -f setattr=1"
 _workout()
 {
 	out=$SCRATCH_MNT/fsstress.$$
-	args=`_scale_fsstress_args -p 1 -n999999999 -f setattr=0 $FSSTRESS_AVOID -d $out`
+	args=`_scale_fsstress_args -p 1 -n999999999 -f setattr=0 -d $out`
 	echo ""
 	echo "Start fsstress.."
 	echo ""
 	echo "fsstress $args" >> $seqres.full
-	$FSSTRESS_PROG $args >> $seqres.full 2>&1 &
-	fs_pid=$!
+	_run_fsstress_bg $args
 	echo "Start fio.."
 	cat $fio_config >>  $seqres.full
 	$FIO_PROG $fio_config >> $seqres.full 2>&1 &
@@ -107,10 +108,8 @@ _workout()
 	    >> $seqres.full 2>&1 && \
 	    _fail "failed: still able to perform integrity fsync on $SCRATCH_MNT"
 
-	kill $fs_pid &> /dev/null
-	wait $fs_pid
+	_kill_fsstress
 	wait $fio_pid
-	unset fs_pid
 	unset fio_pid
 
 	# We expect that broken FS still can be umounted
tests/generic/051 +4 -16
index 65571fdd0..69250cde6 100755
@@ -14,18 +14,8 @@ _begin_fstest shutdown auto stress log metadata repair
 # Import common functions.
 . ./common/filter
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	_scratch_unmount 2>/dev/null
-	rm -f $tmp.*
-}
-
-
 _require_scratch
 _require_scratch_shutdown
-_require_command "$KILLALL_PROG" killall
 
 _scratch_mkfs > $seqres.full 2>&1
 _require_metadata_journaling $SCRATCH_DEV
@@ -37,25 +27,23 @@ PROCS=$((2 * LOAD_FACTOR))
 load_dir=$SCRATCH_MNT/test
 
 # let this run for a while
-$FSSTRESS_PROG $FSSTRESS_AVOID -n10000000 -p $PROCS -d $load_dir >> $seqres.full 2>&1 &
+_run_fsstress_bg -n 10000000 -p $PROCS -d $load_dir
 sleep $SLEEP_TIME
-$KILLALL_PROG -q $FSSTRESS_PROG
-wait
+_kill_fsstress
 sync
 _scratch_unmount
 
 # now mount again, run the load again, this time with a shutdown.
 _scratch_mount
 $XFS_FSR_PROG -v $load_dir >> $seqres.full 2>&1
-$FSSTRESS_PROG -n10000000 -p $PROCS -d $load_dir >> $seqres.full 2>&1 &
+_run_fsstress_bg -n10000000 -p $PROCS -d $load_dir
 sleep $SLEEP_TIME
 sync
 
 # now shutdown and unmount
 sleep 5
 _scratch_shutdown
-$KILLALL_PROG -q $FSSTRESS_PROG
-wait
+_kill_fsstress
 
 # for some reason fsstress processes manage to live on beyond the wait?
 sleep 5
tests/generic/055 +3 -3
index b1126c901..e57f95faf 100755
@@ -23,9 +23,9 @@ _do_meta()
     param="-p 4 -z -f rmdir=10 -f link=10 -f creat=10 -f mkdir=10 \
            -f rename=30 -f stat=30 -f unlink=30 -f truncate=20"
     _echofull "calling fsstress $param -m8 -n $count"
-    FSSTRESS_ARGS=`_scale_fsstress_args $param $FSSTRESS_AVOID -m 8 -n $count -d $out`
-    if ! $FSSTRESS_PROG $FSSTRESS_ARGS >>$seqres.full 2>&1
-    then
+    FSSTRESS_ARGS=`_scale_fsstress_args $param -m 8 -n $count -d $out`
+    _run_fsstress $FSSTRESS_ARGS
+    if [ $? -ne 0 ]; then
 	_echofull "fsstress failed"
     fi
 }
tests/generic/068 +3 -4
index 1e8248b9d..26c5ceea6 100755
@@ -19,8 +19,7 @@ _cleanup()
 {
 	# Make sure $SCRATCH_MNT is unfreezed
 	xfs_freeze -u $SCRATCH_MNT 2>/dev/null
-	[ -n "$pid" ] && kill -9 $pid 2>/dev/null
-	wait $pid
+	_kill_fsstress
 	cd /
 	rm -f $tmp.*
 }
@@ -54,8 +53,8 @@ touch $tmp.running
       do
       # We do both read & write IO - not only is this more realistic,
       # but it also potentially tests atime updates
-      FSSTRESS_ARGS=`_scale_fsstress_args -d $STRESS_DIR -p $procs -n $nops $FSSTRESS_AVOID`
-      $FSSTRESS_PROG $FSSTRESS_ARGS >>$seqres.full
+      FSSTRESS_ARGS=`_scale_fsstress_args -d $STRESS_DIR -p $procs -n $nops`
+      _run_fsstress $FSSTRESS_ARGS >>$seqres.full
     done
 
     rm -r $STRESS_DIR/*
tests/generic/070 +2 -10
index a8e84effd..000fc0482 100755
@@ -9,19 +9,10 @@
 . ./common/preamble
 _begin_fstest attr udf auto quick stress
 
-# Override the default cleanup function.
-_cleanup()
-{
-    cd /
-    rm -rf $TEST_DIR/fsstress
-    rm -f $tmp.*
-}
-
 # Import common functions.
 . ./common/filter
 . ./common/attr
 
-
 _require_test
 _require_attrs
 
@@ -34,7 +25,8 @@ FSSTRESS_ARGS=`_scale_fsstress_args \
 	-f attr_set=100 \
 	-f attr_remove=100 \
         -p 1 -n 10000 -S c`
-$FSSTRESS_PROG $FSSTRESS_ARGS >$seqres.full 2>&1
+_run_fsstress $FSSTRESS_ARGS
+rm -rf $TEST_DIR/fsstress
 
 status=$?
 exit
tests/generic/076 +4 -6
index b50c2df5e..100f19715 100755
@@ -21,16 +21,14 @@ _lets_get_pidst()
 # Override the default cleanup function.
 _cleanup()
 {
-	echo "*** unmount"
-	_scratch_unmount 2>/dev/null
 	_lets_get_pidst
+	cd /
+	rm -f $tmp.*
 }
-_register_cleanup "_cleanup; rm -f $tmp.*"
 
 # Import common functions.
 . ./common/filter
 
-
 _require_scratch
 _require_local_device $SCRATCH_DEV
 
@@ -48,9 +46,9 @@ echo "*** test concurrent block/fs access"
 cat $SCRATCH_DEV >/dev/null &
 pid=$!
 
-FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -p 2 -n 2000 $FSSTRESS_AVOID`
+FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -p 2 -n 2000`
 echo "run fsstress with args: $FSSTRESS_ARGS" >>$seqres.full
-$FSSTRESS_PROG $FSSTRESS_ARGS >>$seqres.full
+_run_fsstress $FSSTRESS_ARGS
 _lets_get_pidst
 
 echo "*** done"
tests/generic/076.out +0 -1
index 54a8f7096..74a65bdec 100644
@@ -2,4 +2,3 @@ QA output created by 076
 *** init fs
 *** test concurrent block/fs access
 *** done
-*** unmount
tests/generic/083 +2 -11
index ff4785eee..6aa0ea06d 100755
@@ -18,18 +18,9 @@
 . ./common/preamble
 _begin_fstest rw auto enospc stress
 
-# Override the default cleanup function.
-_cleanup()
-{
-	echo "*** unmount"
-	_scratch_unmount 2>/dev/null
-	rm -f $tmp.*
-}
-
 # Import common functions.
 . ./common/filter
 
-
 _require_scratch
 _require_no_large_scratch_dev
 
@@ -52,8 +43,8 @@ workout()
 	_scratch_mount
 
 	# -w ensures that the only ops are ones which cause write I/O
-	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs -n $nops $FSSTRESS_AVOID`
-	$FSSTRESS_PROG $FSSTRESS_ARGS >>$seqres.full
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs -n $nops`
+	_run_fsstress $FSSTRESS_ARGS >>$seqres.full
 }
 
 echo "*** test out-of-space handling for random write operations"
tests/generic/083.out +0 -1
index 025afe30e..b853eadb4 100644
@@ -1,4 +1,3 @@
 QA output created by 083
 *** test out-of-space handling for random write operations
 *** done
-*** unmount
tests/generic/117 +1 -3
index f9769be94..a34f86165 100755
@@ -63,9 +63,7 @@ echo Running fsstress in serial:
 i=0
 while [ $i -lt $ITERATIONS ]; do
     echo fsstress iteration: $i | tee -a $seqres.full
-    $FSSTRESS_PROG \
-	-d $SCRATCH_MNT/fsstress \
-	$fss_ops -S c >>$seqres.full 2>&1
+    _run_fsstress -d $SCRATCH_MNT/fsstress $fss_ops -S c
 
     let i=$i+1
 done
tests/generic/232 +2 -3
index 35934cc18..c903a5619 100755
@@ -30,13 +30,12 @@ _fsstress()
 
 	out=$SCRATCH_MNT/fsstress.$$
 	count=2000
-	args=`_scale_fsstress_args -d $out -n $count -p 7 $FSSTRESS_AVOID`
+	args=`_scale_fsstress_args -d $out -n $count -p 7`
 
 	echo "fsstress $args" >> $seqres.full
-	if ! $FSSTRESS_PROG $args | tee -a $seqres.full | _filter_num
+	if ! _run_fsstress $args
 	then
 		echo "    fsstress $args returned $?"
-		cat $tmp.out | tee -a $seqres.full
 		status=1
 	fi
 }
tests/generic/232.out +0 -1
index 5da53d4bf..9214607f0 100644
@@ -2,6 +2,5 @@ QA output created by 232
 
 Testing fsstress
 
-seed = S
 Comparing user usage
 Comparing group usage
tests/generic/269 +3 -5
index 341fcd22f..979cc15cb 100755
@@ -21,10 +21,9 @@ _workout()
 	num_iterations=10
 	enospc_time=2
 	out=$SCRATCH_MNT/fsstress.$$
-	args=`_scale_fsstress_args -p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out`
+	args=`_scale_fsstress_args -p128 -n999999999 -f setattr=1 -d $out`
 	echo "fsstress $args" >> $seqres.full
-	$FSSTRESS_PROG $args &>> $seqres.full &
-	pid=$!
+	_run_fsstress_bg $args
 	echo "Run dd writers in parallel"
 	for ((i=0; i < num_iterations; i++))
 	do
@@ -34,8 +33,7 @@ _workout()
 		sleep $enospc_time
 	done
 	echo "Killing fsstress process..." >> $seqres.full
-	kill $pid >> $seqres.full 2>&1
-	wait $pid
+	_kill_fsstress
 }
 
 _require_scratch
tests/generic/270 +5 -4
index cf523f4ea..aff379ac5 100755
@@ -28,8 +28,8 @@ _workout()
 	args=`_scale_fsstress_args -p128 -n999999999 -f setattr=1 $FSSTRESS_AVOID -d $out`
 	echo "fsstress $args" >> $seqres.full
 	# Grant chown capability 
-	cp $FSSTRESS_PROG  $tmp.fsstress.bin
-	$SETCAP_PROG cap_chown=epi  $tmp.fsstress.bin
+	cp $FSSTRESS_PROG $_FSSTRESS_PROG
+	$SETCAP_PROG cap_chown=epi $_FSSTRESS_PROG
 
 	# io_uring accounts memory it needs under the rlimit memlocked option,
 	# which can be quite low on some setups (especially 64K pagesize). root
@@ -37,7 +37,8 @@ _workout()
 	# io_uring_queue_init fail on ENOMEM, set max locked memory to unlimited
 	# temporarily.
 	ulimit -l unlimited
-	(su $qa_user -c "$tmp.fsstress.bin $args" &) > /dev/null 2>&1
+	su $qa_user -c "$_FSSTRESS_PROG $args" > /dev/null 2>&1 &
+	_FSSTRESS_PID=$!
 
 	echo "Run dd writers in parallel"
 	for ((i=0; i < num_iterations; i++))
@@ -49,7 +50,7 @@ _workout()
 		sleep $enospc_time
 	done
 
-	$KILLALL_PROG -w $tmp.fsstress.bin
+	_kill_fsstress
 }
 
 _require_quota
tests/generic/388 +2 -22
index 89ddda31d..34609a0d3 100755
@@ -17,23 +17,9 @@
 . ./common/preamble
 _begin_fstest shutdown auto log metadata recoveryloop
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-	_scratch_unmount > /dev/null 2>&1
-}
-
-# Import common functions.
-
-# Modify as appropriate.
-
 _require_scratch
 _require_local_device $SCRATCH_DEV
 _require_scratch_shutdown
-_require_command "$KILLALL_PROG" "killall"
 
 echo "Silence is golden."
 
@@ -42,20 +28,14 @@ _require_metadata_journaling $SCRATCH_DEV
 _scratch_mount
 
 while _soak_loop_running $((50 * TIME_FACTOR)); do
-	($FSSTRESS_PROG $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p 4 >> $seqres.full &) \
-		> /dev/null 2>&1
+	_run_fsstress_bg -d $SCRATCH_MNT -n 999999 -p 4
 
 	# purposely include 0 second sleeps to test shutdown immediately after
 	# recovery
 	sleep $((RANDOM % 3))
 	_scratch_shutdown
 
-	ps -e | grep fsstress > /dev/null 2>&1
-	while [ $? -eq 0 ]; do
-		$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-		wait > /dev/null 2>&1
-		ps -e | grep fsstress > /dev/null 2>&1
-	done
+	_kill_fsstress
 
 	# Toggle between rw and ro mounts for recovery. Quit if any mount
 	# attempt fails so we don't shutdown the host fs.
tests/generic/390 +4 -7
index 02f6c5eef..52270782e 100755
@@ -18,8 +18,7 @@ _cleanup()
 	[ -n "$freeze_pids" ] && kill -9 $freeze_pids 2>/dev/null
 	wait $freeze_pids
 	xfs_freeze -u $SCRATCH_MNT 2>/dev/null
-	[ -n "$fsstress_pid" ] && kill -9 $fsstress_pid 2>/dev/null
-	wait $fsstress_pid
+	_kill_fsstress
 	rm -f $tmp.*
 }
 
@@ -49,9 +48,8 @@ fi
 nops=1000
 stress_dir="$SCRATCH_MNT/fsstress_test_dir"
 mkdir "$stress_dir"
-fsstress_args=`_scale_fsstress_args -d $stress_dir -p $procs -n $nops $FSSTRESS_AVOID`
-$FSSTRESS_PROG $fsstress_args >>$seqres.full 2>&1 &
-fsstress_pid=$!
+fsstress_args=`_scale_fsstress_args -d $stress_dir -p $procs -n $nops`
+_run_fsstress_bg $fsstress_args
 
 # Start multi-threads freeze/unfreeze
 for ((i=0; i<$procs; i++)); do
@@ -62,9 +60,8 @@ for ((i=0; i<$procs; i++)); do
 	freeze_pids="$! $freeze_pids"
 done
 
-wait $fsstress_pid
+_wait_for_fsstress
 result=$?
-unset fsstress_pid
 wait $freeze_pids
 unset freeze_pids
 
tests/generic/409 +5 -4
index b7edc2ac6..ab7e7eb55 100755
@@ -25,11 +25,12 @@ _begin_fstest auto quick mount
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
+	_kill_fsstress
 	_clear_mount_stack
 	# make sure there's no bug cause dentry isn't be freed
 	rm -rf $MNTHEAD
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -43,7 +44,7 @@ fs_stress()
 {
 	local target=$1
 
-	$FSSTRESS_PROG -z -n 50 -p 3 \
+	_run_fsstress -z -n 50 -p 3 \
 		       -f creat=5 \
 		       -f mkdir=5 \
 		       -f link=2 \
@@ -56,7 +57,7 @@ fs_stress()
 		       -f chown=1 \
 		       -f getdents=1 \
 		       -f fiemap=1 \
-		       -d $target >>$seqres.full
+		       -d $target
 	sync
 }
 
tests/generic/410 +5 -4
index 902f27144..f0f0921be 100755
@@ -33,11 +33,12 @@ _begin_fstest auto quick mount
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
+	_kill_fsstress
 	_clear_mount_stack
 	# make sure there's no bug cause dentry isn't be freed
 	rm -rf $MNTHEAD
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -51,7 +52,7 @@ fs_stress()
 {
 	local target=$1
 
-	$FSSTRESS_PROG -z -n 50 -p 3 \
+	_run_fsstress -z -n 50 -p 3 \
 		       -f creat=5 \
 		       -f mkdir=5 \
 		       -f link=2 \
@@ -64,7 +65,7 @@ fs_stress()
 		       -f chown=1 \
 		       -f getdents=1 \
 		       -f fiemap=1 \
-		       -d $target >>$seqres.full
+		       -d $target
 	sync
 }
 
tests/generic/411 +5 -4
index c35436c82..3b55b4f91 100755
@@ -14,11 +14,12 @@ _begin_fstest auto quick mount
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
+	_kill_fsstress
 	_clear_mount_stack
 	# make sure there's no bug cause dentry isn't be freed
 	rm -rf $MNTHEAD
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -32,7 +33,7 @@ fs_stress()
 {
 	local target=$1
 
-	$FSSTRESS_PROG -z -n 500 -p 5 \
+	_run_fsstress -z -n 500 -p 5 \
 		       -f creat=5 \
 		       -f mkdir=5 \
 		       -f dwrite=1 \
@@ -47,7 +48,7 @@ fs_stress()
 		       -f chown=1 \
 		       -f getdents=1 \
 		       -f fiemap=1 \
-		       -d $target >>$seqres.full
+		       -d $target
 	sync
 }
 
tests/generic/461 +2 -13
index 468ce46f0..95dd6a34d 100755
@@ -14,18 +14,8 @@ _begin_fstest auto shutdown stress
 # Import common functions.
 . ./common/filter
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	_scratch_unmount 2>/dev/null
-	rm -f $tmp.*
-}
-
-
 _require_scratch_nocheck
 _require_scratch_shutdown
-_require_command "$KILLALL_PROG" killall
 
 _scratch_mkfs > $seqres.full 2>&1
 _scratch_mount
@@ -35,15 +25,14 @@ PROCS=$((4 * LOAD_FACTOR))
 
 load_dir=$SCRATCH_MNT/test
 
-$FSSTRESS_PROG $FSSTRESS_AVOID -n10000000 -p $PROCS -d $load_dir >> $seqres.full 2>&1 &
+_run_fsstress_bg -n10000000 -p $PROCS -d $load_dir
 sleep $SLEEP_TIME
 sync
 
 # now shutdown and unmount
 sleep 5
 _scratch_shutdown
-$KILLALL_PROG -q $FSSTRESS_PROG
-wait
+_kill_fsstress
 
 # for some reason fsstress processes manage to live on beyond the wait?
 sleep 5
tests/generic/475 +5 -12
index 4b854f9ab..9df2051fb 100755
@@ -17,11 +17,11 @@ _begin_fstest shutdown auto log metadata eio recoveryloop smoketest
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
+	_kill_fsstress
 	_dmerror_unmount
 	_dmerror_cleanup
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -31,7 +31,6 @@ _cleanup()
 
 _require_scratch
 _require_dm_target error
-_require_command "$KILLALL_PROG" "killall"
 
 echo "Silence is golden."
 
@@ -41,8 +40,7 @@ _dmerror_init
 _dmerror_mount
 
 while _soak_loop_running $((50 * TIME_FACTOR)); do
-	($FSSTRESS_PROG $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p $((LOAD_FACTOR * 4)) >> $seqres.full &) \
-		> /dev/null 2>&1
+	_run_fsstress_bg -d $SCRATCH_MNT -n 999999 -p $((LOAD_FACTOR * 4))
 
 	# purposely include 0 second sleeps to test shutdown immediately after
 	# recovery
@@ -55,12 +53,7 @@ while _soak_loop_running $((50 * TIME_FACTOR)); do
 	# error table helper *without* 'lockfs'.
 	_dmerror_load_error_table
 
-	ps -e | grep fsstress > /dev/null 2>&1
-	while [ $? -eq 0 ]; do
-		$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-		wait > /dev/null 2>&1
-		ps -e | grep fsstress > /dev/null 2>&1
-	done
+	_kill_fsstress
 
 	# Mount again to replay log after loading working table, so we have a
 	# consistent XFS after test.
tests/generic/476 +1 -14
index cf7402a12..769b3745f 100755
@@ -10,20 +10,7 @@
 . ./common/preamble
 _begin_fstest auto rw long_rw stress soak smoketest
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-}
-
-# Import common functions.
-
-# Modify as appropriate.
-
 _require_scratch
-_require_command "$KILLALL_PROG" "killall"
 
 echo "Silence is golden."
 
@@ -35,7 +22,7 @@ nr_ops=$((25000 * nr_cpus * TIME_FACTOR))
 fsstress_args=(-w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus)
 test -n "$SOAK_DURATION" && fsstress_args+=(--duration="$SOAK_DURATION")
 
-$FSSTRESS_PROG $FSSTRESS_AVOID "${fsstress_args[@]}" >> $seqres.full
+_run_fsstress_bg "${fsstress_args[@]}"
 
 # success, all done
 status=0
tests/generic/482 +5 -7
index 54fee07d4..8c114ee03 100755
@@ -18,10 +18,10 @@ _begin_fstest auto metadata replay thin recoveryloop
 # failure leaves the corpse intact for post-mortem failure analysis.
 _cleanup()
 {
-	cd /
-	$KILLALL_PROG -KILL -q $FSSTRESS_PROG &> /dev/null
+	_kill_fsstress
 	_log_writes_cleanup &> /dev/null
 	_dmthin_cleanup
+	cd /
 	rm -f $tmp.*
 }
 
@@ -29,7 +29,7 @@ _cleanup()
 #_cleanup()
 #{
 #	cd /
-#	$KILLALL_PROG -KILL -q $FSSTRESS_PROG &> /dev/null
+#	[ -n "$fsstress_pid" ] && kill $fsstress_pid
 #	if [ $status -eq 0 ]; then
 #		_log_writes_cleanup &> /dev/null
 #		_dmthin_cleanup
@@ -48,7 +48,6 @@ _cleanup()
 # Modify as appropriate.
 
 _require_no_logdev
-_require_command "$KILLALL_PROG" killall
 # Use thin device as replay device, which requires $SCRATCH_DEV
 _require_scratch_nocheck
 # and we need extra device as log device
@@ -60,8 +59,7 @@ nr_cpus=$("$here/src/feature" -o)
 if [ $nr_cpus -gt 8 ]; then
 	nr_cpus=8
 fi
-fsstress_args=$(_scale_fsstress_args -w -d $SCRATCH_MNT -n 512 -p $nr_cpus \
-		$FSSTRESS_AVOID)
+fsstress_args=$(_scale_fsstress_args -w -d $SCRATCH_MNT -n 512 -p $nr_cpus)
 
 size=$(_small_fs_size_mb 200)           # 200m phys/virt size
 devsize=$((1024*1024*size / 512))
@@ -77,7 +75,7 @@ _log_writes_mkfs >> $seqres.full 2>&1
 _log_writes_mark mkfs
 
 _log_writes_mount
-run_check $FSSTRESS_PROG $fsstress_args
+_run_fsstress $fsstress_args
 _log_writes_unmount
 
 _log_writes_remove
tests/generic/547 +3 -2
index 1e3881db9..14d02b4fd 100755
@@ -14,6 +14,7 @@ _begin_fstest auto quick log
 # Override the default cleanup function.
 _cleanup()
 {
+	_kill_fsstress
 	_cleanup_flakey
 	cd /
 	rm -f $tmp.*
@@ -38,10 +39,10 @@ _init_flakey
 _mount_flakey
 
 mkdir $SCRATCH_MNT/test
-args=`_scale_fsstress_args -p 4 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/test`
+args=`_scale_fsstress_args -p 4 -n 100 -d $SCRATCH_MNT/test`
 args="$args -f mknod=0 -f symlink=0"
 echo "Running fsstress with arguments: $args" >>$seqres.full
-$FSSTRESS_PROG $args >>$seqres.full
+_run_fsstress $args
 
 # Fsync every file and directory.
 find $SCRATCH_MNT/test \( -type f -o -type d \) -exec $XFS_IO_PROG -c fsync {} \;
tests/generic/560 +3 -4
index 62983d69b..067d3ec00 100755
@@ -30,8 +30,8 @@ function iterate_dedup_verify()
 		find $dest -type f -exec md5sum {} \; \
 			> $md5file$index
 		# Make some noise
-		$FSSTRESS_PROG $fsstress_opts -d $noisedir \
-			       -n 200 -p $((5 * LOAD_FACTOR)) >/dev/null 2>&1
+		_run_fsstress $fsstress_opts -d $noisedir \
+			       -n 200 -p $((5 * LOAD_FACTOR))
 		# Too many output, so only save error output
 		$DUPEREMOVE_PROG -dr --dedupe-options=same $dupdir \
 			>/dev/null 2>$seqres.full
@@ -51,8 +51,7 @@ md5file=${tmp}.md5sum
 
 fsstress_opts="-w -r"
 # Create some files to be original data
-$FSSTRESS_PROG $fsstress_opts -d $srcdir \
-	       -n 500 -p $((5 * LOAD_FACTOR)) >/dev/null 2>&1
+_run_fsstress $fsstress_opts -d $srcdir -n 500 -p $((5 * LOAD_FACTOR))
 
 # Calculate how many test cycles will be run
 src_size=`du -ks $srcdir | awk '{print $1}'`
tests/generic/561 +5 -12
index 39e5977a3..3e931b1a7 100755
@@ -13,9 +13,9 @@ _begin_fstest auto stress dedupe
 # Override the default cleanup function.
 _cleanup()
 {
+	end_test
 	cd /
 	rm -f $tmp.*
-	end_test
 }
 
 # Import common functions.
@@ -23,28 +23,20 @@ _cleanup()
 . ./common/reflink
 
 _require_scratch_duperemove
-_require_command "$KILLALL_PROG" killall
 
 _scratch_mkfs > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 
 function end_test()
 {
-	local f=1
+	_kill_fsstress
 
 	# stop duperemove running
 	if [ -e $dupe_run ]; then
 		rm -f $dupe_run
-		$KILLALL_PROG -q $DUPEREMOVE_PROG > /dev/null 2>&1
+		kill $dedup_pids
 		wait $dedup_pids
 	fi
-
-	# Make sure all fsstress get killed
-	while [ $f -ne 0 ]; do
-		$KILLALL_PROG -q $FSSTRESS_PROG > /dev/null 2>&1
-		sleep 1
-		f=`ps -eLf | grep $FSSTRESS_PROG | grep -v "grep" | wc -l`
-	done
 }
 
 sleep_time=$((50 * TIME_FACTOR))
@@ -53,7 +45,8 @@ sleep_time=$((50 * TIME_FACTOR))
 testdir="$SCRATCH_MNT/dir"
 mkdir $testdir
 fsstress_opts="-r -n 1000 -p $((5 * LOAD_FACTOR))"
-$FSSTRESS_PROG $fsstress_opts -d $testdir -l 0 >> $seqres.full 2>&1 &
+_run_fsstress_bg $fsstress_opts -d $testdir -l 0
+
 dedup_pids=""
 dupe_run=$TEST_DIR/${seq}-running
 # Start several dedupe processes on same directory
tests/generic/579 +2 -4
index 3191342b3..4187ab0f0 100755
@@ -16,7 +16,7 @@ _begin_fstest auto stress verity
 _cleanup()
 {
 	# Stop all subprocesses.
-	$KILLALL_PROG -q $FSSTRESS_PROG
+	_kill_fsstress
 	touch $tmp.done
 	wait
 
@@ -29,7 +29,6 @@ _cleanup()
 . ./common/verity
 
 _require_scratch_verity
-_require_command "$KILLALL_PROG" killall
 _disable_fsverity_signatures
 
 _scratch_mkfs_verity &>> $seqres.full
@@ -92,8 +91,7 @@ done
 ) &
 
 # Start the fsstress processes.
-$FSSTRESS_PROG $FSSTRESS_AVOID -p $nproc_stress -l 0 -d $SCRATCH_MNT/stressdir \
-	>> $seqres.full 2>&1 &
+_run_fsstress_bg -p $nproc_stress -l 0 -d $SCRATCH_MNT/stressdir
 
 # Run for a while.
 sleep $runtime
tests/generic/585 +2 -2
index fb675c8d5..b95cd9abf 100755
@@ -23,10 +23,10 @@ _scratch_mount >> $seqres.full 2>&1
 # start a create and rename(rename_whiteout) workload. These processes
 # occur simultaneously may cause the deadlock between AGI and AGF with
 # RENAME_WHITEOUT.
-$FSSTRESS_PROG -z -n 150 -p 100 \
+_run_fsstress -z -n 150 -p 100 \
 		-f mknod=5 \
 		-f rwhiteout=5 \
-		-d $SCRATCH_MNT/fsstress >> $seqres.full 2>&1
+		-d $SCRATCH_MNT/fsstress
 
 echo Silence is golden
 
tests/generic/589 +4 -3
index 0ce16556a..969a8ac61 100755
@@ -28,11 +28,12 @@ _begin_fstest auto mount
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
+	_kill_fsstress
 	_clear_mount_stack
 	# make sure there's no bug cause dentry isn't be freed
 	rm -rf $MNTHEAD
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -46,7 +47,7 @@ fs_stress()
 {
 	local target=$1
 
-	$FSSTRESS_PROG -n 50 -p 3 -d $target >>$seqres.full
+	_run_fsstress -n 50 -p 3 -d $target
 	sync
 }
 
tests/generic/642 +1 -9
index a7112a08f..4b92a9c18 100755
@@ -10,17 +10,9 @@
 . ./common/preamble
 _begin_fstest auto soak attr long_rw stress smoketest
 
-_cleanup()
-{
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-	cd /
-	rm -f $tmp.*
-}
-
 # Modify as appropriate.
 
 _require_scratch
-_require_command "$KILLALL_PROG" "killall"
 
 echo "Silence is golden."
 
@@ -50,7 +42,7 @@ args+=('-f' "setfattr=20")
 args+=('-f' "attr_set=60")	# sets larger xattrs
 test -n "$SOAK_DURATION" && args+=(--duration="$SOAK_DURATION")
 
-$FSSTRESS_PROG "${args[@]}" $FSSTRESS_AVOID -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full
+_run_fsstress "${args[@]}" -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus
 
 # success, all done
 status=0
tests/generic/648 +5 -14
index 29d1b470b..e4c9990e1 100755
@@ -16,16 +16,15 @@ _begin_fstest shutdown auto log metadata eio recoveryloop
 
 _cleanup()
 {
-	cd /
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-	wait
+	_kill_fsstress
 	if [ -n "$loopmnt" ]; then
 		$UMOUNT_PROG $loopmnt 2>/dev/null
 		rm -r -f $loopmnt
 	fi
-	rm -f $tmp.*
 	_dmerror_unmount
 	_dmerror_cleanup
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -37,7 +36,6 @@ _cleanup()
 _require_scratch_reflink
 _require_cp_reflink
 _require_dm_target error
-_require_command "$KILLALL_PROG" "killall"
 _require_loop
 
 echo "Silence is golden."
@@ -71,8 +69,6 @@ snap_loop_fs() {
 	rm -f "$snap_aliveflag"
 }
 
-fsstress=($FSSTRESS_PROG $FSSTRESS_AVOID -d "$loopmnt" -n 999999 -p "$((LOAD_FACTOR * 4))")
-
 while _soak_loop_running $((25 * TIME_FACTOR)); do
 	touch $scratch_aliveflag
 	snap_loop_fs >> $seqres.full 2>&1 &
@@ -84,7 +80,7 @@ while _soak_loop_running $((25 * TIME_FACTOR)); do
 		break
 	fi
 
-	("${fsstress[@]}" >> $seqres.full &) > /dev/null 2>&1
+	_run_fsstress_bg -d "$loopmnt" -n 999999 -p "$((LOAD_FACTOR * 4))"
 
 	# purposely include 0 second sleeps to test shutdown immediately after
 	# recovery
@@ -98,12 +94,7 @@ while _soak_loop_running $((25 * TIME_FACTOR)); do
 	# error table helper *without* 'lockfs'.
 	_dmerror_load_error_table
 
-	ps -e | grep fsstress > /dev/null 2>&1
-	while [ $? -eq 0 ]; do
-		$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-		wait > /dev/null 2>&1
-		ps -e | grep fsstress > /dev/null 2>&1
-	done
+	_kill_fsstress
 	for ((j = 0; j < 10; j++)); do
 		test -e "$snap_aliveflag" || break
 		sleep 1
tests/generic/650 +9 -6
index 5d2cb8977..36a23e48d 100755
@@ -16,14 +16,14 @@ _begin_fstest auto rw stress soak
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
+	_kill_fsstress
 	wait	# for exercise_cpu_hotplug subprocess
 	for i in "$sysfs_cpu_dir/"cpu*/online; do
 		echo 1 > "$i" 2>/dev/null
 	done
 	test -n "$stress_dir" && rm -r -f "$stress_dir"
+	cd /
+	rm -f $tmp.*
 }
 
 exercise_cpu_hotplug()
@@ -39,7 +39,6 @@ exercise_cpu_hotplug()
 }
 
 _require_test
-_require_command "$KILLALL_PROG" "killall"
 
 sysfs_cpu_dir="/sys/devices/system/cpu"
 
@@ -71,13 +70,17 @@ test "$nr_cpus" -gt 1024 && nr_cpus="$nr_hotplug_cpus"
 fsstress_args+=(-p $nr_cpus)
 if [ -n "$SOAK_DURATION" ]; then
 	test "$SOAK_DURATION" -lt 10 && SOAK_DURATION=10
-	fsstress_args+=(--duration="$((SOAK_DURATION / 10))")
+else
+	# run for 30s per iteration max
+	SOAK_DURATION=300
 fi
+fsstress_args+=(--duration="$((SOAK_DURATION / 10))")
 
 nr_ops=$((2500 * TIME_FACTOR))
 fsstress_args+=(-n $nr_ops)
 for ((i = 0; i < 10; i++)); do
-	$FSSTRESS_PROG $FSSTRESS_AVOID -w "${fsstress_args[@]}" >> $seqres.full
+	_run_fsstress_bg -w "${fsstress_args[@]}"
+	_wait_for_fsstress
 	_test_cycle_mount
 done
 
tests/generic/750 +4 -6
index dba8021d6..5c54a5c78 100755
@@ -11,13 +11,12 @@ _begin_fstest auto rw long_rw stress soak smoketest
 
 _cleanup()
 {
-	cd /
+	_kill_fsstress
 	rm -f $runfile
-	rm -f $tmp.*
 	kill -9 $trigger_compaction_pid > /dev/null 2>&1
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-
 	wait > /dev/null 2>&1
+	rm -f $tmp.*
+	cd /
 }
 
 # Import common functions.
@@ -26,7 +25,6 @@ _cleanup()
 
 _require_scratch
 _require_vm_compaction
-_require_command "$KILLALL_PROG" "killall"
 
 # We still deadlock with this test on v6.10-rc2, we need more work.
 # but the below makes things better.
@@ -52,7 +50,7 @@ while [ -e $runfile ]; do
 done &
 trigger_compaction_pid=$!
 
-$FSSTRESS_PROG $FSSTRESS_AVOID "${fsstress_args[@]}" >> $seqres.full
+_run_fsstress "${fsstress_args[@]}"
 
 rm -f $runfile
 wait > /dev/null 2>&1
tests/generic/753 +5 -12
index e427d62d1..f5665320a 100755
@@ -16,11 +16,11 @@ _begin_fstest shutdown auto log metadata eio recoveryloop attr
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
+	_kill_fsstress
 	_dmerror_unmount
 	_dmerror_cleanup
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -30,7 +30,6 @@ _cleanup()
 
 _require_scratch
 _require_dm_target error
-_require_command "$KILLALL_PROG" "killall"
 
 echo "Silence is golden."
 
@@ -59,8 +58,7 @@ args+=('-f' "setfattr=20")
 args+=('-f' "attr_set=60")	# sets larger xattrs
 
 while _soak_loop_running $((50 * TIME_FACTOR)); do
-	($FSSTRESS_PROG "${args[@]}" $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p $((LOAD_FACTOR * 4)) >> $seqres.full &) \
-		> /dev/null 2>&1
+	_run_fsstress_bg "${args[@]}" -d $SCRATCH_MNT -n 999999 -p $((LOAD_FACTOR * 4))
 
 	# purposely include 0 second sleeps to test shutdown immediately after
 	# recovery
@@ -73,12 +71,7 @@ while _soak_loop_running $((50 * TIME_FACTOR)); do
 	# error table helper *without* 'lockfs'.
 	_dmerror_load_error_table
 
-	ps -e | grep fsstress > /dev/null 2>&1
-	while [ $? -eq 0 ]; do
-		$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-		wait > /dev/null 2>&1
-		ps -e | grep fsstress > /dev/null 2>&1
-	done
+	_kill_fsstress
 
 	# Mount again to replay log after loading working table, so we have a
 	# consistent XFS after test.
tests/overlay/019 +33 -15
index ae026604d..b20b7ae52 100755
@@ -9,6 +9,22 @@
 . ./common/preamble
 _begin_fstest auto stress
 
+# This nests multiple background fsstress instances, so we have to
+# do some magic with _FSSTRESS_PID here.
+_cleanup()
+{
+	if [ -n "$fsstress_pid_1" ]; then
+		 FSTRESS_PID=$fsstress_pid_1
+		_kill_fsstress
+	fi
+	if [ -n "$fsstress_pid_2" ]; then
+		 FSTRESS_PID=$fsstress_pid_2
+		_kill_fsstress
+	fi
+	cd /
+	rm -f tmp.*
+}
+
 # Import common functions.
 . ./common/filter
 
@@ -28,29 +44,31 @@ d_low=$lowerdir/fsstress
 d_top=$SCRATCH_MNT/fsstress
 mkdir -p $d_low $d_top
 
-echo $FSSTRESS_PROG -s 42 -d $d_low -p 4 -n 1000 -l100 -v > $seqres.full.1
-$FSSTRESS_PROG -s 42 -d $d_low -p 4 -n 1000 -l100 -v >> $seqres.full.1 2>&1 &
+echo fsstress -s 42 -d $d_low -p 4 -n 1000 -l100 -v >> $seqres.full
+_run_fsstress_bg -s 42 -d $d_low -p 4 -n 1000 -l100 -v
+fsstress_pid_1=$_FSSTRESS_PID
 
-echo $FSSTRESS_PROG -s 42 -d $d_top -p 4 -n 1000 -l100 -v > $seqres.full.2
-$FSSTRESS_PROG -s 42 -d $d_top -p 4 -n 1000 -l100 -v >> $seqres.full.2 2>&1 &
+echo fsstress -s 42 -d $d_top -p 4 -n 1000 -l100 -v >> $seqres.full
+_run_fsstress_bg -s 42 -d $d_top -p 4 -n 1000 -l100 -v
+fsstress_pid_2=$_FSTRESS_PID
+unset _FSSTRESS_PID
 
 ret=0
-if ! wait %1; then
-	echo "--------------------------------------"       >>$seqres.full.1
-	echo "fsstress on lower directory returned $? - see $seqres.full.1"
-	echo "--------------------------------------"       >>$seqres.full.1
+if ! wait $fsstress_pid_1; then
+	echo "--------------------------------------"       >>$seqres.full
+	echo "fsstress on lower directory returned $? - see $seqres.full"
+	echo "--------------------------------------"       >>$seqres.full
 	ret=1
 fi
+unset fsstress_pid_1
 
-if ! wait %2; then
-	echo "--------------------------------------"       >>$seqres.full.2
-	echo "fsstress on overlay directory returned $? - see $seqres.full.2"
-	echo "--------------------------------------"       >>$seqres.full.2
+if ! wait $fsstress_pid_2; then
+	echo "--------------------------------------"       >>$seqres.full
+	echo "fsstress on overlay directory returned $? - see $seqres.full"
+	echo "--------------------------------------"       >>$seqres.full
 	ret=1
 fi
-
-cat $seqres.full.1 $seqres.full.2 > $seqres.full
-rm $seqres.full.1 $seqres.full.2
+unset fsstress_pid_2
 
 if [ "$ret" -eq 1 ]; then
 	status=1
tests/overlay/021 +4 -4
index 95a9ada5d..ee5a51007 100755
@@ -33,8 +33,8 @@ d_low=$lowerdir/$testdir
 mkdir -p $d_low
 
 # Create 4K empty files in 4 directories
-echo $FSSTRESS_PROG -d $d_low -p 4 -z -f creat=1 -n 1024 -v >> $seqres.full
-$FSSTRESS_PROG -d $d_low -p 4 -z -f creat=1 -n 1024 -v >> $seqres.full 2>&1
+echo fsstress -d $d_low -p 4 -z -f creat=1 -n 1024 -v >> $seqres.full
+_run_fsstress -d $d_low -p 4 -z -f creat=1 -n 1024 -v
 echo "--------------------------------------"       >> $seqres.full
 echo "Created 1K files in lower directory.  "       >> $seqres.full
 
@@ -91,9 +91,9 @@ echo "Go team truncate!!                    "       >> $seqres.full
 # Give team 'touch' a 1 second head start.
 # Team 'truncate' players should catch up after few copy up bombs.
 sleep 1
-$FSSTRESS_PROG -d $d_top -p 4 -z -f creat=1 -n 1024 -v >> $seqres.full &
+_run_fsstress -d $d_top -p 4 -z -f creat=1 -n 1024 -v
 
-wait %1 %2 %3 %4 %5
+wait %1 %2 %3 %4
 
 echo "Silence is golden"
 status=0
tests/xfs/006 +4 -3
index 50b36947d..20fd104bc 100755
@@ -13,9 +13,10 @@ _begin_fstest auto quick mount eio
 # Override the default cleanup function.
 _cleanup()
 {
+	_kill_fsstress
+	_dmerror_cleanup
 	cd /
 	rm -f $tmp.*
-	_dmerror_cleanup
 }
 
 # Import common functions.
@@ -45,7 +46,7 @@ fi
 # start a metadata-intensive workload, but no data allocation operation.
 # Because uncompleted new space allocation I/Os may cause XFS to shutdown
 # after loading error table.
-$FSSTRESS_PROG -z -n 5000 -p 10 \
+_run_fsstress -z -n 5000 -p 10 \
 	       -f creat=10 \
 	       -f resvsp=1 \
 	       -f truncate=1 \
@@ -57,7 +58,7 @@ $FSSTRESS_PROG -z -n 5000 -p 10 \
 	       -f unlink=1 \
 	       -f symlink=1 \
 	       -f rename=1 \
-	       -d $SCRATCH_MNT/fsstress >> $seqres.full 2>&1
+	       -d $SCRATCH_MNT/fsstress
 
 # Loading error table without "--nolockfs" option. Because "--nolockfs"
 # won't freeze fs, then some running I/Os may cause XFS to shutdown
tests/xfs/011 +3 -7
index df967f098..ed69879c5 100755
@@ -26,8 +26,7 @@ _cleanup()
 {
 	# Make sure $SCRATCH_MNT is unfreezed
 	xfs_freeze -u $SCRATCH_MNT 2>/dev/null
-	$KILLALL_PROG -9 fsstress 2>/dev/null
-	wait
+	_kill_fsstress
 	cd /
 	rm -f $tmp.*
 }
@@ -102,8 +101,7 @@ _scratch_mount
 
 _check_scratch_log_state
 
-$FSSTRESS_PROG -d $SCRATCH_MNT/fsstress -n 9999999 -p 2 -S t \
-	>> $seqres.full 2>&1 &
+_run_fsstress_bg -d $SCRATCH_MNT/fsstress -n 9999999 -p 2 -S t
 
 iters=5
 while [ $iters -gt 0 ]; do
@@ -112,9 +110,7 @@ while [ $iters -gt 0 ]; do
 	iters=$((iters - 1))
 done
 
-$KILLALL_PROG $FSSTRESS_PROG
-wait
-
+_kill_fsstress
 _scratch_unmount
 
 status=0
tests/xfs/013 +2 -14
index f4f406aa9..c68c6ad85 100755
@@ -16,16 +16,6 @@ _begin_fstest auto metadata stress
 # Import common functions.
 . ./common/filter
 
-# Override the default cleanup function.
-_cleanup()
-{
-	$KILLALL_PROG -9 fsstress 2>/dev/null
-	wait
-	cd /
-	_scratch_unmount 2>/dev/null
-	rm -f $tmp.*
-}
-
 filter_enospc() {
 	sed -e '/^.*No space left on device.*/d'
 }
@@ -103,8 +93,7 @@ _create $SCRATCH_MNT/dir1 $COUNT
 _cleaner $SCRATCH_MNT $LOOPS $MINDIRS &
 
 # start a background stress workload on the fs
-$FSSTRESS_PROG -d $SCRATCH_MNT/fsstress -n 9999999 -p 2 -S t \
-	>> $seqres.full 2>&1 &
+_run_fsstress_bg -d $SCRATCH_MNT/fsstress -n 9999999 -p 2 -S t
 
 # Each cycle clones the current directory and makes a random file replacement
 # pass on the new directory. The directory is copied to the next using hard
@@ -127,8 +116,7 @@ do
 	_rand_replace $SCRATCH_MNT/dir$((i+1)) $COUNT
 done
 
-$KILLALL_PROG fsstress
-wait
+_kill_fsstress
 
 # clean out the competing fsstress allocations, then everything else
 rm -rf $SCRATCH_MNT/fsstress
tests/xfs/017 +2 -11
index c40d9cf09..263ecc753 100755
@@ -9,15 +9,6 @@
 . ./common/preamble
 _begin_fstest mount auto quick stress
 
-_register_cleanup "_cleanup; rm -f $tmp.*"
-
-# Override the default cleanup function.
-_cleanup()
-{
-    echo "*** unmount"
-    _scratch_unmount 2>/dev/null
-}
-
 # Import common functions.
 . ./common/filter
 
@@ -41,8 +32,8 @@ echo "*** test"
 for l in 0 1 2 3 4
 do
         echo "    *** test $l"
-	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -n 1000 $FSSTRESS_AVOID`
-        $FSSTRESS_PROG  $FSSTRESS_ARGS >>$seqres.full
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -n 1000`
+        _run_fsstress $FSSTRESS_ARGS
 
         _try_scratch_mount -o remount,ro \
             || _fail "remount ro failed"
tests/xfs/017.out +0 -1
index 2d11c9492..e61793df1 100644
@@ -7,4 +7,3 @@ QA output created by 017
     *** test 3
     *** test 4
 *** done
-*** unmount
tests/xfs/032 +1 -1
index 75edf0e9c..1ecc02fe0 100755
@@ -50,7 +50,7 @@ while [ $SECTORSIZE -le $PAGESIZE ]; do
 		fi
 		_scratch_mount
 		# light population of the fs
-		$FSSTRESS_PROG -n 100 -d $SCRATCH_MNT >> $seqres.full 2>&1
+		_run_fsstress -n 100 -d $SCRATCH_MNT
 		_scratch_unmount
 
 		# Test "duplicate" copy at first, if $XFS_COPY_PROG won't do it.
tests/xfs/049 +2 -2
index 668ac3745..4163a144f 100755
@@ -68,7 +68,7 @@ mount -t xfs -o loop $SCRATCH_MNT/test.xfs $SCRATCH_MNT/test >> $seqres.full 2>&
     || _fail "!!! failed to loop mount xfs"
 
 _log "stress"
-$FSSTRESS_PROG -d $SCRATCH_MNT/test -n 1000 $FSSTRESS_AVOID >> $seqres.full 2>&1 \
+_run_fsstress -d $SCRATCH_MNT/test -n 1000 \
     || _fail "!!! stress failed"
 
 _log "clean"
@@ -88,7 +88,7 @@ mount -t ext2 -o loop $SCRATCH_MNT/test/test.ext2 $SCRATCH_MNT/test2 >> $seqres.
     || _fail "!!! failed to loop mount xfs"
 
 _log "stress ext2 on xfs via loop"
-$FSSTRESS_PROG -d $SCRATCH_MNT/test2 -n 1000 $FSSTRESS_AVOID >> $seqres.full 2>&1 \
+_run_fsstress -d $SCRATCH_MNT/test2 -n 1000 \
     || _fail "!!! stress ext2 failed"
 
 _log "clean"
tests/xfs/051 +2 -12
index 43fee4c45..bb9c36da8 100755
@@ -11,15 +11,6 @@
 . ./common/preamble
 _begin_fstest shutdown auto log metadata
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 $FSSTRESS_PROG > /dev/null 2>&1
-	_scratch_unmount > /dev/null 2>&1
-}
-
 # Import common functions.
 . ./common/dmflakey
 
@@ -37,11 +28,10 @@ _scratch_mount
 
 # Start a workload and shutdown the fs. The subsequent mount will require log
 # recovery.
-$FSSTRESS_PROG -n 9999 -p 2 -w -d $SCRATCH_MNT &>> $seqres.full &
+_run_fsstress_bg -n 9999 -p 2 -w -d $SCRATCH_MNT
 sleep 5
 _scratch_shutdown -f
-$KILLALL_PROG -q $FSSTRESS_PROG
-wait
+_kill_fsstress
 _scratch_unmount
 
 # Initialize a dm-flakey device that will pass I/Os for 5s and fail thereafter.
tests/xfs/057 +4 -4
index f1c947795..62eb8b93c 100755
@@ -26,12 +26,12 @@ _begin_fstest auto log recoveryloop
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
 	[ -e /sys/fs/xfs/$sdev/errortag/log_item_pin ] &&
 		echo 0 > /sys/fs/xfs/$sdev/errortag/log_item_pin
+	_kill_fsstress
 	wait > /dev/null 2>&1
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -53,7 +53,7 @@ _scratch_mkfs_sized $((1024 * 1024 * 500)) >> $seqres.full 2>&1
 _scratch_mount
 
 # populate the fs with some data and cycle the mount to reset the log head/tail
-$FSSTRESS_PROG -d $SCRATCH_MNT -z -fcreat=1 -p 4 -n 100000 >> $seqres.full
+_run_fsstress -d $SCRATCH_MNT -z -fcreat=1 -p 4 -n 100000
 _scratch_cycle_mount || _fail "cycle mount failed"
 
 # Pin the tail and start a file removal workload. File removal tends to
tests/xfs/077 +1 -1
index 4a4ac4702..3d8ecd9f6 100755
@@ -54,7 +54,7 @@ ORIG_UUID=`_scratch_xfs_db -c "uuid" | awk '{print $NF}'`
 
 _scratch_mount
 # Put some stuff on the fs
-$FSSTRESS_PROG -d $SCRATCH_MNT -n 100 -p 4 >> $seqres.full 2>&1
+_run_fsstress -d $SCRATCH_MNT -n 100 -p 4
 _scratch_unmount
 
 # Can xfs_db change it?
tests/xfs/079 +2 -12
index 46a15ed78..794d2db49 100755
@@ -17,19 +17,9 @@
 . ./common/preamble
 _begin_fstest shutdown auto log quick
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-	wait > /dev/null 2>&1
-}
-
 # Import common functions.
 . ./common/log
 
-
 # Modify as appropriate.
 _require_scratch
 _require_v2log
@@ -43,10 +33,10 @@ _scratch_mkfs >> $seqres.full 2>&1 || _fail "mkfs failed"
 _scratch_mount "-o logbsize=32k"
 
 # Run a workload to dirty the log, wait a bit and shutdown the fs.
-$FSSTRESS_PROG -d $SCRATCH_MNT -p 4 -n 99999999 >> $seqres.full 2>&1 &
+_run_fsstress_bg -d $SCRATCH_MNT -p 4 -n 99999999
 sleep 10
 _scratch_shutdown -f
-wait
+_wait_for_fsstress
 
 # Remount with a different log buffer size. Going from 32k to 64k increases the
 # log record extended header count, as the log record header can only handle 32k
tests/xfs/104 +4 -10
index 7f11f89a5..cd625d6b7 100755
@@ -40,9 +40,8 @@ _stress_scratch()
 	procs=3
 	nops=1000
 	# -w ensures that the only ops are ones which cause write I/O
-	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
-	    -n $nops $FSSTRESS_AVOID`
-	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 &
+	args=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs -n $nops`
+	_run_fsstress_bg $args
 }
 
 _require_scratch
@@ -73,14 +72,11 @@ for i in `seq 125 -1 90`; do
 	break
 done
 
-#
 # Grow the filesystem while actively stressing it...
-# Kick off more stress threads on each iteration, grow; repeat.
-#
 while [ $size -le $endsize ]; do
 	echo "*** stressing filesystem"
 	echo "*** stressing a ${sizeb} block filesystem" >> $seqres.full
-	_stress_scratch
+	_stress_scratch 
 	sleep 1
 	size=`expr $size + $incsize`
 	sizeb=`expr $size / $dbsize`	# in data blocks
@@ -92,10 +88,8 @@ while [ $size -le $endsize ]; do
 	[ `expr $size % $modsize` -eq 0 ] && wait	# every 4th iteration
 	echo AGCOUNT=$agcount | tee -a $seqres.full
 	echo && echo >> $seqres.full
+	_wait_for_fsstress
 done
-wait	# stop for any remaining stress processes
-
-_scratch_unmount
 
 status=0
 exit
tests/xfs/137 +1 -1
index dfc653573..d97942bf6 100755
@@ -29,7 +29,7 @@ _scratch_xfs_db -x -c "logformat -c 3" >> $seqres.full 2>&1
 
 # do some work on the fs to update metadata LSNs
 _scratch_mount
-$FSSTRESS_PROG -d $SCRATCH_MNT -n 999 -p 4 -w >> $seqres.full 2>&1
+_run_fsstress -d $SCRATCH_MNT -n 999 -p 4 -w
 _scratch_unmount
 
 # Reformat to the current cycle and try to mount. This fails in most cases
tests/xfs/141 +1 -11
index 5e9067e24..b630ba10d 100755
@@ -14,19 +14,9 @@
 . ./common/preamble
 _begin_fstest auto log metadata
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-	wait > /dev/null 2>&1
-}
-
 # Import common functions.
 . ./common/inject
 
-
 # Modify as appropriate.
 _require_xfs_io_error_injection "log_bad_crc"
 _require_scratch
@@ -49,7 +39,7 @@ while [ $nr_times -gt 0 ]; do
 
 	# Run fsstress until the filesystem shuts down. It will shut down
 	# automatically when error injection triggers.
-	$FSSTRESS_PROG -d $SCRATCH_MNT -p 4 -n 999999 >> $seqres.full 2>&1
+	_run_fsstress -d $SCRATCH_MNT -p 4 -n 999999
 
 	# Verify that we can remount the fs. Log recovery should handle the torn
 	# write.
tests/xfs/158 +2 -2
index 3c4e60f0e..89bf8c851 100755
@@ -31,7 +31,7 @@ _scratch_mkfs -m crc=1,inobtcount=0 >> $seqres.full
 _scratch_mount
 
 mkdir $SCRATCH_MNT/stress
-$FSSTRESS_PROG -d $SCRATCH_MNT/stress -n 1000 >> $seqres.full
+_run_fsstress -d $SCRATCH_MNT/stress -n 1000
 echo moo > $SCRATCH_MNT/urk
 
 _scratch_unmount
@@ -56,7 +56,7 @@ _check_scratch_xfs_features NEEDSREPAIR INOBTCNT
 
 echo "Filesystem should be usable again"
 _scratch_mount
-$FSSTRESS_PROG -d $SCRATCH_MNT/stress -n 1000 >> $seqres.full
+_run_fsstress -d $SCRATCH_MNT/stress -n 1000
 _scratch_unmount
 _check_scratch_fs
 _check_scratch_xfs_features INOBTCNT
tests/xfs/167 +4 -10
index f9da261db..5ef2aa2ea 100755
@@ -9,20 +9,12 @@
 . ./common/preamble
 _begin_fstest rw metadata auto stress prealloc
 
-# Override the default cleanup function.
-_cleanup()
-{
-	$KILLALL_PROG -r -q -TERM fsstress 2> /dev/null
-	wait	# ensures all fsstress processes died
-}
-
 workout()
 {
 	procs=100
 	nops=15000
-	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -p $procs -n $nops \
-	    $FSSTRESS_AVOID`
-	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full &
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -p $procs -n $nops`
+	_run_fsstress_bg $FSSTRESS_ARGS
 	sleep 2
 }
 
@@ -52,6 +44,8 @@ rm -f $TEST_FILE
 workout
 $TEST_PROG $LOOPS $TEST_FILE
 
+_kill_fsstress
+
 echo "     *** test done"
 
 status=0
tests/xfs/168 +4 -4
index f187a336f..098e0c86a 100755
@@ -38,9 +38,8 @@ stress_scratch()
 	local procs=3
 	local nops=1000
 	# -w ensures that the only ops are ones which cause write I/O
-	local FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w \
-		-p $procs -n $nops $FSSTRESS_AVOID`
-	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1
+	local args=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs -n $nops`
+	_run_fsstress_bg $args
 }
 
 _require_scratch_xfs_shrink
@@ -73,7 +72,7 @@ while [ $totalcount -gt 0 ]; do
 	decsize=`expr  41 \* 1048576 + 1 + $RANDOM \* $RANDOM % 1048576`
 
 	while [ $size -gt $endsize ]; do
-		stress_scratch &
+		stress_scratch
 		sleep 1
 
 		decb=`expr $decsize / $dbsize`    # in data blocks
@@ -95,6 +94,7 @@ while [ $totalcount -gt 0 ]; do
 		. $tmp.growfs
 
 		size=`expr $dblocks \* $dbsize`
+		_kill_fsstress
 		_scratch_unmount
 		_scratch_xfs_repair -n >> $seqres.full 2>&1 || \
 			_fail "xfs_repair failed with shrinking $sizeb"
tests/xfs/264 +2 -2
index 109fecd1c..a6e816d3c 100755
@@ -55,7 +55,7 @@ do_test()
 	# start a metadata-intensive workload, but no data allocation operation.
 	# Because uncompleted new space allocation I/Os may cause XFS to shutdown
 	# after loading error table.
-	$FSSTRESS_PROG -z -n 5000 -p 10 \
+	_run_fsstress -z -n 5000 -p 10 \
 		       -f creat=10 \
 		       -f resvsp=1 \
 		       -f truncate=1 \
@@ -67,7 +67,7 @@ do_test()
 		       -f unlink=1 \
 		       -f symlink=1 \
 		       -f rename=1 \
-		       -d $SCRATCH_MNT/fsstress >> $seqres.full 2>&1
+		       -d $SCRATCH_MNT/fsstress
 
 	# Loading error table without "--nolockfs" option. Because "--nolockfs"
 	# won't freeze fs, then some running I/Os may cause XFS to shutdown
tests/xfs/270 +1 -1
index 3744df5a9..d3bce386a 100755
@@ -80,7 +80,7 @@ if [ $? -ne 0 ]; then
 else
 	# no hang/panic is fine
 	cat $SCRATCH_MNT/testfile > /dev/null
-	$FSSTRESS_PROG -d $SCRATCH_MNT -p 4 -n 400 >>$seqres.full 2>&1
+	_run_fsstress -d $SCRATCH_MNT -p 4 -n 400
 fi
 
 # remount as rw, kernel should reject it
tests/xfs/297 +3 -5
index 2c5b03c5c..66c5d0cc7 100755
@@ -16,8 +16,7 @@ _cleanup()
 {
 	# Make sure $SCRATCH_MNT is unfreezed
 	xfs_freeze -u $SCRATCH_MNT 2>/dev/null
-	$KILLALL_PROG -q -9 $FSSTRESS_PROG
-	wait
+	_kill_fsstress
 	cd /
 	rm -f $tmp.*
 }
@@ -37,7 +36,7 @@ _scratch_mount
 STRESS_DIR="$SCRATCH_MNT/testdir"
 mkdir -p $STRESS_DIR
 
-$FSSTRESS_PROG -d $STRESS_DIR -n 100 -p 1000 $FSSTRESS_AVOID >>$seqres.full &
+_run_fsstress_bg -d $STRESS_DIR -n 1000 -p 1000 $FSSTRESS_AVOID
 
 # Freeze/unfreeze file system randomly
 echo "Start freeze/unfreeze randomly" | tee -a $seqres.full
@@ -60,8 +59,7 @@ while [ $LOOP -gt 0 ];do
 	let LOOP=$LOOP-1
 done
 echo "Test done" | tee -a $seqres.full
-$KILLALL_PROG -q $FSSTRESS_PROG
-wait
+_kill_fsstress
 
 status=0
 exit
tests/xfs/305 +3 -5
index 0ad3ef7fb..6371ed8a6 100755
@@ -4,7 +4,7 @@
 #
 # FS QA Test No. 305
 #
-# Test to verify that turn group/project quotas off while fstress and
+# Test to verify that turn group/project quotas off while fsstress and
 # user quotas are left on.
 #
 . ./common/preamble
@@ -18,7 +18,6 @@ _begin_fstest auto quota
 
 _require_scratch
 _require_xfs_quota
-_require_command "$KILLALL_PROG" killall
 
 _scratch_mkfs_xfs -m crc=1 >/dev/null 2>&1
 
@@ -33,12 +32,11 @@ _exercise()
 	_qmount
 	mkdir -p $QUOTA_DIR
 
-	$FSSTRESS_PROG -d $QUOTA_DIR -n 1000000 -p 100 $FSSTRESS_AVOID >>$seqres.full &
+	_run_fsstress_bg -d $QUOTA_DIR -n 1000000 -p 100
 	sleep 10
 	$XFS_QUOTA_PROG -x -c "disable -$type" $SCRATCH_DEV
 	sleep 5
-	$KILLALL_PROG -q $FSSTRESS_PROG
-	wait
+	_kill_fsstress
 }
 
 echo "*** turn off group quotas"
tests/xfs/442 +1 -9
index 77d08fda5..5cbd8dd19 100755
@@ -12,14 +12,6 @@
 . ./common/preamble
 _begin_fstest auto stress clone quota
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-}
-
 # Import common functions.
 . ./common/quota
 . ./common/filter
@@ -72,7 +64,7 @@ _scratch_mount >> $seqres.full 2>&1
 
 nr_cpus=$((LOAD_FACTOR * 4))
 nr_ops=$((25000 * nr_cpus * TIME_FACTOR))
-$FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full
+_run_fsstress -w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus
 
 echo "Check quota before remount"
 check_quota_du_blocks
tests/xfs/538 +2 -2
index 57113d341..f858cadc3 100755
@@ -46,7 +46,7 @@ echo "Inject bmap_alloc_minlen_extent error tag"
 _scratch_inject_error bmap_alloc_minlen_extent 1
 
 echo "Execute fsstress"
-$FSSTRESS_PROG -d $SCRATCH_MNT \
+_run_fsstress -d $SCRATCH_MNT \
 		$(_scale_fsstress_args -p 75 -n 1000) \
 		-f bulkstat=0 \
 		-f bulkstat1=0 \
@@ -61,7 +61,7 @@ $FSSTRESS_PROG -d $SCRATCH_MNT \
 		-f readv=0 \
 		-f stat=0 \
 		-f aread=0 \
-		-f dread=0 >> $seqres.full
+		-f dread=0
 
 # success, all done
 status=0
tests/xfs/609 +3 -17
index c23b6893d..88dc3c683 100755
@@ -18,21 +18,12 @@ _stress_scratch()
 	nops=999999
 	# -w ensures that the only ops are ones which cause write I/O
 	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
-	    -n $nops $FSSTRESS_AVOID`
-	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 &
+	    -n $nops`
+	_run_fsstress_bg $FSSTRESS_ARGS >> $seqres.full 2>&1
 }
 
 _require_scratch
 _require_command "$XFS_GROWFS_PROG" xfs_growfs
-_require_command "$KILLALL_PROG" killall
-
-_cleanup()
-{
-	$KILLALL_ALL fsstress > /dev/null 2>&1
-	wait
-	cd /
-	rm -f $tmp.*
-}
 
 _scratch_mkfs_xfs | _filter_mkfs >$seqres.full 2>$tmp.mkfs
 . $tmp.mkfs	# extract blocksize and data size for scratch device
@@ -63,12 +54,7 @@ while [ $size -le $endsize ]; do
 
 	sleep $((RANDOM % 3))
 	_scratch_shutdown
-	ps -e | grep fsstress > /dev/null 2>&1
-	while [ $? -eq 0 ]; do
-		$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-		wait > /dev/null 2>&1
-		ps -e | grep fsstress > /dev/null 2>&1
-	done
+	_kill_fsstress
 	_scratch_cycle_mount
 done > /dev/null 2>&1
 wait	# stop for any remaining stress processes
tests/xfs/610 +3 -17
index 23fbd8585..8610b912c 100755
@@ -18,22 +18,13 @@ _stress_scratch()
 	nops=999999
 	# -w ensures that the only ops are ones which cause write I/O
 	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
-	    -n $nops $FSSTRESS_AVOID`
-	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 &
+	    -n $nops`
+	_run_fsstress_bg $FSSTRESS_ARGS >> $seqres.full 2>&1
 }
 
 _require_scratch
 _require_realtime
 _require_command "$XFS_GROWFS_PROG" xfs_growfs
-_require_command "$KILLALL_PROG" killall
-
-_cleanup()
-{
-	$KILLALL_ALL fsstress > /dev/null 2>&1
-	wait
-	cd /
-	rm -f $tmp.*
-}
 
 _scratch_mkfs_xfs | _filter_mkfs >$seqres.full 2>$tmp.mkfs
 . $tmp.mkfs	# extract blocksize and data size for scratch device
@@ -65,12 +56,7 @@ while [ $size -le $endsize ]; do
 
 	sleep $((RANDOM % 3))
 	_scratch_shutdown
-	ps -e | grep fsstress > /dev/null 2>&1
-	while [ $? -eq 0 ]; do
-		$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-		wait > /dev/null 2>&1
-		ps -e | grep fsstress > /dev/null 2>&1
-	done
+	_kill_fsstress
 	_scratch_cycle_mount
 done > /dev/null 2>&1
 wait	# stop for any remaining stress processes
Signature · 1 line
2.45.2
#5 lore raw

Re: [PATCH 02/40] fstests: cleanup fsstress process management

Quoted 23 lines · Zorro Lang
Quoted 9 lines · Dave Chinner
common/rc +9 -0
+# Common execution handling for fsstress invocation.
+#
+# We need per-test fsstress binaries because of the way fsstress forks and
+# tests run it in the background and/or nest it. Trying to kill fsstress
+# tasks is unreliable because killing parent fsstress task does not guarantee
+# that the children get killed. Hence the historic use of killall for stopping
+# execution.
+#
+# However, we can't just kill all fsstress binaries as multiple tests might be

common/rc line 27

Quoted 10 lines
common/rc +10 -0
+# running fsstress at the same time. Hence copy the fsstress binary to a test
+# specific binary on the test device and use pkill to select that only that
+# task name to kill.
+#
+# If tasks want to start fsstress themselves (e.g. under a different uid) then
+# they can set up _FSSTRESS_BIN and record _FSSTRESS_PID themselves. Then if the
+# test is killed then it will get cleaned up automatically.
+
+_FSSTRESS_BIN="$seq.fsstress"
+_FSSTRESS_PROG="$TEST_DIR/$seq.fsstress"

Hi Dave,

Quoted 4 lines

I'm wondering if the "$seq.fsstress" can be unique name? For example, if generic/561, xfs/561, btrfs/561, ext4/561 run fsstress in parallel, won't they have same "561.fsstress", then "pkill -PIPE $_FSSTRESS_BIN" kills all these cases' fsstress processes?

Yeah, it's not entirely unique, but it was "good enough" to solve the problem and move on to the next one. It should be easy enough to change; the reason I used $seq was because it is short, and the pkill man page says:

NOTES
The process name used for matching is limited to the 15 characters present in the output of /proc/<pid>/stat.

It turns out that this 15 character limit doesn't appear to exist anymore. I just checked that file on a 6.10.7 kernel, and it appears that there are full process names in there. pgrep (at least) matches on process names longer than 15 character, so I suspect that the man page is simply out of date here.

Hence we might be able to use something longer and more unique along with 'pkill -x' to match the full name. i.e. this is a solvable problem, but not critical to the effective working of concurrent test running.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#6 lore raw

Re: [PATCH 02/40] fstests: cleanup fsstress process management

Quoted 24 lines · Dave Chinner
Quoted 23 lines · Zorro Lang
Quoted 9 lines · Dave Chinner
common/rc +9 -0
+# Common execution handling for fsstress invocation.
+#
+# We need per-test fsstress binaries because of the way fsstress forks and
+# tests run it in the background and/or nest it. Trying to kill fsstress
+# tasks is unreliable because killing parent fsstress task does not guarantee
+# that the children get killed. Hence the historic use of killall for stopping
+# execution.
+#
+# However, we can't just kill all fsstress binaries as multiple tests might be

common/rc line 27

Quoted 10 lines
common/rc +10 -0
+# running fsstress at the same time. Hence copy the fsstress binary to a test
+# specific binary on the test device and use pkill to select that only that
+# task name to kill.
+#
+# If tasks want to start fsstress themselves (e.g. under a different uid) then
+# they can set up _FSSTRESS_BIN and record _FSSTRESS_PID themselves. Then if the
+# test is killed then it will get cleaned up automatically.
+
+_FSSTRESS_BIN="$seq.fsstress"
+_FSSTRESS_PROG="$TEST_DIR/$seq.fsstress"

Hi Dave,

Quoted 7 lines
Quoted 4 lines

I'm wondering if the "$seq.fsstress" can be unique name? For example, if generic/561, xfs/561, btrfs/561, ext4/561 run fsstress in parallel, won't they have same "561.fsstress", then "pkill -PIPE $_FSSTRESS_BIN" kills all these cases' fsstress processes?

Yeah, it's not entirely unique, but it was "good enough" to solve the problem and move on to the next one. It should be easy enough to

Sure, it's good to move on to the next one, due to it doesn't bring in regression for original testing way. We can improve this part later.

Quoted 13 lines

change; the reason I used $seq was because it is short, and the pkill man page says:

NOTES
The process name used for matching is limited to the 15 characters present in the output of /proc/<pid>/stat.

It turns out that this 15 character limit doesn't appear to exist anymore. I just checked that file on a 6.10.7 kernel, and it appears that there are full process names in there. pgrep (at least) matches on process names longer than 15 character, so I suspect that the man page is simply out of date here.

Quoted 4 lines

Hence we might be able to use something longer and more unique along with 'pkill -x' to match the full name. i.e. this is a solvable problem, but not critical to the effective working of concurrent test running.

Good to know that, thanks! So we can use "xfs-561-fsstress" or "$$-$seq-fsstress" or something better idea you have :)

Thanks,
Zorro

Quoted 7 lines

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#7 lore raw

Re: [PATCH 02/40] fstests: cleanup fsstress process management

In reply to Dave Chinner
Quoted 72 lines · Dave Chinner

From: Dave Chinner <[email protected]>

Lots of tests run fsstress in the background and then have to kill it and/or provide special cleanup functions to kill the background fsstress processes. They typically use $KILLALL_PROG for this.

Use of killall is problematic for running multiple tests in parallel in that one test can kill other tests' processes. However, because fsstress itself forks and runs children, there are very few avenues for shell scripts to ensure all the fsstress processes actually die.

With bash, it is especially nasty, because sending SIGTERM will result in bash outputting error messages ("Killed: ..." that will cause golden output mismatches and hence test failures. Hence we also need to be able to tell the main fstress process to die without triggering these messages.

To avoid the process tracking problems, we change to use pkill rather than killall (more options for process selection) and we stop using the $here/ltp/fsstress binary. Instead, we copy the $here/ltp/fsstress to $TEST_DIR/$seq.fsstress so that the test has a unique fsstress binary name. This allows the pkill filter to select just the fsstress processes the test has run. The fsstress binary name is held in _FSSTRESS_NAME, and the program to run is _FSSTRESS_PROG.

We also track the primary fsstress process ID, and store that in _FSSTRESS_PID. We do this so that we have a PID to wait against so that we don't return before the fsstress processes are dead. To this end, we add a SIGPIPE handler to the primary process so that it dying doesn't trigger bash 'killed' message output. We can send 'pkill -PIPE $_FSSTRESS_NAME' to all the fsstress processes and the primary process will then enter the "wait for children to die" processing loop before it exits. In this way, we can wait for the primary fsstress process and when it exits we know that all it's children have also finished and gone away. This makes killing fsstress invocations reliable and noise free.

This is accomplished by the helpers added to common/rc:

_run_fsstress
_run_fsstress_bg
_wait_for_fsstress
_kill_fstress

This also means that all fsstress invocations now obey FSSTRESS_AVOID environment restrictions, many of which didn't.

We add a call to _kill_fstress into the generic _cleanup() function. This means that tests using fsstress don't need to add a special local _cleanup function just to call _kill_fsstress() so that background fsstress processes are killed when the user interrupts the tests with ctrl-c.

Further, killall in the _cleanup() function is often used to attempt to expedite killing of foreground execution fsstress processes. This doesn't actually work because of the way bash processes interupt signals. That is, it waits for the currently executing process to finish execution, then runs the trap function. Hence a foreground fsstress won't ever be interrupted by ctrl-c. By implementing _run_fsstress() as a background process and a wait call, the wait() call is interrupted by the signal and the cleanup trap is run immediately. Hence the fsstress processes are killed immediately and the test exits cleanly almost immediately.

The result of all this is common, clean handling of fsstress execution and termination. There are a few exceptions for special cases, but the vast majority of tests that run fsstress use the above four wrapper functions exclusively.

---

[snip]

Quoted 24 lines
tests/generic/561 +1 -2
index 39e5977a3..3e931b1a7 100755
@@ -13,9 +13,9 @@ _begin_fstest auto stress dedupe
 # Override the default cleanup function.
 _cleanup()
 {
+	end_test
 	cd /
 	rm -f $tmp.*
-	end_test
 }
 
 # Import common functions.
@@ -23,28 +23,20 @@ _cleanup()
 . ./common/reflink
 
 _require_scratch_duperemove
-_require_command "$KILLALL_PROG" killall
 
 _scratch_mkfs > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 

tests/generic/561 _cleanup() · line 37

Quoted 10 lines
tests/generic/561 +2 -2
 function end_test()
 {
-	local f=1
+	_kill_fsstress
 
 	# stop duperemove running
 	if [ -e $dupe_run ]; then
 		rm -f $dupe_run
-		$KILLALL_PROG -q $DUPEREMOVE_PROG > /dev/null 2>&1
+		kill $dedup_pids

Hi Dave,

The $dedup_pids is a "while loop" bash process, it isn't the $DUPEREMOVE_PROG process itself. From my testing, this change might cause g/561 keep waiting $DUPEREMOVE_PROG processes forever, as $DUPEREMOVE_PROG not always be killed properly.

I'm wondering if you hope to do "pkill $DUPEREMOVE_PROG" directly? Or you'd like to copy $DUPEREMOVE_PROG to $TEST_DIR/${othername}_duperemove, then pkill ${othername}_duperemove ?

Thanks,
Zorro

Quoted 1,155 lines
tests/generic/561 +2 -8
 		wait $dedup_pids
 	fi
-
-	# Make sure all fsstress get killed
-	while [ $f -ne 0 ]; do
-		$KILLALL_PROG -q $FSSTRESS_PROG > /dev/null 2>&1
-		sleep 1
-		f=`ps -eLf | grep $FSSTRESS_PROG | grep -v "grep" | wc -l`
-	done
 }
 
 sleep_time=$((50 * TIME_FACTOR))
@@ -53,7 +45,8 @@ sleep_time=$((50 * TIME_FACTOR))
 testdir="$SCRATCH_MNT/dir"
 mkdir $testdir
 fsstress_opts="-r -n 1000 -p $((5 * LOAD_FACTOR))"
-$FSSTRESS_PROG $fsstress_opts -d $testdir -l 0 >> $seqres.full 2>&1 &
+_run_fsstress_bg $fsstress_opts -d $testdir -l 0
+
 dedup_pids=""
 dupe_run=$TEST_DIR/${seq}-running
 # Start several dedupe processes on same directory
tests/generic/579 +2 -4
index 3191342b3..4187ab0f0 100755
@@ -16,7 +16,7 @@ _begin_fstest auto stress verity
 _cleanup()
 {
 	# Stop all subprocesses.
-	$KILLALL_PROG -q $FSSTRESS_PROG
+	_kill_fsstress
 	touch $tmp.done
 	wait
 
@@ -29,7 +29,6 @@ _cleanup()
 . ./common/verity
 
 _require_scratch_verity
-_require_command "$KILLALL_PROG" killall
 _disable_fsverity_signatures
 
 _scratch_mkfs_verity &>> $seqres.full
@@ -92,8 +91,7 @@ done
 ) &
 
 # Start the fsstress processes.
-$FSSTRESS_PROG $FSSTRESS_AVOID -p $nproc_stress -l 0 -d $SCRATCH_MNT/stressdir \
-	>> $seqres.full 2>&1 &
+_run_fsstress_bg -p $nproc_stress -l 0 -d $SCRATCH_MNT/stressdir
 
 # Run for a while.
 sleep $runtime
tests/generic/585 +2 -2
index fb675c8d5..b95cd9abf 100755
@@ -23,10 +23,10 @@ _scratch_mount >> $seqres.full 2>&1
 # start a create and rename(rename_whiteout) workload. These processes
 # occur simultaneously may cause the deadlock between AGI and AGF with
 # RENAME_WHITEOUT.
-$FSSTRESS_PROG -z -n 150 -p 100 \
+_run_fsstress -z -n 150 -p 100 \
 		-f mknod=5 \
 		-f rwhiteout=5 \
-		-d $SCRATCH_MNT/fsstress >> $seqres.full 2>&1
+		-d $SCRATCH_MNT/fsstress
 
 echo Silence is golden
 
tests/generic/589 +4 -3
index 0ce16556a..969a8ac61 100755
@@ -28,11 +28,12 @@ _begin_fstest auto mount
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
+	_kill_fsstress
 	_clear_mount_stack
 	# make sure there's no bug cause dentry isn't be freed
 	rm -rf $MNTHEAD
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -46,7 +47,7 @@ fs_stress()
 {
 	local target=$1
 
-	$FSSTRESS_PROG -n 50 -p 3 -d $target >>$seqres.full
+	_run_fsstress -n 50 -p 3 -d $target
 	sync
 }
 
tests/generic/642 +1 -9
index a7112a08f..4b92a9c18 100755
@@ -10,17 +10,9 @@
 . ./common/preamble
 _begin_fstest auto soak attr long_rw stress smoketest
 
-_cleanup()
-{
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-	cd /
-	rm -f $tmp.*
-}
-
 # Modify as appropriate.
 
 _require_scratch
-_require_command "$KILLALL_PROG" "killall"
 
 echo "Silence is golden."
 
@@ -50,7 +42,7 @@ args+=('-f' "setfattr=20")
 args+=('-f' "attr_set=60")	# sets larger xattrs
 test -n "$SOAK_DURATION" && args+=(--duration="$SOAK_DURATION")
 
-$FSSTRESS_PROG "${args[@]}" $FSSTRESS_AVOID -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full
+_run_fsstress "${args[@]}" -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus
 
 # success, all done
 status=0
tests/generic/648 +5 -14
index 29d1b470b..e4c9990e1 100755
@@ -16,16 +16,15 @@ _begin_fstest shutdown auto log metadata eio recoveryloop
 
 _cleanup()
 {
-	cd /
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-	wait
+	_kill_fsstress
 	if [ -n "$loopmnt" ]; then
 		$UMOUNT_PROG $loopmnt 2>/dev/null
 		rm -r -f $loopmnt
 	fi
-	rm -f $tmp.*
 	_dmerror_unmount
 	_dmerror_cleanup
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -37,7 +36,6 @@ _cleanup()
 _require_scratch_reflink
 _require_cp_reflink
 _require_dm_target error
-_require_command "$KILLALL_PROG" "killall"
 _require_loop
 
 echo "Silence is golden."
@@ -71,8 +69,6 @@ snap_loop_fs() {
 	rm -f "$snap_aliveflag"
 }
 
-fsstress=($FSSTRESS_PROG $FSSTRESS_AVOID -d "$loopmnt" -n 999999 -p "$((LOAD_FACTOR * 4))")
-
 while _soak_loop_running $((25 * TIME_FACTOR)); do
 	touch $scratch_aliveflag
 	snap_loop_fs >> $seqres.full 2>&1 &
@@ -84,7 +80,7 @@ while _soak_loop_running $((25 * TIME_FACTOR)); do
 		break
 	fi
 
-	("${fsstress[@]}" >> $seqres.full &) > /dev/null 2>&1
+	_run_fsstress_bg -d "$loopmnt" -n 999999 -p "$((LOAD_FACTOR * 4))"
 
 	# purposely include 0 second sleeps to test shutdown immediately after
 	# recovery
@@ -98,12 +94,7 @@ while _soak_loop_running $((25 * TIME_FACTOR)); do
 	# error table helper *without* 'lockfs'.
 	_dmerror_load_error_table
 
-	ps -e | grep fsstress > /dev/null 2>&1
-	while [ $? -eq 0 ]; do
-		$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-		wait > /dev/null 2>&1
-		ps -e | grep fsstress > /dev/null 2>&1
-	done
+	_kill_fsstress
 	for ((j = 0; j < 10; j++)); do
 		test -e "$snap_aliveflag" || break
 		sleep 1
tests/generic/650 +9 -6
index 5d2cb8977..36a23e48d 100755
@@ -16,14 +16,14 @@ _begin_fstest auto rw stress soak
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
+	_kill_fsstress
 	wait	# for exercise_cpu_hotplug subprocess
 	for i in "$sysfs_cpu_dir/"cpu*/online; do
 		echo 1 > "$i" 2>/dev/null
 	done
 	test -n "$stress_dir" && rm -r -f "$stress_dir"
+	cd /
+	rm -f $tmp.*
 }
 
 exercise_cpu_hotplug()
@@ -39,7 +39,6 @@ exercise_cpu_hotplug()
 }
 
 _require_test
-_require_command "$KILLALL_PROG" "killall"
 
 sysfs_cpu_dir="/sys/devices/system/cpu"
 
@@ -71,13 +70,17 @@ test "$nr_cpus" -gt 1024 && nr_cpus="$nr_hotplug_cpus"
 fsstress_args+=(-p $nr_cpus)
 if [ -n "$SOAK_DURATION" ]; then
 	test "$SOAK_DURATION" -lt 10 && SOAK_DURATION=10
-	fsstress_args+=(--duration="$((SOAK_DURATION / 10))")
+else
+	# run for 30s per iteration max
+	SOAK_DURATION=300
 fi
+fsstress_args+=(--duration="$((SOAK_DURATION / 10))")
 
 nr_ops=$((2500 * TIME_FACTOR))
 fsstress_args+=(-n $nr_ops)
 for ((i = 0; i < 10; i++)); do
-	$FSSTRESS_PROG $FSSTRESS_AVOID -w "${fsstress_args[@]}" >> $seqres.full
+	_run_fsstress_bg -w "${fsstress_args[@]}"
+	_wait_for_fsstress
 	_test_cycle_mount
 done
 
tests/generic/750 +4 -6
index dba8021d6..5c54a5c78 100755
@@ -11,13 +11,12 @@ _begin_fstest auto rw long_rw stress soak smoketest
 
 _cleanup()
 {
-	cd /
+	_kill_fsstress
 	rm -f $runfile
-	rm -f $tmp.*
 	kill -9 $trigger_compaction_pid > /dev/null 2>&1
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-
 	wait > /dev/null 2>&1
+	rm -f $tmp.*
+	cd /
 }
 
 # Import common functions.
@@ -26,7 +25,6 @@ _cleanup()
 
 _require_scratch
 _require_vm_compaction
-_require_command "$KILLALL_PROG" "killall"
 
 # We still deadlock with this test on v6.10-rc2, we need more work.
 # but the below makes things better.
@@ -52,7 +50,7 @@ while [ -e $runfile ]; do
 done &
 trigger_compaction_pid=$!
 
-$FSSTRESS_PROG $FSSTRESS_AVOID "${fsstress_args[@]}" >> $seqres.full
+_run_fsstress "${fsstress_args[@]}"
 
 rm -f $runfile
 wait > /dev/null 2>&1
tests/generic/753 +5 -12
index e427d62d1..f5665320a 100755
@@ -16,11 +16,11 @@ _begin_fstest shutdown auto log metadata eio recoveryloop attr
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
+	_kill_fsstress
 	_dmerror_unmount
 	_dmerror_cleanup
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -30,7 +30,6 @@ _cleanup()
 
 _require_scratch
 _require_dm_target error
-_require_command "$KILLALL_PROG" "killall"
 
 echo "Silence is golden."
 
@@ -59,8 +58,7 @@ args+=('-f' "setfattr=20")
 args+=('-f' "attr_set=60")	# sets larger xattrs
 
 while _soak_loop_running $((50 * TIME_FACTOR)); do
-	($FSSTRESS_PROG "${args[@]}" $FSSTRESS_AVOID -d $SCRATCH_MNT -n 999999 -p $((LOAD_FACTOR * 4)) >> $seqres.full &) \
-		> /dev/null 2>&1
+	_run_fsstress_bg "${args[@]}" -d $SCRATCH_MNT -n 999999 -p $((LOAD_FACTOR * 4))
 
 	# purposely include 0 second sleeps to test shutdown immediately after
 	# recovery
@@ -73,12 +71,7 @@ while _soak_loop_running $((50 * TIME_FACTOR)); do
 	# error table helper *without* 'lockfs'.
 	_dmerror_load_error_table
 
-	ps -e | grep fsstress > /dev/null 2>&1
-	while [ $? -eq 0 ]; do
-		$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-		wait > /dev/null 2>&1
-		ps -e | grep fsstress > /dev/null 2>&1
-	done
+	_kill_fsstress
 
 	# Mount again to replay log after loading working table, so we have a
 	# consistent XFS after test.
tests/overlay/019 +33 -15
index ae026604d..b20b7ae52 100755
@@ -9,6 +9,22 @@
 . ./common/preamble
 _begin_fstest auto stress
 
+# This nests multiple background fsstress instances, so we have to
+# do some magic with _FSSTRESS_PID here.
+_cleanup()
+{
+	if [ -n "$fsstress_pid_1" ]; then
+		 FSTRESS_PID=$fsstress_pid_1
+		_kill_fsstress
+	fi
+	if [ -n "$fsstress_pid_2" ]; then
+		 FSTRESS_PID=$fsstress_pid_2
+		_kill_fsstress
+	fi
+	cd /
+	rm -f tmp.*
+}
+
 # Import common functions.
 . ./common/filter
 
@@ -28,29 +44,31 @@ d_low=$lowerdir/fsstress
 d_top=$SCRATCH_MNT/fsstress
 mkdir -p $d_low $d_top
 
-echo $FSSTRESS_PROG -s 42 -d $d_low -p 4 -n 1000 -l100 -v > $seqres.full.1
-$FSSTRESS_PROG -s 42 -d $d_low -p 4 -n 1000 -l100 -v >> $seqres.full.1 2>&1 &
+echo fsstress -s 42 -d $d_low -p 4 -n 1000 -l100 -v >> $seqres.full
+_run_fsstress_bg -s 42 -d $d_low -p 4 -n 1000 -l100 -v
+fsstress_pid_1=$_FSSTRESS_PID
 
-echo $FSSTRESS_PROG -s 42 -d $d_top -p 4 -n 1000 -l100 -v > $seqres.full.2
-$FSSTRESS_PROG -s 42 -d $d_top -p 4 -n 1000 -l100 -v >> $seqres.full.2 2>&1 &
+echo fsstress -s 42 -d $d_top -p 4 -n 1000 -l100 -v >> $seqres.full
+_run_fsstress_bg -s 42 -d $d_top -p 4 -n 1000 -l100 -v
+fsstress_pid_2=$_FSTRESS_PID
+unset _FSSTRESS_PID
 
 ret=0
-if ! wait %1; then
-	echo "--------------------------------------"       >>$seqres.full.1
-	echo "fsstress on lower directory returned $? - see $seqres.full.1"
-	echo "--------------------------------------"       >>$seqres.full.1
+if ! wait $fsstress_pid_1; then
+	echo "--------------------------------------"       >>$seqres.full
+	echo "fsstress on lower directory returned $? - see $seqres.full"
+	echo "--------------------------------------"       >>$seqres.full
 	ret=1
 fi
+unset fsstress_pid_1
 
-if ! wait %2; then
-	echo "--------------------------------------"       >>$seqres.full.2
-	echo "fsstress on overlay directory returned $? - see $seqres.full.2"
-	echo "--------------------------------------"       >>$seqres.full.2
+if ! wait $fsstress_pid_2; then
+	echo "--------------------------------------"       >>$seqres.full
+	echo "fsstress on overlay directory returned $? - see $seqres.full"
+	echo "--------------------------------------"       >>$seqres.full
 	ret=1
 fi
-
-cat $seqres.full.1 $seqres.full.2 > $seqres.full
-rm $seqres.full.1 $seqres.full.2
+unset fsstress_pid_2
 
 if [ "$ret" -eq 1 ]; then
 	status=1
tests/overlay/021 +4 -4
index 95a9ada5d..ee5a51007 100755
@@ -33,8 +33,8 @@ d_low=$lowerdir/$testdir
 mkdir -p $d_low
 
 # Create 4K empty files in 4 directories
-echo $FSSTRESS_PROG -d $d_low -p 4 -z -f creat=1 -n 1024 -v >> $seqres.full
-$FSSTRESS_PROG -d $d_low -p 4 -z -f creat=1 -n 1024 -v >> $seqres.full 2>&1
+echo fsstress -d $d_low -p 4 -z -f creat=1 -n 1024 -v >> $seqres.full
+_run_fsstress -d $d_low -p 4 -z -f creat=1 -n 1024 -v
 echo "--------------------------------------"       >> $seqres.full
 echo "Created 1K files in lower directory.  "       >> $seqres.full
 
@@ -91,9 +91,9 @@ echo "Go team truncate!!                    "       >> $seqres.full
 # Give team 'touch' a 1 second head start.
 # Team 'truncate' players should catch up after few copy up bombs.
 sleep 1
-$FSSTRESS_PROG -d $d_top -p 4 -z -f creat=1 -n 1024 -v >> $seqres.full &
+_run_fsstress -d $d_top -p 4 -z -f creat=1 -n 1024 -v
 
-wait %1 %2 %3 %4 %5
+wait %1 %2 %3 %4
 
 echo "Silence is golden"
 status=0
tests/xfs/006 +4 -3
index 50b36947d..20fd104bc 100755
@@ -13,9 +13,10 @@ _begin_fstest auto quick mount eio
 # Override the default cleanup function.
 _cleanup()
 {
+	_kill_fsstress
+	_dmerror_cleanup
 	cd /
 	rm -f $tmp.*
-	_dmerror_cleanup
 }
 
 # Import common functions.
@@ -45,7 +46,7 @@ fi
 # start a metadata-intensive workload, but no data allocation operation.
 # Because uncompleted new space allocation I/Os may cause XFS to shutdown
 # after loading error table.
-$FSSTRESS_PROG -z -n 5000 -p 10 \
+_run_fsstress -z -n 5000 -p 10 \
 	       -f creat=10 \
 	       -f resvsp=1 \
 	       -f truncate=1 \
@@ -57,7 +58,7 @@ $FSSTRESS_PROG -z -n 5000 -p 10 \
 	       -f unlink=1 \
 	       -f symlink=1 \
 	       -f rename=1 \
-	       -d $SCRATCH_MNT/fsstress >> $seqres.full 2>&1
+	       -d $SCRATCH_MNT/fsstress
 
 # Loading error table without "--nolockfs" option. Because "--nolockfs"
 # won't freeze fs, then some running I/Os may cause XFS to shutdown
tests/xfs/011 +3 -7
index df967f098..ed69879c5 100755
@@ -26,8 +26,7 @@ _cleanup()
 {
 	# Make sure $SCRATCH_MNT is unfreezed
 	xfs_freeze -u $SCRATCH_MNT 2>/dev/null
-	$KILLALL_PROG -9 fsstress 2>/dev/null
-	wait
+	_kill_fsstress
 	cd /
 	rm -f $tmp.*
 }
@@ -102,8 +101,7 @@ _scratch_mount
 
 _check_scratch_log_state
 
-$FSSTRESS_PROG -d $SCRATCH_MNT/fsstress -n 9999999 -p 2 -S t \
-	>> $seqres.full 2>&1 &
+_run_fsstress_bg -d $SCRATCH_MNT/fsstress -n 9999999 -p 2 -S t
 
 iters=5
 while [ $iters -gt 0 ]; do
@@ -112,9 +110,7 @@ while [ $iters -gt 0 ]; do
 	iters=$((iters - 1))
 done
 
-$KILLALL_PROG $FSSTRESS_PROG
-wait
-
+_kill_fsstress
 _scratch_unmount
 
 status=0
tests/xfs/013 +2 -14
index f4f406aa9..c68c6ad85 100755
@@ -16,16 +16,6 @@ _begin_fstest auto metadata stress
 # Import common functions.
 . ./common/filter
 
-# Override the default cleanup function.
-_cleanup()
-{
-	$KILLALL_PROG -9 fsstress 2>/dev/null
-	wait
-	cd /
-	_scratch_unmount 2>/dev/null
-	rm -f $tmp.*
-}
-
 filter_enospc() {
 	sed -e '/^.*No space left on device.*/d'
 }
@@ -103,8 +93,7 @@ _create $SCRATCH_MNT/dir1 $COUNT
 _cleaner $SCRATCH_MNT $LOOPS $MINDIRS &
 
 # start a background stress workload on the fs
-$FSSTRESS_PROG -d $SCRATCH_MNT/fsstress -n 9999999 -p 2 -S t \
-	>> $seqres.full 2>&1 &
+_run_fsstress_bg -d $SCRATCH_MNT/fsstress -n 9999999 -p 2 -S t
 
 # Each cycle clones the current directory and makes a random file replacement
 # pass on the new directory. The directory is copied to the next using hard
@@ -127,8 +116,7 @@ do
 	_rand_replace $SCRATCH_MNT/dir$((i+1)) $COUNT
 done
 
-$KILLALL_PROG fsstress
-wait
+_kill_fsstress
 
 # clean out the competing fsstress allocations, then everything else
 rm -rf $SCRATCH_MNT/fsstress
tests/xfs/017 +2 -11
index c40d9cf09..263ecc753 100755
@@ -9,15 +9,6 @@
 . ./common/preamble
 _begin_fstest mount auto quick stress
 
-_register_cleanup "_cleanup; rm -f $tmp.*"
-
-# Override the default cleanup function.
-_cleanup()
-{
-    echo "*** unmount"
-    _scratch_unmount 2>/dev/null
-}
-
 # Import common functions.
 . ./common/filter
 
@@ -41,8 +32,8 @@ echo "*** test"
 for l in 0 1 2 3 4
 do
         echo "    *** test $l"
-	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -n 1000 $FSSTRESS_AVOID`
-        $FSSTRESS_PROG  $FSSTRESS_ARGS >>$seqres.full
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -n 1000`
+        _run_fsstress $FSSTRESS_ARGS
 
         _try_scratch_mount -o remount,ro \
             || _fail "remount ro failed"
tests/xfs/017.out +0 -1
index 2d11c9492..e61793df1 100644
@@ -7,4 +7,3 @@ QA output created by 017
     *** test 3
     *** test 4
 *** done
-*** unmount
tests/xfs/032 +1 -1
index 75edf0e9c..1ecc02fe0 100755
@@ -50,7 +50,7 @@ while [ $SECTORSIZE -le $PAGESIZE ]; do
 		fi
 		_scratch_mount
 		# light population of the fs
-		$FSSTRESS_PROG -n 100 -d $SCRATCH_MNT >> $seqres.full 2>&1
+		_run_fsstress -n 100 -d $SCRATCH_MNT
 		_scratch_unmount
 
 		# Test "duplicate" copy at first, if $XFS_COPY_PROG won't do it.
tests/xfs/049 +2 -2
index 668ac3745..4163a144f 100755
@@ -68,7 +68,7 @@ mount -t xfs -o loop $SCRATCH_MNT/test.xfs $SCRATCH_MNT/test >> $seqres.full 2>&
     || _fail "!!! failed to loop mount xfs"
 
 _log "stress"
-$FSSTRESS_PROG -d $SCRATCH_MNT/test -n 1000 $FSSTRESS_AVOID >> $seqres.full 2>&1 \
+_run_fsstress -d $SCRATCH_MNT/test -n 1000 \
     || _fail "!!! stress failed"
 
 _log "clean"
@@ -88,7 +88,7 @@ mount -t ext2 -o loop $SCRATCH_MNT/test/test.ext2 $SCRATCH_MNT/test2 >> $seqres.
     || _fail "!!! failed to loop mount xfs"
 
 _log "stress ext2 on xfs via loop"
-$FSSTRESS_PROG -d $SCRATCH_MNT/test2 -n 1000 $FSSTRESS_AVOID >> $seqres.full 2>&1 \
+_run_fsstress -d $SCRATCH_MNT/test2 -n 1000 \
     || _fail "!!! stress ext2 failed"
 
 _log "clean"
tests/xfs/051 +2 -12
index 43fee4c45..bb9c36da8 100755
@@ -11,15 +11,6 @@
 . ./common/preamble
 _begin_fstest shutdown auto log metadata
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 $FSSTRESS_PROG > /dev/null 2>&1
-	_scratch_unmount > /dev/null 2>&1
-}
-
 # Import common functions.
 . ./common/dmflakey
 
@@ -37,11 +28,10 @@ _scratch_mount
 
 # Start a workload and shutdown the fs. The subsequent mount will require log
 # recovery.
-$FSSTRESS_PROG -n 9999 -p 2 -w -d $SCRATCH_MNT &>> $seqres.full &
+_run_fsstress_bg -n 9999 -p 2 -w -d $SCRATCH_MNT
 sleep 5
 _scratch_shutdown -f
-$KILLALL_PROG -q $FSSTRESS_PROG
-wait
+_kill_fsstress
 _scratch_unmount
 
 # Initialize a dm-flakey device that will pass I/Os for 5s and fail thereafter.
tests/xfs/057 +4 -4
index f1c947795..62eb8b93c 100755
@@ -26,12 +26,12 @@ _begin_fstest auto log recoveryloop
 # Override the default cleanup function.
 _cleanup()
 {
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
 	[ -e /sys/fs/xfs/$sdev/errortag/log_item_pin ] &&
 		echo 0 > /sys/fs/xfs/$sdev/errortag/log_item_pin
+	_kill_fsstress
 	wait > /dev/null 2>&1
+	cd /
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -53,7 +53,7 @@ _scratch_mkfs_sized $((1024 * 1024 * 500)) >> $seqres.full 2>&1
 _scratch_mount
 
 # populate the fs with some data and cycle the mount to reset the log head/tail
-$FSSTRESS_PROG -d $SCRATCH_MNT -z -fcreat=1 -p 4 -n 100000 >> $seqres.full
+_run_fsstress -d $SCRATCH_MNT -z -fcreat=1 -p 4 -n 100000
 _scratch_cycle_mount || _fail "cycle mount failed"
 
 # Pin the tail and start a file removal workload. File removal tends to
tests/xfs/077 +1 -1
index 4a4ac4702..3d8ecd9f6 100755
@@ -54,7 +54,7 @@ ORIG_UUID=`_scratch_xfs_db -c "uuid" | awk '{print $NF}'`
 
 _scratch_mount
 # Put some stuff on the fs
-$FSSTRESS_PROG -d $SCRATCH_MNT -n 100 -p 4 >> $seqres.full 2>&1
+_run_fsstress -d $SCRATCH_MNT -n 100 -p 4
 _scratch_unmount
 
 # Can xfs_db change it?
tests/xfs/079 +2 -12
index 46a15ed78..794d2db49 100755
@@ -17,19 +17,9 @@
 . ./common/preamble
 _begin_fstest shutdown auto log quick
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-	wait > /dev/null 2>&1
-}
-
 # Import common functions.
 . ./common/log
 
-
 # Modify as appropriate.
 _require_scratch
 _require_v2log
@@ -43,10 +33,10 @@ _scratch_mkfs >> $seqres.full 2>&1 || _fail "mkfs failed"
 _scratch_mount "-o logbsize=32k"
 
 # Run a workload to dirty the log, wait a bit and shutdown the fs.
-$FSSTRESS_PROG -d $SCRATCH_MNT -p 4 -n 99999999 >> $seqres.full 2>&1 &
+_run_fsstress_bg -d $SCRATCH_MNT -p 4 -n 99999999
 sleep 10
 _scratch_shutdown -f
-wait
+_wait_for_fsstress
 
 # Remount with a different log buffer size. Going from 32k to 64k increases the
 # log record extended header count, as the log record header can only handle 32k
tests/xfs/104 +4 -10
index 7f11f89a5..cd625d6b7 100755
@@ -40,9 +40,8 @@ _stress_scratch()
 	procs=3
 	nops=1000
 	# -w ensures that the only ops are ones which cause write I/O
-	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
-	    -n $nops $FSSTRESS_AVOID`
-	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 &
+	args=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs -n $nops`
+	_run_fsstress_bg $args
 }
 
 _require_scratch
@@ -73,14 +72,11 @@ for i in `seq 125 -1 90`; do
 	break
 done
 
-#
 # Grow the filesystem while actively stressing it...
-# Kick off more stress threads on each iteration, grow; repeat.
-#
 while [ $size -le $endsize ]; do
 	echo "*** stressing filesystem"
 	echo "*** stressing a ${sizeb} block filesystem" >> $seqres.full
-	_stress_scratch
+	_stress_scratch 
 	sleep 1
 	size=`expr $size + $incsize`
 	sizeb=`expr $size / $dbsize`	# in data blocks
@@ -92,10 +88,8 @@ while [ $size -le $endsize ]; do
 	[ `expr $size % $modsize` -eq 0 ] && wait	# every 4th iteration
 	echo AGCOUNT=$agcount | tee -a $seqres.full
 	echo && echo >> $seqres.full
+	_wait_for_fsstress
 done
-wait	# stop for any remaining stress processes
-
-_scratch_unmount
 
 status=0
 exit
tests/xfs/137 +1 -1
index dfc653573..d97942bf6 100755
@@ -29,7 +29,7 @@ _scratch_xfs_db -x -c "logformat -c 3" >> $seqres.full 2>&1
 
 # do some work on the fs to update metadata LSNs
 _scratch_mount
-$FSSTRESS_PROG -d $SCRATCH_MNT -n 999 -p 4 -w >> $seqres.full 2>&1
+_run_fsstress -d $SCRATCH_MNT -n 999 -p 4 -w
 _scratch_unmount
 
 # Reformat to the current cycle and try to mount. This fails in most cases
tests/xfs/141 +1 -11
index 5e9067e24..b630ba10d 100755
@@ -14,19 +14,9 @@
 . ./common/preamble
 _begin_fstest auto log metadata
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-	wait > /dev/null 2>&1
-}
-
 # Import common functions.
 . ./common/inject
 
-
 # Modify as appropriate.
 _require_xfs_io_error_injection "log_bad_crc"
 _require_scratch
@@ -49,7 +39,7 @@ while [ $nr_times -gt 0 ]; do
 
 	# Run fsstress until the filesystem shuts down. It will shut down
 	# automatically when error injection triggers.
-	$FSSTRESS_PROG -d $SCRATCH_MNT -p 4 -n 999999 >> $seqres.full 2>&1
+	_run_fsstress -d $SCRATCH_MNT -p 4 -n 999999
 
 	# Verify that we can remount the fs. Log recovery should handle the torn
 	# write.
tests/xfs/158 +2 -2
index 3c4e60f0e..89bf8c851 100755
@@ -31,7 +31,7 @@ _scratch_mkfs -m crc=1,inobtcount=0 >> $seqres.full
 _scratch_mount
 
 mkdir $SCRATCH_MNT/stress
-$FSSTRESS_PROG -d $SCRATCH_MNT/stress -n 1000 >> $seqres.full
+_run_fsstress -d $SCRATCH_MNT/stress -n 1000
 echo moo > $SCRATCH_MNT/urk
 
 _scratch_unmount
@@ -56,7 +56,7 @@ _check_scratch_xfs_features NEEDSREPAIR INOBTCNT
 
 echo "Filesystem should be usable again"
 _scratch_mount
-$FSSTRESS_PROG -d $SCRATCH_MNT/stress -n 1000 >> $seqres.full
+_run_fsstress -d $SCRATCH_MNT/stress -n 1000
 _scratch_unmount
 _check_scratch_fs
 _check_scratch_xfs_features INOBTCNT
tests/xfs/167 +4 -10
index f9da261db..5ef2aa2ea 100755
@@ -9,20 +9,12 @@
 . ./common/preamble
 _begin_fstest rw metadata auto stress prealloc
 
-# Override the default cleanup function.
-_cleanup()
-{
-	$KILLALL_PROG -r -q -TERM fsstress 2> /dev/null
-	wait	# ensures all fsstress processes died
-}
-
 workout()
 {
 	procs=100
 	nops=15000
-	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -p $procs -n $nops \
-	    $FSSTRESS_AVOID`
-	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full &
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -p $procs -n $nops`
+	_run_fsstress_bg $FSSTRESS_ARGS
 	sleep 2
 }
 
@@ -52,6 +44,8 @@ rm -f $TEST_FILE
 workout
 $TEST_PROG $LOOPS $TEST_FILE
 
+_kill_fsstress
+
 echo "     *** test done"
 
 status=0
tests/xfs/168 +4 -4
index f187a336f..098e0c86a 100755
@@ -38,9 +38,8 @@ stress_scratch()
 	local procs=3
 	local nops=1000
 	# -w ensures that the only ops are ones which cause write I/O
-	local FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w \
-		-p $procs -n $nops $FSSTRESS_AVOID`
-	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1
+	local args=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs -n $nops`
+	_run_fsstress_bg $args
 }
 
 _require_scratch_xfs_shrink
@@ -73,7 +72,7 @@ while [ $totalcount -gt 0 ]; do
 	decsize=`expr  41 \* 1048576 + 1 + $RANDOM \* $RANDOM % 1048576`
 
 	while [ $size -gt $endsize ]; do
-		stress_scratch &
+		stress_scratch
 		sleep 1
 
 		decb=`expr $decsize / $dbsize`    # in data blocks
@@ -95,6 +94,7 @@ while [ $totalcount -gt 0 ]; do
 		. $tmp.growfs
 
 		size=`expr $dblocks \* $dbsize`
+		_kill_fsstress
 		_scratch_unmount
 		_scratch_xfs_repair -n >> $seqres.full 2>&1 || \
 			_fail "xfs_repair failed with shrinking $sizeb"
tests/xfs/264 +2 -2
index 109fecd1c..a6e816d3c 100755
@@ -55,7 +55,7 @@ do_test()
 	# start a metadata-intensive workload, but no data allocation operation.
 	# Because uncompleted new space allocation I/Os may cause XFS to shutdown
 	# after loading error table.
-	$FSSTRESS_PROG -z -n 5000 -p 10 \
+	_run_fsstress -z -n 5000 -p 10 \
 		       -f creat=10 \
 		       -f resvsp=1 \
 		       -f truncate=1 \
@@ -67,7 +67,7 @@ do_test()
 		       -f unlink=1 \
 		       -f symlink=1 \
 		       -f rename=1 \
-		       -d $SCRATCH_MNT/fsstress >> $seqres.full 2>&1
+		       -d $SCRATCH_MNT/fsstress
 
 	# Loading error table without "--nolockfs" option. Because "--nolockfs"
 	# won't freeze fs, then some running I/Os may cause XFS to shutdown
tests/xfs/270 +1 -1
index 3744df5a9..d3bce386a 100755
@@ -80,7 +80,7 @@ if [ $? -ne 0 ]; then
 else
 	# no hang/panic is fine
 	cat $SCRATCH_MNT/testfile > /dev/null
-	$FSSTRESS_PROG -d $SCRATCH_MNT -p 4 -n 400 >>$seqres.full 2>&1
+	_run_fsstress -d $SCRATCH_MNT -p 4 -n 400
 fi
 
 # remount as rw, kernel should reject it
tests/xfs/297 +3 -5
index 2c5b03c5c..66c5d0cc7 100755
@@ -16,8 +16,7 @@ _cleanup()
 {
 	# Make sure $SCRATCH_MNT is unfreezed
 	xfs_freeze -u $SCRATCH_MNT 2>/dev/null
-	$KILLALL_PROG -q -9 $FSSTRESS_PROG
-	wait
+	_kill_fsstress
 	cd /
 	rm -f $tmp.*
 }
@@ -37,7 +36,7 @@ _scratch_mount
 STRESS_DIR="$SCRATCH_MNT/testdir"
 mkdir -p $STRESS_DIR
 
-$FSSTRESS_PROG -d $STRESS_DIR -n 100 -p 1000 $FSSTRESS_AVOID >>$seqres.full &
+_run_fsstress_bg -d $STRESS_DIR -n 1000 -p 1000 $FSSTRESS_AVOID
 
 # Freeze/unfreeze file system randomly
 echo "Start freeze/unfreeze randomly" | tee -a $seqres.full
@@ -60,8 +59,7 @@ while [ $LOOP -gt 0 ];do
 	let LOOP=$LOOP-1
 done
 echo "Test done" | tee -a $seqres.full
-$KILLALL_PROG -q $FSSTRESS_PROG
-wait
+_kill_fsstress
 
 status=0
 exit
tests/xfs/305 +3 -5
index 0ad3ef7fb..6371ed8a6 100755
@@ -4,7 +4,7 @@
 #
 # FS QA Test No. 305
 #
-# Test to verify that turn group/project quotas off while fstress and
+# Test to verify that turn group/project quotas off while fsstress and
 # user quotas are left on.
 #
 . ./common/preamble
@@ -18,7 +18,6 @@ _begin_fstest auto quota
 
 _require_scratch
 _require_xfs_quota
-_require_command "$KILLALL_PROG" killall
 
 _scratch_mkfs_xfs -m crc=1 >/dev/null 2>&1
 
@@ -33,12 +32,11 @@ _exercise()
 	_qmount
 	mkdir -p $QUOTA_DIR
 
-	$FSSTRESS_PROG -d $QUOTA_DIR -n 1000000 -p 100 $FSSTRESS_AVOID >>$seqres.full &
+	_run_fsstress_bg -d $QUOTA_DIR -n 1000000 -p 100
 	sleep 10
 	$XFS_QUOTA_PROG -x -c "disable -$type" $SCRATCH_DEV
 	sleep 5
-	$KILLALL_PROG -q $FSSTRESS_PROG
-	wait
+	_kill_fsstress
 }
 
 echo "*** turn off group quotas"
tests/xfs/442 +1 -9
index 77d08fda5..5cbd8dd19 100755
@@ -12,14 +12,6 @@
 . ./common/preamble
 _begin_fstest auto stress clone quota
 
-# Override the default cleanup function.
-_cleanup()
-{
-	cd /
-	rm -f $tmp.*
-	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-}
-
 # Import common functions.
 . ./common/quota
 . ./common/filter
@@ -72,7 +64,7 @@ _scratch_mount >> $seqres.full 2>&1
 
 nr_cpus=$((LOAD_FACTOR * 4))
 nr_ops=$((25000 * nr_cpus * TIME_FACTOR))
-$FSSTRESS_PROG $FSSTRESS_AVOID -w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus >> $seqres.full
+_run_fsstress -w -d $SCRATCH_MNT -n $nr_ops -p $nr_cpus
 
 echo "Check quota before remount"
 check_quota_du_blocks
tests/xfs/538 +2 -2
index 57113d341..f858cadc3 100755
@@ -46,7 +46,7 @@ echo "Inject bmap_alloc_minlen_extent error tag"
 _scratch_inject_error bmap_alloc_minlen_extent 1
 
 echo "Execute fsstress"
-$FSSTRESS_PROG -d $SCRATCH_MNT \
+_run_fsstress -d $SCRATCH_MNT \
 		$(_scale_fsstress_args -p 75 -n 1000) \
 		-f bulkstat=0 \
 		-f bulkstat1=0 \
@@ -61,7 +61,7 @@ $FSSTRESS_PROG -d $SCRATCH_MNT \
 		-f readv=0 \
 		-f stat=0 \
 		-f aread=0 \
-		-f dread=0 >> $seqres.full
+		-f dread=0
 
 # success, all done
 status=0
tests/xfs/609 +3 -17
index c23b6893d..88dc3c683 100755
@@ -18,21 +18,12 @@ _stress_scratch()
 	nops=999999
 	# -w ensures that the only ops are ones which cause write I/O
 	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
-	    -n $nops $FSSTRESS_AVOID`
-	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 &
+	    -n $nops`
+	_run_fsstress_bg $FSSTRESS_ARGS >> $seqres.full 2>&1
 }
 
 _require_scratch
 _require_command "$XFS_GROWFS_PROG" xfs_growfs
-_require_command "$KILLALL_PROG" killall
-
-_cleanup()
-{
-	$KILLALL_ALL fsstress > /dev/null 2>&1
-	wait
-	cd /
-	rm -f $tmp.*
-}
 
 _scratch_mkfs_xfs | _filter_mkfs >$seqres.full 2>$tmp.mkfs
 . $tmp.mkfs	# extract blocksize and data size for scratch device
@@ -63,12 +54,7 @@ while [ $size -le $endsize ]; do
 
 	sleep $((RANDOM % 3))
 	_scratch_shutdown
-	ps -e | grep fsstress > /dev/null 2>&1
-	while [ $? -eq 0 ]; do
-		$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-		wait > /dev/null 2>&1
-		ps -e | grep fsstress > /dev/null 2>&1
-	done
+	_kill_fsstress
 	_scratch_cycle_mount
 done > /dev/null 2>&1
 wait	# stop for any remaining stress processes
tests/xfs/610 +3 -17
index 23fbd8585..8610b912c 100755
@@ -18,22 +18,13 @@ _stress_scratch()
 	nops=999999
 	# -w ensures that the only ops are ones which cause write I/O
 	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
-	    -n $nops $FSSTRESS_AVOID`
-	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 &
+	    -n $nops`
+	_run_fsstress_bg $FSSTRESS_ARGS >> $seqres.full 2>&1
 }
 
 _require_scratch
 _require_realtime
 _require_command "$XFS_GROWFS_PROG" xfs_growfs
-_require_command "$KILLALL_PROG" killall
-
-_cleanup()
-{
-	$KILLALL_ALL fsstress > /dev/null 2>&1
-	wait
-	cd /
-	rm -f $tmp.*
-}
 
 _scratch_mkfs_xfs | _filter_mkfs >$seqres.full 2>$tmp.mkfs
 . $tmp.mkfs	# extract blocksize and data size for scratch device
@@ -65,12 +56,7 @@ while [ $size -le $endsize ]; do
 
 	sleep $((RANDOM % 3))
 	_scratch_shutdown
-	ps -e | grep fsstress > /dev/null 2>&1
-	while [ $? -eq 0 ]; do
-		$KILLALL_PROG -9 fsstress > /dev/null 2>&1
-		wait > /dev/null 2>&1
-		ps -e | grep fsstress > /dev/null 2>&1
-	done
+	_kill_fsstress
 	_scratch_cycle_mount
 done > /dev/null 2>&1
 wait	# stop for any remaining stress processes
Signature · 1 line
2.45.2
#8 lore raw

Re: [PATCH 02/40] fstests: cleanup fsstress process management

Quoted 38 lines · Zorro Lang
Quoted 24 lines · Dave Chinner
tests/generic/561 +1 -2
index 39e5977a3..3e931b1a7 100755
@@ -13,9 +13,9 @@ _begin_fstest auto stress dedupe
 # Override the default cleanup function.
 _cleanup()
 {
+	end_test
 	cd /
 	rm -f $tmp.*
-	end_test
 }
 
 # Import common functions.
@@ -23,28 +23,20 @@ _cleanup()
 . ./common/reflink
 
 _require_scratch_duperemove
-_require_command "$KILLALL_PROG" killall
 
 _scratch_mkfs > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 

tests/generic/561 _cleanup() · line 37

Quoted 10 lines
tests/generic/561 +2 -2
 function end_test()
 {
-	local f=1
+	_kill_fsstress
 
 	# stop duperemove running
 	if [ -e $dupe_run ]; then
 		rm -f $dupe_run
-		$KILLALL_PROG -q $DUPEREMOVE_PROG > /dev/null 2>&1
+		kill $dedup_pids

Hi Dave,

Quoted 4 lines

The $dedup_pids is a "while loop" bash process, it isn't the $DUPEREMOVE_PROG process itself. From my testing, this change might cause g/561 keep waiting $DUPEREMOVE_PROG processes forever, as $DUPEREMOVE_PROG not always be killed properly.

I have another patch that I didn't send that reworks duperemove killing. I didn't sent it because I ended up simply turning the test off via adding it to the unreliable_in_parallel group because it had a 75% failure rate due to duperemove bugs (e.g. getting stuck in fiemap loops that never terminate)...

Quoted 3 lines

I'm wondering if you hope to do "pkill $DUPEREMOVE_PROG" directly? Or you'd like to copy $DUPEREMOVE_PROG to $TEST_DIR/${othername}_duperemove, then pkill ${othername}_duperemove ?

I came up with the "copy to $seq.binary then pkill $seq.binary" idea after I gave up on this test as fundamentally broken. I'll revisit it now that we have a reliable way of killing only the processes belonging to the specific test. That might make this test reliable enough to run concurrently, but I have my doubts...

I'll have another look at it.

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#9 lore raw

Re: [PATCH 02/40] fstests: cleanup fsstress process management

Quoted 44 lines · Dave Chinner
Quoted 38 lines · Zorro Lang
Quoted 24 lines · Dave Chinner
tests/generic/561 +1 -2
index 39e5977a3..3e931b1a7 100755
@@ -13,9 +13,9 @@ _begin_fstest auto stress dedupe
 # Override the default cleanup function.
 _cleanup()
 {
+	end_test
 	cd /
 	rm -f $tmp.*
-	end_test
 }
 
 # Import common functions.
@@ -23,28 +23,20 @@ _cleanup()
 . ./common/reflink
 
 _require_scratch_duperemove
-_require_command "$KILLALL_PROG" killall
 
 _scratch_mkfs > $seqres.full 2>&1
 _scratch_mount >> $seqres.full 2>&1
 

tests/generic/561 _cleanup() · line 37

Quoted 10 lines
tests/generic/561 +2 -2
 function end_test()
 {
-	local f=1
+	_kill_fsstress
 
 	# stop duperemove running
 	if [ -e $dupe_run ]; then
 		rm -f $dupe_run
-		$KILLALL_PROG -q $DUPEREMOVE_PROG > /dev/null 2>&1
+		kill $dedup_pids

Hi Dave,

Quoted 4 lines

The $dedup_pids is a "while loop" bash process, it isn't the $DUPEREMOVE_PROG process itself. From my testing, this change might cause g/561 keep waiting $DUPEREMOVE_PROG processes forever, as $DUPEREMOVE_PROG not always be killed properly.

Quoted 5 lines

I have another patch that I didn't send that reworks duperemove killing. I didn't sent it because I ended up simply turning the test off via adding it to the unreliable_in_parallel group because it had a 75% failure rate due to duperemove bugs (e.g. getting stuck in fiemap loops that never terminate)...

Great to know you've noticed that :)

Quoted 5 lines
Quoted 4 lines

I'm wondering if you hope to do "pkill $DUPEREMOVE_PROG" directly? Or you'd like to copy $DUPEREMOVE_PROG to $TEST_DIR/${othername}_duperemove, then pkill ${othername}_duperemove ?

Quoted 7 lines

I came up with the "copy to $seq.binary then pkill $seq.binary" idea after I gave up on this test as fundamentally broken. I'll revisit it now that we have a reliable way of killing only the processes belonging to the specific test. That might make this test reliable enough to run concurrently, but I have my doubts...

I'll have another look at it.

Currently only generic/561 runs duperemove in background bash, if you don't have time to change it this week, I can help to change it to use the "copy to $seq.binary then pkill $seq.binary" way temporarily, to avoid a test blocker. Due to I want to merge your patchset ASAP in this week. Then we can move on. I don't want to let you or others do much rebase jobs :)

Thanks,
Zorro

Quoted 6 lines

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#10 lore raw

PATCH 3/40 fuzzy: don't use killall

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Having test cleanup call 'killall xfs_io fsx xfs_scrub' results in a system wide process kill, rather than just the processes the test is running directly.

Make sure we only kill processes the fuzz test directly owns. We can do this with 'pkill --parent $$ <process names>' to limit the search for processes to kill to just the children of the current process.

---
 common/fuzzy | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
common/fuzzy +3 -2
index 9181520a3..3a7f04aae 100644
@@ -1145,7 +1145,6 @@ __stress_scrub_fsstress_loop() {
 _require_xfs_stress_scrub() {
 	_require_xfs_io_command "scrub"
 	_require_test_program "xfsfind"
-	_require_command "$KILLALL_PROG" killall
 	_require_freeze
 	command -v _filter_scratch &>/dev/null || \
 		_notrun 'xfs scrub stress test requires common/filter'
@@ -1178,7 +1177,9 @@ _scratch_xfs_stress_scrub_cleanup() {
 	# distorts the golden output.
 	echo "Killing stressor processes at $(date)" >> $seqres.full
 	_kill_fsstress
-	$KILLALL_PROG -INT xfs_io fsx xfs_scrub >> $seqres.full 2>&1
+	pkill -INT --parent $$ xfs_io >> $seqres.full 2>&1
+	pkill -INT --parent $$ fsx >> $seqres.full 2>&1
+	pkill -INT --parent $$ xfs_scrub >> $seqres.full 2>&1
 
 	# Tests are not allowed to exit with the scratch fs frozen.  If we
 	# started a fs freeze/thaw background loop, wait for that loop to exit
Signature · 1 line
2.45.2
#11 lore raw

PATCH 4/40 fstests: per-test dmflakey instances

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

We can't run two tests that use dmflakey at the same time because the device name is the same. hence they interfere with each other. Given dmflakey devices their own per-test names to avoid this problem.

Also, drop_and_remount is about to fail the fs during unmount, so ensure the filesystem is going to fail the IO during unmount rather than retrying forever.

---
 common/dmflakey | 56 ++++++++++++++++++++++++++++---------------------
 1 file changed, 32 insertions(+), 24 deletions(-)
common/dmflakey +32 -24
index 52da3b100..75a0a67c8 100644
@@ -8,15 +8,19 @@ FLAKEY_ALLOW_WRITES=0
 FLAKEY_DROP_WRITES=1
 FLAKEY_ERROR_WRITES=2
 
+export FLAKEY_NAME="flakey-test.$seq"
+export FLAKEY_RTNAME="flakey-rttest.$seq"
+export FLAKEY_LOGNAME="flakey-logtest.$seq"
+
 _init_flakey()
 {
 	# Scratch device
 	local BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
-	FLAKEY_DEV=/dev/mapper/flakey-test
+	export FLAKEY_DEV="/dev/mapper/$FLAKEY_NAME"
 	FLAKEY_TABLE="0 $BLK_DEV_SIZE flakey $SCRATCH_DEV 0 180 0"
 	FLAKEY_TABLE_DROP="0 $BLK_DEV_SIZE flakey $SCRATCH_DEV 0 0 180 1 drop_writes"
 	FLAKEY_TABLE_ERROR="0 $BLK_DEV_SIZE flakey $SCRATCH_DEV 0 0 180 1 error_writes"
-	_dmsetup_create flakey-test --table "$FLAKEY_TABLE" || \
+	_dmsetup_create $FLAKEY_NAME --table "$FLAKEY_TABLE" || \
 		_fatal "failed to create flakey device"
 
 	# Realtime device
@@ -25,7 +29,7 @@ _init_flakey()
 			# Set up the device switch
 			local backing_dev="$SCRATCH_RTDEV"
 			export NON_FLAKEY_RTDEV="$SCRATCH_RTDEV"
-			SCRATCH_RTDEV=/dev/mapper/flakey-rttest
+			SCRATCH_RTDEV=/dev/mapper/$FLAKEY_RTNAME
 		else
 			# Already set up; recreate tables
 			local backing_dev="$NON_FLAKEY_RTDEV"
@@ -34,7 +38,7 @@ _init_flakey()
 		FLAKEY_RTTABLE="0 $BLK_DEV_SIZE flakey $backing_dev 0 180 0"
 		FLAKEY_RTTABLE_DROP="0 $BLK_DEV_SIZE flakey $backing_dev 0 0 180 1 drop_writes"
 		FLAKEY_RTTABLE_ERROR="0 $BLK_DEV_SIZE flakey $backing_dev 0 0 180 1 error_writes"
-		_dmsetup_create flakey-rttest --table "$FLAKEY_RTTABLE" || \
+		_dmsetup_create $FLAKEY_RTNAME --table "$FLAKEY_RTTABLE" || \
 			_fatal "failed to create flakey rt device"
 	fi
 
@@ -44,7 +48,7 @@ _init_flakey()
 			# Set up the device switch
 			local backing_dev="$SCRATCH_LOGDEV"
 			export NON_FLAKEY_LOGDEV="$SCRATCH_LOGDEV"
-			SCRATCH_LOGDEV=/dev/mapper/flakey-logtest
+			SCRATCH_LOGDEV=/dev/mapper/$FLAKEY_LOGNAME
 		else
 			# Already set up; recreate tables
 			local backing_dev="$NON_FLAKEY_LOGDEV"
@@ -53,7 +57,7 @@ _init_flakey()
 		FLAKEY_LOGTABLE="0 $BLK_DEV_SIZE flakey $backing_dev 0 180 0"
 		FLAKEY_LOGTABLE_DROP="0 $BLK_DEV_SIZE flakey $backing_dev 0 0 180 1 drop_writes"
 		FLAKEY_LOGTABLE_ERROR="0 $BLK_DEV_SIZE flakey $backing_dev 0 0 180 1 error_writes"
-		_dmsetup_create flakey-logtest --table "$FLAKEY_LOGTABLE" || \
+		_dmsetup_create $FLAKEY_LOGNAME --table "$FLAKEY_LOGTABLE" || \
 			_fatal "failed to create flakey log device"
 	fi
 }
@@ -74,15 +78,15 @@ _cleanup_flakey()
 {
 	# If dmsetup load fails then we need to make sure to do resume here
 	# otherwise the umount will hang
-	test -n "$NON_FLAKEY_LOGDEV" && $DMSETUP_PROG resume flakey-logtest &> /dev/null
-	test -n "$NON_FLAKEY_RTDEV" && $DMSETUP_PROG resume flakey-rttest &> /dev/null
+	test -n "$NON_FLAKEY_LOGDEV" && $DMSETUP_PROG resume $FLAKEY_LOGNAME &> /dev/null
+	test -n "$NON_FLAKEY_RTDEV" && $DMSETUP_PROG resume $FLAKEY_RTNAME &> /dev/null
 	$DMSETUP_PROG resume flakey-test > /dev/null 2>&1
 
 	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
 
-	_dmsetup_remove flakey-test
-	test -n "$NON_FLAKEY_LOGDEV" && _dmsetup_remove flakey-logtest
-	test -n "$NON_FLAKEY_RTDEV" && _dmsetup_remove flakey-rttest
+	_dmsetup_remove $FLAKEY_NAME
+	test -n "$NON_FLAKEY_LOGDEV" && _dmsetup_remove $FLAKEY_LOGNAME
+	test -n "$NON_FLAKEY_RTDEV" && _dmsetup_remove $FLAKEY_RTNAME
 
 	if [ -n "$NON_FLAKEY_LOGDEV" ]; then
 		SCRATCH_LOGDEV="$NON_FLAKEY_LOGDEV"
@@ -125,54 +129,58 @@ _load_flakey_table()
 	# Suspend the scratch device before the log and realtime devices so
 	# that the kernel can freeze and flush the filesystem if the caller
 	# wanted a freeze.
-	$DMSETUP_PROG suspend $suspend_opt flakey-test
-	[ $? -ne 0 ] && _fatal "failed to suspend flakey-test"
+	$DMSETUP_PROG suspend $suspend_opt $FLAKEY_NAME
+	[ $? -ne 0 ] && _fatal "failed to suspend $FLAKEY_NAME"
 
 	if [ -n "$NON_FLAKEY_RTDEV" ]; then
-		$DMSETUP_PROG suspend $suspend_opt flakey-rttest
+		$DMSETUP_PROG suspend $suspend_opt $FLAKEY_RTNAME
 		[ $? -ne 0 ] && _fatal "failed to suspend flakey-rttest"
 	fi
 
 	if [ -n "$NON_FLAKEY_LOGDEV" ]; then
-		$DMSETUP_PROG suspend $suspend_opt flakey-logtest
+		$DMSETUP_PROG suspend $suspend_opt $FLAKEY_LOGNAME
 		[ $? -ne 0 ] && _fatal "failed to suspend flakey-logtest"
 	fi
 
 	# There may be multiple dm targets in the table, and these dm targets
 	# will be joined by the newline ("\n"). Option --table can not cope with
 	# the multiple-targets case, so get them by reading from standard input.
-	echo -e "$table" | $DMSETUP_PROG load flakey-test
-	[ $? -ne 0 ] && _fatal "failed to load table into flakey-test"
+	echo -e "$table" | $DMSETUP_PROG load $FLAKEY_NAME
+	[ $? -ne 0 ] && _fatal "failed to load table into $FLAKEY_NAME"
 
 	if [ -n "$NON_FLAKEY_RTDEV" ]; then
-		echo -e "$rttable" | $DMSETUP_PROG load flakey-rttest
+		echo -e "$rttable" | $DMSETUP_PROG load $FLAKEY_RTNAME
 		[ $? -ne 0 ] && _fatal "failed to load table into flakey-rttest"
 	fi
 
 	if [ -n "$NON_FLAKEY_LOGDEV" ]; then
-		echo -e "$logtable" | $DMSETUP_PROG load flakey-logtest
+		echo -e "$logtable" | $DMSETUP_PROG load $FLAKEY_LOGNAME
 		[ $? -ne 0 ] && _fatal "failed to load table into flakey-logtest"
 	fi
 
 	# Resume devices in the opposite order that we suspended them.
 	if [ -n "$NON_FLAKEY_LOGDEV" ]; then
-		$DMSETUP_PROG resume flakey-logtest
+		$DMSETUP_PROG resume $FLAKEY_LOGNAME
 		[ $? -ne 0 ] && _fatal  "failed to resume flakey-logtest"
 	fi
 
 	if [ -n "$NON_FLAKEY_RTDEV" ]; then
-		$DMSETUP_PROG resume flakey-rttest
+		$DMSETUP_PROG resume $FLAKEY_RTNAME
 		[ $? -ne 0 ] && _fatal  "failed to resume flakey-rttest"
 	fi
 
-	$DMSETUP_PROG resume flakey-test
-	[ $? -ne 0 ] && _fatal  "failed to resume flakey-test"
+	$DMSETUP_PROG resume $FLAKEY_NAME
+	[ $? -ne 0 ] && _fatal  "failed to resume $FLAKEY_NAME"
 }
 
 # Silently drop all writes and unmount/remount to simulate a crash/power
 # failure.
 _flakey_drop_and_remount()
 {
+	# If the full environment is set up, configure ourselves for shutdown
+	type _prepare_for_eio_shutdown &>/dev/null && \
+		_prepare_for_eio_shutdown $FLAKEY_DEV
+
 	_load_flakey_table $FLAKEY_DROP_WRITES
 	_unmount_flakey
 
@@ -188,7 +196,7 @@ _require_flakey_with_error_writes()
 {
 	local SIZE
 	local TABLE
-	local NAME=flakey-test
+	local NAME=$FLAKEY_NAME
 
 	_require_dm_target flakey
 
Signature · 1 line
2.45.2
#12 lore raw

PATCH 5/40 fstests: per-test dmerror instances

In reply to Dave Chinner

We can't run two tests that use dmerror at the same time because the device name is the same. hence they interfere with each other. Give dmerror devices their own per-test names to avoid this problem.

Note that we need a hack to pass the test sequence number through to src/dmerror as used by generic/441 so that it can construct the dmerror name correctly.

---
 common/dmerror    | 70 +++++++++++++++++++++++++----------------------
 src/dmerror       |  6 +++-
 tests/generic/441 |  2 +-
 3 files changed, 43 insertions(+), 35 deletions(-)
common/dmerror +37 -33
index 3494b6dd3..97796b38e 100644
@@ -4,6 +4,10 @@
 #
 # common functions for setting up and tearing down a dmerror device
 
+export DMERROR_NAME="error-test.$seq"
+export DMERROR_RTNAME="error-rttest.$seq"
+export DMERROR_LOGNAME="error-logtest.$seq"
+
 _dmerror_setup_vars()
 {
 	local backing_dev="$1"
@@ -31,7 +35,7 @@ _dmerror_setup()
 	done
 
 	# Scratch device
-	export DMERROR_DEV='/dev/mapper/error-test'
+	export DMERROR_DEV="/dev/mapper/$DMERROR_NAME"
 	_dmerror_setup_vars $SCRATCH_DEV
 
 	# Realtime device.  We reassign SCRATCH_RTDEV so that all the scratch
@@ -41,7 +45,7 @@ _dmerror_setup()
 			# Set up the device switch
 			local dm_backing_dev=$SCRATCH_RTDEV
 			export NON_ERROR_RTDEV="$SCRATCH_RTDEV"
-			SCRATCH_RTDEV='/dev/mapper/error-rttest'
+			SCRATCH_RTDEV="/dev/mapper/$DMERROR_RTNAME"
 		else
 			# Already set up; recreate tables
 			local dm_backing_dev="$NON_ERROR_RTDEV"
@@ -57,7 +61,7 @@ _dmerror_setup()
 			# Set up the device switch
 			local dm_backing_dev=$SCRATCH_LOGDEV
 			export NON_ERROR_LOGDEV="$SCRATCH_LOGDEV"
-			SCRATCH_LOGDEV='/dev/mapper/error-logtest'
+			SCRATCH_LOGDEV="/dev/mapper/$DMERROR_LOGNAME"
 		else
 			# Already set up; recreate tables
 			local dm_backing_dev="$NON_ERROR_LOGDEV"
@@ -71,19 +75,19 @@ _dmerror_init()
 {
 	_dmerror_setup "$@"
 
-	_dmsetup_remove error-test
-	_dmsetup_create error-test --table "$DMLINEAR_TABLE" || \
+	_dmsetup_remove $DMERROR_NAME
+	_dmsetup_create $DMERROR_NAME --table "$DMLINEAR_TABLE" || \
 		_fatal "failed to create dm linear device"
 
 	if [ -n "$NON_ERROR_RTDEV" ]; then
-		_dmsetup_remove error-rttest
-		_dmsetup_create error-rttest --table "$DMLINEAR_RTTABLE" || \
+		_dmsetup_remove $DMERROR_RTNAME
+		_dmsetup_create $DMERROR_RTNAME --table "$DMLINEAR_RTTABLE" || \
 			_fatal "failed to create dm linear rt device"
 	fi
 
 	if [ -n "$NON_ERROR_LOGDEV" ]; then
-		_dmsetup_remove error-logtest
-		_dmsetup_create error-logtest --table "$DMLINEAR_LOGTABLE" || \
+		_dmsetup_remove $DMERROR_LOGNAME
+		_dmsetup_create $DMERROR_LOGNAME --table "$DMLINEAR_LOGTABLE" || \
 			_fatal "failed to create dm linear log device"
 	fi
 }
@@ -102,15 +106,15 @@ _dmerror_unmount()
 
 _dmerror_cleanup()
 {
-	test -n "$NON_ERROR_LOGDEV" && $DMSETUP_PROG resume error-logtest &>/dev/null
-	test -n "$NON_ERROR_RTDEV" && $DMSETUP_PROG resume error-rttest &>/dev/null
-	$DMSETUP_PROG resume error-test > /dev/null 2>&1
+	test -n "$NON_ERROR_LOGDEV" && $DMSETUP_PROG resume $DMERROR_LOGNAME &>/dev/null
+	test -n "$NON_ERROR_RTDEV" && $DMSETUP_PROG resume $DMERROR_RTNAME &>/dev/null
+	$DMSETUP_PROG resume $DMERROR_NAME > /dev/null 2>&1
 
 	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
 
-	test -n "$NON_ERROR_LOGDEV" && _dmsetup_remove error-logtest
-	test -n "$NON_ERROR_RTDEV" && _dmsetup_remove error-rttest
-	_dmsetup_remove error-test
+	test -n "$NON_ERROR_LOGDEV" && _dmsetup_remove $DMERROR_LOGNAME
+	test -n "$NON_ERROR_RTDEV" && _dmsetup_remove $DMERROR_RTNAME
+	_dmsetup_remove $DMERROR_NAME
 
 	unset DMERROR_DEV DMLINEAR_TABLE DMERROR_TABLE
 
@@ -145,45 +149,45 @@ _dmerror_load_error_table()
 	# Suspend the scratch device before the log and realtime devices so
 	# that the kernel can freeze and flush the filesystem if the caller
 	# wanted a freeze.
-	$DMSETUP_PROG suspend $suspend_opt error-test
+	$DMSETUP_PROG suspend $suspend_opt $DMERROR_NAME
 	[ $? -ne 0 ] && _fail  "dmsetup suspend failed"
 
 	if [ -n "$NON_ERROR_RTDEV" ]; then
-		$DMSETUP_PROG suspend $suspend_opt error-rttest
+		$DMSETUP_PROG suspend $suspend_opt $DMERROR_RTNAME
 		[ $? -ne 0 ] && _fail "failed to suspend error-rttest"
 	fi
 
 	if [ -n "$NON_ERROR_LOGDEV" ]; then
-		$DMSETUP_PROG suspend $suspend_opt error-logtest
+		$DMSETUP_PROG suspend $suspend_opt $DMERROR_LOGNAME
 		[ $? -ne 0 ] && _fail "failed to suspend error-logtest"
 	fi
 
 	# Load new table
-	$DMSETUP_PROG load error-test --table "$DMERROR_TABLE"
+	$DMSETUP_PROG load $DMERROR_NAME --table "$DMERROR_TABLE"
 	load_res=$?
 
 	if [ -n "$NON_ERROR_RTDEV" ]; then
-		$DMSETUP_PROG load error-rttest --table "$DMERROR_RTTABLE"
+		$DMSETUP_PROG load $DMERROR_RTNAME --table "$DMERROR_RTTABLE"
 		[ $? -ne 0 ] && _fail "failed to load error table into error-rttest"
 	fi
 
 	if [ -n "$NON_ERROR_LOGDEV" ]; then
-		$DMSETUP_PROG load error-logtest --table "$DMERROR_LOGTABLE"
+		$DMSETUP_PROG load $DMERROR_LOGNAME --table "$DMERROR_LOGTABLE"
 		[ $? -ne 0 ] && _fail "failed to load error table into error-logtest"
 	fi
 
 	# Resume devices in the opposite order that we suspended them.
 	if [ -n "$NON_ERROR_LOGDEV" ]; then
-		$DMSETUP_PROG resume error-logtest
+		$DMSETUP_PROG resume $DMERROR_LOGNAME
 		[ $? -ne 0 ] && _fail  "failed to resume error-logtest"
 	fi
 
 	if [ -n "$NON_ERROR_RTDEV" ]; then
-		$DMSETUP_PROG resume error-rttest
+		$DMSETUP_PROG resume $DMERROR_RTNAME
 		[ $? -ne 0 ] && _fail  "failed to resume error-rttest"
 	fi
 
-	$DMSETUP_PROG resume error-test
+	$DMSETUP_PROG resume $DMERROR_NAME
 	resume_res=$?
 
 	[ $load_res -ne 0 ] && _fail "dmsetup failed to load error table"
@@ -206,45 +210,45 @@ _dmerror_load_working_table()
 	# Suspend the scratch device before the log and realtime devices so
 	# that the kernel can freeze and flush the filesystem if the caller
 	# wanted a freeze.
-	$DMSETUP_PROG suspend $suspend_opt error-test
+	$DMSETUP_PROG suspend $suspend_opt $DMERROR_NAME
 	[ $? -ne 0 ] && _fail  "dmsetup suspend failed"
 
 	if [ -n "$NON_ERROR_RTDEV" ]; then
-		$DMSETUP_PROG suspend $suspend_opt error-rttest
+		$DMSETUP_PROG suspend $suspend_opt $DMERROR_RTNAME
 		[ $? -ne 0 ] && _fail "failed to suspend error-rttest"
 	fi
 
 	if [ -n "$NON_ERROR_LOGDEV" ]; then
-		$DMSETUP_PROG suspend $suspend_opt error-logtest
+		$DMSETUP_PROG suspend $suspend_opt $DMERROR_LOGNAME
 		[ $? -ne 0 ] && _fail "failed to suspend error-logtest"
 	fi
 
 	# Load new table
-	$DMSETUP_PROG load error-test --table "$DMLINEAR_TABLE"
+	$DMSETUP_PROG load $DMERROR_NAME --table "$DMLINEAR_TABLE"
 	load_res=$?
 
 	if [ -n "$NON_ERROR_RTDEV" ]; then
-		$DMSETUP_PROG load error-rttest --table "$DMLINEAR_RTTABLE"
+		$DMSETUP_PROG load $DMERROR_RTNAME --table "$DMLINEAR_RTTABLE"
 		[ $? -ne 0 ] && _fail "failed to load working table into error-rttest"
 	fi
 
 	if [ -n "$NON_ERROR_LOGDEV" ]; then
-		$DMSETUP_PROG load error-logtest --table "$DMLINEAR_LOGTABLE"
+		$DMSETUP_PROG load $DMERROR_LOGNAME --table "$DMLINEAR_LOGTABLE"
 		[ $? -ne 0 ] && _fail "failed to load working table into error-logtest"
 	fi
 
 	# Resume devices in the opposite order that we suspended them.
 	if [ -n "$NON_ERROR_LOGDEV" ]; then
-		$DMSETUP_PROG resume error-logtest
+		$DMSETUP_PROG resume $DMERROR_LOGNAME
 		[ $? -ne 0 ] && _fail  "failed to resume error-logtest"
 	fi
 
 	if [ -n "$NON_ERROR_RTDEV" ]; then
-		$DMSETUP_PROG resume error-rttest
+		$DMSETUP_PROG resume $DMERROR_RTNAME
 		[ $? -ne 0 ] && _fail  "failed to resume error-rttest"
 	fi
 
-	$DMSETUP_PROG resume error-test
+	$DMSETUP_PROG resume $DMERROR_NAME
 	resume_res=$?
 
 	[ $load_res -ne 0 ] && _fail "dmsetup failed to load error table"
src/dmerror +5 -1
index cde2b4284..96611b101 100755
@@ -2,6 +2,10 @@
 # SPDX-License-Identifier: GPL-2.0
 # Copyright (c) 2017, Jeff Layton <[email protected]>
 
+# hackity hack hack - first arg is the test number, second arg is op.
+# This is needed so that dmerror names the dm dev correctly.
+seq=$1
+
 . ./common/config
 . ./common/dmerror
 
@@ -10,7 +14,7 @@ if [ -z "$DMERROR_DEV" ]; then
 	exit 1
 fi
 
-case $1 in
+case $2 in
 load_error_table)
 	_dmerror_load_error_table
 	;;
tests/generic/441 +1 -1
index 9851ac219..6b48fc9ed 100755
@@ -59,7 +59,7 @@ _require_fs_space $SCRATCH_MNT 65536
 testfile=$SCRATCH_MNT/fsync-err-test
 
 echo "$here/src/fsync-err $sflag -d $here/src/dmerror $testfile" >> $seqres.full
-$here/src/fsync-err $sflag -d $here/src/dmerror $testfile
+$here/src/fsync-err $sflag -d "$here/src/dmerror $seq" $testfile
 
 # success, all done
 _dmerror_load_working_table
Signature · 1 line
2.45.2
#13 lore raw

PATCH 6/40 fstests: per-test dmhuge instances

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

We can't run two tests that use dmhuge at the same time because the device name is the same. hence they interfere with each other. Give dmhuge devices their own per-test names to avoid this problem.

---
 common/dmhugedisk | 19 +++++++++++--------
 tests/generic/620 |  1 +
 2 files changed, 12 insertions(+), 8 deletions(-)
common/dmhugedisk +11 -8
index 502f02437..0757e1908 100644
@@ -4,6 +4,9 @@
 #
 # Routines for creating huge (fake) disks
 
+export DMHUGE_TEST="huge-test.$seq"
+export DMHUGE_TEST_ZERO="huge-test-zero.$seq"
+
 _require_dmhugedisk()
 {
 	_require_dm_target zero
@@ -20,26 +23,26 @@ _dmhugedisk_init()
 		chunk_size=512
 	fi
 
-	_dmsetup_remove huge-test
-	_dmsetup_remove huge-test-zero
+	_dmsetup_remove $DMHUGE_TEST
+	_dmsetup_remove $DMHUGE_TEST_ZERO
 
 	local blk_dev_size=$1
 
-	DMHUGEDISK_ZERO='/dev/mapper/huge-test-zero'
-	DMHUGEDISK_DEV='/dev/mapper/huge-test'
+	DMHUGEDISK_ZERO="/dev/mapper/$DMHUGE_TEST_ZERO"
+	DMHUGEDISK_DEV="/dev/mapper/$DMHUGE_TEST"
 
 	DMHUGEDISK_ZERO_TABLE="0 $blk_dev_size zero"
 	DMHUGEDISK_DEV_TABLE="0 $blk_dev_size snapshot $DMHUGEDISK_ZERO $SCRATCH_DEV N $chunk_size"
 
-	_dmsetup_create huge-test-zero --table "$DMHUGEDISK_ZERO_TABLE" || \
+	_dmsetup_create $DMHUGE_TEST_ZERO --table "$DMHUGEDISK_ZERO_TABLE" || \
 		_fatal "failed to create dm huge zero device"
-	_dmsetup_create huge-test --table "$DMHUGEDISK_DEV_TABLE" || \
+	_dmsetup_create $DMHUGE_TEST --table "$DMHUGEDISK_DEV_TABLE" || \
 		_fatal "failed to create dm huge device"
 }
 
 _dmhugedisk_cleanup()
 {
 	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
-	_dmsetup_remove huge-test
-	_dmsetup_remove huge-test-zero
+	_dmsetup_remove $DMHUGE_TEST
+	_dmsetup_remove $DMHUGE_TEST_ZERO
 }
tests/generic/620 +1 -0
index bf97328d1..3f1ce45a5 100755
@@ -47,6 +47,7 @@ testfile=$SCRATCH_MNT/testfile-$seq
 
 $XFS_IO_PROG -fc "pwrite -S 0xaa 0 1m" -c "fsync" $testfile | _filter_xfs_io
 
+_dmhugedisk_cleanup
 # success, all done
 status=0
 exit
Signature · 1 line
2.45.2
#14 lore raw

PATCH 7/40 fstests: per-test dmthin instances

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

We can't run two tests that use dmthin at the same time because the device name is the same. hence they interfere with each other. Give dmthin devices their own per-test names to avoid this problem.

---
 common/dmthin | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
common/dmthin +4 -4
index 7107d5080..f2c00f5a7 100644
@@ -9,16 +9,16 @@
 # from a single scratch device
 
 # Backing data dev
-DMTHIN_DATA_NAME="thin-data"
+DMTHIN_DATA_NAME="thin-data.$seq"
 DMTHIN_DATA_DEV="/dev/mapper/$DMTHIN_DATA_NAME"
 # Backing metadata dev
-DMTHIN_META_NAME="thin-meta"
+DMTHIN_META_NAME="thin-meta.$seq"
 DMTHIN_META_DEV="/dev/mapper/$DMTHIN_META_NAME"
 # Backing pool dev (combination of above)
-DMTHIN_POOL_NAME="thin-provision-pool"
+DMTHIN_POOL_NAME="thin-provision-pool.$seq"
 DMTHIN_POOL_DEV="/dev/mapper/$DMTHIN_POOL_NAME"
 # Thin volume
-DMTHIN_VOL_NAME="thin-vol"
+DMTHIN_VOL_NAME="thin-vol.$seq"
 DMTHIN_VOL_DEV="/dev/mapper/$DMTHIN_VOL_NAME"
 
 _dmthin_cleanup()
Signature · 1 line
2.45.2
#15 lore raw

PATCH 8/40 fstests: per-test dmdust instances

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

We can't run two tests that use dmdust at the same time because the device name is the same. hence they interfere with each other. Give dmdust devices their own per-test names to avoid this problem.

---
 common/dmdust | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
common/dmdust +6 -4
index 56fcc0e0f..37bb865c8 100644
@@ -4,12 +4,14 @@
 #
 # common functions for setting up and tearing down a dmdust device
 
+export DUST_NAME="dust-test.$seq"
+
 _init_dust()
 {
 	local DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
-	DUST_DEV=/dev/mapper/dust-test
+	DUST_DEV=/dev/mapper/$DUST_NAME
 	DUST_TABLE="0 $DEV_SIZE dust $SCRATCH_DEV 0 512"
-	_dmsetup_create dust-test --table "$DUST_TABLE" || \
+	_dmsetup_create $DUST_NAME --table "$DUST_TABLE" || \
 		_fatal "failed to create dust device"
 }
 
@@ -29,7 +31,7 @@ _cleanup_dust()
 {
 	# If dmsetup load fails then we need to make sure to do resume here
 	# otherwise the umount will hang
-	$DMSETUP_PROG resume dust-test > /dev/null 2>&1
+	$DMSETUP_PROG resume $DUST_NAME > /dev/null 2>&1
 	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
-	_dmsetup_remove dust-test
+	_dmsetup_remove $DUST_NAME
 }
Signature · 1 line
2.45.2
#16 lore raw

PATCH 9/40 fstests: per-test dmdelay instances

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

We can't run two tests that use dmdelay at the same time because the device name is the same. hence they interfere with each other. Give dmdelay devices their own per-test names to avoid this problem.

---
 common/dmdelay | 20 +++++++++++---------
 1 file changed, 11 insertions(+), 9 deletions(-)
common/dmdelay +11 -9
index 66cac1a70..1c4fca6e9 100644
@@ -7,13 +7,15 @@
 DELAY_NONE=0
 DELAY_READ=1
 
+export DELAY_NAME="delay-test.$seq"
+
 _init_delay()
 {
 	local BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
-	DELAY_DEV=/dev/mapper/delay-test
+	DELAY_DEV=/dev/mapper/$DELAY_NAME
 	DELAY_TABLE="0 $BLK_DEV_SIZE delay $SCRATCH_DEV 0 0"
 	DELAY_TABLE_RDELAY="0 $BLK_DEV_SIZE delay $SCRATCH_DEV 0 10000 $SCRATCH_DEV 0 0"
-	_dmsetup_create delay-test --table "$DELAY_TABLE" || \
+	_dmsetup_create $DELAY_NAME --table "$DELAY_TABLE" || \
 		_fatal "failed to create delay device"
 }
 
@@ -33,9 +35,9 @@ _cleanup_delay()
 {
 	# If dmsetup load fails then we need to make sure to do resume here
 	# otherwise the umount will hang
-	$DMSETUP_PROG resume delay-test > /dev/null 2>&1
+	$DMSETUP_PROG resume $DELAY_NAME > /dev/null 2>&1
 	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
-	_dmsetup_remove delay-test
+	_dmsetup_remove $DELAY_NAME
 }
 
 # _load_delay_table <table> [lockfs]
@@ -52,15 +54,15 @@ _load_delay_table()
 
 	# run a suspend/resume cycle to avoid excessive resume delays once a
 	# delay is introduced below
-	$DMSETUP_PROG suspend $suspend_opt delay-test
-	$DMSETUP_PROG resume $suspend_opt delay-test
+	$DMSETUP_PROG suspend $suspend_opt $DELAY_NAME
+	$DMSETUP_PROG resume $suspend_opt $DELAY_NAME
 
-	$DMSETUP_PROG suspend $suspend_opt delay-test
+	$DMSETUP_PROG suspend $suspend_opt $DELAY_NAME
 	[ $? -ne 0 ] && _fatal "failed to suspend delay-test"
 
-	$DMSETUP_PROG load delay-test --table "$table"
+	$DMSETUP_PROG load $DELAY_NAME --table "$table"
 	[ $? -ne 0 ] && _fatal "failed to load table into delay-test"
 
-	$DMSETUP_PROG resume delay-test
+	$DMSETUP_PROG resume $DELAY_NAME
 	[ $? -ne 0 ] && _fatal  "failed to resume delay-test"
 }
Signature · 1 line
2.45.2
#17 lore raw

PATCH 10/40 fstests: fix DM device creation/removal vs udev races

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

When there is load on the system, newly created DM devices don't seem to be created consistently. When a new device is created, it is supposed to be created as /dev/dm-X, and then a udev rule creates the symlink from /dev/mapper/<dev name> to /dev/dm-X.

Unfortunately, a lot of the tests that use dynamically created dm devices (dmerror, dmflakey) are not being created with this device node structure. This is resulting in getting the wrong short device name for the block device and hence we can't find the filesystem sysfs attribute directory for the filesystem on that block device.

For example, with added debug to check what device name was being passed around and resolved:

eneric/489 - output mismatch (see /mnt/xfs/runner-10/results/xfs/generic/489.out.bad)
--- tests/generic/489.out 2022-12-21 15:53:25.503043574 +1100
+++ /mnt/xfs/runner-10/results/xfs/generic/489.out.bad 2024-10-24 10:27:29.767196340 +1100
@@ -1,4 +1,10 @@
QA output created by 489
+./common/rc: line 4955: /sys/fs/xfs/flakey-test.489/error/fail_at_unmount: No such file or directory
+dev: /dev/mapper/flakey-test.489
+resolved dev: /dev/mapper/flakey-test.489
+brw-rw----. 1 root disk 251, 5 Oct 24 10:27 /dev/mapper/flakey-test.489
+./common/rc: line 4955: /sys/fs/xfs/flakey-test.489/error/metadata/EIO/max_retries: No such file or directory +./common/rc: line 4955: /sys/fs/xfs/flakey-test.489/error/metadata/EIO/retry_timeout_seconds: No such file or directory
...
(Run 'diff -u /home/dave/src/xfstests-dev/tests/generic/489.out /mnt/xfs/runner-10/results/xfs/generic/489.out.bad' to see the entire diff)

Here we see that the block device node is actually at /dev/mapper/flakey-test.489, not a link to a /dev/dm-X device node.

This implies that the udev rule to create the /dev/dm-X node and the symlink to it at /dev/mapper/flakey-test.489 has not run, and something else created the device node.

That looks like a bug in _dmsetup_create(). It creates the new DM device, then runs 'dmsetup mknodes', then waits for udev to settle. This means the mknodes command - which makes sure the dm device nodes exist - is racing with udev to create the device nodes. They don't use the same rules to create nodes, so we end up with this broken situation.

'dmsetup mknodes' is considered legacy functionality, intended for systems that have no udev capability. For systems that have udev enabled (i.e. all modern distros), mknodes should not be run because it creates a different device node structure to what udev creates and can race with udev as we see here.

Fix it by removing the 'dmsetup mknodes' as it is unnecessary to create the correct device node layout the rest of the system is expecting to see.

Additionally,_dmsetup_remove() calls 'dmsetup mknodes' and that can also race with udev and cause issues. Hence we need to remove that call from the remove operation as well.

Further, 'dmsetup remove' is also subject to races with udev which results in device remove failing. This problem is documented in the dmsetup man page and suggests the use of the "--retry" option. This means dmsetup will retry several times over a few seconds before failing the removal.

This reduces the remove failure rate substantially,
but it can still occasionally fail when the system is under heavy load and udev processing is very slow. This is fixable, but requires fstests udev infrastructure changes as it requires udevadm functionality that is relatively new. Hence that will be done as a separate fix.

---
 common/rc | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
common/rc +2 -3
index 391370fd5..a601e2c80 100644
@@ -5162,8 +5162,8 @@ _require_label_get_max()
 _dmsetup_remove()
 {
 	$UDEV_SETTLE_PROG >/dev/null 2>&1
-	$DMSETUP_PROG remove "$@" >>$seqres.full 2>&1
-	$DMSETUP_PROG mknodes >/dev/null 2>&1
+	$DMSETUP_PROG remove --retry "$@" >>$seqres.full 2>&1
+	$UDEV_SETTLE_PROG >/dev/null 2>&1
 }
 
 _dmsetup_create()
@@ -5174,7 +5174,6 @@ _dmsetup_create()
 	# device open won't also fail.
 	$UDEV_SETTLE_PROG >/dev/null 2>&1
 	$DMSETUP_PROG create "$@" >>$seqres.full 2>&1 || return 1
-	$DMSETUP_PROG mknodes >/dev/null 2>&1
 	$UDEV_SETTLE_PROG >/dev/null 2>&1
 }
 
Signature · 1 line
2.45.2
#18 lore raw

PATCH 11/40 fstests: use syncfs rather than sync

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

sync(1) is a system wide sync and is implemented by iterating all the superblocks in the system. In most cases, fstests require just the filesystem under test to be synced - we require syncfs(2) semantics but what we use is sync(2) semantics.

The result of this is that when running many concurrent fstests at the same time, we can have *hundreds* of concurrent sync operations in progress (thanks fsstress!) and this causes excessive interference with other tests that are running on other filesystems.

For example, some tests try to specifically control extent layout via specific write and fsync patterns. All these global syncs perturb them and cause them to spuriously fail.

A random snapshot of running concurrent tests shows just how many tests are explicitly blocked in sync(1):

check-parallel───check───077───077─┬─cut
      │                                    ├─du
      │                                    └─tail
      ├─check-parallel───check───311───xfs_scrub───{xfs_scrub}
      ├─check-parallel───check───531───128*[t_open_tmpfiles]
      ├─check-parallel───check───227
      ├─check-parallel───check───388
      ├─check-parallel───check───070───fsstress───fsstress───{fsstress+
      ├─check-parallel───check───232───fsstress───7*[fsstress───{fsstr+
      ├─check-parallel───check───648───sleep
      ├─check-parallel───check───409───sync
      ├─check-parallel───check───683───sync
      ├─check-parallel───check───013─┬─013───sleep
      │                              └─fsstress───2*[fsstress───{fsstr+
      ├─check-parallel───check───684───sync
      ├─check-parallel───check───673───sync
      ├─check-parallel───check───118───dd
      ├─check-parallel───check───467───open_by_handle
      ├─check-parallel───check─┬─622
      │                        └─check
      ├─check-parallel───check───685───sync
      ├─check-parallel───check───049───fsstress───fsstress───{fsstress+
      ├─check-parallel───check───599
      ├─check-parallel───check───426───open_by_handle
      ├─check-parallel───check───057───umount
      ├─check-parallel───check───390───fsstress─┬─18*[fsstress───{fsst+
      │                                         └─fsstress
      ├─check-parallel───check───158───fsstress───fsstress───{fsstress+
      ├─check-parallel───check───017
      ├─check-parallel───check───032───fsstress───fsstress
      ├─check-parallel───check───076
      ├─check-parallel───check───477───open_by_handle
      ├─check-parallel───check───170───2*[170───170]
      ├─check-parallel───check───112
      ├─check-parallel───check───686───sync
      ├─4*[check-parallel───check───check───xfs_scrub───{xfs_scrub}]
      ├─check-parallel───check───387───xfs_io───{xfs_io}
      ├─check-parallel───check───615───615
      ├─check-parallel───check─┬─051
      │                        └─check───xfs_repair
      ├─check-parallel───check───049
      ├─check-parallel───check───247
      ├─check-parallel───check───674───sync
      ├─check-parallel───check───040
      ├─check-parallel───check───560───fsstress───fsstress───{fsstress+
      ├─check-parallel───check───030─┬─030─┬─030───xfs_repair
      │                              │     └─030───perl
      │                              ├─sed
      │                              └─uniq
      ├─check-parallel───check───055───055
      ├─2*[check-parallel───check───check]
      ├─check-parallel───check───042
      ├─check-parallel───check───204
      ├─check-parallel───check───271─┬─271───sed
      │                              └─md5sum
      ├─check-parallel───check───091─┬─fsx
      │                              └─tee
      ├─check-parallel───check───063───sleep
      ├─check-parallel───check───026
      ├─check-parallel───check───459───lvm
      ├─check-parallel───check───495
      ├─check-parallel───check───141───fsstress───4*[fsstress]
      ├─check-parallel───check───011─┬─fsstress─┬─fsstress───{fsstress+
      │                              │          └─fsstress
      │                              └─sleep
      ├─check-parallel───check───328───sync
      ├─check-parallel───check───507───507
      ├─check-parallel───check
      ├─check-parallel───check───687───sync
      ├─check-parallel───check───109───mkfs.xfs
      ├─check-parallel───check───324
      ├─check-parallel───check───114───aio-dio-eof-rac
      └─check-parallel───check───503───xfs_scrub───2*[{xfs_scrub}]

There are ~10 sync(1) calls blocked and at least half of the 50-odd fsstress processes currently running are also going to be stuck in sync(2) calls.

They are stuck because the superblock iteration has to wait for mount, unmount, freeze, thaw and any other operation that locks a superblock exclusively. When running dozens of tests concurrently, there can be tens of superblocks that are locked exclusively for every second for significant lengths of time.

Hence the use of sync has impact on both performance and test behaviour and we need to minimise the amount of sync(1) and sync(2) usage as much as possible.

Introduce _test_sync() and _scratch_sync() so we can implement a syncfs mechanism with a fallback to sync(1) if it is not supported without dirtying all the test code unnecessarily. Then convert fsstress to use syncfs(2) in preference to sync(2).

This commit changes all the generic and XFS tests to use the new sync functions, other filesystem specific tests will eventually need to be converted to avoid similar problems.

---
 common/encrypt    |  2 +-
 common/log        |  2 +-
 common/quota      |  2 +-
 common/rc         | 24 ++++++++++++++++++++++++
 common/repair     |  2 +-
 common/report     |  2 +-
 common/verity     |  2 +-
 common/xfs        |  2 +-
 ltp/fsstress.c    | 12 ++++++++++++
 tests/generic/015 |  2 +-
 tests/generic/029 |  2 +-
 tests/generic/030 |  2 +-
 tests/generic/032 |  2 +-
 tests/generic/034 |  2 +-
 tests/generic/039 |  2 +-
 tests/generic/040 |  2 +-
 tests/generic/041 |  2 +-
 tests/generic/048 |  2 +-
 tests/generic/049 |  2 +-
 tests/generic/051 |  4 ++--
 tests/generic/054 |  2 +-
 tests/generic/057 |  2 +-
 tests/generic/059 |  2 +-
 tests/generic/065 |  2 +-
 tests/generic/066 |  2 +-
 tests/generic/073 |  2 +-
 tests/generic/090 |  4 ++--
 tests/generic/092 |  2 +-
 tests/generic/098 |  4 ++--
 tests/generic/101 |  2 +-
 tests/generic/104 |  2 +-
 tests/generic/106 |  2 +-
 tests/generic/107 |  2 +-
 tests/generic/150 |  2 +-
 tests/generic/151 |  2 +-
 tests/generic/152 |  2 +-
 tests/generic/157 |  3 ++-
 tests/generic/158 |  3 ++-
 tests/generic/159 |  2 +-
 tests/generic/160 |  2 +-
 tests/generic/171 |  4 ++--
 tests/generic/172 |  4 ++--
 tests/generic/173 |  4 ++--
 tests/generic/174 |  4 ++--
 tests/generic/204 |  8 --------
 tests/generic/247 |  2 +-
 tests/generic/250 |  2 +-
 tests/generic/252 |  2 +-
 tests/generic/265 |  2 +-
 tests/generic/266 |  2 +-
 tests/generic/267 |  2 +-
 tests/generic/268 |  2 +-
 tests/generic/271 |  2 +-
 tests/generic/272 |  2 +-
 tests/generic/273 |  2 +-
 tests/generic/274 |  6 +++---
 tests/generic/275 |  6 +++---
 tests/generic/276 |  2 +-
 tests/generic/278 |  2 +-
 tests/generic/279 |  2 +-
 tests/generic/281 |  2 +-
 tests/generic/282 |  2 +-
 tests/generic/283 |  2 +-
 tests/generic/315 |  2 +-
 tests/generic/317 |  2 +-
 tests/generic/318 |  2 +-
 tests/generic/325 |  2 +-
 tests/generic/328 |  4 ++--
 tests/generic/329 |  2 +-
 tests/generic/330 |  2 +-
 tests/generic/331 |  2 +-
 tests/generic/332 |  2 +-
 tests/generic/335 |  2 +-
 tests/generic/336 |  2 +-
 tests/generic/341 |  2 +-
 tests/generic/342 |  2 +-
 tests/generic/343 |  2 +-
 tests/generic/347 |  2 +-
 tests/generic/348 |  2 +-
 tests/generic/353 |  2 +-
 tests/generic/376 |  2 +-
 tests/generic/382 |  2 +-
 tests/generic/387 |  2 +-
 tests/generic/391 |  2 +-
 tests/generic/409 |  2 +-
 tests/generic/410 |  2 +-
 tests/generic/411 |  2 +-
 tests/generic/416 |  2 +-
 tests/generic/422 |  4 ++--
 tests/generic/425 |  2 +-
 tests/generic/461 |  2 +-
 tests/generic/474 |  4 ++--
 tests/generic/479 |  2 +-
 tests/generic/480 |  2 +-
 tests/generic/483 |  2 +-
 tests/generic/489 |  2 +-
 tests/generic/502 |  2 +-
 tests/generic/505 |  2 +-
 tests/generic/506 |  2 +-
 tests/generic/507 |  2 +-
 tests/generic/508 |  2 +-
 tests/generic/510 |  2 +-
 tests/generic/520 |  2 +-
 tests/generic/526 |  2 +-
 tests/generic/527 |  2 +-
 tests/generic/535 |  2 +-
 tests/generic/546 |  2 +-
 tests/generic/556 |  2 +-
 tests/generic/579 |  2 +-
 tests/generic/589 |  2 +-
 tests/generic/599 |  2 +-
 tests/generic/610 |  2 +-
 tests/generic/628 |  4 ++--
 tests/generic/629 |  4 ++--
 tests/generic/640 |  2 +-
 tests/generic/673 |  2 +-
 tests/generic/674 |  2 +-
 tests/generic/675 |  2 +-
 tests/generic/677 |  2 +-
 tests/generic/683 |  2 +-
 tests/generic/684 |  2 +-
 tests/generic/685 |  2 +-
 tests/generic/686 |  2 +-
 tests/generic/687 |  2 +-
 tests/generic/688 |  2 +-
 tests/generic/690 |  2 +-
 tests/generic/694 |  2 +-
 tests/generic/695 |  2 +-
 tests/generic/703 |  2 +-
 tests/generic/716 |  2 +-
 tests/generic/718 |  2 +-
 tests/generic/719 |  2 +-
 tests/generic/721 |  2 +-
 tests/generic/725 |  2 +-
 tests/generic/726 |  2 +-
 tests/generic/727 |  2 +-
 tests/generic/735 |  2 +-
 tests/generic/738 |  2 +-
 tests/generic/745 |  4 ++--
 tests/generic/746 |  4 ++--
 tests/generic/747 |  4 ++--
 tests/generic/749 |  4 ++--
 tests/xfs/016     |  4 ++--
 tests/xfs/052     |  2 +-
 tests/xfs/110     |  2 +-
 tests/xfs/118     |  4 ++--
 tests/xfs/119     |  2 +-
 tests/xfs/128     |  2 +-
 tests/xfs/133     |  2 +-
 tests/xfs/134     |  2 +-
 tests/xfs/161     |  2 +-
 tests/xfs/176     |  2 +-
 tests/xfs/195     |  2 +-
 tests/xfs/201     |  4 ++--
 tests/xfs/212     |  2 +-
 tests/xfs/227     |  2 +-
 tests/xfs/231     |  4 ++--
 tests/xfs/232     |  4 ++--
 tests/xfs/234     |  2 +-
 tests/xfs/236     |  2 +-
 tests/xfs/237     |  2 +-
 tests/xfs/239     |  2 +-
 tests/xfs/240     |  2 +-
 tests/xfs/241     |  2 +-
 tests/xfs/243     |  4 ++--
 tests/xfs/246     |  2 +-
 tests/xfs/265     |  2 +-
 tests/xfs/272     |  2 +-
 tests/xfs/274     |  2 +-
 tests/xfs/276     |  2 +-
 tests/xfs/291     |  4 ++--
 tests/xfs/309     |  2 +-
 tests/xfs/312     |  2 +-
 tests/xfs/313     |  2 +-
 tests/xfs/314     |  2 +-
 tests/xfs/315     |  4 ++--
 tests/xfs/316     |  2 +-
 tests/xfs/317     |  2 +-
 tests/xfs/318     |  4 ++--
 tests/xfs/319     |  2 +-
 tests/xfs/320     |  2 +-
 tests/xfs/321     |  2 +-
 tests/xfs/322     |  2 +-
 tests/xfs/323     |  2 +-
 tests/xfs/324     |  2 +-
 tests/xfs/325     |  4 ++--
 tests/xfs/326     |  4 ++--
 tests/xfs/327     |  2 +-
 tests/xfs/420     |  2 +-
 tests/xfs/421     |  2 +-
 tests/xfs/423     |  2 +-
 tests/xfs/438     |  2 +-
 tests/xfs/440     |  2 +-
 tests/xfs/442     |  2 +-
 tests/xfs/511     |  2 +-
 tests/xfs/519     |  2 +-
 tests/xfs/520     |  2 +-
 tests/xfs/541     |  2 +-
 tests/xfs/553     |  4 ++--
 tests/xfs/558     |  6 +++---
 tests/xfs/607     |  6 +++---
 tests/xfs/617     |  2 +-
 tests/xfs/629     |  2 +-
 tests/xfs/630     |  2 +-
 tests/xfs/631     |  2 +-
 tests/xfs/790     |  2 +-
 tests/xfs/791     |  2 +-
 tests/xfs/792     |  2 +-
 208 files changed, 279 insertions(+), 249 deletions(-)
common/encrypt +1 -1
index d90a566ac..52f23cd0a 100644
@@ -686,7 +686,7 @@ _get_ciphertext_block_list()
 {
 	local file=$1
 
-	sync
+	_sync_fs $file
 	$XFS_IO_PROG -c fiemap $file | perl -ne '
 		next if not /^\s*\d+: \[\d+\.\.\d+\]: (\d+)\.\.(\d+)/;
 		print $_ . "," foreach $1..$2;' | sed 's/,$//'
common/log +1 -1
index ea7e4158e..ab7bc9f87 100644
@@ -392,7 +392,7 @@ _create_log_sync()
     # add some syncs to get the log flushed to disk 
     for file in $SCRATCH_MNT/{0,1,2,3,4,5,6,7,8,9}{0,1,2,3,4,5,6,7,8,9}; do
 	touch $file
-	sync
+	_scratch_sync
     done
 
     # unmount the FS
common/quota +1 -1
index 3bf7d552e..e87489192 100644
@@ -336,7 +336,7 @@ _qmount_option()
 _check_quota_usage()
 {
 	# Sync to get delalloc to disk
-	sync
+	_scratch_sync
 
 	# kill caches to guarantee removal speculative delalloc
 	# XXX: really need an ioctl instead of this big hammer
common/rc +24 -0
index a601e2c80..6f939e3b5 100644
@@ -6,6 +6,30 @@
 
 BC="$(type -P bc)" || BC=
 
+# Don't use sync(1) directly if at all possible. In most cases we only need to
+# sync the fs under test, so we use syncfs if it is supported to prevent
+# disturbance of other tests that may be running concurrently.
+_sync_fs()
+{
+	local dir=$1
+
+	if [ -e $dir ]; then
+		$XFS_IO_PROG -rxc "syncfs" $dir > /dev/null 2>&1
+		return
+	fi
+	sync
+}
+
+_scratch_sync()
+{
+	_sync_fs $SCRATCH_MNT
+}
+
+_test_sync()
+{
+	_sync_fs $TEST_DIR
+}
+
 # Common execution handling for fsstress invocation.
 #
 # We need per-test fsstress binaries because of the way fsstress forks and
common/repair +1 -1
index 8945d0028..0dae83052 100644
@@ -119,7 +119,7 @@ _check_repair()
 	_scratch_mount
 	POSIXLY_CORRECT=yes \
 	dd if=/bin/bash of=$SCRATCH_MNT/sh 2>&1 |_filter_dd
-	sync
+	_scratch_sync
 	rm -f $SCRATCH_MNT/sh
 	_scratch_unmount
 
common/report +1 -1
index 0e91e481f..7128bbeba 100644
@@ -156,7 +156,7 @@ ENDL
 		cat $report >> "$tmp_fn"
 	fi
 	echo "</testsuite>" >> "$tmp_fn"
-	sync "$tmp_fn" && mv "$tmp_fn" "$out_fn"
+	_sync_fs "$tmp_fn" && mv "$tmp_fn" "$out_fn"
 	echo "Xunit report: $out_fn"
 }
 
common/verity +1 -1
index 59b67e120..11e839d2e 100644
@@ -335,7 +335,7 @@ _fsv_scratch_corrupt_bytes()
 	local dd_cmds=()
 	local cmd
 
-	sync	# Sync to avoid unwritten extents
+	_scratch_sync	# Sync to avoid unwritten extents
 
 	cat > $tmp.bytes
 	local end=$(( offset + $(_get_filesize $tmp.bytes ) ))
common/xfs +1 -1
index 68abfa71d..0417a40ad 100644
@@ -1087,7 +1087,7 @@ _scratch_xfs_unmount_dirty()
 
 	rm -f "$f"
 	echo "test" > "$f"
-	sync
+	_scratch_sync
 	_scratch_shutdown
 	_scratch_unmount
 }
ltp/fsstress.c +12 -0
index a6840f28c..8b8c9035f 100644
@@ -5241,6 +5241,18 @@ symlink_f(opnum_t opno, long r)
 void
 sync_f(opnum_t opno, long r)
 {
+	int	fd;
+
+	fd = open(homedir, O_RDONLY|O_DIRECTORY);
+	if (fd < 0)
+		goto use_sync;
+	syncfs(fd);
+	close(fd);
+	if (verbose)
+		printf("%d/%lld: syncfs\n", procid, opno);
+	return;
+
+use_sync:
 	sync();
 	if (verbose)
 		printf("%d/%lld: sync\n", procid, opno);
tests/generic/015 +1 -1
index d4f81c7f6..3cbb2f2d0 100755
@@ -85,7 +85,7 @@ fi
 
 echo "check free space:"
 
-sync
+_scratch_sync
 
 free2=`_free`
 if [ -z "$free2" ]
tests/generic/029 +1 -1
index 0af46e892..c6162b0b4 100755
@@ -43,7 +43,7 @@ echo "==== Post-Remount =="
 _hexdump $testfile
 
 rm -f $testfile
-sync
+_scratch_sync
 
 # second case is to do a mwrite between the truncate to a block on the
 # same page we are truncating within the EOF. This checks that a mapped
tests/generic/030 +1 -1
index 3f0e5b993..b1b51469c 100755
@@ -48,7 +48,7 @@ echo "==== Post-Remount =="
 _hexdump $testfile
 
 rm -f $testfile
-sync
+_scratch_sync
 
 # second case is to do a mwrite between the truncate to a block on the
 # same page we are truncating within the EOF. This checks that a mapped
tests/generic/032 +1 -1
index 63abd62a8..30290c722 100755
@@ -29,7 +29,7 @@ _cleanup()
 _syncloop()
 {
 	while [ true ]; do
-		sync
+		_scratch_sync
 	done
 }
 
tests/generic/034 +1 -1
index 71aae2a01..cd22f330d 100755
@@ -41,7 +41,7 @@ touch $SCRATCH_MNT/test_dir/foo
 # The intention is that at log recovery time we have a dir entry for 'foo' both
 # in the fs/subvol tree and in the log tree - this is necessary to trigger the
 # bug on btrfs.
-sync
+_scratch_sync
 
 touch $SCRATCH_MNT/test_dir/bar
 $XFS_IO_PROG -c "fsync" $SCRATCH_MNT/test_dir
tests/generic/039 +1 -1
index 5162e85b0..00d4e4afb 100755
@@ -44,7 +44,7 @@ echo "hello world" > $SCRATCH_MNT/a/b/foo
 ln $SCRATCH_MNT/a/b/foo $SCRATCH_MNT/a/b/bar
 
 # Make sure all metadata and data are durably persisted.
-sync
+_scratch_sync
 
 # Now remove one of the hard links and fsync the inode.
 rm -f $SCRATCH_MNT/a/b/bar
tests/generic/040 +1 -1
index eb88e9eed..8b4e3b703 100755
@@ -60,7 +60,7 @@ for i in `seq 1 3000`; do
 done
 
 # Make sure all metadata and data are durably persisted.
-sync
+_scratch_sync
 
 # Add one more link to the inode that ends up being a btrfs extref and fsync
 # the inode.
tests/generic/041 +1 -1
index 447013689..6d42d1a28 100755
@@ -64,7 +64,7 @@ for i in `seq 1 3000`; do
 done
 
 # Make sure all metadata and data are durably persisted.
-sync
+_scratch_sync
 
 # Now remove one link, add a new one with a new name, add another new one with
 # the same name as the one we just removed and fsync the inode.
tests/generic/048 +1 -1
index 7b28cb053..5c554d501 100755
@@ -69,7 +69,7 @@ do
 done
 
 # sync, then shutdown immediately after, then remount and test
-sync
+_scratch_sync
 _scratch_shutdown
 _scratch_unmount
 _scratch_mount
tests/generic/049 +1 -1
index 8e4b99565..94b5afe96 100755
@@ -66,7 +66,7 @@ do
 done
 
 # sync, then shutdown immediately after, then remount and test
-sync
+_scratch_sync
 _scratch_shutdown
 _scratch_unmount
 _scratch_mount
tests/generic/051 +2 -2
index 69250cde6..5ee4dfdba 100755
@@ -30,7 +30,7 @@ load_dir=$SCRATCH_MNT/test
 _run_fsstress_bg -n 10000000 -p $PROCS -d $load_dir
 sleep $SLEEP_TIME
 _kill_fsstress
-sync
+_scratch_sync
 _scratch_unmount
 
 # now mount again, run the load again, this time with a shutdown.
@@ -38,7 +38,7 @@ _scratch_mount
 $XFS_FSR_PROG -v $load_dir >> $seqres.full 2>&1
 _run_fsstress_bg -n10000000 -p $PROCS -d $load_dir
 sleep $SLEEP_TIME
-sync
+_scratch_sync
 
 # now shutdown and unmount
 sleep 5
tests/generic/054 +1 -1
index 20e9a1e96..0649b2593 100755
@@ -67,7 +67,7 @@ for s in sync nosync ; do
 	    # add some syncs to get the log flushed to disk
 	    for file in $SCRATCH_MNT/{0,1,2,3,4,5,6,7,8,9}{0,1,2,3,4,5,6,7,8,9}; do
 		touch $file
-		sync
+		_scratch_sync
 	    done
 	else
 	    # generate some log traffic - but not too much - life gets a little
tests/generic/057 +1 -1
index c92af5ddb..c5db80977 100755
@@ -41,7 +41,7 @@ $XFS_IO_PROG -f -c "pwrite -S 0xaa -b 8K 0 8K" \
 	$SCRATCH_MNT/foo | _filter_xfs_io
 
 # Make sure the file is durably persisted.
-sync
+_scratch_sync
 
 # Append some data to our file, to increase its size.
 $XFS_IO_PROG -f -c "pwrite -S 0xcc -b 4K 8K 4K" \
tests/generic/059 +1 -1
index 0d3270787..db48de378 100755
@@ -58,7 +58,7 @@ $XFS_IO_PROG -c "fsync" $SCRATCH_MNT/foo
 # that modifies the data or metadata of our file, should update those fields in
 # the btrfs inode with values that make the next fsync operation write to the
 # fsync log.
-sync
+_scratch_sync
 
 # Sleep for 1 second, because we want to check that the next punch operations we
 # do update the file's mtime and ctime.
tests/generic/065 +1 -1
index adf55b750..f7e1e276f 100755
@@ -42,7 +42,7 @@ $XFS_IO_PROG -f -c "pwrite -S 0xaa 0 8K" $SCRATCH_MNT/foo | _filter_xfs_io
 mkdir $SCRATCH_MNT/mydir
 
 # Make sure all metadata and data are durably persisted.
-sync
+_scratch_sync
 
 # Add a hard link to 'foo' inside our test directory and fsync only the
 # directory. The btrfs fsync implementation had a bug that caused the new
tests/generic/066 +1 -1
index e32e9658b..37d04e069 100755
@@ -48,7 +48,7 @@ $SETFATTR_PROG -n user.attr2 -v val2 $SCRATCH_MNT/foobar
 $SETFATTR_PROG -n user.attr3 -v val3 $SCRATCH_MNT/foobar
 
 # Make sure everything is durably persisted.
-sync
+_scratch_sync
 
 # Now delete the second xattr and fsync the inode.
 $SETFATTR_PROG -x user.attr2 $SCRATCH_MNT/foobar
tests/generic/073 +1 -1
index 5ebf634fa..05df1ea87 100755
@@ -51,7 +51,7 @@ touch $SCRATCH_MNT/testdir_1/bar
 mkdir $SCRATCH_MNT/testdir_2
 
 # Make sure everything is durably persisted.
-sync
+_scratch_sync
 
 # Write more 8Kb of data to our file.
 $XFS_IO_PROG -c "pwrite -S 0xbb 8K 8K" $SCRATCH_MNT/foo | _filter_xfs_io
tests/generic/090 +2 -2
index 36a9ee865..b1ea27bbd 100755
@@ -41,7 +41,7 @@ _mount_flakey
 $XFS_IO_PROG -f -c "pwrite -S 0xaa 0 32k" \
 		-c "fsync" \
 		$SCRATCH_MNT/foo | _filter_xfs_io
-sync
+_scratch_sync
 
 # Add a hard link to our file.
 # On btrfs this sets the flag BTRFS_INODE_COPY_EVERYTHING on the btrfs inode,
@@ -50,7 +50,7 @@ ln $SCRATCH_MNT/foo $SCRATCH_MNT/bar
 
 # Sync the filesystem to force a commit of the current btrfs transaction, this
 # is a necessary condition to trigger the bug on btrfs.
-sync
+_scratch_sync
 
 # Now append more data to our file, increasing its size, and fsync the file.
 # In btrfs because the inode flag BTRFS_INODE_COPY_EVERYTHING was set and the
tests/generic/092 +1 -1
index d8c223270..3a61f00f7 100755
@@ -36,7 +36,7 @@ _require_congruent_file_oplen $TEST_DIR $((7 * 1048576))
 # past i_size
 $XFS_IO_PROG -f -c "falloc -k 0 10M" -c "pwrite 0 5M" -c "truncate 5M"\
 	$TEST_DIR/testfile.$seq | _filter_xfs_io
-sync
+_test_sync
 $XFS_IO_PROG -c "fiemap -v" $TEST_DIR/testfile.$seq | _filter_fiemap
 
 # Now verify that if we truncate up past i_size we don't trim the preallocated
tests/generic/098 +2 -2
index 57de42e9c..ff6cf908e 100755
@@ -38,7 +38,7 @@ workout()
 
 	# Create our test file with some data and durably persist it.
 	$XFS_IO_PROG -t -f -c "pwrite -S 0xaa 0 128K" $SCRATCH_MNT/foo | _filter_xfs_io
-	sync
+	_scratch_sync
 
 	# Append some data to the file, increasing its size, and leave a hole between
 	# the old size and the start offset if the following write. So our file gets
@@ -49,7 +49,7 @@ workout()
 	# This is required to trigger a bug in btrfs truncate where it updates on-disk
 	# inode size incorrectly.
 	if [ $need_sync -eq 1 ]; then
-		sync
+		_scratch_sync
 	fi
 
 	# Now truncate our file to a smaller size that is in the middle of the hole we
tests/generic/101 +1 -1
index 2e9543579..4295f0801 100755
@@ -49,7 +49,7 @@ $XFS_IO_PROG -f -c "pwrite -S 0xaa 0 64K"         \
 $XFS_IO_PROG -f -c "pwrite -S 0xee 0 64K"         \
 		-c "pwrite -S 0xff 64K 61K"       \
 		$SCRATCH_MNT/bar | _filter_xfs_io
-sync
+_scratch_sync
 
 # Now truncate our file foo to a smaller size (64Kb) and then truncate it to the
 # size it had before the shrinking truncate (125Kb). Then fsync our file. If a
tests/generic/104 +1 -1
index 7f294e1b6..9af3b5582 100755
@@ -38,7 +38,7 @@ touch $SCRATCH_MNT/testdir/foo
 touch $SCRATCH_MNT/testdir/bar
 
 # Make sure everything done so far is durably persisted.
-sync
+_scratch_sync
 
 # Create one hard link for file foo and another one for file bar. After that
 # fsync only the file bar.
tests/generic/106 +1 -1
index 2b89ae69a..8bcc7575e 100755
@@ -37,7 +37,7 @@ touch $SCRATCH_MNT/testdir/foo
 ln $SCRATCH_MNT/testdir/foo $SCRATCH_MNT/testdir/bar
 
 # Make sure everything done so far is durably persisted.
-sync
+_scratch_sync
 
 # Now remove one of the links, trigger inode eviction and then fsync our
 # inode.
tests/generic/107 +1 -1
index 177ca59c5..8a82d1468 100755
@@ -41,7 +41,7 @@ ln $SCRATCH_MNT/foo $SCRATCH_MNT/testdir/foo2
 ln $SCRATCH_MNT/foo $SCRATCH_MNT/testdir/foo3
 
 # Make sure everything done so far is durably persisted.
-sync
+_scratch_sync
 
 # Now we remove one of our file's hardlinks in the directory testdir.
 unlink $SCRATCH_MNT/testdir/foo3
tests/generic/150 +1 -1
index 159a53c4d..d9a53a5b1 100755
@@ -38,7 +38,7 @@ sz=$((blksz * blks))
 nr=7
 filesize=$((blksz * nr))
 _pwrite_byte 0x61 0 $sz $testdir/file1 >> $seqres.full
-sync
+_test_sync
 free_blocks0=$(stat -f $testdir -c '%f')
 
 echo "Create the reflink copies"
tests/generic/151 +1 -1
index a335b9273..d20477ab3 100755
@@ -43,7 +43,7 @@ free_blocks0=$(stat -f $testdir -c '%f')
 nr=7
 filesize=$((blksz * nr))
 _pwrite_byte 0x61 0 $sz $testdir/file1 >> $seqres.full
-sync
+_test_sync
 
 echo "Create the reflink copies"
 for i in `seq 2 $nr`; do
tests/generic/152 +1 -1
index 8d87dbac7..afcbd6d93 100755
@@ -44,7 +44,7 @@ free_blocks0=$(stat -f $testdir -c '%f')
 nr=4
 filesize=$((blksz * nr))
 _pwrite_byte 0x61 0 $sz $testdir/file1 >> $seqres.full
-sync
+_test_sync
 
 echo "Create the reflink copies"
 for i in `seq 2 $nr`; do
tests/generic/157 +2 -1
index cb9143673..24415937c 100755
@@ -54,7 +54,8 @@ seq 1 $((2 * blksz / 250)) | while read f; do
 done
 mknod $testdir1/dev1 c 1 3
 mkfifo $testdir1/fifo1
-sync
+_test_sync
+_scratch_sync
 
 _filter_enotty() {
 	sed -e 's/Inappropriate ioctl for device/Invalid argument/g'
tests/generic/158 +2 -1
index 171d3c0dc..ccee3690a 100755
@@ -55,7 +55,8 @@ seq 1 $((2 * blksz / 250)) | while read f; do
 done
 mknod $testdir1/dev1 c 1 3
 mkfifo $testdir1/fifo1
-sync
+_test_sync
+_scratch_sync
 
 _filter_enotty() {
 	_filter_dedupe_error | \
tests/generic/159 +1 -1
index c4dec17ca..a2d20622a 100755
@@ -39,7 +39,7 @@ nr=4
 filesize=$((blksz * nr))
 _pwrite_byte 0x61 0 $sz $testdir1/file1 >> $seqres.full
 _pwrite_byte 0x61 0 $sz $testdir1/file2 >> $seqres.full
-sync
+_test_sync
 
 do_filter_output()
 {
tests/generic/160 +1 -1
index fc7a161e1..c1f24ddbb 100755
@@ -39,7 +39,7 @@ nr=4
 filesize=$((blksz * nr))
 _pwrite_byte 0x61 0 $sz $testdir1/file1 >> $seqres.full
 _pwrite_byte 0x61 0 $sz $testdir1/file2 >> $seqres.full
-sync
+_test_sync
 
 do_filter_output()
 {
tests/generic/171 +2 -2
index dd56aa792..5c8504d26 100755
@@ -49,12 +49,12 @@ mkdir $testdir
 echo "Create a big file and reflink it"
 _pwrite_byte 0x61 0 $((blksz * nr_blks)) $testdir/bigfile >> $seqres.full 2>&1
 _cp_reflink $testdir/bigfile $testdir/clonefile
-sync
+_scratch_sync
 
 echo "Allocate the rest of the space"
 nr_free=$(stat -f -c '%f' $testdir)
 _fill_fs $((nr_free * blksz)) $testdir/space $blksz 0 >> $seqres.full 2>&1
-sync
+_scratch_sync
 
 echo "CoW the big file"
 out="$(_pwrite_byte 0x62 0 $((blksz * nr_blks)) $testdir/bigfile 2>&1 | \
tests/generic/172 +2 -2
index c23a12284..17f817805 100755
@@ -47,11 +47,11 @@ mkdir $testdir
 echo "Create a big file and reflink it"
 _pwrite_byte 0x61 0 $file_size $testdir/bigfile >> $seqres.full 2>&1
 _cp_reflink $testdir/bigfile $testdir/clonefile
-sync
+_scratch_sync
 
 echo "Allocate the rest of the space"
 _fill_fs $fs_size  $testdir/space $blksz 0 >> $seqres.full 2>&1
-sync
+_scratch_sync
 
 echo "CoW the big file"
 out="$(_pwrite_byte 0x62 0 $file_size $testdir/bigfile 2>&1 | \
tests/generic/173 +2 -2
index 8df3c6df2..d7ac7b15d 100755
@@ -49,12 +49,12 @@ mkdir $testdir
 echo "Create a big file and reflink it"
 _pwrite_byte 0x61 0 $((blksz * nr_blks)) $testdir/bigfile >> $seqres.full 2>&1
 _cp_reflink $testdir/bigfile $testdir/clonefile
-sync
+_scratch_sync
 
 echo "Allocate the rest of the space"
 nr_free=$(stat -f -c '%f' $testdir)
 _fill_fs $((blksz * nr_free)) $testdir/space $blksz 0 >> $seqres.full 2>&1
-sync
+_scratch_sync
 
 echo "mmap CoW the big file"
 core_ulimit="$(ulimit -c)"
tests/generic/174 +2 -2
index b9c292071..c7c62001d 100755
@@ -50,12 +50,12 @@ mkdir $testdir
 echo "Create a big file and reflink it"
 _pwrite_byte 0x61 0 $((blksz * nr_blks)) $testdir/bigfile >> $seqres.full 2>&1
 _cp_reflink $testdir/bigfile $testdir/clonefile
-sync
+_scratch_sync
 
 echo "Allocate the rest of the space"
 nr_free=$(stat -f -c '%f' $testdir)
 _fill_fs $((blksz * nr_free)) $testdir/space $blksz 0 >> $seqres.full 2>&1
-sync
+_scratch_sync
 
 echo "CoW the big file"
 out="$(_pwrite_byte 0x62 0 $((blksz * nr_blks)) $testdir/bigfile -d 2>&1 | \
tests/generic/204 +0 -8
index 9ef54e239..39483a09a 100755
@@ -9,17 +9,9 @@
 . ./common/preamble
 _begin_fstest metadata rw auto
 
-# Override the default cleanup function.
-_cleanup()
-{
-	rm -f $tmp.*
-	sync
-}
-
 # Import common functions.
 . ./common/filter
 
-
 _require_scratch
 
 # For xfs, we need to handle the different default log sizes that different
tests/generic/247 +1 -1
index 47ff5c343..a758ae24c 100755
@@ -39,7 +39,7 @@ iosize=1048576
 # Initialise file
 dd if=/dev/zero of=$testfile bs=$iosize count=$loops &> /dev/null
 [ $? -ne 0 ] && exit
-sync
+_test_sync
 	
 # Direct I/O overwriter
 dd if=/dev/zero of=$testfile oflag=direct bs=$iosize count=$loops conv=notrunc &> /dev/null &
tests/generic/250 +1 -1
index 196a34bb0..605d7c8ad 100755
@@ -61,7 +61,7 @@ md5sum $testdir/file2 | _filter_scratch
 
 echo "Write and unmount"
 $XFS_IO_PROG -f -c "pwrite -S 0x63 $bufsize 1" $testdir/file2 >> $seqres.full
-sync
+_scratch_sync
 _dmerror_load_error_table
 $XFS_IO_PROG -d -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $testdir/file2 \
 	>> $seqres.full 2>&1
tests/generic/252 +1 -1
index 3ee2b0a67..ddf950e34 100755
@@ -64,7 +64,7 @@ md5sum $testdir/file2 | _filter_scratch
 echo "CoW and unmount"
 $XFS_IO_PROG -f -c "pwrite -S 0x63 $bufsize 1" $testdir/file2 >> $seqres.full
 $XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $TEST_DIR/moo >> $seqres.full
-sync
+_scratch_sync
 _dmerror_load_error_table
 $AIO_TEST -a $alignment -f DIRECT -b $bufsize $TEST_DIR/moo $testdir/file2 >> $seqres.full
 _dmerror_load_working_table
tests/generic/265 +1 -1
index 3c84e2a84..0f165e7e6 100755
@@ -53,7 +53,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "CoW and unmount"
-sync
+_scratch_sync
 _dmerror_load_error_table
 urk=$($XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" \
 	-c "fdatasync" $testdir/file2 2>&1)
tests/generic/266 +1 -1
index d5675afbc..02e290de5 100755
@@ -53,7 +53,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "CoW and unmount"
-sync
+_scratch_sync
 _dmerror_load_error_table
 urk=$($XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" \
 	-c "fdatasync" $testdir/file2 2>&1)
tests/generic/267 +1 -1
index ddaf1064c..48b1981f1 100755
@@ -53,7 +53,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "CoW and unmount"
-sync
+_scratch_sync
 _dmerror_load_error_table
 $XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $testdir/file2 >> $seqres.full 2>&1
 _dmerror_load_working_table
tests/generic/268 +1 -1
index c6068cf4e..f46e0b755 100755
@@ -54,7 +54,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "CoW and unmount"
-sync
+_scratch_sync
 _dmerror_load_error_table
 urk=$($XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" \
 	-c "fdatasync" $testdir/file2 2>&1)
tests/generic/271 +1 -1
index ce647d155..854ea1021 100755
@@ -54,7 +54,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "CoW and unmount"
-sync
+_scratch_sync
 _dmerror_load_error_table
 urk=$($XFS_IO_PROG -d -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $testdir/file2 2>&1)
 echo $urk >> $seqres.full
tests/generic/272 +1 -1
index 3d2cace9f..bc77365df 100755
@@ -54,7 +54,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "CoW and unmount"
-sync
+_scratch_sync
 _dmerror_load_error_table
 urk=$($XFS_IO_PROG -d -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $testdir/file2 2>&1)
 echo $urk >> $seqres.full
tests/generic/273 +1 -1
index e0ee0483a..5100e848b 100755
@@ -92,7 +92,7 @@ _porter()
 		echo "_porter $_suffix not complete"
 	fi
 	
-	sync
+	_scratch_sync
 }
 
 _do_workload()
tests/generic/274 +3 -3
index d526a9b71..916c7173c 100755
@@ -50,7 +50,7 @@ echo "Fill fs with 1M IOs; ENOSPC expected" >> $seqres.full
 dd if=/dev/zero of=$SCRATCH_MNT/tmp1 bs=1M >>$seqres.full 2>&1
 echo "Fill fs with 4K IOs; ENOSPC expected" >> $seqres.full
 dd if=/dev/zero of=$SCRATCH_MNT/tmp2 bs=4K >>$seqres.full 2>&1
-sync
+_scratch_sync
 # Last effort, use O_SYNC
 echo "Fill fs with 4K DIOs; ENOSPC expected" >> $seqres.full
 dd if=/dev/zero of=$SCRATCH_MNT/tmp3 bs=4K oflag=sync >>$seqres.full 2>&1
@@ -66,7 +66,7 @@ for i in `seq 1 2 1023`; do
 	dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=4K count=1 conv=notrunc \
 		>>$seqres.full 2>/dev/null || _fail "failed to write to test file"
 done
-sync
+_scratch_sync
 echo >> $seqres.full
 echo "Fill in prealloc space; fill holes at offsets:" >> $seqres.full
 for i in `seq 2 2 1023`; do
@@ -74,7 +74,7 @@ for i in `seq 2 2 1023`; do
 	dd if=/dev/zero of=$SCRATCH_MNT/test seek=$i bs=4K count=1 conv=notrunc \
 		>>$seqres.full 2>/dev/null || _fail "failed to fill test file"
 done
-sync
+_scratch_sync
 echo >> $seqres.full
 
 echo "done"
tests/generic/275 +3 -3
index 57de69b80..fbfe15816 100755
@@ -51,9 +51,9 @@ dd if=/dev/zero of=$SCRATCH_MNT/tmp1 bs=256K count=1 >>$seqres.full 2>&1
 
 # Attempt to completely fill fs
 dd if=/dev/zero of=$SCRATCH_MNT/tmp2 bs=1M >>$seqres.full 2>&1
-sync
+_scratch_sync
 dd if=/dev/zero of=$SCRATCH_MNT/tmp3 bs=4K >>$seqres.full 2>&1
-sync
+_scratch_sync
 # Last effort, use O_SYNC
 dd if=/dev/zero of=$SCRATCH_MNT/tmp4 bs=4K oflag=sync >>$seqres.full 2>&1
 # Save space usage info to the full file
@@ -62,7 +62,7 @@ $DF_PROG $SCRATCH_MNT >>$seqres.full 2>&1
 
 # Should leave approx 256k free
 rm -f $SCRATCH_MNT/tmp1
-sync
+_scratch_sync
 echo "Post rm space:" >> $seqres.full
 $DF_PROG $SCRATCH_MNT >>$seqres.full 2>&1
 _freespace=`$DF_PROG -k $SCRATCH_MNT | tail -n 1 | awk '{print $5}'`
tests/generic/276 +1 -1
index 3c3e75df6..dfd2888ce 100755
@@ -54,7 +54,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "CoW and unmount"
-sync
+_scratch_sync
 _dmerror_load_error_table
 $XFS_IO_PROG -d -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $testdir/file2 \
 	>> $seqres.full 2>&1
tests/generic/278 +1 -1
index 5d9778f9f..e303a74ce 100755
@@ -55,7 +55,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "CoW and unmount"
-sync
+_scratch_sync
 _dmerror_load_error_table
 urk=$($XFS_IO_PROG -d -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $testdir/file2 2>&1)
 echo $urk >> $seqres.full
tests/generic/279 +1 -1
index f4dac950c..599b4b6dc 100755
@@ -53,7 +53,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "CoW and unmount"
-sync
+_scratch_sync
 _dmerror_load_error_table
 # Insulate ourselves against bash reporting the SIGBUS when we try to modify
 # the metadata.
tests/generic/281 +1 -1
index 6d48c4b14..474b8b73a 100755
@@ -53,7 +53,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "CoW and unmount"
-sync
+_scratch_sync
 _dmerror_load_error_table
 # Insulate ourselves against bash reporting the SIGBUS when we try to modify
 # the metadata.
tests/generic/282 +1 -1
index b3eb48063..727978130 100755
@@ -53,7 +53,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "CoW and unmount"
-sync
+_scratch_sync
 _dmerror_load_error_table
 # Insulate ourselves against bash reporting the SIGBUS when we try to modify
 # the metadata.
tests/generic/283 +1 -1
index b9104fe64..cdad47a24 100755
@@ -54,7 +54,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "CoW and unmount"
-sync
+_scratch_sync
 _dmerror_load_error_table
 # Insulate ourselves against bash reporting the SIGBUS when we try to modify
 # the metadata.
tests/generic/315 +1 -1
index 83f46655f..cb660ed2d 100755
@@ -41,7 +41,7 @@ fsize=`_get_filesize $TEST_DIR/testfile.$seq`
 
 # Truncate the file size back to 0
 truncate -s 0 $TEST_DIR/testfile.$seq
-sync
+_test_sync
 
 # Preallocated disk space should be released
 avail_done=`df -P $TEST_DIR | awk 'END {print $4}'`
tests/generic/317 +1 -1
index 177d88525..1afec6a75 100755
@@ -73,7 +73,7 @@ _print_numeric_uid
 echo ""
 echo "*** Remounting ***"
 echo ""
-sync
+_scratch_sync
 _scratch_cycle_mount      >>$seqres.full 2>&1 || _fail "remount failed"
 
 _print_numeric_uid
tests/generic/318 +1 -1
index d31ad2b13..033c0611c 100755
@@ -81,7 +81,7 @@ _print_getfacls
 
 echo "*** Remounting ***"
 echo ""
-sync
+_scratch_sync
 _scratch_cycle_mount      >>$seqres.full 2>&1 || _fail "remount failed"
 
 _print_getfacls
tests/generic/325 +1 -1
index 8fbd0d1d9..f85066337 100755
@@ -42,7 +42,7 @@ $XFS_IO_PROG -f -c "pwrite -S 0xff 0 256K" $SCRATCH_MNT/foo | _filter_xfs_io
 # Now sync the file data to disk using 'sync' and not an fsync. This is because
 # in btrfs the first fsync clears the btrfs inode full fsync flag, which must
 # be set when the first msync below happens in order to trigger the bug.
-sync
+_scratch_sync
 
 # Now update the first 4Kb and the last 4Kb of the file, using memory mapped IO
 # because an msync(), since the linux kernel commit
tests/generic/328 +2 -2
index 25e1f2a02..7071ded25 100755
@@ -62,12 +62,12 @@ _report_quota_blocks $SCRATCH_MNT
 
 echo "Try to dio write the whole file"
 _pwrite_byte 0x62 0 $sz $testdir/file1 -d >> $seqres.full
-sync
+_scratch_sync
 _report_quota_blocks $SCRATCH_MNT
 
 echo "Try to write the whole file"
 _pwrite_byte 0x62 0 $sz $testdir/file3 >> $seqres.full
-sync
+_scratch_sync
 _report_quota_blocks $SCRATCH_MNT
 
 # success, all done
tests/generic/329 +1 -1
index ab37e047f..e4300f92c 100755
@@ -57,7 +57,7 @@ md5sum $testdir/file2 | _filter_scratch
 echo "CoW and unmount"
 $XFS_IO_PROG -f -c "pwrite -S 0x63 $bufsize 1" $testdir/file2 >> $seqres.full
 $XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $TEST_DIR/moo >> $seqres.full
-sync
+_scratch_sync
 _dmerror_load_error_table
 $AIO_TEST -a $alignment -f DIRECT -b $bufsize $TEST_DIR/moo $testdir/file2 >> $seqres.full
 _dmerror_load_working_table
tests/generic/330 +1 -1
index 4fa81f991..c67defc48 100755
@@ -51,7 +51,7 @@ md5sum $testdir/file2 | _filter_scratch
 
 echo "CoW and unmount"
 $XFS_IO_PROG -f -c "pwrite -S 0x63 $bufsize 1" $testdir/file2 >> $seqres.full
-sync
+_scratch_sync
 $XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $TEST_DIR/moo >> $seqres.full
 $AIO_TEST -a $alignment -f DIRECT -b $bufsize $TEST_DIR/moo $testdir/file2 >> $seqres.full
 _scratch_cycle_mount
tests/generic/331 +1 -1
index c1949e1b9..fe12ec4eb 100755
@@ -57,7 +57,7 @@ md5sum $testdir/file2 | _filter_scratch
 echo "CoW and unmount"
 $XFS_IO_PROG -f -c "pwrite -S 0x63 $bufsize 1" $testdir/file2 >> $seqres.full
 $XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $TEST_DIR/moo >> $seqres.full
-sync
+_scratch_sync
 _dmerror_load_error_table
 $AIO_TEST -b $bufsize $TEST_DIR/moo $testdir/file2 >> $seqres.full
 $XFS_IO_PROG -c "fdatasync" $testdir/file2
tests/generic/332 +1 -1
index 4a61e4a02..cb15e77bd 100755
@@ -52,7 +52,7 @@ md5sum $testdir/file2 | _filter_scratch
 echo "CoW and unmount"
 $XFS_IO_PROG -f -c "pwrite -S 0x63 $bufsize 1" $testdir/file2 >> $seqres.full
 $XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $TEST_DIR/moo >> $seqres.full
-sync
+_scratch_sync
 $AIO_TEST -b $bufsize $TEST_DIR/moo $testdir/file2 >> $seqres.full
 _scratch_cycle_mount
 
tests/generic/335 +1 -1
index 280cf8538..f287b5150 100755
@@ -38,7 +38,7 @@ mkdir $SCRATCH_MNT/c
 touch $SCRATCH_MNT/a/b/foo
 
 # Make sure everything is durably persisted.
-sync
+_scratch_sync
 
 # Now move our test file into a new parent directory.
 mv $SCRATCH_MNT/a/b/foo $SCRATCH_MNT/c/
tests/generic/336 +1 -1
index 478081978..06391a93f 100755
@@ -43,7 +43,7 @@ ln $SCRATCH_MNT/a/foo $SCRATCH_MNT/b/foo_link
 touch $SCRATCH_MNT/b/bar
 
 # Make sure everything is durably persisted.
-sync
+_scratch_sync
 
 # Now delete one of the hard links of file foo and move file bar into c/
 unlink $SCRATCH_MNT/b/foo_link
tests/generic/341 +1 -1
index bea36e265..80fdcbac7 100755
@@ -36,7 +36,7 @@ mkdir -p $SCRATCH_MNT/a/x
 $XFS_IO_PROG -f -c "pwrite -S 0xaf 0 32K" $SCRATCH_MNT/a/x/foo | _filter_xfs_io
 $XFS_IO_PROG -f -c "pwrite -S 0xba 0 32K" $SCRATCH_MNT/a/x/bar | _filter_xfs_io
 # Make sure everything done so far is durably persisted.
-sync
+_scratch_sync
 
 echo "File digests before power failure:"
 md5sum $SCRATCH_MNT/a/x/foo | _filter_scratch
tests/generic/342 +1 -1
index 3f5aca673..a7aca860b 100755
@@ -40,7 +40,7 @@ _mount_flakey
 mkdir $SCRATCH_MNT/a
 $XFS_IO_PROG -f -c "pwrite -S 0xf1 0 16K" $SCRATCH_MNT/a/foo | _filter_xfs_io
 # Make sure everything done so far is durably persisted.
-sync
+_scratch_sync
 
 # Now rename file foo to bar and create a new file named foo under the same
 # directory. After a power failure we must see the two files.
tests/generic/343 +1 -1
index 6659f1988..97ff4f984 100755
@@ -41,7 +41,7 @@ mkdir $SCRATCH_MNT/y/z
 touch $SCRATCH_MNT/y/foo2
 
 # Make sure everything is durably persisted.
-sync
+_scratch_sync
 
 # Now add a link to foo at directory x, move directory z and file foo2 from
 # directory y to directory x and fsync foo's inode. We expect that after a
tests/generic/347 +1 -1
index 7e1ea5d90..5c0e3f949 100755
@@ -36,7 +36,7 @@ _workout()
 		$XFS_IO_PROG -f -c "pwrite -W 0 1M" $SCRATCH_MNT/file$I &>/dev/null
 	done
 
-	sync
+	_scratch_sync
 
 	_dmthin_grow  $GROW_SIZE
 
tests/generic/348 +1 -1
index eb0587d20..1905a6e6a 100755
@@ -35,7 +35,7 @@ _mount_flakey
 
 mkdir $SCRATCH_MNT/testdir1
 # Make sure it's durably persisted.
-sync
+_scratch_sync
 
 # Create our symlinks and fsync their parent directories.
 # We test both the case where the parent directory is new (not yet durably
tests/generic/353 +1 -1
index ac5b97880..2114a74cb 100755
@@ -48,7 +48,7 @@ cmp -s $extmap1 $extmap2 || echo "mismatched extent maps before sync"
 
 # sync and recheck, to make sure the fiemap doesn't change just
 # due to sync
-sync
+_scratch_sync
 $XFS_IO_PROG -c "fiemap -v" $file1 | _filter_fiemap_flags > $extmap1
 $XFS_IO_PROG -c "fiemap -v" $file2 | _filter_fiemap_flags > $extmap2
 
tests/generic/376 +1 -1
index cecd4e0d7..17a5f290b 100755
@@ -36,7 +36,7 @@ _mount_flakey
 mkdir $SCRATCH_MNT/dir
 touch $SCRATCH_MNT/dir/foo
 # Make sure everything is durably persisted.
-sync
+_scratch_sync
 mv $SCRATCH_MNT/dir/foo $SCRATCH_MNT/dir/bar
 touch $SCRATCH_MNT/dir/foo
 $XFS_IO_PROG -c "fsync" $SCRATCH_MNT/dir/bar
tests/generic/382 +1 -1
index 162da41d4..1634d85d2 100755
@@ -56,7 +56,7 @@ do_test()
 	for ((i=0; i<30; i++));do
 		_user_do "echo -n > ${SCRATCH_MNT}/file${i}"
 	done
-	sync
+	_scratch_sync
 
 	rm -f ${SCRATCH_MNT}/* >/dev/null 2>&1
 }
tests/generic/387 +1 -1
index 544e7552a..c3b972496 100755
@@ -37,7 +37,7 @@ done
 
 # consume all remaining free space
 dd if=/dev/zero of=$SCRATCH_MNT/space >/dev/null 2>&1
-sync
+_scratch_sync
 
 # fs is full now and fs internal operations may need some free space, for
 # example, in btrfs, transaction will need to reserve space first, so here free
tests/generic/391 +1 -1
index d19c48262..e41f644d3 100755
@@ -38,7 +38,7 @@ for ((off = 0; off < num_extents * extent_size; off += extent_size)); do
 done
 
 # To reproduce the Btrfs bug, the extent map must not be cached in memory.
-sync
+_test_sync
 echo 3 > /proc/sys/vm/drop_caches
 
 "$here/src/dio-interleaved" "$extent_size" "$num_extents" "$testfile"
tests/generic/409 +1 -1
index ab7e7eb55..ac1b14ad6 100755
@@ -58,7 +58,7 @@ fs_stress()
 		       -f getdents=1 \
 		       -f fiemap=1 \
 		       -d $target
-	sync
+	_sync_fs $target
 }
 
 # prepare some mountpoint dir
tests/generic/410 +1 -1
index f0f0921be..e0d0c57eb 100755
@@ -66,7 +66,7 @@ fs_stress()
 		       -f getdents=1 \
 		       -f fiemap=1 \
 		       -d $target
-	sync
+	_sync_fs $target
 }
 
 # prepare some mountpoint dir
tests/generic/411 +1 -1
index 3b55b4f91..0a80554cd 100755
@@ -49,7 +49,7 @@ fs_stress()
 		       -f getdents=1 \
 		       -f fiemap=1 \
 		       -d $target
-	sync
+	_sync_fs $target
 }
 
 # prepare some mountpoint dir
tests/generic/416 +1 -1
index 31a85c83f..f6a516fb6 100755
@@ -52,7 +52,7 @@ _scratch_remount
 # remove all files with odd file names, which should free near half
 # of the space
 rm $SCRATCH_MNT/*[13579]
-sync
+_scratch_sync
 
 # We should be able to write at least 1/8 of the whole fs size
 # The number 1/8 is for btrfs, which only has about 47M for data.
tests/generic/422 +2 -2
index 71a0e5a86..ad37b9cea 100755
@@ -34,7 +34,7 @@ $XFS_IO_PROG -f \
 touch $SCRATCH_MNT/foo4
 
 # Make sure everything done so far is durably persisted.
-sync
+_scratch_sync
 
 # Now overwrite the extent of the first file.
 $XFS_IO_PROG -c "pwrite -S 0xff 0 64K" $SCRATCH_MNT/foo1 | _filter_xfs_io
@@ -77,7 +77,7 @@ space_used > $SCRATCH_MNT/$seq.before
     cat $SCRATCH_MNT/$seq.before
 ) >> $seqres.full
 
-sync
+_scratch_sync
 
 # We expect the same file sizes reported by 'du' after writeback finishes.
 
tests/generic/425 +1 -1
index 3367a9567..352ae32d2 100755
@@ -52,7 +52,7 @@ while [ $i -lt $max_attrs ]; do
 	$SETFATTR_PROG -n "user.$n" -v "$n" $testfile > $seqres.full 2>&1 || break
 	i=$((i + 1))
 done
-sync
+_scratch_sync
 
 echo "Check attr extent counts"
 f1=$(_count_attr_extents $testfile)
tests/generic/461 +1 -1
index 95dd6a34d..a2927f85e 100755
@@ -27,7 +27,7 @@ load_dir=$SCRATCH_MNT/test
 
 _run_fsstress_bg -n10000000 -p $PROCS -d $load_dir
 sleep $SLEEP_TIME
-sync
+_scratch_sync
 
 # now shutdown and unmount
 sleep 5
tests/generic/474 +2 -2
index 2fa087a87..961d0f06f 100755
@@ -36,7 +36,7 @@ _scratch_mount
 # accelerates syncfs on testing filesystem so that test case can finish
 # in 30 seconds.
 
-sync
+_scratch_sync
 
 # Large fs has a huge size .use_space file, will take long time on running
 # fssum $SCRATCH_MNT. So change the target path to a sub-dir of $SCRATCH_MNT.
@@ -47,7 +47,7 @@ $XFS_IO_PROG -f -c "pwrite 0 4K" $localdir/testfile >/dev/null 2>&1
 # fssum used for comparing checksum of test file(data & metedata),
 # exclude checking about atime, block structure, open error.
 $FSSUM_PROG -ugomAcdES -f -w $tmp.fssum $localdir
-$XFS_IO_PROG -c "syncfs" $localdir/testfile >/dev/null 2>&1
+_sync_fs $localdir/testfile
 _scratch_shutdown
 _scratch_cycle_mount
 $FSSUM_PROG -r $tmp.fssum $localdir
tests/generic/479 +1 -1
index 7f4aab4e7..650c921b8 100755
@@ -53,7 +53,7 @@ run_test()
 		_fail "Invalid file type argument: $file_type"
 	esac
 	# Make sure everything done so far is durably persisted.
-	sync
+	_scratch_sync
 
 	# Create a file and fsync it just to create a journal/log. This file
 	# must be in the same directory as our special file "foo".
tests/generic/480 +1 -1
index 975c990ff..6c599446b 100755
@@ -38,7 +38,7 @@ touch $SCRATCH_MNT/testdir/foo
 ln $SCRATCH_MNT/testdir/foo $SCRATCH_MNT/testdir/bar
 
 # Make sure everything done so far is durably persisted.
-sync
+_scratch_sync
 
 # Now remove of the links of our file and create a new file with the same name
 # and in the same parent directory, and finally fsync this new file.
tests/generic/483 +1 -1
index 60f9b1097..a71f96ad0 100755
@@ -57,7 +57,7 @@ $XFS_IO_PROG -f -c "pwrite -S 0xf1 0 256K" \
 	     $SCRATCH_MNT/baz >/dev/null
 
 # Make sure everything done so far is durably persisted.
-sync
+_scratch_sync
 
 # Allocate an extent beyond the size of the first test file and fsync it.
 $XFS_IO_PROG -c "falloc -k 256K 1M"\
tests/generic/489 +1 -1
index 62aa45a86..e76055fa4 100755
@@ -39,7 +39,7 @@ $SETFATTR_PROG -n user.xa3 -v test $SCRATCH_MNT/foobar
 $XFS_IO_PROG -c "fsync" $SCRATCH_MNT/foobar
 
 # Call sync to commit all fileystem metadata.
-sync
+_scratch_sync
 
 $XFS_IO_PROG -c "pwrite -S 0xea 0 64K" \
 	     -c "fsync" \
tests/generic/502 +1 -1
index f6374f677..f488bd06c 100755
@@ -46,7 +46,7 @@ touch $SCRATCH_MNT/testdir/foo
 ln $SCRATCH_MNT/testdir/foo $SCRATCH_MNT/testdir/bar
 
 # Make sure everything done so far is durably persisted.
-sync
+_scratch_sync
 
 # Now rename the first hard link (foo) to a new name and rename the second hard
 # link (bar) to the old name of the first hard link (foo).
tests/generic/505 +1 -1
index a25967352..eb44573b4 100755
@@ -40,7 +40,7 @@ do_check()
 	touch $testfile
 
 	if [ "$1" == "sync" ]; then
-		sync
+		_scratch_sync
 	fi
 
 	chown 100 $testfile
tests/generic/506 +1 -1
index 9a41e5634..397e93d07 100755
@@ -45,7 +45,7 @@ do_check()
 	touch $testfile
 
 	if [ "$1" == "sync" ]; then
-		sync
+		_scratch_sync
 	fi
 
 	$XFS_IO_PROG -x -c "chproj 100" $testfile
tests/generic/507 +1 -1
index 6678e50f7..b9e58067e 100755
@@ -59,7 +59,7 @@ do_check()
 
 	if [ "$2" == "sync" ]; then
 		echo "sync" >> $seqres.full
-		sync
+		_scratch_sync
 	fi
 
 	echo "Test chattr +$1" >> $seqres.full
tests/generic/508 +1 -1
index 4e234160d..02023cf08 100755
@@ -44,7 +44,7 @@ do_check()
 	touch $testfile
 
 	if [ "$1" == "sync" ]; then
-		sync
+		_scratch_sync
 	fi
 
 	before=`$XFS_IO_PROG -f $testfile -c "statx -v" | grep btime`
tests/generic/510 +1 -1
index 5ea0e0677..abf18f1bf 100755
@@ -39,7 +39,7 @@ mkdir $SCRATCH_MNT/testdir/B
 touch $SCRATCH_MNT/testdir/B/bar
 
 # Make sure everything done so far is durably persisted.
-sync
+_scratch_sync
 
 # Now move our file bar from directory B to directory A and then replace
 # directory B with directory A, also renaming directory A to B. Finally
tests/generic/520 +1 -1
index 00fa050e2..f2523fca2 100755
@@ -125,7 +125,7 @@ test_link_sync()
 	mkdir -p "${dest%/*}"
 	touch $src
 	ln $src $dest
-	sync
+	_scratch_sync
 	before=`stat "$stat_opt" $src`
 
 	check_consistency $src 0
tests/generic/526 +1 -1
index 7d47cf111..af77ccaff 100755
@@ -43,7 +43,7 @@ echo -n "foo" > $SCRATCH_MNT/testdir/fname1
 echo -n "hello" > $SCRATCH_MNT/testdir/fname2
 
 # Make sure everything done so far is durably persisted.
-sync
+_scratch_sync
 
 # Rename and link files such that one new name corresponds to the name of
 # another renamed file and one new file has the old name of one of the renamed
tests/generic/527 +1 -1
index 2ba1f319b..90555077f 100755
@@ -49,7 +49,7 @@ ln $SCRATCH_MNT/testdir2/zz $SCRATCH_MNT/testdir2/zz_link
 echo -n "hello" > $SCRATCH_MNT/testdir2/a
 
 # Make sure everything done so far is durably persisted.
-sync
+_scratch_sync
 
 # Rename, remove and link files such that one new name corresponds to the name
 # of a deleted file and one new file has the old name of the renamed file. Then
tests/generic/535 +1 -1
index c2338da6b..98e2f2384 100755
@@ -59,7 +59,7 @@ do_check()
 	echo "Test chmod $target" >> $seqres.full
 
 	chmod 777 $target
-	sync
+	_scratch_sync
 
 	chmod 755 $target
 	$XFS_IO_PROG $target -c "fsync"
tests/generic/546 +1 -1
index 972886858..ab4ea657e 100755
@@ -47,7 +47,7 @@ _pwrite_byte 0x00 0 512m "$SCRATCH_MNT/padding" >> $seqres.full 2>&1
 
 # Sync to ensure that padding file reach disk so that at log recovery we
 # still have no data space
-sync
+_scratch_sync
 
 # This should not fail
 _pwrite_byte 0xcd 1m 16m "$SCRATCH_MNT/foobar" >> $seqres.full
tests/generic/556 +1 -1
index d3396ae3b..279ff58b3 100755
@@ -366,7 +366,7 @@ test_toplevel_dir_rename()
 	# 'f2fs: don't use casefolded comparison for "." and ".."'.
 	mkdir ${dir}
 	_casefold_set_attr ${dir}
-	sync
+	_scratch_sync
 	echo 2 > /proc/sys/vm/drop_caches
 	mv ${dir} ${dir}.new
 }
tests/generic/579 +1 -1
index 4187ab0f0..d5fab4892 100755
@@ -86,7 +86,7 @@ done
 (
 	while [ ! -e $tmp.done ]; do
 		sleep 2.$((RANDOM % 100))
-		sync && echo 3 > /proc/sys/vm/drop_caches
+		_scratch_sync && echo 3 > /proc/sys/vm/drop_caches
 	done
 ) &
 
tests/generic/589 +1 -1
index 969a8ac61..b5860070b 100755
@@ -48,7 +48,7 @@ fs_stress()
 	local target=$1
 
 	_run_fsstress -n 50 -p 3 -d $target
-	sync
+	_sync_fs $target
 }
 
 # prepare some mountpoint dir
tests/generic/599 +1 -1
index e1454584f..a8eb93709 100755
@@ -24,7 +24,7 @@ _scratch_mount
 
 localdir=$SCRATCH_MNT/dir
 mkdir $localdir
-sync
+_scratch_sync
 
 # fssum used for comparing checksum of test file(data & metedata),
 # exclude checking about atime, block structure, open error.
tests/generic/610 +1 -1
index 22f28c6df..00a676e6e 100755
@@ -30,7 +30,7 @@ $here/src/punch-alternating $SCRATCH_MNT/foobar >>$seqres.full
 
 # For btrfs, trigger a transaction commit to force metadata COW for the
 # following fallocate zero range operation.
-sync
+_scratch_sync
 
 $XFS_IO_PROG -c "fzero 0 100M" $SCRATCH_MNT/foobar
 
tests/generic/628 +2 -2
index 8fd44364c..6f891abc2 100755
@@ -54,7 +54,7 @@ $XFS_IO_PROG -f -c 'pwrite -S 0x59 0 1m -b 1m' $SCRATCH_MNT/c >> $seqres.full
 _cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/e
 _cp_reflink $SCRATCH_MNT/c $SCRATCH_MNT/d
 touch $SCRATCH_MNT/b
-sync
+_scratch_sync
 
 # Test that reflink forces dirty data/metadata to disk when destination file
 # opened with O_SYNC
@@ -81,7 +81,7 @@ rm -f $SCRATCH_MNT/b $SCRATCH_MNT/d
 _cp_reflink $SCRATCH_MNT/c $SCRATCH_MNT/d
 touch $SCRATCH_MNT/b
 chattr +S $SCRATCH_MNT/b $SCRATCH_MNT/d
-sync
+_scratch_sync
 
 # Test that reflink forces dirty data/metadata to disk when destination file
 # has the sync iflag set
tests/generic/629 +2 -2
index 92b7839a0..1088e50e4 100755
@@ -50,7 +50,7 @@ md5sum $SCRATCH_MNT/0 | _filter_scratch
 # Set up initial files for copy test
 $XFS_IO_PROG -f -c 'pwrite -S 0x58 0 1m -b 1m' $SCRATCH_MNT/a >> $seqres.full
 touch $SCRATCH_MNT/b
-sync
+_scratch_sync
 
 # Test that unaligned copy file range forces dirty data/metadata to disk when
 # destination file opened with O_SYNC
@@ -66,7 +66,7 @@ md5sum $SCRATCH_MNT/a $SCRATCH_MNT/b | _filter_scratch
 rm -f $SCRATCH_MNT/b
 touch $SCRATCH_MNT/b
 chattr +S $SCRATCH_MNT/b
-sync
+_scratch_sync
 
 # Test that unaligned copy file range forces dirty data/metadata to disk when
 # destination file has the sync iflag set
tests/generic/640 +1 -1
index 038dde975..c3b33746e 100755
@@ -38,7 +38,7 @@ mkdir $SCRATCH_MNT/B
 echo -n "hello world" > $SCRATCH_MNT/A/foo
 
 # Persist everything done so far.
-sync
+_scratch_sync
 
 # Add some new file to directory A and fsync the directory.
 touch $SCRATCH_MNT/A/bar
tests/generic/673 +1 -1
index 1230b51e1..8f6def9c7 100755
@@ -28,7 +28,7 @@ setup_testfile() {
 	_pwrite_byte 0x58 0 1m $SCRATCH_MNT/a >> $seqres.full
 	_pwrite_byte 0x57 0 1m $SCRATCH_MNT/b >> $seqres.full
 	chmod a+r $SCRATCH_MNT/b
-	sync
+	_scratch_sync
 }
 
 commit_and_check() {
tests/generic/674 +1 -1
index 2e395290a..1b711f27f 100755
@@ -29,7 +29,7 @@ setup_testfile() {
 	_pwrite_byte 0x58 0 1m $SCRATCH_MNT/a >> $seqres.full
 	_pwrite_byte 0x58 0 1m $SCRATCH_MNT/b >> $seqres.full
 	chmod a+r $SCRATCH_MNT/b
-	sync
+	_scratch_sync
 }
 
 commit_and_check() {
tests/generic/675 +1 -1
index 7c062db81..e66de84b5 100755
@@ -33,7 +33,7 @@ setup_testfile() {
 	_pwrite_byte 0x57 0 1m $SCRATCH_MNT/b >> $seqres.full
 	chmod a+rwx $SCRATCH_MNT/a $SCRATCH_MNT/b
 	$SETCAP_PROG cap_setgid,cap_setuid+ep $SCRATCH_MNT/a
-	sync
+	_scratch_sync
 }
 
 commit_and_check() {
tests/generic/677 +1 -1
index f2081c34c..86099454f 100755
@@ -62,7 +62,7 @@ $XFS_IO_PROG -c "falloc -k 20M 1M" $SCRATCH_MNT/foo
 # On btrfs this commits the current transaction and it makes all the created
 # extents to have a generation lower than the generation of the transaction used
 # by the next write and fsync.
-sync
+_scratch_sync
 
 # Now overwrite only the first extent.
 # On btrfs, due to COW (both data and metadata), that results in modifying only
tests/generic/683 +1 -1
index cf1ebbc44..cc9a9786b 100755
@@ -36,7 +36,7 @@ chmod a+rw $junk_dir/
 setup_testfile() {
 	rm -f $junk_file
 	_pwrite_byte 0x58 0 192k $junk_file >> $seqres.full
-	sync
+	_test_sync
 }
 
 commit_and_check() {
tests/generic/684 +1 -1
index e1eb4e118..2ca036fe5 100755
@@ -36,7 +36,7 @@ chmod a+rw $junk_dir/
 setup_testfile() {
 	rm -f $junk_file
 	_pwrite_byte 0x58 0 192k $junk_file >> $seqres.full
-	sync
+	_test_sync
 }
 
 commit_and_check() {
tests/generic/685 +1 -1
index bec2c5a0b..de07a798a 100755
@@ -36,7 +36,7 @@ chmod a+rw $junk_dir/
 setup_testfile() {
 	rm -f $junk_file
 	_pwrite_byte 0x58 0 192k $junk_file >> $seqres.full
-	sync
+	_test_sync
 }
 
 commit_and_check() {
tests/generic/686 +1 -1
index efcc3f610..fc6761fe6 100755
@@ -36,7 +36,7 @@ chmod a+rw $junk_dir/
 setup_testfile() {
 	rm -f $junk_file
 	_pwrite_byte 0x58 0 192k $junk_file >> $seqres.full
-	sync
+	_test_sync
 }
 
 commit_and_check() {
tests/generic/687 +1 -1
index e05f0fdcd..82dce88b8 100755
@@ -36,7 +36,7 @@ chmod a+rw $junk_dir/
 setup_testfile() {
 	rm -f $junk_file
 	_pwrite_byte 0x58 0 192k $junk_file >> $seqres.full
-	sync
+	_test_sync
 }
 
 commit_and_check() {
tests/generic/688 +1 -1
index 9c19356d8..e491d5cf3 100755
@@ -40,7 +40,7 @@ setup_testfile() {
 	touch $junk_file
 	chmod a+rwx $junk_file
 	$SETCAP_PROG cap_setgid,cap_setuid+ep $junk_file
-	sync
+	_test_sync
 }
 
 commit_and_check() {
tests/generic/690 +1 -1
index ba0da30d3..ef5bd1983 100755
@@ -52,7 +52,7 @@ _mount_flakey
 mkdir "$SCRATCH_MNT"/testdir
 
 # Commit the current transaction and persist the directory.
-sync
+_scratch_sync
 
 # Create a file in the test directory, so that the next fsync on the directory
 # actually does something (it logs the directory).
tests/generic/694 +1 -1
index a68fc5230..e5c4f88c5 100755
@@ -39,7 +39,7 @@ fi
 
 # make sure indirect block reservations and other temporary block reservations
 # are released before sampling i_blocks
-sync $junk_file
+_test_sync
 
 iblocks=`stat -c '%b' $junk_file`
 
tests/generic/695 +1 -1
index 8179d59a0..df81fdb7f 100755
@@ -51,7 +51,7 @@ $XFS_IO_PROG -f -c "truncate 12M" \
 	     $SCRATCH_MNT/foobar | _filter_xfs_io
 
 # Persist everything, commit the filesystem's transaction.
-sync
+_scratch_sync
 
 # Now punch two holes in the file:
 #
tests/generic/703 +1 -1
index 8ee1d558c..2bace19d6 100755
@@ -70,7 +70,7 @@ fi
 $XFS_IO_PROG -c "falloc 0 256M" $test_file
 
 # Persist everything, make sure the file exists after power failure.
-sync
+_scratch_sync
 
 echo -e "Running fio with config:\n" >> $seqres.full
 cat $fio_config >> $seqres.full
tests/generic/716 +1 -1
index a2b86225a..d3923dc1d 100755
@@ -36,7 +36,7 @@ filesnap() {
 mkfile() {
 	rm -f $dir/a
 	_pwrite_byte 0x58 0 $((blksz * nrblks)) $dir/a >> $seqres.full
-	sync
+	_test_sync
 }
 
 dir=$TEST_DIR/test-$seq
tests/generic/718 +1 -1
index ed3a1fee5..4d322b224 100755
@@ -31,7 +31,7 @@ nrblks=64
 # Create some 4M files to test exchangerange
 _pwrite_byte 0x58 0 $((blksz * nrblks)) $dir/a >> $seqres.full
 _pwrite_byte 0x59 0 $((blksz * nrblks)) $dir/b >> $seqres.full
-sync
+_test_sync
 md5sum $dir/a $dir/b | _filter_test_dir
 
 # Set FSIZE to twice the blocksize (IOWs, 128k)
tests/generic/719 +1 -1
index c82cf9552..9cd6eed91 100755
@@ -34,7 +34,7 @@ filesnap() {
 mkfile() {
 	rm -f $dir/a
 	_pwrite_byte 0x58 0 $((blksz * nrblks)) $dir/a >> $seqres.full
-	sync
+	_test_sync
 }
 
 dir=$TEST_DIR/test-$seq
tests/generic/721 +1 -1
index 98505aac9..a9565f189 100755
@@ -35,7 +35,7 @@ mkfile() {
 	rm -f $dir/a
 	_pwrite_byte 0x58 0 $((blksz * nrblks)) $dir/a >> $seqres.full
 	chown $qa_user $dir/a $dir/
-	sync
+	_test_sync
 }
 
 dir=$TEST_DIR/test-$seq
tests/generic/725 +1 -1
index e1db83b5e..3a9005ddf 100755
@@ -31,7 +31,7 @@ _require_congruent_file_oplen $SCRATCH_MNT 65536
 
 # Create original file
 _pwrite_byte 0x58 0 1m $SCRATCH_MNT/a >> $seqres.full
-sync
+_scratch_sync
 md5sum $SCRATCH_MNT/a | _filter_scratch
 
 # Test atomic scatter-gather file commits.
tests/generic/726 +1 -1
index 00071b1d0..131ac5b50 100755
@@ -35,7 +35,7 @@ chmod a+rw $SCRATCH_MNT/
 setup_testfile() {
 	rm -f $SCRATCH_MNT/a
 	_pwrite_byte 0x58 0 1m $SCRATCH_MNT/a >> $seqres.full
-	sync
+	_scratch_sync
 }
 
 commit_and_check() {
tests/generic/727 +1 -1
index dbed1d45c..ee7ed9760 100755
@@ -42,7 +42,7 @@ setup_testfile() {
 	_pwrite_byte 0x57 0 1m $SCRATCH_MNT/b >> $seqres.full
 	chmod a+rw $SCRATCH_MNT/a $SCRATCH_MNT/b
 	$SETCAP_PROG cap_setgid,cap_setuid+ep $SCRATCH_MNT/a
-	sync
+	_scratch_sync
 }
 
 commit_and_check() {
tests/generic/735 +1 -1
index 1aeeb9a42..9bbdf3a17 100755
@@ -42,7 +42,7 @@ $XFS_IO_PROG -f -c "finsert 1M ${finsert_len}" "${SCRATCH_MNT}/file" >> $seqres.
 # Filling up the free space ensures that the pre-allocated space is the reserved space.
 nr_free=$(stat -f -c '%f' ${SCRATCH_MNT})
 _fill_fs $((nr_free * file_blksz)) ${SCRATCH_MNT}/fill $file_blksz 0 >> $seqres.full 2>&1
-sync
+_scratch_sync
 
 # Remove reserved space to gain free space for allocation
 rm -f ${SCRATCH_MNT}/tmp
tests/generic/738 +1 -1
index 4da2d8879..6f1ea7f8f 100755
@@ -36,7 +36,7 @@ $XFS_IO_PROG -fc "pwrite 0 64k" $SCRATCH_MNT/testfile >> $seqres.full
 for i in $(seq 0 1024); do
 	create_eof_block_file $SCRATCH_MNT/$i
 done
-sync
+_scratch_sync
 xfs_freeze -f $SCRATCH_MNT
 
 # This will hang if bug reproduces
tests/generic/745 +2 -2
index bed43578a..62624b15b 100755
@@ -51,7 +51,7 @@ _mount_flakey
 # Create the test file with some initial data and make sure everything is
 # durably persisted.
 $XFS_IO_PROG -f -c "pwrite -S 0xaa 0 32k" $SCRATCH_MNT/foo | _filter_xfs_io
-sync
+_scratch_sync
 
 # Add many small xattrs to our file.
 # We create such a large amount because it's needed to trigger the issue found
@@ -66,7 +66,7 @@ done
 
 # Sync the filesystem to force a commit of the current btrfs transaction, this
 # is a necessary condition to trigger the bug on btrfs.
-sync
+_scratch_sync
 
 # Now update our file's data and fsync the file.
 # After a successful fsync, if the fsync log/journal is replayed we expect to
tests/generic/746 +2 -2
index 651affe07..5734f254d 100755
@@ -170,7 +170,7 @@ for i in `seq 1 10`; do
 done
 
 # Get reference fiemap, this can contain i.e. uninitialized inode table
-sync
+_sync_fs $loop_mnt
 get_holes > $fiemap_ref
 
 # Delete some files
@@ -184,7 +184,7 @@ echo "done."
 
 echo -n "Detecting interesting holes in image..."
 # Get after-trim fiemap
-sync
+_sync_fs $loop_mnt
 get_holes > $fiemap_after
 echo "done."
 
tests/generic/747 +2 -2
index b92098f9f..448341866 100755
@@ -57,7 +57,7 @@ _used_percent() {
 _delete_random_file() {
 	local to_delete=$(find ${SCRATCH_MNT} -type f | shuf | head -1)
 	rm $to_delete
-	sync ${SCRATCH_MNT}
+	_scratch_sync
 }
 
 _get_random_fsz() {
@@ -110,7 +110,7 @@ echo "Starting mixed write/delete test using buffered IO"
 _mixed_write_delete ""
 
 echo "Syncing"
-sync ${SCRATCH_MNT}/*
+_scratch_sync
 
 echo "Done, all good"
 
tests/generic/749 +2 -2
index 8b2c7feff..fc7477380 100755
@@ -100,7 +100,7 @@ do_mmap_tests()
 	$XFS_IO_PROG -f -c "pwrite -S 0xaa -b 512 $offset $len" \
 		$test_file >> $seqres.full
 
-	sync
+	_scratch_sync
 	new_filelen=$(_get_filesize $test_file)
 	map_len=$(_round_up_to_page_boundary $new_filelen)
 	csum_orig="$(_md5_checksum $test_file)"
@@ -141,7 +141,7 @@ do_mmap_tests()
 			-c "mmap -w 0 $map_len" \
 			-c "mwrite $new_filelen $zero_filled_data_len" \
 			-c "munmap"
-		sync
+		_scratch_sync
 		csum_post="$(_md5_checksum $test_file)"
 		if [[ "$csum_orig" != "$csum_post" ]]; then
 			let failed=$failed+1
tests/xfs/016 +2 -2
index 1abc06c42..f8076cc8d 100755
@@ -101,9 +101,9 @@ _log_traffic()
     while [ $count -ge 0 ]
     do
 	touch $out
-	sync
+	_scratch_sync
 	rm $out
-	sync
+	_scratch_sync
 	let "count = count - 1"
     done
 
tests/xfs/052 +1 -1
index adeab5380..af6e938e0 100755
@@ -54,7 +54,7 @@ fi
 
 # create 100 (fs-blocksize) blocks
 _file_as_id $SCRATCH_MNT/foo $id $type $dbsize 220
-sync
+_scratch_sync
 
 # set limit at 1001 (1k) blocks
 bsoft=1001
tests/xfs/110 +1 -1
index 0c8c87ea9..fc51d183f 100755
@@ -43,7 +43,7 @@ do
 done
 wait
 
-sync
+_scratch_sync
 
 I=10030585
 E=10030599
tests/xfs/118 +2 -2
index 73e429384..b5a1eabf7 100755
@@ -38,7 +38,7 @@ echo "Silence is golden"
 # preallocation added.  Let's say... 64k free chunks.
 
 $XFS_IO_PROG -fs -c "falloc 0 40000k" $SCRATCH_MNT/fill >> $seqres.full 2>&1
-sync
+_scratch_sync
 
 dd if=/dev/zero of=$SCRATCH_MNT/remainder oflag=direct > /dev/null 2>&1
 
@@ -55,7 +55,7 @@ for I in `seq 1 64`; do
 						>> $seqres.full 2>&1
 done
 # sync to get extents on disk so fsr sees them
-sync
+_scratch_sync
 
 # Free up some space for defragmentation temp file
 rm -f $SCRATCH_MNT/fill
tests/xfs/119 +1 -1
index 334d06934..7c1cd6a50 100755
@@ -30,7 +30,7 @@ _require_scratch
 _require_freeze
 
 # this may hang
-sync
+_scratch_sync
 
 export MKFS_OPTIONS="-l version=2,su=64k"
 logblks=$(_scratch_find_xfs_min_logblocks)
tests/xfs/128 +1 -1
index 0f1905295..0bfda0e06 100755
@@ -42,7 +42,7 @@ blksz_factor=$((blksz / real_blksz))
 # preallocate space to try to produce a single extent.
 $XFS_IO_PROG -f -c "falloc 0 $((blks * blksz))" $testdir/file1 >> $seqres.full
 _pwrite_byte 0x61 0 $((blks * blksz)) $testdir/file1 >> $seqres.full
-sync
+_scratch_sync
 
 nextents=$($XFS_IO_PROG -c 'stat' $testdir/file1 | grep 'fsxattr.nextents' | awk '{print $3}')
 
tests/xfs/133 +1 -1
index 39f4f74ae..c9bae5d92 100755
@@ -49,7 +49,7 @@ fi
 echo "Remount, try to append"
 _scratch_mount
 dd if=/dev/zero of=$testdir/a bs=512 count=1 oflag=append conv=notrunc >> $seqres.full 2>&1 || echo "Write did not succeed (ok)."
-sync
+_scratch_sync
 
 # success, all done
 status=0
tests/xfs/134 +1 -1
index b86265211..77505718a 100755
@@ -52,7 +52,7 @@ fi
 echo "Remount, try to append"
 _scratch_mount
 dd if=/dev/zero of=$testdir/a bs=512 count=1 oflag=direct,append conv=notrunc >> $seqres.full 2>&1 || echo "Write did not succeed (ok)."
-sync
+_scratch_sync
 
 # success, all done
 status=0
tests/xfs/161 +1 -1
index 948121c05..e13a646a5 100755
@@ -51,7 +51,7 @@ lim_bhard=$(( 100 * blksz ))
 # Force the block counters for uid 1 and 2 above zero
 _pwrite_byte 0x61 0 $filesz $SCRATCH_MNT/a >> $seqres.full
 _pwrite_byte 0x61 0 $filesz $SCRATCH_MNT/b >> $seqres.full
-sync
+_scratch_sync
 chown 1 $SCRATCH_MNT/a
 chown 2 $SCRATCH_MNT/b
 
tests/xfs/176 +1 -1
index 8d58590fd..e3fa87c43 100755
@@ -151,7 +151,7 @@ for ((ino = icluster_ino; ino < icluster_ino + XFS_INODES_PER_CHUNK; ino++)); do
 	find $SCRATCH_MNT/urk/ -inum "$ino" -print0 | xargs -r -0 mv -t $SCRATCH_MNT/save/
 done
 rm -rf $SCRATCH_MNT/urk/ $SCRATCH_MNT/save/*/*
-sync
+_scratch_sync
 $XFS_IO_PROG -c 'fsmap -vvvvv' $SCRATCH_MNT &>> $seqres.full
 
 # Propose shrinking the filesystem such that the end of the fs ends up in the
tests/xfs/195 +1 -1
index f965d9c38..9d23c32dc 100755
@@ -56,7 +56,7 @@ $CHATTR_PROG +d $TEST_DIR/d/t
 _do_dump
 
 echo "Dump exclude flag set, after sync (should be skipped)"
-sync
+_test_sync
 _do_dump
 
 # success, all done
tests/xfs/201 +2 -2
index 60cc84ed2..ffd05f8a9 100755
@@ -64,8 +64,8 @@ do_pwrite 5000 6455 $min_align
 do_pwrite 1728 4999 $min_align
 do_pwrite 0 1727 $min_align
 
-sync
-sync
+_scratch_sync
+_scratch_sync
 
 # and truncate it again
 > $SCRATCH_MNT/bigfile
tests/xfs/212 +1 -1
index 54ece22ab..51d211164 100755
@@ -52,7 +52,7 @@ echo "CoW and leave leftovers"
 $XFS_IO_PROG -f -c "extsize" $testdir/file2 >> $seqres.full
 $XFS_IO_PROG -f -c "pwrite -S 0x63 $((filesize - 1)) 1" -c "fsync" $testdir/file2 >> $seqres.full
 $XFS_IO_PROG -f -c "pwrite -S 0x63 $((filesize - 1)) 1" -c "fsync" $testdir/file2.chk >> $seqres.full
-sync
+_scratch_sync
 
 echo "Crash and recover"
 $XFS_IO_PROG -x -c "shutdown" $testdir/file2 >> $seqres.full
tests/xfs/227 +1 -1
index 3f01b175e..2ad910cbd 100755
@@ -60,7 +60,7 @@ fragment_freespace()
 		$XFS_IO_PROG -f -c "unresvsp ${i}k 4k" $_file \
 					> /dev/null 2>&1
 	done
-	sync
+	_scratch_sync
 
 	# and now use up all the remaining extents larger than 3 blocks
 	$XFS_IO_PROG -fs -c "resvsp 0 4m" $_file.large > /dev/null 2>&1
tests/xfs/231 +2 -2
index c01682fa8..ea494a152 100755
@@ -71,7 +71,7 @@ seq 2 2 $((nr - 1)) | while read f; do
 	$XFS_IO_PROG -f -c "pwrite -S 0x63 $((blksz * f - 1)) 1" $testdir/file2 >> $seqres.full
 	$XFS_IO_PROG -f -c "pwrite -S 0x63 $((blksz * f - 1)) 1" $testdir/file2.chk >> $seqres.full
 done
-sync
+_scratch_sync
 
 echo "Wait for CoW expiration"
 sleep 3
@@ -88,7 +88,7 @@ seq 2 2 $((nr - 1)) | while read f; do
 	$XFS_IO_PROG -f -c "pwrite -S 0x63 $((blksz * f)) 1" $testdir/file2 >> $seqres.full
 	$XFS_IO_PROG -f -c "pwrite -S 0x63 $((blksz * f)) 1" $testdir/file2.chk >> $seqres.full
 done
-sync
+_scratch_sync
 
 echo "Compare files"
 md5sum $testdir/file1 | _filter_scratch
tests/xfs/232 +2 -2
index c7eba95a4..0eea2c098 100755
@@ -74,7 +74,7 @@ seq 2 2 $((nr - 1)) | while read f; do
 	$XFS_IO_PROG -f -c "pwrite -S 0x63 $((blksz * f - 1)) 1" $testdir/file2 >> $seqres.full
 	$XFS_IO_PROG -f -c "pwrite -S 0x63 $((blksz * f - 1)) 1" $testdir/file2.chk >> $seqres.full
 done
-sync
+_scratch_sync
 
 echo "Wait for CoW expiration"
 $XFS_IO_PROG -f -c "pwrite -S 0x63 $((blksz * 2)) 1" $testdir/file2 >> $seqres.full
@@ -92,7 +92,7 @@ seq 2 2 $((nr - 1)) | while read f; do
 	$XFS_IO_PROG -f -c "pwrite -S 0x63 $((blksz * f)) 1" $testdir/file2 >> $seqres.full
 	$XFS_IO_PROG -f -c "pwrite -S 0x63 $((blksz * f)) 1" $testdir/file2.chk >> $seqres.full
 done
-sync
+_scratch_sync
 
 echo "Compare files"
 md5sum $testdir/file1 | _filter_scratch
tests/xfs/234 +1 -1
index 4d6e30c0a..e57bf38cb 100755
@@ -39,7 +39,7 @@ echo "Create the original file blocks"
 blksz="$(_get_block_size $testdir)"
 nr_blks=$((4 * blksz / 12))
 _pwrite_byte 0x61 0 $((blksz * nr_blks)) $testdir/file1 >> $seqres.full
-sync
+_scratch_sync
 
 echo "Punch every other block"
 seq 1 2 $((nr_blks - 1)) | while read nr; do
tests/xfs/236 +1 -1
index a374a300d..29cab3190 100755
@@ -38,7 +38,7 @@ nr_blks=$((8 * blksz / 12))
 for i in 1 2 x; do
 	_pwrite_byte 0x61 0 $((blksz * nr_blks)) $testdir/file1 >> $seqres.full
 	_pwrite_byte 0x62 0 $((blksz * nr_blks)) $testdir/file2 >> $seqres.full
-	sync
+	_scratch_sync
 
 	echo "$i: Reflink every other block"
 	seq 1 2 $((nr_blks - 1)) | while read nr; do
tests/xfs/237 +1 -1
index 194cd0459..f172aaf59 100755
@@ -60,7 +60,7 @@ md5sum $testdir/file2 | _filter_scratch
 echo "CoW and unmount"
 $XFS_IO_PROG -f -c "pwrite -S 0x63 $bufsize 1" $testdir/file2 >> $seqres.full
 $XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $TEST_DIR/moo >> $seqres.full
-sync
+_scratch_sync
 _dmerror_load_error_table
 $AIO_TEST -a $alignment -f DIRECT -b $bufsize $TEST_DIR/moo $testdir/file2 >> $seqres.full
 _dmerror_load_working_table
tests/xfs/239 +1 -1
index bfe722c0a..11a0d125a 100755
@@ -56,7 +56,7 @@ md5sum $testdir/file2 | _filter_scratch
 
 echo "CoW and unmount"
 $XFS_IO_PROG -f -c "pwrite -S 0x63 $bufsize 1" $testdir/file2 >> $seqres.full
-sync
+_scratch_sync
 $XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $TEST_DIR/moo >> $seqres.full
 $AIO_TEST -a $alignment -f DIRECT -b $bufsize $TEST_DIR/moo $testdir/file2 >> $seqres.full
 _scratch_cycle_mount
tests/xfs/240 +1 -1
index 4f9311e8e..e95cf3f5d 100755
@@ -62,7 +62,7 @@ md5sum $testdir/file2 | _filter_scratch
 echo "CoW and unmount"
 $XFS_IO_PROG -f -c "pwrite -S 0x63 $bufsize 1" $testdir/file2 >> $seqres.full
 $XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $TEST_DIR/moo >> $seqres.full
-sync
+_scratch_sync
 _dmerror_load_error_table
 $AIO_TEST -b $bufsize $TEST_DIR/moo $testdir/file2 >> $seqres.full
 $XFS_IO_PROG -c "fdatasync" $testdir/file2
tests/xfs/241 +1 -1
index 153249397..6d73d48e4 100755
@@ -57,7 +57,7 @@ md5sum $testdir/file2 | _filter_scratch
 echo "CoW and unmount"
 $XFS_IO_PROG -f -c "pwrite -S 0x63 $bufsize 1" $testdir/file2 >> $seqres.full
 $XFS_IO_PROG -f -c "pwrite -S 0x63 -b $bufsize 0 $filesize" $TEST_DIR/moo >> $seqres.full
-sync
+_scratch_sync
 $AIO_TEST -b $bufsize $TEST_DIR/moo $testdir/file2 >> $seqres.full
 _scratch_cycle_mount
 
tests/xfs/243 +2 -2
index ea83a7b44..964e94e1d 100755
@@ -66,7 +66,7 @@ seq 3 5 $nr | while read f; do
 	_pwrite_byte 0x71 $((blksz * f)) $blksz $testdir/file3 >> $seqres.full
 	_pwrite_byte 0x71 $((blksz * f)) $blksz $testdir/file3.chk >> $seqres.full
 done
-sync
+_scratch_sync
 # 4 blocks are delalloc (do later)
 seq 4 5 $nr | while read f; do
 	_pwrite_byte 0x62 $((blksz * f)) $blksz $testdir/file3 >> $seqres.full
@@ -102,7 +102,7 @@ echo "Regular data extents:"
 test $(_xfs_bmapx_find data $testdir/file3 -E '00[01]{4}$') -gt 0 || \
 	echo "Expected to find a regular data extent"
 
-sync
+_scratch_sync
 
 echo "Dump extents after sync"
 $XFS_IO_PROG -c "cowextsize" $testdir/file3 | _filter_scratch
tests/xfs/246 +1 -1
index ce97fad5d..997a7ad47 100755
@@ -25,7 +25,7 @@ mkdir $testdir
 
 echo "Create the original files"
 touch $testdir/file1
-sync
+_scratch_sync
 
 echo "Dump extents after sync"
 echo "Hole CoW extents:"
tests/xfs/265 +1 -1
index 21de4c054..e7f939c14 100755
@@ -44,7 +44,7 @@ for i in 1 2 x; do
 	seq 1 $nr_blks | while read nr; do
 		_cp_reflink $testdir/file1 $testdir/file1.$nr >> $seqres.full
 	done
-	sync
+	_scratch_sync
 
 	echo "$i: Truncate files"
 	seq 1 $nr_blks | while read nr; do
tests/xfs/272 +1 -1
index b65e2fae5..3e3ceec51 100755
@@ -33,7 +33,7 @@ _scratch_mount
 _xfs_force_bdev data $SCRATCH_MNT
 
 _pwrite_byte 0x80 0 737373 $SCRATCH_MNT/urk >> $seqres.full
-sync
+_scratch_sync
 $here/src/punch-alternating $SCRATCH_MNT/urk >> $seqres.full
 ino=$(stat -c '%i' $SCRATCH_MNT/urk)
 
tests/xfs/274 +1 -1
index d06e7fc13..a6bdc7c03 100755
@@ -32,7 +32,7 @@ _scratch_mkfs > "$seqres.full" 2>&1
 _scratch_mount
 
 _pwrite_byte 0x80 0 737373 $SCRATCH_MNT/f1 >> $seqres.full
-sync
+_scratch_sync
 $here/src/punch-alternating $SCRATCH_MNT/f1 >> $seqres.full
 _cp_reflink $SCRATCH_MNT/f1 $SCRATCH_MNT/f2
 ino=$(stat -c '%i' $SCRATCH_MNT/f1)
tests/xfs/276 +1 -1
index f21f7a86e..69de69d86 100755
@@ -37,7 +37,7 @@ _scratch_mount
 test $rtextsz -eq $dbsize || _notrun "Skipping test due to rtextsize > 1 fsb"
 
 $XFS_IO_PROG -f -R -c 'pwrite -S 0x80 0 737373' $SCRATCH_MNT/urk >> $seqres.full
-sync
+_scratch_sync
 $here/src/punch-alternating $SCRATCH_MNT/urk >> $seqres.full
 ino=$(stat -c '%i' $SCRATCH_MNT/urk)
 
tests/xfs/291 +2 -2
index 0141c075b..777968a92 100755
@@ -41,12 +41,12 @@ mkdir $SCRATCH_MNT/fragdir
 for I in `seq 0 26200`; do
         (echo data > $SCRATCH_MNT/fragdir/f$I) >> $seqres.full 2>&1
 done
-sync
+_scratch_sync
 
 for I in `seq 0 2 26200`; do
         rm -f $SCRATCH_MNT/fragdir/f$I >> $seqres.full 2>&1
 done
-sync
+_scratch_sync
 
 # Soak up any remaining freespace
 $XFS_IO_PROG -f -c "pwrite 0 16m" -c "fsync" $SCRATCH_MNT/space_file.large >> $seqres.full 2>&1
tests/xfs/309 +1 -1
index 63ca0a67b..429a506c2 100755
@@ -44,7 +44,7 @@ for i in 1 2 x; do
 	seq 1 $nr_blks | while read nr; do
 		_cp_reflink $testdir/file1 $testdir/file1.$nr >> $seqres.full
 	done
-	sync
+	_scratch_sync
 
 	echo "$i: Truncate files"
 	seq 1 $nr_blks | while read nr; do
tests/xfs/312 +1 -1
index 6e47d442b..4c0fbf74a 100755
@@ -45,7 +45,7 @@ _cp_reflink $SCRATCH_MNT/file1 $SCRATCH_MNT/file3
 seq 1 2 $blks | while read off; do
 	$XFS_IO_PROG -c "fpunch $((off * blksz)) $blksz" $SCRATCH_MNT/file3 >> $seqres.full
 done
-sync
+_scratch_sync
 
 echo "Check files"
 md5sum $SCRATCH_MNT/file1 | _filter_scratch
tests/xfs/313 +1 -1
index d0c77db8b..8f533041b 100755
@@ -44,7 +44,7 @@ _cp_reflink $SCRATCH_MNT/file1 $SCRATCH_MNT/file3
 seq 1 2 $blks | while read off; do
 	$XFS_IO_PROG -c "fpunch $((off * blksz)) $blksz" $SCRATCH_MNT/file3 >> $seqres.full
 done
-sync
+_scratch_sync
 
 echo "Check files"
 md5sum $SCRATCH_MNT/file1 | _filter_scratch
tests/xfs/314 +1 -1
index 07596e00b..b86d06ae7 100755
@@ -45,7 +45,7 @@ _cp_reflink $SCRATCH_MNT/file1 $SCRATCH_MNT/file3
 seq 1 2 $blks | while read off; do
 	$XFS_IO_PROG -c "fpunch $((off * blksz)) $blksz" $SCRATCH_MNT/file3 >> $seqres.full
 done
-sync
+_scratch_sync
 
 echo "Check files"
 md5sum $SCRATCH_MNT/file1 | _filter_scratch
tests/xfs/315 +2 -2
index f6c346e18..af783a7ad 100755
@@ -43,7 +43,7 @@ echo "Create files"
 _pwrite_byte 0x66 0 $sz $SCRATCH_MNT/file1 >> $seqres.full
 _pwrite_byte 0x66 $((sz / 2)) $((sz / 2)) $SCRATCH_MNT/file2 >> $seqres.full
 _reflink_range $SCRATCH_MNT/file1 0 $SCRATCH_MNT/file2 0 $((sz / 2)) >> $seqres.full
-sync
+_scratch_sync
 
 echo "Check files"
 md5sum $SCRATCH_MNT/file1 | _filter_scratch
@@ -54,7 +54,7 @@ _scratch_inject_error "free_extent"
 
 echo "CoW a few blocks"
 $XFS_IO_PROG -c "pwrite -W -S 0x67 -b $sz 0 $sz" $SCRATCH_MNT/file1 >> $seqres.full 2>&1
-sync
+_scratch_sync
 
 echo "FS should be shut down, touch will fail"
 touch $SCRATCH_MNT/badfs 2>&1 | _filter_scratch
tests/xfs/316 +1 -1
index 63711ed47..3c92d1a1d 100755
@@ -45,7 +45,7 @@ _cp_reflink $SCRATCH_MNT/file1 $SCRATCH_MNT/file3
 seq 1 2 $blks | while read off; do
 	$XFS_IO_PROG -c "fpunch $((off * blksz)) $blksz" $SCRATCH_MNT/file3 >> $seqres.full
 done
-sync
+_scratch_sync
 
 echo "Check files"
 md5sum $SCRATCH_MNT/file1 | _filter_scratch
tests/xfs/317 +1 -1
index 192107f59..2ae0b903c 100755
@@ -36,7 +36,7 @@ _scratch_mount >> $seqres.full
 echo "Create files"
 touch $SCRATCH_MNT/file1
 _pwrite_byte 0x67 0 $sz $SCRATCH_MNT/file0 >> $seqres.full
-sync
+_scratch_sync
 
 echo "Check files"
 md5sum $SCRATCH_MNT/file0 | _filter_scratch
tests/xfs/318 +2 -2
index 590d268e6..4ff11be40 100755
@@ -43,7 +43,7 @@ touch $SCRATCH_MNT/file1
 
 echo "Write files"
 $XFS_IO_PROG -c "pwrite -S 0x67 0 $sz" $SCRATCH_MNT/file1 >> $seqres.full
-sync
+_scratch_sync
 
 echo "Check files"
 md5sum $SCRATCH_MNT/file1 2>&1 | _filter_scratch
@@ -53,7 +53,7 @@ _scratch_inject_error "free_extent"
 
 echo "Remove files"
 rm -rf $SCRATCH_MNT/file1
-sync
+_scratch_sync
 $XFS_IO_PROG -x -c 'freeze' -c 'thaw' $SCRATCH_MNT >> $seqres.full 2>&1
 
 echo "FS should be shut down, touch will fail"
tests/xfs/319 +1 -1
index 45b958d8f..66f191127 100755
@@ -38,7 +38,7 @@ echo "Create files"
 _pwrite_byte 0x66 0 $sz $SCRATCH_MNT/file1 >> $seqres.full
 _cp_reflink $SCRATCH_MNT/file1 $SCRATCH_MNT/file2
 _pwrite_byte 0x67 0 $sz $SCRATCH_MNT/file3 >> $seqres.full
-sync
+_scratch_sync
 
 echo "Check files"
 md5sum $SCRATCH_MNT/file1 | _filter_scratch
tests/xfs/320 +1 -1
index f7e4949a3..4cf174b88 100755
@@ -37,7 +37,7 @@ _scratch_mount >> $seqres.full
 echo "Create files"
 _pwrite_byte 0x66 0 $sz $SCRATCH_MNT/file1 >> $seqres.full
 _cp_reflink $SCRATCH_MNT/file1 $SCRATCH_MNT/file2
-sync
+_scratch_sync
 
 echo "Check files"
 md5sum $SCRATCH_MNT/file1 | _filter_scratch
tests/xfs/321 +1 -1
index a09e4a8ce..4ebde3e2a 100755
@@ -37,7 +37,7 @@ _scratch_mount >> $seqres.full
 echo "Create files"
 _pwrite_byte 0x66 0 $sz $SCRATCH_MNT/file1 >> $seqres.full
 $XFS_IO_PROG -f -c "truncate $sz" $SCRATCH_MNT/file3 >> $seqres.full
-sync
+_scratch_sync
 
 echo "Check files"
 md5sum $SCRATCH_MNT/file1 | _filter_scratch
tests/xfs/322 +1 -1
index 4cb36a515..0492b647e 100755
@@ -39,7 +39,7 @@ _require_congruent_file_oplen $SCRATCH_MNT $blksz
 echo "Create files"
 _pwrite_byte 0x66 0 $sz $SCRATCH_MNT/file1 >> $seqres.full
 $XFS_IO_PROG -f -c "truncate $sz" $SCRATCH_MNT/file3 >> $seqres.full
-sync
+_scratch_sync
 
 echo "Check files"
 md5sum $SCRATCH_MNT/file1 | _filter_scratch
tests/xfs/323 +1 -1
index 0579d4a72..ee45ba608 100755
@@ -37,7 +37,7 @@ _scratch_mount >> $seqres.full
 echo "Create files"
 _pwrite_byte 0x66 0 $sz $SCRATCH_MNT/file1 >> $seqres.full
 _pwrite_byte 0x67 0 $sz $SCRATCH_MNT/file3 >> $seqres.full
-sync
+_scratch_sync
 
 echo "Check files"
 md5sum $SCRATCH_MNT/file1 | _filter_scratch
tests/xfs/324 +1 -1
index 485b16215..b86bb0ea8 100755
@@ -45,7 +45,7 @@ _cp_reflink $SCRATCH_MNT/file1 $SCRATCH_MNT/file3
 seq 1 2 $blks | while read off; do
 	$XFS_IO_PROG -c "fpunch $((off * blksz)) $blksz" $SCRATCH_MNT/file3 >> $seqres.full
 done
-sync
+_scratch_sync
 
 echo "Check files"
 md5sum $SCRATCH_MNT/file1 | _filter_scratch
tests/xfs/325 +2 -2
index 2387b8d88..f06eda64e 100755
@@ -39,7 +39,7 @@ _scratch_mount >> $seqres.full
 echo "Create files"
 _pwrite_byte 0x66 0 $((blksz * blks)) $SCRATCH_MNT/file1 >> $seqres.full
 _cp_reflink $SCRATCH_MNT/file1 $SCRATCH_MNT/file2
-sync
+_scratch_sync
 
 echo "Check files"
 md5sum $SCRATCH_MNT/file1 | _filter_scratch
@@ -52,7 +52,7 @@ echo "Inject error"
 _scratch_inject_error "free_extent"
 
 rm $SCRATCH_MNT/file1
-sync
+_scratch_sync
 $XFS_IO_PROG -x -c 'freeze' -c 'thaw' $SCRATCH_MNT >> $seqres.full 2>&1
 
 echo "FS should be shut down, touch will fail"
tests/xfs/326 +2 -2
index 5d7afe9bf..a06879c3e 100755
@@ -62,7 +62,7 @@ _cp_reflink $SCRATCH_MNT/file1 $SCRATCH_MNT/file3
 seq 1 2 $blks | while read off; do
 	$XFS_IO_PROG -c "fpunch $((off * blksz)) $blksz" $SCRATCH_MNT/file3 >> $seqres.full
 done
-sync
+_scratch_sync
 
 echo "Check files"
 md5sum $SCRATCH_MNT/file1 | _filter_scratch
@@ -70,7 +70,7 @@ md5sum $SCRATCH_MNT/file2 | _filter_scratch
 md5sum $SCRATCH_MNT/file3 | _filter_scratch
 
 $XFS_IO_PROG -c "pwrite -W -S 0x67 $((10 * blksz)) 1" $SCRATCH_MNT/file2 >> $seqres.full
-sync
+_scratch_sync
 
 echo "Inject error"
 _scratch_inject_error "refcount_finish_one"
tests/xfs/327 +1 -1
index 44728c117..1f2ee5484 100755
@@ -37,7 +37,7 @@ _pwrite_byte 0x66 0 1 $SCRATCH_MNT/file.0 >> $seqres.full
 seq 1 $nr | while read i; do
 	_cp_reflink $SCRATCH_MNT/file.0 $SCRATCH_MNT/file.$i
 done
-sync
+_scratch_sync
 
 ino_0=$(stat -c '%i' $SCRATCH_MNT/file.0)
 ino_64=$(stat -c '%i' $SCRATCH_MNT/file.64)
tests/xfs/420 +1 -1
index 038682ef9..37fb8c6b8 100755
@@ -109,7 +109,7 @@ md5sum $testdir/file3 | _filter_scratch
 echo 1 > /proc/sys/vm/drop_caches
 
 echo "sync filesystem" | tee -a $seqres.full
-sync
+_scratch_sync
 
 $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file1 >> $seqres.full 2>&1
 $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file2 >> $seqres.full 2>&1
tests/xfs/421 +1 -1
index ab565e1f6..fdee45ce4 100755
@@ -94,7 +94,7 @@ md5sum $testdir/file3 | _filter_scratch
 echo 1 > /proc/sys/vm/drop_caches
 
 echo "sync filesystem" | tee -a $seqres.full
-sync
+_scratch_sync
 
 $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file1 >> $seqres.full 2>&1
 $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file2 >> $seqres.full 2>&1
tests/xfs/423 +1 -1
index 1fbf500d8..78fd91301 100755
@@ -31,7 +31,7 @@ _scratch_mount
 $XFS_IO_PROG -f -c 'falloc 0 10m' $SCRATCH_MNT/a >> $seqres.full
 $XFS_IO_PROG -f -c 'falloc 0 10m' $SCRATCH_MNT/b >> $seqres.full
 $here/src/punch-alternating $SCRATCH_MNT/b
-sync
+_scratch_sync
 
 echo "Set up delalloc extents"
 $XFS_IO_PROG -c 'pwrite -S 0x66 10m 128k' $SCRATCH_MNT/a >> $seqres.full
tests/xfs/438 +1 -1
index 0239778c5..6d1988c8b 100755
@@ -136,7 +136,7 @@ $DMSETUP_PROG table >> $seqres.full
 $XFS_QUOTA_PROG -x -c "limit -u isoft=400 $qa_user" $SCRATCH_MNT
 $XFS_QUOTA_PROG -x -c "report -ih" $SCRATCH_MNT >> $seqres.full
 
-sync
+_scratch_sync
 
 # wait for the push of the dquota log item in AIL and
 # the completion of the retried write of dquota buffer
tests/xfs/440 +1 -1
index fc7319449..0cc679aeb 100755
@@ -43,7 +43,7 @@ $XFS_IO_PROG -c 'stat -r' $SCRATCH_MNT/a | grep stat.size >> $seqres.full
 _report_quota_blocks "-u $SCRATCH_MNT"
 
 echo "Sync"
-sync
+_scratch_sync
 _report_quota_blocks "-u $SCRATCH_MNT"
 
 echo "Chown and check quota"
tests/xfs/442 +1 -1
index 5cbd8dd19..58e56b5d2 100755
@@ -36,7 +36,7 @@ compare_quota_to_du() {
 # This ensures that we did the quota accounting correctly and that we're
 # accurately reporting cow preallocation blocks in stat.
 check_quota_du_blocks() {
-	sync
+	_scratch_sync
 	#$XFS_QUOTA_PROG -x -c 'report' $SCRATCH_MNT >> $seqres.full
 	du_rep=$(du -ks $SCRATCH_MNT | awk '{print $1}')
 	u_rep=$(report_quota_blocks -u)
tests/xfs/511 +1 -1
index b55c34b92..a942e92e3 100755
@@ -33,7 +33,7 @@ _require_prjquota $SCRATCH_DEV
 
 # Create a directory to be project object, and create a file to take 64k space
 mkdir $SCRATCH_MNT/t
-$XFS_IO_PROG -f -c "pwrite 0 65536" -c sync $SCRATCH_MNT/t/file >>$seqres.full
+$XFS_IO_PROG -f -c "pwrite 0 65536" -c syncfs $SCRATCH_MNT/t/file >>$seqres.full
 
 quota_cmd="$XFS_QUOTA_PROG -x"
 $quota_cmd -c "project -s -p $SCRATCH_MNT/t 42" $SCRATCH_MNT >/dev/null 2>&1
tests/xfs/519 +1 -1
index ac815f123..37a139c9e 100755
@@ -28,7 +28,7 @@ $XFS_IO_PROG -f -c 'pwrite -S 0x59 0 1m -b 1m' $SCRATCH_MNT/c >> $seqres.full
 _cp_reflink $SCRATCH_MNT/a $SCRATCH_MNT/e
 _cp_reflink $SCRATCH_MNT/c $SCRATCH_MNT/d
 touch $SCRATCH_MNT/b
-sync
+_scratch_sync
 
 # Test that setting the reflink flag on the dest file forces the log
 echo "test reflink flag not set"
tests/xfs/520 +1 -1
index 3734d8746..2d80188b2 100755
@@ -46,7 +46,7 @@ force_crafted_metadata() {
 	if [ $mounted -ne 0 ]; then
 		dd if=/dev/zero of=$SCRATCH_MNT/test bs=65536 count=1 >> \
 			$seqres.full 2>&1
-		sync
+		_scratch_sync
 	fi
 
 	_dmesg_since_test_start | tac | sed -ne "0,\#${kmsg}#p" | tac | \
tests/xfs/541 +1 -1
index 518373fa8..9be15d1f6 100755
@@ -91,7 +91,7 @@ test $after_rtextsz_blocks -eq $new_rtextsz_blocks || \
 
 # Create a new realtime file to prove that we can.
 echo moo > $SCRATCH_MNT/a
-sync -f $SCRATCH_MNT
+_scratch_sync
 $XFS_IO_PROG -c 'lsattr -v' $SCRATCH_MNT/a | \
 	cut -d ' ' -f 1 | \
 	grep -q realtime || \
tests/xfs/553 +2 -2
index f4294e9c1..13ef629e3 100755
@@ -45,7 +45,7 @@ $XFS_IO_PROG -c "cowextsize $((blksz * 4096))" $destination >> $seqres.full
 echo "Fragment FS"
 $XFS_IO_PROG -f -c "pwrite 0 $((blksz * 16384))" $fragmented_file \
 	     >> $seqres.full
-sync
+_scratch_sync
 $here/src/punch-alternating $fragmented_file >> $seqres.full
 
 echo "Inject bmap_alloc_minlen_extent error tag"
@@ -54,7 +54,7 @@ _scratch_inject_error bmap_alloc_minlen_extent 1
 echo "Create delalloc extent of length 4096 blocks in destination file's CoW fork"
 $XFS_IO_PROG -c "pwrite 0 $blksz" $destination >> $seqres.full
 
-sync
+_scratch_sync
 
 echo "Direct I/O write at 3rd block in destination file"
 $XFS_IO_PROG -d -c "pwrite $((blksz * 3)) $((blksz * 2))" $destination \
tests/xfs/558 +3 -3
index ec2fbcb68..ba137ae8d 100755
@@ -140,13 +140,13 @@ $XFS_IO_PROG -c 'cowextsize 1m' $SCRATCH_MNT
 # Write out a file with the first two blocks unshared and the rest shared.
 _pwrite_byte 0x59 0 $((160 * blksz)) $SCRATCH_MNT/file >> $seqres.full
 _pwrite_byte 0x59 0 $((160 * blksz)) $SCRATCH_MNT/file.compare >> $seqres.full
-sync
+_scratch_sync
 
 _cp_reflink $SCRATCH_MNT/file $SCRATCH_MNT/file.reflink
 
 _pwrite_byte 0x58 0 $((2 * blksz)) $SCRATCH_MNT/file >> $seqres.full
 _pwrite_byte 0x58 0 $((2 * blksz)) $SCRATCH_MNT/file.compare >> $seqres.full
-sync
+_scratch_sync
 
 # Avoid creation of large folios on newer kernels by cycling the mount and
 # immediately writing to the page cache.
@@ -156,7 +156,7 @@ _scratch_cycle_mount
 # before slowing down writeback to avoid unnecessary delay.
 _pwrite_byte 0x57 0 $((2 * blksz)) $SCRATCH_MNT/file.compare >> $seqres.full
 _pwrite_byte 0x56 $((2 * blksz)) $((2 * blksz)) $SCRATCH_MNT/file.compare >> $seqres.full
-sync
+_scratch_sync
 
 # Introduce a half-second wait to each writeback block mapping call.  This
 # gives us a chance to race speculative cow prealloc with writeback.
tests/xfs/607 +3 -3
index 530fea9ed..ae1e19a9a 100755
@@ -53,7 +53,7 @@ fi
 echo "Create source file"
 $XFS_IO_PROG -f -c "pwrite 0 $((blksz * 256))" $file1 >> $seqres.full
 
-sync
+_test_sync
 
 echo "Create Reflinked file"
 _cp_reflink $file1 $file2 >> $seqres.full
@@ -63,7 +63,7 @@ $XFS_IO_PROG -c "cowextsize $((blksz * 128))" -c stat $file1 >> $seqres.full
 
 echo "Fragment FS"
 $XFS_IO_PROG -f -c "pwrite 0 $((blksz * 512))" $fragmentedfile >> $seqres.full
-sync
+_test_sync
 $here/src/punch-alternating $fragmentedfile
 
 echo "Allocate block sized extent from now onwards"
@@ -72,7 +72,7 @@ _test_inject_error bmap_alloc_minlen_extent 1
 echo "Create big delalloc extent in CoW fork"
 $XFS_IO_PROG -c "pwrite 0 $blksz" $file1 >> $seqres.full
 
-sync
+_test_sync
 
 $XFS_IO_PROG -c 'bmap -elpv' -c 'bmap -celpv' $file1 &>> $seqres.full
 
tests/xfs/617 +1 -1
index 2f5bf5233..e0ad7d816 100755
@@ -35,7 +35,7 @@ swap_and_check_contents() {
 	local b="$2"
 	local tag="$3"
 
-	sync
+	_scratch_sync
 
 	# Test exchangerange.  -w means skip holes in /b
 	echo "swap $tag" >> $seqres.full
tests/xfs/629 +1 -1
index 58beedc03..34b3ca0d6 100755
@@ -55,7 +55,7 @@ for ((n=0; n<nfiles; n++)); do
 	write_sync_file $n > /dev/null 2>&1 &
 done
 wait
-sync
+_scratch_sync
 
 for ((n=0; n<nfiles; n++)); do
 	count=$(_count_extents $workfile.$n)
tests/xfs/630 +1 -1
index 939d8a4ac..79dcb44bc 100755
@@ -58,7 +58,7 @@ for ((n=0; n<nfiles; n++)); do
 	write_extsz_file $n > /dev/null 2>&1 &
 done
 wait
-sync
+_scratch_sync
 
 for ((n=0; n<nfiles; n++)); do
 	count=$(_count_extents $workfile.$n)
tests/xfs/631 +1 -1
index 55a742979..4d79b821f 100755
@@ -58,7 +58,7 @@ for ((n=0; n<nfiles; n++)); do
 	write_direct_file $n > /dev/null 2>&1 &
 done
 wait
-sync
+_scratch_sync
 
 for ((n=0; n<nfiles; n++)); do
 	count=$(_count_extents $workfile.$n)
tests/xfs/790 +1 -1
index 53e8906bb..d0bdaef96 100755
@@ -45,7 +45,7 @@ $here/src/punch-alternating $dir/a
 _pwrite_byte 0x59 0 $((blksz * nrblks)) $dir/b >> $seqres.full
 _pwrite_byte 0x59 0 $((blksz * nrblks)) $dir/c >> $seqres.full
 _pwrite_byte 0x58 0 $((blksz * nrblks)) $dir/a >> $seqres.full
-sync
+_test_sync
 
 # Inject a bmap error and trigger it via exchangerange.
 filesnap "before commit"
tests/xfs/791 +1 -1
index bca580420..0fcc48190 100755
@@ -38,7 +38,7 @@ _pwrite_byte 0x58 0 1m $SCRATCH_MNT/a >> $seqres.full
 $XFS_IO_PROG -f -c 'truncate 1m' $SCRATCH_MNT/b
 _pwrite_byte 0x59 64k 64k $SCRATCH_MNT/b >> $seqres.full
 _pwrite_byte 0x57 768k 64k $SCRATCH_MNT/b >> $seqres.full
-sync
+_scratch_sync
 
 md5sum $SCRATCH_MNT/a | _filter_scratch
 md5sum $SCRATCH_MNT/b | _filter_scratch
tests/xfs/792 +1 -1
index cdfa63692..6c7b2db11 100755
@@ -33,7 +33,7 @@ _scratch_mount
 
 # Create original file
 _pwrite_byte 0x58 0 1m $SCRATCH_MNT/a >> $seqres.full
-sync
+_scratch_sync
 md5sum $SCRATCH_MNT/a | _filter_scratch
 
 # Test atomic scatter-gather file commits.
Signature · 1 line
2.45.2
#19 lore raw

PATCH 12/40 fstests: clean up mount and unmount operations

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

The way tests run unmount is, at times, completely random. Sometimes they call the correct _scratch_unmount function, sometimes they open code it with a direct call to UMOUNT_PROG <dir>, sometimes they run umount directly.

This makes it really hard to instrument unmount operations when trying to work out why transient, unpredictable failures like this occur randomly during a test run:

umount: /mnt/xfs/runner-17/test: target is busy.

Sometimes it happens on a test device mount, sometimes a scratch device mount. Sometimes it happens to a test specific dm or loop device mount. But without instrumenting every single unmount call in every test, it's impossible to capture these failures easily.

Solve this problem by introducing the _unmount() wrapper. It is simply a call to UMOUNT_PROG <dir>, but it provides a single point were -every- unmount operation funnels through.

We already have a _mount wrapper for this reason. However, in trying to work out why mounts were failing (because unmounts were failing), I discovered that that_mount() is used inconsistently as well.

Sort this all out by adding and _unmount() wrapper to go with _mount() and use them everywhere consistently.

---
 common/btrfs       |  4 ++--
 common/dmdelay     |  4 ++--
 common/dmdust      |  4 ++--
 common/dmerror     |  4 ++--
 common/dmflakey    |  4 ++--
 common/dmhugedisk  |  2 +-
 common/dmlogwrites |  4 ++--
 common/dmthin      |  4 ++--
 common/overlay     | 10 +++++-----
 common/populate    |  8 ++++----
 common/rc          | 42 +++++++++++++++++++++++++-----------------
 tests/generic/042  |  4 ++--
 tests/generic/050  |  3 ++-
 tests/generic/067  |  6 +++---
 tests/generic/081  |  2 +-
 tests/generic/085  |  6 +++---
 tests/generic/108  |  2 +-
 tests/generic/171  |  2 +-
 tests/generic/172  |  2 +-
 tests/generic/173  |  2 +-
 tests/generic/174  |  2 +-
 tests/generic/306  |  2 +-
 tests/generic/332  |  2 +-
 tests/generic/361  |  2 +-
 tests/generic/373  |  2 +-
 tests/generic/374  |  2 +-
 tests/generic/395  |  2 +-
 tests/generic/459  |  2 +-
 tests/generic/563  | 10 +++++-----
 tests/generic/604  |  2 +-
 tests/generic/631  |  2 +-
 tests/generic/648  |  6 +++---
 tests/generic/698  |  4 ++--
 tests/generic/699  |  8 ++++----
 tests/generic/704  |  2 +-
 tests/generic/717  |  2 +-
 tests/generic/730  |  2 +-
 tests/generic/731  |  2 +-
 tests/generic/732  |  4 ++--
 tests/generic/741  |  2 +-
 tests/generic/744  |  6 +++---
 tests/generic/746  |  8 ++++----
 tests/xfs/014      |  6 +++---
 tests/xfs/073      | 20 ++++++++++----------
 tests/xfs/074      |  2 +-
 tests/xfs/078      |  6 +++---
 tests/xfs/148      |  6 +++---
 tests/xfs/149      |  4 ++--
 tests/xfs/186      |  4 ++--
 tests/xfs/216      | 10 ++++++++--
 tests/xfs/217      |  4 ++--
 tests/xfs/250      |  7 ++++---
 tests/xfs/289      |  4 ++--
 tests/xfs/507      |  2 +-
 tests/xfs/513      |  4 ++--
 tests/xfs/544      |  2 +-
 tests/xfs/606      |  2 +-
 tests/xfs/613      |  4 ++--
 58 files changed, 149 insertions(+), 133 deletions(-)
common/btrfs +2 -2
index 95a9c8e6c..4a2c9886e 100644
@@ -352,7 +352,7 @@ _btrfs_stress_subvolume()
 	while [ ! -e $stop_file ]; do
 		$BTRFS_UTIL_PROG subvolume create $btrfs_mnt/$subvol_name
 		$MOUNT_PROG -o subvol=$subvol_name $btrfs_dev $subvol_mnt
-		$UMOUNT_PROG $subvol_mnt
+		_unmount $subvol_mnt
 		$BTRFS_UTIL_PROG subvolume delete $btrfs_mnt/$subvol_name
 	done
 }
@@ -368,7 +368,7 @@ _btrfs_kill_stress_subvolume_pid()
 	# Ignore if process already died.
 	wait $subvol_pid &> /dev/null
 	rm -f $stop_file
-	$UMOUNT_PROG $subvol_mnt &> /dev/null
+	_unmount $subvol_mnt &> /dev/null
 }
 
 # stress btrfs by running scrub in a loop
common/dmdelay +2 -2
index 1c4fca6e9..848afb993 100644
@@ -28,7 +28,7 @@ _mount_delay()
 
 _unmount_delay()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_unmount $SCRATCH_MNT
 }
 
 _cleanup_delay()
@@ -36,7 +36,7 @@ _cleanup_delay()
 	# If dmsetup load fails then we need to make sure to do resume here
 	# otherwise the umount will hang
 	$DMSETUP_PROG resume $DELAY_NAME > /dev/null 2>&1
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_unmount $SCRATCH_MNT > /dev/null 2>&1
 	_dmsetup_remove $DELAY_NAME
 }
 
common/dmdust +2 -2
index 37bb865c8..55bbc84db 100644
@@ -24,7 +24,7 @@ _mount_dust()
 
 _unmount_dust()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_unmount $SCRATCH_MNT
 }
 
 _cleanup_dust()
@@ -32,6 +32,6 @@ _cleanup_dust()
 	# If dmsetup load fails then we need to make sure to do resume here
 	# otherwise the umount will hang
 	$DMSETUP_PROG resume $DUST_NAME > /dev/null 2>&1
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_unmount $SCRATCH_MNT > /dev/null 2>&1
 	_dmsetup_remove $DUST_NAME
 }
common/dmerror +2 -2
index 97796b38e..309129c03 100644
@@ -101,7 +101,7 @@ _dmerror_mount()
 
 _dmerror_unmount()
 {
-	umount $SCRATCH_MNT
+	_unmount $SCRATCH_MNT
 }
 
 _dmerror_cleanup()
@@ -110,7 +110,7 @@ _dmerror_cleanup()
 	test -n "$NON_ERROR_RTDEV" && $DMSETUP_PROG resume $DMERROR_RTNAME &>/dev/null
 	$DMSETUP_PROG resume $DMERROR_NAME > /dev/null 2>&1
 
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_unmount $SCRATCH_MNT > /dev/null 2>&1
 
 	test -n "$NON_ERROR_LOGDEV" && _dmsetup_remove $DMERROR_LOGNAME
 	test -n "$NON_ERROR_RTDEV" && _dmsetup_remove $DMERROR_RTNAME
common/dmflakey +2 -2
index 75a0a67c8..7368a3e5b 100644
@@ -71,7 +71,7 @@ _mount_flakey()
 
 _unmount_flakey()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_unmount $SCRATCH_MNT
 }
 
 _cleanup_flakey()
@@ -82,7 +82,7 @@ _cleanup_flakey()
 	test -n "$NON_FLAKEY_RTDEV" && $DMSETUP_PROG resume $FLAKEY_RTNAME &> /dev/null
 	$DMSETUP_PROG resume flakey-test > /dev/null 2>&1
 
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_unmount $SCRATCH_MNT > /dev/null 2>&1
 
 	_dmsetup_remove $FLAKEY_NAME
 	test -n "$NON_FLAKEY_LOGDEV" && _dmsetup_remove $FLAKEY_LOGNAME
common/dmhugedisk +1 -1
index 0757e1908..2d23f436c 100644
@@ -42,7 +42,7 @@ _dmhugedisk_init()
 
 _dmhugedisk_cleanup()
 {
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_unmount $SCRATCH_MNT > /dev/null 2>&1
 	_dmsetup_remove $DMHUGE_TEST
 	_dmsetup_remove $DMHUGE_TEST_ZERO
 }
common/dmlogwrites +2 -2
index c1c85de9d..a27e1966a 100644
@@ -110,7 +110,7 @@ _log_writes_mount()
 
 _log_writes_unmount()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_unmount $SCRATCH_MNT
 }
 
 # _log_writes_replay_log <mark>
@@ -141,7 +141,7 @@ _log_writes_remove()
 
 _log_writes_cleanup()
 {
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_unmount $SCRATCH_MNT > /dev/null 2>&1
 	_log_writes_remove
 }
 
common/dmthin +2 -2
index f2c00f5a7..453e43a04 100644
@@ -23,7 +23,7 @@ DMTHIN_VOL_DEV="/dev/mapper/$DMTHIN_VOL_NAME"
 
 _dmthin_cleanup()
 {
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_unmount $SCRATCH_MNT > /dev/null 2>&1
 	_dmsetup_remove $DMTHIN_VOL_NAME
 	_dmsetup_remove $DMTHIN_POOL_NAME
 	_dmsetup_remove $DMTHIN_META_NAME
@@ -32,7 +32,7 @@ _dmthin_cleanup()
 
 _dmthin_check_fs()
 {
-	$UMOUNT_PROG $SCRATCH_MNT > /dev/null 2>&1
+	_unmount $SCRATCH_MNT > /dev/null 2>&1
 	_check_scratch_fs $DMTHIN_VOL_DEV
 }
 
common/overlay +5 -5
index faa9339a6..01b6622f5 100644
@@ -142,18 +142,18 @@ _overlay_base_unmount()
 
 	[ -n "$dev" -a -n "$mnt" ] || return 0
 
-	$UMOUNT_PROG $mnt
+	_unmount $mnt
 }
 
 _overlay_test_unmount()
 {
-	$UMOUNT_PROG $TEST_DIR
+	_unmount $TEST_DIR
 	_overlay_base_unmount "$OVL_BASE_TEST_DEV" "$OVL_BASE_TEST_DIR"
 }
 
 _overlay_scratch_unmount()
 {
-	$UMOUNT_PROG $SCRATCH_MNT
+	_unmount $SCRATCH_MNT
 	_overlay_base_unmount "$OVL_BASE_SCRATCH_DEV" "$OVL_BASE_SCRATCH_MNT"
 }
 
@@ -342,7 +342,7 @@ _overlay_check_scratch_dirs()
 
 	# Need to umount overlay for scratch dir check
 	local ovl_mounted=`_is_dir_mountpoint $SCRATCH_MNT`
-	[ -z "$ovl_mounted" ] || $UMOUNT_PROG $SCRATCH_MNT
+	[ -z "$ovl_mounted" ] || _unmount $SCRATCH_MNT
 
 	# Check dirs with extra overlay options
 	_overlay_check_dirs $lowerdir $upperdir $workdir $*
@@ -387,7 +387,7 @@ _overlay_check_fs()
 	else
 		# Check and umount overlay for dir check
 		ovl_mounted=`_is_dir_mountpoint $ovl_mnt`
-		[ -z "$ovl_mounted" ] || $UMOUNT_PROG $ovl_mnt
+		[ -z "$ovl_mounted" ] || _unmount $ovl_mnt
 	fi
 
 	_overlay_check_dirs $base_mnt/$OVL_LOWER $base_mnt/$OVL_UPPER \
common/populate +4 -4
index 88c8ba2b3..4cf9c0691 100644
@@ -517,7 +517,7 @@ _scratch_xfs_populate() {
 	__populate_fragment_file "${SCRATCH_MNT}/RTRMAPBT"
 	__populate_fragment_file "${SCRATCH_MNT}/REFCOUNTBT"
 
-	umount "${SCRATCH_MNT}"
+	_scratch_unmount
 }
 
 # Populate an ext4 on the scratch device with (we hope) all known
@@ -619,7 +619,7 @@ _scratch_ext4_populate() {
 	# Make sure we get all the fragmentation we asked for
 	__populate_fragment_file "${SCRATCH_MNT}/S_IFREG.FMT_ETREE"
 
-	umount "${SCRATCH_MNT}"
+	_scratch_unmount
 }
 
 # Find the inode number of a file
@@ -772,7 +772,7 @@ _scratch_xfs_populate_check() {
 	dblksz="$(_xfs_get_dir_blocksize "$SCRATCH_MNT")"
 	leaf_lblk="$((32 * 1073741824 / blksz))"
 	node_lblk="$((64 * 1073741824 / blksz))"
-	umount "${SCRATCH_MNT}"
+	_scratch_unmount
 
 	__populate_check_xfs_dformat "${extents_file}" "extents"
 	__populate_check_xfs_dformat "${btree_file}" "btree"
@@ -885,7 +885,7 @@ _scratch_ext4_populate_check() {
 	extents_slink="$(__populate_find_inode "${SCRATCH_MNT}/S_IFLNK.FMT_EXTENTS")"
 	local_attr="$(__populate_find_inode "${SCRATCH_MNT}/ATTR.FMT_LOCAL")"
 	block_attr="$(__populate_find_inode "${SCRATCH_MNT}/ATTR.FMT_BLOCK")"
-	umount "${SCRATCH_MNT}"
+	_scratch_unmount
 
 	__populate_check_ext4_dformat "${extents_file}" "extents"
 	__populate_check_ext4_dformat "${etree_file}" "etree"
common/rc +25 -17

137 lines not on this page · raw message

tests/generic/042 +2 -2
index fd0ef705a..ced145dde 100755
@@ -44,7 +44,7 @@ _crashtest()
 		_filter_xfs_io
 	$here/src/godown -f $mnt
 
-	$UMOUNT_PROG $mnt
+	_unmount $mnt
 	_mount $img $mnt
 
 	# We should /never/ see 0xCD in the file, because we wrote that pattern
@@ -54,7 +54,7 @@ _crashtest()
 		_hexdump $file
 	fi
 
-	$UMOUNT_PROG $mnt
+	_unmount $mnt
 }
 
 # Modify as appropriate.
tests/generic/050 +2 -1
index ca02d3092..8e9456db2 100755
@@ -87,8 +87,9 @@ blockdev --setro $SCRATCH_DEV
 echo "mounting filesystem that needs recovery on a read-only device:"
 _try_scratch_mount 2>&1 | _filter_ro_mount | _filter_scratch
 
+# expects an error, so open code the unmount
 echo "unmounting read-only filesystem"
-_scratch_unmount 2>&1 | _filter_scratch | _filter_ending_dot
+$UMOUNT_PROG $SCRATCH_DEV 2>&1 | _filter_scratch | _filter_ending_dot
 
 #
 # This is the way out if the underlying device really is read-only.
tests/generic/067 +3 -3
index b561b7bc5..ccb1e3fbb 100755
@@ -66,7 +66,7 @@ umount_symlink_device()
 	rm -f $symlink
 	echo "# umount symlink to device, which is not mounted" >>$seqres.full
 	ln -s $SCRATCH_DEV $symlink
-	$UMOUNT_PROG $symlink >>$seqres.full 2>&1
+	_unmount $symlink >>$seqres.full 2>&1
 }
 
 # umount a path name that is 256 bytes long, this should fail gracefully,
@@ -78,7 +78,7 @@ umount_toolong_name()
 	_scratch_mount 2>&1 | tee -a $seqres.full
 
 	echo "# umount a too-long name" >>$seqres.full
-	$UMOUNT_PROG $longname >>$seqres.full 2>&1
+	_unmount $longname >>$seqres.full 2>&1
 	_scratch_unmount 2>&1 | tee -a $seqres.full
 }
 
@@ -93,7 +93,7 @@ lazy_umount_symlink()
 	rm -f $symlink
 	ln -s $SCRATCH_MNT/testdir $symlink
 
-	$UMOUNT_PROG -l $symlink >>$seqres.full 2>&1
+	_unmount -l $symlink >>$seqres.full 2>&1
 	# _scratch_unmount should not be blocked
 	_scratch_unmount 2>&1 | tee -a $seqres.full
 }
tests/generic/081 +1 -1
index 468c87ac9..df17ab6c1 100755
@@ -32,7 +32,7 @@ _cleanup()
 	# other tests to fail.
 	while test -e /dev/mapper/$vgname-$snapname || \
 	      test -e /dev/mapper/$vgname-$lvname; do
-		$UMOUNT_PROG $mnt >> $seqres.full 2>&1
+		_unmount $mnt >> $seqres.full 2>&1
 		$LVM_PROG lvremove -f $vgname/$snapname >>$seqres.full 2>&1
 		$LVM_PROG lvremove -f $vgname/$lvname >>$seqres.full 2>&1
 		$LVM_PROG vgremove -f $vgname >>$seqres.full 2>&1
tests/generic/085 +3 -3
index cfe6112d6..8fbcdce2e 100755
@@ -27,7 +27,7 @@ cleanup_dmdev()
 	$DMSETUP_PROG resume $lvdev >/dev/null 2>&1
 	[ -n "$pid" ] && kill -9 $pid 2>/dev/null
 	wait $pid
-	$UMOUNT_PROG $lvdev >/dev/null 2>&1
+	_unmount $lvdev >/dev/null 2>&1
 	_dmsetup_remove $node
 }
 
@@ -69,8 +69,8 @@ for ((i=0; i<100; i++)); do
 done &
 pid=$!
 for ((i=0; i<100; i++)); do
-	$MOUNT_PROG $lvdev $SCRATCH_MNT >/dev/null 2>&1
-	$UMOUNT_PROG $lvdev >/dev/null 2>&1
+	_mount $lvdev $SCRATCH_MNT >/dev/null 2>&1
+	_unmount $lvdev >/dev/null 2>&1
 done &
 pid="$pid $!"
 
tests/generic/108 +1 -1
index da13715f2..2709472f6 100755
@@ -18,7 +18,7 @@ _cleanup()
 {
 	cd /
 	echo running > /sys/block/`_short_dev $SCSI_DEBUG_DEV`/device/state
-	$UMOUNT_PROG $SCRATCH_MNT >>$seqres.full 2>&1
+	_unmount $SCRATCH_MNT >>$seqres.full 2>&1
 	$LVM_PROG vgremove -f $vgname >>$seqres.full 2>&1
 	$LVM_PROG pvremove -f $SCRATCH_DEV $SCSI_DEBUG_DEV >>$seqres.full 2>&1
 	$UDEV_SETTLE_PROG
tests/generic/171 +1 -1
index 5c8504d26..8726ee61d 100755
@@ -36,7 +36,7 @@ mkdir $testdir
 echo "Reformat with appropriate size"
 blksz="$(_get_block_size $testdir)"
 nr_blks=10240
-umount $SCRATCH_MNT
+_scratch_unmount
 sz_bytes=$((nr_blks * 8 * blksz))
 if [ $sz_bytes -lt $((32 * 1048576)) ]; then
 	sz_bytes=$((32 * 1048576))
tests/generic/172 +1 -1
index 17f817805..b67e817b6 100755
@@ -35,7 +35,7 @@ mkdir $testdir
 
 echo "Reformat with appropriate size"
 blksz="$(_get_block_size $testdir)"
-umount $SCRATCH_MNT
+_scratch_unmount
 
 file_size=$((768 * 1024 * 1024))
 fs_size=$((1024 * 1024 * 1024))
tests/generic/173 +1 -1
index d7ac7b15d..a19ff8070 100755
@@ -36,7 +36,7 @@ mkdir $testdir
 echo "Reformat with appropriate size"
 blksz="$(_get_block_size $testdir)"
 nr_blks=10240
-umount $SCRATCH_MNT
+_scratch_unmount
 sz_bytes=$((nr_blks * 8 * blksz))
 if [ $sz_bytes -lt $((32 * 1048576)) ]; then
 	sz_bytes=$((32 * 1048576))
tests/generic/174 +1 -1
index c7c62001d..32343d911 100755
@@ -37,7 +37,7 @@ mkdir $testdir
 echo "Reformat with appropriate size"
 blksz="$(_get_block_size $testdir)"
 nr_blks=10240
-umount $SCRATCH_MNT
+_scratch_unmount
 sz_bytes=$((nr_blks * 8 * blksz))
 if [ $sz_bytes -lt $((32 * 1048576)) ]; then
 	sz_bytes=$((32 * 1048576))
tests/generic/306 +1 -1
index a6ea654b6..8e118472d 100755
@@ -12,7 +12,7 @@ _begin_fstest auto quick rw
 # Override the default cleanup function.
 _cleanup()
 {
-    umount $BINDFILE
+    _unmount $BINDFILE
     cd /
     rm -f $tmp.*
 }
tests/generic/332 +1 -1
index cb15e77bd..fef4eabb8 100755
@@ -61,7 +61,7 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "Check for damage"
-umount $SCRATCH_MNT
+_scratch_unmount
 _repair_scratch_fs >> $seqres.full
 
 # success, all done
tests/generic/361 +1 -1
index c56157391..7273dd056 100755
@@ -16,7 +16,7 @@ _begin_fstest auto quick
 # Override the default cleanup function.
 _cleanup()
 {
-	$UMOUNT_PROG $fs_mnt
+	_unmount $fs_mnt
 	_destroy_loop_device $loop_dev
 	cd /
 	rm -f $tmp.*
tests/generic/373 +1 -1
index 3bd46963a..e6334398f 100755
@@ -60,7 +60,7 @@ md5sum $testdir/file | _filter_scratch
 md5sum $othertestdir/otherfile | filter_otherdir
 
 echo "Unmount otherdir"
-$UMOUNT_PROG $otherdir
+_unmount $otherdir
 rm -rf $otherdir
 
 # success, all done
tests/generic/374 +1 -1
index acb23d172..e56521aa0 100755
@@ -59,7 +59,7 @@ echo "Check output"
 md5sum $testdir/file $othertestdir/otherfile | filter_md5
 
 echo "Unmount otherdir"
-$UMOUNT_PROG $otherdir
+_unmount $otherdir
 rm -rf $otherdir
 
 # success, all done
tests/generic/395 +1 -1
index 45787fff0..f9c331adb 100755
@@ -75,7 +75,7 @@ mount --bind $SCRATCH_MNT $SCRATCH_MNT/ro_bind_mnt
 mount -o remount,ro,bind $SCRATCH_MNT/ro_bind_mnt
 _set_encpolicy $SCRATCH_MNT/ro_bind_mnt/ro_dir |& _filter_scratch
 _get_encpolicy $SCRATCH_MNT/ro_bind_mnt/ro_dir |& _filter_scratch
-umount $SCRATCH_MNT/ro_bind_mnt
+_unmount $SCRATCH_MNT/ro_bind_mnt
 
 # success, all done
 status=0
tests/generic/459 +1 -1
index 98177f6b5..daccc80ce 100755
@@ -28,7 +28,7 @@ _cleanup()
 	xfs_freeze -u $SCRATCH_MNT 2>/dev/null
 	cd /
 	rm -f $tmp.*
-	$UMOUNT_PROG $SCRATCH_MNT >>$seqres.full 2>&1
+	_unmount $SCRATCH_MNT >>$seqres.full 2>&1
 	$LVM_PROG vgremove -ff $vgname >>$seqres.full 2>&1
 	$LVM_PROG pvremove -ff $SCRATCH_DEV >>$seqres.full 2>&1
 	$UDEV_SETTLE_PROG
tests/generic/563 +5 -5
index e8db8acf0..f0d2f404c 100755
@@ -16,13 +16,13 @@ _begin_fstest auto quick
 # Override the default cleanup function.
 _cleanup()
 {
+	echo $$ > $cgdir/cgroup.procs
+	rmdir $cgdir/$seq-cg* > /dev/null 2>&1
+	_unmount $SCRATCH_MNT > /dev/null 2>&1
+	_destroy_loop_device $LOOP_DEV > /dev/null 2>&1
 	cd /
 	rm -f $tmp.*
 
-	echo $$ > $cgdir/cgroup.procs
-	rmdir $cgdir/$seq-cg* > /dev/null 2>&1
-	umount $SCRATCH_MNT > /dev/null 2>&1
-	_destroy_loop_device $LOOP_DEV > /dev/null 2>&1
 }
 
 # Import common functions.
@@ -80,7 +80,7 @@ reset()
 	rmdir $cgdir/$seq-cg* > /dev/null 2>&1
 	$XFS_IO_PROG -fc "pwrite 0 $iosize" $SCRATCH_MNT/file \
 		>> $seqres.full 2>&1
-	umount $SCRATCH_MNT || _fail "umount failed"
+	_unmount $SCRATCH_MNT || _fail "umount failed"
 	_mount $LOOP_DEV $SCRATCH_MNT || _fail "mount failed"
 	stat $SCRATCH_MNT/file > /dev/null
 }
tests/generic/604 +1 -1
index c2e03c2ea..744d34565 100755
@@ -26,7 +26,7 @@ done
 # mount the base fs.  Delay the mount attempt by a small amount in the hope
 # that the mount() call will try to lock s_umount /after/ umount has already
 # taken it.
-$UMOUNT_PROG $SCRATCH_MNT &
+_unmount $SCRATCH_MNT &
 sleep 0.01s ; _scratch_mount
 wait
 
tests/generic/631 +1 -1
index c7c95e560..642d47863 100755
@@ -84,7 +84,7 @@ worker() {
 		touch $mergedir/etc/access.conf
 		mv $mergedir/etc/access.conf $mergedir/etc/access.conf.bak
 		touch $mergedir/etc/access.conf
-		umount $mergedir
+		_unmount $mergedir
 	done
 	rm -f $SCRATCH_MNT/workers/$tag
 }
tests/generic/648 +3 -3
index e4c9990e1..7473c9d33 100755
@@ -18,7 +18,7 @@ _cleanup()
 {
 	_kill_fsstress
 	if [ -n "$loopmnt" ]; then
-		$UMOUNT_PROG $loopmnt 2>/dev/null
+		_unmount $loopmnt 2>/dev/null
 		rm -r -f $loopmnt
 	fi
 	_dmerror_unmount
@@ -102,7 +102,7 @@ while _soak_loop_running $((25 * TIME_FACTOR)); do
 
 	# Mount again to replay log after loading working table, so we have a
 	# consistent fs after test.
-	$UMOUNT_PROG $loopmnt
+	_unmount $loopmnt
 	is_unmounted=1
 	# We must unmount dmerror at here, or whole later testing will crash.
 	# So try to umount enough times, before we have no choice.
@@ -128,7 +128,7 @@ done
 # Make sure the fs image file is ok
 if [ -f "$loopimg" ]; then
 	if _mount $loopimg $loopmnt -o loop; then
-		$UMOUNT_PROG $loopmnt &> /dev/null
+		_unmount $loopmnt &> /dev/null
 	else
 		_metadump_dev $DMERROR_DEV $seqres.scratch.final.md
 		echo "final scratch mount failed"
tests/generic/698 +2 -2
index 28928b2fb..0029df90f 100755
@@ -17,8 +17,8 @@ _begin_fstest auto quick perms attr idmapped mount
 _cleanup()
 {
 	cd /
-	$UMOUNT_PROG $SCRATCH_MNT/target-mnt 2>/dev/null
-	$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+	_unmount $SCRATCH_MNT/target-mnt 2>/dev/null
+	_unmount $SCRATCH_MNT 2>/dev/null
 	rm -r -f $tmp.*
 }
 
tests/generic/699 +4 -4
index 677307538..3079a861d 100755
@@ -15,9 +15,9 @@ _begin_fstest auto quick perms attr idmapped mount
 _cleanup()
 {
 	cd /
-	$UMOUNT_PROG $SCRATCH_MNT/target-mnt
-	$UMOUNT_PROG $SCRATCH_MNT/ovl-merge 2>/dev/null
-	$UMOUNT_PROG $SCRATCH_MNT 2>/dev/null
+	_unmount $SCRATCH_MNT/target-mnt
+	_unmount $SCRATCH_MNT/ovl-merge 2>/dev/null
+	_unmount $SCRATCH_MNT 2>/dev/null
 	rm -r -f $tmp.*
 }
 
@@ -113,7 +113,7 @@ setup_overlayfs_idmapped_lower_metacopy_on()
 
 reset_overlayfs()
 {
-	$UMOUNT_PROG $SCRATCH_MNT/ovl-merge 2>/dev/null
+	_unmount $SCRATCH_MNT/ovl-merge 2>/dev/null
 	rm -rf $upper $work $merge
 }
 
tests/generic/704 +1 -1
index f39d47066..f452f9e9c 100755
@@ -14,7 +14,7 @@ _cleanup()
 {
 	cd /
 	rm -r -f $tmp.*
-	[ -d "$SCSI_DEBUG_MNT" ] && $UMOUNT_PROG $SCSI_DEBUG_MNT 2>/dev/null
+	[ -d "$SCSI_DEBUG_MNT" ] && _unmount $SCSI_DEBUG_MNT 2>/dev/null
 	_put_scsi_debug_dev
 }
 
tests/generic/717 +1 -1
index 4378e964a..2ecd2888d 100755
@@ -85,7 +85,7 @@ mkdir -p $SCRATCH_MNT/xyz
 mount --bind $dir $SCRATCH_MNT/xyz --bind
 _pwrite_byte 0x60 0 $((blksz * (nrblks + 2))) $dir/c >> $seqres.full
 $XFS_IO_PROG -c "exchangerange $SCRATCH_MNT/xyz/c" $dir/a
-umount $SCRATCH_MNT/xyz
+_unmount $SCRATCH_MNT/xyz
 
 echo Swapping a file with itself
 $XFS_IO_PROG -c "exchangerange $dir/a" $dir/a
tests/generic/730 +1 -1
index 062314ea0..6251980e3 100755
@@ -12,7 +12,7 @@ _begin_fstest auto quick
 _cleanup()
 {
 	cd /
-	$UMOUNT_PROG $SCSI_DEBUG_MNT >>$seqres.full 2>&1
+	_unmount $SCSI_DEBUG_MNT >>$seqres.full 2>&1
 	_put_scsi_debug_dev
 	rm -f $tmp.*
 }
tests/generic/731 +1 -1
index cd39e8b09..4ac5e1a3e 100755
@@ -13,7 +13,7 @@ _begin_fstest auto quick
 _cleanup()
 {
 	cd /
-	$UMOUNT_PROG $SCSI_DEBUG_MNT >>$seqres.full 2>&1
+	_unmount $SCSI_DEBUG_MNT >>$seqres.full 2>&1
 	_put_scsi_debug_dev
 	rm -f $tmp.*
 }
tests/generic/732 +2 -2
index d08028c23..e907a009f 100755
@@ -15,8 +15,8 @@ _begin_fstest auto quick rename
 # Override the default cleanup function.
 _cleanup()
 {
-	$UMOUNT_PROG $testdir1 2>/dev/null
-	$UMOUNT_PROG $testdir2 2>/dev/null
+	_unmount $testdir1 2>/dev/null
+	_unmount $testdir2 2>/dev/null
 	cd /
 	rm -r -f $tmp.*
 }
tests/generic/741 +1 -1
index 4b480b5a1..cac7045eb 100755
@@ -13,7 +13,7 @@ _begin_fstest auto quick volume tempfsid
 # Override the default cleanup function.
 _cleanup()
 {
-	umount $extra_mnt &> /dev/null
+	_unmount $extra_mnt &> /dev/null
 	rm -rf $extra_mnt
 	_unmount_flakey
 	_cleanup_flakey
tests/generic/744 +3 -3
index a31d8d7a4..df8f6ae9b 100755
@@ -15,8 +15,8 @@ _cleanup()
 	cd /
 	rm -r -f $tmp.*
 
-	$UMOUNT_PROG $mnt2 &> /dev/null
-	$UMOUNT_PROG $mnt1 &> /dev/null
+	_unmount $mnt2 &> /dev/null
+	_unmount $mnt1 &> /dev/null
 	[ -b "$loop_dev2" ] && losetup -d $loop_dev2
 	[ -b "$loop_dev1" ] && losetup -d $loop_dev1
 	[ -n "$seq" ] && rm -rf $TEST_DIR/$seq
@@ -42,7 +42,7 @@ clone_filesystem()
 
 	_mount $dev1 $mnt1
 	$XFS_IO_PROG -fc 'pwrite -S 0x61 0 9000' $mnt1/foo >> $seqres.full
-	$UMOUNT_PROG $mnt1
+	_unmount $mnt1
 
 	# device dump of $dev1 to $dev2
 	dd if=$dev1 of=$dev2 conv=fsync status=none || _fail "dd failed: $?"
tests/generic/746 +4 -4
index 5734f254d..671910606 100755
@@ -38,7 +38,7 @@ esac
 # Override the default cleanup function.
 _cleanup()
 {
-	$UMOUNT_PROG $loop_dev &> /dev/null
+	_unmount $loop_mnt &> /dev/null
 	_destroy_loop_device $loop_dev
 	if [ $status -eq 0 ]; then
 		rm -rf $tmp
@@ -53,7 +53,7 @@ get_holes()
 	# in-core state that will perturb the free space map on umount.  Stick
 	# to established convention which requires the filesystem to be
 	# unmounted while we probe the underlying file.
-	$UMOUNT_PROG $loop_mnt
+	_unmount $loop_mnt
 
 	# FIEMAP only works on regular files, so call it on the backing file
 	# and not the loop device like everything else
@@ -66,7 +66,7 @@ get_free_sectors()
 {
 	case $FSTYP in
 	ext4)
-	$UMOUNT_PROG $loop_mnt
+	_unmount $loop_mnt
 	$DUMPE2FS_PROG $loop_dev  2>&1 | grep " Free blocks" | cut -d ":" -f2- | \
 		tr ',' '\n' | $SED_PROG 's/^ //' | \
 		$AWK_PROG -v spb=$sectors_per_block 'BEGIN{FS="-"};
@@ -80,7 +80,7 @@ get_free_sectors()
 	xfs)
 	agsize=`$XFS_INFO_PROG $loop_mnt | $SED_PROG -n 's/.*agsize=\(.*\) blks.*/\1/p'`
 	# Convert free space (agno, block, length) to (start sector, end sector)
-	$UMOUNT_PROG $loop_mnt
+	_unmount $loop_mnt
 	$XFS_DB_PROG -r -c "freesp -d" $loop_dev | $SED_PROG '/^.*from/,$d'| \
 		 $AWK_PROG -v spb=$sectors_per_block -v agsize=$agsize \
 		'{ print spb * ($1 * agsize + $2), spb * ($1 * agsize + $2 + $3) - 1 }'
tests/xfs/014 +3 -3
index 098f64186..f638e4b16 100755
@@ -22,7 +22,7 @@ _begin_fstest auto enospc quick quota prealloc
 _cleanup()
 {
 	cd /
-	umount $LOOP_MNT 2>/dev/null
+	_unmount $LOOP_MNT 2>/dev/null
 	_scratch_unmount 2>/dev/null
 	rm -f $tmp.*
 }
@@ -168,13 +168,13 @@ LOOP_MNT=$SCRATCH_MNT/$seq.mnt
 $MKFS_XFS_PROG -d "file=1,name=$LOOP_FILE,size=10g" >> $seqres.full 2>&1
 
 mkdir -p $LOOP_MNT
-mount -t xfs -o loop,uquota,gquota $LOOP_FILE $LOOP_MNT || \
+_mount -o uquota,gquota $LOOP_FILE $LOOP_MNT || \
 	_fail "Failed to mount loop fs."
 
 _test_enospc $LOOP_MNT
 _test_edquot $LOOP_MNT
 
-umount $LOOP_MNT
+_unmount $LOOP_MNT
 
 echo $orig_sp_time > /proc/sys/fs/xfs/speculative_prealloc_lifetime
 
tests/xfs/073 +10 -10
index 28f1fad08..e4b17c5e7 100755
@@ -21,9 +21,9 @@ _cleanup()
 {
 	cd /
 	_scratch_unmount 2>/dev/null
-	umount $imgs.loop 2>/dev/null
+	_unmount $imgs.loop 2>/dev/null
+	_unmount $imgs.source_dir 2>/dev/null
 	[ -d $imgs.loop ] && rmdir $imgs.loop
-	umount $imgs.source_dir 2>/dev/null
 	[ -d $imgs.source_dir ] && rm -rf $imgs.source_dir
 	rm -f $imgs.* $tmp.* /var/tmp/xfs_copy.log.*
 }
@@ -65,10 +65,10 @@ _verify_copy()
 	rmdir $target_dir 2>/dev/null
 	mkdir $target_dir
 
-	mount -t xfs -o loop $target $target_dir 2>/dev/null
+	_mount -t xfs -o loop $target $target_dir 2>/dev/null
 	if [ $? -ne 0 ]; then
 		echo retrying mount with nouuid option >>$seqres.full
-		mount -t xfs -o loop -o nouuid $target $target_dir
+		_mount -t xfs -o loop -o nouuid $target $target_dir
 		if [ $? -ne 0 ]; then
 			echo mount failed - evil!
 			return
@@ -87,19 +87,19 @@ _verify_copy()
 
 	echo comparing new image geometry to old
 	$XFS_INFO_PROG $source_dir \
-		| _filter_copy $source $source_dir '/dev/loop.' '#' \
+		| _filter_copy $source $source_dir '/dev/loop[0-9]\+' '#' \
 		| tr -s ' ' \
 		> $tmp.geometry1
 	$XFS_INFO_PROG $target_dir \
-		| _filter_copy $target $target_dir '/dev/loop.' '#' \
+		| _filter_copy $target $target_dir '/dev/loop[0-9]\+' '#' \
 		| tr -s ' ' \
 		> $tmp.geometry2
 	[ -s $tmp.geometry1 ] || echo no geometry output
 	diff -u $tmp.geometry1 $tmp.geometry2
 
 	echo unmounting and removing new image
-	umount $source_dir
-	umount $target_dir > /dev/null 2>&1
+	_unmount $source_dir
+	_unmount $target_dir > /dev/null 2>&1
 	rm -f $target
 }
 
@@ -134,10 +134,10 @@ ${MKFS_XFS_PROG} -dfile,name=$imgs.source,size=100g \
 rmdir $imgs.source_dir 2>/dev/null
 mkdir $imgs.source_dir
 
-mount -t xfs -o loop $imgs.source $imgs.source_dir
+_mount -t xfs -o loop $imgs.source $imgs.source_dir
 loop2=`mount | grep $imgs.source | grep -o -e 'loop=.*[^),]' | grep -o -e '/.*$'`
 cp -a $here $imgs.source_dir
-mount -t xfs -o remount,ro $imgs.source $imgs.source_dir
+_mount -t xfs -o remount,ro $imgs.source $imgs.source_dir
 $XFS_COPY_PROG $imgs.source $imgs.image | _filter_copy '#' $imgs.image '#' '#'
 _verify_copy $imgs.image $imgs.source $imgs.source_dir
 
tests/xfs/074 +1 -1
index 278f0ade6..6a03c8111 100755
@@ -72,7 +72,7 @@ $XFS_IO_PROG -ft \
 	-c "falloc 1023m 2g" \
 	$LOOP_MNT/foo >> $seqres.full
 
-umount $LOOP_MNT
+_unmount $LOOP_MNT
 _check_xfs_filesystem $LOOP_DEV none none
 
 # success, all done
tests/xfs/078 +3 -3
index 834c99a00..6b325e05f 100755
@@ -16,7 +16,7 @@ _cleanup()
 {
 	cd /
 	rm -f $tmp.*
-	$UMOUNT_PROG $LOOP_MNT 2>/dev/null
+	_unmount $LOOP_MNT 2>/dev/null
 	[ -n "$LOOP_DEV" ] && _destroy_loop_device $LOOP_DEV 2>/dev/null
 	# try to keep the image file if test fails
 	[ $status -eq 0 ] && rm -f $LOOP_IMG
@@ -75,13 +75,13 @@ _grow_loop()
 	$XFS_IO_PROG -c "pwrite $new_size $bsize" $LOOP_IMG | _filter_io
 	LOOP_DEV=`_create_loop_device $LOOP_IMG`
 	echo "*** mount loop filesystem"
-	$MOUNT_PROG -t xfs $LOOP_DEV $LOOP_MNT
+	_mount -t xfs $LOOP_DEV $LOOP_MNT
 
 	echo "*** grow loop filesystem"
 	$XFS_GROWFS_PROG $LOOP_MNT 2>&1 |  _filter_growfs 2>&1
 
 	echo "*** unmount"
-	$UMOUNT_PROG -d $LOOP_MNT && LOOP_DEV=
+	_unmount $LOOP_MNT
 
 	# Large grows takes forever to check..
 	if [ "$check" -gt "0" ]
tests/xfs/148 +3 -3
index 9e6798f99..c42c9b119 100755
@@ -14,7 +14,7 @@ _begin_fstest auto quick fuzzers
 _cleanup()
 {
 	cd /
-	$UMOUNT_PROG $mntpt > /dev/null 2>&1
+	_unmount $mntpt > /dev/null 2>&1
 	_destroy_loop_device $loopdev > /dev/null 2>&1
 	rm -r -f $tmp.*
 }
@@ -90,7 +90,7 @@ cat $tmp.log >> $seqres.full
 cat $tmp.log | _filter_test_dir
 
 # Corrupt the entries
-$UMOUNT_PROG $mntpt
+_unmount $mntpt
 _destroy_loop_device $loopdev
 cp $imgfile $imgfile.old
 sed -b \
@@ -121,7 +121,7 @@ fi
 echo "does repair complain?" >> $seqres.full
 
 # Does repair complain about this?
-$UMOUNT_PROG $mntpt
+_unmount $mntpt
 $XFS_REPAIR_PROG -n $loopdev >> $seqres.full 2>&1
 res=$?
 test $res -eq 1 || \
tests/xfs/149 +2 -2
index f1b2405e7..f2187109b 100755
@@ -22,7 +22,7 @@ loop_symlink=$TEST_DIR/loop_symlink.$$
 # Override the default cleanup function.
 _cleanup()
 {
-    $UMOUNT_PROG $mntdir
+    _unmount $mntdir
     [ -n "$loop_dev" ] && _destroy_loop_device $loop_dev
     rmdir $mntdir
     rm -f $loop_symlink
@@ -73,7 +73,7 @@ echo "=== xfs_growfs - check device symlink ==="
 $XFS_GROWFS_PROG -D 12288 $loop_symlink > /dev/null
 
 echo "=== unmount ==="
-$UMOUNT_PROG $mntdir || _fail "!!! failed to unmount"
+_unmount $mntdir || _fail "!!! failed to unmount"
 
 echo "=== mount device symlink ==="
 $MOUNT_PROG $loop_symlink $mntdir || _fail "!!! failed to loopback mount"
tests/xfs/186 +2 -2
index 88f02585e..6f82e0cda 100755
@@ -87,7 +87,7 @@ _do_eas()
 		_create_eas $2 $3
 	fi
 	echo ""
-	cd /; $UMOUNT_PROG $SCRATCH_MNT
+	cd /; _unmount $SCRATCH_MNT
 	_print_inode
 }
 
@@ -99,7 +99,7 @@ _do_dirents()
 	echo ""
 	_scratch_mount
 	_create_dirents $1 $2
-	cd /; $UMOUNT_PROG $SCRATCH_MNT
+	cd /; _unmount $SCRATCH_MNT
 	_print_inode
 }
 
tests/xfs/216 +8 -2
index 680239b4e..6b8b2eb22 100755
@@ -12,6 +12,12 @@ _begin_fstest log metadata auto quick
 # Import common functions.
 . ./common/filter
 
+_cleanup()
+{
+	_unmount $LOOP_MNT > /dev/null 2>&1
+	cd /
+	rm -f $tmp.*
+}
 
 _require_scratch
 _scratch_mkfs_xfs >/dev/null 2>&1
@@ -50,9 +56,9 @@ _do_mkfs()
 		echo -n "fssize=${i}g "
 		$MKFS_XFS_PROG -f -b size=4096 -l version=2 \
 			-d name=$LOOP_DEV,size=${i}g $loop_mkfs_opts |grep log
-		mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
+		_mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
 		echo "test write" > $LOOP_MNT/test
-		umount $LOOP_MNT > /dev/null 2>&1
+		_unmount $LOOP_MNT > /dev/null 2>&1
 	done
 }
 # make large holey file
tests/xfs/217 +2 -2
index 41caaf738..b2eb34490 100755
@@ -29,9 +29,9 @@ _do_mkfs()
 		echo -n "fssize=${i}g "
 		$MKFS_XFS_PROG -f -b size=4096 -l version=2 \
 			-d name=$LOOP_DEV,size=${i}g |grep log
-		mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
+		_mount -o loop -t xfs $LOOP_DEV $LOOP_MNT
 		echo "test write" > $LOOP_MNT/test
-		umount $LOOP_MNT > /dev/null 2>&1
+		_unmount $LOOP_MNT > /dev/null 2>&1
 
 		# punch out the previous blocks so that we keep the amount of
 		# disk space the test requires down to a minimum.
tests/xfs/250 +4 -3
index f8846be6e..4e3473ebc 100755
@@ -13,9 +13,10 @@ _begin_fstest auto quick rw prealloc metadata
 _cleanup()
 {
 	cd /
-	umount $LOOP_MNT 2>/dev/null
+	_unmount $LOOP_MNT 2>/dev/null
 	rm -f $LOOP_DEV
 	rmdir $LOOP_MNT
+	rm -f $tmp.*
 }
 
 # Import common functions.
@@ -54,13 +55,13 @@ _test_loop()
 		| _filter_mkfs 2>/dev/null
 
 	echo "*** mount loop filesystem"
-	mount -t xfs -o loop $LOOP_DEV $LOOP_MNT
+	_mount -t xfs -o loop $LOOP_DEV $LOOP_MNT
 
 	echo "*** preallocate large file"
 	$XFS_IO_PROG -f -c "resvsp 0 $fsize" $LOOP_MNT/foo | _filter_io
 
 	echo "*** unmount loop filesystem"
-	umount $LOOP_MNT > /dev/null 2>&1
+	_unmount $LOOP_MNT > /dev/null 2>&1
 
 	echo "*** check loop filesystem"
 	 _check_xfs_filesystem $LOOP_DEV none none
tests/xfs/289 +2 -2
index cf0f2883c..d234f212d 100755
@@ -13,8 +13,8 @@ _begin_fstest growfs auto quick
 # Override the default cleanup function.
 _cleanup()
 {
-    $UMOUNT_PROG $tmpdir
-    $UMOUNT_PROG $tmpbind
+    _unmount $tmpdir
+    _unmount $tmpbind
     rmdir $tmpdir
     rm -f $tmpsymlink
     rmdir $tmpbind
tests/xfs/507 +1 -1
index 75c183c07..52d9b94b4 100755
@@ -22,7 +22,7 @@ _register_cleanup "_cleanup" BUS
 _cleanup()
 {
 	cd /
-	test -n "$loop_mount" && $UMOUNT_PROG $loop_mount > /dev/null 2>&1
+	test -n "$loop_mount" && _unmount $loop_mount > /dev/null 2>&1
 	test -n "$loop_dev" && _destroy_loop_device $loop_dev
 	rm -rf $tmp.*
 }
tests/xfs/513 +2 -2
index 5585a9c8e..5895e6e2d 100755
@@ -14,7 +14,7 @@ _cleanup()
 {
 	cd /
 	rm -f $tmp.*
-	$UMOUNT_PROG $LOOP_MNT 2>/dev/null
+	_unmount $LOOP_MNT 2>/dev/null
 	if [ -n "$LOOP_DEV" ];then
 		_destroy_loop_device $LOOP_DEV 2>/dev/null
 	fi
@@ -89,7 +89,7 @@ get_mount_info()
 
 force_unmount()
 {
-	$UMOUNT_PROG $LOOP_MNT >/dev/null 2>&1
+	_unmount $LOOP_MNT >/dev/null 2>&1
 }
 
 # _do_test <mount options> <should be mounted?> [<key string> <key should be found?>]
tests/xfs/544 +1 -1
index bd694453d..b7eef51c7 100755
@@ -15,7 +15,7 @@ _cleanup()
 	_cleanup_dump
 	cd /
 	rm -r -f $tmp.*
-	$UMOUNT_PROG $TEST_DIR/dest.$seq 2> /dev/null
+	_unmount $TEST_DIR/dest.$seq 2> /dev/null
 	rmdir $TEST_DIR/src.$seq 2> /dev/null
 	rmdir $TEST_DIR/dest.$seq 2> /dev/null
 }
tests/xfs/606 +1 -1
index 02de601d6..f958bddd8 100755
@@ -14,7 +14,7 @@ _begin_fstest auto quick growfs
 _cleanup()
 {
 	local dev
-	$UMOUNT_PROG $LOOP_MNT 2>/dev/null
+	_unmount $LOOP_MNT 2>/dev/null
 	dev=$(losetup -j testfile | cut -d: -f1)
 	losetup -d $dev 2>/dev/null
 	rm -rf $LOOP_IMG $LOOP_MNT
tests/xfs/613 +2 -2
index 1947afec7..6ba3d87bf 100755
@@ -14,7 +14,7 @@ _cleanup()
 {
 	cd /
 	rm -f $tmp.*
-	$UMOUNT_PROG $LOOP_MNT 2>/dev/null
+	_unmount $LOOP_MNT 2>/dev/null
 	if [ -n "$LOOP_DEV" ];then
 		_destroy_loop_device $LOOP_DEV 2>/dev/null
 	fi
@@ -90,7 +90,7 @@ get_mount_info()
 
 force_unmount()
 {
-	$UMOUNT_PROG $LOOP_MNT >/dev/null 2>&1
+	_unmount $LOOP_MNT >/dev/null 2>&1
 }
 
 # _do_test <mount options> <should be mounted?> [<key string> <key should be found?>]
Signature · 1 line
2.45.2