Skip to contentSkip to conversation

xfstests flakiness in generic/095 triggered by fio 3.37

PATCH Theodore Ts'o 2 participants 2 messages Apr 15, 2024

lore
Share & tools
lore
#0 lore raw

I was updating various components in my xfstests test appliance, and it appears that fio 3.37 is causing generic/095 to become flaky. Previously, I had been using fio 3.35 and a quick test shows that fio 3.36 seems to be OK. So the change seems to be somethinig that was introduced between fio 3.36 and 3.37.

The test failures seem to be caused by the following unexpected message in the output file:

fio: first I/O failed. If /xt-vdc/file1 is a zoned block device, consider --zonemode=zbd

The following comment in tests/generic/095 suggests to my suspicious mind that the above warning message isn't getting suppressed when --ignore_error=,EIO is passed to fio:

# There's a known EIO failure to report collisions between directio and buffered

  # writes to userspace, refer to upstream linux 5a9d929d6e13. So ignore EIO error
  # at here.

.... but I could be wrong about that.

For now, I've just switched to using fio 3.36, since there's nothing in my testing that really _requires_ 3.37. It might be that we should just work around this by filtering out the warning message in xfstests. Or maybe we should make a change in fio?

- Ted

tmp/a renamed from tmp/b +10 -8
@@ -1,18 +1,20 @@
-TESTRUNID: tytso-20240415133921
+TESTRUNID: tytso-20240415131932
 KERNEL:    kernel 6.9.0-rc3-xfstests #328 SMP PREEMPT_DYNAMIC Mon Apr 15 13:04:29 EDT 2024 x86_64
 CMDLINE:   -c ext4/4k,xfs/4k -C 20 generic/095
 CPUS:      2
 MEM:       7680
 
-ext4/4k: 20 tests, 86 seconds
-xfs/4k: 20 tests, 75 seconds
-Totals: 40 tests, 0 skipped, 0 failures, 0 errors, 161s
+ext4/4k: 20 tests, 8 failures, 86 seconds
+  Flaky: generic/095: 40% (8/20)
+xfs/4k: 20 tests, 7 failures, 71 seconds
+  Flaky: generic/095: 35% (7/20)
+Totals: 40 tests, 0 skipped, 15 failures, 0 errors, 157s
 
-FSTESTIMG: gce-xfstests/xfstests-amd64-202404151325
+FSTESTIMG: gce-xfstests/xfstests-amd64-202404122347
 FSTESTPRJ: gce-xfstests
 FSTESTVER: blktests 56d7e53 (Fri, 12 Apr 2024 16:03:20 +0900)
 FSTESTVER: e2fsprogs v1.47.0-144-g441741fc (Tue, 8 Aug 2023 16:08:52 -0400)
-FSTESTVER: fio  fio-3.36 (Fri, 20 Oct 2023 04:30:43 -0600)
+FSTESTVER: fio  fio-3.37 (Tue, 26 Mar 2024 15:13:51 -0600)
 FSTESTVER: fsverity v1.6 (Wed, 20 Mar 2024 21:21:46 -0700)
 FSTESTVER: ima-evm-utils v1.5 (Mon, 6 Mar 2023 07:40:07 -0500)
 FSTESTVER: libaio   libaio-0.3.108-82-gb8eadc9 (Thu, 2 Jun 2022 13:33:11 +0200)
@@ -21,10 +23,10 @@
 FSTESTVER: quota  v4.05-56-g00534e7 (Mon, 29 Jan 2024 11:59:57 +0100)
 FSTESTVER: util-linux v2.40 (Wed, 27 Mar 2024 12:46:59 +0100)
 FSTESTVER: xfsprogs v6.6.0 (Mon, 5 Feb 2024 13:43:13 +0100)
-FSTESTVER: xfstests-bld f9077fe2 (Sat, 13 Apr 2024 16:34:23 -0400)
+FSTESTVER: xfstests-bld 4bb62098 12 Apr 2024 20:03:16 -0400)
 FSTESTVER: xfstests v2024.03.31-10-g554b613ae (Fri, 12 Apr 2024 22:06:49 -0400)
 FSTESTVER: zz_build-distro bookworm
 FSTESTCFG: ext4/4k xfs/4k
 FSTESTSET: generic/095
 FSTESTOPT: count 20 fail_loop_count 0 aex
-GCE ID:    7786599483943475363
+GCE ID:    6536244786506030920

Note: the changes between 4bb62098 and f9077fe2 in xfstests-bld do not affect the test appliance. This commits reflect some minor fixes to xfstests-bld's regression test suite, and adding flex and libsqlite3-dev to the build chroot, since that's needed by newer versions of util-linux....

#1 lore raw
Quoted 19 lines · Theodore Ts'o

I was updating various components in my xfstests test appliance, and it appears that fio 3.37 is causing generic/095 to become flaky. Previously, I had been using fio 3.35 and a quick test shows that fio 3.36 seems to be OK. So the change seems to be somethinig that was introduced between fio 3.36 and 3.37.

The test failures seem to be caused by the following unexpected message in the output file:

fio: first I/O failed. If /xt-vdc/file1 is a zoned block device, consider --zonemode=zbd

The following comment in tests/generic/095 suggests to my suspicious mind that the above warning message isn't getting suppressed when --ignore_error=,EIO is passed to fio:

# There's a known EIO failure to report collisions between directio and buffered

  # writes to userspace, refer to upstream linux 5a9d929d6e13. So ignore EIO error
  # at here.
Quoted 6 lines

.... but I could be wrong about that.

For now, I've just switched to using fio 3.36, since there's nothing in my testing that really _requires_ 3.37. It might be that we should just work around this by filtering out the warning message in xfstests. Or maybe we should make a change in fio?

I'll just revert that change, not sure how it slipped by that it also change it from informational to an error message.

Signature · 1 line
Jens Axboe