Skip to contentSkip to conversation

generic: test which tries to exercise AIO/DIO into unwritten space

PATCH In review Theodore Ts'o 4 participants 11 messages Feb 10, 2021 – Mar 15, 2021 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 generic: test which tries to exercise AIO/DIO into unwritten space

This test verifies that the an unwritten extent is properly marked as written after writing into it.

There was a hard-to-hit bug which would occasionally trigger with ext4 for which this test was a reproducer. This has been fixed after moving ext4 to use iomap for Direct I/O's, although as of this writing, there are still some occasional failures on ext4 when block size < page size.

---
 tests/generic/623     | 109 ++++++++++++++++++++++++++++++++++++++++++
 tests/generic/623.out |   4 ++
 tests/generic/group   |   1 +
 3 files changed, 114 insertions(+)
 create mode 100755 tests/generic/623
 create mode 100644 tests/generic/623.out
tests/generic/623 new file +109 -0
new file mode 100755
index 00000000..74136fef
@@ -0,0 +1,109 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+#
+# FSQA Test No. 623
+#
+# AIO/DIO stress test
+# Run random AIO/DIO activity on an file system with unwritten regions
+#
+# This test verifies that the an unwritten extent is properly marked
+# as written after writing into it.
+#
+# There was a hard-to-hit bug which would occasionally trigger with
+# ext4 for which this test was a reproducer.  This has been fixed
+# after moving ext4 to use iomap for Direct I/O's, although as of this
+# writing, there are still some occasional failures on ext4 when block
+# size < page size.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+fio_config=$tmp.fio
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs generic
+_require_test
+_require_scratch
+_require_odirect
+_require_block_device $SCRATCH_DEV
+
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+FILE_SIZE=$(((BLK_DEV_SIZE * 512) * 3 / 4))
+
+max_file_size=$((5 * 1024 * 1024 * 1024))
+if [ $max_file_size -lt $FILE_SIZE ]; then
+	FILE_SIZE=$max_file_size
+fi
+SIZE=$((FILE_SIZE / 2))
+
+cat >$fio_config <<EOF
+###########
+# $seq test fio activity
+# Filenames derived from jobsname and jobid like follows:
+# ${JOB_NAME}.${JOB_ID}.${ITERATION_ID}
+[global]
+ioengine=libaio
+bs=128k
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=${SIZE}
+iodepth=$((128*$LOAD_FACTOR))
+fallocate=native
+
+# Perform direct aio and verify data
+# This test case should check use-after-free issues
+[aio-dio-verifier]
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+direct=1
+blocksize_range=4100k-8200k
+blockalign=4k
+rw=randwrite
+filename=test-file
+
+EOF
+
+rm -f $seqres.full
+
+_require_fio $fio_config
+_require_xfs_io_command "falloc"
+
+_workout()
+{
+	echo ""
+	echo "Run fio with random aio-dio pattern"
+	echo ""
+	cat $fio_config >>  $seqres.full
+	run_check $FIO_PROG $fio_config
+}
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+	_scratch_unmount 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi
+status=0
+exit
tests/generic/623.out new file +4 -0
new file mode 100644
index 00000000..e10c7fd9
@@ -0,0 +1,4 @@
+QA output created by 623
+
+Run fio with random aio-dio pattern
+
tests/generic/group +1 -0
index b10fdea4..24f53ed7 100644
@@ -625,3 +625,4 @@
 620 auto mount quick
 621 auto quick encrypt
 622 auto shutdown metadata atime
+623 aio rw stress
Signature · 1 line
2.30.0
#1 lore raw
Quoted 10 lines · Theodore Ts'o

This test verifies that the an unwritten extent is properly marked as written after writing into it.

There was a hard-to-hit bug which would occasionally trigger with ext4 for which this test was a reproducer. This has been fixed after moving ext4 to use iomap for Direct I/O's, although as of this writing, there are still some occasional failures on ext4 when block size < page size.

Ping?

Quoted 145 lines
---
 tests/generic/623     | 109 ++++++++++++++++++++++++++++++++++++++++++
 tests/generic/623.out |   4 ++
 tests/generic/group   |   1 +
 3 files changed, 114 insertions(+)
 create mode 100755 tests/generic/623
 create mode 100644 tests/generic/623.out
tests/generic/623 new file +109 -0
new file mode 100755
index 00000000..74136fef
@@ -0,0 +1,109 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+#
+# FSQA Test No. 623
+#
+# AIO/DIO stress test
+# Run random AIO/DIO activity on an file system with unwritten regions
+#
+# This test verifies that the an unwritten extent is properly marked
+# as written after writing into it.
+#
+# There was a hard-to-hit bug which would occasionally trigger with
+# ext4 for which this test was a reproducer.  This has been fixed
+# after moving ext4 to use iomap for Direct I/O's, although as of this
+# writing, there are still some occasional failures on ext4 when block
+# size < page size.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+fio_config=$tmp.fio
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs generic
+_require_test
+_require_scratch
+_require_odirect
+_require_block_device $SCRATCH_DEV
+
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+FILE_SIZE=$(((BLK_DEV_SIZE * 512) * 3 / 4))
+
+max_file_size=$((5 * 1024 * 1024 * 1024))
+if [ $max_file_size -lt $FILE_SIZE ]; then
+	FILE_SIZE=$max_file_size
+fi
+SIZE=$((FILE_SIZE / 2))
+
+cat >$fio_config <<EOF
+###########
+# $seq test fio activity
+# Filenames derived from jobsname and jobid like follows:
+# ${JOB_NAME}.${JOB_ID}.${ITERATION_ID}
+[global]
+ioengine=libaio
+bs=128k
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=${SIZE}
+iodepth=$((128*$LOAD_FACTOR))
+fallocate=native
+
+# Perform direct aio and verify data
+# This test case should check use-after-free issues
+[aio-dio-verifier]
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+direct=1
+blocksize_range=4100k-8200k
+blockalign=4k
+rw=randwrite
+filename=test-file
+
+EOF
+
+rm -f $seqres.full
+
+_require_fio $fio_config
+_require_xfs_io_command "falloc"
+
+_workout()
+{
+	echo ""
+	echo "Run fio with random aio-dio pattern"
+	echo ""
+	cat $fio_config >>  $seqres.full
+	run_check $FIO_PROG $fio_config
+}
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+if ! _workout; then
+	_scratch_unmount 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi
+status=0
+exit
tests/generic/623.out new file +4 -0
new file mode 100644
index 00000000..e10c7fd9
@@ -0,0 +1,4 @@
+QA output created by 623
+
+Run fio with random aio-dio pattern
+
tests/generic/group +1 -0
index b10fdea4..24f53ed7 100644
@@ -625,3 +625,4 @@
 620 auto mount quick
 621 auto quick encrypt
 622 auto shutdown metadata atime
+623 aio rw stress
Signature · 1 line
2.30.0
Reviewed-by on v1 1/1
#2 lore raw
In reply to Theodore Ts'o
Quoted 8 lines · Theodore Ts'o

This test verifies that the an unwritten extent is properly marked as written after writing into it.

There was a hard-to-hit bug which would occasionally trigger with ext4 for which this test was a reproducer. This has been fixed after moving ext4 to use iomap for Direct I/O's, although as of this writing, there are still some occasional failures on ext4 when block size < page size.

Are there still failures?

Quoted 103 lines
---
 tests/generic/623     | 109 ++++++++++++++++++++++++++++++++++++++++++
 tests/generic/623.out |   4 ++
 tests/generic/group   |   1 +
 3 files changed, 114 insertions(+)
 create mode 100755 tests/generic/623
 create mode 100644 tests/generic/623.out
tests/generic/623 new file +88 -0
new file mode 100755
index 00000000..74136fef
@@ -0,0 +1,109 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+#
+# FSQA Test No. 623
+#
+# AIO/DIO stress test
+# Run random AIO/DIO activity on an file system with unwritten regions
+#
+# This test verifies that the an unwritten extent is properly marked
+# as written after writing into it.
+#
+# There was a hard-to-hit bug which would occasionally trigger with
+# ext4 for which this test was a reproducer.  This has been fixed
+# after moving ext4 to use iomap for Direct I/O's, although as of this
+# writing, there are still some occasional failures on ext4 when block
+# size < page size.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+fio_config=$tmp.fio
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs generic
+_require_test
+_require_scratch
+_require_odirect
+_require_block_device $SCRATCH_DEV
+
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+FILE_SIZE=$(((BLK_DEV_SIZE * 512) * 3 / 4))
+
+max_file_size=$((5 * 1024 * 1024 * 1024))
+if [ $max_file_size -lt $FILE_SIZE ]; then
+	FILE_SIZE=$max_file_size
+fi
+SIZE=$((FILE_SIZE / 2))
+
+cat >$fio_config <<EOF
+###########
+# $seq test fio activity
+# Filenames derived from jobsname and jobid like follows:
+# ${JOB_NAME}.${JOB_ID}.${ITERATION_ID}
+[global]
+ioengine=libaio
+bs=128k
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=${SIZE}
+iodepth=$((128*$LOAD_FACTOR))
+fallocate=native
+
+# Perform direct aio and verify data
+# This test case should check use-after-free issues
+[aio-dio-verifier]
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+direct=1
+blocksize_range=4100k-8200k
+blockalign=4k
+rw=randwrite
+filename=test-file
+
+EOF
+
+rm -f $seqres.full
+
+_require_fio $fio_config
+_require_xfs_io_command "falloc"
+
+_workout()
+{
+	echo ""

tests/generic/623 line 98

Quoted 10 lines
tests/generic/623 new file +10 -0
+	echo "Run fio with random aio-dio pattern"
+	echo ""
+	cat $fio_config >>  $seqres.full
+	run_check $FIO_PROG $fio_config
+}
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+if ! _workout; then

If _workout fails, will something be printed to the golden output file so that the failure is obvious?

If so,

--D

Quoted 33 lines
tests/generic/623 new file +11 -0
+	_scratch_unmount 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi
+status=0
+exit
tests/generic/623.out new file +4 -0
new file mode 100644
index 00000000..e10c7fd9
@@ -0,0 +1,4 @@
+QA output created by 623
+
+Run fio with random aio-dio pattern
+
tests/generic/group +1 -0
index b10fdea4..24f53ed7 100644
@@ -625,3 +625,4 @@
 620 auto mount quick
 621 auto quick encrypt
 622 auto shutdown metadata atime
+623 aio rw stress
Signature · 1 line
2.30.0
#3 lore raw
Quoted 11 lines · Darrick J. Wong
Quoted 8 lines · Theodore Ts'o

This test verifies that the an unwritten extent is properly marked as written after writing into it.

There was a hard-to-hit bug which would occasionally trigger with ext4 for which this test was a reproducer. This has been fixed after moving ext4 to use iomap for Direct I/O's, although as of this writing, there are still some occasional failures on ext4 when block size < page size.

Are there still failures?

Yes. It's on Ritesh's todo list, but I don't believe it's bubbled up to the top of his list.

The failure rate is substantially less than what I had been seeing with 4k block sizes before the move to iomap, and in practice it was hard to hit in production. We noticed on Google data center workloads (but even then it was impossible to reliably repro before I found this fio recipe); as far as I know no one else had reported seeing this problem.

So in summary, as of this writing, on 1k and 2k block sizes, this test is still occasionally failing for ext4 (but not xfs, where it reliably passes). On 4k block sizes on x86 with ext4, this test is still failing in on 5.4 and earler kernels, unless someone has managed to backport DIO iomap support for ext4 into an enterprise linux kernel.

- Ted

A message that is not in the archive (sent off-list or not yet received): CABnRu57hdKav3Mi8vQYeowZrQtFToMSzK23h4H2DuqGL0Dea2A@mail.gmail.com

#5 lore raw
Quoted 31 lines · Su Yue
Quoted 24 lines · Theodore Ts'o
Quoted 11 lines · Darrick J. Wong
Quoted 8 lines · Theodore Ts'o

This test verifies that the an unwritten extent is properly marked as written after writing into it.

There was a hard-to-hit bug which would occasionally trigger with ext4 for which this test was a reproducer. This has been fixed after moving ext4 to use iomap for Direct I/O's, although as of this writing, there are still some occasional failures on ext4 when block size < page size.

Are there still failures?

Yes. It's on Ritesh's todo list, but I don't believe it's bubbled up to the top of his list.

The failure rate is substantially less than what I had been seeing with 4k block sizes before the move to iomap, and in practice it was hard to hit in production. We noticed on Google data center workloads (but even then it was impossible to reliably repro before I found this fio recipe); as far as I know no one else had reported seeing this problem.

Quoted 6 lines

So in summary, as of this writing, on 1k and 2k block sizes, this test is still occasionally failing for ext4 (but not xfs, where it reliably passes). On 4k block sizes on x86 with ext4, this test is still failing in on 5.4 and earler kernels, unless someone has managed to backport DIO iomap support for ext4 into an enterprise linux kernel.

Quoted 3 lines

But both xfs and ext4 100% fail in v5.12-rc1 with 4k block size and 4k page size on x86_64. The test itself should be okay because btrfs passes. xfstests-dev version is 8cbc48b460b6 applied this patch. logs are attached.

Is this generic/623? That test is new, and I've noticed it'll fail if you have coredumps enabled because the test tries to induce a SIGBUS.

--D

Quoted 13 lines
# fio --version
fio-3.25

#cat local.config
export TEST_DEV=/dev/mapper/test-1
export TEST_DIR=/mnt/test
export SCRATCH_DEV_POOL="/dev/mapper/test-[2-6]"
export SCRATCH_MNT=/mnt/scratch
export LOGWRITES_DEV="/dev/mapper/test-7"
export KEEP_DMESG=yes

Quoted 1 line

- Ted

#6 lore raw
Quoted 33 lines · Darrick J. Wong
Quoted 31 lines · Su Yue
Quoted 30 lines · Theodore Ts'o
Quoted 11 lines · Darrick J. Wong
Quoted 8 lines · Theodore Ts'o

This test verifies that the an unwritten extent is properly marked as written after writing into it.

There was a hard-to-hit bug which would occasionally trigger with ext4 for which this test was a reproducer. This has been fixed after moving ext4 to use iomap for Direct I/O's, although as of this writing, there are still some occasional failures on ext4 when block size < page size.

Are there still failures?

Yes. It's on Ritesh's todo list, but I don't believe it's bubbled up to the top of his list.

The failure rate is substantially less than what I had been seeing with 4k block sizes before the move to iomap, and in practice it was hard to hit in production. We noticed on Google data center workloads (but even then it was impossible to reliably repro before I found this fio recipe); as far as I know no one else had reported seeing this problem.

So in summary, as of this writing, on 1k and 2k block sizes, this test is still occasionally failing for ext4 (but not xfs, where it reliably passes). On 4k block sizes on x86 with ext4, this test is still failing in on 5.4 and earler kernels, unless someone has managed to backport DIO iomap support for ext4 into an enterprise linux kernel.

Quoted 7 lines
Quoted 3 lines

But both xfs and ext4 100% fail in v5.12-rc1 with 4k block size and 4k page size on x86_64. The test itself should be okay because btrfs passes. xfstests-dev version is 8cbc48b460b6 applied this patch. logs are attached.

Is this generic/623? That test is new, and I've noticed it'll fail if you have coredumps enabled because the test tries to induce a SIGBUS.

Hmm.. I think the "generic/623" you mean is introduced by 011bfb01f7f9 ("generic: test
mapped write after shutdown and failed writeback"). The generic/623 which I mean is the commit 8cbc48b460b6(011bfb01f7f9~1) applied Theodore's patch "generic: test which tries to exercise AIO/DIO into unwritten space".

--
Su

Quoted 19 lines

--D

Quoted 13 lines
# fio --version
fio-3.25

#cat local.config
export TEST_DEV=/dev/mapper/test-1
export TEST_DIR=/mnt/test
export SCRATCH_DEV_POOL="/dev/mapper/test-[2-6]"
export SCRATCH_MNT=/mnt/scratch
export LOGWRITES_DEV="/dev/mapper/test-7"
export KEEP_DMESG=yes

Quoted 1 line

- Ted

#7 lore raw
In reply to a message not in the archive
Quoted 3 lines · Su Yue

But both xfs and ext4 100% fail in v5.12-rc1 with 4k block size and 4k page size on x86_64. The test itself should be okay because btrfs passes. xfstests-dev version is 8cbc48b460b6 applied this patch. logs are attached.

Hmm, interesting. Thanks for the report. I've been running most of my tests on a GCE VVM (using gce-xfstests) with a 100GB PD-SSD as the storage device. I just tried, and it's passing now reliably on both 4k and 1k for me, both on v5.11 and v5.12-rc1. (Previously it was failing about 30% of the time on the ext4/1k test scenario, and was passing 100% on ext4/4k.)

Given your report, I tried running it using KVM (with a NVME ssd as the back-end storage device), and it's failing 80% of the time using the 4k block size.

So previously I had assumed that it was passing consistently on with ext4/4k, but it looks like that it's very much dependent on the performance of the TEST_DEV.

- Ted

#8 lore raw
In reply to Theodore Ts'o
Quoted 10 lines · Theodore Ts'o

This test verifies that the an unwritten extent is properly marked as written after writing into it.

There was a hard-to-hit bug which would occasionally trigger with ext4 for which this test was a reproducer. This has been fixed after moving ext4 to use iomap for Direct I/O's, although as of this writing, there are still some occasional failures on ext4 when block size < page size.

Sorry for the late review, it was in my to-review list and I just got to it..

Quoted 31 lines
---
 tests/generic/623     | 109 ++++++++++++++++++++++++++++++++++++++++++
 tests/generic/623.out |   4 ++
 tests/generic/group   |   1 +
 3 files changed, 114 insertions(+)
 create mode 100755 tests/generic/623
 create mode 100644 tests/generic/623.out
tests/generic/623 new file +17 -0
new file mode 100755
index 00000000..74136fef
@@ -0,0 +1,109 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+#
+# FSQA Test No. 623
+#
+# AIO/DIO stress test
+# Run random AIO/DIO activity on an file system with unwritten regions
+#
+# This test verifies that the an unwritten extent is properly marked
+# as written after writing into it.
+#
+# There was a hard-to-hit bug which would occasionally trigger with
+# ext4 for which this test was a reproducer.  This has been fixed
+# after moving ext4 to use iomap for Direct I/O's, although as of this
+# writing, there are still some occasional failures on ext4 when block
+# size < page size.

tests/generic/623 line 27

Quoted 10 lines
tests/generic/623 new file +10 -0
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+fio_config=$tmp.fio
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15

Better to trap a _cleanup function, even we only do "rm -f $tmp.*" in it, so it's consistent to other tests, and it's easier to add more cleanups in _cleanup() function in the future if needed.

tests/generic/623 line 37

Quoted 10 lines
tests/generic/623 new file +10 -0
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs generic
+_require_test
+_require_scratch
+_require_odirect

_require_aio

Quoted 39 lines
tests/generic/623 new file +39 -0
+_require_block_device $SCRATCH_DEV
+
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+FILE_SIZE=$(((BLK_DEV_SIZE * 512) * 3 / 4))
+
+max_file_size=$((5 * 1024 * 1024 * 1024))
+if [ $max_file_size -lt $FILE_SIZE ]; then
+	FILE_SIZE=$max_file_size
+fi
+SIZE=$((FILE_SIZE / 2))
+
+cat >$fio_config <<EOF
+###########
+# $seq test fio activity
+# Filenames derived from jobsname and jobid like follows:
+# ${JOB_NAME}.${JOB_ID}.${ITERATION_ID}
+[global]
+ioengine=libaio
+bs=128k
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=${SIZE}
+iodepth=$((128*$LOAD_FACTOR))
+fallocate=native
+
+# Perform direct aio and verify data
+# This test case should check use-after-free issues
+[aio-dio-verifier]
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+direct=1
+blocksize_range=4100k-8200k
+blockalign=4k
+rw=randwrite

tests/generic/623 line 86

Quoted 10 lines
tests/generic/623 new file +10 -0
+filename=test-file
+
+EOF
+
+rm -f $seqres.full
+
+_require_fio $fio_config
+_require_xfs_io_command "falloc"
+
+_workout()

There's no need to add the leading "_" to local function, it's reserved to common functions.

tests/generic/623 line 92

Quoted 6 lines
tests/generic/623 new file +6 -0
+{
+	echo ""
+	echo "Run fio with random aio-dio pattern"
+	echo ""
+	cat $fio_config >>  $seqres.full
+	run_check $FIO_PROG $fio_config

run_check is not recommanded and should be deprecated (maybe I should send a patch to document it in comment), as it hides failure in $seqres.full and exits if command returns non-zero.

Just call $FIO_PROG command directly and check return value if necessary.

Quoted 5 lines
tests/generic/623 new file +5 -0
+}
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+

tests/generic/623 line 107

Quoted 10 lines
tests/generic/623 new file +10 -0
+if ! _workout; then
+	_scratch_unmount 2>/dev/null
+	exit
+fi
+
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi

Is above _scratch_unmount check really needed? The test harness would umount $SCRATCH_DEV after test anyway.

Thanks,
Eryu

Quoted 23 lines
tests/generic/623 new file +2 -0
+status=0
+exit
tests/generic/623.out new file +4 -0
new file mode 100644
index 00000000..e10c7fd9
@@ -0,0 +1,4 @@
+QA output created by 623
+
+Run fio with random aio-dio pattern
+
tests/generic/group +1 -0
index b10fdea4..24f53ed7 100644
@@ -625,3 +625,4 @@
 620 auto mount quick
 621 auto quick encrypt
 622 auto shutdown metadata atime
+623 aio rw stress
Signature · 1 line
2.30.0
#9 lore raw
Quoted 5 lines · Eryu Guan
Quoted 1 line

+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15

Better to trap a _cleanup function, even we only do "rm -f $tmp.*" in it, so it's consistent to other tests, and it's easier to add more cleanups in _cleanup() function in the future if needed.

Done. I had based this test on generic/299 and generic/300, and a lot of your comments are applicable to them as well. I can send a cleanup patch to fix up those patches as well..

Quoted 3 lines
Quoted 1 line

+_require_odirect

_require_aio

Hmmm, generic/299 and generic/300 are missing _require_aio, while doing async I/O. I'm a bit surprised this hasn't caused problems for other file systems.

Quoted 5 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+_workout()

There's no need to add the leading "_" to local function, it's reserved to common functions.

Done. (Actually, if we're not unmounting $SCRATCH, then we really don't need a workout local function at all.)

Quoted 8 lines

tests/generic/623 line 92

Quoted 1 line
tests/generic/623 new file +1 -0
+	run_check $FIO_PROG $fio_config

run_check is not recommanded and should be deprecated (maybe I should send a patch to document it in comment), as it hides failure in $seqres.full and exits if command returns non-zero.

Just call $FIO_PROG command directly and check return value if necessary.

Thanks for suggesting dropping the run_check. I found a problem in the fio receipe which was causing a FIO warning that I had been missing.

BTW, all but one of the generic are still using run_check, and in the one exception, generic/095, which uses --output=$seqres.full which is causing us to lose all of the output of the earlier commands which redirected their outputs to $seqres.full. So there's clearly a "target rich environment" in terms of test clean up opportunities.

Quoted 8 lines

tests/generic/623 line 107

Quoted 5 lines
tests/generic/623 new file +5 -0
+if ! _scratch_unmount; then
+	echo "failed to umount"
+	status=1
+	exit
+fi

Is above _scratch_unmount check really needed? The test harness would umount $SCRATCH_DEV after test anyway.

Done.

Thanks for the review,

- Ted

#10 lore raw

PATCH 1/1 generic: test which tries to exercise AIO/DIO into unwritten space

This test verifies that the an unwritten extent is properly marked as written after writing into it.

There was a hard-to-hit bug which would occasionally trigger with ext4 for which this test was a reproducer. This has been fixed after moving ext4 to use iomap for Direct I/O's, although as of this writing, there are still some occasional failures on ext4 when block size < page size.

---

Differences between v1 and v2:
* Addressed review comments from Eryu
* Renumbered from generic/623 to generic/630 (since 623 has been assigned)

 tests/generic/630     | 105 ++++++++++++++++++++++++++++++++++++++++++
 tests/generic/630.out |   4 ++
 tests/generic/group   |   1 +
 3 files changed, 110 insertions(+)
 create mode 100755 tests/generic/630
 create mode 100644 tests/generic/630.out
tests/generic/630 new file +105 -0
new file mode 100755
index 00000000..7a04b899
@@ -0,0 +1,105 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+#
+# FSQA Test No. 630
+#
+# AIO/DIO stress test
+# Run random AIO/DIO activity on an file system with unwritten regions
+#
+# This test verifies that the an unwritten extent is properly marked
+# as written after writing into it.
+#
+# There was a hard-to-hit bug which would occasionally trigger with
+# ext4 for which this test was a reproducer.  This has been fixed
+# after moving ext4 to use iomap for Direct I/O's, although as of this
+# writing, there are still some occasional failures on ext4 when block
+# size < page size.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+fio_config=$tmp.fio
+fio_out=$tmp.fio.out
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+    rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs generic
+_require_test
+_require_scratch
+_require_odirect
+_require_aio
+_require_block_device $SCRATCH_DEV
+
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+FILE_SIZE=$(((BLK_DEV_SIZE * 512) * 3 / 4))
+
+max_file_size=$((5 * 1024 * 1024 * 1024))
+if [ $max_file_size -lt $FILE_SIZE ]; then
+	FILE_SIZE=$max_file_size
+fi
+SIZE=$((FILE_SIZE / 2))
+
+cat >$fio_config <<EOF
+###########
+# $seq test fio activity
+# Filenames derived from jobsname and jobid like follows:
+# ${JOB_NAME}.${JOB_ID}.${ITERATION_ID}
+[global]
+ioengine=libaio
+bs=128k
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=${FILE_SIZE}
+io_size=${SIZE}
+iodepth=$((128*$LOAD_FACTOR))
+fallocate=native
+
+# Perform direct aio and verify data
+# This test case should check use-after-free issues
+[aio-dio-verifier]
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+direct=1
+random_generator=lfsr
+blocksize_range=4k-8200k
+rw=randwrite
+filename=test-file
+
+EOF
+
+rm -f $seqres.full
+
+_require_fio $fio_config
+_require_xfs_io_command "falloc"
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+echo ""
+echo "Run fio with random aio-dio pattern"
+echo ""
+cat $fio_config >> $seqres.full
+$FIO_PROG $fio_config --output=$fio_out
+cat $fio_out >> $seqres.full
+
+status=0
+exit
tests/generic/630.out new file +4 -0
new file mode 100644
index 00000000..7f3adb50
@@ -0,0 +1,4 @@
+QA output created by 630
+
+Run fio with random aio-dio pattern
+
tests/generic/group +1 -0
index 84db3789..1ff896a1 100644
@@ -628,3 +628,4 @@
 623 auto quick shutdown
 624 auto quick verity
 625 auto quick verity
+630 aio rw stress
Signature · 1 line
2.30.0
#11 lore raw
Quoted 102 lines · Theodore Ts'o

This test verifies that the an unwritten extent is properly marked as written after writing into it.

There was a hard-to-hit bug which would occasionally trigger with ext4 for which this test was a reproducer. This has been fixed after moving ext4 to use iomap for Direct I/O's, although as of this writing, there are still some occasional failures on ext4 when block size < page size.

---

Differences between v1 and v2:
* Addressed review comments from Eryu
* Renumbered from generic/623 to generic/630 (since 623 has been assigned)

 tests/generic/630     | 105 ++++++++++++++++++++++++++++++++++++++++++
 tests/generic/630.out |   4 ++
 tests/generic/group   |   1 +
 3 files changed, 110 insertions(+)
 create mode 100755 tests/generic/630
 create mode 100644 tests/generic/630.out
tests/generic/630 new file +73 -0
new file mode 100755
index 00000000..7a04b899
@@ -0,0 +1,105 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+
+#
+# FSQA Test No. 630
+#
+# AIO/DIO stress test
+# Run random AIO/DIO activity on an file system with unwritten regions
+#
+# This test verifies that the an unwritten extent is properly marked
+# as written after writing into it.
+#
+# There was a hard-to-hit bug which would occasionally trigger with
+# ext4 for which this test was a reproducer.  This has been fixed
+# after moving ext4 to use iomap for Direct I/O's, although as of this
+# writing, there are still some occasional failures on ext4 when block
+# size < page size.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+fio_config=$tmp.fio
+fio_out=$tmp.fio.out
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+    rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_fs generic
+_require_test
+_require_scratch
+_require_odirect
+_require_aio
+_require_block_device $SCRATCH_DEV
+
+NUM_JOBS=$((4*LOAD_FACTOR))
+BLK_DEV_SIZE=`blockdev --getsz $SCRATCH_DEV`
+FILE_SIZE=$(((BLK_DEV_SIZE * 512) * 3 / 4))
+
+max_file_size=$((5 * 1024 * 1024 * 1024))
+if [ $max_file_size -lt $FILE_SIZE ]; then
+	FILE_SIZE=$max_file_size
+fi
+SIZE=$((FILE_SIZE / 2))
+
+cat >$fio_config <<EOF
+###########
+# $seq test fio activity
+# Filenames derived from jobsname and jobid like follows:
+# ${JOB_NAME}.${JOB_ID}.${ITERATION_ID}
+[global]
+ioengine=libaio
+bs=128k
+directory=${SCRATCH_MNT}
+filesize=${FILE_SIZE}
+size=${FILE_SIZE}
+io_size=${SIZE}
+iodepth=$((128*$LOAD_FACTOR))
+fallocate=native
+
+# Perform direct aio and verify data
+# This test case should check use-after-free issues

tests/generic/630 line 83

Quoted 10 lines
tests/generic/630 new file +10 -0
+[aio-dio-verifier]
+numjobs=1
+verify=crc32c-intel
+verify_fatal=1
+verify_dump=1
+verify_backlog=1024
+verify_async=4
+direct=1
+random_generator=lfsr
+blocksize_range=4k-8200k

Something I hadn't noticed before this got committed -- setting the maximum blocksize to 8200k and the iodepth to a multiple of 128 means that fio will allocate an 8200k buffer per iodepth. Given that the iodepth is (128 * LOAD_FACTOR), this test therefore requires (1GB * LOAD_FACTOR) memory. On my testing VMs which have a generous 1200M of memory, this causes the oom killer to zap fio, which in turn causes the test to fail due to the SIGKILL being reported in the golden output.

Is it necessary to allocate 8MB of memory per io thread? Given that this is a regression test to probe a bug in ext4's unwritten extent conversion I'm loath to touch it too much, but it seems to me that a smaller blocksize would stress the extent mapping structures harder and reduce the memory requirements at the same time.

--D

Quoted 44 lines
tests/generic/630 new file +22 -0
+rw=randwrite
+filename=test-file
+
+EOF
+
+rm -f $seqres.full
+
+_require_fio $fio_config
+_require_xfs_io_command "falloc"
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+echo ""
+echo "Run fio with random aio-dio pattern"
+echo ""
+cat $fio_config >> $seqres.full
+$FIO_PROG $fio_config --output=$fio_out
+cat $fio_out >> $seqres.full
+
+status=0
+exit
tests/generic/630.out new file +4 -0
new file mode 100644
index 00000000..7f3adb50
@@ -0,0 +1,4 @@
+QA output created by 630
+
+Run fio with random aio-dio pattern
+
tests/generic/group +1 -0
index 84db3789..1ff896a1 100644
@@ -628,3 +628,4 @@
 623 auto quick shutdown
 624 auto quick verity
 625 auto quick verity
+630 aio rw stress
Signature · 1 line
2.30.0