Add more tests for multi fs block atomic writes
v6 cover Add more tests for multi fs block atomic writes
Changes in v6:
- Picked up reviews from Darrick, Zorro and John! (Thanks)
- Added _require_fio_atomic_writes helper in patch 3 as a wrapper arounc __require_fio_version
- minor spelling and refactors
Changes in v5: (Thanks to John & Darrick for reviews)
- commor/rc: Add a _require_fio_version helper
- fsx: Switch atomic writes off if direct IO (-Z) not passed
- fio tests: better commit messages to explain what we are testing
- ext4/06{1..2}: Refactor code, also test only a few combinations of bs clustersize rather than every single
Changes in v4: (Thanks to Darrick, John and Zorro for the reviews) [4]
- g/1226,1227: Modify fio threads to not issue overlapping atomic writes
- g/1228: Use xfs_io -c "shutdown" instead of _scratch_shutdown to avoid bash overhead
- g/1229: Remove FSX_AVOID handling for bigalloc from common/rc. It is part of the specific test now
- ext4/063: add more clearer extent diagram
- ext4/064: Drop the test for now as im taking sometime to understand the behavior better.
- Removed test numbers from commit message
- For tests with significant changes I've removed the RVBs
[4] https://lore.kernel.org/fstests/[email protected]/T/#mef34a8c13cbee466bfc162db637d6e1cf0a8b06d
Changes in v3 [3]:
- (2/13) use dumpe2fs to figure out if FS is bigalloc
- (9/13) generic/1230: Detect device speeds for more accurate testing. ALso speeds up the test
- fio tests - switch to write followed by verify approach to avoid false failures due to fio verify reads splitting and racing with atomic writes. Discussion thread:
https://lore.kernel.org/fstests/[email protected]/
[3] https://lore.kernel.org/fstests/[email protected]/
Changes in v2 [1]:
- (1/13) new patch with _min and _max helpers
- (2/13) remove setup_fs_options and add fsx specific helper
- (4/13) skip atomic write instead of falling back to normal write (fsx)
- (4/13) make atomic write default on instead of default off (fsx)
- (5,6/13) refactor and cleanup fio tests
- (7/13) refactored common code
- (8/13) dont ignore mmap writes for fsx with atomic writes
- (9/13) use od instead of xxd. handle cleanup of bg threads in _cleanup()
- (10-13/13) minor refactors
- change all tests use _fail for better consistency
- use higher tests numbers for easier merging
[1] https://lore.kernel.org/fstests/[email protected]/
* Original cover [2] *
These are the tests we were using to verify that filesystems are not tearing multi fs block atomic writes. Infact some of the tests like generic/772 (now: g/1230) actually helped us catch and fix issues in ext4's early implementations of multi fs block atomic writes and hence we feel these tests are useful to have in xfstests.
We have tested these with scsi debug as well as a real nvme device supporting multi fs block atomic writes.
Thoughts and suggestions are welcome!
[2] rfc: https://lore.kernel.org/fstests/[email protected]/
Ojaswin Mujoo (10):
common/rc: Add _min() and _max() helpers
common/rc: Add fio atomic write helpers
common/rc: Add a helper to run fsx on a given file ltp/fsx.c: Add atomic writes support to fsx
generic: Add atomic write test using fio crc check verifier
generic: Add atomic write test using fio verify on file mixed mappings
generic: Add atomic write multi-fsblock O_[D]SYNC tests
generic: Stress fsx with atomic writes enabled
generic: Add sudden shutdown tests for multi block atomic writes
ext4: Atomic write test for extent split across leaf nodes
Ritesh Harjani (IBM) (2):
ext4: Test atomic write and ioend codepaths with bigalloc
ext4: Test atomic writes allocation and write codepaths with bigalloc
common/rc | 88 +++++++++- ltp/fsx.c | 115 ++++++++++++- tests/ext4/061 | 155 +++++++++++++++++ tests/ext4/061.out | 2 + tests/ext4/062 | 203 +++++++++++++++++++++++ tests/ext4/062.out | 2 + tests/ext4/063 | 129 +++++++++++++++ tests/ext4/063.out | 2 + tests/generic/1226 | 108 ++++++++++++ tests/generic/1226.out | 2 + tests/generic/1227 | 132 +++++++++++++++ tests/generic/1227.out | 2 + tests/generic/1228 | 139 ++++++++++++++++ tests/generic/1228.out | 2 + tests/generic/1229 | 68 ++++++++ tests/generic/1229.out | 2 + tests/generic/1230 | 368 +++++++++++++++++++++++++++++++++++++++++ tests/generic/1230.out | 2 + 18 files changed, 1513 insertions(+), 8 deletions(-) create mode 100755 tests/ext4/061 create mode 100644 tests/ext4/061.out create mode 100755 tests/ext4/062 create mode 100644 tests/ext4/062.out create mode 100755 tests/ext4/063 create mode 100644 tests/ext4/063.out create mode 100755 tests/generic/1226 create mode 100644 tests/generic/1226.out create mode 100755 tests/generic/1227 create mode 100644 tests/generic/1227.out create mode 100755 tests/generic/1228 create mode 100644 tests/generic/1228.out create mode 100755 tests/generic/1229 create mode 100644 tests/generic/1229.out create mode 100755 tests/generic/1230 create mode 100644 tests/generic/1230.out
Signature · 1 line
2.49.0
PATCH v6 1/12 common/rc: Add _min() and _max() helpers
Many programs open code these functionalities so add it as a generic helper in common/rc
Reviewed-by Darrick J. Wong <[email protected]>
Signed-off-by Ojaswin Mujoo <[email protected]>
--- common/rc | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
common/rc +22 -0
@@ -5978,6 +5978,28 @@ _require_inplace_writes() fi } +_min() { + local ret + + for arg in "$@"; do + if [ -z "$ret" ] || (( $arg < $ret )); then + ret="$arg" + fi + done + echo $ret +} + +_max() { + local ret + + for arg in "$@"; do + if [ -z "$ret" ] || (( $arg > $ret )); then + ret="$arg" + fi + done + echo $ret +} + ################################################################################ # make sure this script returns success /bin/true
Signature · 1 line
2.49.0
PATCH v6 2/12 common/rc: Add fio atomic write helpers
In reply to Ojaswin MujooThe main motivation of adding this function on top of _require_fio is that there has been a case in fio where atomic= option was added but later it was changed to noop since kernel didn't yet have support for atomic writes. It was then again utilized to do atomic writes in a later version, once kernel got the support. Due to this there is a point in fio where _require_fio w/ atomic=1 will succeed even though it would not be doing atomic writes.
Hence, add an internal helper __require_fio_version to require specific versions of fio to work past such issues. Further, add the high level _require_fio_atomic_writes helper which tests can use to ensure fio has the right version for atomic writes.
Reviewed-by Zorro Lang <[email protected]>
Signed-off-by Ojaswin Mujoo <[email protected]>
--- common/rc | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
common/rc +43 -0
@@ -6000,6 +6000,49 @@ _max() { echo $ret } +# Due to reasons explained in fio commit 40f1fc11d, fio version between +# v3.33 and v3.38 have atomic= feature but it is a no-op and doesn't do +# RWF_ATOMIC write. Hence, use this helper to ensure fio has the +# required support. Currently, the simplest way we have is to ensure +# the version. +_require_fio_atomic_writes() { + __require_fio_version "3.38+" +} + +# Check the required fio version. Examples: +# __require_fio_version 3.38 (matches 3.38 only) +# __require_fio_version 3.38+ (matches 3.38 and above) +# __require_fio_version 3.38- (matches 3.38 and below) +# +# Internal helper, avoid using directly in tests. +__require_fio_version() { + local req_ver="$1" + local fio_ver + + _require_fio + _require_math + + fio_ver=$(fio -v | cut -d"-" -f2) + + case "$req_ver" in + *+) + req_ver=${req_ver%+} + test $(_math "$fio_ver >= $req_ver") -eq 1 || \ + _notrun "need fio >= $req_ver (found $fio_ver)" + ;; + *-) + req_ver=${req_ver%-} + test $(_math "$fio_ver <= $req_ver") -eq 1 || \ + _notrun "need fio <= $req_ver (found $fio_ver)" + ;; + *) + req_ver=${req_ver%-} + test $(_math "$fio_ver == $req_ver") -eq 1 || \ + _notrun "need fio = $req_ver (found $fio_ver)" + ;; + esac +} + ################################################################################ # make sure this script returns success /bin/true
Signature · 1 line
2.49.0
PATCH v6 3/12 common/rc: Add a helper to run fsx on a given file
In reply to Ojaswin MujooCurrently run_fsx is hardcoded to run on a file in $TEST_DIR. Add a helper _run_fsx_on_file so that we can run fsx on any given file including in $SCRATCH_MNT. Also, refactor _run_fsx to use this helper.
No functional change is intended in this patch.
Reviewed-by Darrick J. Wong <[email protected]>
Signed-off-by Ojaswin Mujoo <[email protected]>
--- common/rc | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)
common/rc +20 -3
@@ -5203,13 +5203,24 @@ _require_hugepage_fsx() _notrun "fsx binary does not support MADV_COLLAPSE" } -_run_fsx() +_run_fsx_on_file() { + local testfile=$1 + shift + + if ! [ -f $testfile ] + then + echo "_run_fsx_on_file: $testfile doesn't exist. Creating" >> $seqres.full + touch $testfile + fi + echo "fsx $*" local args=`echo $@ | sed -e "s/ BSIZE / $bsize /g" -e "s/ PSIZE / $psize /g"` - set -- $FSX_PROG $args $FSX_AVOID $TEST_DIR/junk + + set -- $FSX_PROG $args $FSX_AVOID $testfile + echo "$@" >>$seqres.full - rm -f $TEST_DIR/junk + rm -f $testfile "$@" 2>&1 | tee -a $seqres.full >$tmp.fsx local res=${PIPESTATUS[0]} if [ $res -ne 0 ]; then
@@ -5221,6 +5232,12 @@ _run_fsx() return 0 } +_run_fsx() +{ + _run_fsx_on_file $TEST_DIR/junk $@ + return $? +} + # Run fsx with -h(ugepage buffers). If we can't set up a hugepage then skip # the test, but if any other error occurs then exit the test. _run_hugepage_fsx() {
Signature · 1 line
2.49.0
PATCH v6 4/12 ltp/fsx.c: Add atomic writes support to fsx
In reply to Ojaswin MujooImplement atomic write support to help fuzz atomic writes with fsx.
Suggested-by Ritesh Harjani (IBM) <[email protected]>
Reviewed-by Darrick J. Wong <[email protected]>
Reviewed-by John Garry <[email protected]>
Signed-off-by Ojaswin Mujoo <[email protected]>
--- ltp/fsx.c | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 110 insertions(+), 5 deletions(-)
ltp/fsx.c +110 -5
@@ -40,6 +40,7 @@#include <liburing.h> #endif #include <sys/syscall.h> +#include "statx.h" #ifndef MAP_FILE # define MAP_FILE 0
@@ -49,6 +50,10 @@#define RWF_DONTCACHE 0x80#endif +#ifndef RWF_ATOMIC +#define RWF_ATOMIC 0x40 +#endif +#define NUMPRINTCOLUMNS 32/* # columns of data to print on each line */ /* Operation flags (bitmask) */
@@ -110,6 +115,7 @@ enum {OP_READ_DONTCACHE ,OP_WRITE ,OP_WRITE_DONTCACHE , +OP_WRITE_ATOMIC ,OP_MAPREAD ,OP_MAPWRITE , OP_MAX_LITE,
@@ -200,6 +206,11 @@ int uring = 0;int mark_nr =0 ;int dontcache_io =1 ;int hugepages =0 ;/* -h flag */ +int do_atomic_writes =1 ;/* -a flag disables */ + +/* User for atomic writes */ +int awu_min =0 ; +int awu_max =0 ;/* Stores info needed to periodically collapse hugepages */ struct hugepages_collapse_info {
@@ -288,6 +299,7 @@ static const char *op_names[] = { [OP_READ_DONTCACHE] = "read_dontcache", [OP_WRITE] = "write", [OP_WRITE_DONTCACHE] = "write_dontcache", + [OP_WRITE_ATOMIC] = "write_atomic", [OP_MAPREAD] = "mapread", [OP_MAPWRITE] = "mapwrite", [OP_TRUNCATE] = "truncate",
@@ -422,6 +434,7 @@ logdump(void)prt ("\t***RRRR***" );break ;case OP_WRITE_DONTCACHE : +case OP_WRITE_ATOMIC :case OP_WRITE : prt("WRITE 0x%x thru 0x%x\t(0x%x bytes)", lp->args[0], lp->args[0] + lp->args[1] - 1,
@@ -1073,6 +1086,25 @@ update_file_size(unsigned offset, unsigned size) file_size = offset + size; } +static int is_power_of_2 (unsigned n) { +return ((n & (n -1 )) ==0 ); +} + +/* +* Round down n to nearest power of 2. +* If n is already a power of 2, return n; +*/ +static int rounddown_pow_of_2 (int n) { +int i =0 ; + +if (is_power_of_2 (n)) +return n; + +for (; (1 << i) < n; i++); + +return 1 << (i -1 ); +} +void dowrite (unsigned offset,unsigned size,int flags) {
@@ -1081,6 +1113,27 @@ dowrite(unsigned offset, unsigned size, int flags) offset -= offset % writebdy;if (o_direct) size -= size % writebdy; +if (flags &RWF_ATOMIC ) { +/* atomic write len must be between awu_min and awu_max */ +if (size < awu_min) + size = awu_min; +if (size > awu_max) + size = awu_max; + +/* atomic writes need power-of-2 sizes */ + size =rounddown_pow_of_2 (size); + +/* atomic writes need naturally aligned offsets */ + offset -= offset % size; + +/* Skip the write if we are crossing max filesize */ +if ((offset + size) > maxfilelen) { +if (!quiet && testcalls > simulatedopcount) +prt ("skipping atomic write past maxfilelen\n" ); +log4 (OP_WRITE_ATOMIC , offset, size,FL_SKIPPED ); +return ; + } + }if (size ==0 ) {if (!quiet && testcalls > simulatedopcount && !o_direct)prt ("skipping zero size write\n" );
@@ -1088,7 +1141,10 @@ dowrite(unsigned offset, unsigned size, int flags)return ; } -log4 (OP_WRITE , offset, size,FL_NONE ); +if (flags &RWF_ATOMIC ) +log4 (OP_WRITE_ATOMIC , offset, size,FL_NONE ); +else +log4 (OP_WRITE , offset, size,FL_NONE );gendata (original_buf, good_buf, offset, size);if (offset + size > file_size) {
@@ -1108,8 +1164,9 @@ dowrite(unsigned offset, unsigned size, int flags) (monitorstart == -1 || (offset + size > monitorstart && (monitorend == -1 || offset <= monitorend)))))) -prt ("%lld write\t0x%x thru\t0x%x\t(0x%x bytes)\tdontcache=%d\n" , testcalls, - offset, offset + size -1 , size, (flags &RWF_DONTCACHE ) !=0 ); +prt ("%lld write\t0x%x thru\t0x%x\t(0x%x bytes)\tdontcache=%d atomic_wr=%d\n" , testcalls, + offset, offset + size -1 , size, (flags &RWF_DONTCACHE ) !=0 , + (flags &RWF_ATOMIC ) !=0 ); iret =fsxwrite (fd, good_buf + offset, size, offset, flags);if (iret != size) {if (iret ==-1 )
@@ -1785,6 +1842,36 @@ do_dedupe_range(unsigned offset, unsigned length, unsigned dest) }#endif +int test_atomic_writes (void ) { +int ret; +struct statx stx; + +if (o_direct !=O_DIRECT ) { +fprintf (stderr,"main: atomic writes need O_DIRECT (-Z), " +"disabling!\n" ); +return 0 ; + } + + ret =xfstests_statx (AT_FDCWD , fname,0 ,STATX_WRITE_ATOMIC , &stx); +if (ret <0 ) { +fprintf (stderr,"main: Statx failed with %d." +" Failed to determine atomic write limits, " +" disabling!\n" , ret); +return 0 ; + } + +if (stx.stx_attributes &STATX_ATTR_WRITE_ATOMIC && + stx.stx_atomic_write_unit_min >0 ) { + awu_min = stx.stx_atomic_write_unit_min; + awu_max = stx.stx_atomic_write_unit_max; +return 1 ; + } + +fprintf (stderr,"main: IO Stack does not support " +"atomic writes, disabling!\n" ); +return 0 ; +} +#ifdef HAVE_COPY_FILE_RANGE int test_copy_range (void )
@@ -2356,6 +2443,12 @@ have_op: goto out; }break ; +case OP_WRITE_ATOMIC : +if (!do_atomic_writes) { +log4 (OP_WRITE_ATOMIC , offset, size,FL_SKIPPED ); + goto out; + } +break ; }switch (op) {
@@ -2385,6 +2478,11 @@ have_op:dowrite (offset, size,0 );break ; +case OP_WRITE_ATOMIC : +TRIM_OFF_LEN (offset, size, maxfilelen); +dowrite (offset, size,RWF_ATOMIC ); +break ; +case OP_MAPREAD :TRIM_OFF_LEN (offset, size, file_size);domapread (offset, size);
@@ -2511,13 +2609,14 @@ voidusage (void) { fprintf(stdout, "usage: %s", - "fsx [-dfhknqxyzBEFHIJKLORWXZ0]\n\ + "fsx [-adfhknqxyzBEFHIJKLORWXZ0]\n\ [-b opnum] [-c Prob] [-g filldata] [-i logdev] [-j logid]\n\ [-l flen] [-m start:end] [-o oplen] [-p progressinterval]\n\ [-r readbdy] [-s style] [-t truncbdy] [-w writebdy]\n\ [-A|-U] [-D startingop] [-N numops] [-P dirpath] [-S seed]\n\ [--replay-ops=opsfile] [--record-ops[=opsfile]] [--duration=seconds]\n\ ... fname\n\ + -a: disable atomic writes\n\ -b opnum: beginning operation number (default 1)\n\ -c P: 1 in P chance of file close+open at each op (default infinity)\n\ -d: debug output for all operations\n\
@@ -3059,9 +3158,13 @@ main(int argc, char **argv)setvbuf (stdout, (char *)0 , _IOLBF,0 );/* line buffered stdout */ while ((ch =getopt_long (argc, argv, -"0b:c:de:fg:hi:j:kl:m:no:p:qr:s:t:uw:xyABD:EFJKHzCILN:OP:RS:UWXZ" , +"0ab:c:de:fg:hi:j:kl:m:no:p:qr:s:t:uw:xyABD:EFJKHzCILN:OP:RS:UWXZ" , longopts,NULL )) !=EOF )switch (ch) { +case 'a' : +prt ("main(): Atomic writes disabled\n" ); + do_atomic_writes =0 ; +break ;case 'b' : simulatedopcount =getnum (optarg, &endp);if (!quiet)
@@ -3475,6 +3578,8 @@ main(int argc, char **argv) exchange_range_calls =test_exchange_range ();if (dontcache_io) dontcache_io =test_dontcache_io (); +if (do_atomic_writes) + do_atomic_writes =test_atomic_writes ();while (keep_running ())if (!test ())
Signature · 1 line
2.49.0
PATCH v6 5/12 generic: Add atomic write test using fio crc check verifier
In reply to Ojaswin MujooThis adds atomic write test using fio based on it's crc check verifier. fio adds a crc header for each data block, which is verified later to ensure there is no data corruption or torn write.
This test essentially does a lot of parallel RWF_ATOMIC IO on a preallocated file to stress the write and end-io unwritten conversion code paths. The idea is to increase code coverage to ensure RWF_ATOMIC hasn't introduced any issues.
Avoid doing overlapping parallel atomic writes because it might give unexpected results. Use offset_increment=, size= fio options to achieve this behavior.
Co-developed-by Ritesh Harjani (IBM) <[email protected]>
Signed-off-by Ritesh Harjani (IBM) <[email protected]>
Reviewed-by Darrick J. Wong <[email protected]>
Reviewed-by John Garry <[email protected]>
Signed-off-by Ojaswin Mujoo <[email protected]>
--- tests/generic/1226 | 108 +++++++++++++++++++++++++++++++++++++++++ tests/generic/1226.out | 2 + 2 files changed, 110 insertions(+) create mode 100755 tests/generic/1226 create mode 100644 tests/generic/1226.out
tests/generic/1226 new file +108 -0
@@ -0,0 +1,108 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 IBM Corporation. All Rights Reserved. +# +# FS QA Test 1226 +# +# Validate FS atomic write using fio crc check verifier. +# +. ./common/preamble +. ./common/atomicwrites + +_begin_fstest auto aio rw atomicwrites + +_require_scratch_write_atomic +_require_odirect +_require_aio +_require_fio_atomic_writes + +_scratch_mkfs >> $seqres.full 2>&1 +_scratch_mount +_require_xfs_io_command "falloc" + +touch "$SCRATCH_MNT/f1" +awu_min_write=$(_get_atomic_write_unit_min "$SCRATCH_MNT/f1") +awu_max_write=$(_get_atomic_write_unit_max "$SCRATCH_MNT/f1") + +blocksize=$(_max "$awu_min_write" "$((awu_max_write/2))") +threads=$(_min "$(($(nproc) * 2 * LOAD_FACTOR))" "100") +filesize=$((blocksize * threads * 100)) +depth=$threads +io_size=$((filesize / threads)) +io_inc=$io_size +testfile=$SCRATCH_MNT/test-file + +fio_config=$tmp.fio +fio_out=$tmp.fio.out + +fio_aw_config=$tmp.aw.fio +fio_verify_config=$tmp.verify.fio + +function create_fio_configs() +{ + create_fio_aw_config + create_fio_verify_config +} + +function create_fio_verify_config() +{ +cat >$fio_verify_config <<EOF + [verify-job] + direct=1 + ioengine=libaio + rw=read + bs=$blocksize + filename=$testfile + size=$filesize + iodepth=$depth + group_reporting=1 + + verify_only=1 + verify=crc32c + verify_fatal=1 + verify_state_save=0 + verify_write_sequence=0 +EOF +} + +function create_fio_aw_config() +{ +cat >$fio_aw_config <<EOF + [atomicwrite-job] + direct=1 + ioengine=libaio + rw=randwrite + bs=$blocksize + filename=$testfile + size=$io_inc + offset_increment=$io_inc + iodepth=$depth + numjobs=$threads + group_reporting=1 + atomic=1 + + verify_state_save=0 + verify=crc32c + do_verify=0 +EOF +} + +create_fio_configs +_require_fio $fio_aw_config + +cat $fio_aw_config >> $seqres.full +cat $fio_verify_config >> $seqres.full + +$XFS_IO_PROG -fc "falloc 0 $filesize" $testfile >> $seqres.full + +$FIO_PROG $fio_aw_config >> $seqres.full +ret1=$? +$FIO_PROG $fio_verify_config >> $seqres.full +ret2=$? + +[[ $ret1 -eq 0 && $ret2 -eq 0 ]] || _fail "fio with atomic write failed" + +# success, all done +echo Silence is golden +status=0 +exit
tests/generic/1226.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 1226 +Silence is golden
Signature · 1 line
2.49.0
PATCH v6 6/12 generic: Add atomic write test using fio verify on file mixed mappings
In reply to Ojaswin MujooThis test uses fio to first create a file with mixed mappings. Then it does atomic writes using aio dio with parallel jobs to the same file with mixed mappings. Finally, we perform a fio verify step to ensure there is no data corruption or torn write.
The aim is to stress the FS block allocation and extent handling logic to ensure it handles mixed mappings with RWF_ATOMIC correctly without tearing or losing data.
Avoid doing overlapping parallel atomic writes because it might give unexpected results. Use offset_increment=, size= fio options to achieve this behavior.
Co-developed-by Ritesh Harjani (IBM) <[email protected]>
Signed-off-by Ritesh Harjani (IBM) <[email protected]>
Reviewed-by Darrick J. Wong <[email protected]>
Reviewed-by John Garry <[email protected]>
Signed-off-by Ojaswin Mujoo <[email protected]>
--- tests/generic/1227 | 132 +++++++++++++++++++++++++++++++++++++++++ tests/generic/1227.out | 2 + 2 files changed, 134 insertions(+) create mode 100755 tests/generic/1227 create mode 100644 tests/generic/1227.out
tests/generic/1227 new file +132 -0
@@ -0,0 +1,132 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 IBM Corporation. All Rights Reserved. +# +# FS QA Test 1227 +# +# Validate FS atomic write using fio crc check verifier on mixed mappings +# of a file. +# +. ./common/preamble +. ./common/atomicwrites + +_begin_fstest auto aio rw atomicwrites + +_require_scratch_write_atomic_multi_fsblock +_require_odirect +_require_aio +_require_fio_atomic_writes +_require_xfs_io_command "truncate" + +_scratch_mkfs >> $seqres.full 2>&1 +_scratch_mount + +touch "$SCRATCH_MNT/f1" +awu_min_write=$(_get_atomic_write_unit_min "$SCRATCH_MNT/f1") +awu_max_write=$(_get_atomic_write_unit_max "$SCRATCH_MNT/f1") + +aw_bsize=$(_max "$awu_min_write" "$((awu_max_write/4))") +fsbsize=$(_get_block_size $SCRATCH_MNT) + +threads=$(_min "$(($(nproc) * 2 * LOAD_FACTOR))" "100") +filesize=$((aw_bsize * threads * 100)) +depth=$threads +aw_io_size=$((filesize / threads)) +aw_io_inc=$aw_io_size +testfile=$SCRATCH_MNT/test-file + +fio_prep_config=$tmp.prep.fio +fio_aw_config=$tmp.aw.fio +fio_verify_config=$tmp.verify.fio +fio_out=$tmp.fio.out + +cat >$fio_prep_config <<EOF +# prep file to have mixed mappings +[global] +ioengine=libaio +filename=$testfile +size=$filesize +bs=$fsbsize +direct=1 +iodepth=$depth +group_reporting=1 + +# Create written extents +[prep_written_blocks] +ioengine=libaio +rw=randwrite +io_size=$((filesize/3)) +random_generator=lfsr + +# Create unwritten extents +[prep_unwritten_blocks] +ioengine=falloc +rw=randwrite +io_size=$((filesize/3)) +random_generator=lfsr +EOF + +cat >$fio_aw_config <<EOF +# atomic write to mixed mappings of written/unwritten/holes +[atomic_write_job] +ioengine=libaio +rw=randwrite +direct=1 +atomic=1 +random_generator=lfsr +group_reporting=1 + +filename=$testfile +bs=$aw_bsize +size=$aw_io_size +offset_increment=$aw_io_inc +iodepth=$depth +numjobs=$threads + +verify_state_save=0 +verify=crc32c +do_verify=0 +EOF + +cat >$fio_verify_config <<EOF +# verify atomic writes done by previous job +[verify_job] +ioengine=libaio +rw=read +random_generator=lfsr +group_reporting=1 + +filename=$testfile +size=$filesize +bs=$aw_bsize +iodepth=$depth + +verify_state_save=0 +verify_only=1 +verify=crc32c +verify_fatal=1 +verify_write_sequence=0 +EOF + +_require_fio $fio_aw_config +_require_fio $fio_verify_config + +cat $fio_prep_config >> $seqres.full +cat $fio_aw_config >> $seqres.full +cat $fio_verify_config >> $seqres.full + +$XFS_IO_PROG -fc "truncate $filesize" $testfile >> $seqres.full + +#prepare file with mixed mappings +$FIO_PROG $fio_prep_config >> $seqres.full + +# do atomic writes without verifying +$FIO_PROG $fio_aw_config >> $seqres.full + +# verify data is not torn +$FIO_PROG $fio_verify_config >> $seqres.full + +# success, all done +echo Silence is golden +status=0 +exit
tests/generic/1227.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 1227 +Silence is golden
Signature · 1 line
2.49.0
PATCH v6 7/12 generic: Add atomic write multi-fsblock O_[D]SYNC tests
In reply to Ojaswin MujooThis adds various atomic write multi-fsblock stress tests with mixed mappings and O_SYNC, to ensure the data and metadata is atomically persisted even if there is a shutdown.
Suggested-by Ritesh Harjani (IBM) <[email protected]>
Reviewed-by Darrick J. Wong <[email protected]>
Reviewed-by John Garry <[email protected]>
Signed-off-by Ojaswin Mujoo <[email protected]>
--- tests/generic/1228 | 139 +++++++++++++++++++++++++++++++++++++++++ tests/generic/1228.out | 2 + 2 files changed, 141 insertions(+) create mode 100755 tests/generic/1228 create mode 100644 tests/generic/1228.out
tests/generic/1228 new file +139 -0
@@ -0,0 +1,139 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 IBM Corporation. All Rights Reserved. +# +# FS QA Test 1228 +# +# Atomic write multi-fsblock data integrity tests with mixed mappings +# and O_SYNC +# +. ./common/preamble +. ./common/atomicwrites +_begin_fstest auto quick rw atomicwrites + +_require_scratch_write_atomic_multi_fsblock +_require_atomic_write_test_commands +_require_scratch_shutdown +_require_xfs_io_command "truncate" + +_scratch_mkfs >> $seqres.full +_scratch_mount >> $seqres.full + +check_data_integrity() { + actual=$(_hexdump $testfile) + if [[ "$expected" != "$actual" ]] + then + echo "Integrity check failed" + echo "Integrity check failed" >> $seqres.full + echo "# Expected file contents:" >> $seqres.full + echo "$expected" >> $seqres.full + echo "# Actual file contents:" >> $seqres.full + echo "$actual" >> $seqres.full + + _fail "Data integrity check failed. The atomic write was torn." + fi +} + +prep_mixed_mapping() { + $XFS_IO_PROG -c "truncate 0" $testfile >> $seqres.full + local off=0 + local mapping="" + + local operations=("W" "H" "U") + local num_blocks=$((awu_max / blksz)) + for ((i=0; i<num_blocks; i++)); do + local index=$((RANDOM % ${#operations[@]})) + local map="${operations[$index]}" + local mapping="${mapping}${map}" + + case "$map" in + "W") + $XFS_IO_PROG -dc "pwrite -S 0x61 -b $blksz $off $blksz" $testfile > /dev/null + ;; + "H") + # No operation needed for hole + ;; + "U") + $XFS_IO_PROG -c "falloc $off $blksz" $testfile >> /dev/null + ;; + esac + off=$((off + blksz)) + done + + echo "+ + Mixed mapping prep done. Full mapping pattern: $mapping" >> $seqres.full + + sync $testfile +} + +verify_atomic_write() { + test $bytes_written -eq $awu_max || _fail "atomic write len=$awu_max assertion failed" + check_data_integrity +} + +mixed_mapping_test() { + prep_mixed_mapping + + echo -"+ + Performing O_DSYNC atomic write from 0 to $awu_max" >> $seqres.full + if [[ "$1" == "shutdown" ]] + then + bytes_written=$($XFS_IO_PROG -x -dc \ + "pwrite -DA -V1 -b $awu_max 0 $awu_max" \ + -c "shutdown" $testfile | grep wrote | \ + awk -F'[/ ]' '{print $2}') + _scratch_cycle_mount >>$seqres.full 2>&1 || _fail "remount failed" + else + bytes_written=$($XFS_IO_PROG -dc \ + "pwrite -DA -V1 -b $awu_max 0 $awu_max" $testfile | \ + grep wrote | awk -F'[/ ]' '{print $2}') + fi + + verify_atomic_write +} + +testfile=$SCRATCH_MNT/testfile +touch $testfile + +awu_max=$(_get_atomic_write_unit_max $testfile) +blksz=$(_get_block_size $SCRATCH_MNT) + +# Create an expected pattern to compare with +$XFS_IO_PROG -tc "pwrite -b $awu_max 0 $awu_max" $testfile >> $seqres.full +expected=$(_hexdump $testfile) +echo "# Expected file contents:" >> $seqres.full +echo "$expected" >> $seqres.full +echo >> $seqres.full + +echo "# Test 1: Do O_DSYNC atomic write on random mixed mapping:" >> $seqres.full +echo >> $seqres.full + +iterations=10 +for ((i=1; i<=$iterations; i++)); do + echo "=== Mixed Mapping Test Iteration $i ===" >> $seqres.full + + echo "+ Testing without shutdown..." >> $seqres.full + mixed_mapping_test + echo "Passed!" >> $seqres.full + + echo "+ Testing with sudden shutdown..." >> $seqres.full + mixed_mapping_test "shutdown" + echo "Passed!" >> $seqres.full + + echo "Iteration $i completed: OK" >> $seqres.full + echo >> $seqres.full +done +echo "# Test 1: Do O_SYNC atomic write on random mixed mapping ($iterations iterations): OK" >> $seqres.full + + +echo >> $seqres.full +echo "# Test 2: Do extending O_SYNC atomic writes: " >> $seqres.full +bytes_written=$($XFS_IO_PROG -x -dstc "pwrite -A -V1 -b $awu_max 0 $awu_max" \ + -c "shutdown" $testfile | grep wrote | awk -F'[/ ]' '{print $2}') +_scratch_cycle_mount >>$seqres.full 2>&1 || _fail "remount failed" +verify_atomic_write +echo "# Test 2: Do extending O_SYNC atomic writes: OK" >> $seqres.full + +# success, all done +echo "Silence is golden" +status=0 +exit +
tests/generic/1228.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 1228 +Silence is golden
Signature · 1 line
2.49.0
PATCH v6 8/12 generic: Stress fsx with atomic writes enabled
In reply to Ojaswin MujooStress file with atomic writes to ensure we exercise codepaths where we are mixing different FS operations with atomic writes
Suggested-by Ritesh Harjani (IBM) <[email protected]>
Reviewed-by Darrick J. Wong <[email protected]>
Reviewed-by John Garry <[email protected]>
Signed-off-by Ojaswin Mujoo <[email protected]>
--- tests/generic/1229 | 68 ++++++++++++++++++++++++++++++++++++++++++ tests/generic/1229.out | 2 ++ 2 files changed, 70 insertions(+) create mode 100755 tests/generic/1229 create mode 100644 tests/generic/1229.out
tests/generic/1229 new file +68 -0
@@ -0,0 +1,68 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 IBM Corporation. All Rights Reserved. +# +# FS QA Test 1229 +# +# fuzz fsx with atomic writes +# +. ./common/preamble +. ./common/atomicwrites +_begin_fstest rw auto quick atomicwrites + +_require_odirect +_require_scratch_write_atomic + +_scratch_mkfs >> $seqres.full 2>&1 +_scratch_mount >> $seqres.full 2>&1 + +testfile=$SCRATCH_MNT/testfile +touch $testfile + +awu_max=$(_get_atomic_write_unit_max $testfile) +blksz=$(_get_block_size $SCRATCH_MNT) +bsize=`$here/src/min_dio_alignment $SCRATCH_MNT $SCRATCH_DEV` + +set_fsx_avoid() { + local file=$1 + + case "$FSTYP" in + "ext4") + local dev=$(findmnt -n -o SOURCE --target $testfile) + + # fsx insert/collapse range support for ext4+bigalloc is + # currently broken, so disable it. Also disable in case we + # can't detect bigalloc to be on safer side. + if [ -z "$DUMPE2FS_PROG" ]; then + echo "dumpe2fs not found, disabling insert/collapse range" >> $seqres.full + FSX_AVOID+=" -I -C" + return + fi + + $DUMPE2FS_PROG -h $dev 2>&1 | grep -q bigalloc && { + echo "fsx insert/collapse range not supported with bigalloc. Disabling.." >> $seqres.full + FSX_AVOID+=" -I -C" + } + ;; + *) + ;; + esac +} + +# fsx usage: +# +# -N numops: total # operations to do +# -l flen: the upper bound on file size +# -o oplen: the upper bound on operation size (64k default) +# -Z: O_DIRECT () + +set_fsx_avoid +_run_fsx_on_file $testfile -N 10000 -o $awu_max -A -l 500000 -r $bsize -w $bsize -Z $FSX_AVOID >> $seqres.full +if [[ "$?" != "0" ]] +then + _fail "fsx returned error: $?" +fi + +echo "Silence is golden" +status=0 +exit
tests/generic/1229.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 1229 +Silence is golden
Signature · 1 line
2.49.0
PATCH v6 9/12 generic: Add sudden shutdown tests for multi block atomic writes
In reply to Ojaswin MujooThis test is intended to ensure that multi blocks atomic writes maintain atomic guarantees across sudden FS shutdowns.
The way we work is that we lay out a file with random mix of written, unwritten and hole extents. Then we start performing atomic writes sequentially on the file while we parallelly shutdown the FS. Then we note the last offset where the atomic write happened just before shut down and then make sure blocks around it either have completely old data or completely new data, ie the write was not torn during shutdown.
We repeat the same with completely written, completely unwritten and completely empty file to ensure these cases are not torn either. Finally, we have a similar test for append atomic writes
Suggested-by Ritesh Harjani (IBM) <[email protected]>
Reviewed-by Darrick J. Wong <[email protected]>
Signed-off-by Ojaswin Mujoo <[email protected]>
--- tests/generic/1230 | 368 +++++++++++++++++++++++++++++++++++++++++ tests/generic/1230.out | 2 + 2 files changed, 370 insertions(+) create mode 100755 tests/generic/1230 create mode 100644 tests/generic/1230.out
tests/generic/1230 new file +368 -0
@@ -0,0 +1,368 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 IBM Corporation. All Rights Reserved. +# +# FS QA Test No. 1230 +# +# Test multi block atomic writes with sudden FS shutdowns to ensure +# the FS is not tearing the write operation +. ./common/preamble +. ./common/atomicwrites +_begin_fstest auto atomicwrites + +_require_scratch_write_atomic_multi_fsblock +_require_atomic_write_test_commands +_require_scratch_shutdown +_require_xfs_io_command "truncate" + +_scratch_mkfs >> $seqres.full 2>&1 +_scratch_mount >> $seqres.full + +testfile=$SCRATCH_MNT/testfile +touch $testfile + +awu_max=$(_get_atomic_write_unit_max $testfile) +blksz=$(_get_block_size $SCRATCH_MNT) +echo "Awu max: $awu_max" >> $seqres.full + +num_blocks=$((awu_max / blksz)) +# keep initial value high for dry run. This will be +# tweaked in dry_run() based on device write speed. +filesize=$(( 10 * 1024 * 1024 * 1024 )) + +_cleanup() { + [ -n "$awloop_pid" ] && kill $awloop_pid &> /dev/null + wait +} + +atomic_write_loop() { + local off=0 + local size=$awu_max + for ((i=0; i<$((filesize / $size )); i++)); do + # Due to sudden shutdown this can produce errors so just + # redirect them to seqres.full + $XFS_IO_PROG -c "open -fsd $testfile" -c "pwrite -S 0x61 -DA -V1 -b $size $off $size" >> /dev/null 2>>$seqres.full + echo "Written to offset: $off" >> $tmp.aw + off=$((off + $size)) + done +} + +start_atomic_write_and_shutdown() { + atomic_write_loop & + awloop_pid=$! + + local i=0 + # Wait for atleast first write to be recorded or 10s + while [ ! -f "$tmp.aw" -a $i -le 50 ]; do i=$((i + 1)); sleep 0.2; done + + if [[ $i -gt 50 ]] + then + _fail "atomic write process took too long to start" + fi + + echo >> $seqres.full + echo "# Shutting down filesystem while write is running" >> $seqres.full + _scratch_shutdown + + kill $awloop_pid 2>/dev/null # the process might have finished already + wait $awloop_pid + unset $awloop_pid +} + +# This test has the following flow: +# 1. Start doing sequential atomic writes in background, upto $filesize +# 2. Sleep for 0.2s and shutdown the FS +# 3. kill the atomic write process +# 4. verify the writes were not torn +# +# We ideally want the shutdown to happen while an atomic write is ongoing +# but this gets tricky since faster devices can actually finish the whole +# atomic write loop before sleep 0.2s completes, resulting in the shutdown +# happening after the write loop which is not what we want. A simple solution +# to this is to increase $filesize so step 1 takes long enough but a big +# $filesize leads to create_mixed_mappings() taking very long, which is not +# ideal. +# +# Hence, use the dry_run function to figure out the rough device speed and set +# $filesize accordingly. +dry_run() { + echo >> $seqres.full + echo "# Estimating ideal filesize..." >> $seqres.full + + start_atomic_write_and_shutdown + + bytes_written=$(tail -n 1 $tmp.aw | cut -d" " -f4) + echo "# Bytes written in 0.2s: $bytes_written" >> $seqres.full + + filesize=$((bytes_written * 3)) + echo "# Setting \$filesize=$filesize" >> $seqres.full + + rm $tmp.aw + sleep 0.5 + + _scratch_cycle_mount + +} + +create_mixed_mappings() { + local file=$1 + local size_bytes=$2 + + echo "# Filling file $file with alternate mappings till size $size_bytes" >> $seqres.full + #Fill the file with alternate written and unwritten blocks + local off=0 + local operations=("W" "U") + + for ((i=0; i<$((size_bytes / blksz )); i++)); do + index=$(($i % ${#operations[@]})) + map="${operations[$index]}" + + case "$map" in + "W") + $XFS_IO_PROG -fc "pwrite -b $blksz $off $blksz" $file >> /dev/null + ;; + "U") + $XFS_IO_PROG -fc "falloc $off $blksz" $file >> /dev/null + ;; + esac + off=$((off + blksz)) + done + + sync $file +} + +populate_expected_data() { + # create a dummy file with expected old data for different cases + create_mixed_mappings $testfile.exp_old_mixed $awu_max + expected_data_old_mixed=$(od -An -t x1 -j 0 -N $awu_max $testfile.exp_old_mixed) + + $XFS_IO_PROG -fc "falloc 0 $awu_max" $testfile.exp_old_zeroes >> $seqres.full + expected_data_old_zeroes=$(od -An -t x1 -j 0 -N $awu_max $testfile.exp_old_zeroes) + + $XFS_IO_PROG -fc "pwrite -b $awu_max 0 $awu_max" $testfile.exp_old_mapped >> $seqres.full + expected_data_old_mapped=$(od -An -t x1 -j 0 -N $awu_max $testfile.exp_old_mapped) + + # create a dummy file with expected new data + $XFS_IO_PROG -fc "pwrite -S 0x61 -b $awu_max 0 $awu_max" $testfile.exp_new >> $seqres.full + expected_data_new=$(od -An -t x1 -j 0 -N $awu_max $testfile.exp_new) +} + +verify_data_blocks() { + local verify_start=$1 + local verify_end=$2 + local expected_data_old="$3" + local expected_data_new="$4" + + echo >> $seqres.full + echo "# Checking data integrity from $verify_start to $verify_end" >> $seqres.full + + # After an atomic write, for every chunk we ensure that the underlying + # data is either the old data or new data as writes shouldn't get torn. + local off=$verify_start + while [[ "$off" -lt "$verify_end" ]] + do + #actual_data=$(xxd -s $off -l $awu_max -p $testfile) + actual_data=$(od -An -t x1 -j $off -N $awu_max $testfile) + if [[ "$actual_data" != "$expected_data_new" ]] && [[ "$actual_data" != "$expected_data_old" ]] + then + echo "Checksum match failed at off: $off size: $awu_max" + echo "Expected contents: (Either of the 2 below):" + echo + echo "Expected old: " + echo "$expected_data_old" + echo + echo "Expected new: " + echo "$expected_data_new" + echo + echo "Actual contents: " + echo "$actual_data" + + _fail + fi + echo -n "Check at offset $off succeeded! " >> $seqres.full + if [[ "$actual_data" == "$expected_data_new" ]] + then + echo "matched new" >> $seqres.full + elif [[ "$actual_data" == "$expected_data_old" ]] + then + echo "matched old" >> $seqres.full + fi + off=$(( off + awu_max )) + done +} + +# test data integrity for file by shutting down in between atomic writes +test_data_integrity() { + echo >> $seqres.full + echo "# Writing atomically to file in background" >> $seqres.full + + start_atomic_write_and_shutdown + + last_offset=$(tail -n 1 $tmp.aw | cut -d" " -f4) + if [[ -z $last_offset ]] + then + last_offset=0 + fi + + echo >> $seqres.full + echo "# Last offset of atomic write: $last_offset" >> $seqres.full + + rm $tmp.aw + sleep 0.5 + + _scratch_cycle_mount + + # we want to verify all blocks around which the shutdown happened + verify_start=$(( last_offset - (awu_max * 5))) + if [[ $verify_start < 0 ]] + then + verify_start=0 + fi + + verify_end=$(( last_offset + (awu_max * 5))) + if [[ "$verify_end" -gt "$filesize" ]] + then + verify_end=$filesize + fi +} + +# test data integrity for file with written and unwritten mappings +test_data_integrity_mixed() { + $XFS_IO_PROG -fc "truncate 0" $testfile >> $seqres.full + + echo >> $seqres.full + echo "# Creating testfile with mixed mappings" >> $seqres.full + create_mixed_mappings $testfile $filesize + + test_data_integrity + + verify_data_blocks $verify_start $verify_end "$expected_data_old_mixed" "$expected_data_new" +} + +# test data integrity for file with completely written mappings +test_data_integrity_written() { + $XFS_IO_PROG -fc "truncate 0" $testfile >> $seqres.full + + echo >> $seqres.full + echo "# Creating testfile with fully written mapping" >> $seqres.full + $XFS_IO_PROG -c "pwrite -b $filesize 0 $filesize" $testfile >> $seqres.full + sync $testfile + + test_data_integrity + + verify_data_blocks $verify_start $verify_end "$expected_data_old_mapped" "$expected_data_new" +} + +# test data integrity for file with completely unwritten mappings +test_data_integrity_unwritten() { + $XFS_IO_PROG -fc "truncate 0" $testfile >> $seqres.full + + echo >> $seqres.full + echo "# Creating testfile with fully unwritten mappings" >> $seqres.full + $XFS_IO_PROG -c "falloc 0 $filesize" $testfile >> $seqres.full + sync $testfile + + test_data_integrity + + verify_data_blocks $verify_start $verify_end "$expected_data_old_zeroes" "$expected_data_new" +} + +# test data integrity for file with no mappings +test_data_integrity_hole() { + $XFS_IO_PROG -fc "truncate 0" $testfile >> $seqres.full + + echo >> $seqres.full + echo "# Creating testfile with no mappings" >> $seqres.full + $XFS_IO_PROG -c "truncate $filesize" $testfile >> $seqres.full + sync $testfile + + test_data_integrity + + verify_data_blocks $verify_start $verify_end "$expected_data_old_zeroes" "$expected_data_new" +} + +test_append_data_integrity() { + $XFS_IO_PROG -c "truncate 0" $testfile >> $seqres.full + + echo >> $seqres.full + echo "# Performing append atomic writes over file in background" >> $seqres.full + + start_atomic_write_and_shutdown + + local last_offset=$(tail -n 1 $tmp.aw | cut -d" " -f4) + if [[ -z $last_offset ]] + then + last_offset=0 + fi + + echo >> $seqres.full + echo "# Last offset of atomic write: $last_offset" >> $seqres.full + rm $tmp.aw + sleep 0.5 + + _scratch_cycle_mount + local filesize=$(_get_filesize $testfile) + echo >> $seqres.full + echo "# Filesize after shutdown: $filesize" >> $seqres.full + + # To confirm that the write went atomically, we check: + # 1. The last block should be a multiple of awu_max + # 2. The last block should be the completely new data + + if (( $filesize % $awu_max )) + then + echo "Filesize after shutdown ($filesize) not a multiple of atomic write unit ($awu_max)" + fi + + verify_start=$(( filesize - (awu_max * 5))) + if [[ $verify_start < 0 ]] + then + verify_start=0 + fi + + local verify_end=$filesize + + # Here the blocks should always match new data hence, for simplicity of + # code, just corrupt the $expected_data_old buffer so it never matches + local expected_data_old="POISON" + verify_data_blocks $verify_start $verify_end "$expected_data_old" "$expected_data_new" +} + +$XFS_IO_PROG -fc "truncate 0" $testfile >> $seqres.full + +dry_run + +echo >> $seqres.full +echo "# Populating expected data buffers" >> $seqres.full +populate_expected_data + +# Loop 20 times to shake out any races due to shutdown +for ((iter=0; iter<20; iter++)) +do + echo >> $seqres.full + echo "------ Iteration $iter ------" >> $seqres.full + + echo >> $seqres.full + echo "# Starting data integrity test for atomic writes over mixed mapping" >> $seqres.full + test_data_integrity_mixed + + echo >> $seqres.full + echo "# Starting data integrity test for atomic writes over fully written mapping" >> $seqres.full + test_data_integrity_written + + echo >> $seqres.full + echo "# Starting data integrity test for atomic writes over fully unwritten mapping" >> $seqres.full + test_data_integrity_unwritten + + echo >> $seqres.full + echo "# Starting data integrity test for atomic writes over holes" >> $seqres.full + test_data_integrity_hole + + echo >> $seqres.full + echo "# Starting shutdown data integrity test for append atomic writes" >> $seqres.full + test_append_data_integrity +done + +echo "Silence is golden" +status=0 +exit
tests/generic/1230.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 1230 +Silence is golden
Signature · 1 line
2.49.0
PATCH v6 10/12 ext4: Test atomic write and ioend codepaths with bigalloc
In reply to Ojaswin MujooFrom: "Ritesh Harjani (IBM)" <[email protected]>
This test does a lot of parallel RWF_ATOMIC IO on a preallocated file to stress the write and end-io unwritten conversion code paths. We brute force this for different blocksize and clustersizes and after each iteration we ensure the data was not torn or corrupted using fio crc verification.
Note that in this test we use overlapping atomic writes of same io size. Although serializing racing writes is not guaranteed for RWF_ATOMIC, NVMe and SCSI provide this guarantee as an inseparable feature to power-fail atomicity. Keeping the iosize as same also ensures that ext4 doesn't tear the write due to racing ioend unwritten conversion.
The value of this test is that we make sure the RWF_ATOMIC is handled correctly by ext4 as well as test that the block layer doesn't split or only generate multiple bios for an atomic write.
Signed-off-by Ritesh Harjani (IBM) <[email protected]>
Reviewed-by Darrick J. Wong <[email protected]>
Reviewed-by John Garry <[email protected]>
Signed-off-by Ojaswin Mujoo <[email protected]>
--- tests/ext4/061 | 155 +++++++++++++++++++++++++++++++++++++++++++++ tests/ext4/061.out | 2 + 2 files changed, 157 insertions(+) create mode 100755 tests/ext4/061 create mode 100644 tests/ext4/061.out
tests/ext4/061 new file +155 -0
@@ -0,0 +1,155 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 IBM Corporation. All Rights Reserved. +# +# FS QA Test 061 +# +# This test does a lot of parallel RWF_ATOMIC IO on a preallocated file to +# stress the write and end-io unwritten conversion code paths. We brute force +# this for all possible blocksize and clustersizes and after each iteration we +# ensure the data was not torn or corrupted using fio crc verification. +# +# Note that in this test we use overlapping atomic writes of same io size. +# Although serializing racing writes is not guaranteed for RWF_ATOMIC, NVMe and +# SCSI provide this guarantee as an inseparable feature to power-fail +# atomicity. Keeping the iosize as same also ensures that ext4 doesn't tear the +# write due to racing ioend unwritten conversion. +# +# The value of this test is that we make sure the RWF_ATOMIC is handled +# correctly by ext4 as well as test that the block layer doesn't split or only +# generate multiple bios for an atomic write. + +. ./common/preamble +. ./common/atomicwrites + +_begin_fstest auto rw stress atomicwrites + +_require_scratch_write_atomic +_require_fio_atomic_writes +_require_aiodio + +FIO_LOAD=$(($(nproc) * 2 * LOAD_FACTOR)) +SIZE=$((100*1024*1024)) + +# Calculate fsblocksize as per bdev atomic write units. +bdev_awu_min=$(_get_atomic_write_unit_min $SCRATCH_DEV) +bdev_awu_max=$(_get_atomic_write_unit_max $SCRATCH_DEV) +bs=$(_max 4096 "$bdev_awu_min") + +function create_fio_configs() +{ + local bsize=$1 + create_fio_aw_config $bsize + create_fio_verify_config $bsize +} + +function create_fio_verify_config() +{ + local bsize=$1 +cat >$fio_verify_config <<EOF + [aio-dio-aw-verify] + direct=1 + ioengine=libaio + rw=read + bs=$bsize + fallocate=native + filename=$SCRATCH_MNT/test-file + size=$SIZE + iodepth=$FIO_LOAD + numjobs=$FIO_LOAD + atomic=1 + group_reporting=1 + + verify_only=1 + verify_state_save=0 + verify=crc32c + verify_fatal=1 + verify_write_sequence=0 +EOF +} + +function create_fio_aw_config() +{ + local bsize=$1 +cat >$fio_aw_config <<EOF + [aio-dio-aw] + direct=1 + ioengine=libaio + rw=randwrite + bs=$bsize + fallocate=native + filename=$SCRATCH_MNT/test-file + size=$SIZE + iodepth=$FIO_LOAD + numjobs=$FIO_LOAD + group_reporting=1 + atomic=1 + + verify_state_save=0 + verify=crc32c + do_verify=0 + +EOF +} + +run_test_one() { + local bs=$1 + local cs=$2 + local iosize=$3 + + MKFS_OPTIONS="-O bigalloc -b $bs -C $cs" + _scratch_mkfs_ext4 >> $seqres.full 2>&1 || return + if _try_scratch_mount >> $seqres.full 2>&1; then + echo "== Testing: bs=$bs cs=$cs iosize=$iosize ==" >> $seqres.full + + touch $SCRATCH_MNT/f1 + create_fio_configs $iosize + + cat $fio_aw_config >> $seqres.full + echo >> $seqres.full + cat $fio_verify_config >> $seqres.full + + $FIO_PROG $fio_aw_config >> $seqres.full + ret1=$? + + $FIO_PROG $fio_verify_config >> $seqres.full + ret2=$? + + _scratch_unmount + + [[ $ret1 -eq 0 && $ret2 -eq 0 ]] || _fail "fio with atomic write failed" + fi +} + +run_test() { + local bs=$1 + + # cluster sizes above 16 x blocksize are experimental so avoid them + # Also, cap cluster size at 128kb to keep it reasonable for large + # blocks size + max_cs=$(_min $((16 * bs)) "$bdev_awu_max" $((128 * 1024))) + + # Fuzz for combinations of blocksize, clustersize and + # iosize that cover most of the cases + run_test_one $bs $bs $bs + run_test_one $bs $max_cs $bs + run_test_one $bs $max_cs $max_cs + run_test_one $bs $max_cs $(_max "$((max_cs/2))" $bs) +} + +# Let's create a sample fio config to check whether fio supports all options. +fio_aw_config=$tmp.aw.fio +fio_verify_config=$tmp.verify.fio +fio_out=$tmp.fio.out + +create_fio_configs $bs +_require_fio $fio_aw_config + +for ((bs=$bs; bs <= $(_get_page_size); bs = $bs << 1)); do + run_test $bs +done + +# success, all done +echo Silence is golden +status=0 +exit
tests/ext4/061.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 061 +Silence is golden
Signature · 1 line
2.49.0
PATCH v6 11/12 ext4: Test atomic writes allocation and write codepaths with bigalloc
In reply to Ojaswin MujooFrom: "Ritesh Harjani (IBM)" <[email protected]>
This test does a parallel RWF_ATOMIC IO on a multiple truncated files in a small FS. The idea is to stress ext4 allocator to ensure we are able to handle low space scenarios correctly with atomic writes. We brute force this for different blocksize and clustersizes and after each iteration we ensure the data was not torn or corrupted using fio crc verification.
Note that in this test we use overlapping atomic writes of same io size. Although serializing racing writes is not guaranteed for RWF_ATOMIC, NVMe and SCSI provide this guarantee as an inseparable feature to power-fail atomicity. Keeping the iosize as same also ensures that ext4 doesn't tear the write due to racing ioend unwritten conversion.
The value of this test is that we make sure the RWF_ATOMIC is handled correctly by ext4 as well as test that the block layer doesn't split or only generate multiple bios for an atomic write.
Signed-off-by Ritesh Harjani (IBM) <[email protected]>
Reviewed-by Darrick J. Wong <[email protected]>
Reviewed-by John Garry <[email protected]>
Signed-off-by Ojaswin Mujoo <[email protected]>
--- tests/ext4/062 | 203 +++++++++++++++++++++++++++++++++++++++++++++ tests/ext4/062.out | 2 + 2 files changed, 205 insertions(+) create mode 100755 tests/ext4/062 create mode 100644 tests/ext4/062.out
tests/ext4/062 new file +203 -0
@@ -0,0 +1,203 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 IBM Corporation. All Rights Reserved. +# +# FS QA Test 062 +# +# This test does a parallel RWF_ATOMIC IO on a multiple truncated files in a +# small FS. The idea is to stress ext4 allocator to ensure we are able to +# handle low space scenarios correctly with atomic writes.. We brute force this +# for all possible blocksize and clustersizes and after each iteration we +# ensure the data was not torn or corrupted using fio crc verification. +# +# Note that in this test we use overlapping atomic writes of same io size. +# Although serializing racing writes is not guaranteed for RWF_ATOMIC, NVMe and +# SCSI provide this guarantee as an inseparable feature to power-fail +# atomicity. Keeping the iosize as same also ensures that ext4 doesn't tear the +# write due to racing ioend unwritten conversion. +# +# The value of this test is that we make sure the RWF_ATOMIC is handled +# correctly by ext4 as well as test that the block layer doesn't split or only +# generate multiple bios for an atomic write. +# + +. ./common/preamble +. ./common/atomicwrites + +_begin_fstest auto rw stress atomicwrites + +_require_scratch_write_atomic +_require_fio_atomic_writes +_require_aiodio + +FSSIZE=$((360*1024*1024)) +FIO_LOAD=$(($(nproc) * LOAD_FACTOR)) + +# Calculate bs as per bdev atomic write units. +bdev_awu_min=$(_get_atomic_write_unit_min $SCRATCH_DEV) +bdev_awu_max=$(_get_atomic_write_unit_max $SCRATCH_DEV) +bs=$(_max 4096 "$bdev_awu_min") + +function create_fio_configs() +{ + local bsize=$1 + create_fio_aw_config $bsize + create_fio_verify_config $bsize +} + +function create_fio_verify_config() +{ + local bsize=$1 +cat >$fio_verify_config <<EOF + [global] + direct=1 + ioengine=libaio + rw=read + bs=$bsize + fallocate=truncate + size=$((FSSIZE / 12)) + iodepth=$FIO_LOAD + numjobs=$FIO_LOAD + group_reporting=1 + atomic=1 + + verify_only=1 + verify_state_save=0 + verify=crc32c + verify_fatal=1 + verify_write_sequence=0 + + [verify-job1] + filename=$SCRATCH_MNT/testfile-job1 + + [verify-job2] + filename=$SCRATCH_MNT/testfile-job2 + + [verify-job3] + filename=$SCRATCH_MNT/testfile-job3 + + [verify-job4] + filename=$SCRATCH_MNT/testfile-job4 + + [verify-job5] + filename=$SCRATCH_MNT/testfile-job5 + + [verify-job6] + filename=$SCRATCH_MNT/testfile-job6 + + [verify-job7] + filename=$SCRATCH_MNT/testfile-job7 + + [verify-job8] + filename=$SCRATCH_MNT/testfile-job8 + +EOF +} + +function create_fio_aw_config() +{ + local bsize=$1 +cat >$fio_aw_config <<EOF + [global] + direct=1 + ioengine=libaio + rw=randwrite + bs=$bsize + fallocate=truncate + size=$((FSSIZE / 12)) + iodepth=$FIO_LOAD + numjobs=$FIO_LOAD + group_reporting=1 + atomic=1 + + verify_state_save=0 + verify=crc32c + do_verify=0 + + [write-job1] + filename=$SCRATCH_MNT/testfile-job1 + + [write-job2] + filename=$SCRATCH_MNT/testfile-job2 + + [write-job3] + filename=$SCRATCH_MNT/testfile-job3 + + [write-job4] + filename=$SCRATCH_MNT/testfile-job4 + + [write-job5] + filename=$SCRATCH_MNT/testfile-job5 + + [write-job6] + filename=$SCRATCH_MNT/testfile-job6 + + [write-job7] + filename=$SCRATCH_MNT/testfile-job7 + + [write-job8] + filename=$SCRATCH_MNT/testfile-job8 + +EOF +} + +run_test_one() { + local bs=$1 + local cs=$2 + local iosize=$3 + + MKFS_OPTIONS="-O bigalloc -b $bs -C $cs" + _scratch_mkfs_sized "$FSSIZE" >> $seqres.full 2>&1 || return + if _try_scratch_mount >> $seqres.full 2>&1; then + echo "Testing: bs=$bs cs=$cs iosize=$iosize" >> $seqres.full + + touch $SCRATCH_MNT/f1 + create_fio_configs $iosize + + cat $fio_aw_config >> $seqres.full + cat $fio_verify_config >> $seqres.full + + $FIO_PROG $fio_aw_config >> $seqres.full + ret1=$? + + $FIO_PROG $fio_verify_config >> $seqres.full + ret2=$? + + _scratch_unmount + + [[ $ret1 -eq 0 && $ret2 -eq 0 ]] || _fail "fio with atomic write failed" + fi +} + +run_test() { + local bs=$1 + + # cluster sizes above 16 x blocksize are experimental so avoid them + # Also, cap cluster size at 128kb to keep it reasonable for large + # blocks size + max_cs=$(_min $((16 * bs)) "$bdev_awu_max" $((128 * 1024))) + + # Fuzz for combinations of blocksize, clustersize and + # iosize that cover most of the cases + run_test_one $bs $bs $bs + run_test_one $bs $max_cs $bs + run_test_one $bs $max_cs $max_cs + run_test_one $bs $max_cs $(_max "$((max_cs/2))" $bs) +} + +# Let's create a sample fio config to check whether fio supports all options. +fio_aw_config=$tmp.aw.fio +fio_verify_config=$tmp.verify.fio +fio_out=$tmp.fio.out + +create_fio_configs $bs +_require_fio $fio_aw_config + +for ((bs=$bs; bs <= $(_get_page_size); bs = $bs << 1)); do + run_test $bs $cs $iosize +done + +# success, all done +echo Silence is golden +status=0 +exit
tests/ext4/062.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 062 +Silence is golden
Signature · 1 line
2.49.0
PATCH v6 12/12 ext4: Atomic write test for extent split across leaf nodes
In reply to Ojaswin MujooIn ext4, even if an allocated range is physically and logically contiguous, it can still be split into 2 extents. This is because ext4 does not merge extents across leaf nodes. This is an issue for atomic writes since even for a continuous extent the map block could (in rare cases) return a shorter map, hence tearning the write. This test creates such a file and ensures that the atomic write handles this case correctly
Reviewed-by Darrick J. Wong <[email protected]>
Signed-off-by Ojaswin Mujoo <[email protected]>
--- tests/ext4/063 | 129 +++++++++++++++++++++++++++++++++++++++++++++ tests/ext4/063.out | 2 + 2 files changed, 131 insertions(+) create mode 100755 tests/ext4/063 create mode 100644 tests/ext4/063.out
tests/ext4/063 new file +129 -0
@@ -0,0 +1,129 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 IBM Corporation. All Rights Reserved. +# +# In ext4, even if an allocated range is physically and logically contiguous, +# it can still be split into 2 or more extents. This is because ext4 does not +# merge extents across leaf nodes. This is an issue for atomic writes since +# even for a continuous extent the map block could (in rare cases) return a +# shorter map, hence tearing the write. This test creates such a file and +# ensures that the atomic write handles this case correctly +# +. ./common/preamble +. ./common/atomicwrites +_begin_fstest auto atomicwrites + +_require_scratch_write_atomic_multi_fsblock +_require_atomic_write_test_commands +_require_command "$DEBUGFS_PROG" debugfs + +prep() { + local bs=`_get_block_size $SCRATCH_MNT` + local ex_hdr_bytes=12 + local ex_entry_bytes=12 + local entries_per_blk=$(( (bs - ex_hdr_bytes) / ex_entry_bytes )) + + # fill the extent tree leaf with bs len extents at alternate offsets. + # The tree should look as follows + # + # +---------+---------+ + # | index 1 | index 2 | + # +-----+---+-----+---+ + # +------+ +-----------+ + # | | + # +-------+-------+---+---------+ +-----+----+ + # | ex 1 | ex 2 | | ex n | | ex n+1 | + # | off:0 | off:2 |...| off:678 | | off:680 | + # | len:1 | len:1 | | len:1 | | len:1 | + # +-------+-------+---+---------+ +----------+ + # + for i in $(seq 0 $entries_per_blk) + do + $XFS_IO_PROG -fc "pwrite -b $bs $((i * 2 * bs)) $bs" $testfile > /dev/null + done + sync $testfile + + echo >> $seqres.full + echo "Create file with extents spanning 2 leaves. Extents:">> $seqres.full + echo "...">> $seqres.full + $DEBUGFS_PROG -R "ex `basename $testfile`" $SCRATCH_DEV |& tail >> $seqres.full + + # Now try to insert a new extent ex(new) between ex(n) and ex(n+1). + # Since this is a new FS the allocator would find continuous blocks + # such that ex(n) ex(new) ex(n+1) are physically(and logically) + # contiguous. However, since we don't merge extents across leaf we will + # end up with a tree as: + # + # +---------+---------+ + # | index 1 | index 2 | + # +-----+---+-----+---+ + # +------+ +------------+ + # | | + # +-------+-------+---+---------+ +------+-----------+ + # | ex 1 | ex 2 | | ex n | | ex n+1 (merged) | + # | off:0 | off:2 |...| off:678 | | off:679 | + # | len:1 | len:1 | | len:1 | | len:2 | + # +-------+-------+---+---------+ +------------------+ + # + echo >> $seqres.full + torn_ex_offset=$((((entries_per_blk * 2) - 1) * bs)) + $XFS_IO_PROG -c "pwrite $torn_ex_offset $bs" $testfile >> /dev/null + sync $testfile + + echo >> $seqres.full + echo "Perform 1 block write at $torn_ex_offset to create torn extent. Extents:">> $seqres.full + echo "...">> $seqres.full + $DEBUGFS_PROG -R "ex `basename $testfile`" $SCRATCH_DEV |& tail >> $seqres.full + + _scratch_cycle_mount +} + +_scratch_mkfs >> $seqres.full +_scratch_mount >> $seqres.full + +testfile=$SCRATCH_MNT/testfile +touch $testfile +awu_max=$(_get_atomic_write_unit_max $testfile) + +echo >> $seqres.full +echo "# Prepping the file" >> $seqres.full +prep + +torn_aw_offset=$((torn_ex_offset - (torn_ex_offset % awu_max))) + +echo >> $seqres.full +echo "# Performing atomic IO on the torn extent range. Command: " >> $seqres.full +echo $XFS_IO_PROG -c "open -fsd $testfile" -c "pwrite -S 0x61 -DA -V1 -b $awu_max $torn_aw_offset $awu_max" >> $seqres.full +$XFS_IO_PROG -c "open -fsd $testfile" -c "pwrite -S 0x61 -DA -V1 -b $awu_max $torn_aw_offset $awu_max" >> $seqres.full + +echo >> $seqres.full +echo "Extent state after atomic write:">> $seqres.full +echo "...">> $seqres.full +$DEBUGFS_PROG -R "ex `basename $testfile`" $SCRATCH_DEV |& tail >> $seqres.full + +echo >> $seqres.full +echo "# Checking data integrity" >> $seqres.full + +# create a dummy file with expected data +$XFS_IO_PROG -fc "pwrite -S 0x61 -b $awu_max 0 $awu_max" $testfile.exp >> /dev/null +expected_data=$(od -An -t x1 -j 0 -N $awu_max $testfile.exp) + +# We ensure that the data after atomic writes should match the expected data +actual_data=$(od -An -t x1 -j $torn_aw_offset -N $awu_max $testfile) +if [[ "$actual_data" != "$expected_data" ]] +then + echo "Checksum match failed at off: $torn_aw_offset size: $awu_max" + echo + echo "Expected: " + echo "$expected_data" + echo + echo "Actual contents: " + echo "$actual_data" + + _fail +fi + +echo -n "Data verification at offset $torn_aw_offset succeeded!" >> $seqres.full +echo "Silence is golden" +status=0 +exit
tests/ext4/063.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 063 +Silence is golden
Signature · 1 line
2.49.0
Re: [PATCH v6 01/12] common/rc: Add _min() and _max() helpers
In reply to Ojaswin MujooQuoted 5 lines · Ojaswin Mujoo
Many programs open code these functionalities so add it as a generic helper in common/rc
Reviewed-by Darrick J. Wong <[email protected]>Signed-off-by Ojaswin Mujoo <[email protected]>
Reviewed-by John Garry <[email protected]>
I just sent a patch for something similar for blktests to linux-block. I wonder how much commonality there is for such helpers...
BTW, let me know if I should attribute some credit there. cheers
Re: [PATCH v6 01/12] common/rc: Add _min() and _max() helpers
Quoted 7 lines · John Garry
Quoted 5 lines · Ojaswin Mujoo
Many programs open code these functionalities so add it as a generic helper in common/rc
Reviewed-by Darrick J. Wong <[email protected]>Signed-off-by Ojaswin Mujoo <[email protected]>
Quoted 6 lines
Reviewed-by John Garry <[email protected]>I just sent a patch for something similar for blktests to linux-block. I wonder how much commonality there is for such helpers...
BTW, let me know if I should attribute some credit there. cheers
Thanks for the review John!
I think the helpers are simple enough so credit is not needed :)
Thanks,
Ojaswin
Re: [PATCH v6 02/12] common/rc: Add fio atomic write helpers
In reply to Ojaswin MujooQuoted 8 lines · Ojaswin Mujoo
The main motivation of adding this function on top of
_require_fiois that there has been a case in fio where atomic= option was added but later it was changed to noop since kernel didn't yet have support for atomic writes. It was then again utilized to do atomic writes in a later version, once kernel got the support. Due to this there is a point in fio where_require_fiow/ atomic=1 will succeed even though it would not be doing atomic writes.
Quoted 7 lines
Hence, add an internal helper
__require_fio_versionto require specific versions of fio to work past such issues. Further, add the high level_require_fio_atomic_writeshelper which tests can use to ensure fio has the right version for atomic writes.Reviewed-by Zorro Lang <[email protected]>Signed-off-by Ojaswin Mujoo <[email protected]>
Thanks for doing this.
Reviewed-by John Garry <[email protected]>
Quoted 58 lines
--- common/rc | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)common/rc +43
-0@@ -6000,6 +6000,49 @@ _max() { echo $ret } +# Due to reasons explained in fio commit 40f1fc11d, fio version between +# v3.33 and v3.38 have atomic= feature but it is a no-op and doesn't do +# RWF_ATOMIC write. Hence, use this helper to ensure fio has the +# required support. Currently, the simplest way we have is to ensure +# the version. +_require_fio_atomic_writes() { + __require_fio_version "3.38+" +} + +# Check the required fio version. Examples: +# __require_fio_version 3.38 (matches 3.38 only) +# __require_fio_version 3.38+ (matches 3.38 and above) +# __require_fio_version 3.38- (matches 3.38 and below) +# +# Internal helper, avoid using directly in tests. +__require_fio_version() { + local req_ver="$1" + local fio_ver + + _require_fio + _require_math + + fio_ver=$(fio -v | cut -d"-" -f2) + + case "$req_ver" in + *+) + req_ver=${req_ver%+} + test $(_math "$fio_ver >= $req_ver") -eq 1 || \ + _notrun "need fio >= $req_ver (found $fio_ver)" + ;; + *-) + req_ver=${req_ver%-} + test $(_math "$fio_ver <= $req_ver") -eq 1 || \ + _notrun "need fio <= $req_ver (found $fio_ver)" + ;; + *) + req_ver=${req_ver%-} + test $(_math "$fio_ver == $req_ver") -eq 1 || \ + _notrun "need fio = $req_ver (found $fio_ver)" + ;; + esac +} + ################################################################################ # make sure this script returns success /bin/true
Re: [PATCH v6 03/12] common/rc: Add a helper to run fsx on a given file
In reply to Ojaswin MujooQuoted 9 lines · Ojaswin Mujoo
Currently
run_fsxis hardcoded to run on a file in $TEST_DIR. Add a helper_run_fsx_on_fileso that we can run fsx on any given file including in $SCRATCH_MNT. Also, refactor_run_fsxto use this helper.No functional change is intended in this patch.
Reviewed-by Darrick J. Wong <[email protected]>Signed-off-by Ojaswin Mujoo <[email protected]>
Reviewed-by John Garry <[email protected]>
Quoted 49 lines
--- common/rc | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)common/rc +20
-3@@ -5203,13 +5203,24 @@ _require_hugepage_fsx() _notrun "fsx binary does not support MADV_COLLAPSE" } -_run_fsx() +_run_fsx_on_file() { + local testfile=$1 + shift + + if ! [ -f $testfile ] + then + echo "_run_fsx_on_file: $testfile doesn't exist. Creating" >> $seqres.full + touch $testfile + fi + echo "fsx $*" local args=`echo $@ | sed -e "s/ BSIZE / $bsize /g" -e "s/ PSIZE / $psize /g"` - set -- $FSX_PROG $args $FSX_AVOID $TEST_DIR/junk + + set -- $FSX_PROG $args $FSX_AVOID $testfile + echo "$@" >>$seqres.full - rm -f $TEST_DIR/junk + rm -f $testfile "$@" 2>&1 | tee -a $seqres.full >$tmp.fsx local res=${PIPESTATUS[0]} if [ $res -ne 0 ]; then@@ -5221,6 +5232,12 @@ _run_fsx() return 0 } +_run_fsx() +{ + _run_fsx_on_file $TEST_DIR/junk $@ + return $? +} + # Run fsx with -h(ugepage buffers). If we can't set up a hugepage then skip # the test, but if any other error occurs then exit the test. _run_hugepage_fsx() {
Re: [PATCH v6 09/12] generic: Add sudden shutdown tests for multi block atomic writes
In reply to Ojaswin MujooQuoted 10 lines · Ojaswin Mujoo
This test is intended to ensure that multi blocks atomic writes maintain atomic guarantees across sudden FS shutdowns.
The way we work is that we lay out a file with random mix of written, unwritten and hole extents. Then we start performing atomic writes sequentially on the file while we parallelly shutdown the FS. Then we note the last offset where the atomic write happened just before shut down and then make sure blocks around it either have completely old data or completely new data, ie the write was not torn during shutdown.
Quoted 7 lines
We repeat the same with completely written, completely unwritten and completely empty file to ensure these cases are not torn either. Finally, we have a similar test for append atomic writes
Suggested-by Ritesh Harjani (IBM) <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>Signed-off-by Ojaswin Mujoo <[email protected]>
I still have some nits, which are close to being the same as last time. I don't want this series to be held up any longer over my nitpicking, so:
Reviewed-by John Garry <[email protected]>
Quoted 58 lines
--- tests/generic/1230 | 368 +++++++++++++++++++++++++++++++++++++++++ tests/generic/1230.out | 2 + 2 files changed, 370 insertions(+)create mode 100755 tests/generic/1230
create mode 100644 tests/generic/1230.outtests/generic/1230 new file +45
-0@@ -0,0 +1,368 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 IBM Corporation. All Rights Reserved. +# +# FS QA Test No. 1230 +# +# Test multi block atomic writes with sudden FS shutdowns to ensure +# the FS is not tearing the write operation +. ./common/preamble +. ./common/atomicwrites +_begin_fstest auto atomicwrites + +_require_scratch_write_atomic_multi_fsblock +_require_atomic_write_test_commands +_require_scratch_shutdown +_require_xfs_io_command "truncate" + +_scratch_mkfs >> $seqres.full 2>&1 +_scratch_mount >> $seqres.full + +testfile=$SCRATCH_MNT/testfile +touch $testfile + +awu_max=$(_get_atomic_write_unit_max $testfile) +blksz=$(_get_block_size $SCRATCH_MNT) +echo "Awu max: $awu_max" >> $seqres.full + +num_blocks=$((awu_max / blksz)) +# keep initial value high for dry run. This will be +# tweaked in dry_run() based on device write speed. +filesize=$(( 10 * 1024 * 1024 * 1024 )) + +_cleanup() { + [ -n "$awloop_pid" ] && kill $awloop_pid &> /dev/null + wait +} + +atomic_write_loop() { + local off=0 + local size=$awu_max + for ((i=0; i<$((filesize / $size )); i++)); do + # Due to sudden shutdown this can produce errors so just + # redirect them to seqres.full + $XFS_IO_PROG -c "open -fsd $testfile" -c "pwrite -S 0x61 -DA -V1 -b $size $off $size" >> /dev/null 2>>$seqres.full + echo "Written to offset: $off" >> $tmp.aw
tests/generic/1230 line 55
Quoted 10 lines
tests/generic/1230 new file +10
-0+ off=$((off + $size)) + done +} + +start_atomic_write_and_shutdown() { + atomic_write_loop & + awloop_pid=$! + + local i=0 + # Wait for atleast first write to be recorded or 10s
at least
Quoted 15 lines
tests/generic/1230 new file +15
-0+ while [ ! -f "$tmp.aw" -a $i -le 50 ]; do i=$((i + 1)); sleep 0.2; done + + if [[ $i -gt 50 ]] + then + _fail "atomic write process took too long to start" + fi + + echo >> $seqres.full + echo "# Shutting down filesystem while write is running" >> $seqres.full + _scratch_shutdown + + kill $awloop_pid 2>/dev/null # the process might have finished already + wait $awloop_pid + unset $awloop_pid +}
...
Quoted 13 lines
tests/generic/1230 new file +13
-0+ +verify_data_blocks() { + local verify_start=$1 + local verify_end=$2 + local expected_data_old="$3" + local expected_data_new="$4" + + echo >> $seqres.full + echo "# Checking data integrity from $verify_start to $verify_end" >> $seqres.full + + # After an atomic write, for every chunk we ensure that the underlying + # data is either the old data or new data as writes shouldn't get torn. + local off=$verify_start
tests/generic/1230 line 171
Quoted 10 lines
tests/generic/1230 new file +10
-0+ while [[ "$off" -lt "$verify_end" ]] + do + #actual_data=$(xxd -s $off -l $awu_max -p $testfile) + actual_data=$(od -An -t x1 -j $off -N $awu_max $testfile) + if [[ "$actual_data" != "$expected_data_new" ]] && [[ "$actual_data" != "$expected_data_old" ]] + then + echo "Checksum match failed at off: $off size: $awu_max" + echo "Expected contents: (Either of the 2 below):" + echo + echo "Expected old: "
nit: I think that I mentioned this the last time - I would not use the word "expected". We have old data, new data, and actual data. The only thing which we expect is that actual data will be either all old or all new.
Quoted 62 lines
tests/generic/1230 new file +62
-0+ echo "$expected_data_old" + echo + echo "Expected new: " + echo "$expected_data_new" + echo + echo "Actual contents: " + echo "$actual_data" + + _fail + fi + echo -n "Check at offset $off succeeded! " >> $seqres.full + if [[ "$actual_data" == "$expected_data_new" ]] + then + echo "matched new" >> $seqres.full + elif [[ "$actual_data" == "$expected_data_old" ]] + then + echo "matched old" >> $seqres.full + fi + off=$(( off + awu_max )) + done +} + +# test data integrity for file by shutting down in between atomic writes +test_data_integrity() { + echo >> $seqres.full + echo "# Writing atomically to file in background" >> $seqres.full + + start_atomic_write_and_shutdown + + last_offset=$(tail -n 1 $tmp.aw | cut -d" " -f4) + if [[ -z $last_offset ]] + then + last_offset=0 + fi + + echo >> $seqres.full + echo "# Last offset of atomic write: $last_offset" >> $seqres.full + + rm $tmp.aw + sleep 0.5 + + _scratch_cycle_mount + + # we want to verify all blocks around which the shutdown happened + verify_start=$(( last_offset - (awu_max * 5))) + if [[ $verify_start < 0 ]] + then + verify_start=0 + fi + + verify_end=$(( last_offset + (awu_max * 5))) + if [[ "$verify_end" -gt "$filesize" ]] + then + verify_end=$filesize + fi +} + +# test data integrity for file with written and unwritten mappings +test_data_integrity_mixed() { + $XFS_IO_PROG -fc "truncate 0" $testfile >> $seqres.full + + echo >> $seqres.full
tests/generic/1230 line 243
Quoted 10 lines
tests/generic/1230 new file +10
-0+ echo "# Creating testfile with mixed mappings" >> $seqres.full + create_mixed_mappings $testfile $filesize + + test_data_integrity + + verify_data_blocks $verify_start $verify_end "$expected_data_old_mixed" "$expected_data_new" +} + +# test data integrity for file with completely written mappings +test_data_integrity_written() {
nit: again, I am not so keen on using the word "integrity" at all. "integrity" in storage world relates to T10 PI support in Linux. I know that last time I mentioned it's ok to use "integrity" when close to words "atomic write", but I still fear some doubt on whether we are talking about T10 PI when we mention integrity.
Quoted 26 lines
tests/generic/1230 new file +26
-0+ $XFS_IO_PROG -fc "truncate 0" $testfile >> $seqres.full + + echo >> $seqres.full + echo "# Creating testfile with fully written mapping" >> $seqres.full + $XFS_IO_PROG -c "pwrite -b $filesize 0 $filesize" $testfile >> $seqres.full + sync $testfile + + test_data_integrity + + verify_data_blocks $verify_start $verify_end "$expected_data_old_mapped" "$expected_data_new" +} + +# test data integrity for file with completely unwritten mappings +test_data_integrity_unwritten() { + $XFS_IO_PROG -fc "truncate 0" $testfile >> $seqres.full + + echo >> $seqres.full + echo "# Creating testfile with fully unwritten mappings" >> $seqres.full + $XFS_IO_PROG -c "falloc 0 $filesize" $testfile >> $seqres.full + sync $testfile + + test_data_integrity + + verify_data_blocks $verify_start $verify_end "$expected_data_old_zeroes" "$expected_data_new" +} +
Re: [PATCH v6 09/12] generic: Add sudden shutdown tests for multi block atomic writes
Quoted 132 lines · John Garry
Quoted 10 lines · Ojaswin Mujoo
This test is intended to ensure that multi blocks atomic writes maintain atomic guarantees across sudden FS shutdowns.
The way we work is that we lay out a file with random mix of written, unwritten and hole extents. Then we start performing atomic writes sequentially on the file while we parallelly shutdown the FS. Then we note the last offset where the atomic write happened just before shut down and then make sure blocks around it either have completely old data or completely new data, ie the write was not torn during shutdown.
Quoted 7 lines
We repeat the same with completely written, completely unwritten and completely empty file to ensure these cases are not torn either. Finally, we have a similar test for append atomic writes
Suggested-by Ritesh Harjani (IBM) <[email protected]>Reviewed-by Darrick J. Wong <[email protected]>Signed-off-by Ojaswin Mujoo <[email protected]>I still have some nits, which are close to being the same as last time. I don't want this series to be held up any longer over my nitpicking, so:
Reviewed-by John Garry <[email protected]>Quoted 58 lines
--- tests/generic/1230 | 368 +++++++++++++++++++++++++++++++++++++++++ tests/generic/1230.out | 2 + 2 files changed, 370 insertions(+)create mode 100755 tests/generic/1230
create mode 100644 tests/generic/1230.outtests/generic/1230 new file +45
-0@@ -0,0 +1,368 @@ +#! /bin/bash +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 IBM Corporation. All Rights Reserved. +# +# FS QA Test No. 1230 +# +# Test multi block atomic writes with sudden FS shutdowns to ensure +# the FS is not tearing the write operation +. ./common/preamble +. ./common/atomicwrites +_begin_fstest auto atomicwrites + +_require_scratch_write_atomic_multi_fsblock +_require_atomic_write_test_commands +_require_scratch_shutdown +_require_xfs_io_command "truncate" + +_scratch_mkfs >> $seqres.full 2>&1 +_scratch_mount >> $seqres.full + +testfile=$SCRATCH_MNT/testfile +touch $testfile + +awu_max=$(_get_atomic_write_unit_max $testfile) +blksz=$(_get_block_size $SCRATCH_MNT) +echo "Awu max: $awu_max" >> $seqres.full + +num_blocks=$((awu_max / blksz)) +# keep initial value high for dry run. This will be +# tweaked in dry_run() based on device write speed. +filesize=$(( 10 * 1024 * 1024 * 1024 )) + +_cleanup() { + [ -n "$awloop_pid" ] && kill $awloop_pid &> /dev/null + wait +} + +atomic_write_loop() { + local off=0 + local size=$awu_max + for ((i=0; i<$((filesize / $size )); i++)); do + # Due to sudden shutdown this can produce errors so just + # redirect them to seqres.full + $XFS_IO_PROG -c "open -fsd $testfile" -c "pwrite -S 0x61 -DA -V1 -b $size $off $size" >> /dev/null 2>>$seqres.full + echo "Written to offset: $off" >> $tmp.awtests/generic/1230 line 55
Quoted 10 lines
tests/generic/1230 new file +10
-0+ off=$((off + $size)) + done +} + +start_atomic_write_and_shutdown() { + atomic_write_loop & + awloop_pid=$! + + local i=0 + # Wait for atleast first write to be recorded or 10sat least
Quoted 15 lines
tests/generic/1230 new file +15
-0+ while [ ! -f "$tmp.aw" -a $i -le 50 ]; do i=$((i + 1)); sleep 0.2; done + + if [[ $i -gt 50 ]] + then + _fail "atomic write process took too long to start" + fi + + echo >> $seqres.full + echo "# Shutting down filesystem while write is running" >> $seqres.full + _scratch_shutdown + + kill $awloop_pid 2>/dev/null # the process might have finished already + wait $awloop_pid + unset $awloop_pid +}...
Quoted 19 lines
tests/generic/1230 new file +19
-0+ +verify_data_blocks() { + local verify_start=$1 + local verify_end=$2 + local expected_data_old="$3" + local expected_data_new="$4" + + echo >> $seqres.full + echo "# Checking data integrity from $verify_start to $verify_end" >> $seqres.full + + # After an atomic write, for every chunk we ensure that the underlying + # data is either the old data or new data as writes shouldn't get torn. + local off=$verify_start + while [[ "$off" -lt "$verify_end" ]] + do + #actual_data=$(xxd -s $off -l $awu_max -p $testfile) + actual_data=$(od -An -t x1 -j $off -N $awu_max $testfile) + if [[ "$actual_data" != "$expected_data_new" ]] && [[ "$actual_data" != "$expected_data_old" ]] + then
Quoted 8 lines
tests/generic/1230 line 171
Quoted 4 lines
tests/generic/1230 new file +4
-0+ echo "Checksum match failed at off: $off size: $awu_max" + echo "Expected contents: (Either of the 2 below):" + echo + echo "Expected old: "nit: I think that I mentioned this the last time - I would not use the word "expected". We have old data, new data, and actual data. The only thing which we expect is that actual data will be either all old or all new.
Hey John so I mentioned here [1] that the wording "expected new", "expected old", "actual" looked more clear to me than "new", "old" and "actual" and you replied with sure so I though we were good there :)
But no worries I can make this change. I'll keep the wording as new, old and actual.
Quoted 71 lines
Quoted 71 lines
tests/generic/1230 new file +70
-0+ echo "$expected_data_old" + echo + echo "Expected new: " + echo "$expected_data_new" + echo + echo "Actual contents: " + echo "$actual_data" + + _fail + fi + echo -n "Check at offset $off succeeded! " >> $seqres.full + if [[ "$actual_data" == "$expected_data_new" ]] + then + echo "matched new" >> $seqres.full + elif [[ "$actual_data" == "$expected_data_old" ]] + then + echo "matched old" >> $seqres.full + fi + off=$(( off + awu_max )) + done +} + +# test data integrity for file by shutting down in between atomic writes +test_data_integrity() { + echo >> $seqres.full + echo "# Writing atomically to file in background" >> $seqres.full + + start_atomic_write_and_shutdown + + last_offset=$(tail -n 1 $tmp.aw | cut -d" " -f4) + if [[ -z $last_offset ]] + then + last_offset=0 + fi + + echo >> $seqres.full + echo "# Last offset of atomic write: $last_offset" >> $seqres.full + + rm $tmp.aw + sleep 0.5 + + _scratch_cycle_mount + + # we want to verify all blocks around which the shutdown happened + verify_start=$(( last_offset - (awu_max * 5))) + if [[ $verify_start < 0 ]] + then + verify_start=0 + fi + + verify_end=$(( last_offset + (awu_max * 5))) + if [[ "$verify_end" -gt "$filesize" ]] + then + verify_end=$filesize + fi +} + +# test data integrity for file with written and unwritten mappings +test_data_integrity_mixed() { + $XFS_IO_PROG -fc "truncate 0" $testfile >> $seqres.full + + echo >> $seqres.full + echo "# Creating testfile with mixed mappings" >> $seqres.full + create_mixed_mappings $testfile $filesize + + test_data_integrity + + verify_data_blocks $verify_start $verify_end "$expected_data_old_mixed" "$expected_data_new" +} +
Quoted 8 lines
tests/generic/1230 line 243
Quoted 2 lines
tests/generic/1230 new file +2
-0+# test data integrity for file with completely written mappings +test_data_integrity_written() {nit: again, I am not so keen on using the word "integrity" at all. "integrity" in storage world relates to T10 PI support in Linux. I know that last time I mentioned it's ok to use "integrity" when close to words "atomic write", but I still fear some doubt on whether we are talking about T10 PI when we mention integrity.
Okay got it, fine then how about using phrases like "test for torn data for file with completely written mapping" and such?
Quoted 27 lines
Quoted 27 lines
tests/generic/1230 new file +26
-0+ $XFS_IO_PROG -fc "truncate 0" $testfile >> $seqres.full + + echo >> $seqres.full + echo "# Creating testfile with fully written mapping" >> $seqres.full + $XFS_IO_PROG -c "pwrite -b $filesize 0 $filesize" $testfile >> $seqres.full + sync $testfile + + test_data_integrity + + verify_data_blocks $verify_start $verify_end "$expected_data_old_mapped" "$expected_data_new" +} + +# test data integrity for file with completely unwritten mappings +test_data_integrity_unwritten() { + $XFS_IO_PROG -fc "truncate 0" $testfile >> $seqres.full + + echo >> $seqres.full + echo "# Creating testfile with fully unwritten mappings" >> $seqres.full + $XFS_IO_PROG -c "falloc 0 $filesize" $testfile >> $seqres.full + sync $testfile + + test_data_integrity + + verify_data_blocks $verify_start $verify_end "$expected_data_old_zeroes" "$expected_data_new" +} +
Re: [PATCH v6 09/12] generic: Add sudden shutdown tests for multi block atomic writes
Quoted 10 lines · Ojaswin Mujoo
Quoted 3 lines
nit: I think that I mentioned this the last time - I would not use the word "expected". We have old data, new data, and actual data. The only thing which we expect is that actual data will be either all old or all new.
Hey John so I mentioned here [1] that the wording "expected new", "expected old", "actual" looked more clear to me than "new", "old" and "actual" and you replied with sure so I though we were good there 🙂
But no worries I can make this change. I'll keep the wording as new, old and actual.
great, thanks
Quoted 78 lines
Quoted 70 lines
Quoted 70 lines
tests/generic/1230 new file +69
-0+ echo "$expected_data_old" + echo + echo "Expected new: " + echo "$expected_data_new" + echo + echo "Actual contents: " + echo "$actual_data" + + _fail + fi + echo -n "Check at offset $off succeeded! " >> $seqres.full + if [[ "$actual_data" == "$expected_data_new" ]] + then + echo "matched new" >> $seqres.full + elif [[ "$actual_data" == "$expected_data_old" ]] + then + echo "matched old" >> $seqres.full + fi + off=$(( off + awu_max )) + done +} + +# test data integrity for file by shutting down in between atomic writes +test_data_integrity() { + echo >> $seqres.full + echo "# Writing atomically to file in background" >> $seqres.full + + start_atomic_write_and_shutdown + + last_offset=$(tail -n 1 $tmp.aw | cut -d" " -f4) + if [[ -z $last_offset ]] + then + last_offset=0 + fi + + echo >> $seqres.full + echo "# Last offset of atomic write: $last_offset" >> $seqres.full + + rm $tmp.aw + sleep 0.5 + + _scratch_cycle_mount + + # we want to verify all blocks around which the shutdown happened + verify_start=$(( last_offset - (awu_max * 5))) + if [[ $verify_start < 0 ]] + then + verify_start=0 + fi + + verify_end=$(( last_offset + (awu_max * 5))) + if [[ "$verify_end" -gt "$filesize" ]] + then + verify_end=$filesize + fi +} + +# test data integrity for file with written and unwritten mappings +test_data_integrity_mixed() { + $XFS_IO_PROG -fc "truncate 0" $testfile >> $seqres.full + + echo >> $seqres.full + echo "# Creating testfile with mixed mappings" >> $seqres.full + create_mixed_mappings $testfile $filesize + + test_data_integrity + + verify_data_blocks $verify_start $verify_end "$expected_data_old_mixed" "$expected_data_new" +}Quoted 8 lines
tests/generic/1230 line 243
Quoted 3 lines
tests/generic/1230 new file +3
-0+ +# test data integrity for file with completely written mappings +test_data_integrity_written() {nit: again, I am not so keen on using the word "integrity" at all. "integrity" in storage world relates to T10 PI support in Linux. I know that last time I mentioned it's ok to use "integrity" when close to words "atomic write", but I still fear some doubt on whether we are talking about T10 PI when we mention integrity.
Quoted 2 lines
Okay got it, fine then how about using phrases like "test for torn data for file with completely written mapping" and such?
sure, it's ok
Thanks very much