random fixes for fstests
cover random fixes for fstests
These 5 patches are random fixes, except patch 4/5 bases on 3/5.
- [PATCH 1/5] generic/139: require 512 bytes to be the minimum dio size
I tried to help it support 4096 sector size, but it's not simple, especially the golden image broken. So I'm thinking how about limit it in 512 bytes dio testing.
- [PATCH 2/5] generic/506: call _require_quota before _qmount
As subject
- [PATCH 3/5] generic/591: remove redundant output from golden image
I think I found where's the issue from, but I'm not sure if it's the best way to fix it. So welcome suggestions :)
- [PATCH 4/5] generic/591: use proper sector size
Help splice-test to support a sector size option
- [PATCH 5/5] generic/623: add overlay into the blacklist
I think it's simple to exclude overlay from this test directly, or we have to check if $FSTYP=overlay, then think about how to deal with OVL_BASE_SCRATCH_DEV things.
Thanks,
Zorro
PATCH 1/5 generic/139: require 512 bytes to be the minimum dio size
Due to generic/139 tests base on 512 bytes aligned, so skip this test if the minimum dio write size >512. This patch also change the common/rc::_require_dio helper, supports a minimum aligned size argument.
--- common/rc | 11 ++++++++--- tests/generic/139 | 2 +- 2 files changed, 9 insertions(+), 4 deletions(-)
common/rc +8 -3
@@ -2721,7 +2721,8 @@ _require_xfs_io_command() fi } -# check that kernel and filesystem support direct I/O +# check that kernel and filesystem support direct I/O, and check if "$1" size +# aligned (optional) is supported _require_odirect() { if [ $FSTYP = "ext4" ] || [ $FSTYP = "f2fs" ] ; then
@@ -2735,9 +2736,13 @@ _require_odirect() fi fi local testfile=$TEST_DIR/$$.direct - $XFS_IO_PROG -F -f -d -c "pwrite 0 20k" $testfile > /dev/null 2>&1 + local opt + if [ -n "$1" ];then + opt="-b $1" + fi + $XFS_IO_PROG -F -f -d -c "pwrite $opt 0 20k" $testfile > /dev/null 2>&1 if [ $? -ne 0 ]; then - _notrun "O_DIRECT is not supported" + _notrun "O_DIRECT $1 is not supported" fi rm -f $testfile 2>&1 > /dev/null }
tests/generic/139 +1 -1
@@ -26,7 +26,7 @@ _cleanup() # real QA test starts here _require_test_reflink _require_cp_reflink -_require_odirect +_require_odirect 512 testdir=$TEST_DIR/test-$seq rm -rf $testdir
Signature · 1 line
2.31.1
Re: [PATCH 1/5] generic/139: require 512 bytes to be the minimum dio size
Quoted 26 lines · Zorro Lang
Due to generic/139 tests base on 512 bytes aligned, so skip this test if the minimum dio write size >512. This patch also change the common/rc::_require_dio helper, supports a minimum aligned size argument.
Signed-off-by Zorro Lang <[email protected]>--- common/rc | 11 ++++++++--- tests/generic/139 | 2 +- 2 files changed, 9 insertions(+), 4 deletions(-)common/rc +2
-1@@ -2721,7 +2721,8 @@ _require_xfs_io_command() fi } -# check that kernel and filesystem support direct I/O +# check that kernel and filesystem support direct I/O, and check if "$1" size +# aligned (optional) is supported _require_odirect() { if [ $FSTYP = "ext4" ] || [ $FSTYP = "f2fs" ] ; then
common/rc _require_odirect() · line 2743
Quoted 9 lines
common/rc +5
-1@@ -2735,9 +2736,13 @@ _require_odirect() fi fi local testfile=$TEST_DIR/$$.direct - $XFS_IO_PROG -F -f -d -c "pwrite 0 20k" $testfile > /dev/null 2>&1 + local opt + if [ -n "$1" ];then + opt="-b $1" + fi + $XFS_IO_PROG -F -f -d -c "pwrite $opt 0 20k" $testfile > /dev/null 2>&1
Local variables that are defined by incoming parameters need to be declared at the top of the function, not inline. Also, there's a simple way to set this up at initialisation, too:
local alignment=${1:+"-b $1"}If $1 is not set, alignment will be null, otherwise it will be set to "-b $1".
common/rc _require_odirect() · line 2746
Quoted 4 lines
common/rc +1
-1if [ $? -ne 0 ]; then - _notrun "O_DIRECT is not supported" + _notrun "O_DIRECT $1 is not supported" fi
if [ -n "$alignment" ]; then
_notrun "O_DIRECT aligned to $1 bytes is not supported"
else
_notrun "O_DIRECT is not supported"
fi
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
PATCH 2/5 generic/506: call _require_quota before _qmount
In reply to Zorro LangThe g/506 fails on some filesystems (e.g. overlay) which doesn't support prjquota:
MOUNT_OPTIONS = -o prjquota
qmount failed
To avoid this failure, call _require_quota before doing real quota mount.
--- tests/generic/506 | 1 + 1 file changed, 1 insertion(+)
tests/generic/506 +1 -0
@@ -27,6 +27,7 @@ _begin_fstest shutdown auto quick metadata quota _supported_fs generic _require_scratch +_require_quota _require_scratch_shutdown _scratch_mkfs >/dev/null 2>&1
Signature · 1 line
2.31.1
Re: [PATCH 2/5] generic/506: call _require_quota before _qmount
Quoted 10 lines · Zorro Lang
The g/506 fails on some filesystems (e.g. overlay) which doesn't support prjquota:
MOUNT_OPTIONS= -o prjquota
qmount failedTo avoid this failure, call
_require_quotabefore doing real quota mount.Signed-off-by Zorro Lang <[email protected]>
Looks fine.
Signature · 2 lines
Dave Chinner [email protected]
PATCH 3/5 generic/591: remove redundant output from golden image
In reply to Zorro LangIn generic/591.out expects below output:
concurrent reader with O_DIRECT
concurrent reader with O_DIRECT <=== ???
concurrent reader without O_DIRECT
concurrent reader without O_DIRECT <=== ???
sequential reader with O_DIRECT
sequential reader without O_DIRECT
The lines marked "???" are unbelievable, due to the src/splice-test.c only calls printf to output that message once in main function. So Why splice-test prints that message twice sometimes? It seems related with the "-r" option, due to the test lines without "-r" option only print one line each time running.
A stanger thing is this "double output" issue only can be triggered by running g/591, can't reproduce it by running splice-test manually.
By checking the code of splice-test.c, I found a "fork()" in it, and it'll be called if the '-r' option is specified. So I suspect the redundant output come from the child process. By the help of strace tool, I got:
10554 execve("/root/git/xfstests/src/splice-test", ["/root/git/xfstests/src/splice-te"..., "-r", "/mnt/test/a"], 0x7ffcabc2c0a8 /* 202 vars */) = 0
...
10554 clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f937f5d5a10) = 10555
...
10555 read(4, "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"..., 512) = 512
10555 write(1, "concurrent reader with O_DIRECT\n", 32) = 32
10555 exit_group(0) = ?
10555 +++ exited with 0 +++
10554 <... wait4 resumed>NULL, 0, NULL) = 10555
10554 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=10555, si_uid=0, si_status=0, si_utime=0, si_stime=1} ---
10554 unlink("/mnt/test/a") = 0
10554 write(1, "concurrent reader with O_DIRECT\n", 32) = 32
10554 exit_group(0) = ?
10554 +++ exited with 0 +++
We can see the "concurrent reader with O_DIRECT\n" be printed by parent process 10554 and child process 10555 separately.
As the output from child is unexpected and unstable. So I think we shouldn't use it as golden image. So I add a "fflush(stdout)" to clear the output buffer of parent process before forking a child. Then correct the generic/591.out.
--- src/splice-test.c | 1 + tests/generic/591.out | 2 -- 2 files changed, 1 insertion(+), 2 deletions(-)
src/splice-test.c +1 -0
@@ -143,6 +143,7 @@ int main(int argc, char *argv[])printf ("%s reader %s O_DIRECT\n" , do_splice == do_splice1 ?"sequential" :"concurrent" , (open_flags &O_DIRECT ) ?"with" :"without" ); +fflush (stdout); buffer =memalign (SECTOR_SIZE ,BUFFER_SIZE );if (buffer ==NULL )
tests/generic/591.out +0 -2
@@ -1,7 +1,5 @@ QA output created by 591 concurrent reader with O_DIRECT -concurrent reader with O_DIRECT -concurrent reader without O_DIRECT concurrent reader without O_DIRECT sequential reader with O_DIRECT sequential reader without O_DIRECT
Signature · 1 line
2.31.1
Re: [PATCH 3/5] generic/591: remove redundant output from golden image
Quoted 8 lines · Zorro Lang
In generic/591.out expects below output:
concurrent reader withO_DIRECT
concurrent reader withO_DIRECT<=== ???
concurrent reader withoutO_DIRECT
concurrent reader withoutO_DIRECT<=== ???
sequential reader withO_DIRECT
sequential reader withoutO_DIRECT
Quoted 8 lines
The lines marked "???" are unbelievable, due to the
src/splice-test.conly calls printf to output that message once in main function. So Why splice-test prints that message twice sometimes? It seems related with the "-r" option, due to the test lines without "-r" option only print one line each time running.A stanger thing is this "double output" issue only can be triggered by running g/591, can't reproduce it by running splice-test manually.
Huh. I wonder....
src/splice-test.c int main(int argc, char *argv[]) · line 146
Quoted 9 lines
src/splice-test.c +1
-0@@ -143,6 +143,7 @@ int main(int argc, char *argv[])printf ("%s reader %s O_DIRECT\n" , do_splice == do_splice1 ?"sequential" :"concurrent" , (open_flags &O_DIRECT ) ?"with" :"without" ); +fflush (stdout);
Yeah, ok, stdout output is usually line buffered. That printf() statement has a "\n" on the end of it, so it should be flushing immediately, and that's what you are seeing when it is run from the command line.
Hmmmm. I wonder if the redirect to an output file (or pipe) is changing the buffering mode because stdout no longer points to a tty?
# src/xfstests-dev/src/splice-test -r /mnt/test/a concurrent reader with O_DIRECT # src/xfstests-dev/src/splice-test -r /mnt/test/a | less concurrent reader with O_DIRECT concurrent reader with O_DIRECT #
Yup.
# man setbuf .... Normally all files are block buffered. If a stream refers to a terminal (as stdout normally does), it is line buffered. The standard error stream stderr is always unbuffered by default.
Yeah, so the stdout redirection that fstests does is exactly what is changing the behaviour.
Ok, so the correct way to fix this is to add:
setlinebuf(stdout);
before any printf() output to ensure that it is correctly line buffered no matter what the output redirection does with stdout.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 3/5] generic/591: remove redundant output from golden image
Quoted 58 lines · Dave Chinner
Quoted 8 lines · Zorro Lang
In generic/591.out expects below output:
concurrent reader withO_DIRECT
concurrent reader withO_DIRECT<=== ???
concurrent reader withoutO_DIRECT
concurrent reader withoutO_DIRECT<=== ???
sequential reader withO_DIRECT
sequential reader withoutO_DIRECTQuoted 8 lines
The lines marked "???" are unbelievable, due to the
src/splice-test.conly calls printf to output that message once in main function. So Why splice-test prints that message twice sometimes? It seems related with the "-r" option, due to the test lines without "-r" option only print one line each time running.A stanger thing is this "double output" issue only can be triggered by running g/591, can't reproduce it by running splice-test manually.
Huh. I wonder....
src/splice-test.c int main(int argc, char *argv[]) · line 146
Quoted 9 lines
src/splice-test.c +1
-0@@ -143,6 +143,7 @@ int main(int argc, char *argv[])printf ("%s reader %s O_DIRECT\n" , do_splice == do_splice1 ?"sequential" :"concurrent" , (open_flags &O_DIRECT ) ?"with" :"without" ); +fflush (stdout);Yeah, ok, stdout output is usually line buffered. That
printf()statement has a "\n" on the end of it, so it should be flushing immediately, and that's what you are seeing when it is run from the command line.Hmmmm. I wonder if the redirect to an output file (or pipe) is changing the buffering mode because stdout no longer points to a tty?
# src/xfstests-dev/src/splice-test -r /mnt/test/a concurrent reader with O_DIRECT # src/xfstests-dev/src/splice-test -r /mnt/test/a | less concurrent reader with O_DIRECT concurrent reader with O_DIRECT #Yup.
# man setbuf .... Normally all files are block buffered. If a stream refers to a terminal (as stdout normally does), it is line buffered. The standard error stream stderr is always unbuffered by default.Yeah, so the stdout redirection that fstests does is exactly what is changing the behaviour.
Quoted 6 lines
Ok, so the correct way to fix this is to add:
setlinebuf(stdout);before any
printf()output to ensure that it is correctly line buffered no matter what the output redirection does with stdout.
Wow, that's it! You reveal the root cause of this issue :)
Thanks,
Zorro
Quoted 8 lines
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
PATCH 4/5 generic/591: use proper sector size
In reply to Zorro LangThe generic/591 fails if the sector size of TEST_DEV isn't 512:
splice-test: write: /mnt/test/a: Invalid argument
To fix this issue, this patch help src/splice-test.c to get a specify sector size from the test case. Then let g/591 give it a proper sector size which dio aligned.
--- src/dmiperf | Bin 0 -> 31288 bytes src/splice-test.c | 37 +++++++++++++++++++++++++------------ src/t_locks_execve | Bin 0 -> 25904 bytes tests/generic/591 | 6 ++++-- 4 files changed, 29 insertions(+), 14 deletions(-) create mode 100755 src/dmiperf create mode 100755 src/t_locks_execve
src/dmiperf new file binary +0 -0
src/splice-test.c +25 -12
@@ -19,19 +19,20 @@#include <errno.h> #include <malloc.h> -#define SECTOR_SIZE 512 -#define BUFFER_SIZE (150 * SECTOR_SIZE) +unsigned int sector_size; +unsigned int buffer_size;void read_from_pipe (int fd,const char *filename,size_t size) { -char buffer[SECTOR_SIZE ]; +char *buffer;size_t sz;ssize_t ret; + buffer =malloc (buffer_size);while (size) { sz = size; -if (sz >sizeof buffer) - sz =sizeof buffer; +if (sz > buffer_size) + sz = buffer_size; ret =read (fd, buffer, sz);if (ret <0 )err (1 ,"read: %s" , filename);
@@ -41,6 +42,7 @@ void read_from_pipe(int fd, const char *filename, size_t size) } size -= sz; } +free (buffer); }void do_splice1 (int fd,const char *filename,size_t size)
@@ -108,7 +110,7 @@ void do_splice2(int fd, const char *filename, size_t size)void usage (const char *argv0) { -fprintf (stderr,"USAGE: %s [-rd] {filename}\n" ,basename (argv0)); +fprintf (stderr,"USAGE: %s [-rd] [-s sectorsize] {filename}\n" ,basename (argv0));exit (2 ); }
@@ -120,11 +122,22 @@ int main(int argc, char *argv[])int opt, open_flags, fd;ssize_t ret; +/* +* init default sector_size and buffer_size, might be changed if the -s +* option is specified +*/ + sector_size =512 ; + buffer_size =150 * sector_size; + do_splice = do_splice1; open_flags =O_CREAT |O_TRUNC |O_RDWR |O_DIRECT ; -while ((opt =getopt (argc, argv,"rd" )) !=-1 ) { +while ((opt =getopt (argc, argv,"rds:" )) !=-1 ) {switch (opt) { +case 's' : + sector_size =strtol (optarg,NULL ,0 ); + buffer_size =150 * sector_size; +break ;case 'r' : do_splice = do_splice2;break ;
@@ -145,7 +158,7 @@ int main(int argc, char *argv[]) (open_flags & O_DIRECT) ? "with" : "without");fflush (stdout); - buffer =memalign (SECTOR_SIZE ,BUFFER_SIZE ); + buffer =memalign (sector_size, buffer_size);if (buffer ==NULL )err (1 ,"memalign" );
@@ -153,11 +166,11 @@ int main(int argc, char *argv[])if (fd ==-1 )err (1 ,"open: %s" , filename); -memset (buffer,'x' ,BUFFER_SIZE ); - ret =write (fd, buffer,BUFFER_SIZE ); +memset (buffer,'x' , buffer_size); + ret =write (fd, buffer, buffer_size);if (ret <0 )err (1 ,"write: %s" , filename); -if (ret !=BUFFER_SIZE ) { +if (ret != buffer_size) {fprintf (stderr,"%s: short write\n" , filename);exit (1 ); }
@@ -166,7 +179,7 @@ int main(int argc, char *argv[])if (ret !=0 )err (1 ,"lseek: %s" , filename); -do_splice (fd, filename,BUFFER_SIZE ); +do_splice (fd, filename, buffer_size);if (unlink (filename) ==-1 )err (1 ,"unlink: %s" , filename);
src/t_locks_execve new file binary +0 -0
tests/generic/591 +4 -2
@@ -24,9 +24,11 @@ _require_test _require_odirect _require_test_program "splice-test" -$here/src/splice-test -r $TEST_DIR/a +diosize=`_min_dio_alignment $TEST_DEV` + +$here/src/splice-test -s $diosize -r $TEST_DIR/a $here/src/splice-test -rd $TEST_DIR/a -$here/src/splice-test $TEST_DIR/a +$here/src/splice-test -s $diosize $TEST_DIR/a $here/src/splice-test -d $TEST_DIR/a # success, all done
Signature · 1 line
2.31.1
Re: [PATCH 4/5] generic/591: use proper sector size
Quoted 8 lines · Zorro Lang
The generic/591 fails if the sector size of
TEST_DEVisn't 512:splice-test: write:
/mnt/test/a: Invalid argumentTo fix this issue, this patch help
src/splice-test.cto get a specify sector size from the test case. Then let g/591 give it a proper sector size which dio aligned.
Quoted 6 lines
Signed-off-by Zorro Lang <[email protected]>--- src/dmiperf | Bin 0 -> 31288 bytes src/splice-test.c | 37 +++++++++++++++++++++++++------------ src/t_locks_execve | Bin 0 -> 25904 bytes tests/generic/591 | 6 ++++--
I don't think this commit should include built binaries....
-Dave
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 4/5] generic/591: use proper sector size
Quoted 9 lines · Dave Chinner
Quoted 8 lines · Zorro Lang
The generic/591 fails if the sector size of
TEST_DEVisn't 512:splice-test: write:
/mnt/test/a: Invalid argumentTo fix this issue, this patch help
src/splice-test.cto get a specify sector size from the test case. Then let g/591 give it a proper sector size which dio aligned.
Quoted 8 lines
Quoted 6 lines
Signed-off-by Zorro Lang <[email protected]>--- src/dmiperf | Bin 0 -> 31288 bytes src/splice-test.c | 37 +++++++++++++++++++++++++------------ src/t_locks_execve | Bin 0 -> 25904 bytes tests/generic/591 | 6 ++++--I don't think this commit should include built binaries....
That's a ... mistake, sorry, I'll resend.
Thanks,
Zorro
Quoted 6 lines
-Dave
Signature · 2 lines
Dave Chinner [email protected]
PATCH 5/5 generic/623: add overlay into the blacklist
In reply to Zorro LangThe _require_scratch_shutdown can't help this test case, except use _scratch_shutdown or _scratch_shutdown_handle with it. But this test case does 'shutdown' on $SCRATCH_MNT/file directly. It's not suitable for overlay.
--- tests/generic/623 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/generic/623 +1 -1
@@ -11,7 +11,7 @@ _begin_fstest auto quick shutdown . ./common/filter -_supported_fs generic +_supported_fs ^overlay _fixed_by_kernel_commit e4826691cc7e \ "xfs: restore shutdown check in mapped write fault path"
Signature · 1 line
2.31.1
Re: [PATCH 5/5] generic/623: add overlay into the blacklist
Quoted 6 lines · Zorro Lang
The
_require_scratch_shutdowncan't help this test case, except use_scratch_shutdownor_scratch_shutdown_handlewith it. But this test case does 'shutdown' on $SCRATCH_MNT/file directly. It's not suitable for overlay.
This is not about testing overlayfs.
It is about testing FS under overlayfs which can detect bugs in FS that are otherwise hard to trigger.
mmap is an especially odd case of overlayfs so I rather not loose this test coverage. Please do not apply this patch I will send a fix to the test.
Thanks,
Amir.
Quoted 21 lines
Signed-off-by Zorro Lang <[email protected]>--- tests/generic/623 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/623 +1
-2@@ -11,7 +11,7 @@ _begin_fstest auto quick shutdown . ./common/filter -_supported_fs generic +_supported_fs ^overlay _fixed_by_kernel_commit e4826691cc7e \ "xfs: restore shutdown check in mapped write fault path" --2.31.1
Re: [PATCH 5/5] generic/623: add overlay into the blacklist
Quoted 8 lines · Amir Goldstein
Quoted 6 lines · Zorro Lang
The
_require_scratch_shutdowncan't help this test case, except use_scratch_shutdownor_scratch_shutdown_handlewith it. But this test case does 'shutdown' on $SCRATCH_MNT/file directly. It's not suitable for overlay.
Quoted 5 lines
This is not about testing overlayfs.
It is about testing FS under overlayfs which can detect bugs in FS that are otherwise hard to trigger.
mmap is an especially odd case of overlayfs so I rather not loose this test coverage. Please do not apply this patch I will send a fix to the test.
Thanks, if you think it's worth keeping for overlay, I'll drop this patch.
If we change the code as:
if [ $FSTYP = "overlay" ];then
file=$OVL_BASE_SCRATCH_MNT/file
fi
It's actually not testing overlay at all. We might need all testing operations run on overlay, then shutdown the $OVL_BASE_SCRATCH_MNT. But it looks not simple to separate the 'shutdown' from the:
$XFS_IO_PROG -x -c "mmap 0 4k" -c "mwrite 0 4k" -c shutdown -c fsync \
-c "mwrite 0 4k" $file | _filter_xfs_ioSo what's your plan?
Thanks,
Zorro
Quoted 26 lines
Thanks,
Amir.Quoted 21 lines
Signed-off-by Zorro Lang <[email protected]>--- tests/generic/623 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/generic/623 +1
-2@@ -11,7 +11,7 @@ _begin_fstest auto quick shutdown . ./common/filter -_supported_fs generic +_supported_fs ^overlay _fixed_by_kernel_commit e4826691cc7e \ "xfs: restore shutdown check in mapped write fault path" --2.31.1