Optimisations and ENOSPC tests
cover Optimisations and ENOSPC tests
The first patch reduces the number of forks required by some qa tests by using shell builtins. Tests run faster on UML if they fork less.
The other two patches introduce tests that trip over spurious ENOSPC errors. A followup patch series will fix these test failures.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 1/3 Reduce the number of processes forked
One of the big cpu time consumers when running xfsqa on UML is forking of new processes. when looping lots of times, using 'expr' to calculate the loop counter increment means we fork at least once every loop. using shell builtins means that we don't fork and many tests run substantially faster.
Some tests are even runnable with this modification. e.g. 110 went from taking 4500s to run down to 9s with the loop iterators changed to avoid forking.
--- 007 | 2 +- 028 | 2 +- 031 | 2 +- 047 | 2 +- 053 | 4 ++-- 064 | 8 ++++---- 065 | 8 ++++---- 068 | 6 +++--- 071 | 2 +- 089 | 2 +- 109 | 8 ++++---- 110 | 6 +++--- 111 | 2 +- 113 | 2 +- 114 | 4 ++-- 117 | 2 +- 119 | 2 +- 136 | 14 +++++++------- 137 | 4 ++-- 138 | 4 ++-- 139 | 4 ++-- 140 | 4 ++-- 149 | 2 +- 165 | 8 ++++---- 179 | 4 ++-- 180 | 4 ++-- 182 | 4 ++-- common.attr | 6 +++--- common.dump | 12 ++++++------ common.rc | 10 +++++----- 30 files changed, 72 insertions(+), 72 deletions(-)
007 +1 -1
@@ -50,7 +50,7 @@ num_filenames=100 i=1 while [ $i -le $num_filenames ]; do echo "nametest.$i" >>$sourcefile - i=`expr $i + 1` + let i=$i+1 done mkdir $testdir/$seq
028 +1 -1
@@ -44,7 +44,7 @@ while [ $i -lt 5 ]; do fi rm $dump_file sleep 2 - i=`expr $i + 1` + let i=$i+1 done echo "middate = $middate" >>$seq.full
031 +1 -1
@@ -71,7 +71,7 @@ EOF while [ $count -lt $total ] do - count=`expr $count + 1` + let count=$count+1 cat >>$tmp.proto <<EOF ${count}_of_${total}_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ---755 3 1 /bin/true EOF
047 +1 -1
@@ -44,7 +44,7 @@ while [ $i -lt 5 ]; do fi rm $dump_file sleep 2 - i=`expr $i + 1` + let i=$i+1 done echo "middate = $middate" >>$seq.full
053 +2 -2
@@ -54,7 +54,7 @@ for acl in $acls do _do "touch $test.$i" _do "chacl $acl $test.$i" - i=`expr $i + 1` + let i=$i+1 done list_acls()
@@ -63,7 +63,7 @@ list_acls() for acl in $acls do chacl -l $test.$i | _acl_filter_id | sed -e "s!$SCRATCH_MNT!\$SCRATCH_MNT!" - i=`expr $i + 1` + let i=$i+1 done }
064 +4 -4
@@ -58,7 +58,7 @@ while [ $i -le 9 ]; do date >>$seq.full find $SCRATCH_MNT -exec $here/src/lstat64 {} \; | sed 's/(00.*)//' >$tmp.dates.$i if [ $i -gt 0 ]; then - level_1=`expr $i - 1` + let level_1=$i-1 diff -c $tmp.dates.$level_1 $tmp.dates.$i >>$seq.full else cat $tmp.dates.$i >>$seq.full
@@ -66,7 +66,7 @@ while [ $i -le 9 ]; do dump_file=$tmp.df.level$i _do_dump_file -l $i - i=`expr $i + 1` + let i=$i+1 done echo "Listing of what files we start with:"
@@ -79,7 +79,7 @@ while [ $i -le 9 ]; do echo "restoring from df.level$i" dump_file=$tmp.df.level$i _do_restore_toc - i=`expr $i + 1` + let i=$i+1 done echo "Do the cumulative restores"
@@ -91,7 +91,7 @@ while [ $i -le 9 ]; do _do_restore_file_cum -l $i echo "ls -l restore_dir" ls -lR $restore_dir | _ls_size_filter | _check_quota_file - i=`expr $i + 1` + let i=$i+1 done # success, all done
065 +4 -4
@@ -146,7 +146,7 @@ while [ $i -le $num_dumps ]; do dump_file=$tmp.df.level$i _do_dump_file -l $i - i=`expr $i + 1` + let i=$i+1 done echo "Look at what files are contained in the inc. dump"
@@ -156,7 +156,7 @@ while [ $i -le $num_dumps ]; do echo "restoring from df.level$i" dump_file=$tmp.df.level$i _do_restore_toc - i=`expr $i + 1` + let i=$i+1 done echo "Do the cumulative restores"
@@ -168,7 +168,7 @@ while [ $i -le $num_dumps ]; do _do_restore_file_cum -l $i echo "list restore_dir" _list_dir $restore_dir | _check_quota_file | tee $tmp.restorals.$i - i=`expr $i + 1` + let i=$i+1 done echo ""
@@ -178,7 +178,7 @@ while [ $i -le $num_dumps ]; do echo "Comparing ls of FS with restored FS at level $i" diff -s $tmp.ls.$i $tmp.restorals.$i | sed "s#$tmp#TMP#g" echo "" - i=`expr $i + 1` + let i=$i+1 done
068 +3 -3
@@ -75,7 +75,7 @@ touch $tmp.running } & i=0 -ITERATIONS=`expr $ITERATIONS - 1` +let ITERATIONS=$ITERATIONS-1 echo | tee -a $seq.full while [ $i -le $ITERATIONS ]
@@ -94,7 +94,7 @@ do sleep 2 echo | tee -a $seq.full - i=`expr $i + 1` + let i=$i+1 done # stop fsstress iterations
@@ -105,4 +105,4 @@ wait _check_scratch_fs -exit 1 +exit 1
071 +1 -1
@@ -137,7 +137,7 @@ do echo === Iterating, `expr $upperbound - $count` remains echo echo - count=`expr $count + 1` + let count=$count+1 done # success, all done
089 +1 -1
@@ -30,7 +30,7 @@ addentries() while [ $count -gt 0 ]; do touch `printf $pattern $count` - count=`expr $count - 1` + let count=$count-1 done }
109 +4 -4
@@ -38,7 +38,7 @@ populate() while [ $i -le $files -a "X$faststart" = "X" ]; do file=$SCRATCH_MNT/f$i xfs_io -f -d -c 'pwrite -b 64k 0 64k' $file >/dev/null - i=`expr $i + 1` + let i=$i+1 done # remove every second file, freeing up lots of space
@@ -46,7 +46,7 @@ populate() i=1 while [ $i -le $files -a "X$faststart" = "X" ]; do rm $SCRATCH_MNT/f$i - i=`expr $i + 2` + let i=$i+2 done echo "flushing changes via umount/mount."
@@ -67,10 +67,10 @@ allocate() xfs_io -f -c 'pwrite -b 64k 0 16m' $file \ >/dev/null 2>&1 rm $file - j=`expr $j + 1` + let j=$j+1 done } & - i=`expr $i + 1` + let i=$i+1 done wait
110 +3 -3
@@ -52,8 +52,8 @@ E=10030600 while [ $I -le $E ] do echo > $SCRATCH_MNT/test/${STR1}${STR2}${STR3}${I} - I=`expr $I + 1` - [ `expr $I % 1000` -eq 0 ] && echo "Created $I/$E" + let I=$I+1 + [ $[$I % 1000] -eq 0 ] && echo "Created $I/$E" done sync
@@ -63,7 +63,7 @@ E=10030599 while [ $I -le $E ] do rm $SCRATCH_MNT/test/${STR1}${STR2}${STR3}${I} & - I=`expr $I + 1` + let I=$I+1 done _check_scratch_fs
111 +1 -1
@@ -39,7 +39,7 @@ I=0 while [ $I -lt 1000 ] do cp src/itrash.c $SCRATCH_MNT/${I} - I=`expr $I + 1` + let I=$I+1 done umount $SCRATCH_DEV
113 +1 -1
@@ -41,7 +41,7 @@ _do_test() [ $__proc -gt 1 ] && _param="-t $__proc $_param" while [ $__proc -gt 1 ]; do _files="$_files $testdir/aiostress.$$.$_n.$__proc" - __proc=`expr $__proc - 1` + let __proc=$__proc-1 done rm -f $_files
114 +2 -2
@@ -135,8 +135,8 @@ _test_hardlink() paths="$d/l1 $d/l2 $d/l3 $d2/l4 $d2/l5 $d2/l6" i=0 for x in $paths; do - i=`expr $i + 1` - j=`expr $i % 2` + let i=$i+1 + let j=$i%2 if [ $j -eq 0 ]; then echo "rm'ing $x" rm $x
117 +1 -1
@@ -71,7 +71,7 @@ while [ $i -lt $ITERATIONS ]; do -s $seed \ -S -p 1 -n 1000 >>$seq.full 2>&1 - i=`expr $i + 1` + let i=$i+1 done cd /
119 +1 -1
@@ -54,7 +54,7 @@ while [ $i -lt $max ]; do xfs_freeze -f $SCRATCH_MNT xfs_freeze -u $SCRATCH_MNT echo -n . - i=`expr $i + 1` + let i=$i+1 done echo "done"
136 +7 -7
@@ -61,7 +61,7 @@ add_eas() i=$start while [ $i -le $end ]; do attr -s name.$i -V value $file >/dev/null - i=`expr $i + 1` + let i=$i+1 done }
@@ -73,7 +73,7 @@ rm_eas() i=$start while [ $i -le $end ]; do attr -r name.$i $file >/dev/null - i=`expr $i + 1` + let i=$i+1 done }
@@ -171,7 +171,7 @@ _test_add_extents() while [ $j -le 30 ]; do do_extents $j _print_inode - j=`expr $j + 2` + let j=$j+2 done #scale down
@@ -179,7 +179,7 @@ _test_add_extents() while [ $j -ge 1 ]; do do_extents $j _print_inode - j=`expr $j - 2` + let j=$j-2 done #build up
@@ -187,7 +187,7 @@ _test_add_extents() while [ $j -le 30 ]; do do_extents $j _print_inode - j=`expr $j + 2` + let j=$j+2 done }
@@ -211,7 +211,7 @@ _test_extents_eas() _print_inode _print_inode_u > $tmp.u1 for j in `seq 1 $EAs_inc $EAs_max`; do - k=`expr $j + $EAs_inc - 1` + let k=$k+$EAs_inc-1 add_eas $j $k done # should have same extents
@@ -256,7 +256,7 @@ _test_eas_extents() EAs_inc=5 for j in `seq 1 $EAs_inc $EAs_max`; do - k=`expr $j + $EAs_inc - 1` + let k=$k+$EAs_inc-1 add_eas $j $k echo "--- EAs: $j ---"
137 +2 -2
@@ -47,7 +47,7 @@ do echo error creating/writing file $file exit fi - i=`expr $i + 1` + let i=$i+1 done # give the system a chance to write something out
@@ -90,7 +90,7 @@ do rm -f $file fi fi - i=`expr $i + 1` + let i=$i+1 done status=0
138 +2 -2
@@ -53,7 +53,7 @@ do echo error truncating file $file exit fi - i=`expr $i + 1` + let i=$i+1 done # give the system a chance to write something out
@@ -96,7 +96,7 @@ do rm -f $file fi fi - i=`expr $i + 1` + let i=$i+1 done status=0
139 +2 -2
@@ -53,7 +53,7 @@ do echo error truncating file $file exit fi - i=`expr $i + 1` + let i=$i+1 done # give the system a chance to write something out
@@ -96,7 +96,7 @@ do rm -f $file fi fi - i=`expr $i + 1` + let i=$i+1 done status=0
140 +2 -2
@@ -53,7 +53,7 @@ do echo error truncating file $file exit fi - i=`expr $i + 1` + let i=$i+1 done # give the system a chance to write something out
@@ -93,7 +93,7 @@ do rm -f $file fi fi - i=`expr $i + 1` + let i=$i+1 done status=0
149 +1 -1
@@ -74,7 +74,7 @@ EOF while [ $count -lt $total ] do - count=`expr $count + 1` + let count=$count+1 cat >>$tmp.proto <<EOF ${count}_of_${total}_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ---755 3 1 /bin/true EOF
165 +4 -4
@@ -82,8 +82,8 @@ do $XFS_IO_PROG -c "unresvsp $offset $length" -c "bmap -vp" $testfile | _filter_bmap - off=`expr $off + $len` # skip over 1 - off=`expr $off + $len` + let off=$off+$len # skip over 1 + let off=$off+$len done off=0
@@ -111,8 +111,8 @@ do #$XFS_IO_PROG -r -c "pread -v -b $bufsize $offset $length" $testfile #sleep 5 - off=`expr $off + $len` # skip over 1 - off=`expr $off + $len` + let off=$off+$len # skip over 1 + let off=$off+$len done wait
179 +2 -2
@@ -63,7 +63,7 @@ _check_files() else echo file $file missing - fsync failed fi - i=`expr $i + 1` + let i=$i+1 done }
@@ -78,7 +78,7 @@ do echo error creating/writing file $file exit fi - i=`expr $i + 1` + let i=$i+1 done # shutdown immediately after, then remount and test
180 +2 -2
@@ -68,7 +68,7 @@ _check_files() else echo file $file missing - sync failed fi - i=`expr $i + 1` + let i=$i+1 done }
@@ -83,7 +83,7 @@ do echo error creating/writing file $file exit fi - i=`expr $i + 1` + let i=$i+1 done # sync, then shutdown immediately after, then remount and test
182 +2 -2
@@ -63,7 +63,7 @@ _check_files() else echo file $file missing - sync failed fi - i=`expr $i + 1` + let i=$i+1 done }
@@ -78,7 +78,7 @@ do echo error creating/writing file $file exit fi - i=`expr $i + 1` + let i=$i+1 done # sync, then shutdown immediately after, then remount and test
common.attr +3 -3
@@ -77,11 +77,11 @@ _acl_list() # _create_n_aces() { - n=`expr $1 - 4` + let n=$1-4 acl='u::rwx,g::rwx,o::rwx,m::rwx' # 4 ace acl start while [ $n -ne 0 ]; do - acl="$acl,u:$n:rwx" - n=`expr $n - 1` + acl="$acl,u:$n:rwx" + let n=$n-1 done echo $acl }
common.dump +6 -6
@@ -97,7 +97,7 @@ _check_onl() else sleep 1 fi - i=`expr $i + 1` + let i=$i+1 done
@@ -135,7 +135,7 @@ _wait_tape() else sleep 1 fi - i=`expr $i + 1` + let i=$i+1 done }
@@ -681,7 +681,7 @@ _create_hardlinks() _hardlink=$_fname$_suffix echo "creating hardlink $_hardlink to $_fname" ln $_fname $_hardlink - _j=`expr $_j + 1` + let _j=$_j+1 done }
@@ -697,7 +697,7 @@ _create_hardset() _i=1 while [ $_i -le $_numsets ]; do _create_hardlinks file$_i 5 - _i=`expr $_i + 1` + let _i=$_i+1 done }
@@ -973,7 +973,7 @@ _do_dump_multi_file() while [ $i -lt $multi ] do multi_args="$multi_args -f $dump_file.$i -M $media_label.$i" - i=`expr $i + 1` + let i=$i+1 done echo "Dumping to files..."
@@ -1093,7 +1093,7 @@ _do_restore_multi_file() while [ $i -lt $multi ] do multi_args="$multi_args -f $dump_file.$i" - i=`expr $i + 1` + let i=$i+1 done echo "Restoring from file..."
common.rc +5 -5
@@ -152,7 +152,7 @@ _mount_ops_filter() params="$*" #get mount point to handle dmapi mtpt option correctly - last_index=`expr $# - 1` + let last_index=$#-1 [ $last_index -gt 0 ] && shift $last_index FS_ESCAPED=$1
@@ -1114,11 +1114,11 @@ _nfiles() while [ $f -lt $1 ] do file=f$f - touch $file + echo > $file if [ $size -gt 0 ]; then dd if=/dev/zero of=$file bs=1024 count=$size fi - f=`expr $f + 1` + let f=$f+1 done }
@@ -1132,7 +1132,7 @@ _descend() _nfiles $files # files for this dir [ $depth -eq 0 ] && return - deep=`expr $depth - 1` # go 1 down + let deep=$depth-1 # go 1 down [ $verbose = true ] && echo "descending, depth from leaves = $deep"
@@ -1140,7 +1140,7 @@ _descend() while [ $d -lt $dirs ] do _descend d$d $deep & - d=`expr $d + 1` + let d=$d+1 wait done }
Signature · 6 lines
1.6.2 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 1/3] [XFSQA] Reduce the number of processes forked
Quoted 9 lines · Dave Chinner
One of the big cpu time consumers when running xfsqa on UML is forking of new processes. when looping lots of times, using 'expr' to calculate the loop counter increment means we fork at least once every loop. using shell builtins means that we don't fork and many tests run substantially faster.
Some tests are even runnable with this modification. e.g. 110 went from taking 4500s to run down to 9s with the loop iterators changed to avoid forking.
Looks good.
Care to see if any tests should be added to the quick group after this? The above 110 sounds like a candidate for that.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 2/3 Add simple delayed allocation ENOSPC test.
In reply to Dave ChinnerUsing a small (100MB) filesystem and writing lots of single block files can result in spurious ENOSPCs being reported. Reproduce this test case so we can confirm that it gets fixed.
--- 203 | 41 +++++++++++++++++++++++++++++++++++++++++ 203.out | 2 ++ group | 1 + 3 files changed, 44 insertions(+), 0 deletions(-) create mode 100755 203 create mode 100644 203.out
203 new file +41 -0
@@ -0,0 +1,41 @@ +#! /bin/sh +# FS QA Test No. 203 +# +# Test out ENOSPC flushiung on small filesystems. +# +#----------------------------------------------------------------------- +# Copyright (c) 2009 Christoph Hellwig. +#----------------------------------------------------------------------- +# +# creator [email protected] + +seq=`basename $0` +echo "QA output created by $seq" + +here=`pwd` +tmp=/tmp/$$ +status=1 # failure is the default! + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +# real QA test starts here +_supported_fs xfs +_supported_os Linux + +_require_scratch + +_scratch_mkfs_xfs -d size=104m >/dev/null +_scratch_mount + +for i in `seq 1 22500`; do + echo -n > $SCRATCH_MNT/$i + echo XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > $SCRATCH_MNT/$i +done + +# success, all done +echo "*** done" +rm -f $seq.full +status=0
203.out new file +2 -0
@@ -0,0 +1,2 @@ +QA output created by 203 +*** done
group +1 -0
@@ -307,3 +307,4 @@ atime 200 mount auto quick 201 metadata auto quick 202 repair auto quick +203 metadata rw auto
Signature · 6 lines
1.6.2 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 2/3] [XFSQA] Add simple delayed allocation ENOSPC test.
Quoted 4 lines · Dave Chinner
Using a small (100MB) filesystem and writing lots of single block files can result in spurious ENOSPCs being reported. Reproduce this test case so we can confirm that it gets fixed.
Looks good.
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
PATCH 3/3 Test writing to ENOSPC
In reply to Dave ChinnerUse larger files and different writing styles to fill a 100MB filesystem to being full. In each case we should get very close to the filesystem being full before getting ENOSPC. THis tests different types of ENOSPC failures to test 203 and requires more changes to pass.
--- 015 | 11 --------- 204 | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 204.out | 4 +++ common.filter | 11 +++++++++ group | 1 + 5 files changed, 82 insertions(+), 11 deletions(-) mode change 100644 => 100755 141 mode change 100644 => 100755 164 mode change 100644 => 100755 165 mode change 100644 => 100755 166 mode change 100644 => 100755 167 mode change 100644 => 100755 170 mode change 100644 => 100755 171 mode change 100644 => 100755 172 mode change 100644 => 100755 173 mode change 100644 => 100755 174 mode change 100644 => 100755 179 mode change 100644 => 100755 180 mode change 100644 => 100755 182 mode change 100644 => 100755 183 mode change 100644 => 100755 184 mode change 100644 => 100755 185 mode change 100644 => 100755 188 mode change 100644 => 100755 189 mode change 100644 => 100755 194 mode change 100644 => 100755 195 mode change 100644 => 100755 196 mode change 100644 => 100755 197 mode change 100644 => 100755 199 mode change 100644 => 100755 200 create mode 100755 204 create mode 100644 204.out
015 +0 -11
@@ -33,17 +33,6 @@ _free() _df_dir $SCRATCH_MNT | $AWK_PROG '{ print $5 }' } -_filter_dd() -{ - $AWK_PROG ' - /records in/ { next } - /records out/ { next } - /No space left on device/ { print " !!! disk full (expected)" - next } - { print " *** " $0 } - ' -} - # real QA test starts here _supported_fs xfs _supported_os IRIX Linux
141 +0 -0
164 +0 -0
165 +0 -0
166 +0 -0
167 +0 -0
170 +0 -0
171 +0 -0
172 +0 -0
173 +0 -0
174 +0 -0
179 +0 -0
180 +0 -0
182 +0 -0
183 +0 -0
184 +0 -0
185 +0 -0
188 +0 -0
189 +0 -0
194 +0 -0
195 +0 -0
196 +0 -0
197 +0 -0
199 +0 -0
200 +0 -0
204 new file +66 -0
66 lines not on this page · raw message
204.out new file +4 -0
@@ -0,0 +1,4 @@ +QA output created by 204 +*** one file +*** one file, a few bytes at a time +*** done
common.filter +11 -0
@@ -167,5 +167,16 @@ _filter_growfs() }' } +_filter_dd() +{ + $AWK_PROG ' + /records in/ { next } + /records out/ { next } + /No space left on device/ { print " !!! disk full (expected)" + next } + { print " *** " $0 } + ' +} + # make sure this script returns success /bin/true
group +1 -0
@@ -308,3 +308,4 @@ atime 201 metadata auto quick 202 repair auto quick 203 metadata rw auto +204 metadata rw auto
Signature · 6 lines
1.6.2 _______________________________________________ xfs mailing list [email protected] http://oss.sgi.com/mailman/listinfo/xfs
Re: [PATCH 3/3] [XFSQA] Test writing to ENOSPC
Quoted 5 lines · Dave Chinner
Use larger files and different writing styles to fill a 100MB filesystem to being full. In each case we should get very close to the filesystem being full before getting ENOSPC. THis tests different types of ENOSPC failures to test 203 and requires more changes to pass.
Looks good. But don't we actually need a trap handler to unmount the
scratch partitions if the test gets intterupted? I always added them to my tests, copying that fragment from older patches. (also applies to the previous patch)
_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs