Skip to contentSkip to conversation

xfstests: generic --large-fs support

PATCH In review Dave Chinner 3 participants 23 messages Mar 15, 2013 – Mar 19, 2013

lore
Share & tools
lore
#0 lore raw

cover xfstests: generic --large-fs support

Hi folks,

This is a repost of my current version of the large filesystem testing support for xfstests. I've been using it for more than a year, and I think the ext4 guys would like to have it available to them.

Last posted here:

http://oss.sgi.com/archives/xfs/2012-07/msg00361.html

There are changes from last time, but it's months since i've made them, so there's no fine grained changelog. :/

Cheers,

Dave.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#1 lore raw

PATCH 1/13 xfstests: filter new repair/mkfs output for CRCs

From: Dave Chinner <[email protected]>

This patch it to ensure that xfstests passes on non-crc filesystems with a CRC enabled userspace.

Filter out the mkfs/xfs_info CRC line from tests that capture the output of these commands.

Filter out new error noise from xfs_repair that occurs for xfs_repair as a result of the CRC changes.

---
 096           |    1 +
 206           |    9 ++++++---
 common.repair |    3 +++
 3 files changed, 10 insertions(+), 3 deletions(-)
096 +1 -0
index 237edad..5a2273d 100755
@@ -85,6 +85,7 @@ _mkfs_filter()
 	-e '/inode-paths/d' \
 	-e 's/\(log[ 	]*=\).*bsize/\1LOG                    bsize/' \
 	-e 's/\(realtime[ 	]*=\).*extsz/\1REALTIME               extsz/' \
+	-e '/.*crc=/d' \
 	-e 's/ *$//' \
    | grep -v parent
 }
206 +6 -3
index e8bc922..e46e6a3 100755
@@ -77,7 +77,8 @@ echo "=== mkfs.xfs ==="
 mkfs.xfs -f -bsize=4096 -dagsize=76288719b,size=3905982455b -llazy-count=0 $tmpfile \
 	| sed -e "s,^meta-data=.*isize,meta-data=FILE                   isize,g" \
 		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/" \
-		-e "s/, projid32bit=[0-9]//"
+		-e "s/, projid32bit=[0-9]//" \
+		-e "/.*crc=/d"
 
 mount -o loop $tmpfile $tmpdir || _fail "!!! failed to loopback mount"
 
@@ -85,13 +86,15 @@ mount -o loop $tmpfile $tmpdir || _fail "!!! failed to loopback mount"
 echo "=== xfs_growfs ==="
 xfs_growfs $tmpdir \
 	| sed -e "s,^meta-data=.*isize,meta-data=FILE                   isize,g" \
-		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/"
+		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/" \
+		-e "/.*crc=/d"
 
 # and double-check the new geometry
 echo "=== xfs_info ==="
 xfs_info $tmpdir \
 	| sed -e "s,^meta-data=.*isize,meta-data=FILE                   isize,g" \
-		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/"
+		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/" \
+		-e "/.*crc=/d"
 
 # _cleanup cleans up for us
 
common.repair +3 -0
index 23a1ed6..db160bb 100644
@@ -77,6 +77,9 @@ s/\s+- \d+:\d\d:\d\d:.*\n//g;
 /^agi_/ && next; # remove agi counts
 /^sb_/ && next; # remove sb counts
 /^agi unlinked/ && next; # remove agi unlinked bucket warning
+# crc enabled filesystem output
+/XFS_CORRUPTION_ERROR/ && next;
+/^bad uuid/ && next;
 	print;'
 }
 
Signature · 6 lines
1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#3 lore raw

Re: [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs

In reply to Dave Chinner

Hey Rich,

Quoted 10 lines · Dave Chinner

From: Dave Chinner <[email protected]>

This patch it to ensure that xfstests passes on non-crc filesystems with a CRC enabled userspace.

Filter out the mkfs/xfs_info CRC line from tests that capture the output of these commands.

Filter out new error noise from xfs_repair that occurs for xfs_repair as a result of the CRC changes.

IIRC this change was a temporary one so that xfstests could be run with crc enabled filesystems.

Quoted 56 lines
---
 096           |    1 +
 206           |    9 ++++++---
 common.repair |    3 +++
 3 files changed, 10 insertions(+), 3 deletions(-)
096 +1 -0
index 237edad..5a2273d 100755
@@ -85,6 +85,7 @@ _mkfs_filter()
 	-e '/inode-paths/d' \
 	-e 's/\(log[ 	]*=\).*bsize/\1LOG                    bsize/' \
 	-e 's/\(realtime[ 	]*=\).*extsz/\1REALTIME               extsz/' \
+	-e '/.*crc=/d' \
 	-e 's/ *$//' \
    | grep -v parent
 }
206 +6 -3
index e8bc922..e46e6a3 100755
@@ -77,7 +77,8 @@ echo "=== mkfs.xfs ==="
 mkfs.xfs -f -bsize=4096 -dagsize=76288719b,size=3905982455b -llazy-count=0 $tmpfile \
 	| sed -e "s,^meta-data=.*isize,meta-data=FILE                   isize,g" \
 		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/" \
-		-e "s/, projid32bit=[0-9]//"
+		-e "s/, projid32bit=[0-9]//" \
+		-e "/.*crc=/d"
 
 mount -o loop $tmpfile $tmpdir || _fail "!!! failed to loopback mount"
 
@@ -85,13 +86,15 @@ mount -o loop $tmpfile $tmpdir || _fail "!!! failed to loopback mount"
 echo "=== xfs_growfs ==="
 xfs_growfs $tmpdir \
 	| sed -e "s,^meta-data=.*isize,meta-data=FILE                   isize,g" \
-		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/"
+		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/" \
+		-e "/.*crc=/d"
 
 # and double-check the new geometry
 echo "=== xfs_info ==="
 xfs_info $tmpdir \
 	| sed -e "s,^meta-data=.*isize,meta-data=FILE                   isize,g" \
-		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/"
+		-e "s/\(^log.*blocks=\)\([0-9]*,\)/\1XXXXX,/" \
+		-e "/.*crc=/d"
 
 # _cleanup cleans up for us
 
common.repair +0 -0
index 23a1ed6..db160bb 100644

common.repair s/\s+- \d+:\d\d:\d\d:.*\n//g; · line 82

Quoted 6 lines
common.repair +3 -0
@@ -77,6 +77,9 @@ s/\s+- \d+:\d\d:\d\d:.*\n//g;
 /^agi_/ && next; # remove agi counts
 /^sb_/ && next; # remove sb counts
 /^agi unlinked/ && next; # remove agi unlinked bucket warning
+# crc enabled filesystem output
+/XFS_CORRUPTION_ERROR/ && next;
+/^bad uuid/ && next;

Doesn't this filter out xfs repair failures which should be reported?

Thanks,
Ben

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#4 lore raw

Re: [PATCH 01/13] xfstests: filter new repair/mkfs output for CRCs

Quoted 8 lines · Ben Myers

Hey Rich,

Quoted 5 lines · Dave Chinner

From: Dave Chinner <[email protected]>

This patch it to ensure that xfstests passes on non-crc filesystems with a CRC enabled userspace.

Quoted 8 lines
Quoted 5 lines

Filter out the mkfs/xfs_info CRC line from tests that capture the output of these commands.

Filter out new error noise from xfs_repair that occurs for xfs_repair as a result of the CRC changes.

IIRC this change was a temporary one so that xfstests could be run with crc enabled filesystems.

It's going to be a permanent change because we dump mkfs.xfs output into golden output files and that means the "crc=X" line will always need to be filtered out

Quoted 5 lines
Quoted 5 lines
common.repair +0 -0
index 23a1ed6..db160bb 100644
@@ -77,6 +77,9 @@ s/\s+- \d+:\d\d:\d\d:.*\n//g;
Quoted 8 lines

common.repair s/\s+- \d+:\d\d:\d\d:.*\n//g; · line 82

Quoted 6 lines
common.repair +3 -0
 /^agi_/ && next; # remove agi counts
 /^sb_/ && next; # remove sb counts
 /^agi unlinked/ && next; # remove agi unlinked bucket warning
+# crc enabled filesystem output
+/XFS_CORRUPTION_ERROR/ && next;
+/^bad uuid/ && next;

Doesn't this filter out xfs repair failures which should be reported?

No, because repair will issue errors when bad metadata is read off disk now, and as such they are brand new output introduced by the CRC patches. Over time this might get cleaned up, but at least for initial testing and debugging this noise will remain in the repair code....

Cheers,

Dave.

Signature · 7 lines
Dave Chinner
[email protected]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#5 lore raw

PATCH 2/13 xfstests: add --largefs check option

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Make it easier to check large filesystems quickly by adding a --large-fs option to check to turn on shortcuts for large scratch device filesystem testing.

Also, reject invalid command line options with a usage message.

---
 common |   63 +++++++++++++++++++++++++++++++++++++++------------------------
 1 file changed, 39 insertions(+), 24 deletions(-)
common +39 -24
index 0723224..7d13078 100644
@@ -27,6 +27,35 @@ _setenvironment()
     export MSGVERB
 }
 
+usage()
+{
+    echo "Usage: $0 [options] [testlist]"'
+
+common options
+    -v			verbose
+
+check options
+    -xfs                test XFS (default)
+    -udf                test UDF
+    -nfs                test NFS
+    -l			line mode diff
+    -xdiff		graphical mode diff
+    -udiff		show unified diff (default)
+    -n			show me, do not run tests
+    -q			quick [deprecated]
+    -T			output timestamps
+    -r			randomize test order
+    --large-fs		optimise scratch device for large filesystems
+
+testlist options
+    -g group[,group...]	include tests from these groups
+    -x group[,group...]	exclude tests from these groups
+    NNN			include test NNN
+    NNN-NNN		include test range (eg. 012-021)
+'
+	    exit 0
+}
+
 here=`pwd`
 rm -f $here/$iam.out
 _setenvironment
@@ -117,30 +146,7 @@ s/ .*//p
     in
 
 	-\? | -h | --help)	# usage
-	    echo "Usage: $0 [options] [testlist]"'
-
-common options
-    -v			verbose
-
-check options
-    -xfs                test XFS (default)
-    -udf                test UDF
-    -nfs                test NFS
-    -l			line mode diff
-    -xdiff		graphical mode diff
-    -udiff		show unified diff (default)
-    -n			show me, do not run tests
-    -q			quick [deprecated]
-    -T			output timestamps
-    -r 			randomize test order
-    
-testlist options
-    -g group[,group...]	include tests from these groups
-    -x group[,group...]	exclude tests from these groups
-    NNN			include test NNN
-    NNN-NNN		include test range (eg. 012-021)
-'
-	    exit 0
+	    usage
 	    ;;
 
 	-udf)	# -udf ... set FSTYP to udf
@@ -232,6 +238,15 @@ testlist options
 	    fi
 	    ;;
 
+	--large-fs)
+	    export USE_BIG_LOOPFS=yes
+	    xpand=false
+	    ;;
+
+	-*)
+	    usage
+	    ;;
+
 	*)
 	    start=$r
 	    end=$r
Signature · 6 lines
1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#6 lore raw

PATCH 3/13 xfstests: rename USE_BIG_LOOPFS to be more generic

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

USE_BIG_LOOPFS is really misnamed - it can be used on real devices just as easily as loop devices. It really means we are testing a large scratch device and that we should enable the special filesystem filling and checking options that enable xfstests to be run sanely on large XFS filesystems.

---
 004       |    2 +-
 015       |    2 +-
 030       |    2 +-
 031       |    2 +-
 032       |    2 +-
 033       |    4 ++--
 041       |    2 +-
 049       |    2 +-
 083       |    2 +-
 092       |    2 +-
 148       |    2 +-
 149       |    2 +-
 common    |    2 +-
 common.rc |   12 ++++++------
 setup     |    5 +++--
 15 files changed, 23 insertions(+), 22 deletions(-)
004 +1 -1
index 9f28e17..23729da 100755
@@ -64,7 +64,7 @@ _supported_os IRIX Linux
 
 _need_to_be_root
 _require_scratch
-_require_nobigloopfs
+_require_no_large_scratch_dev
 
 rm -f $seq.full
 
015 +1 -1
index 4206b93..a99f1ed 100755
@@ -52,7 +52,7 @@ _supported_fs generic
 _supported_os IRIX Linux
 
 _require_scratch
-_require_nobigloopfs
+_require_no_large_scratch_dev
 
 _scratch_mkfs_sized `expr 50 \* 1024 \* 1024` >/dev/null 2>&1 \
     || _fail "mkfs failed"
030 +1 -1
index fe57372..a8fb92c 100755
@@ -64,8 +64,8 @@ _check_ag()
 _supported_fs xfs
 _supported_os IRIX Linux
 
-_require_nobigloopfs
 _require_scratch
+_require_no_large_scratch_dev
 
 DSIZE="-dsize=100m,agcount=6"
 
031 +1 -1
index b062277..fb6f15b 100755
@@ -97,8 +97,8 @@ EOF
 _supported_fs xfs
 _supported_os IRIX Linux
 
-_require_nobigloopfs
 _require_scratch
+_require_no_large_scratch_dev
 
 # sanity test - default + one root directory entry
 # Note: must do this proto/mkfs now for later inode size calcs
032 +1 -1
index bf88492..09f08c3 100755
@@ -41,8 +41,8 @@ rm -f $seq.full
 _supported_fs xfs
 _supported_os Linux
 
-_require_nobigloopfs
 _require_scratch
+_require_no_large_scratch_dev
 
 echo "Silence is golden."
 for fs in `echo ${MKFS_PROG}.* | sed -e 's/.sbin.mkfs.//g'`
033 +2 -2
index 9651f26..68a688e 100755
@@ -76,9 +76,9 @@ _filter_bad_ids()
 # real QA test starts here
 _supported_fs xfs
 _supported_os IRIX Linux
- 
-_require_nobigloopfs
+
 _require_scratch
+_require_no_large_scratch_dev
 
 # devzero blows away 512byte blocks, so make 512byte inodes (at least)
 _scratch_mkfs_xfs | _filter_mkfs 2>$tmp.mkfs
041 +1 -1
index 2800811..28dcb33 100755
@@ -50,7 +50,7 @@ _supported_fs xfs
 _supported_os IRIX Linux
 
 _require_scratch
-_require_nobigloopfs
+_require_no_large_scratch_dev
 umount $SCRATCH_DEV 2>/dev/null
 
 _fill()
049 +1 -1
index b68821a..8593814 100755
@@ -60,9 +60,9 @@ _log()
     echo "--- $*" >> $seq.full
 }
 
-_require_nobigloopfs
 _require_nonexternal
 _require_scratch
+_require_no_large_scratch_dev
 _require_loop
 _require_ext2
 
083 +1 -1
index f5349a9..a977ffd 100755
@@ -58,7 +58,7 @@ _supported_fs generic
 _supported_os IRIX Linux
 
 _require_scratch
-_require_nobigloopfs
+_require_no_large_scratch_dev
 
 rm -f $seq.full
 
092 +1 -1
index 429fa80..02ccc71 100755
@@ -48,7 +48,7 @@ _cleanup()
 _supported_fs xfs
 _supported_os IRIX Linux
 _require_scratch
-_require_nobigloopfs
+_require_no_large_scratch_dev
 
 MOUNT_OPTIONS="$MOUNT_OPTIONS -o inode64"
 _scratch_mkfs_xfs | _filter_mkfs 2>/dev/null
148 +1 -1
index 76cbf37..7bb1722 100755
@@ -66,8 +66,8 @@ _check_ag()
 _supported_fs xfs
 _supported_os IRIX Linux
 
-_require_nobigloopfs
 _require_scratch
+_require_no_large_scratch_dev
 
 DSIZE="-dsize=100m"
 
149 +1 -1
index 5131a45..193e6d7 100755
@@ -100,8 +100,8 @@ EOF
 _supported_fs xfs
 _supported_os IRIX Linux
 
-_require_nobigloopfs
 _require_scratch
+_require_no_large_scratch_dev
 
 # sanity test - default + one root directory entry
 # Note: must do this proto/mkfs now for later inode size calcs
common +1 -1
index 7d13078..da86cd9 100644
@@ -239,7 +239,7 @@ s/ .*//p
 	    ;;
 
 	--large-fs)
-	    export USE_BIG_LOOPFS=yes
+	    export LARGE_SCRATCH_DEV=yes
 	    xpand=false
 	    ;;
 
common.rc +6 -6
index ef41cbf..edf0318 100644
@@ -310,7 +310,7 @@ _scratch_mkfs_xfs()
     cat $tmp_dir.mkfsstd
     rm -f $tmp_dir.mkfserr $tmp_dir.mkfsstd
 
-    if [ "$USE_BIG_LOOPFS" = yes ]; then
+    if [ "$LARGE_SCRATCH_DEV" = yes ]; then
         [ -z "$RETAIN_AG_BYTES" ] && RETAIN_AG_BYTES=0
         ./tools/ag-wipe -q -r $RETAIN_AG_BYTES $SCRATCH_DEV
     fi
@@ -437,7 +437,7 @@ _scratch_xfs_repair()
         SCRATCH_OPTIONS="-l$SCRATCH_LOGDEV"
     [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_RTDEV" ] && \
         SCRATCH_OPTIONS=$SCRATCH_OPTIONS" -r$SCRATCH_RTDEV"
-    [ "$USE_BIG_LOOPFS" = yes ] && SCRATCH_OPTIONS=$SCRATCH_OPTIONS" -t"
+    [ "$LARGE_SCRATCH_DEV" = yes ] && SCRATCH_OPTIONS=$SCRATCH_OPTIONS" -t"
     $XFS_REPAIR_PROG $SCRATCH_OPTIONS $* $SCRATCH_DEV
 }
 
@@ -826,9 +826,9 @@ _require_ext2()
 
 # this test requires that (large) loopback device files are not in use
 #
-_require_nobigloopfs()
+_require_no_large_scratch_dev()
 {
-    [ "$USE_BIG_LOOPFS" = yes ] && \
+    [ "$LARGE_SCRATCH_DEV" = yes ] && \
 	_notrun "Large filesystem testing in progress, skipped this test"
 }
 
@@ -1237,7 +1237,7 @@ _check_xfs_filesystem()
 
     [ "$FSTYP" != xfs ] && return 0
     testoption=""
-    [ "$USE_BIG_LOOPFS" = yes ] && testoption=-t
+    [ "$LARGE_SCRATCH_DEV" = yes ] && testoption=-t
 
     type=`_fs_type $device`
     ok=1
@@ -1276,7 +1276,7 @@ _check_xfs_filesystem()
         ok=0
     fi
     # repair doesn't scale massively at this stage, optionally skip it for now
-    [ "$USE_BIG_LOOPFS" = yes ] || \
+    [ "$LARGE_SCRATCH_DEV" = yes ] || \
     $XFS_REPAIR_PROG -n $extra_log_options $extra_rt_options $device >$tmp.repair 2>&1
     if [ $? -ne 0 ]
     then
setup +3 -2
index 62798cc..5225951 100755
@@ -23,7 +23,7 @@ fi
 
 [ "$USE_EXTERNAL"    = yes ] || USE_EXTERNAL=no
 [ "$USE_LBD_PATCH"   = yes ] || USE_LBD_PATCH=no
-[ "$USE_BIG_LOOPFS"  = yes ] || USE_BIG_LOOPFS=no
+[ "$LARGE_SCRATCH_DEV"  = yes ] || LARGE_SCRATCH_DEV=no
 [ "$USE_ATTR_SECURE" = yes ] || USE_ATTR_SECURE=no
 [ -z "$FSTYP" ] && FSTYP="xfs"
 
@@ -31,5 +31,6 @@ cat <<EOF
 TEST: DIR=$TEST_DIR DEV=$TEST_DEV rt=[$TEST_RTDEV] log=[$TEST_LOGDEV]
 TAPE: dev=[$TAPE_DEV] rmt=[$RMT_TAPE_DEV] rmtirix=[$RMT_TAPE_USER@$RMT_IRIXTAPE_DEV]
 SCRATCH: MNT=$SCRATCH_MNT DEV=$SCRATCH_DEV rt=[$SCRATCH_RTDEV] log=[$SCRATCH_LOGDEV]
-VARIABLES: external=$USE_EXTERNAL largeblk=$USE_LBD_PATCH fstyp=$FSTYP bigloopfs=$USE_BIG_LOOPFS attrsecure=$USE_ATTR_SECURE
+VARIABLES: external=$USE_EXTERNAL largeblk=$USE_LBD_PATCH fstyp=$FSTYP
+	   large_scratch_dev=$LARGE_SCRATCH_DEV attrsecure=$USE_ATTR_SECURE
 EOF
Signature · 6 lines
1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#8 lore raw

PATCH 4/13 xfstests: rename RETAIN_AG_BYTES

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Rename the $RETAIN_AG_BYTES variable to be more generic so that it reflects the fact that it is designed to retain a certain amount of extra free space above the default amount in the filesystem when doing large scratch device testing.

---
 common.rc |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
common.rc +2 -2
index edf0318..42d2ddc 100644
@@ -311,8 +311,8 @@ _scratch_mkfs_xfs()
     rm -f $tmp_dir.mkfserr $tmp_dir.mkfsstd
 
     if [ "$LARGE_SCRATCH_DEV" = yes ]; then
-        [ -z "$RETAIN_AG_BYTES" ] && RETAIN_AG_BYTES=0
-        ./tools/ag-wipe -q -r $RETAIN_AG_BYTES $SCRATCH_DEV
+        [ -z "$SCRATCH_DEV_EMPTY_SPACE" ] && SCRATCH_DEV_EMPTY_SPACE=0
+        ./tools/ag-wipe -q -r $SCRATCH_DEV_EMPTY_SPACE $SCRATCH_DEV
     fi
 
     return $mkfs_status
Signature · 6 lines
1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#9 lore raw

PATCH 5/13 xfstests: use preallocation for ag-wiper

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

To enable sane testing of large scale filesystems, the --large-fs test option uses xfs_db magic to mark AGs full without doing any IO. This leaves only a small amount of free space left in the filesystem to stress the high AGs of the filesystem rather than the low AGs.

This method requires us to have special filesystem check options to avoid free space checking in xfs_check, and we cannot current run xfs_repair on such a filesystem at all. As it is, free space checking on xfs_check does not scale, so we still need to avoid this checking regardless of how we fill the filesystem.

We can acheive exactly the same fill behaviour by preallocating a single large file in the filesystem immediately after creating it. This is a filesystem independent manner of filling the filesystem, and allows us to do large filesystem testing on more than just XFS.

Further, this preallocation method effectively adds a new "very large file" test. It also enables us to run an unmodified xfs_repair or filesystem specific fsck program to check the filesystem for sanity, so we can now do full sanity checking of such large filesystems.

---
 common.rc |   58 +++++++++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 53 insertions(+), 5 deletions(-)
common.rc +53 -5
index 42d2ddc..d221be1 100644
@@ -276,6 +276,47 @@ _scratch_mkfs_options()
     echo $SCRATCH_OPTIONS $MKFS_OPTIONS $* $SCRATCH_DEV
 }
 
+
+_setup_large_xfs_fs()
+{
+	fs_size=$1
+	local tmp_dir=/tmp/
+
+	[ "$LARGE_SCRATCH_DEV" != yes ] && return 0
+	[ -z "$SCRATCH_DEV_EMPTY_SPACE" ] && SCRATCH_DEV_EMPTY_SPACE=0
+	[ $SCRATCH_DEV_EMPTY_SPACE -ge $fs_size ] && return 0
+
+	# calculate the size of the file we need to allocate.
+	# Default free space in the FS is 50GB, but you can specify more via
+	# SCRATCH_DEV_EMPTY_SPACE
+	file_size=$(($fs_size - 50*1024*1024*1024))
+	file_size=$(($file_size - $SCRATCH_DEV_EMPTY_SPACE))
+
+	# mount the filesystem, create the file, unmount it
+	_scratch_mount 2>&1 >$tmp_dir/mnt.err
+	local status=$?
+	if [ $status -ne 0 ]; then
+		echo "mount failed"
+		cat $tmp_dir/mnt.err >&2
+		rm -f $tmp_dir/mnt.err
+		return $status
+	fi
+	rm -f $tmp_dir/mnt.err
+
+	xfs_io -F -f \
+		-c "truncate $file_size" \
+		-c "falloc -k 0 $file_size" \
+		$SCRATCH_MNT/.use_space 2>&1 > /dev/null
+	status=$?
+	umount $SCRATCH_MNT
+	if [ $status -ne 0 ]; then
+		echo "large file prealloc failed"
+		cat $tmp_dir/mnt.err >&2
+		return $status
+	fi
+	return 0
+}
+
 _scratch_mkfs_xfs()
 {
     # extra mkfs options can be added by tests
@@ -305,16 +346,23 @@ _scratch_mkfs_xfs()
         mkfs_status=$?
     fi
 
+    if [ $mkfs_status -eq 0 -a "$LARGE_SCRATCH_DEV" = yes ]; then
+	# manually parse the mkfs output to get the fs size in bytes
+	local fs_size
+	fs_size=`cat $tmp_dir.mkfsstd | perl -ne '
+	    if (/^data\s+=\s+bsize=(\d+)\s+blocks=(\d+)/) {
+		my $size = $1 * $2;
+		print STDOUT "$size\n";
+	    }'`
+	_setup_large_xfs_fs $fs_size
+	mkfs_status=$?
+    fi
+
     # output stored mkfs output
     cat $tmp_dir.mkfserr >&2
     cat $tmp_dir.mkfsstd
     rm -f $tmp_dir.mkfserr $tmp_dir.mkfsstd
 
-    if [ "$LARGE_SCRATCH_DEV" = yes ]; then
-        [ -z "$SCRATCH_DEV_EMPTY_SPACE" ] && SCRATCH_DEV_EMPTY_SPACE=0
-        ./tools/ag-wipe -q -r $SCRATCH_DEV_EMPTY_SPACE $SCRATCH_DEV
-    fi
-
     return $mkfs_status
 }
 
Signature · 6 lines
1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#11 lore raw

PATCH 6/13 xfstests: use command line option for setting extra space

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Allow the extra free space to leave in large scratch filesystems to be specified by a command line option rather than just via an environment variable.

---
 common |    5 +++++
 1 file changed, 5 insertions(+)
common +5 -0
index da86cd9..78ac654 100644
@@ -247,6 +247,11 @@ s/ .*//p
 	    usage
 	    ;;
 
+	--extra-space=*)
+	    export SCRATCH_DEV_EMPTY_SPACE=${r#*=}
+	    xpand=false
+	    ;;
+
 	*)
 	    start=$r
 	    end=$r
Signature · 6 lines
1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#12 lore raw

PATCH 7/13 xfstest: enable xfs_repair for large filesystem testing

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Now that large filesystem testing does not play free space games to fill the space without IO, we can enable xfs_repair when running in this mode. xfs_repair has had it's scalability problems solved, too, so this is a safe thing to do.

---
 common.rc |   18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)
common.rc +10 -8
index d221be1..ee4f522 100644
@@ -1273,19 +1273,17 @@ _check_xfs_filesystem()
     extra_mount_options=""
     device=$1
     if [ "$2" != "none" ]; then
-       	extra_log_options="-l$2"
+	extra_log_options="-l$2"
         extra_mount_options="-ologdev=$2"
     fi
 
     if [ "$3" != "none" ]; then
-       	extra_rt_options="-r$3"
+	extra_rt_options="-r$3"
         extra_mount_options=$extra_mount_options" -ortdev=$3"
     fi
     extra_mount_options=$extra_mount_options" $MOUNT_OPTIONS"
 
     [ "$FSTYP" != xfs ] && return 0
-    testoption=""
-    [ "$LARGE_SCRATCH_DEV" = yes ] && testoption=-t
 
     type=`_fs_type $device`
     ok=1
@@ -1310,8 +1308,13 @@ _check_xfs_filesystem()
         ok=0
     fi
 
-    $XFS_CHECK_PROG $testoption $extra_log_options $device 2>&1 |\
-	 _fix_malloc >$tmp.fs_check
+    # xfs_check runs out of memory on large files, so even providing the test
+    # option (-t) to avoid indexing the free space trees doesn't make it pass on
+    # large filesystems. Avoid it.
+    if [ "$LARGE_SCRATCH_DEV" != yes ]; then
+	    $XFS_CHECK_PROG $extra_log_options $device 2>&1 |\
+		 _fix_malloc >$tmp.fs_check
+    fi
     if [ -s $tmp.fs_check ]
     then
         echo "_check_xfs_filesystem: filesystem on $device is inconsistent (c) (see $seq.full)"
@@ -1323,8 +1326,7 @@ _check_xfs_filesystem()
 
         ok=0
     fi
-    # repair doesn't scale massively at this stage, optionally skip it for now
-    [ "$LARGE_SCRATCH_DEV" = yes ] || \
+
     $XFS_REPAIR_PROG -n $extra_log_options $extra_rt_options $device >$tmp.repair 2>&1
     if [ $? -ne 0 ]
     then
Signature · 6 lines
1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#13 lore raw

PATCH 8/13 xfstests: always use test option when checking large scratch device

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Some tests call _check_scratch_device directly and when using a large filesystem this needs to run with a -t option to avoid consuming large amounts of memory. Make this happen in all cases that the scratch device is checked.

---
 017       |    7 ++-----
 common.rc |    2 ++
 2 files changed, 4 insertions(+), 5 deletions(-)
017 +2 -5
index 7db4667..fb92375 100755
@@ -48,9 +48,6 @@ _supported_os Linux
 
 _require_scratch
 
-checkopts=""
-[ "$USE_BIG_LOOPFS" = yes ] && checkopts=-t
-
 echo "*** init FS"
 
 rm -f $seq.full
@@ -82,8 +79,8 @@ do
         echo ""                             >>$seq.full
         echo "*** XFS_CHECK ***"            >>$seq.full
         echo ""                             >>$seq.full
-        _scratch_xfs_check $checkopts       >>$seq.full 2>&1 \
-            || _fail "xfs_check $checkopts failed"
+        _scratch_xfs_check                  >>$seq.full 2>&1 \
+            || _fail "xfs_check failed"
         _scratch_mount -o remount,rw \
             || _fail "remount rw failed"
 done
common.rc +2 -0
index ee4f522..1026572 100644
@@ -475,6 +475,8 @@ _scratch_xfs_check()
     SCRATCH_OPTIONS=""
     [ "$USE_EXTERNAL" = yes -a ! -z "$SCRATCH_LOGDEV" ] && \
         SCRATCH_OPTIONS="-l $SCRATCH_LOGDEV"
+    [ "$LARGE_SCRATCH_DEV" = yes ] && \
+        SCRATCH_OPTIONS=$SCRATCH_OPTIONS" -t"
     $XFS_CHECK_PROG $SCRATCH_OPTIONS $* $SCRATCH_DEV
 }
 
Signature · 6 lines
1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#14 lore raw

PATCH 9/13 xfstests: enable large fs testing on ext4

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Now that setting up large filesystem testing on sparse loopback devices uses a generic method for filling the filesystem, extent support to ext4 filesystems.

ext4 is slightly more complex to fill as it does not support files larger than 16TB. Hence a slightly more complex method of using multiple smaller files to fill the space is necessary.

WARNING: be prepeared for ext4 to take forever to prepare large filesystems as allocation of large amounts of space (especially as it approaches ENOSPC) is can take minutes rather than milliseconds

---
 common.rc |   90 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 90 insertions(+)
common.rc +90 -0
index 1026572..690c6a3 100644
@@ -366,6 +366,93 @@ _scratch_mkfs_xfs()
     return $mkfs_status
 }
 
+_setup_large_ext4_fs()
+{
+	fs_size=$1
+	local tmp_dir=/tmp/
+
+	[ "$LARGE_SCRATCH_DEV" != yes ] && return 0
+	[ -z "$SCRATCH_DEV_EMPTY_SPACE" ] && SCRATCH_DEV_EMPTY_SPACE=0
+	[ $SCRATCH_DEV_EMPTY_SPACE -ge $fs_size ] && return 0
+
+	# Default free space in the FS is 50GB, but you can specify more via
+	# SCRATCH_DEV_EMPTY_SPACE
+	space_to_consume=$(($fs_size - 50*1024*1024*1024 - $SCRATCH_DEV_EMPTY_SPACE))
+
+	# mount the filesystem and create 16TB - 4KB files until we consume
+	# all the necessary space.
+	_scratch_mount 2>&1 >$tmp_dir/mnt.err
+	local status=$?
+	if [ $status -ne 0 ]; then
+		echo "mount failed"
+		cat $tmp_dir/mnt.err >&2
+		rm -f $tmp_dir/mnt.err
+		return $status
+	fi
+	rm -f $tmp_dir/mnt.err
+
+	file_size=$((16*1024*1024*1024*1024 - 4096))
+	nfiles=0
+	while [ $space_to_consume -gt $file_size ]; do
+
+		xfs_io -F -f \
+			-c "truncate $file_size" \
+			-c "falloc -k 0 $file_size" \
+			$SCRATCH_MNT/.use_space.$nfiles 2>&1
+		status=$?
+		if [ $status -ne 0 ]; then
+			break;
+		fi
+
+		space_to_consume=$(( $space_to_consume - $file_size ))
+		nfiles=$(($nfiles + 1))
+	done
+
+	# consume the remaining space.
+	if [ $space_to_consume -gt 0 ]; then
+		xfs_io -F -f \
+			-c "truncate $space_to_consume" \
+			-c "falloc -k 0 $space_to_consume" \
+			$SCRATCH_MNT/.use_space.$nfiles 2>&1
+		status=$?
+	fi
+
+	umount $SCRATCH_MNT
+	if [ $status -ne 0 ]; then
+		echo "large file prealloc failed"
+		cat $tmp_dir/mnt.err >&2
+		return $status
+	fi
+	return 0
+}
+_scratch_mkfs_ext4()
+{
+	local tmp_dir=/tmp/
+
+	/sbin/mkfs -t $FSTYP -- $MKFS_OPTIONS $* $SCRATCH_DEV \
+			2>$tmp_dir.mkfserr 1>$tmp_dir.mkfsstd
+	local mkfs_status=$?
+
+	if [ $mkfs_status -eq 0 -a "$LARGE_SCRATCH_DEV" = yes ]; then
+		# manually parse the mkfs output to get the fs size in bytes
+		fs_size=`cat $tmp_dir.mkfsstd | awk ' \
+			/^Block size/ { split($2, a, "="); bs = a[2] ; } \
+			/ inodes, / { blks = $3 } \
+			/reserved for the super user/ { resv = $1 } \
+			END { fssize = bs * blks - resv; print fssize }'`
+
+		_setup_large_ext4_fs $fs_size
+		mkfs_status=$?
+	fi
+
+	# output stored mkfs output
+	cat $tmp_dir.mkfserr >&2
+	cat $tmp_dir.mkfsstd
+	rm -f $tmp_dir.mkfserr $tmp_dir.mkfsstd
+
+	return $mkfs_status
+}
+
 _scratch_mkfs()
 {
     case $FSTYP in
@@ -381,6 +468,9 @@ _scratch_mkfs()
     btrfs)
         $MKFS_BTRFS_PROG $MKFS_OPTIONS $* $SCRATCH_DEV > /dev/null
 	;;
+    ext4)
+	_scratch_mkfs_ext4 $*
+	;;
     *)
 	yes | $MKFS_PROG -t $FSTYP -- $MKFS_OPTIONS $* $SCRATCH_DEV
 	;;
Signature · 6 lines
1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#15 lore raw

PATCH 10/13 xfstests: disable tests that typically fail on large filesystems

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Test 017 typically fails due to the OOM killer kicking in and killing everything but xfs_db so panics the machine.

---
 017 |    4 ++++
 1 file changed, 4 insertions(+)
017 +4 -0
index fb92375..65ddce1 100755
@@ -48,6 +48,10 @@ _supported_os Linux
 
 _require_scratch
 
+# xfs_db will OOM kill the machine if you don't have huge amounts of RAM, so
+# don't run this on large filesystems.
+_require_no_large_scratch_dev
+
 echo "*** init FS"
 
 rm -f $seq.full
Signature · 6 lines
1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#16 lore raw

PATCH 11/13 xfstests: exclude largefs fill files from dump tests

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

The xfsdump/restore tests will see the space filling files and may try to backthem up and restore them, consuming huge amounts of time to do (espcially when diffing the results). Exclude the space filling files by setting the no dump attributes on them and ensure that xfsdump runs with the -e flag to exclude such files.

This also needs a dump filter addition to remove the output that files were skipped, and to decrement the count of files processed by xfsrestore because the inventory still includes excluded files.

---
 common.config |    3 ++-
 common.dump   |   34 ++++++++++++++++++++++++++--------
 common.filter |    4 +++-
 common.rc     |    1 +
 4 files changed, 32 insertions(+), 10 deletions(-)
common.config +2 -1
index 5ac58bf..ab19772 100644
@@ -160,7 +160,8 @@ export XFS_GROWFS_PROG=`set_prog_path xfs_growfs`
 export XFS_IO_PROG="`set_prog_path xfs_io`"
 export XFS_PARALLEL_REPAIR_PROG="`set_prog_path xfs_prepair`"
 export XFS_PARALLEL_REPAIR64_PROG="`set_prog_path xfs_prepair64`"
-export XFSDUMP_PROG="`set_prog_path xfsdump`"
+export __XFSDUMP_PROG="`set_prog_path xfsdump`"
+export XFSDUMP_PROG="$__XFSDUMP_PROG -e"
 export XFSRESTORE_PROG="`set_prog_path xfsrestore`"
 export XFSINVUTIL_PROG="`set_prog_path xfsinvutil`"
 export DBENCH_PROG="`set_prog_path dbench`"
common.dump +26 -8
index e89dedc..be55249 100644
@@ -29,7 +29,8 @@ if [ -n "$DEBUGDUMP" ]; then
 
 	# Use dump/restore in qa directory (copy them here) for debugging
 	export PATH="$here:$PATH"
-	export XFSDUMP_PROG="`set_prog_path xfsdump`"
+	export __XFSDUMP_PROG="`set_prog_path xfsdump`"
+	export XFSDUMP_PROG="$__XFSDUMP_PROG -e"
 	export XFSRESTORE_PROG="`set_prog_path xfsrestore`"
 	export XFSINVUTIL_PROG="`set_prog_path xfsinvutil`"
 	[ -x $here/xfsdump ]    && echo "Using xfstests' xfsdump for debug"
@@ -829,7 +830,7 @@ _dump_filter_main()
 {
   _filter_devchar |\
   sed \
-      -e "s#$XFSDUMP_PROG#xfsdump#"			\
+      -e "s#$__XFSDUMP_PROG#xfsdump#"			\
       -e "s#$XFSRESTORE_PROG#xfsrestore#"		\
       -e "s#$XFSINVUTIL_PROG#xfsinvutil#"		\
       -e "s/`hostname`/HOSTNAME/"			\
@@ -872,13 +873,29 @@ _dump_filter_main()
       -e '/xfsdump: ino map phase [5]/,1d'		\
       -e 's/id:[[:space:]]*[0-9a-f]\{8\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{12\}/ID: ID/'						\
       -e 's/\[y\/n\][- ]----------------------*/\[y\/n\]/'		\
+      -e '/skip attribute set/d'				\
   | perl -ne '
-      if ($_ =~ /(?:Dump|Restore) Summary/) {
-        $skip = 1;
-      } elsif ($_ =~ /(?:Dump|Restore) Status/) {
-        $skip = 0;
-      }
-      print if (! $skip);'
+	# filter out all the output between the lines "Dump Summary:"
+	# and "Dump Status:"
+	if ($_ =~ /(?:Dump|Restore) Summary/) {
+		$skip = 1;
+	} elsif ($_ =~ /(?:Dump|Restore) Status/) {
+		$skip = 0;
+	}
+	print if (! $skip);' \
+  | perl -ne '
+	# correct the file count if large scratch devices are being used
+	$skip = 0;
+	if ($_ =~ /(\S+) directories and (\S+) entries/) {
+		$foo = $2;
+		if ($ENV{'LARGE_SCRATCH_DEV'} && $foo > 0) {
+			$foo -= 1;
+		}
+		printf("xfsrestore: %u directories and %u entries processed\n",
+						$1, $foo);
+		$skip = 1;
+	}
+	print if (! $skip);'
 }
 
 _dump_filter()
@@ -916,6 +933,7 @@ _dir_filter()
     -e "s#$dump_sdir#DUMP_SUBDIR#g"   \
     -e "s#$restore_sdir#RESTORE_SUBDIR#g" \
     -e "s#$$#PID#g" \
+    -e "/Only in SCRATCH_MNT: .use_space/d" \
 
 }
 
common.filter +3 -1
index c34809f..da5675f 100644
@@ -223,7 +223,9 @@ _filter_test_dir()
 
 _filter_scratch()
 {
-	sed -e "s,$SCRATCH_DEV,SCRATCH_DEV,g" -e "s,$SCRATCH_MNT,SCRATCH_MNT,g"
+	sed -e "s,$SCRATCH_DEV,SCRATCH_DEV,g" \
+	    -e "s,$SCRATCH_MNT,SCRATCH_MNT,g" \
+	    -e "/.use_space/d"
 }
 
 # Turn any device in the scratch pool into SCRATCH_DEV
common.rc +1 -0
index 690c6a3..96d5077 100644
@@ -306,6 +306,7 @@ _setup_large_xfs_fs()
 	xfs_io -F -f \
 		-c "truncate $file_size" \
 		-c "falloc -k 0 $file_size" \
+		-c "chattr +d" \
 		$SCRATCH_MNT/.use_space 2>&1 > /dev/null
 	status=$?
 	umount $SCRATCH_MNT
Signature · 6 lines
1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#18 lore raw

PATCH 12/13 xfstests: filter files in 019 for --large-fs

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

Make sure that the .use_space files don't appear in the files dumped to the output files.

---
 019 |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
019 +3 -2
index 42e3c22..88be464 100755
@@ -128,15 +128,16 @@ _verify_fs()
 
 	echo "*** verify FS"
 	(cd $SCRATCH_MNT ; find . | LC_COLLATE=POSIX sort \
+		| grep -v ".use_space" \
 		| xargs $here/src/lstat64 | _filter_stat)
 	diff -q $SCRATCH_MNT/bigfile $tempfile.2 \
 		|| _fail "bigfile corrupted"
-    
+
 	echo "*** unmount FS"
 	_full "umount"
 	umount $SCRATCH_DEV >>$seqfull 2>&1 \
 		|| _fail "umount failed"
-	    
+
 	echo "*** check FS"
 	_check_scratch_fs
 }
Signature · 6 lines
1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#20 lore raw

PATCH 13/13 xfstests: correct quota file count for --large-fs

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

The files used to fill space when using --large-fs contribute towards the quota of uid 0. Modify the quota output filter in test 050 to "hide" these files from the test output.

---
 050       |    8 +++++++-
 common.rc |    2 ++
 2 files changed, 9 insertions(+), 1 deletion(-)
050 +7 -1
index cf70b5a..acca654 100755
@@ -69,7 +69,13 @@ _filter_report()
 	perl -npe '
 		s/^\#'$id' /[NAME] /g;
 		s/^\#0 \d+ /[ROOT] 0 /g;
-		s/6 days/7 days/g'
+		s/6 days/7 days/g' |
+	perl -npe '
+		$val = 0;
+		if ($ENV{'LARGE_SCRATCH_DEV'}) {
+			$val = $ENV{'NUM_SPACE_FILES'};
+		}
+		s/(^\[ROOT\] \S+ \S+ \S+ \S+ \[--------\] )(\S+)/$1@{[$2 - $val]}/g'
 }
 
 # The actual point at which limit enforcement takes place for the
common.rc +2 -0
index 96d5077..319680d 100644
@@ -308,6 +308,7 @@ _setup_large_xfs_fs()
 		-c "falloc -k 0 $file_size" \
 		-c "chattr +d" \
 		$SCRATCH_MNT/.use_space 2>&1 > /dev/null
+	export NUM_SPACE_FILES=1
 	status=$?
 	umount $SCRATCH_MNT
 	if [ $status -ne 0 ]; then
@@ -417,6 +418,7 @@ _setup_large_ext4_fs()
 			$SCRATCH_MNT/.use_space.$nfiles 2>&1
 		status=$?
 	fi
+	export NUM_SPACE_FILES=$nfiles
 
 	umount $SCRATCH_MNT
 	if [ $status -ne 0 ]; then
Signature · 6 lines
1.7.10.4

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#22 lore raw
In reply to Dave Chinner

This patchset has been committed, thanks for submitting it Dave.

Thanks
--Rich

Quoted 14 lines · Dave Chinner

Hi folks,

This is a repost of my current version of the large filesystem testing support for xfstests. I've been using it for more than a year, and I think the ext4 guys would like to have it available to them.

Last posted here:

http://oss.sgi.com/archives/xfs/2012-07/msg00361.html

There are changes from last time, but it's months since i've made them, so there's no fine grained changelog. :/

Quoted 9 lines

Cheers,

Dave.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs