Skip to contentSkip to conversation

xfstests: Provide dumpe2fs via standard common.config interface

PATCH In review Tomas Racek 4 participants 9 messages Feb 3, 2013 – Feb 26, 2013

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 xfstests: Provide dumpe2fs via standard common.config interface

dumpe2fs can be now accessed via $DUMPE2FS_PROG, tests that require it can check for its availability by _require_dumpe2fs function.

---
 common.config | 1 +
 common.rc     | 7 +++++++
 2 files changed, 8 insertions(+)
common.config +1 -0
index 57f505d..7174127 100644
@@ -159,6 +159,7 @@ export KILLALL_PROG="`set_prog_path killall`"
 export INDENT_PROG="`set_prog_path indent`"
 export XFS_COPY_PROG="`set_prog_path xfs_copy`"
 export FSTRIM_PROG="`set_prog_path fstrim`"
+export DUMPE2FS_PROG="`set_prog_path dumpe2fs`"
 
 # Generate a comparable xfsprogs version number in the form of
 # major * 10000 + minor * 100 + release
common.rc +7 -0
index 2e8581e..0cccb94 100644
@@ -1826,6 +1826,13 @@ _test_batched_discard()
 	$FSTRIM_PROG ${1} &>/dev/null
 }
 
+_require_dumpe2fs()
+{
+	if [ -z "$DUMPE2FS_PROG" ]; then
+		_notrun "This test requires dumpe2fs utility."
+	fi
+}
+
 ################################################################################
 
 if [ "$iam" != new -a "$iam" != bench ]
Signature · 6 lines
1.7.11.7

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#1 lore raw

PATCH 2/3 xfstests: Add new standard loop handling functions

Add _create_loop_device and _destroy_loop_device to uniformly handle loopback devices.

---
 common.rc | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
common.rc +14 -0
index 0cccb94..67a5e3a 100644
@@ -1833,6 +1833,20 @@ _require_dumpe2fs()
 	fi
 }
 
+_create_loop_device()
+{
+	file=$1
+	dev=`losetup -f`
+	losetup $dev $file || _fail "Cannot associate $file with $dev"
+	echo $dev
+}
+
+_destroy_loop_device()
+{
+	dev=$1
+	losetup -d $dev || _fail "Cannot destroy loop device $dev"
+}
+
 ################################################################################
 
 if [ "$iam" != new -a "$iam" != bench ]
Signature · 6 lines
1.7.11.7

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
#2 lore raw

Re: [PATCH 2/3] xfstests: Add new standard loop handling functions

Quoted 2 lines · Tomas Racek

Add _create_loop_device and _destroy_loop_device to uniformly handle loopback devices.

Good idea, small comment below

Quoted 10 lines
---
 common.rc | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
common.rc +0 -0
index 0cccb94..67a5e3a 100644

common.rc _require_dumpe2fs() · line 1841

Quoted 9 lines
common.rc +6 -0
@@ -1833,6 +1833,20 @@ _require_dumpe2fs()
 	fi
 }
 
+_create_loop_device()
+{
+	file=$1
+	dev=`losetup -f`
+	losetup $dev $file || _fail "Cannot associate $file with $dev"
+	echo $dev

In theory this is a little racy, right? the loop device could become used in between.

I wonder if something like:

_create_loop_device()

{
	dev=`losetup -f --show $file` || _fail "Cannot associate $file with $dev"
	echo $dev
}

would work better? From the manpage:

-f, --find
find the first unused loop device. If a file argument is present,
use this device. Otherwise, print its name

--show print device name if the -f option and a file argument are present.

So this will set up the first available loop device and print its name.

Thanks,
-Eric

common.rc _require_dumpe2fs() · line 1852

Quoted 12 lines
common.rc +8 -0
+}
+
+_destroy_loop_device()
+{
+	dev=$1
+	losetup -d $dev || _fail "Cannot destroy loop device $dev"
+}
+
 ################################################################################
 
 if [ "$iam" != new -a "$iam" != bench ]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#3 lore raw

Re: [PATCH 2/3] xfstests: Add new standard loop handling functions

----- Original Message -----

Quoted 46 lines · Tomas Racek
Quoted 3 lines

Add _create_loop_device and _destroy_loop_device to uniformly handle
loopback devices.

Good idea, small comment below

Quoted 10 lines
---
 common.rc | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
common.rc +0 -0
index 0cccb94..67a5e3a 100644

common.rc _require_dumpe2fs() · line 1841

Quoted 9 lines
common.rc +6 -0
@@ -1833,6 +1833,20 @@ _require_dumpe2fs()
 	fi
 }
 
+_create_loop_device()
+{
+	file=$1
+	dev=`losetup -f`
+	losetup $dev $file || _fail "Cannot associate $file with $dev"
+	echo $dev

In theory this is a little racy, right? the loop device could become used in between.

I wonder if something like:

_create_loop_device()

{
	dev=`losetup -f --show $file` || _fail "Cannot associate $file with
	$dev"
	echo $dev
}

would work better? From the manpage:

-f, --find
find the first unused loop device. If a file argument is present,
use this device. Otherwise, print its name

Quoted 5 lines

--show print device name if the -f option and a file argument are present.

So this will set up the first available loop device and print its name.

You're right, thanks for the suggestion!

Tom

Quoted 14 lines

Thanks,
-Eric

Quoted 10 lines
common.rc +8 -0
+}
+
+_destroy_loop_device()
+{
+	dev=$1
+	losetup -d $dev || _fail "Cannot destroy loop device $dev"
+}
+
 ################################################################################
 
Quoted 8 lines

common.rc _require_dumpe2fs() · line 1852

Quoted 2 lines
common.rc +0 -0
 if [ "$iam" != new -a "$iam" != bench ]

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Reviewed-by
#4 lore raw

Re: [PATCH 2/3] xfstests: Add new standard loop handling functions

Add _create_loop_device and _destroy_loop_device to uniformly handle loopback devices.

Quoted 8 lines
Quoted 8 lines

_create_loop_device()

{
	dev=`losetup -f --show $file` || _fail "Cannot associate $file with
	$dev"
	echo $dev
}

Looks good.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#5 lore raw

PATCH v2 3/3 xfstests: 297: Test that FS sends discard requests only on free blocks

In reply to Tomas Racek

V1->V2: Change way of testing suggested by Dave Chinner
1. Create image file with FS on it
2. Call fstrim to discard free blocks
3. Check that every punched hole in the image file is in the area that is marked as free

---
 297     | 190 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 297.out |   5 ++
 group   |   1 +
 3 files changed, 196 insertions(+)
 create mode 100755 297
 create mode 100644 297.out
297 new file +190 -0
new file mode 100755
index 0000000..7121c03
@@ -0,0 +1,190 @@
+#! /bin/bash
+# FS QA Test No. 297
+#
+# Test that filesystem sends discard requests only on free blocks
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2013 Red Hat, Inc., Tomas Racek <[email protected]>
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+# creator
[email protected]
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+. common.config
+. common.rc
+
+_supported_fs ext4 xfs
+_supported_os Linux
+_require_fstrim
+_require_fs_space $TEST_DIR 307200
+[ "$FSTYP" = "ext4" ] && _require_dumpe2fs
+
+_cleanup()
+{
+	$UMOUNT_PROG $loop_dev &> /dev/null
+	_destroy_loop_device $loop_dev
+	if [ $status -eq 0 ]; then
+		rm -rf $tmp
+		rm $img_file
+	fi
+}
+
+get_holes()
+{
+	$XFS_IO_PROG -c fiemap $1 | grep hole | $SED_PROG 's/.*\[\(.*\)\.\.\(.*\)\].*/\1 \2/'
+}
+
+get_free_sectors()
+{
+	case $FSTYP in
+	ext4)
+	$DUMPE2FS_PROG $img_file  2>&1 | grep " Free blocks" | cut -d ":" -f2- | \
+		tr ',' '\n' | $SED_PROG 's/^ //' | \
+		$AWK_PROG -v spb=$sectors_per_block 'BEGIN{FS="-"};
+		     NF {
+			if($2 != "") # range of blocks
+				print spb * $1, spb * ($2 + 1) - 1;
+			else		# just single block
+				print spb * $1, spb * ($1 + 1) - 1;
+		     }'
+	;;
+	xfs)
+	agsize=`xfs_info $loop_mnt | $SED_PROG -n 's/.*agsize=\(.*\) blks.*/\1/p'`
+	# Convert free space (agno, block, length) to (start sector, end sector)
+	$UMOUNT_PROG $loop_mnt
+	$XFS_DB_PROG -c "freesp -d" $img_file | $SED_PROG '/^.*from/,$d'| \
+		 $AWK_PROG -v spb=$sectors_per_block -v agsize=$agsize \
+		'{ print spb * ($1 * agsize + $2), spb * ($1 * agsize + $2 + $3) - 1 }'
+	;;
+	esac
+}
+
+merge_ranges()
+{
+	# Merges consecutive ranges from two input files
+	file1=$1
+	file2=$2
+
+	tmp_file=$tmp/sectors.tmp
+
+	cat $file1 $file2 | sort -n > $tmp_file
+
+	read line < $tmp_file
+	start=${line% *}
+	end=${line#* }
+
+	# Continue from second line
+	sed -i "1d" $tmp_file
+	while read line; do
+		curr_start=${line% *}
+		curr_end=${line#* }
+
+		if [ `expr $end + 1` -ge $curr_start ]; then
+			if [ $curr_end -gt $end ]; then
+				end=$curr_end
+			fi
+		else
+			echo $start $end
+			start=$curr_start
+			end=$curr_end
+		fi
+	done < $tmp_file
+
+	# Print last line
+	echo $start $end
+
+	rm $tmp_file
+}
+
+here=`pwd`
+tmp=`mktemp -d`
+
+img_file=$TEST_DIR/$$.fs
+dd if=/dev/zero of=$img_file bs=1M count=300 &> /dev/null
+
+loop_dev=$(_create_loop_device $img_file)
+loop_mnt=$tmp/loop_mnt
+
+fiemap_ref="$tmp/reference"
+fiemap_after="$tmp/after"
+free_sectors="$tmp/free_sectors"
+merged_sectors="$tmp/merged_free_sectors"
+
+mkdir $loop_mnt
+
+[ "$FSTYP" = "xfs" ] && MKFS_OPTIONS="-f $MKFS_OPTIONS"
+
+$MKFS_PROG -t $FSTYP $MKFS_OPTIONS $loop_dev &> /dev/null
+$MOUNT_PROG $loop_dev $loop_mnt
+
+echo -n "Generating garbage on loop..."
+for i in `seq 1 10`; do
+	mkdir $loop_mnt/$i
+	cp -r $here/* $loop_mnt/$i
+done
+
+# Get reference fiemap, this can contain i.e. uninitialized inode table
+sync
+get_holes $img_file > $fiemap_ref
+
+# Delete some files
+find $loop_mnt -type f -print | $AWK_PROG \
+	'BEGIN {srand()}; {if(rand() > 0.7) print $1;}' | xargs rm
+echo "done."
+
+echo -n "Running fstrim..."
+$FSTRIM_PROG $loop_mnt &> /dev/null
+echo "done."
+
+echo -n "Detecting interesting holes in image..."
+# Get after-trim fiemap
+sync
+get_holes $img_file > $fiemap_after
+echo "done."
+
+echo -n "Comparing holes to the reported space from FS..."
+# Get block size
+block_size=$(stat -f -c "%S" $loop_mnt/)
+sectors_per_block=`expr $block_size / 512`
+
+# Obtain free space from filesystem
+get_free_sectors > $free_sectors
+# Merge original holes with free sectors
+merge_ranges $fiemap_ref $free_sectors > $merged_sectors
+
+# Check that all holes after fstrim call were already present before or
+# that they match free space reported from FS
+while read line; do
+        from=${line% *}
+        to=${line#* }
+	if ! $AWK_PROG -v s=$from -v e=$to \
+		'{ if ($1 <= s && e <= $2) found = 1};
+		END { if(found) exit 0; else exit 1}' $merged_sectors
+	then
+		echo "Sectors $from-$to are not marked as free!"
+		exit
+	fi
+done < $fiemap_after
+echo "done."
+
+status=0
+exit
297.out new file +5 -0
new file mode 100644
index 0000000..71d720a
@@ -0,0 +1,5 @@
+QA output created by 297
+Generating garbage on loop...done.
+Running fstrim...done.
+Detecting interesting holes in image...done.
+Comparing holes to the reported space from FS...done.
group +1 -0
index eb4f375..64f322c 100644
@@ -415,3 +415,4 @@ deprecated
 294 auto quick
 295 auto logprint quick
 296 dump auto quick
+297 auto trim
Signature · 6 lines
1.7.11.7

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs
Reviewed-by on v2 3/3
#6 lore raw

Re: [PATCH 3/3 V2] xfstests: 297: Test that FS sends discard requests only on free blocks

Quoted 7 lines · Tomas Racek

V1->V2: Change way of testing suggested by Dave Chinner
1. Create image file with FS on it
2. Call fstrim to discard free blocks
3. Check that every punched hole in the image file is in the area that is marked as free

Looks good.

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

#8 lore raw
In reply to Tomas Racek

Applied this series to xfstests.

Thanks
--Rich

commit 29fe8007e75bf8983fa1267ae2850787262f17c1

Author: Tomas Racek <[email protected]>
Date:   Tue Feb 12 09:21:59 2013 +0000

xfstests: Add new standard loop handling functions

commit 03b0089e9b18ee6b4e5e990db98a1f545cc2521b

Author: Tomas Racek <[email protected]>
Date:   Sun Feb 3 10:19:56 2013 +0000

xfstests: Provide dumpe2fs via standard common.config interface

commit 76765b7e4e425dee394e12789264d6db1fea2876

Author: Tomas Racek <[email protected]>
Date:   Sun Feb 3 10:19:58 2013 +0000

xfstests: 298: Test that FS sends discard requests only on free blocks

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs