Skip to contentSkip to conversation

fstests: fix a major test setup performance regression

PATCH In review Dave Chinner 4 participants 7 messages May 20, 2022

lore
Share & tools
lore
#0 lore raw

cover fstests: fix a major test setup performance regression

Hi folks,

I've been seeing symptoms of this issue described in patch 2 for most of this merge cycle, and it's slowly been getting worse, to the point where I had to fix it to be able to get any amount of testing done. Turns out we're doing something that is not smart with XFS test setup, and recent changes to mkfs.xfs caused that to be exposed. These patches fix it.

The first patch is a cleanup - no test should be including common/rc directly anymore - _being_fstest() does that now. The second patch fixes the XFS issues in init_rc() which is called from _begin_fstest when is sources common/rc....

Cheers,

Dave.

#1 lore raw

PATCH 1/2 fstests: tests should not source common/rc directly

From: Dave Chinner <[email protected]>

_begin_fstest() does this as every test needs it, so remove it from all the tests that source it a second time.

---
 tests/btrfs/243   | 1 -
 tests/btrfs/245   | 1 -
 tests/ext4/053    | 1 -
 tests/generic/644 | 1 -
 tests/generic/645 | 1 -
 tests/generic/647 | 1 -
 tests/generic/677 | 1 -
 tests/generic/678 | 1 -
 tests/generic/679 | 1 -
 tests/generic/690 | 1 -
 10 files changed, 10 deletions(-)
tests/btrfs/243 +0 -1
index 750b4323..09a63bcd 100755
@@ -20,7 +20,6 @@ _cleanup()
 	rm -r -f $tmp.*
 }
 
-. ./common/rc
 . ./common/filter
 . ./common/dmflakey
 
tests/btrfs/245 +0 -1
index dadc8492..426aab1c 100755
@@ -11,7 +11,6 @@
 _begin_fstest auto quick idmapped subvol
 
 # get standard environment, filters and checks
-. ./common/rc
 . ./common/filter
 
 # real QA test starts here
tests/ext4/053 +0 -1
index bf6e3f6b..187a2515 100755
@@ -28,7 +28,6 @@ _cleanup()
 }
 
 # get standard environment, filters and checks
-. ./common/rc
 . ./common/filter
 . ./common/quota
 
tests/generic/644 +0 -1
index edf9b03e..c48338d8 100755
@@ -10,7 +10,6 @@
 _begin_fstest auto quick cap idmapped mount
 
 # get standard environment, filters and checks
-. ./common/rc
 . ./common/filter
 
 # real QA test starts here
tests/generic/645 +0 -1
index 74e0f589..556d2450 100755
@@ -10,7 +10,6 @@
 _begin_fstest auto quick idmapped mount
 
 # get standard environment, filters and checks
-. ./common/rc
 . ./common/filter
 
 # real QA test starts here
tests/generic/647 +0 -1
index fa6edadc..8484fa8d 100755
@@ -18,7 +18,6 @@ _cleanup()
 }
 
 # get standard environment, filters and checks
-. ./common/rc
 . ./common/filter
 
 # real QA test starts here
tests/generic/677 +0 -1
index 1d4eaa53..39af90a9 100755
@@ -18,7 +18,6 @@ _cleanup()
 	rm -r -f $tmp.*
 }
 
-. ./common/rc
 . ./common/filter
 . ./common/dmflakey
 . ./common/punch
tests/generic/678 +0 -1
index 1c3cf6e3..3c3c05b7 100755
@@ -22,7 +22,6 @@ _cleanup()
 }
 
 # get standard environment, filters and checks
-. ./common/rc
 . ./common/filter
 
 # real QA test starts here
tests/generic/679 +0 -1
index c32d42b9..a0094e48 100755
@@ -11,7 +11,6 @@
 . ./common/preamble
 _begin_fstest auto quick prealloc
 
-. ./common/rc
 . ./common/filter
 . ./common/punch
 
tests/generic/690 +0 -1
index f03295a5..cef8d6e8 100755
@@ -24,7 +24,6 @@ _cleanup()
 	rm -r -f $tmp.*
 }
 
-. ./common/rc
 . ./common/filter
 . ./common/dmflakey
 
Signature · 1 line
2.35.1
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] fstests: tests should not source common/rc directly

Quoted 6 lines · Dave Chinner

From: Dave Chinner <[email protected]>

_begin_fstest() does this as every test needs it, so remove it from all the tests that source it a second time.

Looks good,

--D

Quoted 136 lines
---
 tests/btrfs/243   | 1 -
 tests/btrfs/245   | 1 -
 tests/ext4/053    | 1 -
 tests/generic/644 | 1 -
 tests/generic/645 | 1 -
 tests/generic/647 | 1 -
 tests/generic/677 | 1 -
 tests/generic/678 | 1 -
 tests/generic/679 | 1 -
 tests/generic/690 | 1 -
 10 files changed, 10 deletions(-)
tests/btrfs/243 +0 -1
index 750b4323..09a63bcd 100755
@@ -20,7 +20,6 @@ _cleanup()
 	rm -r -f $tmp.*
 }
 
-. ./common/rc
 . ./common/filter
 . ./common/dmflakey
 
tests/btrfs/245 +0 -1
index dadc8492..426aab1c 100755
@@ -11,7 +11,6 @@
 _begin_fstest auto quick idmapped subvol
 
 # get standard environment, filters and checks
-. ./common/rc
 . ./common/filter
 
 # real QA test starts here
tests/ext4/053 +0 -1
index bf6e3f6b..187a2515 100755
@@ -28,7 +28,6 @@ _cleanup()
 }
 
 # get standard environment, filters and checks
-. ./common/rc
 . ./common/filter
 . ./common/quota
 
tests/generic/644 +0 -1
index edf9b03e..c48338d8 100755
@@ -10,7 +10,6 @@
 _begin_fstest auto quick cap idmapped mount
 
 # get standard environment, filters and checks
-. ./common/rc
 . ./common/filter
 
 # real QA test starts here
tests/generic/645 +0 -1
index 74e0f589..556d2450 100755
@@ -10,7 +10,6 @@
 _begin_fstest auto quick idmapped mount
 
 # get standard environment, filters and checks
-. ./common/rc
 . ./common/filter
 
 # real QA test starts here
tests/generic/647 +0 -1
index fa6edadc..8484fa8d 100755
@@ -18,7 +18,6 @@ _cleanup()
 }
 
 # get standard environment, filters and checks
-. ./common/rc
 . ./common/filter
 
 # real QA test starts here
tests/generic/677 +0 -1
index 1d4eaa53..39af90a9 100755
@@ -18,7 +18,6 @@ _cleanup()
 	rm -r -f $tmp.*
 }
 
-. ./common/rc
 . ./common/filter
 . ./common/dmflakey
 . ./common/punch
tests/generic/678 +0 -1
index 1c3cf6e3..3c3c05b7 100755
@@ -22,7 +22,6 @@ _cleanup()
 }
 
 # get standard environment, filters and checks
-. ./common/rc
 . ./common/filter
 
 # real QA test starts here
tests/generic/679 +0 -1
index c32d42b9..a0094e48 100755
@@ -11,7 +11,6 @@
 . ./common/preamble
 _begin_fstest auto quick prealloc
 
-. ./common/rc
 . ./common/filter
 . ./common/punch
 
tests/generic/690 +0 -1
index f03295a5..cef8d6e8 100755
@@ -24,7 +24,6 @@ _cleanup()
 	rm -r -f $tmp.*
 }
 
-. ./common/rc
 . ./common/filter
 . ./common/dmflakey
 
Signature · 1 line
2.35.1
#4 lore raw

PATCH 2/2 common/rc: fix test init performance regression on XFS

In reply to Dave Chinner

From: Dave Chinner <[email protected]>

I've been having problems over the past month with tests randomly running much slower than they should. The overall effect has been that running an auto group iterate has blown out from about 3 hours to almost 5 hours.

The problem has been that the root disks have been thrashing. Thousands of iops and the backing store getting IO bound so all test vms with root drive images hosted on that backing store then also go slow.

I finally got to the bottom of it - it looked for all the world like a qemu image format write amplification problem, because I couldn't capture anything unusual from inside the guest. I converted all the root images to raw format on fast SSD storage, and the problem didn't go away - while some tests were faster, other tests were still randomly slow.

Finally I tracked it down with blktrace/blkparse. All the IO was coming from processes with the command line like:

/sbin/mkfs.xfs -d file name=/tmp/791409.img size=512m

A quick grep then pointed me at common/rc::init_rc() where there is a test for XFS specific on-disk format behaviour that uses that exact command line.

init_rc() is run from _begin_fstest() so it is executed on every test start. We run mkfs to create a 512MB filesystem in $tmp on every test start. That's ..... not smart. And we do it so we can change the setup of $XFS_COPY_PROG, which is only used in a handful of tests. That's also .... not smart.

And then the penny dropped: we recently increased the default log size of small filesystems, so this is now zeroing a 64MB log instead of a 5-10MB log. And, of course it's on the root drive because of the $tmp image prefix, which in this case is ext3, and it's fragmenting the crap out of the writes and so it increases runtime of the mkfs.xfs execution from nothing to 5-10s.

Then when one of these mkfs invocations co-incides with journal flushing driven writeback, everything just stops until the writeback stops thrashing. And so all tests are 5-10s slows, and random tests take anywhere between 20-100s longer to start up....

Fix it by moving the setup of XFS_COPY_PROG to _require_xfs_copy() so only the tests that use xfs_copy run this code.

Numbers don't lie:

generic/001 11s ... 6s
generic/002 9s ... 1s
generic/003 17s ... 10s
generic/004 12s ... 1s
generic/005 9s ... 1s
generic/006 9s ... 2s
generic/007 11s ... 3s
generic/008 5s ... 2s
generic/009 10s ... 2s

generic/010 5s ...  0s
generic/011 9s ...  1s
generic/012 7s ...  2s

generic/013 10s ... 5s
generic/014 11s ... 1s
generic/015 7s ... 1s
generic/016 7s ... 2s
.....

---
 common/rc  | 13 -------------
 common/xfs | 12 ++++++++++++
 2 files changed, 12 insertions(+), 13 deletions(-)
common/rc +0 -13
index f5ead044..67b4cc77 100644
@@ -4522,19 +4522,6 @@ init_rc()
 	# it is supported.
 	$XFS_IO_PROG -i -c quit 2>/dev/null && \
 		export XFS_IO_PROG="$XFS_IO_PROG -i"
-
-	# xfs_copy on v5 filesystems do not require the "-d" option if xfs_db
-	# can change the UUID on v5 filesystems
-	if [ "$FSTYP" == "xfs" ]; then
-		touch /tmp/$$.img
-		$MKFS_XFS_PROG -d file,name=/tmp/$$.img,size=512m >/dev/null 2>&1
-		# xfs_db will return 0 even if it can't generate a new uuid, so
-		# check the output to make sure if it can change UUID of V5 xfs
-		$XFS_DB_PROG -x -c "uuid generate" /tmp/$$.img \
-			| grep -q "invalid UUID\|supported on V5 fs" \
-			&& export XFS_COPY_PROG="$XFS_COPY_PROG -d"
-		rm -f /tmp/$$.img
-	fi
 }
 
 # get real device path name by following link
common/xfs +12 -0
index ac1d021e..2123a4ab 100644
@@ -1160,6 +1160,18 @@ _require_xfs_copy()
 	[ -n "$XFS_COPY_PROG" ] || _notrun "xfs_copy binary not yet installed"
 	[ "$USE_EXTERNAL" = yes ] && \
 		_notrun "Cannot xfs_copy with external devices"
+
+	# xfs_copy on v5 filesystems do not require the "-d" option if xfs_db
+	# can change the UUID on v5 filesystems
+	touch /tmp/$$.img
+	$MKFS_XFS_PROG -d file,name=/tmp/$$.img,size=64m >/dev/null 2>&1
+
+	# xfs_db will return 0 even if it can't generate a new uuid, so
+	# check the output to make sure if it can change UUID of V5 xfs
+	$XFS_DB_PROG -x -c "uuid generate" /tmp/$$.img \
+		| grep -q "invalid UUID\|supported on V5 fs" \
+		&& export XFS_COPY_PROG="$XFS_COPY_PROG -d"
+	rm -f /tmp/$$.img
 }
 
 __xfs_cowgc_interval_knob1="/proc/sys/fs/xfs/speculative_cow_prealloc_lifetime"
Signature · 1 line
2.35.1
Reviewed-by on v1 2/2
#5 lore raw

Re: [PATCH 2/2] common/rc: fix test init performance regression on XFS

Quoted 63 lines · Dave Chinner

From: Dave Chinner <[email protected]>

I've been having problems over the past month with tests randomly running much slower than they should. The overall effect has been that running an auto group iterate has blown out from about 3 hours to almost 5 hours.

The problem has been that the root disks have been thrashing. Thousands of iops and the backing store getting IO bound so all test vms with root drive images hosted on that backing store then also go slow.

I finally got to the bottom of it - it looked for all the world like a qemu image format write amplification problem, because I couldn't capture anything unusual from inside the guest. I converted all the root images to raw format on fast SSD storage, and the problem didn't go away - while some tests were faster, other tests were still randomly slow.

Finally I tracked it down with blktrace/blkparse. All the IO was coming from processes with the command line like:

/sbin/mkfs.xfs -d file name=/tmp/791409.img size=512m

A quick grep then pointed me at common/rc::init_rc() where there is a test for XFS specific on-disk format behaviour that uses that exact command line.

init_rc() is run from _begin_fstest() so it is executed on every test start. We run mkfs to create a 512MB filesystem in $tmp on every test start. That's ..... not smart. And we do it so we can change the setup of $XFS_COPY_PROG, which is only used in a handful of tests. That's also .... not smart.

And then the penny dropped: we recently increased the default log size of small filesystems, so this is now zeroing a 64MB log instead of a 5-10MB log. And, of course it's on the root drive because of the $tmp image prefix, which in this case is ext3, and it's fragmenting the crap out of the writes and so it increases runtime of the mkfs.xfs execution from nothing to 5-10s.

Then when one of these mkfs invocations co-incides with journal flushing driven writeback, everything just stops until the writeback stops thrashing. And so all tests are 5-10s slows, and random tests take anywhere between 20-100s longer to start up....

Fix it by moving the setup of XFS_COPY_PROG to _require_xfs_copy() so only the tests that use xfs_copy run this code.

Numbers don't lie:

generic/001 11s ... 6s
generic/002 9s ... 1s
generic/003 17s ... 10s
generic/004 12s ... 1s
generic/005 9s ... 1s
generic/006 9s ... 2s
generic/007 11s ... 3s
generic/008 5s ... 2s
generic/009 10s ... 2s

generic/010 5s ...  0s
generic/011 9s ...  1s
generic/012 7s ...  2s
Quoted 7 lines

generic/013 10s ... 5s
generic/014 11s ... 1s
generic/015 7s ... 1s
generic/016 7s ... 2s
.....

Ah, nice. I hadn't even realized that was there!

--D

Quoted 55 lines
---
 common/rc  | 13 -------------
 common/xfs | 12 ++++++++++++
 2 files changed, 12 insertions(+), 13 deletions(-)
common/rc +0 -13
index f5ead044..67b4cc77 100644
@@ -4522,19 +4522,6 @@ init_rc()
 	# it is supported.
 	$XFS_IO_PROG -i -c quit 2>/dev/null && \
 		export XFS_IO_PROG="$XFS_IO_PROG -i"
-
-	# xfs_copy on v5 filesystems do not require the "-d" option if xfs_db
-	# can change the UUID on v5 filesystems
-	if [ "$FSTYP" == "xfs" ]; then
-		touch /tmp/$$.img
-		$MKFS_XFS_PROG -d file,name=/tmp/$$.img,size=512m >/dev/null 2>&1
-		# xfs_db will return 0 even if it can't generate a new uuid, so
-		# check the output to make sure if it can change UUID of V5 xfs
-		$XFS_DB_PROG -x -c "uuid generate" /tmp/$$.img \
-			| grep -q "invalid UUID\|supported on V5 fs" \
-			&& export XFS_COPY_PROG="$XFS_COPY_PROG -d"
-		rm -f /tmp/$$.img
-	fi
 }
 
 # get real device path name by following link
common/xfs +12 -0
index ac1d021e..2123a4ab 100644
@@ -1160,6 +1160,18 @@ _require_xfs_copy()
 	[ -n "$XFS_COPY_PROG" ] || _notrun "xfs_copy binary not yet installed"
 	[ "$USE_EXTERNAL" = yes ] && \
 		_notrun "Cannot xfs_copy with external devices"
+
+	# xfs_copy on v5 filesystems do not require the "-d" option if xfs_db
+	# can change the UUID on v5 filesystems
+	touch /tmp/$$.img
+	$MKFS_XFS_PROG -d file,name=/tmp/$$.img,size=64m >/dev/null 2>&1
+
+	# xfs_db will return 0 even if it can't generate a new uuid, so
+	# check the output to make sure if it can change UUID of V5 xfs
+	$XFS_DB_PROG -x -c "uuid generate" /tmp/$$.img \
+		| grep -q "invalid UUID\|supported on V5 fs" \
+		&& export XFS_COPY_PROG="$XFS_COPY_PROG -d"
+	rm -f /tmp/$$.img
 }
 
 __xfs_cowgc_interval_knob1="/proc/sys/fs/xfs/speculative_cow_prealloc_lifetime"
Signature · 1 line
2.35.1
Reviewed-by on v1 2/2
#6 lore raw

Re: [PATCH 2/2] common/rc: fix test init performance regression on XFS

In reply to Dave Chinner
Quoted 63 lines · Dave Chinner

From: Dave Chinner <[email protected]>

I've been having problems over the past month with tests randomly running much slower than they should. The overall effect has been that running an auto group iterate has blown out from about 3 hours to almost 5 hours.

The problem has been that the root disks have been thrashing. Thousands of iops and the backing store getting IO bound so all test vms with root drive images hosted on that backing store then also go slow.

I finally got to the bottom of it - it looked for all the world like a qemu image format write amplification problem, because I couldn't capture anything unusual from inside the guest. I converted all the root images to raw format on fast SSD storage, and the problem didn't go away - while some tests were faster, other tests were still randomly slow.

Finally I tracked it down with blktrace/blkparse. All the IO was coming from processes with the command line like:

/sbin/mkfs.xfs -d file name=/tmp/791409.img size=512m

A quick grep then pointed me at common/rc::init_rc() where there is a test for XFS specific on-disk format behaviour that uses that exact command line.

init_rc() is run from _begin_fstest() so it is executed on every test start. We run mkfs to create a 512MB filesystem in $tmp on every test start. That's ..... not smart. And we do it so we can change the setup of $XFS_COPY_PROG, which is only used in a handful of tests. That's also .... not smart.

And then the penny dropped: we recently increased the default log size of small filesystems, so this is now zeroing a 64MB log instead of a 5-10MB log. And, of course it's on the root drive because of the $tmp image prefix, which in this case is ext3, and it's fragmenting the crap out of the writes and so it increases runtime of the mkfs.xfs execution from nothing to 5-10s.

Then when one of these mkfs invocations co-incides with journal flushing driven writeback, everything just stops until the writeback stops thrashing. And so all tests are 5-10s slows, and random tests take anywhere between 20-100s longer to start up....

Fix it by moving the setup of XFS_COPY_PROG to _require_xfs_copy() so only the tests that use xfs_copy run this code.

Numbers don't lie:

generic/001 11s ... 6s
generic/002 9s ... 1s
generic/003 17s ... 10s
generic/004 12s ... 1s
generic/005 9s ... 1s
generic/006 9s ... 2s
generic/007 11s ... 3s
generic/008 5s ... 2s
generic/009 10s ... 2s

generic/010 5s ...  0s
generic/011 9s ...  1s
generic/012 7s ...  2s
Quoted 8 lines

generic/013 10s ... 5s
generic/014 11s ... 1s
generic/015 7s ... 1s
generic/016 7s ... 2s
.....

---

Nice catch! I just checked all cases use XFS_COPY_PROG, they all contains _require_xfs_copy at beginning, so I think this change is good. I'll give it more test before pushing.

That remind me better to check the common code which runs before each case real starting. And carefully review patches might add more operations at each case beginning. Anyway, thanks for this performance improvement.

Thanks,
Zorro

Quoted 54 lines
 common/rc  | 13 -------------
 common/xfs | 12 ++++++++++++
 2 files changed, 12 insertions(+), 13 deletions(-)
common/rc +0 -13
index f5ead044..67b4cc77 100644
@@ -4522,19 +4522,6 @@ init_rc()
 	# it is supported.
 	$XFS_IO_PROG -i -c quit 2>/dev/null && \
 		export XFS_IO_PROG="$XFS_IO_PROG -i"
-
-	# xfs_copy on v5 filesystems do not require the "-d" option if xfs_db
-	# can change the UUID on v5 filesystems
-	if [ "$FSTYP" == "xfs" ]; then
-		touch /tmp/$$.img
-		$MKFS_XFS_PROG -d file,name=/tmp/$$.img,size=512m >/dev/null 2>&1
-		# xfs_db will return 0 even if it can't generate a new uuid, so
-		# check the output to make sure if it can change UUID of V5 xfs
-		$XFS_DB_PROG -x -c "uuid generate" /tmp/$$.img \
-			| grep -q "invalid UUID\|supported on V5 fs" \
-			&& export XFS_COPY_PROG="$XFS_COPY_PROG -d"
-		rm -f /tmp/$$.img
-	fi
 }
 
 # get real device path name by following link
common/xfs +12 -0
index ac1d021e..2123a4ab 100644
@@ -1160,6 +1160,18 @@ _require_xfs_copy()
 	[ -n "$XFS_COPY_PROG" ] || _notrun "xfs_copy binary not yet installed"
 	[ "$USE_EXTERNAL" = yes ] && \
 		_notrun "Cannot xfs_copy with external devices"
+
+	# xfs_copy on v5 filesystems do not require the "-d" option if xfs_db
+	# can change the UUID on v5 filesystems
+	touch /tmp/$$.img
+	$MKFS_XFS_PROG -d file,name=/tmp/$$.img,size=64m >/dev/null 2>&1
+
+	# xfs_db will return 0 even if it can't generate a new uuid, so
+	# check the output to make sure if it can change UUID of V5 xfs
+	$XFS_DB_PROG -x -c "uuid generate" /tmp/$$.img \
+		| grep -q "invalid UUID\|supported on V5 fs" \
+		&& export XFS_COPY_PROG="$XFS_COPY_PROG -d"
+	rm -f /tmp/$$.img
 }
 
 __xfs_cowgc_interval_knob1="/proc/sys/fs/xfs/speculative_cow_prealloc_lifetime"
Signature · 1 line
2.35.1