cpu scaling fixes v2
Hi all,
this series fully disables the mkfs automatic CPU count scaling for two tests, where that would lead to differences in the gold output.
Changes since v1:
- only disabling scaling in xfs/078 when it is supported
PATCH 1/2 xfs/078: disable all concurrency scaling
Disable concurrency scaling to avoid mismatching output on systems with a large CPU count.
--- tests/xfs/078 | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)
tests/xfs/078 +10 -3
@@ -57,9 +57,16 @@ _grow_loop() $XFS_IO_PROG -f -c "truncate $original" $LOOP_IMG loop_dev=`_create_loop_device $LOOP_IMG $bsize` - dparam="" + # Disable concurrency scaling to avoid output differences due to large + # CPU counts. + if _scratch_mkfs_xfs_supports_concurrency -l >> $seqres.full 2>&1; then + mkfs_opts="-d concurrency=0 -l concurrency=0 -r concurrency=0" + else + mkfs_opts="" + fi + if [ -n "$agsize" ]; then - dparam="-d agsize=$agsize" + mkfs_opts="$mkfs_opts -d agsize=$agsize" fi echo
@@ -67,7 +74,7 @@ _grow_loop() echo echo "*** mkfs loop file (size=$original)" - $MKFS_XFS_PROG -b size=$bsize $dparam $loop_dev | \ + $MKFS_XFS_PROG -b size=$bsize $mkfs_opts $loop_dev | \ _filter_mkfs 2>/dev/null echo "*** extend loop file"
Signature · 1 line
2.53.0
Re: [PATCH 1/2] xfs/078: disable all concurrency scaling
Quoted 6 lines · Christoph Hellwig
Disable concurrency scaling to avoid mismatching output on systems with a large CPU count.
Reported-by Shin'ichiro Kawasaki <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>Tested-by Shin'ichiro Kawasaki <[email protected]>
Quoted 36 lines
--- tests/xfs/078 | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-)tests/xfs/078 +10
-3@@ -57,9 +57,16 @@ _grow_loop() $XFS_IO_PROG -f -c "truncate $original" $LOOP_IMG loop_dev=`_create_loop_device $LOOP_IMG $bsize` - dparam="" + # Disable concurrency scaling to avoid output differences due to large + # CPU counts. + if _scratch_mkfs_xfs_supports_concurrency -l >> $seqres.full 2>&1; then + mkfs_opts="-d concurrency=0 -l concurrency=0 -r concurrency=0" + else + mkfs_opts="" + fi + if [ -n "$agsize" ]; then - dparam="-d agsize=$agsize" + mkfs_opts="$mkfs_opts -d agsize=$agsize" fi echo@@ -67,7 +74,7 @@ _grow_loop() echo echo "*** mkfs loop file (size=$original)" - $MKFS_XFS_PROG -b size=$bsize $dparam $loop_dev | \ + $MKFS_XFS_PROG -b size=$bsize $mkfs_opts $loop_dev | \ _filter_mkfs 2>/dev/null echo "*** extend loop file"
PATCH 2/2 xfs/216: disable all concurrency scaling
In reply to Christoph HellwigThis test currently disables log concurrency scaling, but even the data device concurrency scaling can create mismatching output on systems with a large CPU count.
--- tests/xfs/216 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/xfs/216 +1 -1
@@ -23,7 +23,7 @@ _cleanup() _require_scratch _scratch_mkfs_xfs >/dev/null 2>&1 if _scratch_mkfs_xfs_supports_concurrency -l >> $seqres.full 2>&1; then - loop_mkfs_opts="-l concurrency=0" + loop_mkfs_opts="-d concurrency=0 -l concurrency=0 -r concurrency=0" else loop_mkfs_opts="" fi
Signature · 1 line
2.53.0
Re: [PATCH 2/2] xfs/216: disable all concurrency scaling
Quoted 7 lines · Christoph Hellwig
This test currently disables log concurrency scaling, but even the data device concurrency scaling can create mismatching output on systems with a large CPU count.
Reported-by Shin'ichiro Kawasaki <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>Tested-by Shin'ichiro Kawasaki <[email protected]>
Quoted 17 lines
--- tests/xfs/216 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/xfs/216 +1
-1@@ -23,7 +23,7 @@ _cleanup() _require_scratch _scratch_mkfs_xfs >/dev/null 2>&1 if _scratch_mkfs_xfs_supports_concurrency -l >> $seqres.full 2>&1; then - loop_mkfs_opts="-l concurrency=0" + loop_mkfs_opts="-d concurrency=0 -l concurrency=0 -r concurrency=0" else loop_mkfs_opts="" fi
Re: [PATCH 2/2] xfs/216: disable all concurrency scaling
In reply to Christoph HellwigQuoted 16 lines · Christoph Hellwig
This test currently disables log concurrency scaling, but even the data device concurrency scaling can create mismatching output on systems with a large CPU count.
Reported-by Shin'ichiro Kawasaki <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>Tested-by Shin'ichiro Kawasaki <[email protected]>--- tests/xfs/216 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/xfs/216 +0
-0
tests/xfs/216 _cleanup() · line 26
Quoted 5 lines
tests/xfs/216 +1
-1@@ -23,7 +23,7 @@ _cleanup() _require_scratch _scratch_mkfs_xfs >/dev/null 2>&1 if _scratch_mkfs_xfs_supports_concurrency -l >> $seqres.full 2>&1; then - loop_mkfs_opts="-l concurrency=0" + loop_mkfs_opts="-d concurrency=0 -l concurrency=0 -r concurrency=0"
/me notes that -lconcurrency is not compatible with -llogdev and the loopdev is not formatted with SCRATCH_MKFS_OPTIONS, so this won't work to disable the concurrency= mkfs options if fstests is being run with SCRATCH_LOGDEV set.
--D
Quoted 7 lines
tests/xfs/216 +0
-0else loop_mkfs_opts="" fiSignature · 1 line
2.53.0
Re: [PATCH 2/2] xfs/216: disable all concurrency scaling
Quoted 12 lines · Darrick J. Wong
Quoted 11 lines · Christoph Hellwig
This test currently disables log concurrency scaling, but even the data device concurrency scaling can create mismatching output on systems with a large CPU count.
Reported-by Shin'ichiro Kawasaki <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>Tested-by Shin'ichiro Kawasaki <[email protected]>--- tests/xfs/216 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 15 lines
tests/xfs/216 _cleanup() · line 26
Quoted 10 lines
tests/xfs/216 +1
-1@@ -23,7 +23,7 @@ _cleanup() _require_scratch _scratch_mkfs_xfs >/dev/null 2>&1 if _scratch_mkfs_xfs_supports_concurrency -l >> $seqres.full 2>&1; then - loop_mkfs_opts="-l concurrency=0" + loop_mkfs_opts="-d concurrency=0 -l concurrency=0 -r concurrency=0"/me notes that -lconcurrency is not compatible with -llogdev and the loopdev is not formatted with
SCRATCH_MKFS_OPTIONS, so this won't work to disable the concurrency= mkfs options if fstests is being run withSCRATCH_LOGDEVset.
Arrg. I hate our option parsing mess. Any good idea how we'd get the desired result? Unless there's something easy I'd be tempted to leave it to the first person using an external log device and large cpu counts..
Re: [PATCH 2/2] xfs/216: disable all concurrency scaling
Quoted 29 lines · Christoph Hellwig
Quoted 12 lines · Darrick J. Wong
Quoted 11 lines · Christoph Hellwig
This test currently disables log concurrency scaling, but even the data device concurrency scaling can create mismatching output on systems with a large CPU count.
Reported-by Shin'ichiro Kawasaki <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>Tested-by Shin'ichiro Kawasaki <[email protected]>--- tests/xfs/216 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)Quoted 15 lines
tests/xfs/216 _cleanup() · line 26
Quoted 10 lines
tests/xfs/216 +1
-1@@ -23,7 +23,7 @@ _cleanup() _require_scratch _scratch_mkfs_xfs >/dev/null 2>&1 if _scratch_mkfs_xfs_supports_concurrency -l >> $seqres.full 2>&1; then - loop_mkfs_opts="-l concurrency=0" + loop_mkfs_opts="-d concurrency=0 -l concurrency=0 -r concurrency=0"/me notes that -lconcurrency is not compatible with -llogdev and the loopdev is not formatted with
SCRATCH_MKFS_OPTIONS, so this won't work to disable the concurrency= mkfs options if fstests is being run withSCRATCH_LOGDEVset.
Quoted 4 lines
Arrg. I hate our option parsing mess. Any good idea how we'd get the desired result? Unless there's something easy I'd be tempted to leave it to the first person using an external log device and large cpu counts..
Yeah, that's me. :)
I just sent my latest batch of random fixes:
https://lore.kernel.org/fstests/178240619625.1665857.3954516357764795627.stgit@frogsfrogsfrogs/
I don't know that it's necessary to add -dconcurrency=0 for xfs/216 after that (it seemed to work for me) but I don't have any objections to adding more:
if _mkfs_xfs_supported $loop_mkfs_opts -l concurrency=0 $loop_dev &>> $seqres.full; then
loop_mkfs_opts="$loop_mkfs_opts -l concurrency=0"
fi
if _mkfs_xfs_supported $loop_mkfs_opts -d concurrency=0 $loop_dev &>> $seqres.full; then
loop_mkfs_opts="$loop_mkfs_opts -d concurrency=0"
fi
--D
Re: [PATCH 2/2] xfs/216: disable all concurrency scaling
In reply to Christoph HellwigI'll retract this patch in favour of the btrfs fix from Darrick. We'll still need patch 1, though.
Re: [PATCH 2/2] xfs/216: disable all concurrency scaling
Quoted 2 lines · Christoph Hellwig
I'll retract this patch in favour of the btrfs fix from Darrick. We'll still need patch 1, though.
Let me make sure I have the current status mapped out correctly. So we should apply/merge the following patches:
[PATCH 1/2] common/xfs: fix _scratch_mkfs_xfs_supports_concurrency [PATCH 2/2] xfs/21[67]: fix mkfs log concurrency detection [PATCH 1/2] xfs/078: disable all concurrency scaling
And revert:
[PATCH 2/2] xfs/216: disable all concurrency scaling
Is that correct?
Quoted 1 line
Re: [PATCH 2/2] xfs/216: disable all concurrency scaling
Quoted 7 lines · Zorro Lang
Quoted 2 lines · Christoph Hellwig
I'll retract this patch in favour of the btrfs fix from Darrick. We'll still need patch 1, though.
Let me make sure I have the current status mapped out correctly. So we should apply/merge the following patches:
Quoted 8 lines
[PATCH 1/2] common/xfs: fix
_scratch_mkfs_xfs_supports_concurrency[PATCH 2/2] xfs/21[67]: fix mkfs log concurrency detection [PATCH 1/2] xfs/078: disable all concurrency scalingAnd revert:
[PATCH 2/2] xfs/216: disable all concurrency scalingIs that correct?
Sounds fine to me.
--D
Re: [PATCH 2/2] xfs/216: disable all concurrency scaling
Quoted 7 lines · Darrick J. Wong
Quoted 7 lines
Let me make sure I have the current status mapped out correctly. So we should apply/merge the following patches:
[PATCH 1/2] common/xfs: fix
_scratch_mkfs_xfs_supports_concurrency[PATCH 2/2] xfs/21[67]: fix mkfs log concurrency detection [PATCH 1/2] xfs/078: disable all concurrency scaling
Quoted 6 lines
Quoted 4 lines
And revert:
[PATCH 2/2] xfs/216: disable all concurrency scalingIs that correct?
Sounds fine to me.
Same here.