Fixes for mkfs.xfs with high amount of CPUs and SSDs
v2 cover Fixes for mkfs.xfs with high amount of CPUs and SSDs
Since xfsprogs v6.13.0 mkfs.xfs scales the log size on SSDs with the amount of CPUs. This leads to various failures from ENOSPC in generic/102 and generic/172 (the xfs log can take up to 256MB on 1GB test FS) or depleting the dmthin metadata area in generic/347. Downsize of the percentage change is we cannot longer check the exact amount of data xfs_io wrote in the generic/102 test.
The xfs/21{6,7} are testing the old behavior before the concurrency option was introduced. Forcing the old behavior on newer mkfs.xfs.
Lukas Herbolt (4):
common/rc: Add helper to calculate percentage of free space available common/xfs: helper function to check if -l/-d/-r concurrency flags. generic/{102,172,347}: Adapt test for XFS on systems with 128+CPUs +
SSDs
xfs/21{6,7} Use default -l concurrency=0 on mkfs.xfs that supports it
common/rc | 10 ++++++++++ common/xfs | 9 +++++++++ tests/generic/102 | 5 +++-- tests/generic/102.out | 20 ++++++++++---------- tests/generic/172 | 8 ++++---- tests/generic/347 | 9 ++++++++- tests/xfs/216 | 6 +++++- tests/xfs/217 | 7 ++++++- 8 files changed, 55 insertions(+), 19 deletions(-)
Signature · 1 line
2.54.0
PATCH v2 1/4 common/rc: Add helper to calculate percentage of free space available
It calculates percentage of filesystem available space and returns it in MB. This helper will be used in the following test/generic fixes.
---
changes v2:
spelling and description
common/rc | 10 ++++++++++ 1 file changed, 10 insertions(+)
common/rc +10 -0
@@ -6189,6 +6189,16 @@ _require_fanotify_ioerrors() _notrun "$FSTYP does not support fanotify ioerrors" } +# Computes a percentage of the available space in a filesystem and +# returns that quantity in MB. The percentage must not contain a percent +# sign ("%"). +# usage example: +# _mb_pct_of_available_space <mount point> <percent> +# +_mb_pct_of_available_space() +{ + _df_device $1 | $AWK_PROG -v pct=$2 '{printf "%.f", (($5*(pct/100))/1024)}' +} ################################################################################ # make sure this script returns success /bin/true
Signature · 1 line
2.54.0
Re: [PATCH v2 1/4] common/rc: Add helper to calculate percentage of free space available
[fix linux-xfs cc on everything]
Quoted 4 lines · Lukas Herbolt
It calculates percentage of filesystem available space and returns it in MB. This helper will be used in the following test/generic fixes.
Signed-off-by Lukas Herbolt <[email protected]>
Looks fine to me
--D
Quoted 31 lines
---
changes v2:
spelling and descriptioncommon/rc | 10 ++++++++++ 1 file changed, 10 insertions(+)common/rc +10
-0@@ -6189,6 +6189,16 @@ _require_fanotify_ioerrors() _notrun "$FSTYP does not support fanotify ioerrors" } +# Computes a percentage of the available space in a filesystem and +# returns that quantity in MB. The percentage must not contain a percent +# sign ("%"). +# usage example: +# _mb_pct_of_available_space <mount point> <percent> +# +_mb_pct_of_available_space() +{ + _df_device $1 | $AWK_PROG -v pct=$2 '{printf "%.f", (($5*(pct/100))/1024)}' +} ################################################################################ # make sure this script returns success /bin/trueSignature · 1 line
2.54.0
Re: [PATCH v2 1/4] common/rc: Add helper to calculate percentage of free space available
In reply to Lukas HerboltQuoted 17 lines · Lukas Herbolt
It calculates percentage of filesystem available space and returns it in MB. This helper will be used in the following test/generic fixes.
Signed-off-by Lukas Herbolt <[email protected]>---
changes v2:
spelling and descriptioncommon/rc | 10 ++++++++++ 1 file changed, 10 insertions(+)common/rc +0
-0@@ -6189,6 +6189,16 @@ _require_fanotify_ioerrors() _notrun "$FSTYP does not support fanotify ioerrors" }
common/rc _require_fanotify_ioerrors() · line 6200
Quoted 10 lines
common/rc +9
-0+# Computes a percentage of the available space in a filesystem and +# returns that quantity in MB. The percentage must not contain a percent +# sign ("%"). +# usage example: +# _mb_pct_of_available_space <mount point> <percent> +# +_mb_pct_of_available_space() +{ + _df_device $1 | $AWK_PROG -v pct=$2 '{printf "%.f", (($5*(pct/100))/1024)}'
Good to me,
Quoted 7 lines
common/rc +1
-0+} ################################################################################ # make sure this script returns success /bin/trueSignature · 1 line
2.54.0
PATCH v2 2/4 common/xfs: helper function to check if -l/-d/-r concurrecy flags.
In reply to Lukas HerboltCheck if the requested flag is supported by the mkfs.xfs on scratch device.
---
---
changes v2:
-l/-d were introduced at same release no need to check both
common/xfs | 9 +++++++++ 1 file changed, 9 insertions(+)
common/xfs +9 -0
@@ -2399,3 +2399,12 @@ _require_xfs_healer() _xfs_healer --supported "$@" &>/dev/null || \ _notrun "health monitoring not supported on this kernel" } + +# -l/-d concurrency came in same xfsprogs release v6.7 + _scratch_mkfs_xfs_supports_concurrency() +{ + local arg="${1:-d}" + + test "$arg" = "-l" && arg="-d" + _scratch_mkfs_xfs_supported "$arg" concurrency=0 +}
Signature · 1 line
2.54.0
Re: [PATCH v2 2/4] common/xfs: helper function to check if -l/-d/-r concurrecy flags.
Quoted 8 lines · Lukas Herbolt
Check if the requested flag is supported by the mkfs.xfs on scratch device.
Signed-off-by Lukas Herbolt <[email protected]>---
---
changes v2:
-l/-d were introduced at same release no need to check both
--D
Quoted 24 lines
common/xfs | 9 +++++++++ 1 file changed, 9 insertions(+)common/xfs +9
-0@@ -2399,3 +2399,12 @@ _require_xfs_healer() _xfs_healer --supported "$@" &>/dev/null || \ _notrun "health monitoring not supported on this kernel" } + +# -l/-d concurrency came in same xfsprogs release v6.7 + _scratch_mkfs_xfs_supports_concurrency() +{ + local arg="${1:-d}" + + test "$arg" = "-l" && arg="-d" + _scratch_mkfs_xfs_supported "$arg" concurrency=0 +}Signature · 1 line
2.54.0
Re: [PATCH v2 2/4] common/xfs: helper function to check if -l/-d/-r concurrecy flags.
In reply to Lukas HerboltQuoted 17 lines · Lukas Herbolt
Check if the requested flag is supported by the mkfs.xfs on scratch device.
Signed-off-by Lukas Herbolt <[email protected]>---
---
changes v2:
-l/-d were introduced at same release no need to check bothcommon/xfs | 9 +++++++++ 1 file changed, 9 insertions(+)common/xfs +0
-0@@ -2399,3 +2399,12 @@ _require_xfs_healer() _xfs_healer --supported "$@" &>/dev/null || \
common/xfs _require_xfs_healer() · line 2409
Quoted 10 lines
common/xfs +8
-0_notrun "health monitoring not supported on this kernel" } + +# -l/-d concurrency came in same xfsprogs release v6.7 + _scratch_mkfs_xfs_supports_concurrency() +{ + local arg="${1:-d}" + + test "$arg" = "-l" && arg="-d" + _scratch_mkfs_xfs_supported "$arg" concurrency=0
Fair enough. It seems this 'compromise' is a workaround to avoid passing a log device to _scratch_mkfs_xfs_supported.
Quoted 4 lines
common/xfs +1
-0+}Signature · 1 line
2.54.0
PATCH v2 3/4 generic/{102,172,347}: Adapt test for XFS on systems with 128+CPUs + SSDs
In reply to Lukas HerboltXFS will now scale by default its log size according the amount of CPUs. This can significantly increase the log area and using hard-coded size of file size will make the test file. Using the X percent of the free space to create the file will let the test proceed.
For generic/102 we also have to change the 102.out as we cannot know the amount written data and it has to be filtered out.
For generic/172 we just use the new function to calculate file big enough to consume most of the available space.
The generic/347 will use -l concurrency=0 in case it is being run on the XFS filesystem and mkfs.xfs already supports this option.
---
changes v2:
test: generic/102 change the file size to 80% as it's closer to
original size of 800MB
tests/generic/102 | 5 +++-- tests/generic/102.out | 20 ++++++++++---------- tests/generic/172 | 8 ++++---- tests/generic/347 | 9 ++++++++- 4 files changed, 25 insertions(+), 17 deletions(-)
tests/generic/102 +3 -2
@@ -23,12 +23,13 @@ _require_scratch dev_size=$((1024 * 1024 * 1024)) # 1GB filesystem _scratch_mkfs_sized $dev_size >>$seqres.full 2>&1 _scratch_mount +file_size=`_mb_pct_of_available_space $SCRATCH_DEV 80` for ((i = 0; i < 10; i++)); do echo "loop $i" >>$seqres.full - $XFS_IO_PROG -f -c "pwrite -b 1m 0 800m" "$SCRATCH_MNT"/file | \ -_filter_xfs_io | _filter_scratch + $XFS_IO_PROG -f -c "pwrite -b 1m 0 ${file_size}m" "$SCRATCH_MNT"/file | \ +_filter_xfs_io_numbers | _filter_scratch rm -f "$SCRATCH_MNT"/file done
tests/generic/102.out +10 -10
@@ -1,21 +1,21 @@ QA output created by 102 -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
tests/generic/172 +4 -4
@@ -37,15 +37,15 @@ echo "Reformat with appropriate size" blksz="$(_get_block_size $testdir)" _scratch_unmount -file_size=$((768 * 1024 * 1024)) fs_size=$((1024 * 1024 * 1024)) _scratch_mkfs_sized $fs_size >> $seqres.full 2>&1 _scratch_mount >> $seqres.full 2>&1 rm -rf $testdir mkdir $testdir +file_size=`_mb_pct_of_available_space $SCRATCH_DEV 70` echo "Create a big file and reflink it" -_pwrite_byte 0x61 0 $file_size $testdir/bigfile >> $seqres.full 2>&1 +_pwrite_byte 0x61 0 ${file_size}m $testdir/bigfile >> $seqres.full 2>&1 _cp_reflink $testdir/bigfile $testdir/clonefile _scratch_sync
@@ -54,7 +54,7 @@ _fill_fs $fs_size $testdir/space $blksz 0 >> $seqres.full 2>&1 _scratch_sync echo "CoW the big file" -out="$(_pwrite_byte 0x62 0 $file_size $testdir/bigfile 2>&1 | \ +out="$(_pwrite_byte 0x62 0 ${file_size}m $testdir/bigfile 2>&1 | \ _filter_xfs_io_error)" echo ${out} | grep -q "No space left on device" || echo "CoW should have failed with ENOSPC" echo ${out} >> $seqres.full 2>&1
@@ -63,7 +63,7 @@ echo ${out} echo "Remount and try CoW again" _scratch_cycle_mount -out="$(_pwrite_byte 0x62 0 $file_size $testdir/bigfile 2>&1 | \ +out="$(_pwrite_byte 0x62 0 ${file_size}m $testdir/bigfile 2>&1 | \ _filter_xfs_io_error)" echo ${out} | grep -q "No space left on device" || echo "CoW should have failed with ENOSPC" echo ${out} >> $seqres.full 2>&1
tests/generic/347 +8 -1
@@ -14,6 +14,13 @@ BACKING_SIZE=$((500 * 1024 * 1024 / 512)) # 500M VIRTUAL_SIZE=$((10 * $BACKING_SIZE)) # 5000M GROW_SIZE=$((100 * 1024 * 1024 / 512)) # 100M +dmthin_mkfs_opts="" +if [ $FSTYP == "xfs" ]; then + if _scratch_mkfs_xfs_supports_concurrency -l >> $seqres.full 2>&1; then + dmthin_mkfs_opts="-l concurrency=0" + fi +fi + # Override the default cleanup function. _cleanup() {
@@ -25,7 +32,7 @@ _setup_thin() { _dmthin_init $BACKING_SIZE $VIRTUAL_SIZE _dmthin_set_queue - _dmthin_mkfs + _dmthin_mkfs $dmthin_mkfs_opts _dmthin_mount }
Signature · 1 line
2.54.0
Re: [PATCH v2 3/4] generic/{102,172,347}: Adapt test for XFS on systems with 128+CPUs + SSDs
Quoted 11 lines · Lukas Herbolt
XFS will now scale by default its log size according the amount of CPUs. This can significantly increase the log area and using hard-coded size of file size will make the test file. Using the X percent of the free space to create the file will let the test proceed.
For generic/102 we also have to change the 102.out as we cannot know the amount written data and it has to be filtered out.
For generic/172 we just use the new function to calculate file big enough to consume most of the available space.
Quoted 8 lines
The generic/347 will use -l concurrency=0 in case it is being run on the XFS filesystem and mkfs.xfs already supports this option.
Signed-off-by Lukas Herbolt <[email protected]>---
changes v2:
test: generic/102 change the file size to 80% as it's closer to
original size of 800MB
Cool, thanks.
--D
Quoted 133 lines
tests/generic/102 | 5 +++-- tests/generic/102.out | 20 ++++++++++---------- tests/generic/172 | 8 ++++---- tests/generic/347 | 9 ++++++++- 4 files changed, 25 insertions(+), 17 deletions(-)tests/generic/102 +3
-2@@ -23,12 +23,13 @@ _require_scratch dev_size=$((1024 * 1024 * 1024)) # 1GB filesystem _scratch_mkfs_sized $dev_size >>$seqres.full 2>&1 _scratch_mount +file_size=`_mb_pct_of_available_space $SCRATCH_DEV 80` for ((i = 0; i < 10; i++)); do echo "loop $i" >>$seqres.full - $XFS_IO_PROG -f -c "pwrite -b 1m 0 800m" "$SCRATCH_MNT"/file | \ -_filter_xfs_io | _filter_scratch + $XFS_IO_PROG -f -c "pwrite -b 1m 0 ${file_size}m" "$SCRATCH_MNT"/file | \ +_filter_xfs_io_numbers | _filter_scratch rm -f "$SCRATCH_MNT"/file donetests/generic/102.out +10
-10@@ -1,21 +1,21 @@ QA output created by 102 -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)tests/generic/172 +4
-4@@ -37,15 +37,15 @@ echo "Reformat with appropriate size" blksz="$(_get_block_size $testdir)" _scratch_unmount -file_size=$((768 * 1024 * 1024)) fs_size=$((1024 * 1024 * 1024)) _scratch_mkfs_sized $fs_size >> $seqres.full 2>&1 _scratch_mount >> $seqres.full 2>&1 rm -rf $testdir mkdir $testdir +file_size=`_mb_pct_of_available_space $SCRATCH_DEV 70` echo "Create a big file and reflink it" -_pwrite_byte 0x61 0 $file_size $testdir/bigfile >> $seqres.full 2>&1 +_pwrite_byte 0x61 0 ${file_size}m $testdir/bigfile >> $seqres.full 2>&1 _cp_reflink $testdir/bigfile $testdir/clonefile _scratch_sync@@ -54,7 +54,7 @@ _fill_fs $fs_size $testdir/space $blksz 0 >> $seqres.full 2>&1 _scratch_sync echo "CoW the big file" -out="$(_pwrite_byte 0x62 0 $file_size $testdir/bigfile 2>&1 | \ +out="$(_pwrite_byte 0x62 0 ${file_size}m $testdir/bigfile 2>&1 | \ _filter_xfs_io_error)" echo ${out} | grep -q "No space left on device" || echo "CoW should have failed with ENOSPC" echo ${out} >> $seqres.full 2>&1@@ -63,7 +63,7 @@ echo ${out} echo "Remount and try CoW again" _scratch_cycle_mount -out="$(_pwrite_byte 0x62 0 $file_size $testdir/bigfile 2>&1 | \ +out="$(_pwrite_byte 0x62 0 ${file_size}m $testdir/bigfile 2>&1 | \ _filter_xfs_io_error)" echo ${out} | grep -q "No space left on device" || echo "CoW should have failed with ENOSPC" echo ${out} >> $seqres.full 2>&1tests/generic/347 +8
-1@@ -14,6 +14,13 @@ BACKING_SIZE=$((500 * 1024 * 1024 / 512)) # 500M VIRTUAL_SIZE=$((10 * $BACKING_SIZE)) # 5000M GROW_SIZE=$((100 * 1024 * 1024 / 512)) # 100M +dmthin_mkfs_opts="" +if [ $FSTYP == "xfs" ]; then + if _scratch_mkfs_xfs_supports_concurrency -l >> $seqres.full 2>&1; then + dmthin_mkfs_opts="-l concurrency=0" + fi +fi + # Override the default cleanup function. _cleanup() {@@ -25,7 +32,7 @@ _setup_thin() { _dmthin_init $BACKING_SIZE $VIRTUAL_SIZE _dmthin_set_queue - _dmthin_mkfs + _dmthin_mkfs $dmthin_mkfs_opts _dmthin_mount }Signature · 1 line
2.54.0
Re: [PATCH v2 3/4] generic/{102,172,347}: Adapt test for XFS on systems with 128+CPUs + SSDs
In reply to Lukas HerboltQuoted 8 lines · Lukas Herbolt
XFS will now scale by default its log size according the amount of CPUs. This can significantly increase the log area and using hard-coded size of file size will make the test file. Using the X percent of the free space to create the file will let the test proceed.
For generic/102 we also have to change the 102.out as we cannot know the amount written data and it has to be filtered out.
Quoted 8 lines
For generic/172 we just use the new function to calculate file big enough to consume most of the available space.
The generic/347 will use -l concurrency=0 in case it is being run on the XFS filesystem and mkfs.xfs already supports this option.
Signed-off-by Lukas Herbolt <[email protected]>---
Looks and tests good to me,
Quoted 135 lines
changes v2:
test: generic/102 change the file size to 80% as it's closer to
original size of 800MBtests/generic/102 | 5 +++-- tests/generic/102.out | 20 ++++++++++---------- tests/generic/172 | 8 ++++---- tests/generic/347 | 9 ++++++++- 4 files changed, 25 insertions(+), 17 deletions(-)tests/generic/102 +3
-2@@ -23,12 +23,13 @@ _require_scratch dev_size=$((1024 * 1024 * 1024)) # 1GB filesystem _scratch_mkfs_sized $dev_size >>$seqres.full 2>&1 _scratch_mount +file_size=`_mb_pct_of_available_space $SCRATCH_DEV 80` for ((i = 0; i < 10; i++)); do echo "loop $i" >>$seqres.full - $XFS_IO_PROG -f -c "pwrite -b 1m 0 800m" "$SCRATCH_MNT"/file | \ -_filter_xfs_io | _filter_scratch + $XFS_IO_PROG -f -c "pwrite -b 1m 0 ${file_size}m" "$SCRATCH_MNT"/file | \ +_filter_xfs_io_numbers | _filter_scratch rm -f "$SCRATCH_MNT"/file donetests/generic/102.out +10
-10@@ -1,21 +1,21 @@ QA output created by 102 -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) -wrote 838860800/838860800 bytes at offset 0 +wrote XXXX/XXXX bytes at offset XXXX XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)tests/generic/172 +4
-4@@ -37,15 +37,15 @@ echo "Reformat with appropriate size" blksz="$(_get_block_size $testdir)" _scratch_unmount -file_size=$((768 * 1024 * 1024)) fs_size=$((1024 * 1024 * 1024)) _scratch_mkfs_sized $fs_size >> $seqres.full 2>&1 _scratch_mount >> $seqres.full 2>&1 rm -rf $testdir mkdir $testdir +file_size=`_mb_pct_of_available_space $SCRATCH_DEV 70` echo "Create a big file and reflink it" -_pwrite_byte 0x61 0 $file_size $testdir/bigfile >> $seqres.full 2>&1 +_pwrite_byte 0x61 0 ${file_size}m $testdir/bigfile >> $seqres.full 2>&1 _cp_reflink $testdir/bigfile $testdir/clonefile _scratch_sync@@ -54,7 +54,7 @@ _fill_fs $fs_size $testdir/space $blksz 0 >> $seqres.full 2>&1 _scratch_sync echo "CoW the big file" -out="$(_pwrite_byte 0x62 0 $file_size $testdir/bigfile 2>&1 | \ +out="$(_pwrite_byte 0x62 0 ${file_size}m $testdir/bigfile 2>&1 | \ _filter_xfs_io_error)" echo ${out} | grep -q "No space left on device" || echo "CoW should have failed with ENOSPC" echo ${out} >> $seqres.full 2>&1@@ -63,7 +63,7 @@ echo ${out} echo "Remount and try CoW again" _scratch_cycle_mount -out="$(_pwrite_byte 0x62 0 $file_size $testdir/bigfile 2>&1 | \ +out="$(_pwrite_byte 0x62 0 ${file_size}m $testdir/bigfile 2>&1 | \ _filter_xfs_io_error)" echo ${out} | grep -q "No space left on device" || echo "CoW should have failed with ENOSPC" echo ${out} >> $seqres.full 2>&1tests/generic/347 +8
-1@@ -14,6 +14,13 @@ BACKING_SIZE=$((500 * 1024 * 1024 / 512)) # 500M VIRTUAL_SIZE=$((10 * $BACKING_SIZE)) # 5000M GROW_SIZE=$((100 * 1024 * 1024 / 512)) # 100M +dmthin_mkfs_opts="" +if [ $FSTYP == "xfs" ]; then + if _scratch_mkfs_xfs_supports_concurrency -l >> $seqres.full 2>&1; then + dmthin_mkfs_opts="-l concurrency=0" + fi +fi + # Override the default cleanup function. _cleanup() {@@ -25,7 +32,7 @@ _setup_thin() { _dmthin_init $BACKING_SIZE $VIRTUAL_SIZE _dmthin_set_queue - _dmthin_mkfs + _dmthin_mkfs $dmthin_mkfs_opts _dmthin_mount }Signature · 1 line
2.54.0
PATCH v2 4/4 xfs/21{6,7} Use default -l concurrency=0 on mkfs.xfs that supports it
In reply to Lukas HerboltThe XFS concurrency optimization breaks the log sizing check on systems with non-rotational disks and high amount of CPUs. Default to the old behavior with -l concurrency=0 and with -d concurrency=0.
---
changes v2:
use default mkfs.xfs for scratch device
remove the double check -l/-d as those were introduced together
tests/xfs/216 | 6 +++++- tests/xfs/217 | 7 ++++++- 2 files changed, 11 insertions(+), 2 deletions(-)
tests/xfs/216 +5 -1
@@ -22,13 +22,17 @@ _cleanup() _require_scratch _scratch_mkfs_xfs >/dev/null 2>&1 +if _scratch_mkfs_xfs_supports_concurrency -l >> $seqres.full 2>&1; then + loop_mkfs_opts="-l concurrency=0" +else + loop_mkfs_opts="" +fi _scratch_mount _require_loop LOOP_IMG=$SCRATCH_MNT/test_fs LOOP_MNT=$SCRATCH_MNT/test_fs_dir -loop_mkfs_opts= $MKFS_XFS_PROG 2>&1 | grep -q rmapbt && \ loop_mkfs_opts="$loop_mkfs_opts -m rmapbt=0" $MKFS_XFS_PROG 2>&1 | grep -q reflink && \
tests/xfs/217 +6 -1
@@ -21,6 +21,11 @@ _cleanup() _require_scratch _scratch_mkfs_xfs >/dev/null 2>&1 +if _scratch_mkfs_xfs_supports_concurrency -l >> $seqres.full 2>&1; then + loop_mkfs_opts="-l concurrency=0 -d concurrency=0" +else + loop_mkfs_opts="" +fi _scratch_mount # 16T mkfs requires a bit over 2G free _require_fs_space $SCRATCH_MNT 2202000
@@ -34,7 +39,7 @@ _do_mkfs() for i in $*; do echo -n "fssize=${i}g " $MKFS_XFS_PROG -f -b size=4096 -l version=2 \ - -d size=${i}g $loop_dev |grep log + -d size=${i}g $loop_dev $loop_mkfs_opts |grep log _mount $loop_dev $LOOP_MNT echo "test write" > $LOOP_MNT/test _unmount $LOOP_MNT > /dev/null 2>&1
Signature · 1 line
2.54.0
Re: [PATCH v2 4/4] xfs/21{6,7} Use default -l concurrency=0 on mkfs.xfs that supports it
Quoted 9 lines · Lukas Herbolt
The XFS concurrency optimization breaks the log sizing check on systems with non-rotational disks and high amount of CPUs. Default to the old behavior with -l concurrency=0 and with -d concurrency=0.
Signed-off-by Lukas Herbolt <[email protected]>---
changes v2:
use default mkfs.xfs for scratch device
remove the double check -l/-d as those were introduced together
Yeah, sorry about breaking that. :/
--D
Quoted 56 lines
tests/xfs/216 | 6 +++++- tests/xfs/217 | 7 ++++++- 2 files changed, 11 insertions(+), 2 deletions(-)tests/xfs/216 +5
-1@@ -22,13 +22,17 @@ _cleanup() _require_scratch _scratch_mkfs_xfs >/dev/null 2>&1 +if _scratch_mkfs_xfs_supports_concurrency -l >> $seqres.full 2>&1; then + loop_mkfs_opts="-l concurrency=0" +else + loop_mkfs_opts="" +fi _scratch_mount _require_loop LOOP_IMG=$SCRATCH_MNT/test_fs LOOP_MNT=$SCRATCH_MNT/test_fs_dir -loop_mkfs_opts= $MKFS_XFS_PROG 2>&1 | grep -q rmapbt && \ loop_mkfs_opts="$loop_mkfs_opts -m rmapbt=0" $MKFS_XFS_PROG 2>&1 | grep -q reflink && \tests/xfs/217 +6
-1@@ -21,6 +21,11 @@ _cleanup() _require_scratch _scratch_mkfs_xfs >/dev/null 2>&1 +if _scratch_mkfs_xfs_supports_concurrency -l >> $seqres.full 2>&1; then + loop_mkfs_opts="-l concurrency=0 -d concurrency=0" +else + loop_mkfs_opts="" +fi _scratch_mount # 16T mkfs requires a bit over 2G free _require_fs_space $SCRATCH_MNT 2202000@@ -34,7 +39,7 @@ _do_mkfs() for i in $*; do echo -n "fssize=${i}g " $MKFS_XFS_PROG -f -b size=4096 -l version=2 \ - -d size=${i}g $loop_dev |grep log + -d size=${i}g $loop_dev $loop_mkfs_opts |grep log _mount $loop_dev $LOOP_MNT echo "test write" > $LOOP_MNT/test _unmount $LOOP_MNT > /dev/null 2>&1Signature · 1 line
2.54.0
Re: [PATCH v2 4/4] xfs/21{6,7} Use default -l concurrency=0 on mkfs.xfs that supports it
In reply to Lukas HerboltQuoted 9 lines · Lukas Herbolt
The XFS concurrency optimization breaks the log sizing check on systems with non-rotational disks and high amount of CPUs. Default to the old behavior with -l concurrency=0 and with -d concurrency=0.
Signed-off-by Lukas Herbolt <[email protected]>---
changes v2:
use default mkfs.xfs for scratch device
remove the double check -l/-d as those were introduced together
This patch is good to me, thanks for fixing.
Quoted 56 lines
tests/xfs/216 | 6 +++++- tests/xfs/217 | 7 ++++++- 2 files changed, 11 insertions(+), 2 deletions(-)tests/xfs/216 +5
-1@@ -22,13 +22,17 @@ _cleanup() _require_scratch _scratch_mkfs_xfs >/dev/null 2>&1 +if _scratch_mkfs_xfs_supports_concurrency -l >> $seqres.full 2>&1; then + loop_mkfs_opts="-l concurrency=0" +else + loop_mkfs_opts="" +fi _scratch_mount _require_loop LOOP_IMG=$SCRATCH_MNT/test_fs LOOP_MNT=$SCRATCH_MNT/test_fs_dir -loop_mkfs_opts= $MKFS_XFS_PROG 2>&1 | grep -q rmapbt && \ loop_mkfs_opts="$loop_mkfs_opts -m rmapbt=0" $MKFS_XFS_PROG 2>&1 | grep -q reflink && \tests/xfs/217 +6
-1@@ -21,6 +21,11 @@ _cleanup() _require_scratch _scratch_mkfs_xfs >/dev/null 2>&1 +if _scratch_mkfs_xfs_supports_concurrency -l >> $seqres.full 2>&1; then + loop_mkfs_opts="-l concurrency=0 -d concurrency=0" +else + loop_mkfs_opts="" +fi _scratch_mount # 16T mkfs requires a bit over 2G free _require_fs_space $SCRATCH_MNT 2202000@@ -34,7 +39,7 @@ _do_mkfs() for i in $*; do echo -n "fssize=${i}g " $MKFS_XFS_PROG -f -b size=4096 -l version=2 \ - -d size=${i}g $loop_dev |grep log + -d size=${i}g $loop_dev $loop_mkfs_opts |grep log _mount $loop_dev $LOOP_MNT echo "test write" > $LOOP_MNT/test _unmount $LOOP_MNT > /dev/null 2>&1Signature · 1 line
2.54.0