common/rc: add _parse_size_from_string
PATCH 1/2 common/rc: add _parse_size_from_string
Add a helper to convert size value to bytes. This is used to handle value as bytes, such as 4k to 4096.
--- common/rc | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+)
common/rc +48 -0
@@ -1028,6 +1028,54 @@ _check_minimal_fs_size() fi } +# Convert size value to bytes +# _parse_size_from_string <size> +_parse_size_from_string() +{ + local str=$1 + local mult=1 + local size + local endchar + + if [[ $str =~ ^[0-9]+[a-zA-Z]$ ]] ; then + size=${str:: -1} + endchar=${str: -1} + case $endchar in + e|E) + mult=$(($mult * 1024)) + ;& + p|P) + mult=$(($mult * 1024)) + ;& + t|T) + mult=$(($mult * 1024)) + ;& + g|G) + mult=$(($mult * 1024)) + ;& + m|M) + mult=$(($mult * 1024)) + ;& + k|K) + mult=$(($mult * 1024)) + ;& + b|B) + ;; + *) + echo "unknown size descriptor $endchar" + exit 1 + esac + elif [[ $str =~ ^[0-9]+$ ]] ; then + size=$str + else + echo "size value $str is invalid" + exit 1 + fi + + size=$(($size * $mult)) + echo $size +} + # Create fs of certain size on scratch device # _scratch_mkfs_sized <size in bytes> [optional blocksize] _scratch_mkfs_sized()
Signature · 1 line
2.35.3
PATCH 2/2 common/rc: fix input value to _scratch_mkfs_sized
_scratch_mkfs_sized only receive integer number of bytes as a valid input. But if the MKFS_OPTIONS variable exists, it will use the value of block size in MKFS_OPTIONS to override input. In case of MKFS_OPTIONS="-b 4k", would result in blocksize=4 but not 4096. This will give errors to ext2/3/4 etc, and brings potential bugs to xfs or btrfs.
In addition, since we can receive various strings, so remove integer number check.
--- common/rc | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-)
common/rc +6 -12
@@ -1077,7 +1077,7 @@ _parse_size_from_string() } # Create fs of certain size on scratch device -# _scratch_mkfs_sized <size in bytes> [optional blocksize] +# _scratch_mkfs_sized <size> [optional blocksize] _scratch_mkfs_sized() { local fssize=$1
@@ -1086,13 +1086,13 @@ _scratch_mkfs_sized() case $FSTYP in xfs) - def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?size= ?+([0-9]+).*/\1/p'` + def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?size= ?+([0-9]+[a-zA-Z]?).*/\1/p'` ;; btrfs) - def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-s ?+([0-9]+).*/\1/p'` + def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-s ?+([0-9]+[a-zA-Z]?).*/\1/p'` ;; ext2|ext3|ext4|ext4dev|udf|reiser4|ocfs2|reiserfs) - def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?+([0-9]+).*/\1/p'` + def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?+([0-9]+[a-zA-Z]?).*/\1/p'` ;; jfs) def_blksz=4096
@@ -1101,14 +1101,8 @@ _scratch_mkfs_sized() [ -n "$def_blksz" ] && blocksize=$def_blksz [ -z "$blocksize" ] && blocksize=4096 - - local re='^[0-9]+$' - if ! [[ $fssize =~ $re ]] ; then - _notrun "error: _scratch_mkfs_sized: fs size \"$fssize\" not an integer." - fi - if ! [[ $blocksize =~ $re ]] ; then - _notrun "error: _scratch_mkfs_sized: block size \"$blocksize\" not an integer." - fi + blocksize=$(_parse_size_from_string $blocksize) + fssize=$(_parse_size_from_string $fssize) local blocks=`expr $fssize / $blocksize`
Signature · 1 line
2.35.3
Re: [PATCH 2/2] common/rc: fix input value to _scratch_mkfs_sized
Quoted 4 lines · An Long
_scratch_mkfs_sizedonly receive integer number of bytes as a valid input. But if theMKFS_OPTIONSvariable exists, it will use the value of block size inMKFS_OPTIONSto override input. In case of MKFS_OPTIONS="-b 4k", would result in blocksize=4 but not 4096. This
Hi An,
Good catch. Can you provide an example that a case fails on this issue, and then it tests passed after having this patch? I'm wondering why no one notice that before.
Quoted 45 lines
will give errors to ext2/3/4 etc, and brings potential bugs to xfs or btrfs.
In addition, since we can receive various strings, so remove integer number check.
Signed-off-by An Long <[email protected]>--- common/rc | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-)common/rc +4
-4@@ -1077,7 +1077,7 @@ _parse_size_from_string() } # Create fs of certain size on scratch device -# _scratch_mkfs_sized <size in bytes> [optional blocksize] +# _scratch_mkfs_sized <size> [optional blocksize] _scratch_mkfs_sized() { local fssize=$1@@ -1086,13 +1086,13 @@ _scratch_mkfs_sized() case $FSTYP in xfs) - def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?size= ?+([0-9]+).*/\1/p'` + def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?size= ?+([0-9]+[a-zA-Z]?).*/\1/p'` ;; btrfs) - def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-s ?+([0-9]+).*/\1/p'` + def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-s ?+([0-9]+[a-zA-Z]?).*/\1/p'` ;; ext2|ext3|ext4|ext4dev|udf|reiser4|ocfs2|reiserfs) - def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?+([0-9]+).*/\1/p'` + def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?+([0-9]+[a-zA-Z]?).*/\1/p'` ;; jfs) def_blksz=4096@@ -1101,14 +1101,8 @@ _scratch_mkfs_sized() [ -n "$def_blksz" ] && blocksize=$def_blksz [ -z "$blocksize" ] && blocksize=4096
common/rc _scratch_mkfs_sized() · line 1105
Quoted 10 lines
common/rc +2
-8- - local re='^[0-9]+$' - if ! [[ $fssize =~ $re ]] ; then - _notrun "error: _scratch_mkfs_sized: fs size \"$fssize\" not an integer." - fi - if ! [[ $blocksize =~ $re ]] ; then - _notrun "error: _scratch_mkfs_sized: block size \"$blocksize\" not an integer." - fi + blocksize=$(_parse_size_from_string $blocksize) + fssize=$(_parse_size_from_string $fssize)
For now, the _parse_size_from_string is only used for this patch, I think these two patches can be merged into one patch, as it trys to fix one single problem.
Thanks,
Zorro
Quoted 6 lines
common/rc +0
-0local blocks=`expr $fssize / $blocksize`Signature · 1 line
2.35.3
Quoted 13 lines · An Long
Add a helper to convert size value to bytes. This is used to handle value as bytes, such as 4k to 4096.
Signed-off-by An Long <[email protected]>--- common/rc | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+)common/rc +0
-0
common/rc _check_minimal_fs_size() · line 1033
Quoted 6 lines
common/rc +3
-0@@ -1028,6 +1028,54 @@ _check_minimal_fs_size() fi } +# Convert size value to bytes +# _parse_size_from_string <size> +_parse_size_from_string()
I think this name is better to be shorter, as it might be a lower level common function. How about _convert_num() or _cvtnum()?
Quoted 32 lines
common/rc +32
-0+{ + local str=$1 + local mult=1 + local size + local endchar + + if [[ $str =~ ^[0-9]+[a-zA-Z]$ ]] ; then + size=${str:: -1} + endchar=${str: -1} + case $endchar in + e|E) + mult=$(($mult * 1024)) + ;& + p|P) + mult=$(($mult * 1024)) + ;& + t|T) + mult=$(($mult * 1024)) + ;& + g|G) + mult=$(($mult * 1024)) + ;& + m|M) + mult=$(($mult * 1024)) + ;& + k|K) + mult=$(($mult * 1024)) + ;& + b|B) + ;; + *) + echo "unknown size descriptor $endchar"
common/rc _check_minimal_fs_size() · line 1075
Quoted 10 lines
common/rc +10
-0+ exit 1 + esac + elif [[ $str =~ ^[0-9]+$ ]] ; then + size=$str + else + echo "size value $str is invalid" + exit 1 + fi + + size=$(($size * $mult))
^ ^
I think above two '$' isn't necessary, although it doesn't make something wrong if keep it. Same as all above (($mult * 1024)) .
Thanks,
Zorro
^
Quoted 9 lines
common/rc +3
-0+ echo $size +} + # Create fs of certain size on scratch device # _scratch_mkfs_sized <size in bytes> [optional blocksize] _scratch_mkfs_sized()Signature · 1 line
2.35.3
Re: [PATCH 2/2] common/rc: fix input value to _scratch_mkfs_sized
In reply to Zorro LangQuoted 8 lines · Zorro Lang
Quoted 6 lines · An Long
_scratch_mkfs_sizedonly receive integer number of bytes as a valid input. But if theMKFS_OPTIONSvariable exists, it will use the value of
block size inMKFS_OPTIONSto override input. In case of MKFS_OPTIONS="-b 4k", would result in blocksize=4 but not 4096. This
Quoted 8 lines
Hi An,
Good catch. Can you provide an example that a case fails on this issue, and
then it tests passed after having this patch? I'm wondering why no one notice
that before.
For example, I set MKFS_OPTIONS="-b 4k" and then run generic/416 on ext4 filesystem.
# ./check tests/generic/416 FSTYP -- ext4 PLATFORM -- Linux/x86_64 bogon 5.3.18-57-default #1 SMP Wed Apr 28 10:54:41 UTC 2021 (ba3c2e9) MKFS_OPTIONS -- -b 4k /dev/loop1 MOUNT_OPTIONS -- -o acl,user_xattr /dev/loop1 /mnt/scratch
generic/416 90s ... [failed, exit status 1]- output mismatch (see /home/lan/work/xfstests/results//generic/416.out.bad)
--- tests/generic/416.out 2021-06-25 23:18:09.595254118 +0800
+++ /home/lan/work/xfstests/results//generic/416.out.bad 2022-06-15 16:33:19.584206316 +0800
@@ -1,3 +1,4 @@
QA output created by 416
-wrote 16777216/16777216 bytes at offset 0
-XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +mount: /mnt/scratch: wrong fs type, bad option, bad superblock on/dev/loop1, missing codepage or helper program, or other error.
+mount -o acl,user_xattr /dev/loop1 /mnt/scratch failed +(see /home/lan/work/xfstests/results//generic/416.full for
details)
...
(Run 'diff -u /home/lan/work/xfstests/tests/generic/416.out/home/lan/work/xfstests/results//generic/416.out.bad' to see the entire diff)
Ran: generic/416
Failures: generic/416
Failed 1 of 1 tests
But the more common result as below:
# ./check tests/generic/416 FSTYP -- ext4 PLATFORM -- Linux/x86_64 bogon 5.3.18-57-default #1 SMP Wed Apr 28 10:54:41 UTC 2021 (ba3c2e9) MKFS_OPTIONS -- -b 4k /dev/loop1 MOUNT_OPTIONS -- -o acl,user_xattr /dev/loop1 /mnt/scratch
umount: /mnt/scratch: target is busy.
generic/416 77s ... 90s
Ran: generic/416
Passed all 1 tests
Test passed due to _require_scratch_nocheck() ignore the umount error. However, "mkfs.ext4: invalid block size - 4" was found in the 416.full
Apart from this, the error only occurs when MKFS_OPTIONS is set, and doesn't affect xfs or btrfs. Since ext4 only supports 4k blocksize on most platforms, we usually don't set blocksize in MKFS_OPTIONS.
I think that's why this problem hasn't been found before. In fact, I found this issue when I tested on ppc64 with 64k kernel and 4k subblocksize fs.
Quoted 62 lines
Quoted 62 lines
will give errors to ext2/3/4 etc, and brings potential bugs to xfs or
btrfs.In addition, since we can receive various strings, so remove integer
number check.Signed-off-by An Long <[email protected]>--- common/rc | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-)common/rc +1
-2@@ -1077,7 +1077,7 @@ _parse_size_from_string() } # Create fs of certain size on scratch device -# _scratch_mkfs_sized <size in bytes> [optional blocksize] +# _scratch_mkfs_sized <size> [optional blocksize] _scratch_mkfs_sized() { local fssize=$1@@ -1086,13 +1086,13 @@ _scratch_mkfs_sized() case $FSTYP in xfs) - def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?size=?+([0-9]+).*/\1/p'`
common/rc +1
-0+ def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?size=?+([0-9]+[a-zA-Z]?).*/\1/p'`
common/rc +0
-1;; btrfs) - def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-s ?+([0-9]+).*/\1/p'`
common/rc +1
-0+ def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-s ?+([0-9]+[a-zA-Z]?).*/\1/p'`
common/rc +0
-1;; ext2|ext3|ext4|ext4dev|udf|reiser4|ocfs2|reiserfs) - def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?+([0-9]+).*/\1/p'`
common/rc +1
-0+ def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?+([0-9]+[a-zA-Z]?).*/\1/p'`
common/rc +0
-4;; jfs) def_blksz=4096@@ -1101,14 +1101,8 @@ _scratch_mkfs_sized() [ -n "$def_blksz" ] && blocksize=$def_blksz [ -z "$blocksize" ] && blocksize=4096 - - local re='^[0-9]+$' - if ! [[ $fssize =~ $re ]] ; then - _notrun "error: _scratch_mkfs_sized: fs size\"$fssize\" not an integer."
common/rc +0
-3- fi - if ! [[ $blocksize =~ $re ]] ; then - _notrun "error: _scratch_mkfs_sized: block size\"$blocksize\" not an integer."
Quoted 9 lines
common/rc _scratch_mkfs_sized() · line 1105
Quoted 3 lines
common/rc +2
-1- fi + blocksize=$(_parse_size_from_string $blocksize) + fssize=$(_parse_size_from_string $fssize)For now, the
_parse_size_from_stringis only used for this patch, I think these
two patches can be merged into one patch, as it trys to fix one single problem.
There are a total of two logical changes. In particular it also include a Function update. So it's better to keep two separate patch. But I'll add patch depends on info to commit message.
Thank you for your time!
Quoted 9 lines
Thanks,
ZorroQuoted 6 lines
local blocks=
expr $fssize / $blocksizeSignature · 1 line
2.35.3
Signature · 2 lines
An Long <[email protected]> SUSE QE LSG, QE 2, Beijing
Quoted 17 lines · Zorro Lang
Quoted 16 lines · An Long
Add a helper to convert size value to bytes. This is used to handle value as bytes, such as 4k to 4096.
Signed-off-by An Long <[email protected]>--- common/rc | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+)common/rc +0
-0@@ -1028,6 +1028,54 @@ _check_minimal_fs_size() fi }
Quoted 8 lines
common/rc _check_minimal_fs_size() · line 1033
Quoted 3 lines
common/rc +3
-0+# Convert size value to bytes +# _parse_size_from_string <size> +_parse_size_from_string()I think this name is better to be shorter, as it might be a lower level common
function. How about_convert_num()or_cvtnum()?
Rename _parse_size_from_string() to _parse_size_tring(). So that as much as possible to keep function meaning.
Quoted 48 lines
Quoted 33 lines
common/rc +32
-0+{ + local str=$1 + local mult=1 + local size + local endchar + + if [[ $str =~ ^[0-9]+[a-zA-Z]$ ]] ; then + size=${str:: -1} + endchar=${str: -1} + case $endchar in + e|E) + mult=$(($mult * 1024)) + ;& + p|P) + mult=$(($mult * 1024)) + ;& + t|T) + mult=$(($mult * 1024)) + ;& + g|G) + mult=$(($mult * 1024)) + ;& + m|M) + mult=$(($mult * 1024)) + ;& + k|K) + mult=$(($mult * 1024)) + ;& + b|B) + ;; + *) + echo "unknown size descriptor$endchar"
common/rc _check_minimal_fs_size() · line 1075
Quoted 10 lines
common/rc +10
-0+ exit 1 + esac + elif [[ $str =~ ^[0-9]+$ ]] ; then + size=$str + else + echo "size value $str is invalid" + exit 1 + fi + + size=$(($size * $mult))^ ^
I think above two '$' isn't necessary, although it doesn't make something
wrong if keep it. Same as all above (($mult * 1024)) .
Quoted 5 lines
Thanks,
Zorro^
Removed.
Quoted 9 lines
Quoted 9 lines
common/rc +3
-0+ echo $size +} + # Create fs of certain size on scratch device # _scratch_mkfs_sized <size in bytes> [optional blocksize] _scratch_mkfs_sized()Signature · 1 line
2.35.3
Signature · 2 lines
An Long <[email protected]> SUSE QE LSG, QE 2, Beijing