Skip to contentSkip to conversation

common/rc: add _parse_size_from_string

PATCH Superseded An Long 2 participants 6 messages Jun 14, 2022 – Jun 16, 2022

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

An Long

author
#0 lore raw

PATCH 1/2 common/rc: add _parse_size_from_string

Add a helper to convert size value to bytes. This is used to handle value as bytes, such as 4k to 4096.

---
 common/rc | 48 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)
common/rc +48 -0
index 3c072c16..22050bc2 100644
@@ -1028,6 +1028,54 @@ _check_minimal_fs_size()
 	fi
 }
 
+# Convert size value to bytes
+# _parse_size_from_string <size>
+_parse_size_from_string()
+{
+        local str=$1
+        local mult=1
+        local size
+        local endchar
+
+        if [[ $str =~ ^[0-9]+[a-zA-Z]$ ]] ; then
+                size=${str:: -1}
+                endchar=${str: -1}
+                case $endchar in
+                e|E)
+                                mult=$(($mult * 1024))
+                                ;&
+                p|P)
+                                mult=$(($mult * 1024))
+                                ;&
+                t|T)
+                                mult=$(($mult * 1024))
+                                ;&
+                g|G)
+                                mult=$(($mult * 1024))
+                                ;&
+                m|M)
+                                mult=$(($mult * 1024))
+                                ;&
+                k|K)
+                                mult=$(($mult * 1024))
+                                ;&
+                b|B)
+                                ;;
+                *)
+                                echo "unknown size descriptor $endchar"
+                                exit 1
+                esac
+        elif [[ $str =~ ^[0-9]+$ ]] ; then
+                size=$str
+        else
+                echo "size value $str is invalid"
+                exit 1
+        fi
+
+        size=$(($size * $mult))
+        echo $size
+}
+
 # Create fs of certain size on scratch device
 # _scratch_mkfs_sized <size in bytes> [optional blocksize]
 _scratch_mkfs_sized()
Signature · 1 line
2.35.3
author
#1 lore raw

PATCH 2/2 common/rc: fix input value to _scratch_mkfs_sized

_scratch_mkfs_sized only receive integer number of bytes as a valid input. But if the MKFS_OPTIONS variable exists, it will use the value of block size in MKFS_OPTIONS to override input. In case of MKFS_OPTIONS="-b 4k", would result in blocksize=4 but not 4096. This will give errors to ext2/3/4 etc, and brings potential bugs to xfs or btrfs.

In addition, since we can receive various strings, so remove integer number check.

---
 common/rc | 18 ++++++------------
 1 file changed, 6 insertions(+), 12 deletions(-)
common/rc +6 -12
index 22050bc2..026007d3 100644
@@ -1077,7 +1077,7 @@ _parse_size_from_string()
 }
 
 # Create fs of certain size on scratch device
-# _scratch_mkfs_sized <size in bytes> [optional blocksize]
+# _scratch_mkfs_sized <size> [optional blocksize]
 _scratch_mkfs_sized()
 {
 	local fssize=$1
@@ -1086,13 +1086,13 @@ _scratch_mkfs_sized()
 
 	case $FSTYP in
 	xfs)
-		def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?size= ?+([0-9]+).*/\1/p'`
+		def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?size= ?+([0-9]+[a-zA-Z]?).*/\1/p'`
 		;;
 	btrfs)
-		def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-s ?+([0-9]+).*/\1/p'`
+		def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-s ?+([0-9]+[a-zA-Z]?).*/\1/p'`
 		;;
 	ext2|ext3|ext4|ext4dev|udf|reiser4|ocfs2|reiserfs)
-		def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?+([0-9]+).*/\1/p'`
+		def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?+([0-9]+[a-zA-Z]?).*/\1/p'`
 		;;
 	jfs)
 		def_blksz=4096
@@ -1101,14 +1101,8 @@ _scratch_mkfs_sized()
 
 	[ -n "$def_blksz" ] && blocksize=$def_blksz
 	[ -z "$blocksize" ] && blocksize=4096
-
-	local re='^[0-9]+$'
-	if ! [[ $fssize =~ $re ]] ; then
-		_notrun "error: _scratch_mkfs_sized: fs size \"$fssize\" not an integer."
-	fi
-	if ! [[ $blocksize =~ $re ]] ; then
-		_notrun "error: _scratch_mkfs_sized: block size \"$blocksize\" not an integer."
-	fi
+	blocksize=$(_parse_size_from_string $blocksize)
+	fssize=$(_parse_size_from_string $fssize)
 
 	local blocks=`expr $fssize / $blocksize`
 
Signature · 1 line
2.35.3
#2 lore raw

Re: [PATCH 2/2] common/rc: fix input value to _scratch_mkfs_sized

Quoted 4 lines · An Long

_scratch_mkfs_sized only receive integer number of bytes as a valid input. But if the MKFS_OPTIONS variable exists, it will use the value of block size in MKFS_OPTIONS to override input. In case of MKFS_OPTIONS="-b 4k", would result in blocksize=4 but not 4096. This

Hi An,

Good catch. Can you provide an example that a case fails on this issue, and then it tests passed after having this patch? I'm wondering why no one notice that before.

Quoted 45 lines

will give errors to ext2/3/4 etc, and brings potential bugs to xfs or btrfs.

In addition, since we can receive various strings, so remove integer number check.

---
 common/rc | 18 ++++++------------
 1 file changed, 6 insertions(+), 12 deletions(-)
common/rc +4 -4
index 22050bc2..026007d3 100644
@@ -1077,7 +1077,7 @@ _parse_size_from_string()
 }
 
 # Create fs of certain size on scratch device
-# _scratch_mkfs_sized <size in bytes> [optional blocksize]
+# _scratch_mkfs_sized <size> [optional blocksize]
 _scratch_mkfs_sized()
 {
 	local fssize=$1
@@ -1086,13 +1086,13 @@ _scratch_mkfs_sized()
 
 	case $FSTYP in
 	xfs)
-		def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?size= ?+([0-9]+).*/\1/p'`
+		def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?size= ?+([0-9]+[a-zA-Z]?).*/\1/p'`
 		;;
 	btrfs)
-		def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-s ?+([0-9]+).*/\1/p'`
+		def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-s ?+([0-9]+[a-zA-Z]?).*/\1/p'`
 		;;
 	ext2|ext3|ext4|ext4dev|udf|reiser4|ocfs2|reiserfs)
-		def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?+([0-9]+).*/\1/p'`
+		def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?+([0-9]+[a-zA-Z]?).*/\1/p'`
 		;;
 	jfs)
 		def_blksz=4096
@@ -1101,14 +1101,8 @@ _scratch_mkfs_sized()
 
 	[ -n "$def_blksz" ] && blocksize=$def_blksz
 	[ -z "$blocksize" ] && blocksize=4096

common/rc _scratch_mkfs_sized() · line 1105

Quoted 10 lines
common/rc +2 -8
-
-	local re='^[0-9]+$'
-	if ! [[ $fssize =~ $re ]] ; then
-		_notrun "error: _scratch_mkfs_sized: fs size \"$fssize\" not an integer."
-	fi
-	if ! [[ $blocksize =~ $re ]] ; then
-		_notrun "error: _scratch_mkfs_sized: block size \"$blocksize\" not an integer."
-	fi
+	blocksize=$(_parse_size_from_string $blocksize)
+	fssize=$(_parse_size_from_string $fssize)

For now, the _parse_size_from_string is only used for this patch, I think these two patches can be merged into one patch, as it trys to fix one single problem.

Thanks,
Zorro

Quoted 6 lines
common/rc +0 -0
 	local blocks=`expr $fssize / $blocksize`
 
Signature · 1 line
2.35.3
#4 lore raw
In reply to An Long
Quoted 13 lines · An Long

Add a helper to convert size value to bytes. This is used to handle value as bytes, such as 4k to 4096.

---
 common/rc | 48 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)
common/rc +0 -0
index 3c072c16..22050bc2 100644

common/rc _check_minimal_fs_size() · line 1033

Quoted 6 lines
common/rc +3 -0
@@ -1028,6 +1028,54 @@ _check_minimal_fs_size()
 	fi
 }
 
+# Convert size value to bytes
+# _parse_size_from_string <size>
+_parse_size_from_string()

I think this name is better to be shorter, as it might be a lower level common function. How about _convert_num() or _cvtnum()?

Quoted 32 lines
common/rc +32 -0
+{
+        local str=$1
+        local mult=1
+        local size
+        local endchar
+
+        if [[ $str =~ ^[0-9]+[a-zA-Z]$ ]] ; then
+                size=${str:: -1}
+                endchar=${str: -1}
+                case $endchar in
+                e|E)
+                                mult=$(($mult * 1024))
+                                ;&
+                p|P)
+                                mult=$(($mult * 1024))
+                                ;&
+                t|T)
+                                mult=$(($mult * 1024))
+                                ;&
+                g|G)
+                                mult=$(($mult * 1024))
+                                ;&
+                m|M)
+                                mult=$(($mult * 1024))
+                                ;&
+                k|K)
+                                mult=$(($mult * 1024))
+                                ;&
+                b|B)
+                                ;;
+                *)
+                                echo "unknown size descriptor $endchar"

common/rc _check_minimal_fs_size() · line 1075

Quoted 10 lines
common/rc +10 -0
+                                exit 1
+                esac
+        elif [[ $str =~ ^[0-9]+$ ]] ; then
+                size=$str
+        else
+                echo "size value $str is invalid"
+                exit 1
+        fi
+
+        size=$(($size * $mult))

^ ^
I think above two '$' isn't necessary, although it doesn't make something wrong if keep it. Same as all above (($mult * 1024)) .

Thanks,
Zorro

                                      ^
Quoted 9 lines
common/rc +3 -0
+        echo $size
+}
+
 # Create fs of certain size on scratch device
 # _scratch_mkfs_sized <size in bytes> [optional blocksize]
 _scratch_mkfs_sized()
Signature · 1 line
2.35.3
author
#3 lore raw

Re: [PATCH 2/2] common/rc: fix input value to _scratch_mkfs_sized

In reply to Zorro Lang
Quoted 8 lines · Zorro Lang
Quoted 6 lines · An Long

_scratch_mkfs_sized only receive integer number of bytes as a valid input. But if the MKFS_OPTIONS variable exists, it will use the value of
block size in MKFS_OPTIONS to override input. In case of MKFS_OPTIONS="-b 4k", would result in blocksize=4 but not 4096. This

Quoted 8 lines

Hi An,

Good catch. Can you provide an example that a case fails on this issue, and
then it tests passed after having this patch? I'm wondering why no one notice
that before.

For example, I set MKFS_OPTIONS="-b 4k" and then run generic/416 on ext4 filesystem.

# ./check tests/generic/416
FSTYP         -- ext4
PLATFORM      -- Linux/x86_64 bogon 5.3.18-57-default #1 SMP Wed Apr 28
10:54:41 UTC 2021 (ba3c2e9)
MKFS_OPTIONS  -- -b 4k /dev/loop1
MOUNT_OPTIONS -- -o acl,user_xattr /dev/loop1 /mnt/scratch

generic/416 90s ... [failed, exit status 1]- output mismatch (see /home/lan/work/xfstests/results//generic/416.out.bad)

    --- tests/generic/416.out	2021-06-25 23:18:09.595254118 +0800
    +++ /home/lan/work/xfstests/results//generic/416.out.bad	2022-

06-15 16:33:19.584206316 +0800
@@ -1,3 +1,4 @@
QA output created by 416
-wrote 16777216/16777216 bytes at offset 0
-XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +mount: /mnt/scratch: wrong fs type, bad option, bad superblock on
/dev/loop1, missing codepage or helper program, or other error.
+mount -o acl,user_xattr /dev/loop1 /mnt/scratch failed +(see /home/lan/work/xfstests/results//generic/416.full for
details)
...
(Run 'diff -u /home/lan/work/xfstests/tests/generic/416.out
/home/lan/work/xfstests/results//generic/416.out.bad' to see the entire diff)
Ran: generic/416
Failures: generic/416
Failed 1 of 1 tests

But the more common result as below:

# ./check tests/generic/416
FSTYP         -- ext4
PLATFORM      -- Linux/x86_64 bogon 5.3.18-57-default #1 SMP Wed Apr 28
10:54:41 UTC 2021 (ba3c2e9)
MKFS_OPTIONS  -- -b 4k /dev/loop1
MOUNT_OPTIONS -- -o acl,user_xattr /dev/loop1 /mnt/scratch

umount: /mnt/scratch: target is busy.
generic/416 77s ... 90s
Ran: generic/416
Passed all 1 tests

Test passed due to _require_scratch_nocheck() ignore the umount error. However, "mkfs.ext4: invalid block size - 4" was found in the 416.full

Apart from this, the error only occurs when MKFS_OPTIONS is set, and doesn't affect xfs or btrfs. Since ext4 only supports 4k blocksize on most platforms, we usually don't set blocksize in MKFS_OPTIONS.

I think that's why this problem hasn't been found before. In fact, I found this issue when I tested on ppc64 with 64k kernel and 4k subblocksize fs.

Quoted 62 lines
Quoted 62 lines

will give errors to ext2/3/4 etc, and brings potential bugs to xfs or
btrfs.

In addition, since we can receive various strings, so remove integer
number check.

---
 common/rc | 18 ++++++------------
 1 file changed, 6 insertions(+), 12 deletions(-)
common/rc +1 -2
index 22050bc2..026007d3 100644
@@ -1077,7 +1077,7 @@ _parse_size_from_string()
 }
 
 # Create fs of certain size on scratch device
-# _scratch_mkfs_sized <size in bytes> [optional blocksize]
+# _scratch_mkfs_sized <size> [optional blocksize]
 _scratch_mkfs_sized()
 {
 	local fssize=$1
@@ -1086,13 +1086,13 @@ _scratch_mkfs_sized()
 
 	case $FSTYP in
 	xfs)
-		def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?size=

?+([0-9]+).*/\1/p'`

common/rc +1 -0
+		def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?size=

?+([0-9]+[a-zA-Z]?).*/\1/p'`

common/rc +0 -1
 		;;
 	btrfs)
-		def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-s ?+([0-

9]+).*/\1/p'`

common/rc +1 -0
+		def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-s ?+([0-

9]+[a-zA-Z]?).*/\1/p'`

common/rc +0 -1
 		;;
 	ext2|ext3|ext4|ext4dev|udf|reiser4|ocfs2|reiserfs)
-		def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?+([0-

9]+).*/\1/p'`

common/rc +1 -0
+		def_blksz=`echo $MKFS_OPTIONS | sed -rn 's/.*-b ?+([0-

9]+[a-zA-Z]?).*/\1/p'`

common/rc +0 -4
 		;;
 	jfs)
 		def_blksz=4096
@@ -1101,14 +1101,8 @@ _scratch_mkfs_sized()
 
 	[ -n "$def_blksz" ] && blocksize=$def_blksz
 	[ -z "$blocksize" ] && blocksize=4096
-
-	local re='^[0-9]+$'
-	if ! [[ $fssize =~ $re ]] ; then
-		_notrun "error: _scratch_mkfs_sized: fs size

\"$fssize\" not an integer."

common/rc +0 -3
-	fi
-	if ! [[ $blocksize =~ $re ]] ; then
-		_notrun "error: _scratch_mkfs_sized: block size

\"$blocksize\" not an integer."

Quoted 9 lines

common/rc _scratch_mkfs_sized() · line 1105

Quoted 3 lines
common/rc +2 -1
-	fi
+	blocksize=$(_parse_size_from_string $blocksize)
+	fssize=$(_parse_size_from_string $fssize)

For now, the _parse_size_from_string is only used for this patch, I think these
two patches can be merged into one patch, as it trys to fix one single problem.

There are a total of two logical changes. In particular it also include a Function update. So it's better to keep two separate patch. But I'll add patch depends on info to commit message.

Thank you for your time!

Quoted 9 lines

Thanks,
Zorro

Quoted 6 lines

local blocks=expr $fssize / $blocksize

Signature · 1 line
2.35.3
Signature · 2 lines
An Long <[email protected]>
SUSE QE LSG, QE 2, Beijing
author
#5 lore raw
In reply to Zorro Lang
Quoted 17 lines · Zorro Lang
Quoted 16 lines · An Long

Add a helper to convert size value to bytes. This is used to handle value as bytes, such as 4k to 4096.

---
 common/rc | 48 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)
common/rc +0 -0
index 3c072c16..22050bc2 100644
@@ -1028,6 +1028,54 @@ _check_minimal_fs_size()
 	fi
 }
 
Quoted 8 lines

common/rc _check_minimal_fs_size() · line 1033

Quoted 3 lines
common/rc +3 -0
+# Convert size value to bytes
+# _parse_size_from_string <size>
+_parse_size_from_string()

I think this name is better to be shorter, as it might be a lower level common
function. How about _convert_num() or _cvtnum()?

Rename _parse_size_from_string() to _parse_size_tring(). So that as much as possible to keep function meaning.

Quoted 48 lines
Quoted 33 lines
common/rc +32 -0
+{
+        local str=$1
+        local mult=1
+        local size
+        local endchar
+
+        if [[ $str =~ ^[0-9]+[a-zA-Z]$ ]] ; then
+                size=${str:: -1}
+                endchar=${str: -1}
+                case $endchar in
+                e|E)
+                                mult=$(($mult * 1024))
+                                ;&
+                p|P)
+                                mult=$(($mult * 1024))
+                                ;&
+                t|T)
+                                mult=$(($mult * 1024))
+                                ;&
+                g|G)
+                                mult=$(($mult * 1024))
+                                ;&
+                m|M)
+                                mult=$(($mult * 1024))
+                                ;&
+                k|K)
+                                mult=$(($mult * 1024))
+                                ;&
+                b|B)
+                                ;;
+                *)
+                                echo "unknown size descriptor

$endchar"

common/rc _check_minimal_fs_size() · line 1075

Quoted 10 lines
common/rc +10 -0
+                                exit 1
+                esac
+        elif [[ $str =~ ^[0-9]+$ ]] ; then
+                size=$str
+        else
+                echo "size value $str is invalid"
+                exit 1
+        fi
+
+        size=$(($size * $mult))

^ ^
I think above two '$' isn't necessary, although it doesn't make something
wrong if keep it. Same as all above (($mult * 1024)) .

Quoted 5 lines

Thanks,
Zorro

                                      ^

Removed.

Quoted 9 lines
Quoted 9 lines
common/rc +3 -0
+        echo $size
+}
+
 # Create fs of certain size on scratch device
 # _scratch_mkfs_sized <size in bytes> [optional blocksize]
 _scratch_mkfs_sized()
Signature · 1 line
2.35.3
Signature · 2 lines
An Long <[email protected]>
SUSE QE LSG, QE 2, Beijing