lib/kstrtox: introduce kstrtoull_suffix() helper
Subsystems (2)
v2 cover lib/kstrtox: introduce kstrtoull_suffix() helper
[CHANGELOG]
v2:
- Use enum bitmap to define suffixes
This get rid of the upper/lower case problem, and using enum for each suffix still provides a somewhat readable result.
- Do proper suffix overflow check
The previous check for suffix overflow is incorrect at all, would still lead to incorrect values.
- Slight refactor for kstrtoull_suffix() code
Recently David Sterba <[email protected]> exposed some weird behavior of btrfs sysfs interface, which is utilizing memparse().
One example is using "echo 25e >
/sys/fs/btrfs/<uuid>/devinfo/scrub_speed_max".
The result value is not 0x25e, because there is no "0x" prefix provided. Nor (25 << 60), as that would overflow.
All these are the caveats of memparse(), which lacks:
- Overflow check
- Reasonable suffix selection
I know there may be some niche cases for memory layout, but for most callers, they just want a kstrtoull() with reasonable suffix selection.
And I don't think exabytes is a reasonable suffix.
So here we introduce a new helper, "kstrtoull_suffix", which has some the following two abilities added:
- Allow caller to select the suffix they want to enable The default list is "KkMmGgTtPp", no unreasonable "Ee" ones.
- Allow suffix parsing with overflow detection.
The int part detection is already done by _kstrtoull(), and with the extra left shift, do the check again before returning the value.
Unfortunately this new helper is still not a drop-in replacement for memparse():
- No @retptr support
It's possible to add, but I'm not sure how many call sites need that for extra separators, and even if there are, they may want to go strsep() and strtok().
- Need to check the failure properly
Which memparse() callers never seems to check anyway.
Thus the existing memparse() callers need to opt-in. For now, btrfs usage of memparse() can be easily converted to kstrtoull_suffix(), in the 2nd patch.
Qu Wenruo (2):
lib/strtox: introduce kstrtoull_suffix() helper
btrfs: sysfs: use kstrtoull_suffix() to replace memparse()
fs/btrfs/sysfs.c | 20 +++----- include/linux/kstrtox.h | 20 ++++++++ lib/kstrtox.c | 108 ++++++++++++++++++++++++++++++++++++++-- 3 files changed, 131 insertions(+), 17 deletions(-)
Signature · 1 line
2.43.0
PATCH v2 1/2 lib/strtox: introduce kstrtoull_suffix() helper
Just as mentioned in the comment of memparse(), the simple_stroull() usage can lead to overflow all by itself.
Furthermore, the suffix calculation is also super overflow prone because that some suffix like "E" itself would eat 60bits, leaving only 4 bits available.
And that suffix "E" can also lead to confusion since it's using the same char of hex Ox'E'.
One simple example to expose all the problem is to use memparse() on "25E".
The correct value should be 28823037615171174400, but the suffix E makes it super simple to overflow, resulting the incorrect value 10376293541461622784 (9E).
So here we introduce a new helper to address the problem, kstrtoull_suffix():
- Enhance _kstrtoull()
This allow _kstrtoull() to return even if it hits an invalid char, as long as the optional parameter @retptr is provided.
If @retptr is provided, _kstrtoull() would try its best to parse the valid part, and leave the remaining to be handled by the caller.
If @retptr is not provided, the behavior is not altered.
- New kstrtoull_suffix() helper
This new helper utilize the new @retptr capability of _kstrtoull(), and provides 2 new ability:
* Allow certain suffixes to be chosen
The recommended suffix list is "KMGTP" (using the new unit_suffix enum as a bitmap), excluding the overflow prone "E". Undermost cases there is really no need to use "E" suffix anyway. And for those who really need that exabytes suffix, they can enable that suffix pretty easily.
* Add overflow checks for the suffixes
If the original number string is fine, but with the extra left shift overflow happens, then -EOVERFLOW is returned.
----
Changelog:
v2:
- Use enum bitmap to describe the suffixes
This gets rid of the upper/lower case problem, and enum makes it a little more readable.
- Fix the suffix overflow detection
- Move the left shift out of the switch block
- Remove the "done" tag
Since no tailing character can already be handled properly.
--- include/linux/kstrtox.h | 20 ++++++++ lib/kstrtox.c | 108 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 123 insertions(+), 5 deletions(-)
include/linux/kstrtox.h +20 -0
@@ -9,6 +9,26 @@int __must_check _kstrtoul (const char *s,unsigned int base,unsigned long *res);int __must_check _kstrtol (const char *s,unsigned int base,long *res); +/* +* The default suffix list would not include "E" since it's too easy to overflow +* and not much real world usage. +*/ +enum unit_suffix { +SUFFIX_K = (1 <<0 ), +SUFFIX_M = (1 <<1 ), +SUFFIX_G = (1 <<2 ), +SUFFIX_T = (1 <<3 ), +SUFFIX_P = (1 <<4 ), +SUFFIX_E = (1 <<5 ), +}; + +/* +* The default suffix list would not include "E" since it's too easy to overflow +* and not much real world usage. +*/ +#define KSTRTOULL_SUFFIX_DEFAULT (SUFFIX_K | SUFFIX_M | SUFFIX_G | SUFFIX_T | SUFFIX_P) +int kstrtoull_suffix (const char *s,unsigned int base,unsigned long long *res, +enum unit_suffix suffixes);int __must_check kstrtoull (const char *s,unsigned int base,unsigned long long *res);int __must_check kstrtoll (const char *s,unsigned int base,long long *res);
lib/kstrtox.c +103 -5
138 lines not on this page · raw message
Signature · 1 line
2.43.0
PATCH v2 2/2 btrfs: sysfs: use kstrtoull_suffix() to replace memparse()
In reply to Qu WenruoSince memparse() itself can not handle overflow at all, use kstrtoull_suffix() to be extra safe.
This would introduce behavior changes to the following sysfs interfaces:
- /sys/fs/btrfs/<uuid>/allocation/(data|metadata|system)/chunk_size
- /sys/fs/btrfs/<uuid>/devinfo/<devid>/scrub_speed_max
The behavior changes would include:
- More overflow detection
- No support for "E" suffix anymore
- More strict tailing character requirement
Now the only allowed tailing character is newline. Other characters including space are no longer supported.
Although for regular "echo NUMBER >" usage it should be totally fine.
---
Changelog:
v2:
- None
--- fs/btrfs/sysfs.c | 20 ++++++++------------ 1 file changed, 8 insertions(+), 12 deletions(-)
fs/btrfs/sysfs.c +8 -12
@@ -760,7 +760,7 @@ static ssize_t btrfs_chunk_size_store(struct kobject *kobj, {struct btrfs_space_info *space_info =to_space_info (kobj);struct btrfs_fs_info *fs_info =to_fs_info (get_btrfs_kobj (kobj)); -char *retptr; +int ret;u64 val;if (!capable (CAP_SYS_ADMIN ))
@@ -776,11 +776,9 @@ static ssize_t btrfs_chunk_size_store(struct kobject *kobj,if (space_info->flags &BTRFS_BLOCK_GROUP_SYSTEM )return -EPERM ; - val =memparse (buf, &retptr); -/* There could be trailing '\n', also catch any typos after the value */ - retptr =skip_spaces (retptr); -if (*retptr !=0 || val ==0 ) -return -EINVAL ; + ret =kstrtoull_suffix (buf,0 , &val,KSTRTOULL_SUFFIX_DEFAULT ); +if (ret <0 ) +return ret; val =min (val,BTRFS_MAX_DATA_CHUNK_SIZE );
@@ -1779,14 +1777,12 @@ static ssize_t btrfs_devinfo_scrub_speed_max_store(struct kobject *kobj, { struct btrfs_device *device = container_of(kobj, struct btrfs_device, devid_kobj); -char *endptr;unsigned long long limit; +int ret; - limit =memparse (buf, &endptr); -/* There could be trailing '\n', also catch any typos after the value. */ - endptr =skip_spaces (endptr); -if (*endptr !=0 ) -return -EINVAL ; + ret =kstrtoull_suffix (buf,0 , &limit,KSTRTOULL_SUFFIX_DEFAULT ); +if (ret <0 ) +return ret;WRITE_ONCE (device->scrub_speed_max, limit);return len; }
Signature · 1 line
2.43.0
Re: [PATCH v2 1/2] lib/strtox: introduce kstrtoull_suffix() helper
In reply to Qu WenruoQuoted 57 lines · Qu Wenruo
Just as mentioned in the comment of
memparse(), thesimple_stroull()usage can lead to overflow all by itself.Furthermore, the suffix calculation is also super overflow prone because that some suffix like "E" itself would eat 60bits, leaving only 4 bits available.
And that suffix "E" can also lead to confusion since it's using the same char of hex Ox'E'.
One simple example to expose all the problem is to use
memparse()on "25E".
The correct value should be 28823037615171174400, but the suffix E makes it super simple to overflow, resulting the incorrect value 10376293541461622784 (9E).So here we introduce a new helper to address the problem,
kstrtoull_suffix():- Enhance
_kstrtoull()
This allow_kstrtoull()to return even if it hits an invalid char, as long as the optional parameter @retptr is provided.If @retptr is provided,
_kstrtoull()would try its best to parse the valid part, and leave the remaining to be handled by the caller.If @retptr is not provided, the behavior is not altered.
- New
kstrtoull_suffix()helper
This new helper utilize the new @retptr capability of_kstrtoull(), and provides 2 new ability:* Allow certain suffixes to be chosen
The recommended suffix list is "KMGTP" (using the newunit_suffixenum as a bitmap), excluding the overflow prone "E". Undermost cases there is really no need to use "E" suffix anyway. And for those who really need that exabytes suffix, they can enable that suffix pretty easily.* Add overflow checks for the suffixes
If the original number string is fine, but with the extra left shift overflow happens, then-EOVERFLOWis returned.Cc Andrew Morton <[email protected]>Cc Christophe JAILLET <[email protected]>Cc Andy Shevchenko <[email protected]>Cc David Laight <[email protected]>Signed-off-by Qu Wenruo <[email protected]>----
Changelog:
v2:
- Use enum bitmap to describe the suffixes
This gets rid of the upper/lower case problem, and enum makes it a little more readable.
Quoted 6 lines
- Fix the suffix overflow detection
- Move the left shift out of the switch block
- Remove the "done" tag
Since no tailing character can already be handled properly.
nit: git am puts this changelog in the commit message when applied. Please use git send-email --annotate and put it next to the diffstat, so that it gets discarded.
Quoted 20 lines
--- include/linux/kstrtox.h | 20 ++++++++ lib/kstrtox.c | 108 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 123 insertions(+), 5 deletions(-)include/linux/kstrtox.h +7
-0@@ -9,6 +9,26 @@int __must_check _kstrtoul (const char *s,unsigned int base,unsigned long *res);int __must_check _kstrtol (const char *s,unsigned int base,long *res); +/* +* The default suffix list would not include "E" since it's too easy to overflow +* and not much real world usage. +*/ +enum unit_suffix { +SUFFIX_K = (1 <<0 ), +SUFFIX_M = (1 <<1 ),
include/linux/kstrtox.h line 28
Quoted 10 lines
include/linux/kstrtox.h +10
-0+SUFFIX_G = (1 <<2 ), +SUFFIX_T = (1 <<3 ), +SUFFIX_P = (1 <<4 ), + SUFFIX_E = (1 << 5), +}; + +/* +* The default suffix list would not include "E" since it's too easy to overflow +* and not much real world usage. +*/
^ this comment is a duplicate.
include/linux/kstrtox.h line 29
Quoted 1 line
include/linux/kstrtox.h +1
-0+#define KSTRTOULL_SUFFIX_DEFAULT (SUFFIX_K | SUFFIX_M | SUFFIX_G | SUFFIX_T | SUFFIX_P)
I think it'd be clearer if you dropped this default and had callers explicitly provide the desired suffix mask.
include/linux/kstrtox.h line 31
Quoted 2 lines
include/linux/kstrtox.h +2
-0+int kstrtoull_suffix (const char *s,unsigned int base,unsigned long long *res, +enum unit_suffix suffixes);
__must_check would be consistent with the other helpers...
Quoted 122 lines
include/linux/kstrtox.h +0
-0int __must_check kstrtoull (const char *s,unsigned int base,unsigned long long *res);int __must_check kstrtoll (const char *s,unsigned int base,long long *res);lib/kstrtox.c +87
-4@@ -93,7 +93,8 @@ unsigned int _parse_integer(const char *s, unsigned int base, unsigned long longreturn _parse_integer_limit (s, base, p,INT_MAX ); } -static int _kstrtoull (const char *s,unsigned int base,unsigned long long *res) +static int _kstrtoull (const char *s,unsigned int base,unsigned long long *res, +char **retptr) {unsigned long long _res;unsigned int rv;@@ -105,11 +106,19 @@ static int _kstrtoull(const char *s, unsigned int base, unsigned long long *res)if (rv ==0 )return -EINVAL ; s += rv; -if (*s =='\n' ) + +/* +* If @retptr is provided, caller is responsible to detect +* the extra chars, otherwise we can skip one newline. +*/ +if (!retptr && *s =='\n' ) s++; -if (*s) +if (!retptr && *s)return -EINVAL ; + *res = _res; +if (retptr) + *retptr = (char *)s;return 0 ; }@@ -133,10 +142,99 @@ int kstrtoull(const char *s, unsigned int base, unsigned long long *res) {if (s[0 ] =='+' ) s++; -return _kstrtoull (s, base, res); +return _kstrtoull (s, base, res,NULL ); }EXPORT_SYMBOL (kstrtoull); +/** +* kstrtoull_suffix - convert a string to ull with suffixes support +* @s: The start of the string. The string must be null-terminated, and may also +* include a single newline before its terminating null. +* @base: The number base to use. The maximum supported base is 16. If base is +* given as 0, then the base of the string is automatically detected with the +* conventional semantics - If it begins with 0x the number will be parsed as a +* hexadecimal (case insensitive), if it otherwise begins with 0, it will be +* parsed as an octal number. Otherwise it will be parsed as a decimal. +* @res: Where to write the result of the conversion on success. +* @suffixes: bitmap of acceptable suffixes, unknown bits would be ignored. +* +* Return 0 on success. +* +* Return -ERANGE on overflow or -EINVAL if invalid chars found. +* Return value must be checked. +*/ +int kstrtoull_suffix (const char *s,unsigned int base,unsigned long long *res, +enum unit_suffix suffixes) +{ +unsigned long long value; +int shift =0 ; +char *endptr; +int ret; + + ret =_kstrtoull (s, base, &value, &endptr); +/* Either already overflow or no number string at all. */ +if (ret <0 ) +return ret; + +switch (*endptr) { +case 'K' : +case 'k' : +if (!(suffixes &SUFFIX_K )) +return -EINVAL ; + shift =10 ; +break ; +case 'M' : +case 'm' : +if (!(suffixes &SUFFIX_M )) +return -EINVAL ; + shift =20 ; +break ; +case 'G' : +case 'g' : +if (!(suffixes &SUFFIX_G )) +return -EINVAL ; + shift =30 ; +break ; +case 'T' : +case 't' : +if (!(suffixes &SUFFIX_T )) +return -EINVAL ; + shift =40 ; +break ; +case 'P' : +case 'p' : +if (!(suffixes &SUFFIX_P )) +return -EINVAL ; + shift =50 ; +break ; +case 'E' : +case 'e' : +if (!(suffixes &SUFFIX_E )) +return -EINVAL ; + shift =60 ; +break ; + } +if (shift) + endptr++; +if (*endptr =='\n' ) + endptr++; +if (*endptr) +return -EINVAL ;
lib/kstrtox.c int kstrtoull(const char *s, unsigned int base, unsigned long long *res) · line 232
Quoted 10 lines
lib/kstrtox.c +10
-0+ +/* +* Overflow check. +* +* Check @shift before doing right shift, as if right shift bit is +* greater than or equal to the number of bits, the result is undefined. +*/ +if (shift && value >> (64 - shift)) +return -EOVERFLOW ; + value <<= shift;
Might as well roll this logic in with the existing "if (shift)" above, e.g.
if (shift) {
/*
* Overflow check...
*/
if (value >> (64 - shift))
return -EOVERFLOW;
value <<= shift;
endptr++;
}
if (*endptr == '\n')
...Quoted 9 lines
lib/kstrtox.c +5
-0+ *res = value; +return 0 ; +} +EXPORT_SYMBOL (kstrtoull_suffix); +/** * kstrtoll - convert a string to a long long * @s: The start of the string. The string must be null-terminated, and may also
lib/kstrtox.c int kstrtoll(const char *s, unsigned int base, long long *res) · line 263
Quoted 8 lines
lib/kstrtox.c +1
-1@@ -159,7 +257,7 @@ int kstrtoll(const char *s, unsigned int base, long long *res)int rv;if (s[0 ] =='-' ) { - rv =_kstrtoull (s +1 , base, &tmp); + rv =_kstrtoull (s +1 , base, &tmp,NULL );if (rv <0 )return rv;if ((long long )-tmp >0 )
With the above changes made, feel free to add
I'll leave the review of patch 2/2 up to others, as I'm still a little worried about sysfs trailing whitespace regressions.
Re: [PATCH v2 1/2] lib/strtox: introduce kstrtoull_suffix() helper
In reply to Qu WenruoQuoted 39 lines · Qu Wenruo
Just as mentioned in the comment of
memparse(), thesimple_stroull()usage can lead to overflow all by itself.Furthermore, the suffix calculation is also super overflow prone because that some suffix like "E" itself would eat 60bits, leaving only 4 bits available.
And that suffix "E" can also lead to confusion since it's using the same char of hex Ox'E'.
One simple example to expose all the problem is to use
memparse()on "25E".
The correct value should be 28823037615171174400, but the suffix E makes it super simple to overflow, resulting the incorrect value 10376293541461622784 (9E).So here we introduce a new helper to address the problem,
kstrtoull_suffix():- Enhance
_kstrtoull()
This allow_kstrtoull()to return even if it hits an invalid char, as long as the optional parameter @retptr is provided.If @retptr is provided,
_kstrtoull()would try its best to parse the valid part, and leave the remaining to be handled by the caller.If @retptr is not provided, the behavior is not altered.
- New
kstrtoull_suffix()helper
This new helper utilize the new @retptr capability of_kstrtoull(), and provides 2 new ability:* Allow certain suffixes to be chosen
The recommended suffix list is "KMGTP" (using the newunit_suffixenum as a bitmap), excluding the overflow prone "E". Undermost cases there is really no need to use "E" suffix anyway. And for those who really need that exabytes suffix, they can enable that suffix pretty easily.
Quoted 3 lines
* Add overflow checks for the suffixes
If the original number string is fine, but with the extra left shift overflow happens, then-EOVERFLOWis returned.
NAK. Read the v1 discussion why.
Signature · 2 lines
With Best Regards, Andy Shevchenko
Re: [PATCH v2 1/2] lib/strtox: introduce kstrtoull_suffix() helper
In reply to David DisseldorpQuoted 2 lines · David Disseldorp
On Wed, 20 Dec 2023 10:40:00 +1030, Qu Wenruo wrote:
[...]
Quoted 10 lines
Quoted 10 lines
----
Changelog:
v2:
- Use enum bitmap to describe the suffixes
This gets rid of the upper/lower case problem, and enum makes it a little more readable.- Fix the suffix overflow detection
Quoted 8 lines
Quoted 4 lines
- Move the left shift out of the switch block
- Remove the "done" tag
Since no tailing character can already be handled properly.nit: git am puts this changelog in the commit message when applied. Please use
git send-email --annotateand put it next to the diffstat, so that it gets discarded.
Got it.
[...]
Quoted 9 lines
include/linux/kstrtox.h line 28
Quoted 6 lines
include/linux/kstrtox.h +6
-0+}; + +/* +* The default suffix list would not include "E" since it's too easy to overflow +* and not much real world usage. +*/ ^ this comment is a duplicate.
Quoted 4 lines
Quoted 1 line
+#define
KSTRTOULL_SUFFIX_DEFAULT(SUFFIX_K|SUFFIX_M|SUFFIX_G|SUFFIX_T|SUFFIX_P)I think it'd be clearer if you dropped this default and had callers explicitly provide the desired suffix mask.
Well, that would be long, and would be even longer as the newer naming would be MEMPARSE_SUFFIX_*, to be more explicit on what the suffix is for...
And I really want callers to choose a saner default suffix, thus here comes the default one.
In fact, in my next version, I also found that there are some memparse() call sites benefits from the newer suffixes (although won't for the "E" one).
The example is the call site setup_elfcorehdr(). Where the comment only mentions KMG, but since memparse() silently added "PE" suffixes, maybe on some mainframes we saved some time for one or two lucky admins.
[...]
Quoted 4 lines
With the above changes made, feel free to add
Reviewed-by David Disseldorp <[email protected]>
Thanks for the review, but I'm afraid the newer version would be another beast.
All the ommitted comments would be addressed a in new series.
Quoted 3 lines
I'll leave the review of patch 2/2 up to others, as I'm still a little worried about sysfs trailing whitespace regressions.
That won't be a problem anymore, the new series would keep the old @retptr behavior, thus for btrfs part it won't be changed at all.
Thanks,
Qu
Quoted 1 line
Re: [PATCH v2 1/2] lib/strtox: introduce kstrtoull_suffix() helper
Quoted 1 line · Qu Wenruo
On 2023/12/20 16:08, David Disseldorp wrote:
...
Quoted 11 lines
Quoted 4 lines
Quoted 1 line
+#define
KSTRTOULL_SUFFIX_DEFAULT(SUFFIX_K|SUFFIX_M|SUFFIX_G|SUFFIX_T|SUFFIX_P)I think it'd be clearer if you dropped this default and had callers explicitly provide the desired suffix mask.
Well, that would be long, and would be even longer as the newer naming would be MEMPARSE_SUFFIX_*, to be more explicit on what the suffix is for...
And I really want callers to choose a saner default suffix, thus here comes the default one.
Quoted 6 lines
In fact, in my next version, I also found that there are some
memparse()call sites benefits from the newer suffixes (although won't for the "E" one).
The example is the call sitesetup_elfcorehdr(). Where the comment only mentions KMG, but sincememparse()silently added "PE" suffixes, maybe on some mainframes we saved some time for one or two lucky admins.
I think it's a sane default, my concern is that _DEFAULT says nothing about supported units from the caller's perspective. Perhaps MEMPARSE_SUFFIX_KMGTP or MEMPARSE_UNITS_KMGTP would be clearer.
...
Quoted 6 lines
Quoted 2 lines
With the above changes made, feel free to add
Reviewed-by David Disseldorp <[email protected]>Thanks for the review, but I'm afraid the newer version would be another beast.
Quoted 7 lines
All the ommitted comments would be addressed a in new series.
Quoted 3 lines
I'll leave the review of patch 2/2 up to others, as I'm still a little worried about sysfs trailing whitespace regressions.
That won't be a problem anymore, the new series would keep the old @retptr behavior, thus for btrfs part it won't be changed at all.
Sounds good. Will follow up there.
Cheers, David