lib: vsprintf: Avoid 32-bit truncation in vsscanf number parsing
Subsystems (2)
PATCH 1/1 lib: vsprintf: Avoid 32-bit truncation in vsscanf number parsing
Number conversion in vsscanf converts a whole string of digits and then extracts the field width part from the converted value. The maximum run of digits is limited by overflow. Conversion was using either simple_strto[u]l or simple_strto[u]ll based on the 'L' qualifier. This created a difference in truncation between builds where long is 32-bit and builds where it is 64-bit. This especially affects parsing a run of contiguous digits into separate fields - the maximum length of the run is 16 digits if long is 64-bit but only 8 digits if long is 32-bits. For example a conversion "%6x%6x" would convert both fields correctly if long is 64-bit but not if long is 32-bit.
It is undesirable for vsscanf to parse numbers differently depending on the size of long on the target build.
As simple_strto[u]l just calls simple_strto[u]ll anyway the conversion is always 64-bit, and the result is manipulated as a u64, so this is an avoidable behaviour difference between 32-bit and 64-bit builds. The conversion can call simple_strto[u]ll directly and preserve the full 64-bits that were parsed out of the string.
--- lib/vsprintf.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)
lib/vsprintf.c +2 -6
@@ -3444,13 +3444,9 @@ int vsscanf(const char *buf, const char *fmt, va_list args)break ;if (is_sign) - val.s = qualifier !='L' ? -simple_strtol (str, &next, base) : -simple_strtoll (str, &next, base); + val.s =simple_strtoll (str, &next, base);else - val.u = qualifier !='L' ? -simple_strtoul (str, &next, base) : -simple_strtoull (str, &next, base); + val.u =simple_strtoull (str, &next, base);if (field_width >0 && next - str > field_width) {if (base ==0 )
Signature · 1 line
2.20.1
Steven Rostedt
maintainerQuoted 35 lines · On Thu, 12 Nov 2020 11:17:59 +0000 Richard Fitzgerald…
Number conversion in vsscanf converts a whole string of digits and then extracts the field width part from the converted value. The maximum run of digits is limited by overflow. Conversion was using either simple_strto[u]l or simple_strto[u]ll based on the 'L' qualifier. This created a difference in truncation between builds where long is 32-bit and builds where it is 64-bit. This especially affects parsing a run of contiguous digits into separate fields - the maximum length of the run is 16 digits if long is 64-bit but only 8 digits if long is 32-bits. For example a conversion "%6x%6x" would convert both fields correctly if long is 64-bit but not if long is 32-bit.
It is undesirable for vsscanf to parse numbers differently depending on the size of long on the target build.
As simple_strto[u]l just calls simple_strto[u]ll anyway the conversion is always 64-bit, and the result is manipulated as a u64, so this is an avoidable behaviour difference between 32-bit and 64-bit builds. The conversion can call simple_strto[u]ll directly and preserve the full 64-bits that were parsed out of the string.
Signed-off-by Richard Fitzgerald <[email protected]>--- lib/vsprintf.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)lib/vsprintf.c +0
-2@@ -3444,13 +3444,9 @@ int vsscanf(const char *buf, const char *fmt, va_list args)break ;if (is_sign) - val.s = qualifier !='L' ? - simple_strtol(str, &next, base) :
lib/vsprintf.c int vsscanf(const char *buf, const char *fmt, va_list args) · line 3452
Quoted 10 lines
lib/vsprintf.c +2
-4-simple_strtoll (str, &next, base); + val.s =simple_strtoll (str, &next, base); else - val.u = qualifier != 'L' ? -simple_strtoul (str, &next, base) : -simple_strtoull (str, &next, base); + val.u = simple_strtoull(str, &next, base);if (field_width >0 && next - str > field_width) {if (base ==0 )
It looks like this is fixing the symptom and not the disease. The real issue I see here is that vsscanf is not honoring the '6' of '%6x' here. It should only read the 6 characters then do the conversion, not the other way around! This looks to me that the design is of issue.
-- Steve
Quoted 54 lines · Steven Rostedt
Quoted 35 lines · On Thu, 12 Nov 2020 11:17:59 +0000 Richard Fitzgerald…
Number conversion in vsscanf converts a whole string of digits and then extracts the field width part from the converted value. The maximum run of digits is limited by overflow. Conversion was using either simple_strto[u]l or simple_strto[u]ll based on the 'L' qualifier. This created a difference in truncation between builds where long is 32-bit and builds where it is 64-bit. This especially affects parsing a run of contiguous digits into separate fields - the maximum length of the run is 16 digits if long is 64-bit but only 8 digits if long is 32-bits. For example a conversion "%6x%6x" would convert both fields correctly if long is 64-bit but not if long is 32-bit.
It is undesirable for vsscanf to parse numbers differently depending on the size of long on the target build.
As simple_strto[u]l just calls simple_strto[u]ll anyway the conversion is always 64-bit, and the result is manipulated as a u64, so this is an avoidable behaviour difference between 32-bit and 64-bit builds. The conversion can call simple_strto[u]ll directly and preserve the full 64-bits that were parsed out of the string.
Signed-off-by Richard Fitzgerald <[email protected]>--- lib/vsprintf.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)lib/vsprintf.c +0
-2@@ -3444,13 +3444,9 @@ int vsscanf(const char *buf, const char *fmt, va_list args)break ;if (is_sign) - val.s = qualifier !='L' ? - simple_strtol(str, &next, base) :lib/vsprintf.c int vsscanf(const char *buf, const char *fmt, va_list args) · line 3452
Quoted 10 lines
lib/vsprintf.c +2
-4-simple_strtoll (str, &next, base); + val.s =simple_strtoll (str, &next, base); else - val.u = qualifier != 'L' ? -simple_strtoul (str, &next, base) : -simple_strtoull (str, &next, base); + val.u = simple_strtoull(str, &next, base);if (field_width >0 && next - str > field_width) {if (base ==0 )It looks like this is fixing the symptom and not the disease. The real issue I see here is that vsscanf is not honoring the '6' of '%6x' here. It should only read the 6 characters then do the conversion, not the other way around! This looks to me that the design is of issue.
Quoted 2 lines
-- Steve
See this thread from 2014 where the field width problem was raised and explained:
http://lkml.iu.edu/hypermail/linux/kernel/1401.1/03443.html
and the reply from Linus Torvalds that was against fixing field width handling:
http://lkml.iu.edu/hypermail/linux/kernel/1401.1/03488.html
which I assume is why the field handling wasn't unoptimized to be strictly correct.
Nevertheless, I see no reason not to remove avoidable inconsistencies from the current design.
Petr Mladek
maintainerAdding other vsprintf maintainers and reviewes into CC.
Quoted 10 lines · Richard Fitzgerald
Number conversion in vsscanf converts a whole string of digits and then extracts the field width part from the converted value. The maximum run of digits is limited by overflow. Conversion was using either simple_strto[u]l or simple_strto[u]ll based on the 'L' qualifier. This created a difference in truncation between builds where long is 32-bit and builds where it is 64-bit. This especially affects parsing a run of contiguous digits into separate fields - the maximum length of the run is 16 digits if long is 64-bit but only 8 digits if long is 32-bits. For example a conversion "%6x%6x" would convert both fields correctly if long is 64-bit but not if long is 32-bit.
I might be just slow today. But it took me really long time to understand what exactly is the problem and how it is caused. The description is nicely detailed but somehow cryptic.
My understanding is that there is a bug when parsing numbers with a limited width like the above mentioned "%6x%6x".
The problem is how the width is handled:
1. The width is ignored in the 1st step. The entire number is read using simple_strto[u]l[l] functions.
2. The width limit is achieved by dividing the result from the first step until it fits the width.
It gives wrong result when there was an overflow in the 1st step. The high bits were lost even when the limited number would not overflow.
This patch improves the situation on 32-bit system because it reduces the chance of overflow there. It also makes it work the same on 32-bit and 64-bit systems.
But the problem is still there when the 64-bit number overflows!
IMHO, the right solution is to copy only the limited number of characters into a buffer and call simple_strto[u]l[l] on this buffer. Then we do not need to divide the result at all.
That said, I agree that simple_strto[u]ll() might always get called. It is used internally by simple_strto[u]l() anyway. The value is always stored into a local 64-bit variable. It is properly casted later when the value is copied into the caller's variable. But I do not consider this a proper fix of the width handling.
Best Regards,
Petr
Quoted 36 lines
It is undesirable for vsscanf to parse numbers differently depending on the size of long on the target build.
As simple_strto[u]l just calls simple_strto[u]ll anyway the conversion is always 64-bit, and the result is manipulated as a u64, so this is an avoidable behaviour difference between 32-bit and 64-bit builds. The conversion can call simple_strto[u]ll directly and preserve the full 64-bits that were parsed out of the string.
Signed-off-by Richard Fitzgerald <[email protected]>--- lib/vsprintf.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)lib/vsprintf.c +2
-6@@ -3444,13 +3444,9 @@ int vsscanf(const char *buf, const char *fmt, va_list args)break ;if (is_sign) - val.s = qualifier !='L' ? -simple_strtol (str, &next, base) : -simple_strtoll (str, &next, base); + val.s =simple_strtoll (str, &next, base);else - val.u = qualifier !='L' ? -simple_strtoul (str, &next, base) : -simple_strtoull (str, &next, base); + val.u =simple_strtoull (str, &next, base);if (field_width >0 && next - str > field_width) {if (base ==0 )Signature · 1 line
2.20.1
From: Petr Mladek
Quoted 15 lines
Sent: 12 November 2020 16:18
Adding other vsprintf maintainers and reviewes into CC.
Quoted 10 lines · Richard Fitzgerald
Number conversion in vsscanf converts a whole string of digits and then extracts the field width part from the converted value. The maximum run of digits is limited by overflow. Conversion was using either simple_strto[u]l or simple_strto[u]ll based on the 'L' qualifier. This created a difference in truncation between builds where long is 32-bit and builds where it is 64-bit. This especially affects parsing a run of contiguous digits into separate fields - the maximum length of the run is 16 digits if long is 64-bit but only 8 digits if long is 32-bits. For example a conversion "%6x%6x" would convert both fields correctly if long is 64-bit but not if long is 32-bit.
If %6x%6x works one might expect %6x%6x%6x to also work.
Quoted 12 lines
I might be just slow today. But it took me really long time to understand what exactly is the problem and how it is caused. The description is nicely detailed but somehow cryptic.
My understanding is that there is a bug when parsing numbers with a limited width like the above mentioned "%6x%6x".
The problem is how the width is handled:
1. The width is ignored in the 1st step. The entire number is read using simple_strto[u]l[l] functions.
Quoted 6 lines
2. The width limit is achieved by dividing the result from the first step until it fits the width.
It gives wrong result when there was an overflow in the 1st step. The high bits were lost even when the limited number would not overflow.
What happens if there are leading zeros on the hex input? From the description I think 'something terrible happens'. (Well, horribly unexpected anyway.)
I'd also expect strtoull() to 'eat' all the digits that exist, not stop when the value got too large.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Steven Rostedt
maintainerQuoted 7 lines · On Thu, 12 Nov 2020 15:46:46 +0000 Richard Fitzgerald…
See this thread from 2014 where the field width problem was raised and explained:
http://lkml.iu.edu/hypermail/linux/kernel/1401.1/03443.htmland the reply from Linus Torvalds that was against fixing field width handling:
http://lkml.iu.edu/hypermail/linux/kernel/1401.1/03488.html
Thanks for the pointers, but note, that references to older emails should use https://lore.kernel.org/ as these links format the output really horribly.
Quoted 6 lines
which I assume is why the field handling wasn't unoptimized to be strictly correct.
Nevertheless, I see no reason not to remove avoidable inconsistencies from the current design.
Yes, but perhaps its time to fix the real problem and not just add band-aids. That thread is over 6 years old (the email was from Jan 14, 2014)
$ git diff `git rev-list --before 'Jan 14 2014' HEAD --max-count=1` | grep '^+' | grep sscanf | wc -l 622
There's been over 600 new additions of sscanf(). Now is the time to just fix it correctly.
-- Steve
Petr Mladek
maintainerQuoted 7 lines · Steven Rostedt
Quoted 4 lines · On Thu, 12 Nov 2020 15:46:46 +0000 Richard Fitzgerald…
See this thread from 2014 where the field width problem was raised and explained:
http://lkml.iu.edu/hypermail/linux/kernel/1401.1/03443.html
Quoted 11 lines
Quoted 3 lines
and the reply from Linus Torvalds that was against fixing field width handling:
http://lkml.iu.edu/hypermail/linux/kernel/1401.1/03488.htmlThanks for the pointers, but note, that references to older emails should use https://lore.kernel.org/ as these links format the output really horribly.
Quoted 3 lines
which I assume is why the field handling wasn't unoptimized to be strictly correct.
Honestly, the handling of the number width by div does not look like a real optimization to me. It avoids the need of the temporary buffer by expensive and error-prone operation.
IMHO, it is safe to assume that the width will be limited so that the value would never overflow.
The longest supported number would be (2^64 - 1) in octal. If I am counting correctly, it is
01777777777777777777777
and it fits into buf[24] including the trailing '\0'.
We could call WARN_ON_ONCE() when the width >= 24 is higher. And we could add a compiler check when long long is bigger than 64-bit.
Quoted 9 lines
Yes, but perhaps its time to fix the real problem and not just add band-aids. That thread is over 6 years old (the email was from Jan 14, 2014)
$ git diff `git rev-list --before 'Jan 14 2014' HEAD --max-count=1` | grep '^+' | grep sscanf | wc -l 622There's been over 600 new additions of
sscanf(). Now is the time to just fix it correctly.
And the following one might suffer from this problem:
drivers/soundwire/slave.c: ret = sscanf(compat, "sdw%01x%04hx%04hx%02hhx", &sdw_version,
I agree with Steven that it is time to fix it properly.
Best Regards,
Petr
Quoted 45 lines · Petr Mladek
Quoted 7 lines · Steven Rostedt
Quoted 4 lines · On Thu, 12 Nov 2020 15:46:46 +0000 Richard Fitzgerald…
See this thread from 2014 where the field width problem was raised and explained:
http://lkml.iu.edu/hypermail/linux/kernel/1401.1/03443.htmlQuoted 11 lines
Quoted 3 lines
and the reply from Linus Torvalds that was against fixing field width handling:
http://lkml.iu.edu/hypermail/linux/kernel/1401.1/03488.htmlThanks for the pointers, but note, that references to older emails should use https://lore.kernel.org/ as these links format the output really horribly.
Quoted 3 lines
which I assume is why the field handling wasn't unoptimized to be strictly correct.
Honestly, the handling of the number width by div does not look like a real optimization to me. It avoids the need of the temporary buffer by expensive and error-prone operation.
IMHO, it is safe to assume that the width will be limited so that the value would never overflow.
The longest supported number would be (2^64 - 1) in octal. If I am counting correctly, it is
01777777777777777777777and it fits into buf[24] including the trailing '\0'.
We could call
WARN_ON_ONCE()when the width >= 24 is higher. And we could add a compiler check when long long is bigger than 64-bit.Quoted 7 lines
Yes, but perhaps its time to fix the real problem and not just add band-aids. That thread is over 6 years old (the email was from Jan 14, 2014)
$ git diff `git rev-list --before 'Jan 14 2014' HEAD --max-count=1` | grep '^+' | grep sscanf | wc -l 622
Quoted 7 lines
Quoted 2 lines
There's been over 600 new additions of
sscanf(). Now is the time to just fix it correctly.And the following one might suffer from this problem:
drivers/soundwire/slave.c: ret = sscanf(compat, "sdw%01x%04hx%04hx%02hhx",&sdw_version,
That's exactly the bug I have.
I'll look at reworking the code to handle number field widths properly.
Quoted 5 lines
I agree with Steven that it is time to fix it properly.
Best Regards,
Petr