Skip to contentSkip to conversation

panic: prevent panic_timeout * 1000 from overflow

PATCH Superseded Changming 3 participants 5 messages Jul 11, 2020 – Jul 16, 2020 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 0d4601d65709a0e2d80e2a8880gmail.com

Changming

author
#1 lore raw

PATCH v2 1/1 panic: prevent panic_timeout * 1000 from overflow

From: Changming Liu <[email protected]>

Since panic_timeout is an integer passed-in through sysctl, the loop boundary panic_timeout * 1000 could overflow and result in a zero-delay panic when panic_timeout is greater than INT_MAX/1000.

Fix this by moving 1000 to the left, also in case i/1000 might never be greater than panic_timeout, change i to long long so that it strictly has more bits.

---
Changes in v2:
- change the loop in panic, instead of change the sysctl

 kernel/panic.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/panic.c +3 -2
index e2157ca387c8..941848aac0ee 100644
@@ -178,7 +178,8 @@ void panic(const char *fmt, ...)
 {
 	static char buf[1024];
 	va_list args;
-	long i, i_next = 0, len;
+	long long i;
+	long i_next = 0, len;
 	int state = 0;
 	int old_cpu, this_cpu;
 	bool _crash_kexec_post_notifiers = crash_kexec_post_notifiers;
@@ -315,7 +316,7 @@ void panic(const char *fmt, ...)
 		 */
 		pr_emerg("Rebooting in %d seconds..\n", panic_timeout);
 
-		for (i = 0; i < panic_timeout * 1000; i += PANIC_TIMER_STEP) {
+		for (i = 0; i / 1000 < panic_timeout; i += PANIC_TIMER_STEP) {
 			touch_nmi_watchdog();
 			if (i >= i_next) {
 				i += panic_blink(state ^= 1);
Signature · 1 line
2.17.1
#2 lore raw
Quoted 26 lines · On Sat, 11 Jul 2020 01:17:28 -0400 Changming <[email protected]>…

From: Changming Liu <[email protected]>

Since panic_timeout is an integer passed-in through sysctl, the loop boundary panic_timeout * 1000 could overflow and result in a zero-delay panic when panic_timeout is greater than INT_MAX/1000.

Fix this by moving 1000 to the left, also in case i/1000 might never be greater than panic_timeout, change i to long long so that it strictly has more bits.

...

kernel/panic.c +2 -1
@@ -178,7 +178,8 @@ void panic(const char *fmt, ...)
 {
 	static char buf[1024];
 	va_list args;
-	long i, i_next = 0, len;
+	long long i;
+	long i_next = 0, len;
 	int state = 0;
 	int old_cpu, this_cpu;
 	bool _crash_kexec_post_notifiers = crash_kexec_post_notifiers;

kernel/panic.c void panic(const char *fmt, ...) · line 319

Quoted 5 lines
kernel/panic.c +1 -1
@@ -315,7 +316,7 @@ void panic(const char *fmt, ...)
 		 */
 		pr_emerg("Rebooting in %d seconds..\n", panic_timeout);
 
-		for (i = 0; i < panic_timeout * 1000; i += PANIC_TIMER_STEP) {
+		for (i = 0; i / 1000 < panic_timeout; i += PANIC_TIMER_STEP) {

Problem is, 32-bit machines generally cannot perform 64-bit divides. So a call is emitted to the library function __divsi64() (I forget the exact name) which Linux doesn't implement (because it's so slow, and we don't want to be calling it by accident).

So a fix would be to call do_div() or something from include/linux/div64.h but it's all a great mess.

However we can do native 64-bit multiplication on 32-bit! So how about something like

kernel/panic.c renamed from kernel/panic.c~a +7 -4
@@ -313,13 +313,16 @@ void panic(const char *fmt, ...)
 		 * Delay timeout seconds before rebooting the machine.
 		 * We can't use the "normal" timers since we just panicked.
 		 */
+		u64 timeout = panic_timeout * 1000;	/* avoid overflow */
+		u64 timer;
+
 		pr_emerg("Rebooting in %d seconds..\n", panic_timeout);
 
-		for (i = 0; i < panic_timeout * 1000; i += PANIC_TIMER_STEP) {
+		for (timer = 0; timer < timeout; timer += PANIC_TIMER_STEP) {
 			touch_nmi_watchdog();
-			if (i >= i_next) {
-				i += panic_blink(state ^= 1);
-				i_next = i + 3600 / PANIC_BLINK_SPD;
+			if (timer >= i_next) {
+				timer += panic_blink(state ^= 1);
+				i_next = timer + 3600 / PANIC_BLINK_SPD;
 			}
 			mdelay(PANIC_TIMER_STEP);
 		}

(untested)

There's still the 3600/PANIC_BLINK_SPD in there, but a) that will be done at compile-time and b) the 64-bit promotion should be done after the division.

And... oh crap, i_next needs to be 64-bit as well.

#3 lore raw

kernel/panic.c void panic(const char *fmt, ...) · line 316

Quoted 6 lines · Andrew Morton

+++ a/kernel/panic.c

kernel/panic.c renamed from kernel/panic.c~a +1 -0
@@ -313,13 +313,16 @@ void panic(const char *fmt, ...)
 		 * Delay timeout seconds before rebooting the machine.
 		 * We can't use the "normal" timers since we just panicked.
 		 */
+		u64 timeout = panic_timeout * 1000;	/* avoid overflow */

1000ULL to not truncate before the assignment.

kernel/panic.c void panic(const char *fmt, ...) · line 317

Quoted 1 line
kernel/panic.c renamed from kernel/panic.c~a +1 -0
+		u64 timer;

... as you implied later

		u64 timer, timer_next;

Review void panic(const char *fmt, ...) · line 318

Quoted 1 line
(quoted diff) +0 -0
 (untested)

Well, there's the rub. Testing requires 49 days (2^32 / 1000 seconds).

#4 lore raw

PATCH v3 1/1 panic: prevent panic_timeout * 1000 from overflow

Since panic_timeout is an s32 integer passed in through sysctl, the loop boundary panic_timeout * 1000 could overflow and result in a zero-delay panic when panic_timeout is greater than INT_MAX/1000.

Fix this by elevating the precision of the loop boundary via assigning the result to a u64 integer, also in case the loop counter i might never be greater than u64 timeout = panic_timeout*1000, elevate its precision to u64(timer) as well. The same applies to timer_next replacing i_next which is initialized to 0.

---
Changes in v3:
- change the loop in panic, instead of change the sysctl
- avoid using 64-bit division, doing 64-bit mult instead

 kernel/panic.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
kernel/panic.c +7 -4
index e2157ca..ef6cd57 100644
@@ -313,13 +313,16 @@ void panic(const char *fmt, ...)
 		 * Delay timeout seconds before rebooting the machine.
 		 * We can't use the "normal" timers since we just panicked.
 		 */
+		u64 timeout = panic_timeout * 1000ULL;	/* avoid overflow */
+		u64 timer, timer_next = 0;
+
 		pr_emerg("Rebooting in %d seconds..\n", panic_timeout);
 
-		for (i = 0; i < panic_timeout * 1000; i += PANIC_TIMER_STEP) {
+		for (timer = 0; timer < timeout; timer += PANIC_TIMER_STEP) {
 			touch_nmi_watchdog();
-			if (i >= i_next) {
-				i += panic_blink(state ^= 1);
-				i_next = i + 3600 / PANIC_BLINK_SPD;
+			if (timer >= timer_next) {
+				timer += panic_blink(state ^= 1);
+				timer_next = timer + 3600 / PANIC_BLINK_SPD;
 			}
 			mdelay(PANIC_TIMER_STEP);
 		}
Signature · 1 line
2.7.4
#5 lore raw
In reply to Andrew Morton
Quoted 27 lines
Quoted 27 lines

Since panic_timeout is an integer passed-in through sysctl, the loop boundary panic_timeout * 1000 could overflow and result in a zero-delay panic when panic_timeout is greater than INT_MAX/1000.

Fix this by moving 1000 to the left, also in case i/1000 might never be greater than panic_timeout, change i to long long so that it strictly has more bits.

...

kernel/panic.c +2 -1
@@ -178,7 +178,8 @@ void panic(const char *fmt, ...)
 {
 	static char buf[1024];
 	va_list args;
-	long i, i_next = 0, len;
+	long long i;
+	long i_next = 0, len;
 	int state = 0;
 	int old_cpu, this_cpu;
 	bool _crash_kexec_post_notifiers = crash_kexec_post_notifiers;
@@ -315,7 +316,7 @@ void panic(const char *fmt, ...)
 		 */
 		pr_emerg("Rebooting in %d seconds..\n", panic_timeout);
 
Quoted 1 line

kernel/panic.c void panic(const char *fmt, ...) · line 318

Quoted 1 line
kernel/panic.c +0 -1
-		for (i = 0; i < panic_timeout * 1000; i += PANIC_TIMER_STEP)

{

Quoted 1 line

kernel/panic.c void panic(const char *fmt, ...) · line 319

Quoted 1 line
kernel/panic.c +1 -0
+		for (i = 0; i / 1000 < panic_timeout; i += PANIC_TIMER_STEP)

{

Quoted 3 lines

Problem is, 32-bit machines generally cannot perform 64-bit divides. So a call is emitted to the library function __divsi64() (I forget the

exact

Quoted 3 lines

name) which Linux doesn't implement (because it's so slow, and we don't want to be calling it by accident).

It's good to know, thanks for letting me know why 64-bit division is slow, and 64-multiplication is fast, surely doing so many 64-bit division will drag a lot, and should be prevented.

Quoted 9 lines

So a fix would be to call do_div() or something from include/linux/div64.h but it's all a great mess.

However we can do native 64-bit multiplication on 32-bit! So how about something like

kernel/panic.c void panic(const char *fmt, ...) · line 317

Quoted 7 lines
kernel/panic.c renamed from kernel/panic.c~a +2 -1
@@ -313,13 +313,16 @@ void panic(const char *fmt, ...)
 		 * Delay timeout seconds before rebooting the machine.
 		 * We can't use the "normal" timers since we just panicked.
 		 */
+		u64 timeout = panic_timeout * 1000;	/* avoid overflow */
+		u64 timer;
 		pr_emerg("Rebooting in %d seconds..\n", panic_timeout);
-		for (i = 0; i < panic_timeout * 1000; i += PANIC_TIMER_STEP)

{

kernel/panic.c void panic(const char *fmt, ...) · line 319

Quoted 1 line
kernel/panic.c renamed from kernel/panic.c~a +1 -0
+		for (timer = 0; timer < timeout; timer += PANIC_TIMER_STEP)

{

If using u64 as the loop boundary, would it be a problem if panic_timeout is negative? Since in the current code, if panic_timeout is negative, the loop will not be executed; as in the patched code, the loop boundary will be a huge unsigned value. I guess s64 should do?

If it's not a problem, I'll submit another patch enforcing the change, including the changes suggested by Matthew here:

Quoted 5 lines

kernel/panic.c void panic(const char *fmt, ...) · line 316

Quoted 1 line
kernel/panic.c +1 -0
+		u64 timeout = panic_timeout * 1000;	/* avoid overflow */

1000ULL to not truncate before the assignment.

kernel/panic.c void panic(const char *fmt, ...) · line 317

Quoted 1 line
kernel/panic.c +1 -0
+		u64 timer;

... as you implied lateru64 timer, timer_next;

Thank you guys so much for your valuable feedback, I learned a lot!

Best,
Changming