Skip to contentSkip to conversation

x86_64 system lockup from userspace using setitimer()

PATCH In review Johannes Bauer 7 participants 16 messages Mar 13, 2007 – Apr 9, 2007

lore
Share & tools
lore
#0 lore raw

Dear Community,

I think I've encountered a bug with the Linux kernel which results in a complete system lockup and which can be started without root priviliges. It's reproducible with 2.6.20.1 and 2.6.20.2 and only x64_64 seems affected.

Here's the code which triggers the bug (originally found by me using an only partly initialized "struct itimerval" structure - hence the strange values in it_interval):

-----8<-----8<-----8<-----8<-----8<-----8<-----8<-----8<-----8<----- #include <stdio.h>
#include <sys/time.h>
#include <unistd.h>

int main(int argc, char **argv) {
    struct itimerval tim = {
        .it_interval = {
            .tv_sec = 140735669863712,
            .tv_usec = 4199521
        },
        .it_value = {
            .tv_sec = 0,
            .tv_usec =  100000
        }
    };
    setitimer(ITIMER_REAL, &tim, NULL);
    while (1) sleep(1);
    return 0;
}

-----8<-----8<-----8<-----8<-----8<-----8<-----8<-----8<-----8<-----

Compiled with gcc 4.1.1 with "gcc -O2 -Wall -o crash crash.c".

The sourcecode can be found at http://www.johannes-bauer.com/crash/crash.c and my kernel configuration is at http://www.johannes-bauer.com/crash/config

Any further questions: feel free to ask. Please CC me for any posts in this thread.

Greetings,
Johannes

Signature · 4 lines
"A PC without Windows is like a chocolate cake without mustard."

Johannes Bauer
91054 Erlangen
#1 lore raw
Quoted 6 lines · Johannes Bauer

Dear Community,

I think I've encountered a bug with the Linux kernel which results in a complete system lockup and which can be started without root priviliges. It's reproducible with 2.6.20.1 and 2.6.20.2 and only x64_64 seems affected.

I can also reproduce it on ia64 with 2.6.20. 2.6.16.42 is ok.

Andreas.

Signature · 4 lines
Andreas Schwab, SuSE Labs, [email protected]
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
PGP key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."
#2 lore raw
In reply to Johannes Bauer
Quoted 33 lines · Johannes Bauer

Dear Community,

I think I've encountered a bug with the Linux kernel which results in a complete system lockup and which can be started without root priviliges. It's reproducible with 2.6.20.1 and 2.6.20.2 and only x64_64 seems affected.

Here's the code which triggers the bug (originally found by me using an only partly initialized "struct itimerval" structure - hence the strange values in it_interval):

-----8<-----8<-----8<-----8<-----8<-----8<-----8<-----8<-----8<----- #include <stdio.h>
#include <sys/time.h>
#include <unistd.h>

int main(int argc, char **argv) {
    struct itimerval tim = {
        .it_interval = {
            .tv_sec = 140735669863712,
            .tv_usec = 4199521
        },
        .it_value = {
            .tv_sec = 0,
            .tv_usec =  100000
        }
    };
    setitimer(ITIMER_REAL, &tim, NULL);
    while (1) sleep(1);
    return 0;
}

-----8<-----8<-----8<-----8<-----8<-----8<-----8<-----8<-----8<-----

Quoted 8 lines

Compiled with gcc 4.1.1 with "gcc -O2 -Wall -o crash crash.c".

The sourcecode can be found at
http://www.johannes-bauer.com/crash/crash.c and my kernel configuration is at http://www.johannes-bauer.com/crash/config

Any further questions: feel free to ask. Please CC me for any posts in this thread.

Could this be fixed by:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=8bfd9a7a229b5f3d3eda5d7d45c2eebec5b4ba16

[PATCH] hrtimers: prevent possible itimer DoS

?

#3 lore raw
Quoted 10 lines · Chuck Ebbert
Quoted 5 lines
    struct itimerval tim = {
        .it_interval = {
            .tv_sec = 140735669863712,
            .tv_usec = 4199521
        },

Could this be fixed by:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=8bfd9a7a229b5f3d3eda5d7d45c2eebec5b4ba16

[PATCH] hrtimers: prevent possible itimer DoS

No. The possible DoS is only when high res timers are enabled, which is not the case in 2.6.20.

Looking at the values

140735669863712 = 0x7FFF 939C 0520

We convert second to nanoseconds:

140735669863712 * 1e9 = 0x1DCD 4BC3 6B82 914B 4000

The seconds value is limited to LONG_MAX, but on a 64 bit machine, the 140735669863712 is inside LONG_MAX and we have an multiplication overflow.

I'm not sure, how this results in a DoS, but I will look into this tomorrow morning, when I'm more awake.

tglx

#4 lore raw

PATCH 1/1 hrtimer: prevent overrun DoS in hrtimer_forward()

hrtimer_forward() does not check for the possible overflow of timer->expires. This can happen on 64 bit machines with large interval values and results currently in an endless loop in the softirq because the expiry value becomes negative and therefor the timer is expired all the time.

Check for this condition and set the expiry value to the max. expiry time in the future.

The fix should be applied to stable kernel series as well.

kernel/hrtimer.c +6 -0
index ec4cb9f..5e7122d 100644
@@ -644,6 +644,12 @@ hrtimer_forward(struct hrtimer *timer, k
 		orun++;
 	}
 	timer->expires = ktime_add(timer->expires, interval);
+	/*
+	 * Make sure, that the result did not wrap with a very large
+	 * interval.
+	 */
+	if (timer->expires.tv64 < 0)
+		timer->expires = ktime_set(KTIME_SEC_MAX, 0);
 
 	return orun;
 }
Acked-by on v1 1/1
#5 lore raw

Re: [PATCH] hrtimer: prevent overrun DoS in hrtimer_forward()

Quoted 6 lines · Thomas Gleixner

hrtimer_forward() does not check for the possible overflow of timer->expires. This can happen on 64 bit machines with large interval values and results currently in an endless loop in the softirq because the expiry value becomes negative and therefor the timer is expired all the time.

Quoted 6 lines

Check for this condition and set the expiry value to the max. expiry time in the future.

The fix should be applied to stable kernel series as well.

ouch ... nice one.

Ingo

#6 lore raw

Re: [PATCH] hrtimer: prevent overrun DoS in hrtimer_forward()

In reply to Thomas Gleixner
Quoted 20 lines · On Wed, 14 Mar 2007 11:00:12 +0100 Thomas Gleixner <[email protected]> wrote:

rtimer_forward() does not check for the possible overflow of timer->expires. This can happen on 64 bit machines with large interval values and results currently in an endless loop in the softirq because the expiry value becomes negative and therefor the timer is expired all the time.

Check for this condition and set the expiry value to the max. expiry time in the future.

The fix should be applied to stable kernel series as well.

kernel/hrtimer.c +0 -0
index ec4cb9f..5e7122d 100644
@@ -644,6 +644,12 @@ hrtimer_forward(struct hrtimer *timer, k
 		orun++;
 	}

kernel/hrtimer.c hrtimer_forward(struct hrtimer *timer, k · line 655

Quoted 10 lines
kernel/hrtimer.c +6 -0
 	timer->expires = ktime_add(timer->expires, interval);
+	/*
+	 * Make sure, that the result did not wrap with a very large
+	 * interval.
+	 */
+	if (timer->expires.tv64 < 0)
+		timer->expires = ktime_set(KTIME_SEC_MAX, 0);
 
 	return orun;
 }

kernel/hrtimer.c: In function 'hrtimer_forward':
kernel/hrtimer.c:652: warning: overflow in implicit constant conversion

problem is, KTIME_SEC_MAX is 9,223,372,036 and ktime_set() takes a `long'.

This?

include/linux/ktime.h renamed from include/linux/ktime.h~ktime_set-fix-arg-type +2 -2
@@ -72,13 +72,13 @@ typedef union {
  *
  * Return the ktime_t representation of the value
  */
-static inline ktime_t ktime_set(const long secs, const unsigned long nsecs)
+static inline ktime_t ktime_set(const s64 secs, const unsigned long nsecs)
 {
 #if (BITS_PER_LONG == 64)
 	if (unlikely(secs >= KTIME_SEC_MAX))
 		return (ktime_t){ .tv64 = KTIME_MAX };
 #endif
-	return (ktime_t) { .tv64 = (s64)secs * NSEC_PER_SEC + (s64)nsecs };
+	return (ktime_t) { .tv64 = secs * NSEC_PER_SEC + (s64)nsecs };
 }
 
 /* Subtract two ktime_t variables. rem = lhs -rhs: */

_

I worry about that `secs >= KTIME_SEC_MAX' comparison in there, too. Both operands are signed.

#7 lore raw

Re: [PATCH] hrtimer: prevent overrun DoS in hrtimer_forward()

Quoted 30 lines · Andrew Morton
Quoted 28 lines · On Wed, 14 Mar 2007 11:00:12 +0100 Thomas Gleixner <[email protected]> wrote:

rtimer_forward() does not check for the possible overflow of timer->expires. This can happen on 64 bit machines with large interval values and results currently in an endless loop in the softirq because the expiry value becomes negative and therefor the timer is expired all the time.

Check for this condition and set the expiry value to the max. expiry time in the future.

The fix should be applied to stable kernel series as well.

kernel/hrtimer.c +6 -0
index ec4cb9f..5e7122d 100644
@@ -644,6 +644,12 @@ hrtimer_forward(struct hrtimer *timer, k
 		orun++;
 	}
 	timer->expires = ktime_add(timer->expires, interval);
+	/*
+	 * Make sure, that the result did not wrap with a very large
+	 * interval.
+	 */
+	if (timer->expires.tv64 < 0)
+		timer->expires = ktime_set(KTIME_SEC_MAX, 0);
 
Quoted 7 lines

kernel/hrtimer.c hrtimer_forward(struct hrtimer *timer, k · line 655

Quoted 2 lines
kernel/hrtimer.c +0 -0
 	return orun;
 }

kernel/hrtimer.c: In function 'hrtimer_forward':
kernel/hrtimer.c:652: warning: overflow in implicit constant conversion

problem is, KTIME_SEC_MAX is 9,223,372,036 and ktime_set() takes a `long'.

Stupid me :(

Quoted 22 lines

This?

include/linux/ktime.h renamed from include/linux/ktime.h~ktime_set-fix-arg-type +2 -2
@@ -72,13 +72,13 @@ typedef union {
  *
  * Return the ktime_t representation of the value
  */
-static inline ktime_t ktime_set(const long secs, const unsigned long nsecs)
+static inline ktime_t ktime_set(const s64 secs, const unsigned long nsecs)
 {
 #if (BITS_PER_LONG == 64)
 	if (unlikely(secs >= KTIME_SEC_MAX))
 		return (ktime_t){ .tv64 = KTIME_MAX };
 #endif
-	return (ktime_t) { .tv64 = (s64)secs * NSEC_PER_SEC + (s64)nsecs };
+	return (ktime_t) { .tv64 = secs * NSEC_PER_SEC + (s64)nsecs };
 }
 
 /* Subtract two ktime_t variables. rem = lhs -rhs: */

_

Quoted 2 lines

I worry about that `secs >= KTIME_SEC_MAX' comparison in there, too. Both operands are signed.

I'd prefer this one: The maximum seconds value we can handle on 32bit is LONG_MAX.

include/linux/ktime.h +5 -1
index c68c7ac..248305b 100644
@@ -57,7 +57,11 @@ typedef union {
 } ktime_t;
 
 #define KTIME_MAX			((s64)~((u64)1 << 63))
-#define KTIME_SEC_MAX			(KTIME_MAX / NSEC_PER_SEC)
+#if (BITS_PER_LONG == 64)
+# define KTIME_SEC_MAX			(KTIME_MAX / NSEC_PER_SEC)
+#else
+# define KTIME_SEC_MAX			LONG_MAX
+#endif
 
 /*
  * ktime_t definitions when using the 64-bit scalar representation:
#8 lore raw

Re: [PATCH] hrtimer: prevent overrun DoS in hrtimer_forward()

Quoted 12 lines · Thomas Gleixner

I'd prefer this one: The maximum seconds value we can handle on 32bit is LONG_MAX.

include/linux/ktime.h +0 -0
index c68c7ac..248305b 100644
@@ -57,7 +57,11 @@ typedef union {
 } ktime_t;
 

include/linux/ktime.h typedef union { · line 67

Quoted 10 lines
include/linux/ktime.h +5 -1
 #define KTIME_MAX			((s64)~((u64)1 << 63))
-#define KTIME_SEC_MAX			(KTIME_MAX / NSEC_PER_SEC)
+#if (BITS_PER_LONG == 64)
+# define KTIME_SEC_MAX			(KTIME_MAX / NSEC_PER_SEC)
+#else
+# define KTIME_SEC_MAX			LONG_MAX
+#endif
 
 /*
  * ktime_t definitions when using the 64-bit scalar representation:

Just to be clear: this replaces the earlier patch, right?

#9 lore raw

Re: [PATCH] hrtimer: prevent overrun DoS in hrtimer_forward()

Quoted 20 lines · Chuck Ebbert
Quoted 19 lines · Thomas Gleixner

I'd prefer this one: The maximum seconds value we can handle on 32bit is LONG_MAX.

include/linux/ktime.h +5 -1
index c68c7ac..248305b 100644
@@ -57,7 +57,11 @@ typedef union {
 } ktime_t;
 
 #define KTIME_MAX			((s64)~((u64)1 << 63))
-#define KTIME_SEC_MAX			(KTIME_MAX / NSEC_PER_SEC)
+#if (BITS_PER_LONG == 64)
+# define KTIME_SEC_MAX			(KTIME_MAX / NSEC_PER_SEC)
+#else
+# define KTIME_SEC_MAX			LONG_MAX
+#endif
 
Quoted 5 lines

include/linux/ktime.h typedef union { · line 67

Quoted 3 lines
include/linux/ktime.h +0 -0
 /*
  * ktime_t definitions when using the 64-bit scalar representation:

Just to be clear: this replaces the earlier patch, right?

This replaces the fix Andrew did.

http://marc.info/?l=linux-kernel&m=117407812411997&w=2

tglx

#10 lore raw

Re: [PATCH] hrtimer: prevent overrun DoS in hrtimer_forward()

Quoted 20 lines · Thomas Gleixner
Quoted 19 lines · Chuck Ebbert
Quoted 18 lines · Thomas Gleixner

I'd prefer this one: The maximum seconds value we can handle on 32bit is LONG_MAX.

include/linux/ktime.h +5 -1
index c68c7ac..248305b 100644
@@ -57,7 +57,11 @@ typedef union {
 } ktime_t;
 
 #define KTIME_MAX			((s64)~((u64)1 << 63))
-#define KTIME_SEC_MAX			(KTIME_MAX / NSEC_PER_SEC)
+#if (BITS_PER_LONG == 64)
+# define KTIME_SEC_MAX			(KTIME_MAX / NSEC_PER_SEC)
+#else
+# define KTIME_SEC_MAX			LONG_MAX
+#endif
 
Quoted 9 lines
Quoted 4 lines

include/linux/ktime.h typedef union { · line 67

Quoted 3 lines
include/linux/ktime.h +0 -0
 /*
  * ktime_t definitions when using the 64-bit scalar representation:

Just to be clear: this replaces the earlier patch, right?

This replaces the fix Andrew did.

http://marc.info/?l=linux-kernel&m=117407812411997&w=2

Right, but is the original "Prevent DOS" patch from you still needed? Or did Andrew's patch replace that one, and now this replaces his?

#11 lore raw

Re: [PATCH] hrtimer: prevent overrun DoS in hrtimer_forward()

Quoted 9 lines · Chuck Ebbert
Quoted 6 lines
Quoted 1 line

Just to be clear: this replaces the earlier patch, right?

This replaces the fix Andrew did.

http://marc.info/?l=linux-kernel&m=117407812411997&w=2

Right, but is the original "Prevent DOS" patch from you still needed? Or did Andrew's patch replace that one, and now this replaces his?

The original patch is still needed - it handles the problem in the first place.

I missed to compile it for 32bit and Andrew did a fix, which I replaced.

tglx

#12 lore raw

Re: [PATCH] hrtimer: prevent overrun DoS in hrtimer_forward()

Quoted 6 lines · Thomas Gleixner
Quoted 5 lines · Chuck Ebbert
Quoted 5 lines
Quoted 1 line

Just to be clear: this replaces the earlier patch, right?

This replaces the fix Andrew did.

http://marc.info/?l=linux-kernel&m=117407812411997&w=2

Quoted 7 lines
Quoted 2 lines

Right, but is the original "Prevent DOS" patch from you still needed? Or did Andrew's patch replace that one, and now this replaces his?

The original patch is still needed - it handles the problem in the first place.

I missed to compile it for 32bit and Andrew did a fix, which I replaced.

Ah, OK, and both of those are now in the queue for 2.6.20-stable.

#13 lore raw

Re: [PATCH] hrtimer: prevent overrun DoS in hrtimer_forward()

In reply to Thomas Gleixner
Quoted 10 lines · Thomas Gleixner

hrtimer_forward() does not check for the possible overflow of timer->expires. This can happen on 64 bit machines with large interval values and results currently in an endless loop in the softirq because the expiry value becomes negative and therefor the timer is expired all the time.

Check for this condition and set the expiry value to the max. expiry time in the future.

The fix should be applied to stable kernel series as well.

Is this relevant for 2.6.16?

I'm asking since KTIME_SEC_MAX is not used in 2.6.16, and therefore the check in ktime_set() is also missing.

Quoted 10 lines
kernel/hrtimer.c +0 -0
index ec4cb9f..5e7122d 100644
@@ -644,6 +644,12 @@ hrtimer_forward(struct hrtimer *timer, k
 		orun++;
 	}

kernel/hrtimer.c hrtimer_forward(struct hrtimer *timer, k · line 655

Quoted 10 lines
kernel/hrtimer.c +6 -0
 	timer->expires = ktime_add(timer->expires, interval);
+	/*
+	 * Make sure, that the result did not wrap with a very large
+	 * interval.
+	 */
+	if (timer->expires.tv64 < 0)
+		timer->expires = ktime_set(KTIME_SEC_MAX, 0);
 
 	return orun;
 }

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#14 lore raw

Re: [PATCH] hrtimer: prevent overrun DoS in hrtimer_forward()

Quoted 10 lines · Adrian Bunk
Quoted 9 lines · Thomas Gleixner

hrtimer_forward() does not check for the possible overflow of timer->expires. This can happen on 64 bit machines with large interval values and results currently in an endless loop in the softirq because the expiry value becomes negative and therefor the timer is expired all the time.

Check for this condition and set the expiry value to the max. expiry time in the future.

Quoted 7 lines
Quoted 1 line

The fix should be applied to stable kernel series as well.

Is this relevant for 2.6.16?

I'm asking since KTIME_SEC_MAX is not used in 2.6.16, and therefore the check in ktime_set() is also missing.

KTIME_SEC_MAX was introduced with commit
96dd7421a06a5bc6eb731323b95efcb2fd864854

to fix a conversion problem on 64 bit machines, which is also present in 2.6.16 AFAICT.

The patch just makes use of this constant. So you need to pull it as well.

tglx

#15 lore raw

Re: [PATCH] hrtimer: prevent overrun DoS in hrtimer_forward()

Quoted 19 lines · Thomas Gleixner
Quoted 10 lines · Adrian Bunk
Quoted 9 lines · Thomas Gleixner

hrtimer_forward() does not check for the possible overflow of timer->expires. This can happen on 64 bit machines with large interval values and results currently in an endless loop in the softirq because the expiry value becomes negative and therefor the timer is expired all the time.

Check for this condition and set the expiry value to the max. expiry time in the future.

Quoted 7 lines
Quoted 1 line

The fix should be applied to stable kernel series as well.

Is this relevant for 2.6.16?

I'm asking since KTIME_SEC_MAX is not used in 2.6.16, and therefore the check in ktime_set() is also missing.

Quoted 8 lines

KTIME_SEC_MAX was introduced with commit
96dd7421a06a5bc6eb731323b95efcb2fd864854

to fix a conversion problem on 64 bit machines, which is also present in 2.6.16 AFAICT.

The patch just makes use of this constant. So you need to pull it as well.

Thanks, below is what I applied.

Quoted 1 line

tglx

cu
Adrian

Thomas Gleixner (3):
prevent timespec/timeval to ktime_t overflow fix MTIME_SEC_MAX on 32-bit
hrtimer: prevent overrun DoS in hrtimer_forward()

include/linux/ktime.h +10 -1
index f3dec45..4548ddb 100644
@@ -56,7 +56,12 @@ typedef union {
 #endif
 } ktime_t;
 
-#define KTIME_MAX			(~((u64)1 << 63))
+#define KTIME_MAX			((s64)~((u64)1 << 63))
+#if (BITS_PER_LONG == 64)
+# define KTIME_SEC_MAX			(KTIME_MAX / NSEC_PER_SEC)
+#else
+# define KTIME_SEC_MAX			LONG_MAX
+#endif
 
 /*
  * ktime_t definitions when using the 64-bit scalar representation:
@@ -77,6 +82,10 @@ typedef union {
  */
 static inline ktime_t ktime_set(const long secs, const unsigned long nsecs)
 {
+#if (BITS_PER_LONG == 64)
+	if (unlikely(secs >= KTIME_SEC_MAX))
+		return (ktime_t){ .tv64 = KTIME_MAX };
+#endif
 	return (ktime_t) { .tv64 = (s64)secs * NSEC_PER_SEC + (s64)nsecs };
 }
 
kernel/hrtimer.c +6 -0
index 14bc9cf..a29ceb0 100644
@@ -316,6 +316,12 @@ hrtimer_forward(struct hrtimer *timer, ktime_t interval)
 		orun++;
 	}
 	timer->expires = ktime_add(timer->expires, interval);
+	/*
+	 * Make sure, that the result did not wrap with a very large
+	 * interval.
+	 */
+	if (timer->expires.tv64 < 0)
+		timer->expires = ktime_set(KTIME_SEC_MAX, 0);
 
 	return orun;
 }