sched: clock wrap bug in 2.6.35-stable kills scheduling
Commit 305e683 introduced a wrap bug that causes task scheduling to fail after sched_clock() wrap. On a 1000 HZ system with 32bit jiffies, this occurs after 49.7 days.
Bug was introduced in 2.6.35.12 and is still present in linux-2.6.35.y HEAD.
Symptoms include one task getting all available cpu time while others get _none_. Setting niceness seems to make things even worse. Running this code in a new process after wrap completely lock up user space, thus triggering a watchdog reboot:
{ nice(1); while(1); }
To reproduce bug in reasonable time, one can up HZ. With 16000 HZ, bug occurs after 3.1 days.
Modifying sched_clock() to wrap when jiffies does triggers bug after 5 mins.
The basic problem seems to be that rq->clock_task get stuck forever with a really high value when rq->clock starts over from 0.
This fix solves that problem:
kernel/sched.c +2 -0
@@ -657,6 +657,8 @@ inline void update_rq_clock(struct rq *rq)if (!rq->skip_clock_update) rq->clock =sched_clock_cpu (cpu_of (rq)); irq_time =irq_time_cpu (cpu); +if (rq->clock < rq->clock_task) + rq->clock_task =0 ;if (rq->clock - irq_time > rq->clock_task) rq->clock_task = rq->clock - irq_time;
I can create a proper patch if the above is acceptable.
A more appropriate solution would perhaps be to pull some additional sched commits into stable branch, like fe44d62 and friends. I don't know enough about scheduler internals to tell.
All tests were performed on mips32 systems, but all systems with 32bit jiffies should be affected.
/Thomas
Quoted 5 lines · Thomas Lange
Commit 305e683 introduced a wrap bug that causes task scheduling to fail after
sched_clock()wrap. On a 1000 HZ system with 32bit jiffies, this occurs after 49.7 days.Bug was introduced in 2.6.35.12 and is still present in linux-2.6.35.y HEAD.
Is this an issue in 3.5-rc3?
2.6.35-stable really isn't maintained anymore, I would suggest upgrading to a more modern kernel version.
thanks,
greg k-h
Peter Zijlstra
maintainerQuoted 1 line · Thomas Lange
Bug was introduced in 2.6.35.12 and is still present in linux-2.6.35.y HEAD.
Ok, so nobody cares about that.. what does something recent like 3.5-rc4 do?
If that's fixed, find the patch that fixes it. If not, we'll have a look.
Peter Zijlstra
maintainerQuoted 8 lines · Peter Zijlstra
Quoted 1 line · Thomas Lange
Bug was introduced in 2.6.35.12 and is still present in linux-2.6.35.y HEAD.
Ok, so nobody cares about that.. what does something recent like 3.5-rc4 do?
If that's fixed, find the patch that fixes it. If not, we'll have a look.
If anything, I think something like the below ought to cure things.
--- kernel/sched/clock.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/clock.c +1 -1
@@ -74,7 +74,7 @@*/ unsigned long long __attribute__((weak))sched_clock (void ) { -return (unsigned long long )(jiffies -INITIAL_JIFFIES ) +return (unsigned long long )(get_jiffies_64 () -INITIAL_JIFFIES ) * (NSEC_PER_SEC /HZ ); }EXPORT_SYMBOL_GPL (sched_clock);
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Peter Zijlstra
Quoted 1 line · Thomas Lange
Bug was introduced in 2.6.35.12 and is still present in linux-2.6.35.y HEAD.
Ok, so nobody cares about that.. what does something recent like 3.5-rc4 do?
I haven't tested yet. I will try to find the time to do that.
Quoted 2 lines
Quoted 2 lines
If that's fixed, find the patch that fixes it. If not, we'll have a look.
Commit fe44d62 removed the if-clause that caused the problem. Most likely, this solved the problem.
Quoted 3 lines
If anything, I think something like the below ought to cure things.
I agree. Either we make sure that sched_clock() never wraps or if it wraps, that it wraps shortly after boot to catch problems early.
Quoted 10 lines
--- kernel/sched/clock.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)kernel/sched/clock.c +0
-0
kernel/sched/clock.c line 80
Quoted 8 lines
kernel/sched/clock.c +1
-1@@ -74,7 +74,7 @@*/ unsigned long long __attribute__((weak))sched_clock (void ) { -return (unsigned long long )(jiffies -INITIAL_JIFFIES ) +return (unsigned long long )(get_jiffies_64 () -INITIAL_JIFFIES ) * (NSEC_PER_SEC /HZ ); }EXPORT_SYMBOL_GPL (sched_clock);
/Thomas
Peter Zijlstra
maintainerQuoted 2 lines · Thomas Lange
Commit fe44d62 removed the if-clause that caused the problem. Most likely, this solved the problem.
Ah, probably. IIRC we also fixed the short wrap of those arm clocks since.
Quoted 4 lines
Quoted 1 line
If anything, I think something like the below ought to cure things.
I agree. Either we make sure that
sched_clock()never wraps or if it wraps, that it wraps shortly after boot to catch problems early.
Right, sched_clock() should only ever wrap on the full u64.