2.6.12-rc3-mm1
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi Andrew,
Quoted 10 lines · Andrew Morton
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.12-rc3/2.6.12-rc3-mm1/
- There's still a bug in the new timer code. If you think you hit it, please revert
timers-fixes-improvements-fix.patch then
timers-fixes-improvements-smp_processor_id-fix.patch then
timers-fixes-improvements.patchor, better, fix the bug.
FWIW, I can reproduce this timer bug fairly consistently, by simply rebooting my cisco router. That means that my linux box has no default gateway, and hence the networking blows up within about 30s and dies with a stack trace which has references to timers.
I'll back out those three patches and see if it continues, but hopefully my little discovery is useful to someone in terms of coming up with a fix....
Box is an Intel 2.8/SMP
reuben
Quoted 15 lines · Reuben Farrelly
Hi Andrew,
Quoted 10 lines · Andrew Morton
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.12-rc3/2.6.12-rc3-mm1/
- There's still a bug in the new timer code. If you think you hit it, please revert
timers-fixes-improvements-fix.patch then
timers-fixes-improvements-smp_processor_id-fix.patch then
timers-fixes-improvements.patchor, better, fix the bug.
Quoted 9 lines
FWIW, I can reproduce this timer bug fairly consistently, by simply rebooting my cisco router. That means that my linux box has no default gateway, and hence the networking blows up within about 30s and dies with a stack trace which has references to timers.
I'll back out those three patches and see if it continues, but hopefully my little discovery is useful to someone in terms of coming up with a fix....
Rather than backing things out, please add this instead:
From: Oleg Nesterov <[email protected]>
The bug was identified by Maneesh Soni.
When __mod_timer() changes timer's base it waits for the completion of timer->function. It is just stupid: the caller of __mod_timer() can held locks which would prevent completion of the timer's handler.
Solution: do not change the base of the currently running timer.
Side effect: __mod_timer() doesn't garantees anymore that timer will run on the local cpu.
---
kernel/timer.c | 42 ++++++++++++++++++++---------------------- 1 files changed, 20 insertions(+), 22 deletions(-)
kernel/timer.c renamed from kernel/timer.c~timers-fix-__mod_timer-vs-__run_timers-deadlock +20 -22
@@ -211,41 +211,39 @@ int __mod_timer(struct timer_list *timertimer_base_t *base;tvec_base_t *new_base;unsigned long flags; -int ret =-1 ; +int ret;BUG_ON (!timer->function);check_timer (timer); -do { - base =lock_timer_base (timer, &flags); - new_base = &__get_cpu_var (tvec_bases); + base =lock_timer_base (timer, &flags); -/* Ensure the timer is serialized. */ -if (base != &new_base->t_base - && base->running_timer == timer) - goto unlock; + ret =0 ; +if (timer_pending (timer)) { +detach_timer (timer,0 ); + ret =1 ; + } - ret =0 ; -if (timer_pending (timer)) { -detach_timer (timer,0 ); - ret =1 ; - } + new_base = &__get_cpu_var (tvec_bases); -if (base != &new_base->t_base) { +if (base != &new_base->t_base) { +if (unlikely (base->running_timer == timer)) +/* Don't change timer's base while it is running. +* Needed for serialization of timer wrt itself. */ + new_base =container_of (base, tvec_base_t, t_base); +else { timer->base =NULL ;/* Safe: the timer can't be seen via ->entry, * and lock_timer_base checks ->base != 0. */ spin_unlock (&base->lock); - base = &new_base->t_base; -spin_lock (&base->lock); - timer->base = base; +spin_lock (&new_base->t_base.lock); + timer->base = &new_base->t_base; } + } - timer->expires = expires; -internal_add_timer (new_base, timer); -unlock: -spin_unlock_irqrestore (&base->lock, flags); - }while (ret <0 ); + timer->expires = expires; +internal_add_timer (new_base, timer); +spin_unlock_irqrestore (&new_base->t_base.lock, flags);return ret; }
_
Hi,
Quoted 23 lines · Andrew Morton
Quoted 13 lines · Reuben Farrelly
Hi Andrew,
Quoted 9 lines · Andrew Morton
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.12-rc3/2.6.12-rc3-mm1/
- There's still a bug in the new timer code. If you think you hit it, please revert
timers-fixes-improvements-fix.patch then
timers-fixes-improvements-smp_processor_id-fix.patch then
timers-fixes-improvements.patchQuoted 6 lines
Quoted 1 line
or, better, fix the bug.
FWIW, I can reproduce this timer bug fairly consistently, by simply rebooting my cisco router. That means that my linux box has no default gateway, and hence the networking blows up within about 30s and dies with a stack trace which has references to timers.
I'll back out those three patches and see if it continues, but hopefully my little discovery is useful to someone in terms of coming up with a fix....
Quoted 6 lines
Rather than backing things out, please add this instead:
From: Oleg Nesterov <[email protected]>
The bug was identified by Maneesh Soni.
Yup, this patch seems to fix it. I did 3 router reboots, and my linux box held up fine. Previous to this patch it had fatally oopsed 5 out of the 6 times I reloaded the router..
Thanks :)
reuben