Skip to contentSkip to conversation

2.6.12-rc3-mm1

DISCUSSION Reuben Farrelly 2 participants 3 messages May 3, 2005 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Hi Andrew,

Quoted 10 lines · Andrew Morton

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.12-rc3/2.6.12-rc3-mm1/

- There's still a bug in the new timer code. If you think you hit it, please revert

timers-fixes-improvements-fix.patch then
timers-fixes-improvements-smp_processor_id-fix.patch then
timers-fixes-improvements.patch

or, better, fix the bug.

FWIW, I can reproduce this timer bug fairly consistently, by simply rebooting my cisco router. That means that my linux box has no default gateway, and hence the networking blows up within about 30s and dies with a stack trace which has references to timers.

I'll back out those three patches and see if it continues, but hopefully my little discovery is useful to someone in terms of coming up with a fix....

Box is an Intel 2.8/SMP

reuben

#2 lore raw
Quoted 15 lines · Reuben Farrelly

Hi Andrew,

Quoted 10 lines · Andrew Morton

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.12-rc3/2.6.12-rc3-mm1/

- There's still a bug in the new timer code. If you think you hit it, please revert

timers-fixes-improvements-fix.patch then
timers-fixes-improvements-smp_processor_id-fix.patch then
timers-fixes-improvements.patch

or, better, fix the bug.

Quoted 9 lines

FWIW, I can reproduce this timer bug fairly consistently, by simply rebooting my cisco router. That means that my linux box has no default gateway, and hence the networking blows up within about 30s and dies with a stack trace which has references to timers.

I'll back out those three patches and see if it continues, but hopefully my little discovery is useful to someone in terms of coming up with a fix....

Rather than backing things out, please add this instead:

From: Oleg Nesterov <[email protected]>

The bug was identified by Maneesh Soni.

When __mod_timer() changes timer's base it waits for the completion of timer->function. It is just stupid: the caller of __mod_timer() can held locks which would prevent completion of the timer's handler.

Solution: do not change the base of the currently running timer.

Side effect: __mod_timer() doesn't garantees anymore that timer will run on the local cpu.

---

 kernel/timer.c |   42 ++++++++++++++++++++----------------------
 1 files changed, 20 insertions(+), 22 deletions(-)
kernel/timer.c renamed from kernel/timer.c~timers-fix-__mod_timer-vs-__run_timers-deadlock +20 -22
diff -puN kernel/timer.c~timers-fix-__mod_timer-vs-__run_timers-deadlock kernel/timer.c
@@ -211,41 +211,39 @@ int __mod_timer(struct timer_list *timer
 	timer_base_t *base;
 	tvec_base_t *new_base;
 	unsigned long flags;
-	int ret = -1;
+	int ret;
 
 	BUG_ON(!timer->function);
 	check_timer(timer);
 
-	do {
-		base = lock_timer_base(timer, &flags);
-		new_base = &__get_cpu_var(tvec_bases);
+	base = lock_timer_base(timer, &flags);
 
-		/* Ensure the timer is serialized. */
-		if (base != &new_base->t_base
-			&& base->running_timer == timer)
-			goto unlock;
+	ret = 0;
+	if (timer_pending(timer)) {
+		detach_timer(timer, 0);
+		ret = 1;
+	}
 
-		ret = 0;
-		if (timer_pending(timer)) {
-			detach_timer(timer, 0);
-			ret = 1;
-		}
+	new_base = &__get_cpu_var(tvec_bases);
 
-		if (base != &new_base->t_base) {
+	if (base != &new_base->t_base) {
+		if (unlikely(base->running_timer == timer))
+			/* Don't change timer's base while it is running.
+			 * Needed for serialization of timer wrt itself. */
+			new_base = container_of(base, tvec_base_t, t_base);
+		else {
 			timer->base = NULL;
 			/* Safe: the timer can't be seen via ->entry,
 			 * and lock_timer_base checks ->base != 0. */
 			spin_unlock(&base->lock);
-			base = &new_base->t_base;
-			spin_lock(&base->lock);
-			timer->base = base;
+			spin_lock(&new_base->t_base.lock);
+			timer->base = &new_base->t_base;
 		}
+	}
 
-		timer->expires = expires;
-		internal_add_timer(new_base, timer);
-unlock:
-		spin_unlock_irqrestore(&base->lock, flags);
-	} while (ret < 0);
+	timer->expires = expires;
+	internal_add_timer(new_base, timer);
+	spin_unlock_irqrestore(&new_base->t_base.lock, flags);
 
 	return ret;
 }

_

#3 lore raw

Hi,

Quoted 23 lines · Andrew Morton
Quoted 13 lines · Reuben Farrelly

Hi Andrew,

Quoted 9 lines · Andrew Morton

ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.12-rc3/2.6.12-rc3-mm1/

- There's still a bug in the new timer code. If you think you hit it, please revert

timers-fixes-improvements-fix.patch then
timers-fixes-improvements-smp_processor_id-fix.patch then
timers-fixes-improvements.patch

Quoted 6 lines
Quoted 1 line

or, better, fix the bug.

FWIW, I can reproduce this timer bug fairly consistently, by simply rebooting my cisco router. That means that my linux box has no default gateway, and hence the networking blows up within about 30s and dies with a stack trace which has references to timers.

I'll back out those three patches and see if it continues, but hopefully my little discovery is useful to someone in terms of coming up with a fix....

Quoted 6 lines

Rather than backing things out, please add this instead:

From: Oleg Nesterov <[email protected]>

The bug was identified by Maneesh Soni.

Yup, this patch seems to fix it. I did 3 router reboots, and my linux box held up fine. Previous to this patch it had fatally oopsed 5 out of the 6 times I reloaded the router..

Thanks :)

reuben