Skip to contentSkip to conversation

Race condition with it_real_fn in kernel/itimer.c

BUG Steven Rostedt 2 participants 4 messages Jun 15, 2005

lore
Share & tools
lore
#0 lore raw

OK, I found this bug on an older version of Ingo's RT kernel with my own customizations. This is a very hard to get race condition but my logging traced it pretty good and this looks like it may also be a bug for both Ingo's RT kernel and the vanilla kernel. This was on an SMP machine.

Here's the race (since this was initiated with XFree86, I'll use it as the userland process that started this):

XFree86: calls sys_call

    -> sys_setitimer
       -> do_setitimer 
           (grabs tsk->sighand->siglock)
           -> del_timer_sync
     which has the following code:

	for_each_online_cpu(i) {
		base = &per_cpu(tvec_bases, i);
		if (base->running_timer == timer) {
			while (base->running_timer == timer) {
				cpu_relax();
				preempt_check_resched();
			}
			break;
		}
	}

If the timer hasn't gone off yet on another cpu, it will spin until it is finished. Now here's the problem:

ksoftirqd: calls do_softirq -> ... -> run_timer_softirq

      -> __run_timers
        -> it_real_fn
            -> send_group_sig_info
              -> group_send_sig_info
                  (grabs p->sighand->siglock)

Now, since the ksoftirqd is what changes running_timer, we have a deadlock!

What would be the harm in doing something like:

kernel/itimer.c renamed from kernel/itimer.c.orig +5 -1
@@ -153,11 +153,15 @@
 
 	switch (which) {
 	case ITIMER_REAL:
+	try_again:
 		spin_lock_irq(&tsk->sighand->siglock);
 		interval = tsk->signal->it_real_incr;
 		val = it_real_value(tsk->signal);
-		if (val)
+		if (val) {
+			spin_unlock_irq(&tsk->sighand->siglock);
 			del_timer_sync(&tsk->signal->real_timer);
+			goto try_again;
+		}
 		tsk->signal->it_real_incr =
 			timeval_to_jiffies(&value->it_interval);
 		it_real_arm(tsk, timeval_to_jiffies(&value->it_value));

-- Steve

#1 lore raw
Quoted 12 lines · Steven Rostedt
	for_each_online_cpu(i) {
		base = &per_cpu(tvec_bases, i);
		if (base->running_timer == timer) {
			while (base->running_timer == timer) {
				cpu_relax();
				preempt_check_resched();
			}
			break;
		}
	}
Quoted 2 lines

If the timer hasn't gone off yet on another cpu, it will spin until it is finished. Now here's the problem:

This was suppose to say...

If the timer is currently going off on another cpu.

-- Steve

#2 lore raw
In reply to Steven Rostedt
Quoted 31 lines · Steven Rostedt

OK, I found this bug on an older version of Ingo's RT kernel with my own customizations. This is a very hard to get race condition but my logging traced it pretty good and this looks like it may also be a bug for both Ingo's RT kernel and the vanilla kernel. This was on an SMP machine.

Here's the race (since this was initiated with XFree86, I'll use it as the userland process that started this):

XFree86: calls sys_call

    -> sys_setitimer
       -> do_setitimer 
           (grabs tsk->sighand->siglock)
           -> del_timer_sync
     which has the following code:

	for_each_online_cpu(i) {
		base = &per_cpu(tvec_bases, i);
		if (base->running_timer == timer) {
			while (base->running_timer == timer) {
				cpu_relax();
				preempt_check_resched();
			}
			break;
		}
	}

If the timer hasn't gone off yet on another cpu, it will spin until it is finished. Now here's the problem:

ksoftirqd: calls do_softirq -> ... -> run_timer_softirq

Quoted 8 lines
      -> __run_timers
        -> it_real_fn
            -> send_group_sig_info
              -> group_send_sig_info
                  (grabs p->sighand->siglock)

Now, since the ksoftirqd is what changes running_timer, we have a deadlock!

Yes, that's a deadlock.

Quoted 13 lines

What would be the harm in doing something like:

kernel/itimer.c renamed from kernel/itimer.c.orig +1 -0
@@ -153,11 +153,15 @@
 
 	switch (which) {
 	case ITIMER_REAL:
+	try_again:
 		spin_lock_irq(&tsk->sighand->siglock);
 		interval = tsk->signal->it_real_incr;

kernel/itimer.c line 167

Quoted 10 lines
kernel/itimer.c renamed from kernel/itimer.c.orig +4 -1
 		val = it_real_value(tsk->signal);
-		if (val)
+		if (val) {
+			spin_unlock_irq(&tsk->sighand->siglock);
 			del_timer_sync(&tsk->signal->real_timer);
+			goto try_again;
+		}
 		tsk->signal->it_real_incr =
 			timeval_to_jiffies(&value->it_interval);
 		it_real_arm(tsk, timeval_to_jiffies(&value->it_value));

And that will fix it. (Labels start in column zero, and a comment is needed here).

However I wonder if it would be sufficient to remove the del_timer_sync() call altogether and just do mod_timer() in it_real_arm().

If the handler happens to be running on another CPU and if the handler tries to run mod_timer() _after_ the do_setitimer() has run mod_timer(), the handler will use the desired value of it_real_incr anyway.

#3 lore raw
Quoted 3 lines · Andrew Morton

And that will fix it. (Labels start in column zero, and a comment is needed here).

I blame emacs for that bad label :-)

Quoted 8 lines

However I wonder if it would be sufficient to remove the del_timer_sync() call altogether and just do mod_timer() in it_real_arm().

If the handler happens to be running on another CPU and if the handler tries to run mod_timer() _after_ the do_setitimer() has run mod_timer(), the handler will use the desired value of it_real_incr anyway.

So do you prefer a patch like the following?

kernel/itimer.c renamed from kernel/itimer.c.orig +4 -4
@@ -118,6 +118,8 @@
  */
 static inline void it_real_arm(struct task_struct *p, unsigned long interval)
 {
+	unsigned long expires;
+
 	p->signal->it_real_value = interval; /* XXX unnecessary field?? */
 	if (interval == 0)
 		return;
@@ -127,8 +129,8 @@
 	 * the interval requested. This could happen if
 	 * time requested % (usecs per jiffy) is more than the usecs left
 	 * in the current jiffy */
-	p->signal->real_timer.expires = jiffies + interval + 1;
-	add_timer(&p->signal->real_timer);
+	expires = jiffies + interval + 1;
+	mod_timer(&p->signal->real_timer, expires);
 }
 
 void it_real_fn(unsigned long __data)
@@ -156,8 +158,6 @@
 		spin_lock_irq(&tsk->sighand->siglock);
 		interval = tsk->signal->it_real_incr;
 		val = it_real_value(tsk->signal);
-		if (val)
-			del_timer_sync(&tsk->signal->real_timer);
 		tsk->signal->it_real_incr =
 			timeval_to_jiffies(&value->it_interval);
 		it_real_arm(tsk, timeval_to_jiffies(&value->it_value));

Now the question is, what happens on the following scenario?

ksoftirqd:

calls it_real_func

process:

calls do_setitimer blocks on siglock;

ksoftirqd: unlocks siglock calls it_real_arm and after it assigns expires it takes an interrupt before calling mod_timer.

process:

calls it_real_arm and does the changes to mod_timer first.

ksoftirqd: comes back from interrupt and then calls mod_timer with the wrong value.

This may be a small chance in hell of happening, and the result may not be to drastic, but this is still a race condition. So far I think that my unconditional calling of del_timer_sync, although inefficient, it doesn't have any races.

-- Steve