Skip to contentSkip to conversation

del_timer_sync needed for UP RT systems.

DISCUSSION George Anzinger 2 participants 7 messages Apr 26, 2005 – Apr 27, 2005

lore
Share & tools
lore
#0 lore raw

Ingo,

In tracking down the failure of a system running the RT patch we have found a preemption between the time run_timer_list clears its spinlock and the call back function (in this case in posix-timers.c) gets its spinlock. The bad news is that it is possible for the timer to be released at this point leaving the call back code with a pointer to a bogus timer.

This was/is possible, of course, in SMP systems and is why del_timer_sync() exists. I suspect that del_timer_sync() needs to also do the "right thing" in UP RT systems.

This means removing the #ifdef CONFIG_SMP at about line 56 of kernel/timer.c thus setting up base->running_timer in all cases (or at least in SMP and RT cases) and also the #ifdef CONFIG_SMP around del_timer_sync() and, of course, the defines that redirect calls to these functions.

Does this make sense?

Signature · 2 lines
George Anzinger   [email protected]
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
#1 lore raw

Basically , there is a race condition in sys_timer_delete() and posix_timer_event() .

Quoted 1 line

From sys_timer_delete() :

/*

	 * This keeps any tasks waiting on the spin lock from thinking
	 * they got something (see the lock code above).
	 */
	if (timer->it_process) {
		if (timer->it_sigev_notify ==
(SIGEV_SIGNAL|SIGEV_THREAD_ID))
			put_task_struct(timer->it_process);
		timer->it_process = NULL;
	}
	unlock_timer(timer, flags);
	/* Preemption happens here. */
	release_posix_timer(timer, IT_ID_SET);

So when the timer is getting triggered , right before it takes the timer lock, preemption happens. You finish the code above. Then your preempted again right after unlock timer, shown above.

At this point, your triggering a timer that is half deleted, in posix_timer_fn() . timer->it_process = NULL , so when you try to send the signal to the timer owner you crash with an OOPS , cause the timer owner was just set to NULL.

George, at least CC me, after all I found/documented this bug ..

Preliminary fix included ..

Daniel

Quoted 12 lines · George Anzinger

Ingo,

In tracking down the failure of a system running the RT patch we have found a preemption between the time run_timer_list clears its spinlock and the call back function (in this case in posix-timers.c) gets its spinlock. The bad news is that it is possible for the timer to be released at this point leaving the call back code with a pointer to a bogus timer.

This was/is possible, of course, in SMP systems and is why del_timer_sync() exists. I suspect that del_timer_sync() needs to also do the "right thing" in UP RT systems.

Quoted 6 lines

This means removing the #ifdef CONFIG_SMP at about line 56 of kernel/timer.c thus setting up base->running_timer in all cases (or at least in SMP and RT cases) and also the #ifdef CONFIG_SMP around del_timer_sync() and, of course, the defines that redirect calls to these functions.

Does this make sense?

Source: MontaVista Software, Inc.
MR: 11506
Type: Defect Fix
Disposition: needs submitting to LKML

Description:
Ok, so here is the run down.. Basically , there is a race condition in
sys_timer_delete() and posix_timer_event() .

Quoted 1 line

From sys_timer_delete() :

		timer->it_process = NULL;
	}
	unlock_timer(timer, flags);
	/* Preemption happens here. */
	release_posix_timer(timer, IT_ID_SET);

So when the timer is getting triggered , right before it takes the timer lock, preemption happens. You finish the code above. Then your preempted again right after unlock timer, shown above.

At this point, your triggering a timer that is half deleted, in posix_timer_fn() . timer->it_process = NULL , so when you try to send the signal to the timer owner you crash with an OOPS , because the timer owner was just set to NULL.

kernel/posix-timers.c +8 -0
Index: linux-2.6.10/kernel/posix-timers.c
===================================================================
@@ -433,6 +433,14 @@ exit:
 
 int posix_timer_event(struct k_itimer *timr,int si_private)
 {
+	/*
+	 * If it_process is NULL then this timer is
+	 * in the process of being deleted. At this
+	 * point we can't do very much. So we
+	 * try to return gracefuly.
+	 */
+	if (timr->it_process == NULL) return 1;
+
 	memset(&timr->sigq->info, 0, sizeof(siginfo_t));
 	timr->sigq->info.si_sys_private = si_private;
 	/*
#2 lore raw
Quoted 24 lines · Daniel Walker

Basically , there is a race condition in sys_timer_delete() and posix_timer_event() .

Quoted 1 line

From sys_timer_delete() :

   /*
	 * This keeps any tasks waiting on the spin lock from thinking
	 * they got something (see the lock code above).
	 */
	if (timer->it_process) {
		if (timer->it_sigev_notify ==
(SIGEV_SIGNAL|SIGEV_THREAD_ID))
			put_task_struct(timer->it_process);
		timer->it_process = NULL;
	}
	unlock_timer(timer, flags);
	/* Preemption happens here. */
	release_posix_timer(timer, IT_ID_SET);

So when the timer is getting triggered , right before it takes the timer lock, preemption happens. You finish the code above. Then your preempted again right after unlock timer, shown above.

Quoted 6 lines

At this point, your triggering a timer that is half deleted, in posix_timer_fn() . timer->it_process = NULL , so when you try to send the signal to the timer owner you crash with an OOPS , cause the timer owner was just set to NULL.

George, at least CC me, after all I found/documented this bug ..

Sorry, my bad :(

Quoted 50 lines

Preliminary fix included ..

Daniel

Quoted 43 lines · George Anzinger

Ingo,

In tracking down the failure of a system running the RT patch we have found a preemption between the time run_timer_list clears its spinlock and the call back function (in this case in posix-timers.c) gets its spinlock. The bad news is that it is possible for the timer to be released at this point leaving the call back code with a pointer to a bogus timer.

This was/is possible, of course, in SMP systems and is why del_timer_sync() exists. I suspect that del_timer_sync() needs to also do the "right thing" in UP RT systems.

This means removing the #ifdef CONFIG_SMP at about line 56 of kernel/timer.c thus setting up base->running_timer in all cases (or at least in SMP and RT cases) and also the #ifdef CONFIG_SMP around del_timer_sync() and, of course, the defines that redirect calls to these functions.

Does this make sense?

------------------------------------------------------------------------

Source: MontaVista Software, Inc.
MR: 11506 Type: Defect Fix
Disposition: needs submitting to LKML Signed-off-by: Daniel Walker <[email protected]> Description:
Ok, so here is the run down.. Basically , there is a race condition in
sys_timer_delete() and posix_timer_event() .

Quoted 1 line

From sys_timer_delete() :

		timer->it_process = NULL;
	}
	unlock_timer(timer, flags);
	/* Preemption happens here. */
	release_posix_timer(timer, IT_ID_SET);

So when the timer is getting triggered , right before it takes the timer lock, preemption happens. You finish the code above. Then your preempted again right after unlock timer, shown above.

At this point, your triggering a timer that is half deleted, in posix_timer_fn() . timer->it_process = NULL , so when you try to send the signal to the timer owner you crash with an OOPS , because the timer owner was just set to NULL.

Index: linux-2.6.10/kernel/posix-timers.c
===================================================================

kernel/posix-timers.c +0 -0
@@ -433,6 +433,14 @@ exit:
 
Quoted 13 lines

kernel/posix-timers.c exit: · line 446

Quoted 13 lines
kernel/posix-timers.c +8 -0
 int posix_timer_event(struct k_itimer *timr,int si_private)
 {
+	/*
+	 * If it_process is NULL then this timer is
+	 * in the process of being deleted. At this
+	 * point we can't do very much. So we
+	 * try to return gracefuly.
+	 */
+	if (timr->it_process == NULL) return 1;
+
 	memset(&timr->sigq->info, 0, sizeof(siginfo_t));
 	timr->sigq->info.si_sys_private = si_private;
 	/*

The problem here is that the reference is to timr, a pointer to something which has been deleted. The memory may well be used elsewhere by this time which will make the test of it_process wrong. It also means we could mess with someone elses memory in the memset above.

Signature · 2 lines
George Anzinger   [email protected]
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
#3 lore raw
Quoted 4 lines · George Anzinger

The problem here is that the reference is to timr, a pointer to something which has been deleted. The memory may well be used elsewhere by this time which will make the test of it_process wrong. It also means we could mess with someone elses memory in the memset above.

Bottom line, you can use sys_timer_delete() on a timer, and trigger the same timer your deleting .. Those operations should be serialized, which they currently aren't ..

Daniel

#4 lore raw
Quoted 10 lines · Daniel Walker
Quoted 4 lines · George Anzinger

The problem here is that the reference is to timr, a pointer to something which has been deleted. The memory may well be used elsewhere by this time which will make the test of it_process wrong. It also means we could mess with someone elses memory in the memset above.

Bottom line, you can use sys_timer_delete() on a timer, and trigger the same timer your deleting .. Those operations should be serialized, which they currently aren't ..

Um, sorry typo, you _can't_ delete a timer, and trigger it at the same time. That's bad.

Daniel

#5 lore raw
In reply to Daniel Walker
Quoted 9 lines · Daniel Walker
Quoted 1 line · George Anzinger

The problem here is that the reference is to timr, a pointer to something which has been deleted. The memory may well be used elsewhere by this time which will make the test of it_process wrong. It also means we could mess with someone elses memory in the memset above.

Bottom line, you can use sys_timer_delete() on a timer, and trigger the same timer your deleting .. Those operations should be serialized, which they currently aren't ..

I agree. The change to do this is to use the del_timer_sync() or the del_singleshot_timer() code.

It is possible and desirable to be able to delete a running timer. We don't want to take it away from the timer call back routine, however, as that leads to "bad things". That is why these two del_* routines were written.

Quoted 1 line
Signature · 2 lines
George Anzinger   [email protected]
High-res-timers:  http://sourceforge.net/projects/high-res-timers/
#6 lore raw
Quoted 6 lines · George Anzinger

I agree. The change to do this is to use the del_timer_sync() or the del_singleshot_timer() code.

It is possible and desirable to be able to delete a running timer. We don't want to take it away from the timer call back routine, however, as that leads to "bad things". That is why these two del_* routines were written.

I'll defer to you on that, since I don't really know what timer people need..

After reviewing, del_timer_sync() I don't think that will stop this race. Because the wakeup happens before posix_timer_fn() is called in __run_timers() .

Daniel