Avoid race w/ posix-cpu-timer and exiting tasks
Quoted 17 lines · john stultz
Hey Ingo,
We've occasionally come across OOPSes in posix-cpu-timer thread (as
well as tripping over theBUG_ON(tsk->exit_state there)where it appears the task we're processing exits out on us while we're using it.Thus this fix tries to avoid running the posix-cpu-timers on a task that is exiting.
kernel/posix-cpu-timers.c +0
-2@@ -1290,12 +1290,15 @@#undef UNEXPIRED -BUG_ON (tsk->exit_state); -
kernel/posix-cpu-timers.c line 1302
Quoted 10 lines
kernel/posix-cpu-timers.c +5
-0/* * Double-check with locks held. */ read_lock (&tasklist_lock); +/* Make sure the task doesn't exit under us. */ +if (unlikely (tsk->exit_state)) { +read_unlock (&tasklist_lock); +return ; + }spin_lock (&tsk->sighand->siglock);
I strongly believe this BUG_ON() is indeed wrong, and I did a similar patch a long ago:
[PATCH] posix-timers: remove false BUG_ON() from run_posix_cpu_timers() Commit 3de463c7d9d58f8cf3395268230cb20a4c15bffa
However it was reverted due to some unclear problems (I think those problems were not related to this patch).
Instead this patch was added:
[PATCH] Yet more posix-cpu-timer fixes Commit 3de463c7d9d58f8cf3395268230cb20a4c15bffa
and I still think this patch is not correct.
Quoting myself:
Quoted 13 lines
Review static void arm_timer(struct k_itimer *t · line 571
Quoted 8 lines · Roland McGrath
(quoted diff) +3
-0@@ -566,6 +566,9 @@ static void arm_timer(struct k_itimer *t struct cpu_timer_list *next; unsigned long i; + if (CPUCLOCK_PERTHREAD(timer->it_clock) && (p->flags & PF_EXITING)) + return; +Why
CPUCLOCK_PERTHREAD()?.
Quoted 4 lines
Also, this is racy, no? Why should
arm_timer()seePF_EXITINGwhich is set on another cpu without any barriers/locking? After all,arm_timer()can testPF_EXITINGbeforedo_exit()sets this flag, but set ->it_xxx_expires afterdo_exit()resets it.
Oleg.
Quoted 32 lines · Oleg Nesterov
Quoted 17 lines · john stultz
Hey Ingo,
We've occasionally come across OOPSes in posix-cpu-timer thread (as
well as tripping over theBUG_ON(tsk->exit_state there)where it appears the task we're processing exits out on us while we're using it.Thus this fix tries to avoid running the posix-cpu-timers on a task that is exiting.
kernel/posix-cpu-timers.c +0
-2@@ -1290,12 +1290,15 @@#undef UNEXPIRED -BUG_ON (tsk->exit_state); -kernel/posix-cpu-timers.c line 1302
Quoted 10 lines
kernel/posix-cpu-timers.c +5
-0/* * Double-check with locks held. */ read_lock (&tasklist_lock); +/* Make sure the task doesn't exit under us. */ +if (unlikely (tsk->exit_state)) { +read_unlock (&tasklist_lock); +return ; + }spin_lock (&tsk->sighand->siglock);I strongly believe this
BUG_ON()is indeed wrong, and I did a similar patch a long ago:
Quoted 2 lines
[PATCH] posix-timers: remove false BUG_ON() from run_posix_cpu_timers() Commit 3de463c7d9d58f8cf3395268230cb20a4c15bffa
Thanks for the pointer! Hmm. That patch is very similar to what I'm trying to avoid.
Just to be clear for everyone else, that commit id is for Linus' git tree. The patch I'm proposing is against the -rt tree, where the run_posix_timers() function has been converted to a kthread instead of being run from the timer interrupt context.
This allows the possibility of the the run_posix_timers_thread racing against the task its running on behalf and the task exiting, and I hope that is what my patch resolves (although I'm not confident its 100%).
The tsk->signal check from the patch above looks like it would avoid this as well. Is there a specific benefit to checking that over exit_state?
If anyone has further feedback or info about why the above was reverted it would be appreciated.
thanks
-john
Quoted 4 lines · john stultz
The
tsk->signalcheck from the patch above looks like it would avoid this as well. Is there a specific benefit to checking that overexit_state?
->exit_state is protected by tasklist_lock, and it would be nice to avoid it in run_posix_cpu_timers(). (I guess we could remove it right now, but I forgot the code). Yes, currently it doesn't matter because tsk == current.
Personally I dislike the testing of ->exit_state != 0 because unlike PF_EXITING or ->sighand/->signal it is changed from 0 to 1 in the middle of do_exit() path. Imho it should be used only by do_exit/do_wait path, but maybe this is just me.
Btw, I think there is another problem,
check_process_timers:
t = tsk;
do {
...
do {
t = next_thread(t);
} while (unlikely(t->flags & PF_EXITING));
} while (t != tsk);This can hang if the local timer interrupt happens right after do_exit() sets PF_EXITING ?
Oleg.