signal: break out of wait loops on kthread_stop()
PATCH 1/1 signal: break out of wait loops on kthread_stop()
I was recently surprised to learn that msleep_interruptible(), wait_for_completion_interruptible_timeout(), and related functions simply hung when I called kthread_stop() on kthreads using them. The solution to fixing the case with msleep_interruptible() was more simply to move to schedule_timeout_interruptible(). Why?
The reason is that msleep_interruptible(), and many functions just like it, has a loop like this:
while (timeout && !signal_pending(current))
timeout = schedule_timeout_interruptible(timeout);The call to kthread_stop() woke up the thread, so schedule_timeout_ interruptible() returned early, but because signal_pending() returned true, it went back into another timeout, which was never woken up.
This wait loop pattern is common to various pieces of code, and I suspect that subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.
So this commit causes signal_pending() to return true when kthread_stop() is called. This is already what's done for TIF_NOTIFY_SIGNAL, for these same purposes of breaking out of wait loops, so a similar KTHREAD_SHOULD_STOP check isn't too much different.
--- include/linux/kthread.h | 1 + include/linux/sched/signal.h | 9 +++++++++ kernel/kthread.c | 6 ++++++ 3 files changed, 16 insertions(+)
include/linux/kthread.h +1 -0
@@ -87,6 +87,7 @@ void kthread_bind(struct task_struct *k, unsigned int cpu);void kthread_bind_mask (struct task_struct *k,const struct cpumask *mask);int kthread_stop (struct task_struct *k);bool kthread_should_stop (void ); +bool __kthread_should_stop (struct task_struct *k);bool kthread_should_park (void );bool __kthread_should_park (struct task_struct *k);bool kthread_freezable_should_stop (bool *was_frozen);
include/linux/sched/signal.h +9 -0
@@ -11,6 +11,7 @@#include <linux/refcount.h> #include <linux/posix-timers.h> #include <linux/mm_types.h> +#include <linux/kthread.h> #include <asm/ptrace.h> /*
@@ -397,6 +398,14 @@ static inline int signal_pending(struct task_struct *p)*/ if (unlikely (test_tsk_thread_flag (p,TIF_NOTIFY_SIGNAL )))return 1 ; + +/* +* Likewise, KTHREAD_SHOULD_STOP isn't really a signal, but it also +* requires the same behavior, lest wait loops go forever. +*/ +if (unlikely (__kthread_should_stop (p))) +return 1 ; +return task_sigpending (p); }
kernel/kthread.c +6 -0
@@ -145,6 +145,12 @@ void free_kthread_struct(struct task_struct *k)kfree (kthread); } +bool __kthread_should_stop (struct task_struct *k) +{ +return (k->flags &PF_KTHREAD ) && +test_bit (KTHREAD_SHOULD_STOP , &to_kthread (k)->flags); +} +/** * kthread_should_stop - should this kthread return now? *
Signature · 1 line
2.35.1
kernel/kthread.c void free_kthread_struct(struct task_struct *k) · line 152
Quoted 5 lines · Jason A. Donenfeld
kernel/kthread.c +5
-0+bool __kthread_should_stop (struct task_struct *k) +{ +return (k->flags &PF_KTHREAD ) && +test_bit (KTHREAD_SHOULD_STOP , &to_kthread (k)->flags); +}
This used to be a racy pattern; not sure it still is since Eric poked at this last, but please use __to_kthread() instead.
Quoted 10 lines · Peter Zijlstra
kernel/kthread.c void free_kthread_struct(struct task_struct *k) · line 152
Quoted 5 lines · Jason A. Donenfeld
kernel/kthread.c +5
-0+bool __kthread_should_stop (struct task_struct *k) +{ +return (k->flags &PF_KTHREAD ) && +test_bit (KTHREAD_SHOULD_STOP , &to_kthread (k)->flags); +}This used to be a racy pattern; not sure it still is since Eric poked at this last, but please use
__to_kthread()instead.
Ah, indeed. Will send a v2.
Jason
PATCH v2 1/1 signal: break out of wait loops on kthread_stop()
I was recently surprised to learn that msleep_interruptible(), wait_for_completion_interruptible_timeout(), and related functions simply hung when I called kthread_stop() on kthreads using them. The solution to fixing the case with msleep_interruptible() was more simply to move to schedule_timeout_interruptible(). Why?
The reason is that msleep_interruptible(), and many functions just like it, has a loop like this:
while (timeout && !signal_pending(current))
timeout = schedule_timeout_interruptible(timeout);The call to kthread_stop() woke up the thread, so schedule_timeout_ interruptible() returned early, but because signal_pending() returned true, it went back into another timeout, which was never woken up.
This wait loop pattern is common to various pieces of code, and I suspect that subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.
So this commit causes signal_pending() to return true when kthread_stop() is called. This is already what's done for TIF_NOTIFY_SIGNAL, for these same purposes of breaking out of wait loops, so a similar KTHREAD_SHOULD_STOP check isn't too much different.
--- include/linux/kthread.h | 1 + include/linux/sched/signal.h | 9 +++++++++ kernel/kthread.c | 8 ++++++++ 3 files changed, 18 insertions(+)
include/linux/kthread.h +1 -0
@@ -87,6 +87,7 @@ void kthread_bind(struct task_struct *k, unsigned int cpu);void kthread_bind_mask (struct task_struct *k,const struct cpumask *mask);int kthread_stop (struct task_struct *k);bool kthread_should_stop (void ); +bool __kthread_should_stop (struct task_struct *k);bool kthread_should_park (void );bool __kthread_should_park (struct task_struct *k);bool kthread_freezable_should_stop (bool *was_frozen);
include/linux/sched/signal.h +9 -0
@@ -11,6 +11,7 @@#include <linux/refcount.h> #include <linux/posix-timers.h> #include <linux/mm_types.h> +#include <linux/kthread.h> #include <asm/ptrace.h> /*
@@ -397,6 +398,14 @@ static inline int signal_pending(struct task_struct *p)*/ if (unlikely (test_tsk_thread_flag (p,TIF_NOTIFY_SIGNAL )))return 1 ; + +/* +* Likewise, KTHREAD_SHOULD_STOP isn't really a signal, but it also +* requires the same behavior, lest wait loops go forever. +*/ +if (unlikely (__kthread_should_stop (p))) +return 1 ; +return task_sigpending (p); }
kernel/kthread.c +8 -0
@@ -145,6 +145,14 @@ void free_kthread_struct(struct task_struct *k)kfree (kthread); } +bool __kthread_should_stop (struct task_struct *k) +{ +struct kthread *kthread =__to_kthread (k); + +return kthread &&test_bit (KTHREAD_SHOULD_STOP , &kthread->flags); +} +EXPORT_SYMBOL_GPL (__kthread_should_stop); +/** * kthread_should_stop - should this kthread return now? *
Signature · 1 line
2.35.1
Quoted 20 lines · Jason A. Donenfeld
I was recently surprised to learn that
msleep_interruptible(),wait_for_completion_interruptible_timeout(), and related functions simply hung when I calledkthread_stop()on kthreads using them. The solution to fixing the case withmsleep_interruptible()was more simply to move toschedule_timeout_interruptible(). Why?The reason is that
msleep_interruptible(), and many functions just like it, has a loop like this:while (timeout && !signal_pending(current)) timeout = schedule_timeout_interruptible(timeout);The call to
kthread_stop()woke up the thread, soschedule_timeout_interruptible()returned early, but becausesignal_pending()returned true, it went back into another timeout, which was never woken up.This wait loop pattern is common to various pieces of code, and I suspect that subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.
Quoted 5 lines
So this commit causes
signal_pending()to return true whenkthread_stop()is called. This is already what's done forTIF_NOTIFY_SIGNAL, for these same purposes of breaking out of wait loops, so a similarKTHREAD_SHOULD_STOPcheck isn't too much different.
Semantically this makes a lot of sense.
Bloating up signal_pending which is mainly called in non-kthread contexts is undesirable.
Instead could you modify kthread_stop to call set_notify_signal().
That is exactly what set_notify_signal is there for. When you don't actually have a signal but you want to break out of an interruptible loop. My last round of work in the area decoupled set_notify_signal from any other semantics.
It would be nice to get everything down so that we only need to test TIF_NOTIFY_SIGNAL in signal_pending. Unfortunately to do that I need to do something with task_sigpending, and it hasn't been important enough to weed through all of those details yet.
Eric
Quoted 71 lines
Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Eric W. Biederman <[email protected]>Cc Toke Høiland-Jørgensen <[email protected]>Cc Kalle Valo <[email protected]>Cc Johannes Berg <[email protected]>Signed-off-by Jason A. Donenfeld <[email protected]>--- include/linux/kthread.h | 1 + include/linux/sched/signal.h | 9 +++++++++ kernel/kthread.c | 8 ++++++++ 3 files changed, 18 insertions(+)include/linux/kthread.h +1
-0@@ -87,6 +87,7 @@ void kthread_bind(struct task_struct *k, unsigned int cpu);void kthread_bind_mask (struct task_struct *k,const struct cpumask *mask);int kthread_stop (struct task_struct *k);bool kthread_should_stop (void ); +bool __kthread_should_stop (struct task_struct *k);bool kthread_should_park (void );bool __kthread_should_park (struct task_struct *k);bool kthread_freezable_should_stop (bool *was_frozen);include/linux/sched/signal.h +9
-0@@ -11,6 +11,7 @@#include <linux/refcount.h> #include <linux/posix-timers.h> #include <linux/mm_types.h> +#include <linux/kthread.h> #include <asm/ptrace.h> /* @@ -397,6 +398,14 @@ static inline int signal_pending(struct task_struct *p)*/ if (unlikely (test_tsk_thread_flag (p,TIF_NOTIFY_SIGNAL )))return 1 ; + +/* +* Likewise, KTHREAD_SHOULD_STOP isn't really a signal, but it also +* requires the same behavior, lest wait loops go forever. +*/ +if (unlikely (__kthread_should_stop (p))) +return 1 ; +return task_sigpending (p); }kernel/kthread.c +8
-0@@ -145,6 +145,14 @@ void free_kthread_struct(struct task_struct *k)kfree (kthread); } +bool __kthread_should_stop (struct task_struct *k) +{ +struct kthread *kthread =__to_kthread (k); + +return kthread &&test_bit (KTHREAD_SHOULD_STOP , &kthread->flags); +} +EXPORT_SYMBOL_GPL (__kthread_should_stop); +/** * kthread_should_stop - should this kthread return now? *
Hi Eric,
Quoted 4 lines · Eric W. Biederman
Semantically this makes a lot of sense.
Bloating up
signal_pendingwhich is mainly called in non-kthread contexts is undesirable.
I guess I understand that concern, but does it really matter here? This is called by code that waits anyway, so it's not like performance matters at all, right?
Quoted 6 lines
Instead could you modify
kthread_stopto callset_notify_signal().That is exactly what
set_notify_signalis there for. When you don't actually have a signal but you want to break out of an interruptible loop. My last round of work in the area decoupledset_notify_signalfrom any other semantics.
This sounds like the best option here, if in fact it does work. I'll send in a patch for that and we can see how it interacts with the other work you're doing.
Jason
PATCH v3 1/1 signal: break out of wait loops on kthread_stop()
I was recently surprised to learn that msleep_interruptible(), wait_for_completion_interruptible_timeout(), and related functions simply hung when I called kthread_stop() on kthreads using them. The solution to fixing the case with msleep_interruptible() was more simply to move to schedule_timeout_interruptible(). Why?
The reason is that msleep_interruptible(), and many functions just like it, has a loop like this:
while (timeout && !signal_pending(current))
timeout = schedule_timeout_interruptible(timeout);The call to kthread_stop() woke up the thread, so schedule_timeout_ interruptible() returned early, but because signal_pending() returned true, it went back into another timeout, which was never woken up.
This wait loop pattern is common to various pieces of code, and I suspect that the subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.
So this commit causes signal_pending() to return true when kthread_stop() is called, by setting TIF_NOTIFY_SIGNAL.
The same also applies to the similar kthread_park() functionality.
--- kernel/kthread.c | 4 ++++ 1 file changed, 4 insertions(+)
kernel/kthread.c +4 -0
@@ -661,12 +661,14 @@ int kthread_park(struct task_struct *k)set_bit (KTHREAD_SHOULD_PARK , &kthread->flags);if (k != current) { +test_and_set_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );wake_up_process (k);/* * Wait for __kthread_parkme() to complete(), this means we * _will_ have TASK_PARKED and are about to call schedule(). */ wait_for_completion (&kthread->parked); +clear_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );/* * Now wait for that schedule() to complete and the task to * get scheduled out.
@@ -704,8 +706,10 @@ int kthread_stop(struct task_struct *k) kthread =to_kthread (k);set_bit (KTHREAD_SHOULD_STOP , &kthread->flags);kthread_unpark (k); +test_and_set_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );wake_up_process (k);wait_for_completion (&kthread->exited); +clear_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL ); ret = kthread->result;put_task_struct (k);
Signature · 1 line
2.35.1
Hey Eric,
Quoted 69 lines · Jason A. Donenfeld
I was recently surprised to learn that
msleep_interruptible(),wait_for_completion_interruptible_timeout(), and related functions simply hung when I calledkthread_stop()on kthreads using them. The solution to fixing the case withmsleep_interruptible()was more simply to move toschedule_timeout_interruptible(). Why?The reason is that
msleep_interruptible(), and many functions just like it, has a loop like this:while (timeout && !signal_pending(current)) timeout = schedule_timeout_interruptible(timeout);The call to
kthread_stop()woke up the thread, soschedule_timeout_interruptible()returned early, but becausesignal_pending()returned true, it went back into another timeout, which was never woken up.This wait loop pattern is common to various pieces of code, and I suspect that the subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.
So this commit causes
signal_pending()to return true whenkthread_stop()is called, by settingTIF_NOTIFY_SIGNAL.The same also applies to the similar
kthread_park()functionality.Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Eric W. Biederman <[email protected]>Cc Toke Høiland-Jørgensen <[email protected]>Cc Kalle Valo <[email protected]>Cc Johannes Berg <[email protected]>Signed-off-by Jason A. Donenfeld <[email protected]>--- kernel/kthread.c | 4 ++++ 1 file changed, 4 insertions(+)kernel/kthread.c +4
-0@@ -661,12 +661,14 @@ int kthread_park(struct task_struct *k)set_bit (KTHREAD_SHOULD_PARK , &kthread->flags);if (k != current) { +test_and_set_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );wake_up_process (k);/* * Wait for __kthread_parkme() to complete(), this means we * _will_ have TASK_PARKED and are about to call schedule(). */ wait_for_completion (&kthread->parked); +clear_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );/* * Now wait for that schedule() to complete and the task to * get scheduled out. @@ -704,8 +706,10 @@ int kthread_stop(struct task_struct *k) kthread =to_kthread (k);set_bit (KTHREAD_SHOULD_STOP , &kthread->flags);kthread_unpark (k); +test_and_set_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );wake_up_process (k);wait_for_completion (&kthread->exited); +clear_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL ); ret = kthread->result;put_task_struct (k);Signature · 1 line
2.35.1
Is this more to the tune of what you had in mind in your message [1]?
Jason
Hi Eric,
Quoted 76 lines · Jason A. Donenfeld
Hey Eric,
Quoted 67 lines · Jason A. Donenfeld
I was recently surprised to learn that
msleep_interruptible(),wait_for_completion_interruptible_timeout(), and related functions simply hung when I calledkthread_stop()on kthreads using them. The solution to fixing the case withmsleep_interruptible()was more simply to move toschedule_timeout_interruptible(). Why?The reason is that
msleep_interruptible(), and many functions just like it, has a loop like this:while (timeout && !signal_pending(current)) timeout = schedule_timeout_interruptible(timeout);The call to
kthread_stop()woke up the thread, soschedule_timeout_interruptible()returned early, but becausesignal_pending()returned true, it went back into another timeout, which was never woken up.This wait loop pattern is common to various pieces of code, and I suspect that the subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.
So this commit causes
signal_pending()to return true whenkthread_stop()is called, by settingTIF_NOTIFY_SIGNAL.The same also applies to the similar
kthread_park()functionality.Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Eric W. Biederman <[email protected]>Cc Toke Høiland-Jørgensen <[email protected]>Cc Kalle Valo <[email protected]>Cc Johannes Berg <[email protected]>Signed-off-by Jason A. Donenfeld <[email protected]>--- kernel/kthread.c | 4 ++++ 1 file changed, 4 insertions(+)kernel/kthread.c +4
-1@@ -661,12 +661,14 @@ int kthread_park(struct task_struct *k)set_bit (KTHREAD_SHOULD_PARK , &kthread->flags);if (k != current) { +test_and_set_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );wake_up_process (k);/* * Wait for __kthread_parkme() to complete(), this means we * _will_ have TASK_PARKED and are about to call schedule(). */ wait_for_completion (&kthread->parked); +clear_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );/* * Now wait for that schedule() to complete and the task to * get scheduled out. @@ -704,8 +706,10 @@ int kthread_stop(struct task_struct *k) kthread =to_kthread (k);set_bit (KTHREAD_SHOULD_STOP , &kthread->flags);kthread_unpark (k); +test_and_set_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );wake_up_process (k);wait_for_completion (&kthread->exited); +clear_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL ); ret = kthread->result;put_task_struct (k); --Quoted 2 lines
2.35.1
Is this more to the tune of what you had in mind in your message [1]?
Quoted 3 lines
Jason
Paging again...
Jason
Quoted 79 lines · Jason A. Donenfeld
Hi Eric,
Quoted 76 lines · Jason A. Donenfeld
Hey Eric,
Quoted 67 lines · Jason A. Donenfeld
I was recently surprised to learn that
msleep_interruptible(),wait_for_completion_interruptible_timeout(), and related functions simply hung when I calledkthread_stop()on kthreads using them. The solution to fixing the case withmsleep_interruptible()was more simply to move toschedule_timeout_interruptible(). Why?The reason is that
msleep_interruptible(), and many functions just like it, has a loop like this:while (timeout && !signal_pending(current)) timeout = schedule_timeout_interruptible(timeout);The call to
kthread_stop()woke up the thread, soschedule_timeout_interruptible()returned early, but becausesignal_pending()returned true, it went back into another timeout, which was never woken up.This wait loop pattern is common to various pieces of code, and I suspect that the subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.
So this commit causes
signal_pending()to return true whenkthread_stop()is called, by settingTIF_NOTIFY_SIGNAL.The same also applies to the similar
kthread_park()functionality.Cc Ingo Molnar <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Eric W. Biederman <[email protected]>Cc Toke Høiland-Jørgensen <[email protected]>Cc Kalle Valo <[email protected]>Cc Johannes Berg <[email protected]>Signed-off-by Jason A. Donenfeld <[email protected]>--- kernel/kthread.c | 4 ++++ 1 file changed, 4 insertions(+)kernel/kthread.c +4
-1@@ -661,12 +661,14 @@ int kthread_park(struct task_struct *k)set_bit (KTHREAD_SHOULD_PARK , &kthread->flags);if (k != current) { +test_and_set_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );wake_up_process (k);/* * Wait for __kthread_parkme() to complete(), this means we * _will_ have TASK_PARKED and are about to call schedule(). */ wait_for_completion (&kthread->parked); +clear_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );/* * Now wait for that schedule() to complete and the task to * get scheduled out. @@ -704,8 +706,10 @@ int kthread_stop(struct task_struct *k) kthread =to_kthread (k);set_bit (KTHREAD_SHOULD_STOP , &kthread->flags);kthread_unpark (k); +test_and_set_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );wake_up_process (k);wait_for_completion (&kthread->exited); +clear_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL ); ret = kthread->result;put_task_struct (k); --Quoted 2 lines
2.35.1
Is this more to the tune of what you had in mind in your message [1]?
Quoted 5 lines
Quoted 3 lines
Jason
Paging again...
Overall it looks reasonable.
For kthread_stop clearing TIF_NOTIFY_SIGNAL seems pointless as the process has exited.
I wonder if we should clear TIF_NOTIFY_SIGNAL in kthread_parkme.
Ordinarily TIF_NOTIFY_SIGNAL gets cleared when the target process calls get_signal. Which doesn't happen for kernel threads. So I am not certain what the best pattern is, but my sense is that keeping things as close to how TIF_NOTIFY_SIGNAL is processed in the rest of the kernel seems the least error prone pattern.
So I am thinking the code should do something like:
kernel/kthread.c +1 -0
@@ -266,6 +266,7 @@ static void __kthread_parkme(struct kthread *self)* changin from TASK_PARKED and us failing the * wait_task_inactive() in kthread_park(). */ +clear_notify_signal ();set_special_state (TASK_PARKED );if (!test_bit (KTHREAD_SHOULD_PARK , &self->flags))break ;
To ensure that TIF_NOTIFY_SIGNAL is not set.
I am trying to think about how things interact if two threads call kthread_park. If the target thread is not responsible for clearing TIF_NOTIFY_SIGNAL it feels like two threads could get into a race. A race where one thread sees the target thread has parked itself right after another thread calls kthread_park and sets TIF_NOTIFY_SIGNAL.
Given the nature of kthread_park that scenario is probably completely ridiculous, but it is all I can think of at the moment to demonstrate my concerns.
In practice kthread_park is pretty specialized. Do you have any evidence that we need to break out of interruptible loops to make it more reliable. Perhaps it is best just to handle kthread_stop, and leave kthread_park for when it actually matters. Which would ensure there are examples that people care about to help sort through exactly how the code should behave in the kthread_park case.
Which I guess my long way of saying I think you can just change kthread_stop to say:
kernel/kthread.c +1 -0
@@ -704,6 +704,7 @@ int kthread_stop(struct task_struct *k) kthread =to_kthread (k);set_bit (KTHREAD_SHOULD_STOP , &kthread->flags);kthread_unpark (k); +set_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );wake_up_process (k);wait_for_completion (&kthread->exited); ret = kthread->result;
Your patch is correct that most of set_notify_signal is redundant with wake_up_process(k). Further if you aren't going to use the return value like set_notify_signal does there is no need to test if the flag gets set.
Eric
Hi Eric,
Thanks for the review.
Quoted 2 lines · Eric W. Biederman
For
kthread_stopclearingTIF_NOTIFY_SIGNALseems pointless as the process has exited.
Alright, I'll get rid of the clear there.
Quoted 15 lines
I wonder if we should clear
TIF_NOTIFY_SIGNALinkthread_parkme.Ordinarily
TIF_NOTIFY_SIGNALgets cleared when the target process callsget_signal. Which doesn't happen for kernel threads. So I am not certain what the best pattern is, but my sense is that keeping things as close to howTIF_NOTIFY_SIGNALis processed in the rest of the kernel seems the least error prone pattern.So I am thinking the code should do something like:
kernel/kthread.c +0
-0
kernel/kthread.c static void __kthread_parkme(struct kthread *self) · line 272
Quoted 7 lines
kernel/kthread.c +1
-0@@ -266,6 +266,7 @@ static void __kthread_parkme(struct kthread *self)* changin from TASK_PARKED and us failing the * wait_task_inactive() in kthread_park(). */ +clear_notify_signal ();set_special_state (TASK_PARKED );if (!test_bit (KTHREAD_SHOULD_PARK , &self->flags))break ;
That seems at least semantically correct for how parkers usually work.
Quoted 9 lines
I am trying to think about how things interact if two threads call
kthread_park. If the target thread is not responsible for clearingTIF_NOTIFY_SIGNALit feels like two threads could get into a race. A race where one thread sees the target thread has parked itself right after another thread callskthread_parkand setsTIF_NOTIFY_SIGNAL.Given the nature of
kthread_parkthat scenario is probably completely ridiculous, but it is all I can think of at the moment to demonstrate my concerns.
Right, it's a bit of a stretch to consider two threads racing on kthread_park(), and were it to happen, maybe the right response would be, "take a mutex so you don't race for that weird case." But it still seems semantically correct to clear the flag as your diff does.
Quoted 6 lines
In practice
kthread_parkis pretty specialized. Do you have any evidence that we need to break out of interruptible loops to make it more reliable. Perhaps it is best just to handlekthread_stop, and leavekthread_parkfor when it actually matters. Which would ensure there are examples that people care about to help sort through exactly how the code should behave in thekthread_parkcase.
Oh, okay. I guess I can do that and we'll address the park case sometime later if it ever comes up? But actually, upon very very cursory look, it might actually be an issue now... The uses of it currently break down into:
Core kernel things:
- kernel/stop_machine.c
- kernel/smpboot.c
- kernel/cpu.c
Drivers:
- drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
- drivers/gpu/drm/msm/adreno/adreno_device.c
- drivers/gpu/drm/scheduler/sched_main.c
- drivers/net/wireless/mediatek/mt76/util.h
- drivers/firmware/psci/psci_checker.c
- drivers/md/raid5-cache.c
- fs/btrfs/disk-io.c
btrfs makes calls to _interruptible() functions (though I'm not sure how the context stack traces work out). The drm scheduler appears to use wait_event_interruptible() in the first couple lines of the thread's loop, though it's wait condition does check for kthread_should_stop().
So I don't know...
Quoted 9 lines
Which I guess my long way of saying I think you can just change
kthread_stopto say:kernel/kthread.c +0
-0
kernel/kthread.c int kthread_stop(struct task_struct *k) · line 710
Quoted 7 lines
kernel/kthread.c +1
-0@@ -704,6 +704,7 @@ int kthread_stop(struct task_struct *k) kthread =to_kthread (k);set_bit (KTHREAD_SHOULD_STOP , &kthread->flags);kthread_unpark (k); +set_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );wake_up_process (k);wait_for_completion (&kthread->exited); ret = kthread->result;
Okay. I'll constrain it to just kthread_stop(). But please file away in the back of your mind the potential for kthread_park() to be problematic down the line, in case we have to fix that later.
Jason
PATCH v4 1/1 signal: break out of wait loops on kthread_stop()
I was recently surprised to learn that msleep_interruptible(), wait_for_completion_interruptible_timeout(), and related functions simply hung when I called kthread_stop() on kthreads using them. The solution to fixing the case with msleep_interruptible() was more simply to move to schedule_timeout_interruptible(). Why?
The reason is that msleep_interruptible(), and many functions just like it, has a loop like this:
while (timeout && !signal_pending(current))
timeout = schedule_timeout_interruptible(timeout);The call to kthread_stop() woke up the thread, so schedule_timeout_ interruptible() returned early, but because signal_pending() returned true, it went back into another timeout, which was never woken up.
This wait loop pattern is common to various pieces of code, and I suspect that the subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.
So this commit causes signal_pending() to return true when kthread_stop() is called, by setting TIF_NOTIFY_SIGNAL.
The same also probably applies to the similar kthread_park() functionality, but that can be addressed later, as its semantics are slightly different.
---
Changes v3->v4:
- Don't address park() for now.
- Don't bother clearing the flag, since the task is about to be freed anyway.
kernel/kthread.c | 1 + 1 file changed, 1 insertion(+)
kernel/kthread.c +1 -0
@@ -704,6 +704,7 @@ int kthread_stop(struct task_struct *k) kthread =to_kthread (k);set_bit (KTHREAD_SHOULD_STOP , &kthread->flags);kthread_unpark (k); +test_and_set_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );wake_up_process (k);wait_for_completion (&kthread->exited); ret = kthread->result;
Signature · 1 line
2.35.1
Quoted 43 lines · Jason A. Donenfeld
I was recently surprised to learn that
msleep_interruptible(),wait_for_completion_interruptible_timeout(), and related functions simply hung when I calledkthread_stop()on kthreads using them. The solution to fixing the case withmsleep_interruptible()was more simply to move toschedule_timeout_interruptible(). Why?The reason is that
msleep_interruptible(), and many functions just like it, has a loop like this:while (timeout && !signal_pending(current)) timeout = schedule_timeout_interruptible(timeout);The call to
kthread_stop()woke up the thread, soschedule_timeout_interruptible()returned early, but becausesignal_pending()returned true, it went back into another timeout, which was never woken up.This wait loop pattern is common to various pieces of code, and I suspect that the subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.
So this commit causes
signal_pending()to return true whenkthread_stop()is called, by settingTIF_NOTIFY_SIGNAL.The same also probably applies to the similar
kthread_park()functionality, but that can be addressed later, as its semantics are slightly different.Cc Eric W. Biederman <[email protected]>Signed-off-by Jason A. Donenfeld <[email protected]>---
Changesv3->v4:
- Don't addresspark()for now.
- Don't bother clearing the flag, since the task is about to be freed anyway.kernel/kthread.c | 1 + 1 file changed, 1 insertion(+)kernel/kthread.c +0
-0
kernel/kthread.c int kthread_stop(struct task_struct *k) · line 710
Quoted 7 lines
kernel/kthread.c +1
-0@@ -704,6 +704,7 @@ int kthread_stop(struct task_struct *k) kthread =to_kthread (k);set_bit (KTHREAD_SHOULD_STOP , &kthread->flags);kthread_unpark (k); +test_and_set_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );wake_up_process (k);wait_for_completion (&kthread->exited); ret = kthread->result;
Minor it. Unless I have missed something that should just be set_tsk_thread_flag. You aren't using the return value so I don't think there is any point in testing the previous state of TIF_NOTIFY_SIGNAL.
Eric
PATCH v5 1/1 signal: break out of wait loops on kthread_stop()
I was recently surprised to learn that msleep_interruptible(), wait_for_completion_interruptible_timeout(), and related functions simply hung when I called kthread_stop() on kthreads using them. The solution to fixing the case with msleep_interruptible() was more simply to move to schedule_timeout_interruptible(). Why?
The reason is that msleep_interruptible(), and many functions just like it, has a loop like this:
while (timeout && !signal_pending(current))
timeout = schedule_timeout_interruptible(timeout);The call to kthread_stop() woke up the thread, so schedule_timeout_ interruptible() returned early, but because signal_pending() returned true, it went back into another timeout, which was never woken up.
This wait loop pattern is common to various pieces of code, and I suspect that the subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.
So this commit causes signal_pending() to return true when kthread_stop() is called, by setting TIF_NOTIFY_SIGNAL.
The same also probably applies to the similar kthread_park() functionality, but that can be addressed later, as its semantics are slightly different.
---
Changes v4->v5:
- Use set_tsk_thread_flag instead of test_and_set_tsk_thread_flag. Must have been a copy and paste mistarget.
Changes v3->v4:
- Don't address park() for now.
- Don't bother clearing the flag, since the task is about to be freed anyway.
kernel/kthread.c | 1 + 1 file changed, 1 insertion(+)
kernel/kthread.c +1 -0
@@ -704,6 +704,7 @@ int kthread_stop(struct task_struct *k) kthread =to_kthread (k);set_bit (KTHREAD_SHOULD_STOP , &kthread->flags);kthread_unpark (k); +set_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );wake_up_process (k);wait_for_completion (&kthread->exited); ret = kthread->result;
Signature · 1 line
2.35.1
Quoted 20 lines · Jason A. Donenfeld
I was recently surprised to learn that
msleep_interruptible(),wait_for_completion_interruptible_timeout(), and related functions simply hung when I calledkthread_stop()on kthreads using them. The solution to fixing the case withmsleep_interruptible()was more simply to move toschedule_timeout_interruptible(). Why?The reason is that
msleep_interruptible(), and many functions just like it, has a loop like this:while (timeout && !signal_pending(current)) timeout = schedule_timeout_interruptible(timeout);The call to
kthread_stop()woke up the thread, soschedule_timeout_interruptible()returned early, but becausesignal_pending()returned true, it went back into another timeout, which was never woken up.This wait loop pattern is common to various pieces of code, and I suspect that the subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.
Quoted 6 lines
So this commit causes
signal_pending()to return true whenkthread_stop()is called, by settingTIF_NOTIFY_SIGNAL.The same also probably applies to the similar
kthread_park()functionality, but that can be addressed later, as its semantics are slightly different.
Do I need to pick this up and put it on a topic branch? Or does this patch have another patch to get merged?
Eric
Quoted 26 lines
Cc Eric W. Biederman <[email protected]>Signed-off-by Jason A. Donenfeld <[email protected]>---
Changesv4->v5:
- Useset_tsk_thread_flaginstead oftest_and_set_tsk_thread_flag. Must have been a copy and paste mistarget.
Changesv3->v4:
- Don't addresspark()for now.
- Don't bother clearing the flag, since the task is about to be freed anyway.kernel/kthread.c | 1 + 1 file changed, 1 insertion(+)kernel/kthread.c +1
-0@@ -704,6 +704,7 @@ int kthread_stop(struct task_struct *k) kthread =to_kthread (k);set_bit (KTHREAD_SHOULD_STOP , &kthread->flags);kthread_unpark (k); +set_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );wake_up_process (k);wait_for_completion (&kthread->exited); ret = kthread->result;
Hi Eric,
Quoted 25 lines · Eric W. Biederman
Quoted 23 lines · Jason A. Donenfeld
I was recently surprised to learn that
msleep_interruptible(),wait_for_completion_interruptible_timeout(), and related functions simply hung when I calledkthread_stop()on kthreads using them. The solution to fixing the case withmsleep_interruptible()was more simply to move toschedule_timeout_interruptible(). Why?The reason is that
msleep_interruptible(), and many functions just like it, has a loop like this:while (timeout && !signal_pending(current)) timeout = schedule_timeout_interruptible(timeout);The call to
kthread_stop()woke up the thread, soschedule_timeout_interruptible()returned early, but becausesignal_pending()returned true, it went back into another timeout, which was never woken up.This wait loop pattern is common to various pieces of code, and I suspect that the subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.
So this commit causes
signal_pending()to return true whenkthread_stop()is called, by settingTIF_NOTIFY_SIGNAL.
Quoted 8 lines
Quoted 3 lines
The same also probably applies to the similar
kthread_park()functionality, but that can be addressed later, as its semantics are slightly different.Do I need to pick this up and put it on a topic branch? Or does this patch have another patch to get merged?
I think it'd make sense to put this in a topic branch.
I discovered this in the process of developing [1] (which could use an Ack for the wake_up_state EXPORT_SYMBOL, by the way). That's marked for stable@ because the breakage there is kind of bad -- people can't put their laptops to sleep right now. After both this patch and that one land, I may then revisit the ath9k one and make changes for non-stable@. That is, of course, if [1] even lands; right now I fear it might be relegated to scheduler bikeshed purgatory and I don't have the time right now to deal with that.
Longer term, this patch here will let me rework [1] to get rid of the set_notify_signal() trick and use a proper condition variable wait with wait_for_condition_interruptible_timeout, since this patch makes that function work right for both contexts in which hwrng devices are called (kthread and process). But that will mean reworking the hwrng API, which means writing patches for every single hwrng driver, so that's work for another time.
In the mean time, [1] is a good way forward (provided it gets acked). And then this patch puts things in a good position to improve down the line. I did a bunch of testing of this patch when trying out different candidates for [1] before settling on [1] as a good-enough intermediate solution.
So, anyway, feel free to put this in a topic branch.
Jason
Quoted 16 lines · Jason A. Donenfeld
Hi Eric,
Thanks for the review.
Quoted 12 lines
Which I guess my long way of saying I think you can just change
kthread_stopto say:kernel/kthread.c +1
-0@@ -704,6 +704,7 @@ int kthread_stop(struct task_struct *k) kthread =to_kthread (k);set_bit (KTHREAD_SHOULD_STOP , &kthread->flags);kthread_unpark (k); +set_tsk_thread_flag (k,TIF_NOTIFY_SIGNAL );
Quoted 8 lines
kernel/kthread.c int kthread_stop(struct task_struct *k) · line 710
Quoted 4 lines
kernel/kthread.c +0
-0wake_up_process (k);wait_for_completion (&kthread->exited); ret = kthread->result;Okay. I'll constrain it to just
kthread_stop(). But please file away in the back of your mind the potential forkthread_park()to be problematic down the line, in case we have to fix that later.
Definitely. Right now I am certain you are motivated to test and make certain the kthread_stop case will work. I just have the feeling that we don't care enough about kthread_park, and so attempting to solve it now is as likely to cause problems as solve them.
Eric