Skip to contentSkip to conversation

signal: break out of wait loops on kthread_stop()

PATCH Superseded Jason A. Donenfeld 3 participants 17 messages Jun 27, 2022 – Jul 12, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 signal: break out of wait loops on kthread_stop()

I was recently surprised to learn that msleep_interruptible(), wait_for_completion_interruptible_timeout(), and related functions simply hung when I called kthread_stop() on kthreads using them. The solution to fixing the case with msleep_interruptible() was more simply to move to schedule_timeout_interruptible(). Why?

The reason is that msleep_interruptible(), and many functions just like it, has a loop like this:

        while (timeout && !signal_pending(current))
                timeout = schedule_timeout_interruptible(timeout);

The call to kthread_stop() woke up the thread, so schedule_timeout_ interruptible() returned early, but because signal_pending() returned true, it went back into another timeout, which was never woken up.

This wait loop pattern is common to various pieces of code, and I suspect that subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.

So this commit causes signal_pending() to return true when kthread_stop() is called. This is already what's done for TIF_NOTIFY_SIGNAL, for these same purposes of breaking out of wait loops, so a similar KTHREAD_SHOULD_STOP check isn't too much different.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Eric W. Biederman <[email protected]>
Cc Toke Høiland-Jørgensen <[email protected]>
Cc Kalle Valo <[email protected]>
Cc Johannes Berg <[email protected]>
---
 include/linux/kthread.h      | 1 +
 include/linux/sched/signal.h | 9 +++++++++
 kernel/kthread.c             | 6 ++++++
 3 files changed, 16 insertions(+)
include/linux/kthread.h +1 -0
index 30e5bec81d2b..7061dde23237 100644
@@ -87,6 +87,7 @@ void kthread_bind(struct task_struct *k, unsigned int cpu);
 void kthread_bind_mask(struct task_struct *k, const struct cpumask *mask);
 int kthread_stop(struct task_struct *k);
 bool kthread_should_stop(void);
+bool __kthread_should_stop(struct task_struct *k);
 bool kthread_should_park(void);
 bool __kthread_should_park(struct task_struct *k);
 bool kthread_freezable_should_stop(bool *was_frozen);
include/linux/sched/signal.h +9 -0
index cafbe03eed01..08700c65b806 100644
@@ -11,6 +11,7 @@
 #include <linux/refcount.h>
 #include <linux/posix-timers.h>
 #include <linux/mm_types.h>
+#include <linux/kthread.h>
 #include <asm/ptrace.h>
 
 /*
@@ -397,6 +398,14 @@ static inline int signal_pending(struct task_struct *p)
 	 */
 	if (unlikely(test_tsk_thread_flag(p, TIF_NOTIFY_SIGNAL)))
 		return 1;
+
+	/*
+	 * Likewise, KTHREAD_SHOULD_STOP isn't really a signal, but it also
+	 * requires the same behavior, lest wait loops go forever.
+	 */
+	if (unlikely(__kthread_should_stop(p)))
+		return 1;
+
 	return task_sigpending(p);
 }
 
kernel/kthread.c +6 -0
index 3c677918d8f2..7e0743330cd4 100644
@@ -145,6 +145,12 @@ void free_kthread_struct(struct task_struct *k)
 	kfree(kthread);
 }
 
+bool __kthread_should_stop(struct task_struct *k)
+{
+	return (k->flags & PF_KTHREAD) &&
+	       test_bit(KTHREAD_SHOULD_STOP, &to_kthread(k)->flags);
+}
+
 /**
  * kthread_should_stop - should this kthread return now?
  *
Signature · 1 line
2.35.1
#1 lore raw

kernel/kthread.c void free_kthread_struct(struct task_struct *k) · line 152

Quoted 5 lines · Jason A. Donenfeld
kernel/kthread.c +5 -0
+bool __kthread_should_stop(struct task_struct *k)
+{
+	return (k->flags & PF_KTHREAD) &&
+	       test_bit(KTHREAD_SHOULD_STOP, &to_kthread(k)->flags);
+}

This used to be a racy pattern; not sure it still is since Eric poked at this last, but please use __to_kthread() instead.

#2 lore raw
Quoted 10 lines · Peter Zijlstra

kernel/kthread.c void free_kthread_struct(struct task_struct *k) · line 152

Quoted 5 lines · Jason A. Donenfeld
kernel/kthread.c +5 -0
+bool __kthread_should_stop(struct task_struct *k)
+{
+	return (k->flags & PF_KTHREAD) &&
+	       test_bit(KTHREAD_SHOULD_STOP, &to_kthread(k)->flags);
+}

This used to be a racy pattern; not sure it still is since Eric poked at this last, but please use __to_kthread() instead.

Ah, indeed. Will send a v2.

Jason

#3 lore raw

PATCH v2 1/1 signal: break out of wait loops on kthread_stop()

I was recently surprised to learn that msleep_interruptible(), wait_for_completion_interruptible_timeout(), and related functions simply hung when I called kthread_stop() on kthreads using them. The solution to fixing the case with msleep_interruptible() was more simply to move to schedule_timeout_interruptible(). Why?

The reason is that msleep_interruptible(), and many functions just like it, has a loop like this:

        while (timeout && !signal_pending(current))
                timeout = schedule_timeout_interruptible(timeout);

The call to kthread_stop() woke up the thread, so schedule_timeout_ interruptible() returned early, but because signal_pending() returned true, it went back into another timeout, which was never woken up.

This wait loop pattern is common to various pieces of code, and I suspect that subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.

So this commit causes signal_pending() to return true when kthread_stop() is called. This is already what's done for TIF_NOTIFY_SIGNAL, for these same purposes of breaking out of wait loops, so a similar KTHREAD_SHOULD_STOP check isn't too much different.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Eric W. Biederman <[email protected]>
Cc Toke Høiland-Jørgensen <[email protected]>
Cc Kalle Valo <[email protected]>
Cc Johannes Berg <[email protected]>
---
 include/linux/kthread.h      | 1 +
 include/linux/sched/signal.h | 9 +++++++++
 kernel/kthread.c             | 8 ++++++++
 3 files changed, 18 insertions(+)
include/linux/kthread.h +1 -0
index 30e5bec81d2b..7061dde23237 100644
@@ -87,6 +87,7 @@ void kthread_bind(struct task_struct *k, unsigned int cpu);
 void kthread_bind_mask(struct task_struct *k, const struct cpumask *mask);
 int kthread_stop(struct task_struct *k);
 bool kthread_should_stop(void);
+bool __kthread_should_stop(struct task_struct *k);
 bool kthread_should_park(void);
 bool __kthread_should_park(struct task_struct *k);
 bool kthread_freezable_should_stop(bool *was_frozen);
include/linux/sched/signal.h +9 -0
index cafbe03eed01..08700c65b806 100644
@@ -11,6 +11,7 @@
 #include <linux/refcount.h>
 #include <linux/posix-timers.h>
 #include <linux/mm_types.h>
+#include <linux/kthread.h>
 #include <asm/ptrace.h>
 
 /*
@@ -397,6 +398,14 @@ static inline int signal_pending(struct task_struct *p)
 	 */
 	if (unlikely(test_tsk_thread_flag(p, TIF_NOTIFY_SIGNAL)))
 		return 1;
+
+	/*
+	 * Likewise, KTHREAD_SHOULD_STOP isn't really a signal, but it also
+	 * requires the same behavior, lest wait loops go forever.
+	 */
+	if (unlikely(__kthread_should_stop(p)))
+		return 1;
+
 	return task_sigpending(p);
 }
 
kernel/kthread.c +8 -0
index 3c677918d8f2..80f6ba323060 100644
@@ -145,6 +145,14 @@ void free_kthread_struct(struct task_struct *k)
 	kfree(kthread);
 }
 
+bool __kthread_should_stop(struct task_struct *k)
+{
+	struct kthread *kthread = __to_kthread(k);
+
+	return kthread && test_bit(KTHREAD_SHOULD_STOP, &kthread->flags);
+}
+EXPORT_SYMBOL_GPL(__kthread_should_stop);
+
 /**
  * kthread_should_stop - should this kthread return now?
  *
Signature · 1 line
2.35.1
#4 lore raw
Quoted 20 lines · Jason A. Donenfeld

I was recently surprised to learn that msleep_interruptible(), wait_for_completion_interruptible_timeout(), and related functions simply hung when I called kthread_stop() on kthreads using them. The solution to fixing the case with msleep_interruptible() was more simply to move to schedule_timeout_interruptible(). Why?

The reason is that msleep_interruptible(), and many functions just like it, has a loop like this:

        while (timeout && !signal_pending(current))
                timeout = schedule_timeout_interruptible(timeout);

The call to kthread_stop() woke up the thread, so schedule_timeout_ interruptible() returned early, but because signal_pending() returned true, it went back into another timeout, which was never woken up.

This wait loop pattern is common to various pieces of code, and I suspect that subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.

Quoted 5 lines

So this commit causes signal_pending() to return true when kthread_stop() is called. This is already what's done for TIF_NOTIFY_SIGNAL, for these same purposes of breaking out of wait loops, so a similar KTHREAD_SHOULD_STOP check isn't too much different.

Semantically this makes a lot of sense.

Bloating up signal_pending which is mainly called in non-kthread contexts is undesirable.

Instead could you modify kthread_stop to call set_notify_signal().

That is exactly what set_notify_signal is there for. When you don't actually have a signal but you want to break out of an interruptible loop. My last round of work in the area decoupled set_notify_signal from any other semantics.

It would be nice to get everything down so that we only need to test TIF_NOTIFY_SIGNAL in signal_pending. Unfortunately to do that I need to do something with task_sigpending, and it hasn't been important enough to weed through all of those details yet.

Eric

Quoted 71 lines
Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Eric W. Biederman <[email protected]>
Cc Toke Høiland-Jørgensen <[email protected]>
Cc Kalle Valo <[email protected]>
Cc Johannes Berg <[email protected]>
---
 include/linux/kthread.h      | 1 +
 include/linux/sched/signal.h | 9 +++++++++
 kernel/kthread.c             | 8 ++++++++
 3 files changed, 18 insertions(+)
include/linux/kthread.h +1 -0
index 30e5bec81d2b..7061dde23237 100644
@@ -87,6 +87,7 @@ void kthread_bind(struct task_struct *k, unsigned int cpu);
 void kthread_bind_mask(struct task_struct *k, const struct cpumask *mask);
 int kthread_stop(struct task_struct *k);
 bool kthread_should_stop(void);
+bool __kthread_should_stop(struct task_struct *k);
 bool kthread_should_park(void);
 bool __kthread_should_park(struct task_struct *k);
 bool kthread_freezable_should_stop(bool *was_frozen);
include/linux/sched/signal.h +9 -0
index cafbe03eed01..08700c65b806 100644
@@ -11,6 +11,7 @@
 #include <linux/refcount.h>
 #include <linux/posix-timers.h>
 #include <linux/mm_types.h>
+#include <linux/kthread.h>
 #include <asm/ptrace.h>
 
 /*
@@ -397,6 +398,14 @@ static inline int signal_pending(struct task_struct *p)
 	 */
 	if (unlikely(test_tsk_thread_flag(p, TIF_NOTIFY_SIGNAL)))
 		return 1;
+
+	/*
+	 * Likewise, KTHREAD_SHOULD_STOP isn't really a signal, but it also
+	 * requires the same behavior, lest wait loops go forever.
+	 */
+	if (unlikely(__kthread_should_stop(p)))
+		return 1;
+
 	return task_sigpending(p);
 }
 
kernel/kthread.c +8 -0
index 3c677918d8f2..80f6ba323060 100644
@@ -145,6 +145,14 @@ void free_kthread_struct(struct task_struct *k)
 	kfree(kthread);
 }
 
+bool __kthread_should_stop(struct task_struct *k)
+{
+	struct kthread *kthread = __to_kthread(k);
+
+	return kthread && test_bit(KTHREAD_SHOULD_STOP, &kthread->flags);
+}
+EXPORT_SYMBOL_GPL(__kthread_should_stop);
+
 /**
  * kthread_should_stop - should this kthread return now?
  *
#5 lore raw

Hi Eric,

Quoted 4 lines · Eric W. Biederman

Semantically this makes a lot of sense.

Bloating up signal_pending which is mainly called in non-kthread contexts is undesirable.

I guess I understand that concern, but does it really matter here? This is called by code that waits anyway, so it's not like performance matters at all, right?

Quoted 6 lines

Instead could you modify kthread_stop to call set_notify_signal().

That is exactly what set_notify_signal is there for. When you don't actually have a signal but you want to break out of an interruptible loop. My last round of work in the area decoupled set_notify_signal from any other semantics.

This sounds like the best option here, if in fact it does work. I'll send in a patch for that and we can see how it interacts with the other work you're doing.

Jason

#6 lore raw

PATCH v3 1/1 signal: break out of wait loops on kthread_stop()

I was recently surprised to learn that msleep_interruptible(), wait_for_completion_interruptible_timeout(), and related functions simply hung when I called kthread_stop() on kthreads using them. The solution to fixing the case with msleep_interruptible() was more simply to move to schedule_timeout_interruptible(). Why?

The reason is that msleep_interruptible(), and many functions just like it, has a loop like this:

        while (timeout && !signal_pending(current))
                timeout = schedule_timeout_interruptible(timeout);

The call to kthread_stop() woke up the thread, so schedule_timeout_ interruptible() returned early, but because signal_pending() returned true, it went back into another timeout, which was never woken up.

This wait loop pattern is common to various pieces of code, and I suspect that the subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.

So this commit causes signal_pending() to return true when kthread_stop() is called, by setting TIF_NOTIFY_SIGNAL.

The same also applies to the similar kthread_park() functionality.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Eric W. Biederman <[email protected]>
Cc Toke Høiland-Jørgensen <[email protected]>
Cc Kalle Valo <[email protected]>
Cc Johannes Berg <[email protected]>
---
 kernel/kthread.c | 4 ++++
 1 file changed, 4 insertions(+)
kernel/kthread.c +4 -0
index 3c677918d8f2..63d5a1f4cb93 100644
@@ -661,12 +661,14 @@ int kthread_park(struct task_struct *k)
 
 	set_bit(KTHREAD_SHOULD_PARK, &kthread->flags);
 	if (k != current) {
+		test_and_set_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
 		wake_up_process(k);
 		/*
 		 * Wait for __kthread_parkme() to complete(), this means we
 		 * _will_ have TASK_PARKED and are about to call schedule().
 		 */
 		wait_for_completion(&kthread->parked);
+		clear_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
 		/*
 		 * Now wait for that schedule() to complete and the task to
 		 * get scheduled out.
@@ -704,8 +706,10 @@ int kthread_stop(struct task_struct *k)
 	kthread = to_kthread(k);
 	set_bit(KTHREAD_SHOULD_STOP, &kthread->flags);
 	kthread_unpark(k);
+	test_and_set_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
 	wake_up_process(k);
 	wait_for_completion(&kthread->exited);
+	clear_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
 	ret = kthread->result;
 	put_task_struct(k);
 
Signature · 1 line
2.35.1
#7 lore raw

Hey Eric,

Quoted 69 lines · Jason A. Donenfeld

I was recently surprised to learn that msleep_interruptible(), wait_for_completion_interruptible_timeout(), and related functions simply hung when I called kthread_stop() on kthreads using them. The solution to fixing the case with msleep_interruptible() was more simply to move to schedule_timeout_interruptible(). Why?

The reason is that msleep_interruptible(), and many functions just like it, has a loop like this:

        while (timeout && !signal_pending(current))
                timeout = schedule_timeout_interruptible(timeout);

The call to kthread_stop() woke up the thread, so schedule_timeout_ interruptible() returned early, but because signal_pending() returned true, it went back into another timeout, which was never woken up.

This wait loop pattern is common to various pieces of code, and I suspect that the subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.

So this commit causes signal_pending() to return true when kthread_stop() is called, by setting TIF_NOTIFY_SIGNAL.

The same also applies to the similar kthread_park() functionality.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Eric W. Biederman <[email protected]>
Cc Toke Høiland-Jørgensen <[email protected]>
Cc Kalle Valo <[email protected]>
Cc Johannes Berg <[email protected]>
---
 kernel/kthread.c | 4 ++++
 1 file changed, 4 insertions(+)
kernel/kthread.c +4 -0
index 3c677918d8f2..63d5a1f4cb93 100644
@@ -661,12 +661,14 @@ int kthread_park(struct task_struct *k)
 
 	set_bit(KTHREAD_SHOULD_PARK, &kthread->flags);
 	if (k != current) {
+		test_and_set_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
 		wake_up_process(k);
 		/*
 		 * Wait for __kthread_parkme() to complete(), this means we
 		 * _will_ have TASK_PARKED and are about to call schedule().
 		 */
 		wait_for_completion(&kthread->parked);
+		clear_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
 		/*
 		 * Now wait for that schedule() to complete and the task to
 		 * get scheduled out.
@@ -704,8 +706,10 @@ int kthread_stop(struct task_struct *k)
 	kthread = to_kthread(k);
 	set_bit(KTHREAD_SHOULD_STOP, &kthread->flags);
 	kthread_unpark(k);
+	test_and_set_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
 	wake_up_process(k);
 	wait_for_completion(&kthread->exited);
+	clear_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
 	ret = kthread->result;
 	put_task_struct(k);
 
Signature · 1 line
2.35.1

Is this more to the tune of what you had in mind in your message [1]?

Jason

[1] https://lore.kernel.org/lkml/[email protected]/

#8 lore raw

Hi Eric,

Quoted 76 lines · Jason A. Donenfeld

Hey Eric,

Quoted 67 lines · Jason A. Donenfeld

I was recently surprised to learn that msleep_interruptible(), wait_for_completion_interruptible_timeout(), and related functions simply hung when I called kthread_stop() on kthreads using them. The solution to fixing the case with msleep_interruptible() was more simply to move to schedule_timeout_interruptible(). Why?

The reason is that msleep_interruptible(), and many functions just like it, has a loop like this:

        while (timeout && !signal_pending(current))
                timeout = schedule_timeout_interruptible(timeout);

The call to kthread_stop() woke up the thread, so schedule_timeout_ interruptible() returned early, but because signal_pending() returned true, it went back into another timeout, which was never woken up.

This wait loop pattern is common to various pieces of code, and I suspect that the subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.

So this commit causes signal_pending() to return true when kthread_stop() is called, by setting TIF_NOTIFY_SIGNAL.

The same also applies to the similar kthread_park() functionality.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Eric W. Biederman <[email protected]>
Cc Toke Høiland-Jørgensen <[email protected]>
Cc Kalle Valo <[email protected]>
Cc Johannes Berg <[email protected]>
---
 kernel/kthread.c | 4 ++++
 1 file changed, 4 insertions(+)
kernel/kthread.c +4 -1
index 3c677918d8f2..63d5a1f4cb93 100644
@@ -661,12 +661,14 @@ int kthread_park(struct task_struct *k)
 
      set_bit(KTHREAD_SHOULD_PARK, &kthread->flags);
      if (k != current) {
+             test_and_set_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
              wake_up_process(k);
              /*
               * Wait for __kthread_parkme() to complete(), this means we
               * _will_ have TASK_PARKED and are about to call schedule().
               */
              wait_for_completion(&kthread->parked);
+             clear_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
              /*
               * Now wait for that schedule() to complete and the task to
               * get scheduled out.
@@ -704,8 +706,10 @@ int kthread_stop(struct task_struct *k)
      kthread = to_kthread(k);
      set_bit(KTHREAD_SHOULD_STOP, &kthread->flags);
      kthread_unpark(k);
+     test_and_set_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
      wake_up_process(k);
      wait_for_completion(&kthread->exited);
+     clear_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
      ret = kthread->result;
      put_task_struct(k);
 
--
Quoted 2 lines

2.35.1

Is this more to the tune of what you had in mind in your message [1]?

Quoted 3 lines

Jason

[1] https://lore.kernel.org/lkml/[email protected]/

Paging again...

Jason

#9 lore raw
Quoted 79 lines · Jason A. Donenfeld

Hi Eric,

Quoted 76 lines · Jason A. Donenfeld

Hey Eric,

Quoted 67 lines · Jason A. Donenfeld

I was recently surprised to learn that msleep_interruptible(), wait_for_completion_interruptible_timeout(), and related functions simply hung when I called kthread_stop() on kthreads using them. The solution to fixing the case with msleep_interruptible() was more simply to move to schedule_timeout_interruptible(). Why?

The reason is that msleep_interruptible(), and many functions just like it, has a loop like this:

        while (timeout && !signal_pending(current))
                timeout = schedule_timeout_interruptible(timeout);

The call to kthread_stop() woke up the thread, so schedule_timeout_ interruptible() returned early, but because signal_pending() returned true, it went back into another timeout, which was never woken up.

This wait loop pattern is common to various pieces of code, and I suspect that the subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.

So this commit causes signal_pending() to return true when kthread_stop() is called, by setting TIF_NOTIFY_SIGNAL.

The same also applies to the similar kthread_park() functionality.

Cc Ingo Molnar <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Eric W. Biederman <[email protected]>
Cc Toke Høiland-Jørgensen <[email protected]>
Cc Kalle Valo <[email protected]>
Cc Johannes Berg <[email protected]>
---
 kernel/kthread.c | 4 ++++
 1 file changed, 4 insertions(+)
kernel/kthread.c +4 -1
index 3c677918d8f2..63d5a1f4cb93 100644
@@ -661,12 +661,14 @@ int kthread_park(struct task_struct *k)
 
      set_bit(KTHREAD_SHOULD_PARK, &kthread->flags);
      if (k != current) {
+             test_and_set_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
              wake_up_process(k);
              /*
               * Wait for __kthread_parkme() to complete(), this means we
               * _will_ have TASK_PARKED and are about to call schedule().
               */
              wait_for_completion(&kthread->parked);
+             clear_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
              /*
               * Now wait for that schedule() to complete and the task to
               * get scheduled out.
@@ -704,8 +706,10 @@ int kthread_stop(struct task_struct *k)
      kthread = to_kthread(k);
      set_bit(KTHREAD_SHOULD_STOP, &kthread->flags);
      kthread_unpark(k);
+     test_and_set_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
      wake_up_process(k);
      wait_for_completion(&kthread->exited);
+     clear_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
      ret = kthread->result;
      put_task_struct(k);
 
--
Quoted 2 lines

2.35.1

Is this more to the tune of what you had in mind in your message [1]?

Quoted 5 lines
Quoted 3 lines

Jason

[1] https://lore.kernel.org/lkml/[email protected]/

Paging again...

Overall it looks reasonable.

For kthread_stop clearing TIF_NOTIFY_SIGNAL seems pointless as the process has exited.

I wonder if we should clear TIF_NOTIFY_SIGNAL in kthread_parkme.

Ordinarily TIF_NOTIFY_SIGNAL gets cleared when the target process calls get_signal. Which doesn't happen for kernel threads. So I am not certain what the best pattern is, but my sense is that keeping things as close to how TIF_NOTIFY_SIGNAL is processed in the rest of the kernel seems the least error prone pattern.

So I am thinking the code should do something like:

kernel/kthread.c +1 -0
index 544fd4097406..c80e8d44e405 100644
@@ -266,6 +266,7 @@ static void __kthread_parkme(struct kthread *self)
 		 * changin from TASK_PARKED and us failing the
 		 * wait_task_inactive() in kthread_park().
 		 */
+		clear_notify_signal();
 		set_special_state(TASK_PARKED);
 		if (!test_bit(KTHREAD_SHOULD_PARK, &self->flags))
 			break;

To ensure that TIF_NOTIFY_SIGNAL is not set.

I am trying to think about how things interact if two threads call kthread_park. If the target thread is not responsible for clearing TIF_NOTIFY_SIGNAL it feels like two threads could get into a race. A race where one thread sees the target thread has parked itself right after another thread calls kthread_park and sets TIF_NOTIFY_SIGNAL.

Given the nature of kthread_park that scenario is probably completely ridiculous, but it is all I can think of at the moment to demonstrate my concerns.

In practice kthread_park is pretty specialized. Do you have any evidence that we need to break out of interruptible loops to make it more reliable. Perhaps it is best just to handle kthread_stop, and leave kthread_park for when it actually matters. Which would ensure there are examples that people care about to help sort through exactly how the code should behave in the kthread_park case.

Which I guess my long way of saying I think you can just change kthread_stop to say:

kernel/kthread.c +1 -0
index 544fd4097406..52e9b3432496 100644
@@ -704,6 +704,7 @@ int kthread_stop(struct task_struct *k)
 	kthread = to_kthread(k);
 	set_bit(KTHREAD_SHOULD_STOP, &kthread->flags);
 	kthread_unpark(k);
+	set_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
 	wake_up_process(k);
 	wait_for_completion(&kthread->exited);
 	ret = kthread->result;

Your patch is correct that most of set_notify_signal is redundant with wake_up_process(k). Further if you aren't going to use the return value like set_notify_signal does there is no need to test if the flag gets set.

Eric

#10 lore raw

Hi Eric,

Thanks for the review.

Quoted 2 lines · Eric W. Biederman

For kthread_stop clearing TIF_NOTIFY_SIGNAL seems pointless as the process has exited.

Alright, I'll get rid of the clear there.

Quoted 15 lines

I wonder if we should clear TIF_NOTIFY_SIGNAL in kthread_parkme.

Ordinarily TIF_NOTIFY_SIGNAL gets cleared when the target process calls get_signal. Which doesn't happen for kernel threads. So I am not certain what the best pattern is, but my sense is that keeping things as close to how TIF_NOTIFY_SIGNAL is processed in the rest of the kernel seems the least error prone pattern.

So I am thinking the code should do something like:

kernel/kthread.c +0 -0
index 544fd4097406..c80e8d44e405 100644

kernel/kthread.c static void __kthread_parkme(struct kthread *self) · line 272

Quoted 7 lines
kernel/kthread.c +1 -0
@@ -266,6 +266,7 @@ static void __kthread_parkme(struct kthread *self)
 		 * changin from TASK_PARKED and us failing the
 		 * wait_task_inactive() in kthread_park().
 		 */
+		clear_notify_signal();
 		set_special_state(TASK_PARKED);
 		if (!test_bit(KTHREAD_SHOULD_PARK, &self->flags))
 			break;

That seems at least semantically correct for how parkers usually work.

Quoted 9 lines

I am trying to think about how things interact if two threads call kthread_park. If the target thread is not responsible for clearing TIF_NOTIFY_SIGNAL it feels like two threads could get into a race. A race where one thread sees the target thread has parked itself right after another thread calls kthread_park and sets TIF_NOTIFY_SIGNAL.

Given the nature of kthread_park that scenario is probably completely ridiculous, but it is all I can think of at the moment to demonstrate my concerns.

Right, it's a bit of a stretch to consider two threads racing on kthread_park(), and were it to happen, maybe the right response would be, "take a mutex so you don't race for that weird case." But it still seems semantically correct to clear the flag as your diff does.

Quoted 6 lines

In practice kthread_park is pretty specialized. Do you have any evidence that we need to break out of interruptible loops to make it more reliable. Perhaps it is best just to handle kthread_stop, and leave kthread_park for when it actually matters. Which would ensure there are examples that people care about to help sort through exactly how the code should behave in the kthread_park case.

Oh, okay. I guess I can do that and we'll address the park case sometime later if it ever comes up? But actually, upon very very cursory look, it might actually be an issue now... The uses of it currently break down into:

Core kernel things:
- kernel/stop_machine.c
- kernel/smpboot.c
- kernel/cpu.c

Drivers:
- drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
- drivers/gpu/drm/msm/adreno/adreno_device.c
- drivers/gpu/drm/scheduler/sched_main.c
- drivers/net/wireless/mediatek/mt76/util.h
- drivers/firmware/psci/psci_checker.c
- drivers/md/raid5-cache.c
- fs/btrfs/disk-io.c

btrfs makes calls to _interruptible() functions (though I'm not sure how the context stack traces work out). The drm scheduler appears to use wait_event_interruptible() in the first couple lines of the thread's loop, though it's wait condition does check for kthread_should_stop().

So I don't know...

Quoted 9 lines

Which I guess my long way of saying I think you can just change kthread_stop to say:

kernel/kthread.c +0 -0
index 544fd4097406..52e9b3432496 100644

kernel/kthread.c int kthread_stop(struct task_struct *k) · line 710

Quoted 7 lines
kernel/kthread.c +1 -0
@@ -704,6 +704,7 @@ int kthread_stop(struct task_struct *k)
 	kthread = to_kthread(k);
 	set_bit(KTHREAD_SHOULD_STOP, &kthread->flags);
 	kthread_unpark(k);
+	set_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
 	wake_up_process(k);
 	wait_for_completion(&kthread->exited);
 	ret = kthread->result;

Okay. I'll constrain it to just kthread_stop(). But please file away in the back of your mind the potential for kthread_park() to be problematic down the line, in case we have to fix that later.

Jason

#11 lore raw

PATCH v4 1/1 signal: break out of wait loops on kthread_stop()

I was recently surprised to learn that msleep_interruptible(), wait_for_completion_interruptible_timeout(), and related functions simply hung when I called kthread_stop() on kthreads using them. The solution to fixing the case with msleep_interruptible() was more simply to move to schedule_timeout_interruptible(). Why?

The reason is that msleep_interruptible(), and many functions just like it, has a loop like this:

        while (timeout && !signal_pending(current))
                timeout = schedule_timeout_interruptible(timeout);

The call to kthread_stop() woke up the thread, so schedule_timeout_ interruptible() returned early, but because signal_pending() returned true, it went back into another timeout, which was never woken up.

This wait loop pattern is common to various pieces of code, and I suspect that the subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.

So this commit causes signal_pending() to return true when kthread_stop() is called, by setting TIF_NOTIFY_SIGNAL.

The same also probably applies to the similar kthread_park() functionality, but that can be addressed later, as its semantics are slightly different.

---
Changes v3->v4:
- Don't address park() for now.
- Don't bother clearing the flag, since the task is about to be freed anyway.

 kernel/kthread.c | 1 +
 1 file changed, 1 insertion(+)
kernel/kthread.c +1 -0
index 3c677918d8f2..8888987f2b25 100644
@@ -704,6 +704,7 @@ int kthread_stop(struct task_struct *k)
 	kthread = to_kthread(k);
 	set_bit(KTHREAD_SHOULD_STOP, &kthread->flags);
 	kthread_unpark(k);
+	test_and_set_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
 	wake_up_process(k);
 	wait_for_completion(&kthread->exited);
 	ret = kthread->result;
Signature · 1 line
2.35.1
#12 lore raw
Quoted 43 lines · Jason A. Donenfeld

I was recently surprised to learn that msleep_interruptible(), wait_for_completion_interruptible_timeout(), and related functions simply hung when I called kthread_stop() on kthreads using them. The solution to fixing the case with msleep_interruptible() was more simply to move to schedule_timeout_interruptible(). Why?

The reason is that msleep_interruptible(), and many functions just like it, has a loop like this:

        while (timeout && !signal_pending(current))
                timeout = schedule_timeout_interruptible(timeout);

The call to kthread_stop() woke up the thread, so schedule_timeout_ interruptible() returned early, but because signal_pending() returned true, it went back into another timeout, which was never woken up.

This wait loop pattern is common to various pieces of code, and I suspect that the subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.

So this commit causes signal_pending() to return true when kthread_stop() is called, by setting TIF_NOTIFY_SIGNAL.

The same also probably applies to the similar kthread_park() functionality, but that can be addressed later, as its semantics are slightly different.

---
Changes v3->v4:
- Don't address park() for now.
- Don't bother clearing the flag, since the task is about to be freed anyway.

 kernel/kthread.c | 1 +
 1 file changed, 1 insertion(+)
kernel/kthread.c +0 -0
index 3c677918d8f2..8888987f2b25 100644

kernel/kthread.c int kthread_stop(struct task_struct *k) · line 710

Quoted 7 lines
kernel/kthread.c +1 -0
@@ -704,6 +704,7 @@ int kthread_stop(struct task_struct *k)
 	kthread = to_kthread(k);
 	set_bit(KTHREAD_SHOULD_STOP, &kthread->flags);
 	kthread_unpark(k);
+	test_and_set_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
 	wake_up_process(k);
 	wait_for_completion(&kthread->exited);
 	ret = kthread->result;

Minor it. Unless I have missed something that should just be set_tsk_thread_flag. You aren't using the return value so I don't think there is any point in testing the previous state of TIF_NOTIFY_SIGNAL.

Eric

#13 lore raw

PATCH v5 1/1 signal: break out of wait loops on kthread_stop()

I was recently surprised to learn that msleep_interruptible(), wait_for_completion_interruptible_timeout(), and related functions simply hung when I called kthread_stop() on kthreads using them. The solution to fixing the case with msleep_interruptible() was more simply to move to schedule_timeout_interruptible(). Why?

The reason is that msleep_interruptible(), and many functions just like it, has a loop like this:

        while (timeout && !signal_pending(current))
                timeout = schedule_timeout_interruptible(timeout);

The call to kthread_stop() woke up the thread, so schedule_timeout_ interruptible() returned early, but because signal_pending() returned true, it went back into another timeout, which was never woken up.

This wait loop pattern is common to various pieces of code, and I suspect that the subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.

So this commit causes signal_pending() to return true when kthread_stop() is called, by setting TIF_NOTIFY_SIGNAL.

The same also probably applies to the similar kthread_park() functionality, but that can be addressed later, as its semantics are slightly different.

---
Changes v4->v5:
- Use set_tsk_thread_flag instead of test_and_set_tsk_thread_flag. Must have been a copy and paste mistarget.
Changes v3->v4:
- Don't address park() for now.
- Don't bother clearing the flag, since the task is about to be freed anyway.

 kernel/kthread.c | 1 +
 1 file changed, 1 insertion(+)
kernel/kthread.c +1 -0
index 3c677918d8f2..7243a010f433 100644
@@ -704,6 +704,7 @@ int kthread_stop(struct task_struct *k)
 	kthread = to_kthread(k);
 	set_bit(KTHREAD_SHOULD_STOP, &kthread->flags);
 	kthread_unpark(k);
+	set_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
 	wake_up_process(k);
 	wait_for_completion(&kthread->exited);
 	ret = kthread->result;
Signature · 1 line
2.35.1
Acked-by on v5 1/1
#14 lore raw
Quoted 20 lines · Jason A. Donenfeld

I was recently surprised to learn that msleep_interruptible(), wait_for_completion_interruptible_timeout(), and related functions simply hung when I called kthread_stop() on kthreads using them. The solution to fixing the case with msleep_interruptible() was more simply to move to schedule_timeout_interruptible(). Why?

The reason is that msleep_interruptible(), and many functions just like it, has a loop like this:

        while (timeout && !signal_pending(current))
                timeout = schedule_timeout_interruptible(timeout);

The call to kthread_stop() woke up the thread, so schedule_timeout_ interruptible() returned early, but because signal_pending() returned true, it went back into another timeout, which was never woken up.

This wait loop pattern is common to various pieces of code, and I suspect that the subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.

Quoted 6 lines

So this commit causes signal_pending() to return true when kthread_stop() is called, by setting TIF_NOTIFY_SIGNAL.

The same also probably applies to the similar kthread_park() functionality, but that can be addressed later, as its semantics are slightly different.

Do I need to pick this up and put it on a topic branch? Or does this patch have another patch to get merged?

Eric

Quoted 26 lines

---
Changes v4->v5:
- Use set_tsk_thread_flag instead of test_and_set_tsk_thread_flag. Must have been a copy and paste mistarget.
Changes v3->v4:
- Don't address park() for now.
- Don't bother clearing the flag, since the task is about to be freed anyway.

 kernel/kthread.c | 1 +
 1 file changed, 1 insertion(+)
kernel/kthread.c +1 -0
index 3c677918d8f2..7243a010f433 100644
@@ -704,6 +704,7 @@ int kthread_stop(struct task_struct *k)
 	kthread = to_kthread(k);
 	set_bit(KTHREAD_SHOULD_STOP, &kthread->flags);
 	kthread_unpark(k);
+	set_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
 	wake_up_process(k);
 	wait_for_completion(&kthread->exited);
 	ret = kthread->result;
#15 lore raw

Hi Eric,

Quoted 25 lines · Eric W. Biederman
Quoted 23 lines · Jason A. Donenfeld

I was recently surprised to learn that msleep_interruptible(), wait_for_completion_interruptible_timeout(), and related functions simply hung when I called kthread_stop() on kthreads using them. The solution to fixing the case with msleep_interruptible() was more simply to move to schedule_timeout_interruptible(). Why?

The reason is that msleep_interruptible(), and many functions just like it, has a loop like this:

        while (timeout && !signal_pending(current))
                timeout = schedule_timeout_interruptible(timeout);

The call to kthread_stop() woke up the thread, so schedule_timeout_ interruptible() returned early, but because signal_pending() returned true, it went back into another timeout, which was never woken up.

This wait loop pattern is common to various pieces of code, and I suspect that the subtle misuse in a kthread that caused a deadlock in the code I looked at last week is also found elsewhere.

So this commit causes signal_pending() to return true when kthread_stop() is called, by setting TIF_NOTIFY_SIGNAL.

Quoted 8 lines
Quoted 3 lines

The same also probably applies to the similar kthread_park() functionality, but that can be addressed later, as its semantics are slightly different.

Do I need to pick this up and put it on a topic branch? Or does this patch have another patch to get merged?

I think it'd make sense to put this in a topic branch.

I discovered this in the process of developing [1] (which could use an Ack for the wake_up_state EXPORT_SYMBOL, by the way). That's marked for stable@ because the breakage there is kind of bad -- people can't put their laptops to sleep right now. After both this patch and that one land, I may then revisit the ath9k one and make changes for non-stable@. That is, of course, if [1] even lands; right now I fear it might be relegated to scheduler bikeshed purgatory and I don't have the time right now to deal with that.

Longer term, this patch here will let me rework [1] to get rid of the set_notify_signal() trick and use a proper condition variable wait with wait_for_condition_interruptible_timeout, since this patch makes that function work right for both contexts in which hwrng devices are called (kthread and process). But that will mean reworking the hwrng API, which means writing patches for every single hwrng driver, so that's work for another time.

In the mean time, [1] is a good way forward (provided it gets acked). And then this patch puts things in a good position to improve down the line. I did a bunch of testing of this patch when trying out different candidates for [1] before settling on [1] as a good-enough intermediate solution.

So, anyway, feel free to put this in a topic branch.

Jason

[1] https://lore.kernel.org/lkml/[email protected]/

#16 lore raw
In reply to Jason A. Donenfeld
Quoted 16 lines · Jason A. Donenfeld

Hi Eric,

Thanks for the review.

Quoted 12 lines

Which I guess my long way of saying I think you can just change kthread_stop to say:

kernel/kthread.c +1 -0
index 544fd4097406..52e9b3432496 100644
@@ -704,6 +704,7 @@ int kthread_stop(struct task_struct *k)
 	kthread = to_kthread(k);
 	set_bit(KTHREAD_SHOULD_STOP, &kthread->flags);
 	kthread_unpark(k);
+	set_tsk_thread_flag(k, TIF_NOTIFY_SIGNAL);
Quoted 8 lines

kernel/kthread.c int kthread_stop(struct task_struct *k) · line 710

Quoted 4 lines
kernel/kthread.c +0 -0
 	wake_up_process(k);
 	wait_for_completion(&kthread->exited);
 	ret = kthread->result;

Okay. I'll constrain it to just kthread_stop(). But please file away in the back of your mind the potential for kthread_park() to be problematic down the line, in case we have to fix that later.

Definitely. Right now I am certain you are motivated to test and make certain the kthread_stop case will work. I just have the feeling that we don't care enough about kthread_park, and so attempting to solve it now is as likely to cause problems as solve them.

Eric