Skip to contentSkip to conversation

Various fixes for nested sleep stuff

PATCH In review Peter Zijlstra 2 participants 15 messages Oct 31, 2014 – Nov 3, 2014

Subsystems (7)
lore
Share & tools
lore

Peter Zijlstra

author maintainer
#0 lore raw

cover Various fixes for nested sleep stuff

Here's a few patches fixing reported nested sleep warns and issues.

author maintainer
#1 lore raw

PATCH 1/7 sched,wait: Fix a kthread race with wait_woken()

There is a race between kthread_stop() and the new wait_woken() that can result in a lack of progress.

CPU 0                                    | CPU 1
                                         |
rfcomm_run()                             | kthread_stop()
  ...                                    |
  if (!test_bit(KTHREAD_SHOULD_STOP))    |
                                         |   set_bit(KTHREAD_SHOULD_STOP)
                                         |   wake_up_process()
    wait_woken()                         |   wait_for_completion()
      set_current_state(INTERRUPTIBLE)   |
      if (!WQ_FLAG_WOKEN)                |
        schedule_timeout()               |
                                         |

After which both tasks will wait.. forever.

Fix this by having wait_woken() check for kthread_should_stop() but only for kthreads (obviously).

---
 kernel/sched/wait.c |    7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
kernel/sched/wait.c +6 -1
@@ -9,6 +9,7 @@
 #include <linux/mm.h>
 #include <linux/wait.h>
 #include <linux/hash.h>
+#include <linux/kthread.h>
 
 void __init_waitqueue_head(wait_queue_head_t *q, const char *name, struct lock_class_key *key)
 {
@@ -297,6 +298,10 @@ int autoremove_wake_function(wait_queue_
 }
 EXPORT_SYMBOL(autoremove_wake_function);
 
+static inline bool is_kthread_should_stop(void)
+{
+	return (current->flags & PF_KTHREAD) && kthread_should_stop();
+}
 
 /*
  * DEFINE_WAIT_FUNC(wait, woken_wake_func);
@@ -326,7 +331,7 @@ long wait_woken(wait_queue_t *wait, unsi
 	 * woken_wake_function() such that if we observe WQ_FLAG_WOKEN we must
 	 * also observe all state before the wakeup.
 	 */
-	if (!(wait->flags & WQ_FLAG_WOKEN))
+	if (!(wait->flags & WQ_FLAG_WOKEN) && !is_kthread_should_stop())
 		timeout = schedule_timeout(timeout);
 	__set_current_state(TASK_RUNNING);
 
#2 lore raw

Re: [PATCH 1/7] sched,wait: Fix a kthread race with wait_woken()

Quoted 3 lines · Peter Zijlstra

There is a race between kthread_stop() and the new wait_woken() that can result in a lack of progress.

Likewise, the user of wait_woken() can miss any other event which is not associated with wq we are going to sleep on. Please see below.

Quoted 8 lines
kernel/sched/wait.c +4 -0
+static inline bool is_kthread_should_stop(void)
+{
+	return (current->flags & PF_KTHREAD) && kthread_should_stop();
+}
 
 /*
  * DEFINE_WAIT_FUNC(wait, woken_wake_func);

kernel/sched/wait.c long wait_woken(wait_queue_t *wait, unsi · line 335

Quoted 6 lines
kernel/sched/wait.c +1 -1
@@ -326,7 +331,7 @@ long wait_woken(wait_queue_t *wait, unsi
 	 * woken_wake_function() such that if we observe WQ_FLAG_WOKEN we must
 	 * also observe all state before the wakeup.
 	 */
-	if (!(wait->flags & WQ_FLAG_WOKEN))
+	if (!(wait->flags & WQ_FLAG_WOKEN) && !is_kthread_should_stop())
 		timeout = schedule_timeout(timeout);

Well yes, this is more straightforward than other hacks we discussed before. But see above, this doesn't look flexible enough.

And. This assumes that the user must also check kthread_should_stop(), otherwise the waiting loop becomes a busy-wait loop.

So I won't argue, but I still think it would be better to allow the user to do set_task_state() by hand if it needs to check the additional conditions.

Oleg.

author maintainer
#3 lore raw

PATCH 2/7 wait: Reimplement wait_event_freezable()

In reply to Peter Zijlstra

Provide better implementations of wait_event_freezable() APIs.

The problem is with freezer_do_not_count(), it hides the thread from the freezer, even though this thread might not actually freeze/sleep at all.

---
 include/linux/freezer.h |   38 --------------------------------
 include/linux/wait.h    |   56 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+), 38 deletions(-)
include/linux/freezer.h +0 -38
@@ -265,35 +265,6 @@ static inline int freezable_schedule_hrt
 	__retval;							\
 })
 
-#define wait_event_freezable(wq, condition)				\
-({									\
-	int __retval;							\
-	freezer_do_not_count();						\
-	__retval = wait_event_interruptible(wq, (condition));		\
-	freezer_count();						\
-	__retval;							\
-})
-
-#define wait_event_freezable_timeout(wq, condition, timeout)		\
-({									\
-	long __retval = timeout;					\
-	freezer_do_not_count();						\
-	__retval = wait_event_interruptible_timeout(wq,	(condition),	\
-				__retval);				\
-	freezer_count();						\
-	__retval;							\
-})
-
-#define wait_event_freezable_exclusive(wq, condition)			\
-({									\
-	int __retval;							\
-	freezer_do_not_count();						\
-	__retval = wait_event_interruptible_exclusive(wq, condition);	\
-	freezer_count();						\
-	__retval;							\
-})
-
-
 #else /* !CONFIG_FREEZER */
 static inline bool frozen(struct task_struct *p) { return false; }
 static inline bool freezing(struct task_struct *p) { return false; }
@@ -331,15 +302,6 @@ static inline void set_freezable(void) {
 #define freezable_schedule_hrtimeout_range(expires, delta, mode)	\
 	schedule_hrtimeout_range(expires, delta, mode)
 
-#define wait_event_freezable(wq, condition)				\
-		wait_event_interruptible(wq, condition)
-
-#define wait_event_freezable_timeout(wq, condition, timeout)		\
-		wait_event_interruptible_timeout(wq, condition, timeout)
-
-#define wait_event_freezable_exclusive(wq, condition)			\
-		wait_event_interruptible_exclusive(wq, condition)
-
 #define wait_event_freezekillable(wq, condition)		\
 		wait_event_killable(wq, condition)
 
include/linux/wait.h +56 -0

74 lines not on this page · raw message

author maintainer
#4 lore raw

Re: [PATCH 2/7] wait: Reimplement wait_event_freezable()

Quoted 10 lines · Peter Zijlstra

Provide better implementations of wait_event_freezable() APIs.

The problem is with freezer_do_not_count(), it hides the thread from the freezer, even though this thread might not actually freeze/sleep at all.

---

This would maybe compile better.. gotta run, will test later.

include/linux/wait.h +6 -5
@@ -284,12 +284,13 @@ do {									\
  * change the result of the wait condition.
  */
 #define wait_event_freezable(wq, condition)				\
-do {									\
+({									\
+	int __ret = 0;							\
 	might_sleep();							\
-	if (condition)							\
-		break;							\
-	__wait_event_freezable(wq, condition);				\
-} while (0)
+	if (!(condition))						\
+		__ret = __wait_event_freezable(wq, condition);		\
+	__ret;								\
+})
 
 #define __wait_event_timeout(wq, condition, timeout)			\
 	___wait_event(wq, ___wait_cond_timeout(condition),		\
#5 lore raw

Re: [PATCH 2/7] wait: Reimplement wait_event_freezable()

In reply to Peter Zijlstra
Quoted 6 lines · Peter Zijlstra

Provide better implementations of wait_event_freezable() APIs.

The problem is with freezer_do_not_count(), it hides the thread from the freezer, even though this thread might not actually freeze/sleep at all.

I agree, wait_event_freezable() is awful. But could you clarify "at all" ?

Sure, the task can be preempted right after it sets, it can do a lot of things before it calls schedule(), it can be woken after that and it can run again and do something else before freezer_count() calls try_to_freeze(), etc.

Is this what you meant?

include/linux/wait.h do { \ · line 272

Quoted 3 lines
include/linux/wait.h +3 -0
+#define __wait_event_freezable(wq, condition)				\
+	(void)___wait_event(wq, condition, TASK_INTERRUPTIBLE, 0, 0,	\
+			    schedule(); try_to_freeze())

I don't think this can work. wait_event_freezable() should be used by kernel threads and thus we can't rely on TIF_SIGPENDING, freeze_task() simply does wake_up_state(TASK_INTERRUPTIBLE) in this case.

Just for example, suppose that try_to_freeze_tasks() calls freeze_task() before this kthread sets current->state = TASK_INTERRUPTIBLE. In this case __wait_event_freezable()->schedule() will happily sleep and try_to_freeze_tasks() will fail.

That is why I tried to suggest cmd == freezable_schedule(). Still not good, but at least this narrows the window and (perhaps) we can improve this freezable_schedule() later.

But on a second thought... Probably cmd => try_to_freeze(); schedule(); should work. Or just

	#define __wait_event_freezable(wq, condition)	\
		__wait_event_interruptible(wq, ({ try_to_freeze(); (condition); }))

which looks simpler.

Oleg.

author maintainer
#6 lore raw

Re: [PATCH 2/7] wait: Reimplement wait_event_freezable()

Quoted 8 lines · Oleg Nesterov
Quoted 6 lines · Peter Zijlstra

Provide better implementations of wait_event_freezable() APIs.

The problem is with freezer_do_not_count(), it hides the thread from the freezer, even though this thread might not actually freeze/sleep at all.

Quoted 8 lines

I agree, wait_event_freezable() is awful. But could you clarify "at all" ?

Sure, the task can be preempted right after it sets, it can do a lot of things before it calls schedule(), it can be woken after that and it can run again and do something else before freezer_count() calls try_to_freeze(), etc.

Is this what you meant?

Yes.

Quoted 5 lines

include/linux/wait.h do { \ · line 272

Quoted 3 lines
include/linux/wait.h +3 -0
+#define __wait_event_freezable(wq, condition)				\
+	(void)___wait_event(wq, condition, TASK_INTERRUPTIBLE, 0, 0,	\
+			    schedule(); try_to_freeze())

I don't think this can work.

Yeah, that was horribly broken. defconfig didn't seem to find a usage site. Wu's robot offered a .config quickly enough though ;-)

#7 lore raw

Re: [PATCH 2/7] wait: Reimplement wait_event_freezable()

In reply to Oleg Nesterov
Quoted 36 lines · Oleg Nesterov
Quoted 6 lines · Peter Zijlstra

Provide better implementations of wait_event_freezable() APIs.

The problem is with freezer_do_not_count(), it hides the thread from the freezer, even though this thread might not actually freeze/sleep at all.

I agree, wait_event_freezable() is awful. But could you clarify "at all" ?

Sure, the task can be preempted right after it sets, it can do a lot of things before it calls schedule(), it can be woken after that and it can run again and do something else before freezer_count() calls try_to_freeze(), etc.

Is this what you meant?

include/linux/wait.h do { \ · line 272

Quoted 3 lines
include/linux/wait.h +3 -0
+#define __wait_event_freezable(wq, condition)				\
+	(void)___wait_event(wq, condition, TASK_INTERRUPTIBLE, 0, 0,	\
+			    schedule(); try_to_freeze())

I don't think this can work. wait_event_freezable() should be used by kernel threads and thus we can't rely on TIF_SIGPENDING, freeze_task() simply does wake_up_state(TASK_INTERRUPTIBLE) in this case.

Just for example, suppose that try_to_freeze_tasks() calls freeze_task() before this kthread sets current->state = TASK_INTERRUPTIBLE. In this case __wait_event_freezable()->schedule() will happily sleep and try_to_freeze_tasks() will fail.

That is why I tried to suggest cmd == freezable_schedule(). Still not good, but at least this narrows the window and (perhaps) we can improve this freezable_schedule() later.

Quoted 7 lines

But on a second thought... Probably cmd => try_to_freeze(); schedule(); should work. Or just

	#define __wait_event_freezable(wq, condition)	\
		__wait_event_interruptible(wq, ({ try_to_freeze(); (condition); }))

which looks simpler.

Ah, but I forgot about another problem... can't we finally kill this ugly "long save = current->state" code in __refrigerator() ? Nobody seem to understand why we are doing this, and this looks just wrong.

And this doesn't allow us to do try_to_freeze() + schedule().

Oleg.

author maintainer
#8 lore raw

PATCH 3/7 wait: Remove wait_event_freezekillable()

In reply to Peter Zijlstra

There is no user.. make it go away.

---
 include/linux/freezer.h |   12 ------------
 1 file changed, 12 deletions(-)
include/linux/freezer.h +0 -12
@@ -246,15 +246,6 @@ static inline int freezable_schedule_hrt
  * defined in <linux/wait.h>
  */
 
-#define wait_event_freezekillable(wq, condition)			\
-({									\
-	int __retval;							\
-	freezer_do_not_count();						\
-	__retval = wait_event_killable(wq, (condition));		\
-	freezer_count();						\
-	__retval;							\
-})
-
 /* DO NOT ADD ANY NEW CALLERS OF THIS FUNCTION */
 #define wait_event_freezekillable_unsafe(wq, condition)			\
 ({									\
@@ -302,9 +293,6 @@ static inline void set_freezable(void) {
 #define freezable_schedule_hrtimeout_range(expires, delta, mode)	\
 	schedule_hrtimeout_range(expires, delta, mode)
 
-#define wait_event_freezekillable(wq, condition)		\
-		wait_event_killable(wq, condition)
-
 #define wait_event_freezekillable_unsafe(wq, condition)			\
 		wait_event_killable(wq, condition)
 
author maintainer
#9 lore raw

PATCH 4/7 audit,wait: Fixup kauditd_thread wait loop

In reply to Peter Zijlstra

The kauditd_thread wait loop is a bit iffy; it has a number of problems:

- calls try_to_freeze() before schedule(); you typically want the thread to re-evaluate the sleep condition when unfreezing, also freeze_task() issues a wakeup.

- it unconditionally does the {add,remove}_wait_queue(), even when the sleep condition is false.

Use wait_event_freezable() that does the right thing.

---
 kernel/audit.c |   11 +----------
 1 file changed, 1 insertion(+), 10 deletions(-)
kernel/audit.c +1 -10
@@ -499,7 +499,6 @@ static int kauditd_thread(void *dummy)
 	set_freezable();
 	while (!kthread_should_stop()) {
 		struct sk_buff *skb;
-		DECLARE_WAITQUEUE(wait, current);
 
 		flush_hold_queue();
 
@@ -514,16 +513,8 @@ static int kauditd_thread(void *dummy)
 				audit_printk_skb(skb);
 			continue;
 		}
-		set_current_state(TASK_INTERRUPTIBLE);
-		add_wait_queue(&kauditd_wait, &wait);
 
-		if (!skb_queue_len(&audit_skb_queue)) {
-			try_to_freeze();
-			schedule();
-		}
-
-		__set_current_state(TASK_RUNNING);
-		remove_wait_queue(&kauditd_wait, &wait);
+		wait_event_freezable(kauditd_wait, skb_queue_len(&audit_skb_queue));
 	}
 	return 0;
 }
author maintainer
#10 lore raw

PATCH 5/7 rfcomm: Fix broken wait construct

In reply to Peter Zijlstra

rfcomm_run() is a tad broken in that is has a nested wait loop. One cannot rely on p->state for the outer wait because the inner wait will overwrite it.

Cc Marcel Holtmann <[email protected]>
Cc Peter Hurley <[email protected]>
---
 net/bluetooth/rfcomm/core.c |   18 ++++++++----------
 1 file changed, 8 insertions(+), 10 deletions(-)
net/bluetooth/rfcomm/core.c +8 -10
@@ -101,11 +101,11 @@ static struct rfcomm_session *rfcomm_ses
 #define __get_rpn_stop_bits(line) (((line) >> 2) & 0x1)
 #define __get_rpn_parity(line)    (((line) >> 3) & 0x7)
 
+static DECLARE_WAIT_QUEUE_HEAD(rfcomm_wq);
+
 static void rfcomm_schedule(void)
 {
-	if (!rfcomm_thread)
-		return;
-	wake_up_process(rfcomm_thread);
+	wake_up_all(&rfcomm_wq);
 }
 
 /* ---- RFCOMM FCS computation ---- */
@@ -2086,24 +2086,22 @@ static void rfcomm_kill_listener(void)
 
 static int rfcomm_run(void *unused)
 {
+	DEFINE_WAIT_FUNC(wait, woken_wake_function);
 	BT_DBG("");
 
 	set_user_nice(current, -10);
 
 	rfcomm_add_listener(BDADDR_ANY);
 
-	while (1) {
-		set_current_state(TASK_INTERRUPTIBLE);
-
-		if (kthread_should_stop())
-			break;
+	add_wait_queue(&rfcomm_wq, &wait);
+	while (!kthread_should_stop()) {
 
 		/* Process stuff */
 		rfcomm_process_sessions();
 
-		schedule();
+		wait_woken(&wait, TASK_INTERRUPTIBLE, MAX_SCHEDULE_TIMEOUT);
 	}
-	__set_current_state(TASK_RUNNING);
+	remove_wait_queue(&rfcomm_wq, &wait);
 
 	rfcomm_kill_listener();
 
author maintainer
#11 lore raw

PATCH 6/7 netdev: Fix sleeping inside wait event

In reply to Peter Zijlstra

rtnl_lock_unregistering*() take rtnl_lock() -- a mutex -- inside a wait loop. The wait loop relies on current->state to function, but so does mutex_lock(), nesting them makes for the inner to destroy the outer state.

Fix this using the new wait_woken() bits.

---
 net/core/dev.c       |   10 +++++-----
 net/core/rtnetlink.c |   10 +++++-----
 2 files changed, 10 insertions(+), 10 deletions(-)
net/core/dev.c +5 -5
@@ -7196,11 +7196,10 @@ static void __net_exit rtnl_lock_unregis
 	 */
 	struct net *net;
 	bool unregistering;
-	DEFINE_WAIT(wait);
+	DEFINE_WAIT_FUNC(wait, woken_wake_function);
 
+	add_wait_queue(&netdev_unregistering_wq, &wait);
 	for (;;) {
-		prepare_to_wait(&netdev_unregistering_wq, &wait,
-				TASK_UNINTERRUPTIBLE);
 		unregistering = false;
 		rtnl_lock();
 		list_for_each_entry(net, net_list, exit_list) {
@@ -7212,9 +7211,10 @@ static void __net_exit rtnl_lock_unregis
 		if (!unregistering)
 			break;
 		__rtnl_unlock();
-		schedule();
+
+		wait_woken(&wait, TASK_UNINTERRUPTIBLE, MAX_SCHEDULE_TIMEOUT);
 	}
-	finish_wait(&netdev_unregistering_wq, &wait);
+	remove_wait_queue(&netdev_unregistering_wq, &wait);
 }
 
 static void __net_exit default_device_exit_batch(struct list_head *net_list)
net/core/rtnetlink.c +5 -5
@@ -365,11 +365,10 @@ static void rtnl_lock_unregistering_all(
 {
 	struct net *net;
 	bool unregistering;
-	DEFINE_WAIT(wait);
+	DEFINE_WAIT_FUNC(wait, woken_wake_function);
 
+	add_wait_queue(&netdev_unregistering_wq, &wait);
 	for (;;) {
-		prepare_to_wait(&netdev_unregistering_wq, &wait,
-				TASK_UNINTERRUPTIBLE);
 		unregistering = false;
 		rtnl_lock();
 		for_each_net(net) {
@@ -381,9 +380,10 @@ static void rtnl_lock_unregistering_all(
 		if (!unregistering)
 			break;
 		__rtnl_unlock();
-		schedule();
+
+		wait_woken(&wait, TASK_UNINTERRUPTIBLE, MAX_SCHEDULE_TIMEOUT);
 	}
-	finish_wait(&netdev_unregistering_wq, &wait);
+	remove_wait_queue(&netdev_unregistering_wq, &wait);
 }
 
 /**
author maintainer
#12 lore raw

PATCH 7/7 sched: Use WARN_ONCE for the might_sleep() TASK_RUNNING test

In reply to Peter Zijlstra

In some cases this can trigger a true flood of output.

---
 kernel/sched/core.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/core.c +1 -1
@@ -7301,7 +7301,7 @@ void __might_sleep(const char *file, int
 	 * since we will exit with TASK_RUNNING make sure we enter with it,
 	 * otherwise we will destroy state.
 	 */
-	if (WARN(current->state != TASK_RUNNING,
+	if (WARN_ONCE(current->state != TASK_RUNNING,
 			"do not call blocking ops when !TASK_RUNNING; "
 			"state=%lx set at [<%p>] %pS\n",
 			current->state,
#13 lore raw

Re: [PATCH 7/7] sched: Use WARN_ONCE for the might_sleep() TASK_RUNNING test

Quoted 12 lines · Peter Zijlstra

In some cases this can trigger a true flood of output.

---
 kernel/sched/core.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

kernel/sched/core.c void __might_sleep(const char *file, int · line 7304

Quoted 5 lines
kernel/sched/core.c +1 -1
@@ -7301,7 +7301,7 @@ void __might_sleep(const char *file, int
 	 * since we will exit with TASK_RUNNING make sure we enter with it,
 	 * otherwise we will destroy state.
 	 */
-	if (WARN(current->state != TASK_RUNNING,
+	if (WARN_ONCE(current->state != TASK_RUNNING,

Agreed, but sorry for off-topic, can't resist.

Sometimes I hate WARN_ONCE() because you can't reproduce the problem once again without reboot.

Perhaps WARN_ON_RATELIMIT() should be used more often (not sure about this particular case). Or, perhaps, we can add a special section for these "__warned" variables and add, say, sysctl which clears that section ?

Oleg.

author maintainer
#14 lore raw

Re: [PATCH 7/7] sched: Use WARN_ONCE for the might_sleep() TASK_RUNNING test

Quoted 21 lines · Oleg Nesterov
Quoted 12 lines · Peter Zijlstra

In some cases this can trigger a true flood of output.

---
 kernel/sched/core.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

kernel/sched/core.c void __might_sleep(const char *file, int · line 7304

Quoted 5 lines
kernel/sched/core.c +1 -1
@@ -7301,7 +7301,7 @@ void __might_sleep(const char *file, int
 	 * since we will exit with TASK_RUNNING make sure we enter with it,
 	 * otherwise we will destroy state.
 	 */
-	if (WARN(current->state != TASK_RUNNING,
+	if (WARN_ONCE(current->state != TASK_RUNNING,

Agreed, but sorry for off-topic, can't resist.

Quoted 2 lines

Sometimes I hate WARN_ONCE() because you can't reproduce the problem once again without reboot.

Yes, and the fact that you can only see the first fail, even if more are present.

Quoted 4 lines

Perhaps WARN_ON_RATELIMIT() should be used more often (not sure about this particular case). Or, perhaps, we can add a special section for these "__warned" variables and add, say, sysctl which clears that section ?

Yeah, maybe, /debug/warn_once/file/line/enable or whatnot. For now I'll continue removing ONCEs whenever I feel like it though ;-)