Skip to contentSkip to conversation

WARNING: CPU: 0 PID: 1444 at kernel/sched/core.c:7088 __might_sleep+0x51/0x16f()

setsockopt Applied Fengguang Wu 5 participants 9 messages Aug 5, 2014 – Aug 12, 2014

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Greetings,

Here is a microcode/load_module error triggered by debug check commit 64c2181bc433b17f04da8fe8592aa83cceac9606 ("sched: Debug nested sleeps"):

[main] Setsockopt(1 8 80d1000 4) on fd 21 [1:2:1] [main] Setsockopt(1 2f 80d1000 4) on fd 22 [4:2:60]

[   14.027148] ------------[ cut here ]------------
[   14.027864] WARNING: CPU: 0 PID: 210 at kernel/sched/core.c:7088 __might_sleep+0x40/0x68()
[   14.029295] do not call blocking ops when !TASK_RUNNING; state=2 set at [<c144e379>] prepare_to_wait+0x35/0x56
[   14.030590] Modules linked in:
[   14.031136] CPU: 0 PID: 210 Comm: trinity-main Not tainted 3.16.0-02167-g254135e #972
[   14.032263]  00000000 c0f4de4c c0f4de24 c196630c c0f4de3c c142f01a c1447632 c0f1dbb0
[   14.033480]  00000002 b0066140 c0f4de54 c142f057 00000009 c0f4de4c c1b3bac8 c0f4de68
[   14.034640]  c0f4de88 c1447632 c1b3bb12 00001bb0 c1b3bac8 00000002 c144e379 c144e379
[   14.035983] Call Trace:
[   14.036355]  [<c196630c>] dump_stack+0x16/0x18
[   14.037005]  [<c142f01a>] warn_slowpath_common+0x55/0x6c
[   14.037715]  [<c1447632>] ? __might_sleep+0x40/0x68
[   14.038372]  [<c142f057>] warn_slowpath_fmt+0x26/0x2a
[   14.039097]  [<c1447632>] __might_sleep+0x40/0x68
[   14.039787]  [<c144e379>] ? prepare_to_wait+0x35/0x56
[   14.040595]  [<c144e379>] ? prepare_to_wait+0x35/0x56
[   14.041272]  [<c14a837e>] kmem_cache_alloc+0x39/0xb0
[   14.041934]  [<c18fa2de>] ? __alloc_skb+0x3c/0x154
[   14.042572]  [<c18fa2de>] __alloc_skb+0x3c/0x154
[   14.043339]  [<c145117a>] ? mark_held_locks+0x44/0x60
[   14.044141]  [<c1946093>] sigd_enq2+0x2a/0xff
[   14.044836]  [<c1946188>] sigd_enq+0x20/0x2a
[   14.045405]  [<c19467fb>] svc_listen+0x8b/0x11f
[   14.046009]  [<c144e5a6>] ? __wake_up_sync+0xd/0xd
[   14.046653]  [<c18f4132>] SyS_listen+0x37/0x51
[   14.047423]  [<c18f4ce5>] SyS_socketcall+0x90/0x1c0
[   14.048328]  [<c145136e>] ? trace_hardirqs_on+0xb/0xd
[   14.049061]  [<c19729f6>] ? restore_all+0xf/0xf
[   14.049665]  [<c19729bd>] syscall_call+0x7/0x7
[   14.050253]  [<c1970000>] ? __ww_mutex_lock_interruptible+0x165/0x573
[   14.051147] ---[ end trace 6f1365c63eafedde ]---

[main] Setsockopt(1 2d 80d1000 f0) on fd 25 [1:1:1]

Thanks,
Fengguang

#1 lore raw
Quoted 3 lines · Fengguang Wu

Greetings,

Here is a microcode/load_module error triggered by debug check commit

Sorry for the copy&paste error! This warning is not related to microcode/load_module..

Thanks,
Fengguang

#2 lore raw
Quoted 8 lines · Fengguang Wu
Quoted 3 lines · Fengguang Wu

Greetings,

Here is a microcode/load_module error triggered by debug check commit

Sorry for the copy&paste error! This warning is not related to microcode/load_module..

Quoted 7 lines

Thanks,
Fengguang
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Hey Fenngaug,
I looking through this trace.

213 skb = kmem_cache_alloc_node(cache, gfp_mask & ~__GFP_DMA, node); 214 if (!skb)
215 goto out;
This lines seem to be an issue as after tracing to were you are getting a warning,
might_sleep is written as

might_sleep_if(flags & __GFP_WAIT);
This means that the flag for GFP_WAIT is probably set and sure enough you would be getting an error after sleeping in __alloc_skb.
Regards Nick

#3 lore raw
In reply to Fengguang Wu
Quoted 38 lines · Fengguang Wu

Greetings,

Here is a microcode/load_module error triggered by debug check commit 64c2181bc433b17f04da8fe8592aa83cceac9606 ("sched: Debug nested sleeps"):

[main] Setsockopt(1 8 80d1000 4) on fd 21 [1:2:1] [main] Setsockopt(1 2f 80d1000 4) on fd 22 [4:2:60]

[   14.027148] ------------[ cut here ]------------
[   14.027864] WARNING: CPU: 0 PID: 210 at kernel/sched/core.c:7088 __might_sleep+0x40/0x68()
[   14.029295] do not call blocking ops when !TASK_RUNNING; state=2 set at [<c144e379>] prepare_to_wait+0x35/0x56
[   14.030590] Modules linked in:
[   14.031136] CPU: 0 PID: 210 Comm: trinity-main Not tainted 3.16.0-02167-g254135e #972
[   14.032263]  00000000 c0f4de4c c0f4de24 c196630c c0f4de3c c142f01a c1447632 c0f1dbb0
[   14.033480]  00000002 b0066140 c0f4de54 c142f057 00000009 c0f4de4c c1b3bac8 c0f4de68
[   14.034640]  c0f4de88 c1447632 c1b3bb12 00001bb0 c1b3bac8 00000002 c144e379 c144e379
[   14.035983] Call Trace:
[   14.036355]  [<c196630c>] dump_stack+0x16/0x18
[   14.037005]  [<c142f01a>] warn_slowpath_common+0x55/0x6c
[   14.037715]  [<c1447632>] ? __might_sleep+0x40/0x68
[   14.038372]  [<c142f057>] warn_slowpath_fmt+0x26/0x2a
[   14.039097]  [<c1447632>] __might_sleep+0x40/0x68
[   14.039787]  [<c144e379>] ? prepare_to_wait+0x35/0x56
[   14.040595]  [<c144e379>] ? prepare_to_wait+0x35/0x56
[   14.041272]  [<c14a837e>] kmem_cache_alloc+0x39/0xb0
[   14.041934]  [<c18fa2de>] ? __alloc_skb+0x3c/0x154
[   14.042572]  [<c18fa2de>] __alloc_skb+0x3c/0x154
[   14.043339]  [<c145117a>] ? mark_held_locks+0x44/0x60
[   14.044141]  [<c1946093>] sigd_enq2+0x2a/0xff
[   14.044836]  [<c1946188>] sigd_enq+0x20/0x2a
[   14.045405]  [<c19467fb>] svc_listen+0x8b/0x11f
[   14.046009]  [<c144e5a6>] ? __wake_up_sync+0xd/0xd
[   14.046653]  [<c18f4132>] SyS_listen+0x37/0x51
[   14.047423]  [<c18f4ce5>] SyS_socketcall+0x90/0x1c0
[   14.048328]  [<c145136e>] ? trace_hardirqs_on+0xb/0xd
[   14.049061]  [<c19729f6>] ? restore_all+0xf/0xf
[   14.049665]  [<c19729bd>] syscall_call+0x7/0x7
[   14.050253]  [<c1970000>] ? __ww_mutex_lock_interruptible+0x165/0x573
[   14.051147] ---[ end trace 6f1365c63eafedde ]---
Quoted 1 line

[main] Setsockopt(1 2d 80d1000 f0) on fd 25 [1:1:1]

---
Subject: atm: Fix blocking in wait loop

One should not call blocking primitives inside a wait loop, since both require task_struct::state to sleep, so the inner will destroy the outer state.

In this instance sigd_enq() will possible sleep for alloc_skb(), now if I understand the code right, we do not actually need to call sigd_enq() after the initial prepare_to_wait(), because we test the termination condition before schedule() anyhow.

So we can simply move it up a bit and avoid the entire confusion.

---
 net/atm/svc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/atm/svc.c +1 -1
index d8e5d0c2ebbc..445ac238b69b 100644
@@ -297,8 +297,8 @@ static int svc_listen(struct socket *sock, int backlog)
 		goto out;
 	}
 	set_bit(ATM_VF_WAITING, &vcc->flags);
-	prepare_to_wait(sk_sleep(sk), &wait, TASK_UNINTERRUPTIBLE);
 	sigd_enq(vcc, as_listen, NULL, NULL, &vcc->local);
+	prepare_to_wait(sk_sleep(sk), &wait, TASK_UNINTERRUPTIBLE);
 	while (test_bit(ATM_VF_WAITING, &vcc->flags) && sigd) {
 		schedule();
 		prepare_to_wait(sk_sleep(sk), &wait, TASK_UNINTERRUPTIBLE);
#4 lore raw
Quoted 23 lines · On Thu, 7 Aug 2014 17:17:41 +0200 Peter Zijlstra <[email protected]> wrote:

Subject: atm: Fix blocking in wait loop

One should not call blocking primitives inside a wait loop, since both require task_struct::state to sleep, so the inner will destroy the outer state.

In this instance sigd_enq() will possible sleep for alloc_skb(), now if I understand the code right, we do not actually need to call sigd_enq() after the initial prepare_to_wait(), because we test the termination condition before schedule() anyhow.

So we can simply move it up a bit and avoid the entire confusion.

---
 net/atm/svc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/atm/svc.c +0 -0
index d8e5d0c2ebbc..445ac238b69b 100644

net/atm/svc.c static int svc_listen(struct socket *sock, int backlog) · line 304

Quoted 9 lines
net/atm/svc.c +1 -1
@@ -297,8 +297,8 @@ static int svc_listen(struct socket *sock, int backlog)
 		goto out;
 	}
 	set_bit(ATM_VF_WAITING, &vcc->flags);
-	prepare_to_wait(sk_sleep(sk), &wait, TASK_UNINTERRUPTIBLE);
 	sigd_enq(vcc, as_listen, NULL, NULL, &vcc->local);
+	prepare_to_wait(sk_sleep(sk), &wait, TASK_UNINTERRUPTIBLE);
 	while (test_bit(ATM_VF_WAITING, &vcc->flags) && sigd) {
 		schedule();
 		prepare_to_wait(sk_sleep(sk), &wait, TASK_UNINTERRUPTIBLE);

This isn't the only place that we queue a message for the signalling daemon after a prepare_to_wait() uninterruptibly so this patch would be incomplete as is.

What bothers me is the TASK_UNINTERRUPTIBLE -- I don't have a good reason why any of these should be sleeping uninterruptibly.

#5 lore raw
Quoted 32 lines · chas williams - CONTRACTOR
Quoted 29 lines · On Thu, 7 Aug 2014 17:17:41 +0200 Peter Zijlstra <[email protected]> wrote:

Subject: atm: Fix blocking in wait loop

One should not call blocking primitives inside a wait loop, since both require task_struct::state to sleep, so the inner will destroy the outer state.

In this instance sigd_enq() will possible sleep for alloc_skb(), now if I understand the code right, we do not actually need to call sigd_enq() after the initial prepare_to_wait(), because we test the termination condition before schedule() anyhow.

So we can simply move it up a bit and avoid the entire confusion.

---
 net/atm/svc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
net/atm/svc.c +1 -1
index d8e5d0c2ebbc..445ac238b69b 100644
@@ -297,8 +297,8 @@ static int svc_listen(struct socket *sock, int backlog)
 		goto out;
 	}
 	set_bit(ATM_VF_WAITING, &vcc->flags);
-	prepare_to_wait(sk_sleep(sk), &wait, TASK_UNINTERRUPTIBLE);
 	sigd_enq(vcc, as_listen, NULL, NULL, &vcc->local);
+	prepare_to_wait(sk_sleep(sk), &wait, TASK_UNINTERRUPTIBLE);
Quoted 8 lines

net/atm/svc.c static int svc_listen(struct socket *sock, int backlog) · line 304

Quoted 3 lines
net/atm/svc.c +0 -0
 	while (test_bit(ATM_VF_WAITING, &vcc->flags) && sigd) {
 		schedule();
 		prepare_to_wait(sk_sleep(sk), &wait, TASK_UNINTERRUPTIBLE);

This isn't the only place that we queue a message for the signalling daemon after a prepare_to_wait() uninterruptibly so this patch would be incomplete as is.

I'm not sure I follow, this is the only place you do so while then going to sleep. All other sites don't sleep while they're enqueued on the waitqueue.

Quoted 2 lines

What bothers me is the TASK_UNINTERRUPTIBLE -- I don't have a good reason why any of these should be sleeping uninterruptibly.

That's a whole different story, there's tons of ugly in there, but its all ancient code.

#6 lore raw
Quoted 7 lines · Peter Zijlstra
Quoted 3 lines

This isn't the only place that we queue a message for the signalling daemon after a prepare_to_wait() uninterruptibly so this patch would be incomplete as is.

I'm not sure I follow, this is the only place you do so while then going to sleep. All other sites don't sleep while they're enqueued on the waitqueue.

Argh, my brain is fried, you're quite right. I'll go have another stab at it tomorrow or so.

#7 lore raw

PATCH 1/1 atm/svc: Fix blocking in wait loop

One should not call blocking primitives inside a wait loop, since both require task_struct::state to sleep, so the inner will destroy the outer state.

sigd_enq() will possibly sleep for alloc_skb(). Move sigd_enq() before prepare_to_wait() to avoid sleeping while waiting interruptibly. You do not actually need to call sigd_enq() after the initial prepare_to_wait() because we test the termination condition before calling schedule().

Based on suggestions from Peter Zijlstra.

---
 net/atm/svc.c | 60 +++++++++++++++++++++++++++++++----------------------------
 1 file changed, 32 insertions(+), 28 deletions(-)
net/atm/svc.c +32 -28
index d8e5d0c2..1ba23f5 100644
@@ -50,12 +50,12 @@ static void svc_disconnect(struct atm_vcc *vcc)
 
 	pr_debug("%p\n", vcc);
 	if (test_bit(ATM_VF_REGIS, &vcc->flags)) {
-		prepare_to_wait(sk_sleep(sk), &wait, TASK_UNINTERRUPTIBLE);
 		sigd_enq(vcc, as_close, NULL, NULL, NULL);
-		while (!test_bit(ATM_VF_RELEASED, &vcc->flags) && sigd) {
+		for (;;) {
+			prepare_to_wait(sk_sleep(sk), &wait, TASK_UNINTERRUPTIBLE);
+			if (test_bit(ATM_VF_RELEASED, &vcc->flags) || !sigd)
+				break;
 			schedule();
-			prepare_to_wait(sk_sleep(sk), &wait,
-					TASK_UNINTERRUPTIBLE);
 		}
 		finish_wait(sk_sleep(sk), &wait);
 	}
@@ -126,11 +126,12 @@ static int svc_bind(struct socket *sock, struct sockaddr *sockaddr,
 	}
 	vcc->local = *addr;
 	set_bit(ATM_VF_WAITING, &vcc->flags);
-	prepare_to_wait(sk_sleep(sk), &wait, TASK_UNINTERRUPTIBLE);
 	sigd_enq(vcc, as_bind, NULL, NULL, &vcc->local);
-	while (test_bit(ATM_VF_WAITING, &vcc->flags) && sigd) {
-		schedule();
+	for (;;) {
 		prepare_to_wait(sk_sleep(sk), &wait, TASK_UNINTERRUPTIBLE);
+		if (!test_bit(ATM_VF_WAITING, &vcc->flags) || !sigd)
+			break;
+		schedule();
 	}
 	finish_wait(sk_sleep(sk), &wait);
 	clear_bit(ATM_VF_REGIS, &vcc->flags); /* doesn't count */
@@ -202,15 +203,14 @@ static int svc_connect(struct socket *sock, struct sockaddr *sockaddr,
 		}
 		vcc->remote = *addr;
 		set_bit(ATM_VF_WAITING, &vcc->flags);
-		prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
 		sigd_enq(vcc, as_connect, NULL, NULL, &vcc->remote);
 		if (flags & O_NONBLOCK) {
-			finish_wait(sk_sleep(sk), &wait);
 			sock->state = SS_CONNECTING;
 			error = -EINPROGRESS;
 			goto out;
 		}
 		error = 0;
+		prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
 		while (test_bit(ATM_VF_WAITING, &vcc->flags) && sigd) {
 			schedule();
 			if (!signal_pending(current)) {
@@ -297,11 +297,12 @@ static int svc_listen(struct socket *sock, int backlog)
 		goto out;
 	}
 	set_bit(ATM_VF_WAITING, &vcc->flags);
-	prepare_to_wait(sk_sleep(sk), &wait, TASK_UNINTERRUPTIBLE);
 	sigd_enq(vcc, as_listen, NULL, NULL, &vcc->local);
-	while (test_bit(ATM_VF_WAITING, &vcc->flags) && sigd) {
-		schedule();
+	for (;;) {
 		prepare_to_wait(sk_sleep(sk), &wait, TASK_UNINTERRUPTIBLE);
+		if (!test_bit(ATM_VF_WAITING, &vcc->flags) || !sigd)
+			break;
+		schedule();
 	}
 	finish_wait(sk_sleep(sk), &wait);
 	if (!sigd) {
@@ -387,15 +388,15 @@ static int svc_accept(struct socket *sock, struct socket *newsock, int flags)
 		}
 		/* wait should be short, so we ignore the non-blocking flag */
 		set_bit(ATM_VF_WAITING, &new_vcc->flags);
-		prepare_to_wait(sk_sleep(sk_atm(new_vcc)), &wait,
-				TASK_UNINTERRUPTIBLE);
 		sigd_enq(new_vcc, as_accept, old_vcc, NULL, NULL);
-		while (test_bit(ATM_VF_WAITING, &new_vcc->flags) && sigd) {
+		for (;;) {
+			prepare_to_wait(sk_sleep(sk_atm(new_vcc)), &wait,
+					TASK_UNINTERRUPTIBLE);
+			if (!test_bit(ATM_VF_WAITING, &new_vcc->flags) || !sigd)
+				break;
 			release_sock(sk);
 			schedule();
 			lock_sock(sk);
-			prepare_to_wait(sk_sleep(sk_atm(new_vcc)), &wait,
-					TASK_UNINTERRUPTIBLE);
 		}
 		finish_wait(sk_sleep(sk_atm(new_vcc)), &wait);
 		if (!sigd) {
@@ -433,12 +434,14 @@ int svc_change_qos(struct atm_vcc *vcc, struct atm_qos *qos)
 	DEFINE_WAIT(wait);
 
 	set_bit(ATM_VF_WAITING, &vcc->flags);
-	prepare_to_wait(sk_sleep(sk), &wait, TASK_UNINTERRUPTIBLE);
 	sigd_enq2(vcc, as_modify, NULL, NULL, &vcc->local, qos, 0);
-	while (test_bit(ATM_VF_WAITING, &vcc->flags) &&
-	       !test_bit(ATM_VF_RELEASED, &vcc->flags) && sigd) {
-		schedule();
+	for (;;) {
 		prepare_to_wait(sk_sleep(sk), &wait, TASK_UNINTERRUPTIBLE);
+		if (!test_bit(ATM_VF_WAITING, &vcc->flags) ||
+		    test_bit(ATM_VF_RELEASED, &vcc->flags) || !sigd) {
+			break;
+		}
+		schedule();
 	}
 	finish_wait(sk_sleep(sk), &wait);
 	if (!sigd)
@@ -529,18 +532,18 @@ static int svc_addparty(struct socket *sock, struct sockaddr *sockaddr,
 
 	lock_sock(sk);
 	set_bit(ATM_VF_WAITING, &vcc->flags);
-	prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
 	sigd_enq(vcc, as_addparty, NULL, NULL,
 		 (struct sockaddr_atmsvc *) sockaddr);
 	if (flags & O_NONBLOCK) {
-		finish_wait(sk_sleep(sk), &wait);
 		error = -EINPROGRESS;
 		goto out;
 	}
 	pr_debug("added wait queue\n");
-	while (test_bit(ATM_VF_WAITING, &vcc->flags) && sigd) {
-		schedule();
+	for (;;) {
 		prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+		if (!test_bit(ATM_VF_WAITING, &vcc->flags) || !sigd)
+			break;
+		schedule();
 	}
 	finish_wait(sk_sleep(sk), &wait);
 	error = xchg(&sk->sk_err_soft, 0);
@@ -558,11 +561,12 @@ static int svc_dropparty(struct socket *sock, int ep_ref)
 
 	lock_sock(sk);
 	set_bit(ATM_VF_WAITING, &vcc->flags);
-	prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
 	sigd_enq2(vcc, as_dropparty, NULL, NULL, NULL, NULL, ep_ref);
-	while (test_bit(ATM_VF_WAITING, &vcc->flags) && sigd) {
-		schedule();
+	for (;;) {
 		prepare_to_wait(sk_sleep(sk), &wait, TASK_INTERRUPTIBLE);
+		if (!test_bit(ATM_VF_WAITING, &vcc->flags) || !sigd)
+			break;
+		schedule();
 	}
 	finish_wait(sk_sleep(sk), &wait);
 	if (!sigd) {
Signature · 1 line
1.9.3
maintainer Applied
#8 lore raw

Re: [PATCH] atm/svc: Fix blocking in wait loop

From: chas williams - CONTRACTOR <[email protected]>
Date: Tue, 12 Aug 2014 08:12:26 -0400

Quoted 9 lines

One should not call blocking primitives inside a wait loop, since both require task_struct::state to sleep, so the inner will destroy the outer state.

sigd_enq() will possibly sleep for alloc_skb(). Move sigd_enq() before prepare_to_wait() to avoid sleeping while waiting interruptibly. You do not actually need to call sigd_enq() after the initial prepare_to_wait() because we test the termination condition before calling schedule().

Quoted 4 lines

Based on suggestions from Peter Zijlstra.

Applied.