Skip to contentSkip to conversation

futex_wait_setup sleeping while atomic bug.

DISCUSSION Dave Jones 5 participants 7 messages Sep 11, 2014 – Sep 12, 2014

lore
Share & tools
lore

Dave Jones

author
#0 lore raw

Hit this overnight on Linus tree from yesterday.

BUG: sleeping function called from invalid context at include/linux/freezer.h:56

in_atomic(): 1, irqs_disabled(): 0, pid: 31948, name: trinity-c121 Preemption disabled at:[<ffffffffb90f2512>] futex_wait_setup+0xb2/0x140

CPU: 0 PID: 31948 Comm: trinity-c121 Not tainted 3.17.0-rc4+ #52
 ffffffffb9a52f2a 00000000d3f40479 ffff88008a32fd90 ffffffffb9771ba4
 0000000000000000 ffff88008a32fdb8 ffffffffb909f02c ffff88008a32ff58
 ffff88008a32ff58 0000000000000010 ffff88008a32fe30 ffffffffb90887bc
Call Trace:
 [<ffffffffb9771ba4>] dump_stack+0x4e/0x7a
 [<ffffffffb909f02c>] __might_sleep+0x11c/0x1b0
 [<ffffffffb90887bc>] get_signal+0x6c/0x720
 [<ffffffffb9002577>] do_signal+0x37/0x6f0
 [<ffffffffb9126ddb>] ? __acct_update_integrals+0x8b/0x120
 [<ffffffffb912728c>] ? acct_account_cputime+0x1c/0x20
 [<ffffffffb90ab1ab>] ? account_user_time+0x9b/0xb0
 [<ffffffffb9002c8c>] do_notify_resume+0x5c/0xa0
 [<ffffffffb977be23>] retint_signal+0x46/0x83
BUG: scheduling while atomic: trinity-c121/31948/0x00000002
Preemption disabled at:[<ffffffffb90f2512>] futex_wait_setup+0xb2/0x140

CPU: 0 PID: 31948 Comm: trinity-c121 Not tainted 3.17.0-rc4+ #52
 ffff88024d013740 00000000d3f40479 ffff88008a32fec8 ffffffffb9771ba4
 ffff8802403eee00 ffff88008a32fee0 ffffffffb909e725 0000000000000000
 ffff88008a32ff60 ffffffffb9774db7 000000000000a000 ffff8802403eee00
Call Trace:
 [<ffffffffb9771ba4>] dump_stack+0x4e/0x7a
 [<ffffffffb909e725>] __schedule_bug+0x65/0xc0
 [<ffffffffb9774db7>] __schedule+0x6a7/0xa70
 [<ffffffffb97756ee>] schedule_user+0x2e/0x90
 [<ffffffffb977bdc1>] retint_careful+0x12/0x2e

note: trinity-c121[31948] exited with preempt_count 1

#1 lore raw
Quoted 5 lines · Dave Jones

Hit this overnight on Linus tree from yesterday.

BUG: sleeping function called from invalid context at include/linux/freezer.h:56

in_atomic(): 1, irqs_disabled(): 0, pid: 31948, name: trinity-c121 Preemption disabled at:[<ffffffffb90f2512>] futex_wait_setup+0xb2/0x140

Huch? So we are in a preempt disabled region in futex_wait_setup and we get interrupted and end up in the signal delivery path?

I really love that futex stuff....

But fortunately the preemption disabled hint made it into mainline so looking at the callsites of futex_wait_setup() makes it pretty clear where the shit hits the fan. Patch below.

Thanks,

tglx

------------------->
From: Thomas Gleixner <[email protected]>
Date: Thu, 11 Sep 2014 23:44:35 +0200
Subject: futex: Unlock hb->lock in futex_wait_requeue_pi() error path

futex_wait_requeue_pi() calls futex_wait_setup(). If futex_wait_setup() succeeds it returns with hb->lock held and preemption disabled. Now the sanity check after this does:

        if (match_futex(&q.key, &key2)) {
	   	ret = -EINVAL;
		goto out_put_keys;
	}

which releases the keys but does not release hb->lock. So we happily return to user space with hb->lock held and therefor preemption disabled.

Unlock hb->lock before taking the exit route.

---

kernel/futex.c +1 -0
index d3a9d946d0b7..815d7af2ffe8 100644
@@ -2592,6 +2592,7 @@ static int futex_wait_requeue_pi(u32 __user *uaddr, unsigned int flags,
 	 * shared futexes. We need to compare the keys:
 	 */
 	if (match_futex(&q.key, &key2)) {
+		queue_unlock(hb);
 		ret = -EINVAL;
 		goto out_put_keys;
 	}
Reviewed-by
#2 lore raw
Quoted 49 lines · Thomas Gleixner
Quoted 5 lines · Dave Jones

Hit this overnight on Linus tree from yesterday.

BUG: sleeping function called from invalid context at include/linux/freezer.h:56

in_atomic(): 1, irqs_disabled(): 0, pid: 31948, name: trinity-c121 Preemption disabled at:[<ffffffffb90f2512>] futex_wait_setup+0xb2/0x140

Huch? So we are in a preempt disabled region in futex_wait_setup and we get interrupted and end up in the signal delivery path?

I really love that futex stuff....

But fortunately the preemption disabled hint made it into mainline so looking at the callsites of futex_wait_setup() makes it pretty clear where the shit hits the fan. Patch below.

Thanks,

tglx

------------------->
From: Thomas Gleixner <[email protected]>
Date: Thu, 11 Sep 2014 23:44:35 +0200
Subject: futex: Unlock hb->lock in futex_wait_requeue_pi() error path

futex_wait_requeue_pi() calls futex_wait_setup(). If futex_wait_setup() succeeds it returns with hb->lock held and preemption disabled. Now the sanity check after this does:

        if (match_futex(&q.key, &key2)) {
	   	ret = -EINVAL;
		goto out_put_keys;
	}

which releases the keys but does not release hb->lock. So we happily return to user space with hb->lock held and therefor preemption disabled.

Unlock hb->lock before taking the exit route.

---

kernel/futex.c +0 -0
index d3a9d946d0b7..815d7af2ffe8 100644

kernel/futex.c static int futex_wait_requeue_pi(u32 __user *uaddr, unsigned int flags, · line 2598

Quoted 7 lines
kernel/futex.c +1 -0
@@ -2592,6 +2592,7 @@ static int futex_wait_requeue_pi(u32 __user *uaddr, unsigned int flags,
 	 * shared futexes. We need to compare the keys:
 	 */
 	if (match_futex(&q.key, &key2)) {
+		queue_unlock(hb);
 		ret = -EINVAL;
 		goto out_put_keys;
 	}

I was hoping to move this before the key equivalence test so we could exit as early as possible, but some idgit (that would be me) embedded the get_futex_key() for uaddr into futex_wait_setup() to avoid so much duplication of code... sigh.

Thomas's fix is the best immediate solution in my opinion.

Signature · 2 lines
Darren Hart
Intel Open Source Technology Center
Reviewed-by
#3 lore raw
In reply to Thomas Gleixner
Quoted 3 lines · Thomas Gleixner

From: Thomas Gleixner <[email protected]>
Date: Thu, 11 Sep 2014 23:44:35 +0200
Subject: futex: Unlock hb->lock in futex_wait_requeue_pi() error path

That's the second time we are bitten by bugs in when requeing, now pi. We need to reconsider some of our testing tools to stress these paths better, imo.

Quoted 9 lines

futex_wait_requeue_pi() calls futex_wait_setup(). If futex_wait_setup() succeeds it returns with hb->lock held and preemption disabled. Now the sanity check after this does:

        if (match_futex(&q.key, &key2)) {
	   	ret = -EINVAL;
		goto out_put_keys;
	}
Quoted 8 lines

which releases the keys but does not release hb->lock. So we happily return to user space with hb->lock held and therefor preemption disabled.

Unlock hb->lock before taking the exit route.

#4 lore raw
Quoted 8 lines · Davidlohr Bueso
Quoted 3 lines · Thomas Gleixner

From: Thomas Gleixner <[email protected]>
Date: Thu, 11 Sep 2014 23:44:35 +0200
Subject: futex: Unlock hb->lock in futex_wait_requeue_pi() error path

That's the second time we are bitten by bugs in when requeing, now pi. We need to reconsider some of our testing tools to stress these paths better, imo.

We do, yes. Per the kselftest discussion at kernel summit, I agreed to move the futextest testsuite into the kernel, function into kselftest and performance into perf, then futextest can go away. From there we can look at how to improve these tests.

Sadly, the best testing we seem to have is trinity - which does a fantastic job at finding nasties.

If someone wanted to start having a look at migrating the futextest tests over... I certainly wouldn't object to the help! ;-)

git://git.kernel.org/pub/scm/linux/kernel/git/dvhart/futextest.git

Signature · 2 lines
Darren Hart
Intel Open Source Technology Center
#5 lore raw
In reply to Davidlohr Bueso
Quoted 8 lines · Davidlohr Bueso
Quoted 3 lines · Thomas Gleixner

From: Thomas Gleixner <[email protected]>
Date: Thu, 11 Sep 2014 23:44:35 +0200
Subject: futex: Unlock hb->lock in futex_wait_requeue_pi() error path

That's the second time we are bitten by bugs in when requeing, now pi. We need to reconsider some of our testing tools to stress these paths better, imo.

Testing tools are nice. What we really need is more competent eyes looking at that code ...

Quoted 22 lines
Quoted 9 lines

futex_wait_requeue_pi() calls futex_wait_setup(). If futex_wait_setup() succeeds it returns with hb->lock held and preemption disabled. Now the sanity check after this does:

        if (match_futex(&q.key, &key2)) {
	   	ret = -EINVAL;
		goto out_put_keys;
	}
Quoted 8 lines

which releases the keys but does not release hb->lock. So we happily return to user space with hb->lock held and therefor preemption disabled.

Unlock hb->lock before taking the exit route.

#6 lore raw

[tip:locking/urgent] futex: Unlock hb-> lock in futex_wait_requeue_pi() error path

In reply to Thomas Gleixner
futex: Unlock hb->lock in futex_wait_requeue_pi() error path
Commit-ID:  13c42c2f43b19aab3195f2d357db00d1e885eaa8
Gitweb:     http://git.kernel.org/tip/13c42c2f43b19aab3195f2d357db00d1e885eaa8
Author:     Thomas Gleixner <[email protected]>
AuthorDate: Thu, 11 Sep 2014 23:44:35 +0200
Committer:  Thomas Gleixner <[email protected]>
CommitDate: Fri, 12 Sep 2014 22:04:36 +0200

futex: Unlock hb->lock in futex_wait_requeue_pi() error path

futex_wait_requeue_pi() calls futex_wait_setup(). If futex_wait_setup() succeeds it returns with hb->lock held and preemption disabled. Now the sanity check after this does:

        if (match_futex(&q.key, &key2)) {
	   	ret = -EINVAL;
		goto out_put_keys;
	}

which releases the keys but does not release hb->lock.

So we happily return to user space with hb->lock held and therefor preemption disabled.

Unlock hb->lock before taking the exit route.

---
 kernel/futex.c | 1 +
 1 file changed, 1 insertion(+)
kernel/futex.c +1 -0
index d3a9d94..815d7af 100644
@@ -2592,6 +2592,7 @@ static int futex_wait_requeue_pi(u32 __user *uaddr, unsigned int flags,
 	 * shared futexes. We need to compare the keys:
 	 */
 	if (match_futex(&q.key, &key2)) {
+		queue_unlock(hb);
 		ret = -EINVAL;
 		goto out_put_keys;
 	}