Skip to contentSkip to conversation

FAILED: patch "[PATCH] epoll: fix race between ep_poll_callback(POLLFREE) and" failed to apply to 3.18-stable tree

PATCH [email protected] 2 participants 2 messages Sep 4, 2017

Subsystems (2)
Stable branch VFS
lore
Share & tools
lore

[email protected]

author maintainer
#0 lore raw

The patch below does not apply to the 3.18-stable tree.
If someone wants it applied there, or to any other stable or longterm tree, then please email the backport, including the original git commit id to <[email protected]>.

thanks,

greg k-h

------------------ original commit in Linus's tree ------------------

Quoted 1 line

From 138e4ad67afd5c6c318b056b4d17c17f2c0ca5c0 Mon Sep 17 00:00:00 2001

From: Oleg Nesterov <[email protected]>
Date: Fri, 1 Sep 2017 18:55:33 +0200
Subject: [PATCH] epoll: fix race between ep_poll_callback(POLLFREE) and
ep_free()/ep_remove()
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The race was introduced by me in commit 971316f0503a ("epoll:
ep_unregister_pollwait() can use the freed pwq->whead"). I did not realize that nothing can protect eventpoll after ep_poll_callback() sets ->whead = NULL, only whead->lock can save us from the race with ep_free() or ep_remove().

Move ->whead = NULL to the end of ep_poll_callback() and add the necessary barriers.

TODO: cleanup the ewake/EPOLLEXCLUSIVE logic, it was confusing even before this patch.

Hopefully this explains use-after-free reported by syzcaller:

	BUG: KASAN: use-after-free in debug_spin_lock_before
	...
	 _raw_spin_lock_irqsave+0x4a/0x60 kernel/locking/spinlock.c:159
	 ep_poll_callback+0x29f/0xff0 fs/eventpoll.c:1148

this is spin_lock(eventpoll->lock),

	...
	Freed by task 17774:
	...
	 kfree+0xe8/0x2c0 mm/slub.c:3883
	 ep_free+0x22c/0x2a0 fs/eventpoll.c:865
Fixes 971316f0503a ("epoll: ep_unregister_pollwait() can use the freed pwq->whead")
Reported-by 范龙飞 <[email protected]>
Cc [email protected]
Signed-off-by Oleg Nesterov <[email protected]>
Signed-off-by Linus Torvalds <[email protected]>
fs/eventpoll.c +26 -16
index e767e4389cb1..adbe328b957c 100644
@@ -600,8 +600,13 @@ static void ep_remove_wait_queue(struct eppoll_entry *pwq)
 	wait_queue_head_t *whead;
 
 	rcu_read_lock();
-	/* If it is cleared by POLLFREE, it should be rcu-safe */
-	whead = rcu_dereference(pwq->whead);
+	/*
+	 * If it is cleared by POLLFREE, it should be rcu-safe.
+	 * If we read NULL we need a barrier paired with
+	 * smp_store_release() in ep_poll_callback(), otherwise
+	 * we rely on whead->lock.
+	 */
+	whead = smp_load_acquire(&pwq->whead);
 	if (whead)
 		remove_wait_queue(whead, &pwq->wait);
 	rcu_read_unlock();

@@ -1134,17 +1139,6 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v
 	struct eventpoll *ep = epi->ep;
 	int ewake = 0;
 
-	if ((unsigned long)key & POLLFREE) {
-		ep_pwq_from_wait(wait)->whead = NULL;
-		/*
-		 * whead = NULL above can race with ep_remove_wait_queue()
-		 * which can do another remove_wait_queue() after us, so we
-		 * can't use __remove_wait_queue(). whead->lock is held by
-		 * the caller.
-		 */
-		list_del_init(&wait->entry);
-	}
-
 	spin_lock_irqsave(&ep->lock, flags);
 
 	ep_set_busy_poll_napi_id(epi);

@@ -1228,10 +1222,26 @@ static int ep_poll_callback(wait_queue_entry_t *wait, unsigned mode, int sync, v
 	if (pwake)
 		ep_poll_safewake(&ep->poll_wait);
 
-	if (epi->event.events & EPOLLEXCLUSIVE)
-		return ewake;
+	if (!(epi->event.events & EPOLLEXCLUSIVE))
+		ewake = 1;
+
+	if ((unsigned long)key & POLLFREE) {
+		/*
+		 * If we race with ep_remove_wait_queue() it can miss
+		 * ->whead = NULL and do another remove_wait_queue() after
+		 * us, so we can't use __remove_wait_queue().
+		 */
+		list_del_init(&wait->entry);
+		/*
+		 * ->whead != NULL protects us from the race with ep_free()
+		 * or ep_remove(), ep_remove_wait_queue() takes whead->lock
+		 * held by the caller. Once we nullify it, nothing protects
+		 * ep/epi or even wait.
+		 */
+		smp_store_release(&ep_pwq_from_wait(wait)->whead, NULL);
+	}
 
-	return 1;
+	return ewake;
 }
 
 /*
#1 lore raw
Quoted 2 lines · On 09/04, [email protected] wrote:

The patch below does not apply to the 3.18-stable tree.

conflict with commit 2055da97389a605c8a00d163d40903afbe413921 "sched/wait: Disambiguate wq_entry->task_list and wq_head->task_list naming" and commit df0108c5da561c66c333bb46bfe3c1fc65905898
"epoll: add EPOLLEXCLUSIVE flag"

Quoted 10 lines

------------------ original commit in Linus's tree ------------------

From 138e4ad67afd5c6c318b056b4d17c17f2c0ca5c0 Mon Sep 17 00:00:00 2001
From: Oleg Nesterov <[email protected]>
Date: Fri, 1 Sep 2017 18:55:33 +0200
Subject: [PATCH] epoll: fix race between ep_poll_callback(POLLFREE) and
ep_free()/ep_remove()

From 4397c043a0dd0e93d4ed0db701cd97818b1ae3bb Mon Sep 17 00:00:00 2001

From: Oleg Nesterov <[email protected]>
Date: Fri, 1 Sep 2017 18:55:33 +0200
Subject: [PATCH] epoll: fix race between ep_poll_callback(POLLFREE) and
ep_free()/ep_remove()
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The race was introduced by me in commit 971316f0503a ("epoll:
ep_unregister_pollwait() can use the freed pwq->whead"). I did not realize that nothing can protect eventpoll after ep_poll_callback() sets ->whead = NULL, only whead->lock can save us from the race with ep_free() or ep_remove().

Move ->whead = NULL to the end of ep_poll_callback() and add the necessary barriers.

TODO: cleanup the ewake/EPOLLEXCLUSIVE logic, it was confusing even before this patch.

Hopefully this explains use-after-free reported by syzcaller:

	BUG: KASAN: use-after-free in debug_spin_lock_before
	...
	 _raw_spin_lock_irqsave+0x4a/0x60 kernel/locking/spinlock.c:159
	 ep_poll_callback+0x29f/0xff0 fs/eventpoll.c:1148

this is spin_lock(eventpoll->lock),

	...
	Freed by task 17774:
	...
	 kfree+0xe8/0x2c0 mm/slub.c:3883
	 ep_free+0x22c/0x2a0 fs/eventpoll.c:865
Fixes 971316f0503a ("epoll: ep_unregister_pollwait() can use the freed pwq->whead")
Reported-by 范龙飞 <[email protected]>
Cc [email protected]
Signed-off-by Oleg Nesterov <[email protected]>
Signed-off-by Linus Torvalds <[email protected]>
---
 fs/eventpoll.c | 37 ++++++++++++++++++++++++-------------
 1 file changed, 24 insertions(+), 13 deletions(-)
fs/eventpoll.c +24 -13
index 7bcfff9..2b5285c 100644
@@ -518,8 +518,13 @@ static void ep_remove_wait_queue(struct eppoll_entry *pwq)
 	wait_queue_head_t *whead;
 
 	rcu_read_lock();
-	/* If it is cleared by POLLFREE, it should be rcu-safe */
-	whead = rcu_dereference(pwq->whead);
+	/*
+	 * If it is cleared by POLLFREE, it should be rcu-safe.
+	 * If we read NULL we need a barrier paired with
+	 * smp_store_release() in ep_poll_callback(), otherwise
+	 * we rely on whead->lock.
+	 */
+	whead = smp_load_acquire(&pwq->whead);
 	if (whead)
 		remove_wait_queue(whead, &pwq->wait);
 	rcu_read_unlock();

@@ -1006,17 +1011,6 @@ static int ep_poll_callback(wait_queue_t *wait, unsigned mode, int sync, void *k
 	struct epitem *epi = ep_item_from_wait(wait);
 	struct eventpoll *ep = epi->ep;
 
-	if ((unsigned long)key & POLLFREE) {
-		ep_pwq_from_wait(wait)->whead = NULL;
-		/*
-		 * whead = NULL above can race with ep_remove_wait_queue()
-		 * which can do another remove_wait_queue() after us, so we
-		 * can't use __remove_wait_queue(). whead->lock is held by
-		 * the caller.
-		 */
-		list_del_init(&wait->task_list);
-	}
-
 	spin_lock_irqsave(&ep->lock, flags);
 
 	/*

@@ -1081,6 +1075,23 @@ out_unlock:
 	if (pwake)
 		ep_poll_safewake(&ep->poll_wait);
 
+
+	if ((unsigned long)key & POLLFREE) {
+		/*
+		 * If we race with ep_remove_wait_queue() it can miss
+		 * ->whead = NULL and do another remove_wait_queue() after
+		 * us, so we can't use __remove_wait_queue().
+		 */
+		list_del_init(&wait->task_list);
+		/*
+		 * ->whead != NULL protects us from the race with ep_free()
+		 * or ep_remove(), ep_remove_wait_queue() takes whead->lock
+		 * held by the caller. Once we nullify it, nothing protects
+		 * ep/epi or even wait.
+		 */
+		smp_store_release(&ep_pwq_from_wait(wait)->whead, NULL);
+	}
+
 	return 1;
 }
Signature · 1 line
2.5.0