Skip to contentSkip to conversation

mm: migrate: don't rely on __PageMovable() of newpage after unlocking it

PATCH Applied David Hildenbrand 3 participants 6 messages Jan 29, 2019 – Feb 1, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 mm: migrate: don't rely on __PageMovable() of newpage after unlocking it

We had a race in the old balloon compaction code before commit b1123ea6d3b3 ("mm: balloon: use general non-lru movable page feature") refactored it that became visible after backporting commit 195a8c43e93d ("virtio-balloon: deflate via a page list") without the refactoring.

The bug existed from commit d6d86c0a7f8d ("mm/balloon_compaction: redesign ballooned pages management") till commit b1123ea6d3b3 ("mm: balloon: use general non-lru movable page feature"). commit d6d86c0a7f8d ("mm/balloon_compaction: redesign ballooned pages management") was backported to 3.12, so the broken kernels are stable kernels [3.12 - 4.7].

There was a subtle race between dropping the page lock of the newpage in __unmap_and_move() and checking for __is_movable_balloon_page(newpage).

Just after dropping this page lock, virtio-balloon could go ahead and deflate the newpage, effectively dequeueing it and clearing PageBalloon, in turn making __is_movable_balloon_page(newpage) fail.

This resulted in dropping the reference of the newpage via putback_lru_page(newpage) instead of put_page(newpage), leading to page->lru getting modified and a !LRU page ending up in the LRU lists. With commit 195a8c43e93d ("virtio-balloon: deflate via a page list") backported, one would suddenly get corrupted lists in release_pages_balloon():
- WARNING: CPU: 13 PID: 6586 at lib/list_debug.c:59 __list_del_entry+0xa1/0xd0
- list_del corruption. prev->next should be ffffe253961090a0, but was dead000000000100

Nowadays this race is no longer possible, but it is hidden behind very ugly handling of __ClearPageMovable() and __PageMovable().

__ClearPageMovable() will not make __PageMovable() fail, only PageMovable(). So the new check (__PageMovable(newpage)) will still hold even after newpage was dequeued by virtio-balloon.

If anybody would ever change that special handling, the BUG would be introduced again. So instead, make it explicit and use the information of the original isolated page before migration.

This patch can be backported fairly easy to stable kernels (in contrast to the refactoring).

Cc Andrew Morton <[email protected]>
Cc Mel Gorman <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Naoya Horiguchi <[email protected]>
Cc Jan Kara <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Dominik Brodowski <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Vratislav Bendel <[email protected]>
Cc Rafael Aquini <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Minchan Kim <[email protected]>
Cc [email protected] # 3.12 - 4.7
Fixes d6d86c0a7f8d ("mm/balloon_compaction: redesign ballooned pages management")
---
 mm/migrate.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
mm/migrate.c +5 -2
index 4512afab46ac..402198816d1a 100644
@@ -1135,10 +1135,13 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
 	 * If migration is successful, decrease refcount of the newpage
 	 * which will not free the page because new page owner increased
 	 * refcounter. As well, if it is LRU page, add the page to LRU
-	 * list in here.
+	 * list in here. Use the old state of the isolated source page to
+	 * determine if we migrated a LRU page. newpage was already unlocked
+	 * and possibly modified by its owner - don't rely on the page
+	 * state.
 	 */
 	if (rc == MIGRATEPAGE_SUCCESS) {
-		if (unlikely(__PageMovable(newpage)))
+		if (unlikely(!is_lru))
 			put_page(newpage);
 		else
 			putback_lru_page(newpage);
Signature · 1 line
2.17.2
maintainer
#1 lore raw

Hi,

[This is an automated email]

This commit has been processed because it contains a "Fixes:" tag,
fixing commit: d6d86c0a7f8d mm/balloon_compaction: redesign ballooned pages management.

The bot has tested the following trees: v4.20.5, v4.19.18, v4.14.96, v4.9.153, v4.4.172, v3.18.133.

v4.20.5: Build OK!
v4.19.18: Build OK!
v4.14.96: Build OK!
v4.9.153: Build OK!
v4.4.172: Failed to apply! Possible dependencies:
1031bc589228 ("lib/vsprintf: add %*pg format specifier") 14e0a214d62d ("tools, perf: make gfp_compact_table up to date") 1f7866b4aebd ("mm, tracing: make show_gfp_flags() up to date") 420adbe9fc1a ("mm, tracing: unify mm flags handling in tracepoints and printk") 53f9263baba6 ("mm: rework mapcount accounting to enable 4k mapping of THPs") 7cd12b4abfd2 ("mm, page_owner: track and print last migrate reason") 7d2eba0557c1 ("mm: add tracepoint for scanning pages") c6c919eb90e0 ("mm: use put_page() to free page instead of putback_lru_page()") d435edca9288 ("mm, page_owner: copy page owner info during migration") d8c1bdeb5d6b ("page-flags: trivial cleanup for PageTrans* helpers") eca56ff906bd ("mm, shmem: add internal shmem resident memory accounting") edf14cdbf9a0 ("mm, printk: introduce new format string for flags")

v3.18.133: Failed to apply! Possible dependencies:
2847cf95c68f ("mm/debug-pagealloc: cleanup page guard code") 48c96a368579 ("mm/page_owner: keep track of page owners") 7cd12b4abfd2 ("mm, page_owner: track and print last migrate reason") 94f759d62b2c ("mm/page_owner.c: remove unnecessary stack_trace field") c6c919eb90e0 ("mm: use put_page() to free page instead of putback_lru_page()") d435edca9288 ("mm, page_owner: copy page owner info during migration") e2cfc91120fa ("mm/page_owner: set correct gfp_mask on page_owner") e30825f1869a ("mm/debug-pagealloc: prepare boottime configurable on/off") eefa864b701d ("mm/page_ext: resurrect struct page extending code for debugging")

How should we proceed with this patch?

--
Thanks,
Sasha

#2 lore raw

PATCH v2 1/1 mm: migrate: don't rely on __PageMovable() of newpage after unlocking it

This is the backport for 4.4-stable.

We had a race in the old balloon compaction code before commit b1123ea6d3b3 ("mm: balloon: use general non-lru movable page feature") refactored it that became visible after backporting commit 195a8c43e93d ("virtio-balloon: deflate via a page list") without the refactoring.

The bug existed from commit d6d86c0a7f8d ("mm/balloon_compaction: redesign ballooned pages management") till commit b1123ea6d3b3 ("mm: balloon: use general non-lru movable page feature"). commit d6d86c0a7f8d ("mm/balloon_compaction: redesign ballooned pages management") was backported to 3.12, so the broken kernels are stable kernels [3.12 - 4.7].

There was a subtle race between dropping the page lock of the newpage in __unmap_and_move() and checking for __is_movable_balloon_page(newpage).

Just after dropping this page lock, virtio-balloon could go ahead and deflate the newpage, effectively dequeueing it and clearing PageBalloon, in turn making __is_movable_balloon_page(newpage) fail.

This resulted in dropping the reference of the newpage via putback_lru_page(newpage) instead of put_page(newpage), leading to page->lru getting modified and a !LRU page ending up in the LRU lists. With commit 195a8c43e93d ("virtio-balloon: deflate via a page list") backported, one would suddenly get corrupted lists in release_pages_balloon():
- WARNING: CPU: 13 PID: 6586 at lib/list_debug.c:59 __list_del_entry+0xa1/0xd0
- list_del corruption. prev->next should be ffffe253961090a0, but was dead000000000100

Nowadays this race is no longer possible, but it is hidden behind very ugly handling of __ClearPageMovable() and __PageMovable().

__ClearPageMovable() will not make __PageMovable() fail, only PageMovable(). So the new check (__PageMovable(newpage)) will still hold even after newpage was dequeued by virtio-balloon.

If anybody would ever change that special handling, the BUG would be introduced again. So instead, make it explicit and use the information of the original isolated page before migration.

This patch can be backported fairly easy to stable kernels (in contrast to the refactoring).

Cc Andrew Morton <[email protected]>
Cc Mel Gorman <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Naoya Horiguchi <[email protected]>
Cc Jan Kara <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Dominik Brodowski <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Vratislav Bendel <[email protected]>
Cc Rafael Aquini <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Sasha Levin <[email protected]>
Cc [email protected] # 3.12 - 4.7
Fixes d6d86c0a7f8d ("mm/balloon_compaction: redesign ballooned pages management")
---
 mm/migrate.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
mm/migrate.c +6 -1
index afedcfab60e2..3304c98f9a78 100644
@@ -936,6 +936,7 @@ static ICE_noinline int unmap_and_move(new_page_t get_new_page,
 	int rc = MIGRATEPAGE_SUCCESS;
 	int *result = NULL;
 	struct page *newpage;
+	bool is_lru = !isolated_balloon_page(page);
 
 	newpage = get_new_page(page, private, &result);
 	if (!newpage)
@@ -984,10 +985,14 @@ out:
 	 * If migration was not successful and there's a freeing callback, use
 	 * it.  Otherwise, putback_lru_page() will drop the reference grabbed
 	 * during isolation.
+	 *
+	 * Use the old state of the isolated source page to determine if we
+	 * migrated a LRU page. newpage was already unlocked and possibly
+	 * modified by its owner - don't rely on the page state.
 	 */
 	if (put_new_page)
 		put_new_page(newpage, private);
-	else if (unlikely(__is_movable_balloon_page(newpage))) {
+	else if (unlikely(!is_lru)) {
 		/* drop our reference, page already in the balloon */
 		put_page(newpage);
 	} else
Signature · 1 line
2.17.2
#3 lore raw
Quoted 64 lines · David Hildenbrand

This is the backport for 4.4-stable.

We had a race in the old balloon compaction code before commit b1123ea6d3b3 ("mm: balloon: use general non-lru movable page feature") refactored it that became visible after backporting commit 195a8c43e93d ("virtio-balloon: deflate via a page list") without the refactoring.

The bug existed from commit d6d86c0a7f8d ("mm/balloon_compaction: redesign ballooned pages management") till commit b1123ea6d3b3 ("mm: balloon: use general non-lru movable page feature"). commit d6d86c0a7f8d ("mm/balloon_compaction: redesign ballooned pages management") was backported to 3.12, so the broken kernels are stable kernels [3.12 - 4.7].

There was a subtle race between dropping the page lock of the newpage in __unmap_and_move() and checking for __is_movable_balloon_page(newpage).

Just after dropping this page lock, virtio-balloon could go ahead and deflate the newpage, effectively dequeueing it and clearing PageBalloon, in turn making __is_movable_balloon_page(newpage) fail.

This resulted in dropping the reference of the newpage via putback_lru_page(newpage) instead of put_page(newpage), leading to page->lru getting modified and a !LRU page ending up in the LRU lists. With commit 195a8c43e93d ("virtio-balloon: deflate via a page list") backported, one would suddenly get corrupted lists in release_pages_balloon():
- WARNING: CPU: 13 PID: 6586 at lib/list_debug.c:59 __list_del_entry+0xa1/0xd0
- list_del corruption. prev->next should be ffffe253961090a0, but was dead000000000100

Nowadays this race is no longer possible, but it is hidden behind very ugly handling of __ClearPageMovable() and __PageMovable().

__ClearPageMovable() will not make __PageMovable() fail, only PageMovable(). So the new check (__PageMovable(newpage)) will still hold even after newpage was dequeued by virtio-balloon.

If anybody would ever change that special handling, the BUG would be introduced again. So instead, make it explicit and use the information of the original isolated page before migration.

This patch can be backported fairly easy to stable kernels (in contrast to the refactoring).

Cc Andrew Morton <[email protected]>
Cc Mel Gorman <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Naoya Horiguchi <[email protected]>
Cc Jan Kara <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Dominik Brodowski <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Vratislav Bendel <[email protected]>
Cc Rafael Aquini <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Sasha Levin <[email protected]>
Cc [email protected] # 3.12 - 4.7
Fixes d6d86c0a7f8d ("mm/balloon_compaction: redesign ballooned pages management")
Quoted 3 lines
---
 mm/migrate.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

What is the git commit id of this patch in Linus's tree?

thanks,

greg k-h

#4 lore raw
Quoted 69 lines · Greg KH
Quoted 64 lines · David Hildenbrand

This is the backport for 4.4-stable.

We had a race in the old balloon compaction code before commit b1123ea6d3b3 ("mm: balloon: use general non-lru movable page feature") refactored it that became visible after backporting commit 195a8c43e93d ("virtio-balloon: deflate via a page list") without the refactoring.

The bug existed from commit d6d86c0a7f8d ("mm/balloon_compaction: redesign ballooned pages management") till commit b1123ea6d3b3 ("mm: balloon: use general non-lru movable page feature"). commit d6d86c0a7f8d ("mm/balloon_compaction: redesign ballooned pages management") was backported to 3.12, so the broken kernels are stable kernels [3.12 - 4.7].

There was a subtle race between dropping the page lock of the newpage in __unmap_and_move() and checking for __is_movable_balloon_page(newpage).

Just after dropping this page lock, virtio-balloon could go ahead and deflate the newpage, effectively dequeueing it and clearing PageBalloon, in turn making __is_movable_balloon_page(newpage) fail.

This resulted in dropping the reference of the newpage via putback_lru_page(newpage) instead of put_page(newpage), leading to page->lru getting modified and a !LRU page ending up in the LRU lists. With commit 195a8c43e93d ("virtio-balloon: deflate via a page list") backported, one would suddenly get corrupted lists in release_pages_balloon():
- WARNING: CPU: 13 PID: 6586 at lib/list_debug.c:59 __list_del_entry+0xa1/0xd0
- list_del corruption. prev->next should be ffffe253961090a0, but was dead000000000100

Nowadays this race is no longer possible, but it is hidden behind very ugly handling of __ClearPageMovable() and __PageMovable().

__ClearPageMovable() will not make __PageMovable() fail, only PageMovable(). So the new check (__PageMovable(newpage)) will still hold even after newpage was dequeued by virtio-balloon.

If anybody would ever change that special handling, the BUG would be introduced again. So instead, make it explicit and use the information of the original isolated page before migration.

This patch can be backported fairly easy to stable kernels (in contrast to the refactoring).

Cc Andrew Morton <[email protected]>
Cc Mel Gorman <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Naoya Horiguchi <[email protected]>
Cc Jan Kara <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Dominik Brodowski <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Vratislav Bendel <[email protected]>
Cc Rafael Aquini <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Sasha Levin <[email protected]>
Cc [email protected] # 3.12 - 4.7
Fixes d6d86c0a7f8d ("mm/balloon_compaction: redesign ballooned pages management")
Quoted 3 lines
---
 mm/migrate.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
Quoted 1 line

What is the git commit id of this patch in Linus's tree?

It's still in Andrew's tree as far as I know. I just wanted to share the backport right away (to show that it is easy :) ). Will resend it again (with proper commit idea) once upstream.

Quoted 5 lines

thanks,

greg k-h

Signature · 3 lines
Thanks,

David / dhildenb
#5 lore raw
In reply to David Hildenbrand
Quoted 82 lines · David Hildenbrand

This is the backport for 4.4-stable.

We had a race in the old balloon compaction code before commit b1123ea6d3b3 ("mm: balloon: use general non-lru movable page feature") refactored it that became visible after backporting commit 195a8c43e93d ("virtio-balloon: deflate via a page list") without the refactoring.

The bug existed from commit d6d86c0a7f8d ("mm/balloon_compaction: redesign ballooned pages management") till commit b1123ea6d3b3 ("mm: balloon: use general non-lru movable page feature"). commit d6d86c0a7f8d ("mm/balloon_compaction: redesign ballooned pages management") was backported to 3.12, so the broken kernels are stable kernels [3.12 - 4.7].

There was a subtle race between dropping the page lock of the newpage in __unmap_and_move() and checking for __is_movable_balloon_page(newpage).

Just after dropping this page lock, virtio-balloon could go ahead and deflate the newpage, effectively dequeueing it and clearing PageBalloon, in turn making __is_movable_balloon_page(newpage) fail.

This resulted in dropping the reference of the newpage via putback_lru_page(newpage) instead of put_page(newpage), leading to page->lru getting modified and a !LRU page ending up in the LRU lists. With commit 195a8c43e93d ("virtio-balloon: deflate via a page list") backported, one would suddenly get corrupted lists in release_pages_balloon():
- WARNING: CPU: 13 PID: 6586 at lib/list_debug.c:59 __list_del_entry+0xa1/0xd0
- list_del corruption. prev->next should be ffffe253961090a0, but was dead000000000100

Nowadays this race is no longer possible, but it is hidden behind very ugly handling of __ClearPageMovable() and __PageMovable().

__ClearPageMovable() will not make __PageMovable() fail, only PageMovable(). So the new check (__PageMovable(newpage)) will still hold even after newpage was dequeued by virtio-balloon.

If anybody would ever change that special handling, the BUG would be introduced again. So instead, make it explicit and use the information of the original isolated page before migration.

This patch can be backported fairly easy to stable kernels (in contrast to the refactoring).

Cc Andrew Morton <[email protected]>
Cc Mel Gorman <[email protected]>
Cc "Kirill A. Shutemov" <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Naoya Horiguchi <[email protected]>
Cc Jan Kara <[email protected]>
Cc Andrea Arcangeli <[email protected]>
Cc Dominik Brodowski <[email protected]>
Cc Matthew Wilcox <[email protected]>
Cc Vratislav Bendel <[email protected]>
Cc Rafael Aquini <[email protected]>
Cc Konstantin Khlebnikov <[email protected]>
Cc Minchan Kim <[email protected]>
Cc Sasha Levin <[email protected]>
Cc [email protected] # 3.12 - 4.7
Fixes d6d86c0a7f8d ("mm/balloon_compaction: redesign ballooned pages management")
---
 mm/migrate.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
mm/migrate.c +1 -0
index afedcfab60e2..3304c98f9a78 100644
@@ -936,6 +936,7 @@ static ICE_noinline int unmap_and_move(new_page_t get_new_page,
 	int rc = MIGRATEPAGE_SUCCESS;
 	int *result = NULL;
 	struct page *newpage;
+	bool is_lru = !isolated_balloon_page(page);
 
 	newpage = get_new_page(page, private, &result);
 	if (!newpage)
@@ -984,10 +985,14 @@ out:
 	 * If migration was not successful and there's a freeing callback, use

mm/migrate.c out: · line 990

Quoted 10 lines
mm/migrate.c +4 -1
 	 * it.  Otherwise, putback_lru_page() will drop the reference grabbed
 	 * during isolation.
+	 *
+	 * Use the old state of the isolated source page to determine if we
+	 * migrated a LRU page. newpage was already unlocked and possibly
+	 * modified by its owner - don't rely on the page state.
 	 */
 	if (put_new_page)
 		put_new_page(newpage, private);
-	else if (unlikely(__is_movable_balloon_page(newpage))) {

And to be save, we should turn this into

else if (rc == MIGRATEPAGE_SUCCESS && unlikely(!is_lru)) {

But will resend this either way as already mentioned to Greg.

Quoted 5 lines
mm/migrate.c +1 -0
+	else if (unlikely(!is_lru)) {
 		/* drop our reference, page already in the balloon */
 		put_page(newpage);
 	} else
Signature · 3 lines
Thanks,

David / dhildenb