hwpoison: fix race with compound page allocation
Subsystems (2)
Naoya Horiguchi
reviewerv4 cover hwpoison: fix race with compound page allocation
Hi,
I updated the patchset based on previous discussion [1],
I fixed wrong patch separation, remaining race issue, and build failure.
If you still find some issue in this series, please let me know.
Thanks,
Naoya Horiguchi
[1] https://lore.kernel.org/linux-mm/[email protected]/T/#u
---
Summary:
Naoya Horiguchi (2):
mm,hwpoison: fix race with compound page allocation mm,hwpoison: make get_hwpoison_page call get_any_page()
mm/memory-failure.c | 122 +++++++++++++++++++++++++++++++--------------------- 1 file changed, 72 insertions(+), 50 deletions(-)
Naoya Horiguchi
reviewerPATCH v4 1/2 mm,hwpoison: fix race with compound page allocation
From: Naoya Horiguchi <[email protected]>
When hugetlb page fault (under overcommitting situation) and memory_failure() race, VM_BUG_ON_PAGE() is triggered by the following race:
CPU0: CPU1:
gather_surplus_pages()
page = alloc_surplus_huge_page()
memory_failure_hugetlb()
get_hwpoison_page(page)
__get_hwpoison_page(page)
get_page_unless_zero(page)
zero = put_page_testzero(page) VM_BUG_ON_PAGE(!zero, page) enqueue_huge_page(h, page)
put_page(page)
__get_hwpoison_page() only checks page refcount before taking additional one for memory error handling, which is wrong because there's a time window where compound pages have non-zero refcount during initialization.
So makes __get_hwpoison_page() check page status a bit more for a few types of compound pages. PageSlab() check is added because otherwise "non anonymous thp" path is wrongly chosen.
---
ChangeLog v4:
- all hugetlb related check in hugetlb_lock,
- fix build error with #ifdef CONFIG_HUGETLB_PAGE
--- mm/memory-failure.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
mm/memory-failure.c +10 -0
@@ -1094,6 +1094,16 @@ static int page_action(struct page_state *ps, struct page *p,static int __get_hwpoison_page (struct page *page) {struct page *head =compound_head (page); +int ret =0 ; + +#ifdef CONFIG_HUGETLB_PAGE +spin_lock (&hugetlb_lock); +if (PageHuge (head) && (HPageFreed (head) ||HPageMigratable (head))) + ret =get_page_unless_zero (head); +spin_unlock (&hugetlb_lock); +if (ret >0 ) +return ret; +#endif if (!PageHuge (head) &&PageTransHuge (head)) {/*
Signature · 1 line
2.25.1
Re: [PATCH v4 1/2] mm,hwpoison: fix race with compound page allocation
Quoted 18 lines · Naoya Horiguchi
From: Naoya Horiguchi <[email protected]>
When hugetlb page fault (under overcommitting situation) and
memory_failure()race,VM_BUG_ON_PAGE()is triggered by the following race:CPU0: CPU1:
gather_surplus_pages()
page =alloc_surplus_huge_page()
memory_failure_hugetlb()
get_hwpoison_page(page)
__get_hwpoison_page(page)
get_page_unless_zero(page)
zero =put_page_testzero(page)VM_BUG_ON_PAGE(!zero, page)enqueue_huge_page(h, page)
put_page(page)
Quoted 1 line
__get_hwpoison_page()only checks page refcount before taking additional
^^ the? ^^ an
Quoted 4 lines
one for memory error handling, which is wrong because there's a time window where compound pages have non-zero refcount during initialization.
So makes
__get_hwpoison_page()check page status a bit more for a few
^^ make
Quoted 2 lines
types of compound pages.
PageSlab()check is added because otherwise "non anonymous thp" path is wrongly chosen.
This is no longer true with this patch, is it? What happened here?
mm/memory-failure.c static int page_action(struct page_state *ps, struct page *p, · line 1106
Quoted 13 lines
static int
__get_hwpoison_page(struct page *page){mm/memory-failure.c +10
-0struct page *head =compound_head (page); +int ret =0 ; + +#ifdef CONFIG_HUGETLB_PAGE +spin_lock (&hugetlb_lock); +if (PageHuge (head) && (HPageFreed (head) ||HPageMigratable (head))) + ret =get_page_unless_zero (head); +spin_unlock (&hugetlb_lock); +if (ret >0 ) +return ret; +#endif
I am kind of fine with this, but I wonder whether it makes sense to hide this details into helper (with an empty stub for non-hugetlb pages)?
Quoted 1 line
if (!PageHuge(head) && PageTransHuge(head)) {
This !PageHuge could go?
Signature · 2 lines
Oscar Salvador SUSE L3
Naoya Horiguchi
reviewerRe: [PATCH v4 1/2] mm,hwpoison: fix race with compound page allocation
Quoted 24 lines · Oscar Salvador
Quoted 18 lines · Naoya Horiguchi
From: Naoya Horiguchi <[email protected]>
When hugetlb page fault (under overcommitting situation) and
memory_failure()race,VM_BUG_ON_PAGE()is triggered by the following race:CPU0: CPU1:
gather_surplus_pages()
page =alloc_surplus_huge_page()
memory_failure_hugetlb()
get_hwpoison_page(page)
__get_hwpoison_page(page)
get_page_unless_zero(page)
zero =put_page_testzero(page)VM_BUG_ON_PAGE(!zero, page)enqueue_huge_page(h, page)
put_page(page)Quoted 1 line
__get_hwpoison_page()only checks page refcount before taking additional^^ the? ^^ an
Quoted 3 lines
one for memory error handling, which is wrong because there's a time window where compound pages have non-zero refcount during initialization.
Quoted 6 lines
Quoted 1 line
So makes
__get_hwpoison_page()check page status a bit more for a few^^ makeQuoted 2 lines
types of compound pages.
PageSlab()check is added because otherwise "non anonymous thp" path is wrongly chosen.This is no longer true with this patch, is it? What happened here?
Sorry, we need remove this. I dropped the changes for PageSlab because I'm still not sure about what to do (I'd like to do it in a separate work).
Quoted 17 lines
Quoted 4 lines
static int
__get_hwpoison_page(struct page *page){mm/memory-failure.c +0
-0struct page *head =compound_head (page);mm/memory-failure.c static int page_action(struct page_state *ps, struct page *p, · line 1106
Quoted 10 lines
mm/memory-failure.c +10
-0+int ret =0 ; + +#ifdef CONFIG_HUGETLB_PAGE +spin_lock (&hugetlb_lock); +if (PageHuge (head) && (HPageFreed (head) ||HPageMigratable (head))) + ret =get_page_unless_zero (head); +spin_unlock (&hugetlb_lock); +if (ret >0 ) +return ret; +#endif I am kind of fine with this, but I wonder whether it makes sense to hide this details into helper (with an empty stub for non-hugetlb pages)?
OK, I will do this.
Quoted 3 lines
Quoted 2 lines
if (!PageHuge(head) && PageTransHuge(head)) {
This !PageHuge could go?
I'll update this, too.
Thanks,
Naoya Horiguchi
Re: [PATCH v4 1/2] mm,hwpoison: fix race with compound page allocation
In reply to Naoya HoriguchiQuoted 46 lines · Naoya Horiguchi
From: Naoya Horiguchi <[email protected]>
When hugetlb page fault (under overcommitting situation) and
memory_failure()race,VM_BUG_ON_PAGE()is triggered by the following race:CPU0: CPU1:
gather_surplus_pages()
page =alloc_surplus_huge_page()
memory_failure_hugetlb()
get_hwpoison_page(page)
__get_hwpoison_page(page)
get_page_unless_zero(page)
zero =put_page_testzero(page)VM_BUG_ON_PAGE(!zero, page)enqueue_huge_page(h, page)
put_page(page)
__get_hwpoison_page()only checks page refcount before taking additional one for memory error handling, which is wrong because there's a time window where compound pages have non-zero refcount during initialization.So makes
__get_hwpoison_page()check page status a bit more for a few types of compound pages.PageSlab()check is added because otherwise "non anonymous thp" path is wrongly chosen.Fixes ead07f6a867b ("mm/memory-failure: introduce get_hwpoison_page() for consistent refcount handling")Signed-off-by Naoya Horiguchi <[email protected]>Reported-by Muchun Song <[email protected]>Cc [email protected] # 5.12+---
ChangeLog v4:
- all hugetlb related check inhugetlb_lock,
- fix build error with #ifdefCONFIG_HUGETLB_PAGE--- mm/memory-failure.c | 10 ++++++++++ 1 file changed, 10 insertions(+)mm/memory-failure.c +0
-0@@ -1094,6 +1094,16 @@ static int page_action(struct page_state *ps, struct page *p,static int __get_hwpoison_page (struct page *page) {struct page *head =compound_head (page);
mm/memory-failure.c static int page_action(struct page_state *ps, struct page *p, · line 1106
Quoted 10 lines
mm/memory-failure.c +10
-0+int ret =0 ; + +#ifdef CONFIG_HUGETLB_PAGE +spin_lock (&hugetlb_lock); +if (PageHuge (head) && (HPageFreed (head) ||HPageMigratable (head))) + ret =get_page_unless_zero (head); +spin_unlock (&hugetlb_lock); +if (ret >0 ) +return ret; +#endif
I must be missing something.
The above code makes sure the page is not in one of these transitive hugetlb states as mentioned in the commit message. It only attempts to take a reference on the page if it is not in one of these states.
However, if it is in such a transitive state (!HPageFreed(head) && !HPageMigratable(head)) we will fall through and execute the code:
if (get_page_unless_zero(head)) {
if (head == compound_head(page))
return 1;So, it seems like we will always do a get_page_unless_zero() for PageHuge() pages?
Also, due to commit db71ef79b59b ("hugetlb: make free_huge_page irq safe") you need to make sure interrupts are disabled when taking hugetlb_lock.
Signature · 1 line
Mike Kravetz
Re: [PATCH v4 1/2] mm,hwpoison: fix race with compound page allocation
Quoted 67 lines · Mike Kravetz
Quoted 46 lines · Naoya Horiguchi
From: Naoya Horiguchi <[email protected]>
When hugetlb page fault (under overcommitting situation) and
memory_failure()race,VM_BUG_ON_PAGE()is triggered by the following race:CPU0: CPU1:
gather_surplus_pages()
page =alloc_surplus_huge_page()
memory_failure_hugetlb()
get_hwpoison_page(page)
__get_hwpoison_page(page)
get_page_unless_zero(page)
zero =put_page_testzero(page)VM_BUG_ON_PAGE(!zero, page)enqueue_huge_page(h, page)
put_page(page)
__get_hwpoison_page()only checks page refcount before taking additional one for memory error handling, which is wrong because there's a time window where compound pages have non-zero refcount during initialization.So makes
__get_hwpoison_page()check page status a bit more for a few types of compound pages.PageSlab()check is added because otherwise "non anonymous thp" path is wrongly chosen.Fixes ead07f6a867b ("mm/memory-failure: introduce get_hwpoison_page() for consistent refcount handling")Signed-off-by Naoya Horiguchi <[email protected]>Reported-by Muchun Song <[email protected]>Cc [email protected] # 5.12+---
ChangeLog v4:
- all hugetlb related check inhugetlb_lock,
- fix build error with #ifdefCONFIG_HUGETLB_PAGE--- mm/memory-failure.c | 10 ++++++++++ 1 file changed, 10 insertions(+)mm/memory-failure.c +0
-0@@ -1094,6 +1094,16 @@ static int page_action(struct page_state *ps, struct page *p,static int __get_hwpoison_page (struct page *page) {struct page *head =compound_head (page);mm/memory-failure.c static int page_action(struct page_state *ps, struct page *p, · line 1106
Quoted 10 lines
mm/memory-failure.c +10
-0+int ret =0 ; + +#ifdef CONFIG_HUGETLB_PAGE +spin_lock (&hugetlb_lock); +if (PageHuge (head) && (HPageFreed (head) ||HPageMigratable (head))) + ret =get_page_unless_zero (head); +spin_unlock (&hugetlb_lock); +if (ret >0 ) +return ret; +#endif I must be missing something.
The above code makes sure the page is not in one of these transitive hugetlb states as mentioned in the commit message. It only attempts to take a reference on the page if it is not in one of these states.
However, if it is in such a transitive state (!HPageFreed(head) && !HPageMigratable(head)) we will fall through and execute the code:
Quoted 6 lines
if (get_page_unless_zero(head)) { if (head == compound_head(page)) return 1;So, it seems like we will always do a
get_page_unless_zero()forPageHuge()pages?
Right, no need to fall through in such a case.
Quoted 4 lines
Also, due to commit
db71ef79b59b("hugetlb: makefree_huge_pageirq safe") you need to make sure interrupts are disabled when takinghugetlb_lock.
Thanks, I'll rebase to latest mainline and comply with new semantics.
- Naoya
Naoya Horiguchi
reviewerPATCH v4 2/2 mm,hwpoison: make get_hwpoison_page call get_any_page()
In reply to Naoya HoriguchiFrom: Naoya Horiguchi <[email protected]>
__get_hwpoison_page() could fail to grab refcount by some race condition, so it's helpful if we can handle it by retrying. We already have retry logic, so make get_hwpoison_page() call get_any_page() when called from memory_failure().
---
ChangeLog v4:
- __get_hwpoison_page() returns -EBUSY (not in the previous patch),
- moved "if (ret == -EBUSY)" block,
- improved comment.
--- mm/memory-failure.c | 114 ++++++++++++++++++++++++-------------------- 1 file changed, 63 insertions(+), 51 deletions(-)
mm/memory-failure.c +63 -51
@@ -1084,13 +1084,6 @@ static int page_action(struct page_state *ps, struct page *p,return (result ==MF_RECOVERED || result ==MF_DELAYED ) ?0 : -EBUSY ; } -/** -* __get_hwpoison_page() - Get refcount for memory error handling: -* @page: raw error page (hit by memory error) -* -* Return: return 0 if failed to grab the refcount, otherwise true (some -* non-zero value.) -*/ static int __get_hwpoison_page (struct page *page) {struct page *head =compound_head (page);
@@ -1098,7 +1091,9 @@ static int __get_hwpoison_page(struct page *page)#ifdef CONFIG_HUGETLB_PAGE spin_lock (&hugetlb_lock); -if (PageHuge (head) && (HPageFreed (head) ||HPageMigratable (head))) +if (!PageHuge (head)) + ret = -EBUSY ; +else if (HPageFreed (head) ||HPageMigratable (head)) ret =get_page_unless_zero (head);spin_unlock (&hugetlb_lock);if (ret >0 )
@@ -1131,15 +1126,6 @@ static int __get_hwpoison_page(struct page *page)return 0 ; } -/* -* Safely get reference count of an arbitrary page. -* -* Returns 0 for a free page, 1 for an in-use page, -* -EIO for a page-type we cannot handle and -EBUSY if we raced with an -* allocation. -* We only incremented refcount in case the page was already in-use and it -* is a known type we can handle. -*/ static int get_any_page (struct page *p,unsigned long flags) {int ret =0 , pass =0 ;
@@ -1149,50 +1135,76 @@ static int get_any_page(struct page *p, unsigned long flags) count_increased = true; try_again: -if (!count_increased && !__get_hwpoison_page (p)) { -if (page_count (p)) { -/* We raced with an allocation, retry. */ -if (pass++ <3 ) - goto try_again; - ret = -EBUSY ; - }else if (!PageHuge (p) && !is_free_buddy_page (p)) { -/* We raced with put_page, retry. */ +if (!count_increased) { + ret =__get_hwpoison_page (p); +if (!ret) { +if (page_count (p)) { +/* We raced with an allocation, retry. */ +if (pass++ <3 ) + goto try_again; + ret = -EBUSY ; + }else if (!PageHuge (p) && !is_free_buddy_page (p)) { +/* We raced with put_page, retry. */ +if (pass++ <3 ) + goto try_again; + ret = -EIO ; + } + goto out; + }else if (ret == -EBUSY ) { +/* We raced with freeing huge page to buddy, retry. */ if (pass++ <3 ) goto try_again; - ret = -EIO ; } + } + +if (PageHuge (p) ||PageLRU (p) ||__PageMovable (p)) { + ret =1 ; }else { -if (PageHuge (p) ||PageLRU (p) ||__PageMovable (p)) { - ret =1 ; - }else { -/* -* A page we cannot handle. Check whether we can turn -* it into something we can handle. -*/ -if (pass++ <3 ) { -put_page (p); -shake_page (p,1 ); - count_increased = false; - goto try_again; - } +/* +* A page we cannot handle. Check whether we can turn +* it into something we can handle. +*/ +if (pass++ <3 ) {put_page (p); - ret = -EIO ; +shake_page (p,1 ); + count_increased = false; + goto try_again; } +put_page (p); + ret = -EIO ; } - +out:return ret; } -static int get_hwpoison_page (struct page *p,unsigned long flags, -enum mf_flags ctxt) +/** +* get_hwpoison_page() - Get refcount for memory error handling +* @p: Raw error page (hit by memory error) +* @flags: Flags controlling behavior of error handling +* +* get_hwpoison_page() takes a page refcount of an error page to handle memory +* error on it, after checking that the error page is in a well-defined state +* (defined as a page-type we can successfully handle the memor error on it, +* such as LRU page and hugetlb page). +* +* Memory error handling could be triggered at any time on any type of page, +* so it's prone to race with typical memory management lifecycle (like +* allocation and free). So to avoid such races, get_hwpoison_page() takes +* extra care for the error page's state (as done in __get_hwpoison_page()), +* and has some retry logic in get_any_page(). +* +* Return: 0 on failure, +* 1 on success for in-use pages in a well-defined state, +* -EIO for pages on which we can not handle memory errors, +* -EBUSY when get_hwpoison_page() has raced with page lifecycle +* operations like allocation and free. +*/ +static int get_hwpoison_page (struct page *p,unsigned long flags) {int ret;zone_pcp_disable (page_zone (p)); -if (ctxt ==MF_SOFT_OFFLINE ) - ret =get_any_page (p, flags); -else - ret =__get_hwpoison_page (p); + ret =get_any_page (p, flags);zone_pcp_enable (page_zone (p));return ret;
@@ -1381,7 +1393,7 @@ static int memory_failure_hugetlb(unsigned long pfn, int flags)num_poisoned_pages_inc (); -if (!(flags &MF_COUNT_INCREASED ) && !get_hwpoison_page (p, flags,0 )) { +if (!(flags &MF_COUNT_INCREASED ) && !get_hwpoison_page (p, flags)) {/* * Check "filter hit" and "race with other subpage." */
@@ -1605,7 +1617,7 @@ int memory_failure(unsigned long pfn, int flags)* In fact it's dangerous to directly bump up page count from 0, * that may make page_ref_freeze()/page_ref_unfreeze() mismatch. */ -if (!(flags &MF_COUNT_INCREASED ) && !get_hwpoison_page (p, flags,0 )) { +if (!(flags &MF_COUNT_INCREASED ) && !get_hwpoison_page (p, flags)) {if (is_free_buddy_page (p)) {if (take_page_off_buddy (p)) {page_ref_inc (p);
@@ -1897,7 +1909,7 @@ int unpoison_memory(unsigned long pfn)return 0 ; } -if (!get_hwpoison_page (p, flags,0 )) { +if (!get_hwpoison_page (p, flags)) {if (TestClearPageHWPoison (p))num_poisoned_pages_dec (); unpoison_pr_info("Unpoison: Software-unpoisoned free page %#lx\n",
@@ -2113,7 +2125,7 @@ int soft_offline_page(unsigned long pfn, int flags) retry:get_online_mems (); - ret =get_hwpoison_page (page, flags,MF_SOFT_OFFLINE ); + ret =get_hwpoison_page (page, flags);put_online_mems ();if (ret >0 ) {
Signature · 1 line
2.25.1
Re: [PATCH v4 2/2] mm,hwpoison: make get_hwpoison_page call get_any_page()
mm/memory-failure.c static int __get_hwpoison_page(struct page *page) · line 1095
Quoted 10 lines · Naoya Horiguchi
static int
__get_hwpoison_page(struct page *page){mm/memory-failure.c +2
-1struct page *head =compound_head (page);@@ -1098,7 +1091,9 @@ static int __get_hwpoison_page(struct page *page)#ifdef CONFIG_HUGETLB_PAGE spin_lock (&hugetlb_lock); -if (PageHuge (head) && (HPageFreed (head) ||HPageMigratable (head))) +if (!PageHuge (head)) + ret = -EBUSY ;
Unless I'm missing something, we will be returning -EBUSY for any non-hugetlb page, which is not right.
I think what you want is to return -EBUSY in case the page is a hugetlb page but we cannot grab a refcount because the page is in a floating state, meaning !HPageFreed && !HPageMigratable.
Right?
Signature · 2 lines
Oscar Salvador SUSE L3
Re: [PATCH v4 2/2] mm,hwpoison: make get_hwpoison_page call get_any_page()
Quoted 6 lines · Oscar Salvador
Quoted 5 lines · Naoya Horiguchi
static int
__get_hwpoison_page(struct page *page){mm/memory-failure.c +0
-0struct page *head =compound_head (page);@@ -1098,7 +1091,9 @@ static int __get_hwpoison_page(struct page *page)
Quoted 8 lines
mm/memory-failure.c static int __get_hwpoison_page(struct page *page) · line 1095
Quoted 5 lines
mm/memory-failure.c +2
-1#ifdef CONFIG_HUGETLB_PAGE spin_lock (&hugetlb_lock); -if (PageHuge (head) && (HPageFreed (head) ||HPageMigratable (head))) +if (!PageHuge (head)) + ret = -EBUSY ;Unless I'm missing something, we will be returning
-EBUSYfor any non-hugetlb page, which is not right.
(sigh, hard to say why I wrote this, really sorry)
Quoted 5 lines
I think what you want is to return
-EBUSYin case the page is a hugetlb page but we cannot grab a refcount because the page is in a floating state, meaning !HPageFreed && !HPageMigratable.Right?
Totally right, so the below should what's wanted.
spin_lock(&hugetlb_lock);
if (PageHuge(head))
if (!HPageFreed(head) && !HPageMigratable(head))
ret = -EBUSY;
else
ret = get_page_unless_zero(head);
spin_unlock(&hugetlb_lock);- Naoya