mm,hwpoison: fix sending SIGBUS for Action Required MCE
Subsystems (5)
Naoya Horiguchi
reviewerv5 cover mm,hwpoison: fix sending SIGBUS for Action Required MCE
I updated the series again with the following changes based on the discussion over v4:
- separated v4's 2/2 into two as done in the former version,
- switched to "first found" approach in getting error virtual address, which could report wrong error address to applications but that's rare and not critical,
- rebased onto v5.13-rc2.
v1: https://lore.kernel.org/linux-mm/[email protected]/T
v2 (only 3/3 is posted): https://lore.kernel.org/linux-mm/20210419023658.GA1962954@u2004/
v3: https://lore.kernel.org/linux-mm/[email protected]/T
v4: https://lore.kernel.org/linux-mm/[email protected]/T
Thanks,
Naoya Horiguchi
--- quote from cover letter of v1 ---
I wrote this patchset to materialize what I think is the current allowable solution mentioned by the previous discussion [1]. I simply borrowed Tony's mutex patch and Aili's return code patch, then I queued another one to find error virtual address in the best effort manner. I know that this is not a perfect solution, but should work for some typical case.
[1]: https://lore.kernel.org/linux-mm/20210331192540.2141052f@alex-virtual-machine/
---
Summary:
Aili Yao (1):
mm,hwpoison: Return -EHWPOISON to denote that the page has already been poisoned
Naoya Horiguchi (1):
mm,hwpoison: Send SIGBUS with error virutal address
Tony Luck (1):
mm/memory-failure: Use a mutex to avoid memory_failure() races
arch/x86/kernel/cpu/mce/core.c | 13 ++- include/linux/swapops.h | 5 ++ mm/memory-failure.c | 188 ++++++++++++++++++++++++++++++++++++++--- 3 files changed, 190 insertions(+), 16 deletions(-)
Naoya Horiguchi
reviewerPATCH v5 1/3 mm/memory-failure: Use a mutex to avoid memory_failure() races
From: Tony Luck <[email protected]>
There can be races when multiple CPUs consume poison from the same page. The first into memory_failure() atomically sets the HWPoison page flag and begins hunting for tasks that map this page. Eventually it invalidates those mappings and may send a SIGBUS to the affected tasks.
But while all that work is going on, other CPUs see a "success" return code from memory_failure() and so they believe the error has been handled and continue executing.
Fix by wrapping most of the internal parts of memory_failure() in a mutex.
--- mm/memory-failure.c | 37 ++++++++++++++++++++++++------------- 1 file changed, 24 insertions(+), 13 deletions(-)
mm/memory-failure.c +24 -13
@@ -1400,6 +1400,8 @@ static int memory_failure_dev_pagemap(unsigned long pfn, int flags,return rc; } +static DEFINE_MUTEX (mf_mutex); +/** * memory_failure - Handle memory failure of a page. * @pfn: Page Number of the corrupted page
@@ -1423,7 +1425,7 @@ int memory_failure(unsigned long pfn, int flags)struct page *hpage;struct page *orig_head;struct dev_pagemap *pgmap; -int res; +int res =0 ;unsigned long page_flags;bool retry = true;
@@ -1443,13 +1445,18 @@ int memory_failure(unsigned long pfn, int flags)return -ENXIO ; } +mutex_lock (&mf_mutex); + try_again: -if (PageHuge (p)) -return memory_failure_hugetlb (pfn, flags); +if (PageHuge (p)) { + res =memory_failure_hugetlb (pfn, flags); + goto unlock_mutex; + } +if (TestSetPageHWPoison (p)) {pr_err ("Memory failure: %#lx: already hardware poisoned\n" , pfn); -return 0 ; + goto unlock_mutex; } orig_head = hpage =compound_head (p);
@@ -1482,17 +1489,19 @@ int memory_failure(unsigned long pfn, int flags) res =MF_FAILED ; }action_result (pfn,MF_MSG_BUDDY , res); -return res ==MF_RECOVERED ?0 : -EBUSY ; + res = res ==MF_RECOVERED ?0 : -EBUSY ; }else {action_result (pfn,MF_MSG_KERNEL_HIGH_ORDER ,MF_IGNORED ); -return -EBUSY ; + res = -EBUSY ; } + goto unlock_mutex; }if (PageTransHuge (hpage)) {if (try_to_split_thp_page (p,"Memory Failure" ) <0 ) {action_result (pfn,MF_MSG_UNSPLIT_THP ,MF_IGNORED ); -return -EBUSY ; + res = -EBUSY ; + goto unlock_mutex; }VM_BUG_ON_PAGE (!page_count (p), p); }
@@ -1516,7 +1525,7 @@ int memory_failure(unsigned long pfn, int flags)if (PageCompound (p) &&compound_head (p) != orig_head) {action_result (pfn,MF_MSG_DIFFERENT_COMPOUND ,MF_IGNORED ); res = -EBUSY ; - goto out; + goto unlock_page; }/*
@@ -1536,14 +1545,14 @@ int memory_failure(unsigned long pfn, int flags)num_poisoned_pages_dec ();unlock_page (p);put_page (p); -return 0 ; + goto unlock_mutex; }if (hwpoison_filter (p)) {if (TestClearPageHWPoison (p))num_poisoned_pages_dec ();unlock_page (p);put_page (p); -return 0 ; + goto unlock_mutex; }if (!PageTransTail (p) && !PageLRU (p))
@@ -1562,7 +1571,7 @@ int memory_failure(unsigned long pfn, int flags)if (!hwpoison_user_mappings (p, pfn, flags, &p)) {action_result (pfn,MF_MSG_UNMAP_FAILED ,MF_IGNORED ); res = -EBUSY ; - goto out; + goto unlock_page; }/*
@@ -1571,13 +1580,15 @@ int memory_failure(unsigned long pfn, int flags)if (PageLRU (p) && !PageSwapCache (p) && p->mapping ==NULL ) {action_result (pfn,MF_MSG_TRUNCATED_LRU ,MF_IGNORED ); res = -EBUSY ; - goto out; + goto unlock_page; } identify_page_state: res =identify_page_state (pfn, p, page_flags); -out: +unlock_page:unlock_page (p); +unlock_mutex: +mutex_unlock (&mf_mutex);return res; }EXPORT_SYMBOL_GPL (memory_failure);
Signature · 1 line
2.25.1
Andrew Morton
maintainerRe: [PATCH v5 1/3] mm/memory-failure: Use a mutex to avoid memory_failure() races
Quoted 6 lines · On Fri, 21 May 2021 12:01:54 +0900 Naoya Horiguchi <[email protected]>…
There can be races when multiple CPUs consume poison from the same page. The first into
memory_failure()atomically sets the HWPoison page flag and begins hunting for tasks that map this page. Eventually it invalidates those mappings and may send a SIGBUS to the affected tasks.
Quoted 6 lines
But while all that work is going on, other CPUs see a "success" return code from
memory_failure()and so they believe the error has been handled and continue executing.Fix by wrapping most of the internal parts of
memory_failure()in a mutex.
We can reduce the scope of that mutex, which helps readability at least.
mm/memory-failure.c renamed from mm/memory-failure.c~mm-memory-failure-use-a-mutex-to-avoid-memory_failure-races-fix +1 -2
@@ -1397,8 +1397,6 @@ out:return rc; } -static DEFINE_MUTEX (mf_mutex); -/** * memory_failure - Handle memory failure of a page. * @pfn: Page Number of the corrupted page
@@ -1425,6 +1423,7 @@ int memory_failure(unsigned long pfn, inint res =0 ;unsigned long page_flags;bool retry = true; +static DEFINE_MUTEX (mf_mutex);if (!sysctl_memory_failure_recovery)panic ("Memory failure on page %lx" , pfn);
_
Re: [PATCH v5 1/3] mm/memory-failure: Use a mutex to avoid memory_failure() races
Quoted 8 lines · Andrew Morton
Quoted 6 lines · On Fri, 21 May 2021 12:01:54 +0900 Naoya Horiguchi <[email protected]>…
There can be races when multiple CPUs consume poison from the same page. The first into
memory_failure()atomically sets the HWPoison page flag and begins hunting for tasks that map this page. Eventually it invalidates those mappings and may send a SIGBUS to the affected tasks.
Quoted 8 lines
Quoted 6 lines
But while all that work is going on, other CPUs see a "success" return code from
memory_failure()and so they believe the error has been handled and continue executing.Fix by wrapping most of the internal parts of
memory_failure()in a mutex.We can reduce the scope of that mutex, which helps readability at least.
Thanks, this change is totally fine to me.
Quoted 22 lines
mm/memory-failure.c renamed from mm/memory-failure.c~mm-memory-failure-use-a-mutex-to-avoid-memory_failure-races-fix +1
-2@@ -1397,8 +1397,6 @@ out:return rc; } -static DEFINE_MUTEX (mf_mutex); -/** * memory_failure - Handle memory failure of a page. * @pfn: Page Number of the corrupted page @@ -1425,6 +1423,7 @@ int memory_failure(unsigned long pfn, inint res =0 ;unsigned long page_flags;bool retry = true; +static DEFINE_MUTEX (mf_mutex);if (!sysctl_memory_failure_recovery)panic ("Memory failure on page %lx" , pfn);_
Re: [PATCH v5 1/3] mm/memory-failure: Use a mutex to avoid memory_failure() races
In reply to Naoya HoriguchiQuoted 12 lines · Naoya Horiguchi
From: Tony Luck <[email protected]>
There can be races when multiple CPUs consume poison from the same page. The first into
memory_failure()atomically sets the HWPoison page flag and begins hunting for tasks that map this page. Eventually it invalidates those mappings and may send a SIGBUS to the affected tasks.But while all that work is going on, other CPUs see a "success" return code from
memory_failure()and so they believe the error has been handled and continue executing.
Quoted 6 lines
Fix by wrapping most of the internal parts of
memory_failure()in a mutex.Signed-off-by Tony Luck <[email protected]>Signed-off-by Naoya Horiguchi <[email protected]>Reviewed-by Borislav Petkov <[email protected]>
Signature · 2 lines
Oscar Salvador SUSE L3
Naoya Horiguchi
reviewerPATCH v5 2/3 mm,hwpoison: Return -EHWPOISON to denote that the page has already been poisoned
In reply to Naoya HoriguchiFrom: Aili Yao <[email protected]>
When memory_failure() is called with MF_ACTION_REQUIRED on the page that has already been hwpoisoned, memory_failure() could fail to send SIGBUS to the affected process, which results in infinite loop of MCEs.
Currently memory_failure() returns 0 if it's called for already hwpoisoned page, then the caller, kill_me_maybe(), could return without sending SIGBUS to current process. An action required MCE is raised when the current process accesses to the broken memory, so no SIGBUS means that the current process continues to run and access to the error page again soon, so running into MCE loop.
This issue can arise for example in the following scenarios:
- Two or more threads access to the poisoned page concurrently. If local MCE is enabled, MCE handler independently handles the MCE events. So there's a race among MCE events, and the second or latter threads fall into the situation in question.
- If there was a precedent memory error event and memory_failure() for the event failed to unmap the error page for some reason, the subsequent memory access to the error page triggers the MCE loop situation.
To fix the issue, make memory_failure() return an error code when the error page has already been hwpoisoned. This allows memory error handler to control how it sends signals to userspace. And make sure that any process touching a hwpoisoned page should get a SIGBUS even in "already hwpoisoned" path of memory_failure() as is done in page fault path.
---
ChangeLog v5:
- update patch description.
--- mm/memory-failure.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/memory-failure.c +2 -1
@@ -1247,7 +1247,7 @@ static int memory_failure_hugetlb(unsigned long pfn, int flags)if (TestSetPageHWPoison (head)) {pr_err ("Memory failure: %#lx: already hardware poisoned\n" , pfn); -return 0 ; +return -EHWPOISON ; }num_poisoned_pages_inc ();
@@ -1456,6 +1456,7 @@ int memory_failure(unsigned long pfn, int flags)if (TestSetPageHWPoison (p)) {pr_err ("Memory failure: %#lx: already hardware poisoned\n" , pfn); + res = -EHWPOISON ; goto unlock_mutex; }
Signature · 1 line
2.25.1
Re: [PATCH v5 2/3] mm,hwpoison: Return -EHWPOISON to denote that the page has already been poisoned
Quoted 33 lines · Naoya Horiguchi
From: Aili Yao <[email protected]>
When
memory_failure()is called withMF_ACTION_REQUIREDon the page that has already been hwpoisoned,memory_failure()could fail to send SIGBUS to the affected process, which results in infinite loop of MCEs.Currently
memory_failure()returns 0 if it's called for already hwpoisoned page, then the caller,kill_me_maybe(), could return without sending SIGBUS to current process. An action required MCE is raised when the current process accesses to the broken memory, so no SIGBUS means that the current process continues to run and access to the error page again soon, so running into MCE loop.This issue can arise for example in the following scenarios:
- Two or more threads access to the poisoned page concurrently. If local MCE is enabled, MCE handler independently handles the MCE events. So there's a race among MCE events, and the second or latter threads fall into the situation in question.
- If there was a precedent memory error event and
memory_failure()for the event failed to unmap the error page for some reason, the subsequent memory access to the error page triggers the MCE loop situation.To fix the issue, make
memory_failure()return an error code when the error page has already been hwpoisoned. This allows memory error handler to control how it sends signals to userspace. And make sure that any process touching a hwpoisoned page should get a SIGBUS even in "already hwpoisoned" path ofmemory_failure()as is done in page fault path.
Quoted 2 lines
Signed-off-by Aili Yao <[email protected]>Signed-off-by Naoya Horiguchi <[email protected]>
Signature · 2 lines
Oscar Salvador SUSE L3
Naoya Horiguchi
reviewerPATCH v5 3/3 mm,hwpoison: Send SIGBUS with error virutal address
In reply to Naoya HoriguchiFrom: Naoya Horiguchi <[email protected]>
Now an action required MCE in already hwpoisoned address surely sends a SIGBUS to current process, but the SIGBUS doesn't convey error virtual address. That's not optimal for hwpoison-aware applications.
To fix the issue, make memory_failure() call kill_accessing_process(), that does pagetable walk to find the error virtual address. It could find multiple virtual addresses for the same error page, and it seems hard to tell which virtual address is correct one. But that's rare and sending incorrect virtual address could be better than no address. So let's report the first found virtual address for now.
---
change log v4 -> v5:
- switched to first found approach,
- introduced check_hwpoisoned_pmd_entry() to fix build failure on arch without thp support.
change log v3 -> v4:
- refactored hwpoison_pte_range to save indentation,
- updated patch description
change log v1 -> v2:
- initialize local variables in check_hwpoisoned_entry() and hwpoison_pte_range()
- fix and improve logic to calculate error address offset.
--- arch/x86/kernel/cpu/mce/core.c | 13 ++- include/linux/swapops.h | 5 ++ mm/memory-failure.c | 150 ++++++++++++++++++++++++++++++++- 3 files changed, 165 insertions(+), 3 deletions(-)
arch/x86/kernel/cpu/mce/core.c +11 -2
@@ -1257,19 +1257,28 @@ static void kill_me_maybe(struct callback_head *cb) { struct task_struct *p = container_of(cb, struct task_struct, mce_kill_me);int flags =MF_ACTION_REQUIRED ; +int ret;pr_err ("Uncorrected hardware memory error in user-access at %llx" , p->mce_addr);if (!p->mce_ripv) flags |=MF_MUST_KILL ; -if (!memory_failure (p->mce_addr >>PAGE_SHIFT , flags) && - !(p->mce_kflags &MCE_IN_KERNEL_COPYIN )) { + ret =memory_failure (p->mce_addr >>PAGE_SHIFT , flags); +if (!ret && !(p->mce_kflags &MCE_IN_KERNEL_COPYIN )) {set_mce_nospec (p->mce_addr >>PAGE_SHIFT , p->mce_whole_page);sync_core ();return ; } +/* +* -EHWPOISON from memory_failure() means that it already sent SIGBUS +* to the current process with the proper error info, so no need to +* send SIGBUS here again. +*/ +if (ret == -EHWPOISON ) +return ; +if (p->mce_vaddr != (void __user *)-1l ) {force_sig_mceerr (BUS_MCEERR_AR , p->mce_vaddr,PAGE_SHIFT ); }else {
include/linux/swapops.h +5 -0
@@ -323,6 +323,11 @@ static inline int is_hwpoison_entry(swp_entry_t entry)return swp_type (entry) ==SWP_HWPOISON ; } +static inline unsigned long hwpoison_entry_to_pfn (swp_entry_t entry) +{ +return swp_offset (entry); +} +static inline void num_poisoned_pages_inc (void ) {atomic_long_inc (&num_poisoned_pages);
mm/memory-failure.c +149 -1
174 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [PATCH v5 3/3] mm,hwpoison: Send SIGBUS with error virutal address
Quoted 29 lines · Naoya Horiguchi
From: Naoya Horiguchi <[email protected]>
Now an action required MCE in already hwpoisoned address surely sends a SIGBUS to current process, but the SIGBUS doesn't convey error virtual address. That's not optimal for hwpoison-aware applications.
To fix the issue, make
memory_failure()callkill_accessing_process(), that does pagetable walk to find the error virtual address. It could find multiple virtual addresses for the same error page, and it seems hard to tell which virtual address is correct one. But that's rare and sending incorrect virtual address could be better than no address. So let's report the first found virtual address for now.Signed-off-by Naoya Horiguchi <[email protected]>---
change log v4 -> v5:
- switched to first found approach,
- introducedcheck_hwpoisoned_pmd_entry()to fix build failure on arch without thp support.change log v3 -> v4:
- refactoredhwpoison_pte_rangeto save indentation,
- updated patch descriptionchange log v1 -> v2:
- initialize local variables incheck_hwpoisoned_entry()andhwpoison_pte_range()
- fix and improve logic to calculate error address offset.
---
...
mm/memory-failure.c static void collect_procs(struct page *page, struct list_head *tokill, · line 694
Quoted 13 lines
mm/memory-failure.c +13
-0+static int kill_accessing_process (struct task_struct *p,unsigned long pfn, +int flags) +{ +int ret; +struct hwp_walk priv = { + .pfn = pfn, + }; + priv.tk.tsk = p; + +mmap_read_lock (p->mm); + ret =walk_page_range (p->mm,0 ,TASK_SIZE , &hwp_walk_ops, + (void *)&priv); +if (!ret && priv.tk.addr)
Sorry, I found a silly mistake, the walk_page_range() got to return 1 when it found at least error virtual address since v5, so this if-condition should be like this.
@@ -691,7 +691,8 @@ static int kill_accessing_process(struct task_struct *p, unsigned long pfn,
mmap_read_lock(p->mm);
ret = walk_page_range(p->mm, 0, TASK_SIZE, &hwp_walk_ops,
(void *)&priv);
- if (!ret && priv.tk.addr)
+ if (ret == 1 && priv.tk.addr)
kill_proc(&priv.tk, pfn, flags);
mmap_read_unlock(p->mm);
return ret ? -EFAULT : -EHWPOISON;Andrew, this patch is now in linux-mm, so could you apply this fix onto mmhwpoison-send-sigbus-with-error-virutal-address.patch ?
Or if it's better to resend a whole patch, please let me know.
Thanks,
Naoya Horiguchi