PAGE_SIZE Unmapping in Memory Failure Recovery for HugeTLB Pages
Subsystems (7)
cover PAGE_SIZE Unmapping in Memory Failure Recovery for HugeTLB Pages
Goal
====
Currently once a byte in a HugeTLB hugepage becomes HWPOISON, the whole hugepage will be unmapped from the page table because that is the finest granularity of the mapping.
High granularity mapping (HGM) [1], the functionality to map memory addresses at finer granularities (extreme case is PAGE_SIZE), is recently proposed upstream, and provides the opportunity to handle memory error more
efficiently: instead of unmapping the whole hugepage, only the raw subpage in the hugepage needs to be thrown away and all the healthy subpages can still be kept available for users.
Idea
====
Today memory failure recovery for HugeTLB pages (hugepage) is different from raw and THP pages. We are only interested in in-use hugepages, which is dealt with in these simplified steps:
1. Increment the refcount on the compound head of the hugepage.
2. Insert the raw HWPOISON page to the compound head’s raw_hwp_list (_hugetlb_hwpoison) if it is not already in the list.
3. Unmap the entire hugepage from HugeTLB’s page table.
4. Kill the processes that are accessing the poisoned hugepage.
HGM can greatly improve this recovery mechanism. Step #3 (unmapping entire hugepage) can be replaced by
3.1 Map the entire hugepage at finer granularity, so that the exact
HWPOISON address is mapped by a PAGE_SIZE PTE, and the rest of the address spaces optimally mapped by either smaller P*Ds or PTEs. In other words, the original HugeTLB PTE is split into smaller P*Ds and PTEs.
3.2 Only unmap the newly mapped PTE that maps the HWPOISON address.
For shared mappings, current HGM patches is already a solid basis for splitting functionality in step #3.1. This RFC drafts a complete solution for shared mapping. The splitting-based idea can be applied to private mappings as well, but additional subtle complexity needs to be dealt with. We defer the private mapping case as future work.
Splitting HugeTLB PTEs (Step #3.1)
==================================
The general process of splitting a present leaf HugeTLB PTE is
1. Get and clear the original HugeTLB PTE old_pte.
2. Initialize curr with the start address range corresponding to old_pte.
3. Find the optimal level we should map curr at.
4. Perform HGM walk on curr with the optimal level found in step 3, potentially allocating a new PTE at the optimal level.
5. Populate the newly allocated PTE with bits from old_pte, including dirty, write, and UFFD_WP.
6. Update curr += the newly created PTE size, repeat step 3 until the entire VMA is covered.
The functionality of splitting hugepage mapping is not meaningful for mostly none PTEs. We handle none or userfaultfd write protect (UFFD_WP) marker HugeTLB PTEs at the time of page faulting. Migration and HWPOISON PTEs are better left not touched.
Memory Failure Recovery and Unmapping (Step #3.2)
=================================================
A few changes are made in memory_failure and rmap to only unmap raw HWPOISON pages:
1. as long as HGM is turned on in CONFIG, memory_failure attempts to enable HGM on the VMA containing the poisoned hugepage
2. memory_failure attempts to split the HugeTLB PTE so that poisoned address is mapped by a PAGE_SIZE PTE, for all the VMAs containing the poisoned hugepage.
3. get_huge_page_for_hwpoison only returns -EHWPOISON if the raw page is already in the compound head’s raw_hwp_list. This makes unmapping work correctly when multiple raw pages in the same hugepage become HWPOISON.
4. rmap utilizes compound head’s raw_hwp_list to 1) avoid unmapping raw pages not in the list, and 2) keep track if the raw pages in the list are already unmapped.
5. page refcount check in me_huge_page is skipped.
Between mmap() and Page Fault
==========================
Memory error can occur between the time when userspace maps a hugepage and the time when userspace faults in the mapped hugepage. General idea is to not create any raw-page-size page table entry for HWPOISON memory, and render memory in healthy raw pages still available to userspace (via normal fault handling). At the time of hugetlb_no_page:
- If the entire hugepage doesn’t contain any HWPOISON page, the normal page fault handler continues.
- If the memory address being faulted is within a HWPOISON raw page, hugetlb_no_page returns VM_FAULT_HWPOISON_LARGE (so that page fault handler sends a BUS_MCEERR_AR SIGBUS to the faulting process).
- If the memory address being faulted is within a healthy raw page, hugetlb_no_page utilize HGM to create a new HugeTLB PTE so that its hugetlb_pte_size cannot be larger and at the same time it doesn’t map any HWPOISON address. Then the normal page fault handler continues.
Failure Handling
================
- If the kernel still fails to allocate a new raw_hwp_page after a retry, memory_failure returns MF_IGNORED with MF_MSG_UNKNOWN.
- For each VMA that maps the HWPOISON hugepage
- If the VMA is not eligible for HGM, the old behavior is taken: unmap the entire hugepage from that VMA.
- If memory_failure fails to enable HGM on the VMA, or if memory_failure fails to split any VMA that mapped the HWPOISON page, the recovery returns MF_IGNORED with MF_MSG_UNMAP_FAILED.
- For a particular VMA, if splitting HugeTLB PTE fails, the original PTE will be restored to the page table.
Code Changes
============
The code patches in this RFC is based on HGM patchset V2 [1], composed of two parts. The first part implements the idea laid out in the cover letter; the second part tests two major scenarios: HWPOISON on already faulted pages and HWPOISON between mapped and faulted.
Future Changes
==============
There is a pending improvement to hugetlbfs_read_iter. If a hugepage is found from page cache and it contains HWPOISON subpages, today kernel returns -EIO immediately. With the new splitting-then-unmap behavior, kernel can return userspace every byte until up to the first raw HWPOISON byte. If userspace wants the read to start within a raw HWPOISON page, kernel will have to return -EIO. This improvement and its selftest will be done in the future patch series.
[1] https://lore.kernel.org/all/[email protected]/
Jiaqi Yan (7):
hugetlb: add HugeTLB splitting functionality
hugetlb: create PTE level mapping when possible
mm: publish raw_hwp_page in mm.h
mm/memory_failure: unmap raw HWPoison PTEs when possible
hugetlb: only VM_FAULT_HWPOISON_LARGE raw page selftest/mm: test PAGESIZE unmapping HWPOISON pages selftest/mm: test PAGESIZE unmapping UFFD WP marker HWPOISON pages
include/linux/hugetlb.h | 14 + include/linux/mm.h | 36 ++ mm/hugetlb.c | 405 ++++++++++++++++++++++- mm/memory-failure.c | 206 ++++++++++-- mm/rmap.c | 38 ++- tools/testing/selftests/mm/hugetlb-hgm.c | 364 ++++++++++++++++++-- 6 files changed, 1004 insertions(+), 59 deletions(-)
Signature · 1 line
2.40.1.495.gc816e09b53d-goog
RFC PATCH 1/7 hugetlb: add HugeTLB splitting functionality
The new function, hugetlb_split_to_shift, optimally splits the page table to map a particular address at a paricular granularity. This is useful for punching a hole in the mapping and for mapping (and unmapping) small sections of a HugeTLB page.
Splitting is for present leaf HugeTLB PTE only. None HugeTLB PTEs and other non-present HugeTLB PTE types are not supported as they are better left untouched:
* None PTEs
* Migration PTEs
* HWPOISON PTEs
* UFFD writeprotect PTEs
--- include/linux/hugetlb.h | 9 ++ mm/hugetlb.c | 249 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 258 insertions(+)
include/linux/hugetlb.h +9 -0
@@ -1266,6 +1266,9 @@ int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm, unsigned long end);int hugetlb_collapse (struct mm_struct *mm,unsigned long start,unsigned long end); +int hugetlb_split_to_shift (struct mm_struct *mm,struct vm_area_struct *vma, +struct hugetlb_pte *hpte,unsigned long addr, +unsigned int desired_shift);#else static inline bool hugetlb_hgm_enabled (struct vm_area_struct *vma) {
@@ -1292,6 +1295,12 @@ int hugetlb_collapse(struct mm_struct *mm, unsigned long start, {return -EINVAL ; } +int hugetlb_split_to_shift (struct mm_struct *mm,struct vm_area_struct *vma, +const struct hugetlb_pte *hpte,unsigned long addr, +unsigned int desired_shift) +{ +return -EINVAL ; +}#endif static inline
mm/hugetlb.c +249 -0
255 lines not on this page · raw message
Signature · 1 line
2.40.1.495.gc816e09b53d-goog
RFC PATCH 2/7 hugetlb: create PTE level mapping when possible
In reply to Jiaqi YanIn memory_failure handling, for each VMA that the HWPOISON HugeTLB page mapped to, enable HGM if eligible, then split the P*D mapped hugepage to smaller PTEs. try_to_unmap still unmaps the entire hugetlb page, one PTE by one PTE, at levels smaller than original P*D. For example, if a hugepage was original mapped at PUD size, it will be split into PMDs and PTEs, and all of these PMDs and PTEs will be unmapped. The next commit will only unmap the raw HWPOISON PTE.
For VMA that is not HGM eligible, or failed to enable HGM, or failed to split hugepage mapping, the hugepage is still mapped by its original P*D then unmapped at this P*D.
--- include/linux/hugetlb.h | 5 +++ mm/hugetlb.c | 27 ++++++++++++++++ mm/memory-failure.c | 68 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 100 insertions(+)
include/linux/hugetlb.h +5 -0
@@ -1266,6 +1266,7 @@ int hugetlb_alloc_largest_pte(struct hugetlb_pte *hpte, struct mm_struct *mm, unsigned long end);int hugetlb_collapse (struct mm_struct *mm,unsigned long start,unsigned long end); +int hugetlb_enable_hgm_vma (struct vm_area_struct *vma);int hugetlb_split_to_shift (struct mm_struct *mm,struct vm_area_struct *vma,struct hugetlb_pte *hpte,unsigned long addr,unsigned int desired_shift);
@@ -1295,6 +1296,10 @@ int hugetlb_collapse(struct mm_struct *mm, unsigned long start, {return -EINVAL ; } +int hugetlb_enable_hgm_vma (struct vm_area_struct *vma) +{ +return -EINVAL ; +}int hugetlb_split_to_shift (struct mm_struct *mm,struct vm_area_struct *vma,const struct hugetlb_pte *hpte,unsigned long addr,unsigned int desired_shift)
mm/hugetlb.c +27 -0
@@ -8203,6 +8203,33 @@ int hugetlb_collapse(struct mm_struct *mm, unsigned long start,return ret; } +int hugetlb_enable_hgm_vma (struct vm_area_struct *vma) +{ +if (hugetlb_hgm_enabled (vma)) +return 0 ; + +if (!is_vm_hugetlb_page (vma)) { +pr_warn ("VMA=[%#lx, %#lx) is not HugeTLB\n" , + vma->vm_start, vma->vm_end); +return -EINVAL ; + } + +if (!hugetlb_hgm_eligible (vma)) { +pr_warn ("VMA=[%#lx, %#lx) is not HGM eligible\n" , + vma->vm_start, vma->vm_end); +return -EINVAL ; + } + +hugetlb_unshare_all_pmds (vma); + +/* +* TODO: add the ability to tell if HGM is enabled by kernel +* (for HWPOISON unmapping) or by userspace (via MADV_SPLIT). +*/ +vm_flags_set (vma,VM_HUGETLB_HGM ); +return 0 ; +} +/* * Find the optimal HugeTLB PTE shift that @desired_addr could be mapped at. */
mm/memory-failure.c +68 -0
80 lines not on this page · raw message
Signature · 1 line
2.40.1.495.gc816e09b53d-goog
RFC PATCH 3/7 mm: publish raw_hwp_page in mm.h
In reply to Jiaqi Yanraw_hwp_page will be needed by HugeTLB to determine if a raw subpage in a hugepage is poisoned and either should be unmapped or not faulted in at PAGE_SIZE PTE level
--- include/linux/mm.h | 16 ++++++++++++++++ mm/memory-failure.c | 13 ------------- 2 files changed, 16 insertions(+), 13 deletions(-)
include/linux/mm.h +16 -0
@@ -3522,6 +3522,22 @@ enum mf_action_page_type {*/ extern const struct attribute_group memory_failure_attr_group; +#ifdef CONFIG_HUGETLB_PAGE +/* +* Struct raw_hwp_page represents information about "raw error page", +* constructing singly linked list from ->_hugetlb_hwpoison field of folio. +*/ +struct raw_hwp_page { +struct llist_node node; +struct page *page; +}; + +static inline struct llist_head *raw_hwp_list_head (struct folio *folio) +{ +return (struct llist_head *)&folio->_hugetlb_hwpoison; +} +#endif +#if defined(CONFIG_TRANSPARENT_HUGEPAGE ) || defined(CONFIG_HUGETLBFS ) extern void clear_huge_page(struct page *page, unsigned long addr_hint,
mm/memory-failure.c +0 -13
@@ -1826,19 +1826,6 @@ EXPORT_SYMBOL_GPL(mf_dax_kill_procs);#endif /* CONFIG_FS_DAX */ #ifdef CONFIG_HUGETLB_PAGE -/* -* Struct raw_hwp_page represents information about "raw error page", -* constructing singly linked list from ->_hugetlb_hwpoison field of folio. -*/ -struct raw_hwp_page { -struct llist_node node; -struct page *page; -}; - -static inline struct llist_head *raw_hwp_list_head (struct folio *folio) -{ -return (struct llist_head *)&folio->_hugetlb_hwpoison; -}static unsigned long __folio_free_raw_hwp (struct folio *folio,bool move_flag) {
Signature · 1 line
2.40.1.495.gc816e09b53d-goog
RFC PATCH 4/7 mm/memory_failure: unmap raw HWPoison PTEs when possible
In reply to Jiaqi YanWhen a folio's VMA is HGM eligible, try_to_unmap_one now only unmaps the raw HWPOISON page (previously split and mapped at PTE size). If HGM failed to be enabled on eligible VMA or splitting failed, try_to_unmap_one fails.
For VMS that is not HGM eligible, try_to_unmap_one still unmaps the whole P*D.
When only the raw HWPOISON subpage is unmapped but others keep mapped, the old way in memory_failure to check if unmapping successful doesn't work. So introduce is_unmapping_successful() to cover both existing and new unmapping behavior.
For the new unmapping behavior, store how many times a raw HWPOISON page is expected to be unmapped, and how many times it is actually unmapped in try_to_unmap_one(). A HWPOISON raw page is expected to be unmapped from a VMA if splitting succeeded in try_to_split_huge_mapping(), so unmap_success = (nr_expected_unamps == nr_actual_unmaps).
Old folio_set_hugetlb_hwpoison returns -EHWPOISON if a folio has any raw HWPOISON subpage, and try_memory_failure_hugetlb won't attempt recovery actions again because recovery used to be done on the entire hugepage. With the new unmapping behavior, this doesn't hold. More subpages in the hugepage can become corrupted, and needs to be recovered (i.e. unmapped) individually. New folio_set_hugetlb_hwpoison returns 0 after adding a new raw subpage to raw_hwp_list.
Unmapping raw HWPOISON page requires allocating raw_hwp_page successfully in folio_set_hugetlb_hwpoison, so try_memory_failure_hugetlb now may fail due to OOM.
--- include/linux/mm.h | 20 ++++++- mm/memory-failure.c | 140 ++++++++++++++++++++++++++++++++++++++------ mm/rmap.c | 38 +++++++++++- 3 files changed, 175 insertions(+), 23 deletions(-)
include/linux/mm.h +19 -1
@@ -3522,20 +3522,38 @@ enum mf_action_page_type {*/ extern const struct attribute_group memory_failure_attr_group; -#ifdef CONFIG_HUGETLB_PAGE /* * Struct raw_hwp_page represents information about "raw error page", * constructing singly linked list from ->_hugetlb_hwpoison field of folio. +* @node: the node in folio->_hugetlb_hwpoison list. +* @page: the raw HWPOISON page struct. +* @nr_vmas_mapped: the number of VMAs that map @page when detected. +* @nr_expected_unmaps: if a VMA that maps @page when detected is eligible +* for high granularity mapping, @page is expected to be unmapped. +* @nr_actual_unmaps: how many times the raw page is actually unmapped. */ struct raw_hwp_page {struct llist_node node;struct page *page; +int nr_vmas_mapped; +int nr_expected_unmaps; +int nr_actual_unmaps; }; +#ifdef CONFIG_HUGETLB_PAGE static inline struct llist_head *raw_hwp_list_head (struct folio *folio) {return (struct llist_head *)&folio->_hugetlb_hwpoison; } + +struct raw_hwp_page *find_in_raw_hwp_list (struct folio *folio, +struct page *subpage); +#else +static inline struct raw_hwp_page *find_in_raw_hwp_list (struct folio *folio, +struct page *subpage) +{ +return NULL ; +}#endif #if defined(CONFIG_TRANSPARENT_HUGEPAGE ) || defined(CONFIG_HUGETLBFS )
mm/memory-failure.c +121 -19
256 lines not on this page · raw message
mm/rmap.c +35 -3
@@ -1453,6 +1453,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,struct mmu_notifier_range range;enum ttu_flags flags = (enum ttu_flags )(long )arg;bool page_poisoned; +bool hgm_eligible =hugetlb_hgm_eligible (vma); +struct raw_hwp_page *hwp_page;/* * When racing against e.g. zap_pte_range() on another cpu,
@@ -1525,6 +1527,29 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,* in the case where the hugetlb page is poisoned. */ VM_BUG_ON_FOLIO (!page_poisoned, folio); + +/* +* When VMA is not HGM eligible, unmap at hugepage's +* original P*D. +* +* When HGM is eligible: +* - if original P*D is split to smaller P*Ds and +* PTEs, we skip subpage if it is not raw HWPoison +* page, or it was but was already unmapped. +* - if original P*D is not split, skip unmapping +* and memory_failure result will be MF_IGNORED. +*/ +if (hgm_eligible) { +if (pvmw.pte_order >0 ) +continue ; + hwp_page =find_in_raw_hwp_list (folio, subpage); +if (hwp_page ==NULL ) +continue ; +if (hwp_page->nr_expected_unmaps == + hwp_page->nr_actual_unmaps) +continue ; + } +/* * huge_pmd_unshare may unmap an entire PMD page. * There is no way of knowing exactly which PMDs may
@@ -1760,12 +1785,19 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,* * See Documentation/mm/mmu_notifier.rst */ -if (folio_test_hugetlb (folio)) +if (!folio_test_hugetlb (folio)) +page_remove_rmap (subpage, vma, false); +else {hugetlb_remove_rmap (subpage, pvmw.pte_order +PAGE_SHIFT ,hstate_vma (vma), vma); -else -page_remove_rmap (subpage, vma, false); +if (hgm_eligible) { +VM_BUG_ON_FOLIO (pvmw.pte_order >0 , folio); +VM_BUG_ON_FOLIO (!hwp_page, folio); +VM_BUG_ON_FOLIO (subpage != hwp_page->page, folio); + ++hwp_page->nr_actual_unmaps; + } + }if (vma->vm_flags &VM_LOCKED )mlock_drain_local ();
Signature · 1 line
2.40.1.495.gc816e09b53d-goog
Re: [RFC PATCH v1 4/7] mm/memory_failure: unmap raw HWPoison PTEs when possible
Quoted 33 lines · Jiaqi Yan
When a folio's VMA is HGM eligible,
try_to_unmap_onenow only unmaps the raw HWPOISON page (previously split and mapped at PTE size). If HGM failed to be enabled on eligible VMA or splitting failed,try_to_unmap_onefails.For VMS that is not HGM eligible,
try_to_unmap_onestill unmaps the whole P*D.When only the raw HWPOISON subpage is unmapped but others keep mapped, the old way in
memory_failureto check if unmapping successful doesn't work. So introduceis_unmapping_successful()to cover both existing and new unmapping behavior.For the new unmapping behavior, store how many times a raw HWPOISON page is expected to be unmapped, and how many times it is actually unmapped in
try_to_unmap_one(). A HWPOISON raw page is expected to be unmapped from a VMA if splitting succeeded intry_to_split_huge_mapping(), sounmap_success= (nr_expected_unamps==nr_actual_unmaps).Old
folio_set_hugetlb_hwpoisonreturns-EHWPOISONif a folio has any raw HWPOISON subpage, andtry_memory_failure_hugetlbwon't attempt recovery actions again because recovery used to be done on the entire hugepage. With the new unmapping behavior, this doesn't hold. More subpages in the hugepage can become corrupted, and needs to be recovered (i.e. unmapped) individually. Newfolio_set_hugetlb_hwpoisonreturns 0 after adding a new raw subpage toraw_hwp_list.Unmapping raw HWPOISON page requires allocating
raw_hwp_pagesuccessfully infolio_set_hugetlb_hwpoison, sotry_memory_failure_hugetlbnow may fail due to OOM.Signed-off-by Jiaqi Yan <[email protected]>---
...
Quoted 7 lines
mm/memory-failure.c +3
-0@@ -1827,6 +1879,31 @@ EXPORT_SYMBOL_GPL(mf_dax_kill_procs);#ifdef CONFIG_HUGETLB_PAGE +/* +* Given a HWPOISON @subpage as raw page, find its location in @folio's +* _hugetlb_hwpoison. Return NULL if @subpage is not in the list.
mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1894
Quoted 10 lines
mm/memory-failure.c +10
-0+*/ +struct raw_hwp_page *find_in_raw_hwp_list (struct folio *folio, +struct page *subpage) +{ +struct llist_node *t, *tnode; +struct llist_head *raw_hwp_head =raw_hwp_list_head (folio); +struct raw_hwp_page *hwp_page =NULL ; +struct raw_hwp_page *p; + +VM_BUG_ON_PAGE (PageHWPoison (subpage), subpage);
I'm testing the series (on top of v6.2-rc4 + HGM v2 patchset) and found the following error triggered by this VM_BUG_ON_PAGE(). The testcase is just to inject hwpoison on an anonymous page (it's not hugetlb-related one).
[ 790.610985] ===> testcase 'mm/hwpoison/base/backend-anonymous_error-hard-offline_access-avoid.auto3' start [ 793.304927] page:000000006743177b refcount:1 mapcount:0 mapping:0000000000000000 index:0x700000000 pfn:0x14d739 [ 793.309322] memcg:ffff8a30c50b6000 [ 793.310934] anon flags: 0x57ffffe08a001d(locked|uptodate|dirty|lru|mappedtodisk|swapbacked|hwpoison|node=1|zone=2|lastcpupid=0x1fffff) [ 793.316665] raw: 0057ffffe08a001d ffffe93cc5353c88 ffffe93cc5685fc8 ffff8a30c91878f1 [ 793.320211] raw: 0000000700000000 0000000000000000 00000001ffffffff ffff8a30c50b6000 [ 793.323665] page dumped because: VM_BUG_ON_PAGE(PageHWPoison(subpage)) [ 793.326764] ------------[ cut here ]------------ [ 793.329080] kernel BUG at mm/memory-failure.c:1894! [ 793.331895] invalid opcode: 0000 [#1] PREEMPT SMP PTI [ 793.334854] CPU: 4 PID: 2644 Comm: mceinj.sh Tainted: G E N 6.2.0-rc4-v6.2-rc2-230529-1404+ #63 [ 793.340710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.1-2.fc37 04/01/2014 [ 793.345875] RIP: 0010:hwpoison_user_mappings+0x654/0x780 [ 793.349066] Code: ef 89 de e8 6e bc f8 ff 48 8b 7c 24 20 48 83 c7 58 e8 10 bb d9 ff e9 5f fb ff ff 48 c7 c6 80 ce 4c b1 4c 89 ef e8 1c 38 f6 ff <0f> 0b 48 c7 c6 7b c8 4c b1 4c 89 ef e8 0b 38 f6 ff 0f 0b 8b 45 58 [ 793.359732] RSP: 0018:ffffa3ff85ed3d28 EFLAGS: 00010296 [ 793.362367] RAX: 000000000000003a RBX: 0000000000000018 RCX: 0000000000000000 [ 793.365763] RDX: 0000000000000001 RSI: ffffffffb14ac451 RDI: 00000000ffffffff [ 793.368698] RBP: ffffe93cc535ce40 R08: 0000000000000000 R09: ffffa3ff85ed3ba0 [ 793.370837] R10: 0000000000000003 R11: ffffffffb1d3ed28 R12: 000000000014d739 [ 793.372903] R13: ffffe93cc535ce40 R14: ffffe93cc535ce40 R15: ffffe93cc535ce40 [ 793.374931] FS: 00007f6ccc42a740(0000) GS:ffff8a31bbc00000(0000) knlGS:0000000000000000 [ 793.377136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 793.378656] CR2: 0000561aad6474b2 CR3: 00000001492d4005 CR4: 0000000000170ee0 [ 793.380514] DR0: ffffffffb28ed7d0 DR1: ffffffffb28ed7d1 DR2: ffffffffb28ed7d2 [ 793.382296] DR3: ffffffffb28ed7d3 DR6: 00000000ffff0ff0 DR7: 0000000000000600 [ 793.384028] Call Trace: [ 793.384655] <TASK> [ 793.385210] ? __lru_add_drain_all+0x164/0x1f0 [ 793.386316] memory_failure+0x352/0xaa0 [ 793.387249] ? __pfx_bpf_lsm_capable+0x10/0x10 [ 793.388323] ? __pfx_security_capable+0x10/0x10 [ 793.389350] hard_offline_page_store+0x46/0x80 [ 793.390397] kernfs_fop_write_iter+0x11e/0x200 [ 793.391441] vfs_write+0x1e4/0x3a0 [ 793.392221] ksys_write+0x53/0xd0 [ 793.392976] do_syscall_64+0x3a/0x90 [ 793.393790] entry_SYSCALL_64_after_hwframe+0x72/0xdc
I'm wondering how this code path is called, one possible path is like this:
hwpoison_user_mappings
if PageHuge(hpage) && !PageAnon(hpage)
try_to_split_huge_mapping()
find_in_raw_hwp_list
VM_BUG_ON_PAGE(PageHWPoison(subpage), subpage)but this looks unlikely because the precheck "PageHuge(hpage) && !PageAnon(hpage)" is false for anonymous pages.
Another possible code path is:
hwpoison_user_mappings
if PageHuge(hpage) && !PageAnon(hpage)
...
else
try_to_unmap
rmap_walk
rmap_walk_anon
try_to_unmap_one
if folio_test_hugetlb
if hgm_eligible
find_in_raw_hwp_list
VM_BUG_ON_PAGE(PageHWPoison(subpage), subpage)but this looks also unlikely because of checking folio_test_hugetlb and hgm_eligible (I think both are false in this testcase.)
Maybe I miss something (and I'll dig this more), but let me share the issue.
Thanks,
Naoya Horiguchi
Re: [RFC PATCH v1 4/7] mm/memory_failure: unmap raw HWPoison PTEs when possible
Quoted 50 lines · HORIGUCHI NAOYA(堀口 直也)
Quoted 33 lines · Jiaqi Yan
When a folio's VMA is HGM eligible,
try_to_unmap_onenow only unmaps the raw HWPOISON page (previously split and mapped at PTE size). If HGM failed to be enabled on eligible VMA or splitting failed,try_to_unmap_onefails.For VMS that is not HGM eligible,
try_to_unmap_onestill unmaps the whole P*D.When only the raw HWPOISON subpage is unmapped but others keep mapped, the old way in
memory_failureto check if unmapping successful doesn't work. So introduceis_unmapping_successful()to cover both existing and new unmapping behavior.For the new unmapping behavior, store how many times a raw HWPOISON page is expected to be unmapped, and how many times it is actually unmapped in
try_to_unmap_one(). A HWPOISON raw page is expected to be unmapped from a VMA if splitting succeeded intry_to_split_huge_mapping(), sounmap_success= (nr_expected_unamps==nr_actual_unmaps).Old
folio_set_hugetlb_hwpoisonreturns-EHWPOISONif a folio has any raw HWPOISON subpage, andtry_memory_failure_hugetlbwon't attempt recovery actions again because recovery used to be done on the entire hugepage. With the new unmapping behavior, this doesn't hold. More subpages in the hugepage can become corrupted, and needs to be recovered (i.e. unmapped) individually. Newfolio_set_hugetlb_hwpoisonreturns 0 after adding a new raw subpage toraw_hwp_list.Unmapping raw HWPOISON page requires allocating
raw_hwp_pagesuccessfully infolio_set_hugetlb_hwpoison, sotry_memory_failure_hugetlbnow may fail due to OOM.Signed-off-by Jiaqi Yan <[email protected]>---
...
Quoted 13 lines
mm/memory-failure.c +9
-0@@ -1827,6 +1879,31 @@ EXPORT_SYMBOL_GPL(mf_dax_kill_procs);#ifdef CONFIG_HUGETLB_PAGE +/* +* Given a HWPOISON @subpage as raw page, find its location in @folio's +* _hugetlb_hwpoison. Return NULL if @subpage is not in the list. +*/ +struct raw_hwp_page *find_in_raw_hwp_list (struct folio *folio, +struct page *subpage) +{ +struct llist_node *t, *tnode; +struct llist_head *raw_hwp_head =raw_hwp_list_head (folio);
Quoted 8 lines
mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1894
Quoted 4 lines
mm/memory-failure.c +4
-0+struct raw_hwp_page *hwp_page =NULL ; +struct raw_hwp_page *p; + +VM_BUG_ON_PAGE (PageHWPoison (subpage), subpage);I'm testing the series (on top of v6.2-rc4 + HGM v2 patchset) and found the following error triggered by this
VM_BUG_ON_PAGE(). The testcase is just to inject hwpoison on an anonymous page (it's not hugetlb-related one).
Thanks for reporting this problem, Naoya!
My mistake, this assertion meant to be "if !PageHWPoison(subpage)", to make sure the caller of find_in_raw_hwp_list is sure that subpage is hw corrupted.
Quoted 65 lines
[ 790.610985] ===> testcase 'mm/hwpoison/base/backend-anonymous_error-hard-offline_access-avoid.auto3' start [ 793.304927] page:000000006743177b refcount:1 mapcount:0 mapping:0000000000000000 index:0x700000000 pfn:0x14d739 [ 793.309322] memcg:ffff8a30c50b6000 [ 793.310934] anon flags: 0x57ffffe08a001d(locked|uptodate|dirty|lru|mappedtodisk|swapbacked|hwpoison|node=1|zone=2|lastcpupid=0x1fffff) [ 793.316665] raw: 0057ffffe08a001d ffffe93cc5353c88 ffffe93cc5685fc8 ffff8a30c91878f1 [ 793.320211] raw: 0000000700000000 0000000000000000 00000001ffffffff ffff8a30c50b6000 [ 793.323665] page dumped because: VM_BUG_ON_PAGE(PageHWPoison(subpage)) [ 793.326764] ------------[ cut here ]------------ [ 793.329080] kernel BUG at mm/memory-failure.c:1894! [ 793.331895] invalid opcode: 0000 [#1] PREEMPT SMP PTI [ 793.334854] CPU: 4 PID: 2644 Comm: mceinj.sh Tainted: G E N 6.2.0-rc4-v6.2-rc2-230529-1404+ #63 [ 793.340710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.1-2.fc37 04/01/2014 [ 793.345875] RIP: 0010:hwpoison_user_mappings+0x654/0x780 [ 793.349066] Code: ef 89 de e8 6e bc f8 ff 48 8b 7c 24 20 48 83 c7 58 e8 10 bb d9 ff e9 5f fb ff ff 48 c7 c6 80 ce 4c b1 4c 89 ef e8 1c 38 f6 ff <0f> 0b 48 c7 c6 7b c8 4c b1 4c 89 ef e8 0b 38 f6 ff 0f 0b 8b 45 58 [ 793.359732] RSP: 0018:ffffa3ff85ed3d28 EFLAGS: 00010296 [ 793.362367] RAX: 000000000000003a RBX: 0000000000000018 RCX: 0000000000000000 [ 793.365763] RDX: 0000000000000001 RSI: ffffffffb14ac451 RDI: 00000000ffffffff [ 793.368698] RBP: ffffe93cc535ce40 R08: 0000000000000000 R09: ffffa3ff85ed3ba0 [ 793.370837] R10: 0000000000000003 R11: ffffffffb1d3ed28 R12: 000000000014d739 [ 793.372903] R13: ffffe93cc535ce40 R14: ffffe93cc535ce40 R15: ffffe93cc535ce40 [ 793.374931] FS: 00007f6ccc42a740(0000) GS:ffff8a31bbc00000(0000) knlGS:0000000000000000 [ 793.377136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 793.378656] CR2: 0000561aad6474b2 CR3: 00000001492d4005 CR4: 0000000000170ee0 [ 793.380514] DR0: ffffffffb28ed7d0 DR1: ffffffffb28ed7d1 DR2: ffffffffb28ed7d2 [ 793.382296] DR3: ffffffffb28ed7d3 DR6: 00000000ffff0ff0 DR7: 0000000000000600 [ 793.384028] Call Trace: [ 793.384655] <TASK> [ 793.385210] ? __lru_add_drain_all+0x164/0x1f0 [ 793.386316] memory_failure+0x352/0xaa0 [ 793.387249] ? __pfx_bpf_lsm_capable+0x10/0x10 [ 793.388323] ? __pfx_security_capable+0x10/0x10 [ 793.389350] hard_offline_page_store+0x46/0x80 [ 793.390397] kernfs_fop_write_iter+0x11e/0x200 [ 793.391441] vfs_write+0x1e4/0x3a0 [ 793.392221] ksys_write+0x53/0xd0 [ 793.392976] do_syscall_64+0x3a/0x90 [ 793.393790] entry_SYSCALL_64_after_hwframe+0x72/0xdcI'm wondering how this code path is called, one possible path is like this:
hwpoison_user_mappings if PageHuge(hpage) && !PageAnon(hpage) try_to_split_huge_mapping() find_in_raw_hwp_list VM_BUG_ON_PAGE(PageHWPoison(subpage), subpage)but this looks unlikely because the precheck "PageHuge(hpage) && !PageAnon(hpage)" is false for anonymous pages.
Another possible code path is:
hwpoison_user_mappings if PageHuge(hpage) && !PageAnon(hpage) ... else try_to_unmap rmap_walk rmap_walk_anon try_to_unmap_one if folio_test_hugetlb if hgm_eligible find_in_raw_hwp_list VM_BUG_ON_PAGE(PageHWPoison(subpage), subpage)
Quoted 3 lines
but this looks also unlikely because of checking
folio_test_hugetlbandhgm_eligible(I think both are false in this testcase.)
Maybe I miss something (and I'll dig this more), but let me share the issue.
I bet it is in "is_unmapping_successful". So another problem with this patch is, "is_unmapping_successful" should only calls find_in_raw_hwp_list after it handles non hugetlb and non shared mapping, i.e.:
struct raw_hwp_page *hwp_page = NULL;
if (!folio_test_hugetlb(folio) ||
folio_test_anon(folio) ||
!IS_ENABLED(CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING)) {
...
}hwp_page = find_in_raw_hwp_list(folio, poisoned_page); VM_BUG_ON_PAGE(!hwp_page, poisoned_page);
I will make sure these two issues get fixed up in follow-up revisions.
Quoted 3 lines
Thanks,
Naoya Horiguchi
Re: [RFC PATCH v1 4/7] mm/memory_failure: unmap raw HWPoison PTEs when possible
Quoted 42 lines · Jiaqi Yan
Quoted 39 lines · HORIGUCHI NAOYA(堀口 直也)
Quoted 33 lines · Jiaqi Yan
When a folio's VMA is HGM eligible,
try_to_unmap_onenow only unmaps the raw HWPOISON page (previously split and mapped at PTE size). If HGM failed to be enabled on eligible VMA or splitting failed,try_to_unmap_onefails.For VMS that is not HGM eligible,
try_to_unmap_onestill unmaps the whole P*D.When only the raw HWPOISON subpage is unmapped but others keep mapped, the old way in
memory_failureto check if unmapping successful doesn't work. So introduceis_unmapping_successful()to cover both existing and new unmapping behavior.For the new unmapping behavior, store how many times a raw HWPOISON page is expected to be unmapped, and how many times it is actually unmapped in
try_to_unmap_one(). A HWPOISON raw page is expected to be unmapped from a VMA if splitting succeeded intry_to_split_huge_mapping(), sounmap_success= (nr_expected_unamps==nr_actual_unmaps).Old
folio_set_hugetlb_hwpoisonreturns-EHWPOISONif a folio has any raw HWPOISON subpage, andtry_memory_failure_hugetlbwon't attempt recovery actions again because recovery used to be done on the entire hugepage. With the new unmapping behavior, this doesn't hold. More subpages in the hugepage can become corrupted, and needs to be recovered (i.e. unmapped) individually. Newfolio_set_hugetlb_hwpoisonreturns 0 after adding a new raw subpage toraw_hwp_list.Unmapping raw HWPOISON page requires allocating
raw_hwp_pagesuccessfully infolio_set_hugetlb_hwpoison, sotry_memory_failure_hugetlbnow may fail due to OOM.Signed-off-by Jiaqi Yan <[email protected]>---
...
Quoted 2 lines
(quoted diff) +0
-0@@ -1827,6 +1879,31 @@ EXPORT_SYMBOL_GPL(mf_dax_kill_procs);
Quoted 7 lines
Quoted 7 lines
mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1886
Quoted 7 lines
mm/memory-failure.c +5
-0#ifdef CONFIG_HUGETLB_PAGE +/* +* Given a HWPOISON @subpage as raw page, find its location in @folio's +* _hugetlb_hwpoison. Return NULL if @subpage is not in the list. +*/ +struct raw_hwp_page *find_in_raw_hwp_list(struct folio *folio,
BTW, per our discussion here[1], this routine will probably reuse what comes out of the refactored routine.
It should be safe for try_to_unmap_one to hold a raw_hwp_page returned from find_in_raw_hwp_list as long as raw_hwp_list is protected by mf_mutex.
Quoted 109 lines
Quoted 4 lines
Quoted 4 lines
mm/memory-failure.c +4
-0+ struct page *subpage) +{ +struct llist_node *t, *tnode; +struct llist_head *raw_hwp_head =raw_hwp_list_head (folio);Quoted 8 lines
mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1894
Quoted 4 lines
mm/memory-failure.c +4
-0+struct raw_hwp_page *hwp_page =NULL ; +struct raw_hwp_page *p; + +VM_BUG_ON_PAGE (PageHWPoison (subpage), subpage);I'm testing the series (on top of v6.2-rc4 + HGM v2 patchset) and found the following error triggered by this
VM_BUG_ON_PAGE(). The testcase is just to inject hwpoison on an anonymous page (it's not hugetlb-related one).Thanks for reporting this problem, Naoya!
My mistake, this assertion meant to be "if !PageHWPoison(subpage)", to make sure the caller of
find_in_raw_hwp_listis sure that subpage is hw corrupted.Quoted 65 lines
[ 790.610985] ===> testcase 'mm/hwpoison/base/backend-anonymous_error-hard-offline_access-avoid.auto3' start [ 793.304927] page:000000006743177b refcount:1 mapcount:0 mapping:0000000000000000 index:0x700000000 pfn:0x14d739 [ 793.309322] memcg:ffff8a30c50b6000 [ 793.310934] anon flags: 0x57ffffe08a001d(locked|uptodate|dirty|lru|mappedtodisk|swapbacked|hwpoison|node=1|zone=2|lastcpupid=0x1fffff) [ 793.316665] raw: 0057ffffe08a001d ffffe93cc5353c88 ffffe93cc5685fc8 ffff8a30c91878f1 [ 793.320211] raw: 0000000700000000 0000000000000000 00000001ffffffff ffff8a30c50b6000 [ 793.323665] page dumped because: VM_BUG_ON_PAGE(PageHWPoison(subpage)) [ 793.326764] ------------[ cut here ]------------ [ 793.329080] kernel BUG at mm/memory-failure.c:1894! [ 793.331895] invalid opcode: 0000 [#1] PREEMPT SMP PTI [ 793.334854] CPU: 4 PID: 2644 Comm: mceinj.sh Tainted: G E N 6.2.0-rc4-v6.2-rc2-230529-1404+ #63 [ 793.340710] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.1-2.fc37 04/01/2014 [ 793.345875] RIP: 0010:hwpoison_user_mappings+0x654/0x780 [ 793.349066] Code: ef 89 de e8 6e bc f8 ff 48 8b 7c 24 20 48 83 c7 58 e8 10 bb d9 ff e9 5f fb ff ff 48 c7 c6 80 ce 4c b1 4c 89 ef e8 1c 38 f6 ff <0f> 0b 48 c7 c6 7b c8 4c b1 4c 89 ef e8 0b 38 f6 ff 0f 0b 8b 45 58 [ 793.359732] RSP: 0018:ffffa3ff85ed3d28 EFLAGS: 00010296 [ 793.362367] RAX: 000000000000003a RBX: 0000000000000018 RCX: 0000000000000000 [ 793.365763] RDX: 0000000000000001 RSI: ffffffffb14ac451 RDI: 00000000ffffffff [ 793.368698] RBP: ffffe93cc535ce40 R08: 0000000000000000 R09: ffffa3ff85ed3ba0 [ 793.370837] R10: 0000000000000003 R11: ffffffffb1d3ed28 R12: 000000000014d739 [ 793.372903] R13: ffffe93cc535ce40 R14: ffffe93cc535ce40 R15: ffffe93cc535ce40 [ 793.374931] FS: 00007f6ccc42a740(0000) GS:ffff8a31bbc00000(0000) knlGS:0000000000000000 [ 793.377136] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 793.378656] CR2: 0000561aad6474b2 CR3: 00000001492d4005 CR4: 0000000000170ee0 [ 793.380514] DR0: ffffffffb28ed7d0 DR1: ffffffffb28ed7d1 DR2: ffffffffb28ed7d2 [ 793.382296] DR3: ffffffffb28ed7d3 DR6: 00000000ffff0ff0 DR7: 0000000000000600 [ 793.384028] Call Trace: [ 793.384655] <TASK> [ 793.385210] ? __lru_add_drain_all+0x164/0x1f0 [ 793.386316] memory_failure+0x352/0xaa0 [ 793.387249] ? __pfx_bpf_lsm_capable+0x10/0x10 [ 793.388323] ? __pfx_security_capable+0x10/0x10 [ 793.389350] hard_offline_page_store+0x46/0x80 [ 793.390397] kernfs_fop_write_iter+0x11e/0x200 [ 793.391441] vfs_write+0x1e4/0x3a0 [ 793.392221] ksys_write+0x53/0xd0 [ 793.392976] do_syscall_64+0x3a/0x90 [ 793.393790] entry_SYSCALL_64_after_hwframe+0x72/0xdcI'm wondering how this code path is called, one possible path is like this:
hwpoison_user_mappings if PageHuge(hpage) && !PageAnon(hpage) try_to_split_huge_mapping() find_in_raw_hwp_list VM_BUG_ON_PAGE(PageHWPoison(subpage), subpage)but this looks unlikely because the precheck "PageHuge(hpage) && !PageAnon(hpage)" is false for anonymous pages.
Another possible code path is:
hwpoison_user_mappings if PageHuge(hpage) && !PageAnon(hpage) ... else try_to_unmap rmap_walk rmap_walk_anon try_to_unmap_one if folio_test_hugetlb if hgm_eligible find_in_raw_hwp_list VM_BUG_ON_PAGE(PageHWPoison(subpage), subpage)Quoted 3 lines
but this looks also unlikely because of checking
folio_test_hugetlbandhgm_eligible(I think both are false in this testcase.)
Maybe I miss something (and I'll dig this more), but let me share the issue.I bet it is in "
is_unmapping_successful". So another problem with this patch is, "is_unmapping_successful" should only callsfind_in_raw_hwp_listafter it handles non hugetlb and non shared mapping, i.e.:struct
raw_hwp_page*hwp_page= NULL;if (!folio_test_hugetlb(folio) ||
folio_test_anon(folio) || !IS_ENABLED(CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING)) { ... }
hwp_page=find_in_raw_hwp_list(folio, poisoned_page);VM_BUG_ON_PAGE(!hwp_page, poisoned_page);I will make sure these two issues get fixed up in follow-up revisions.
Quoted 3 lines
Thanks,
Naoya Horiguchi
Re: [RFC PATCH v1 4/7] mm/memory_failure: unmap raw HWPoison PTEs when possible
In reply to Jiaqi YanQuoted 13 lines · Jiaqi Yan
When a folio's VMA is HGM eligible,
try_to_unmap_onenow only unmaps the raw HWPOISON page (previously split and mapped at PTE size). If HGM failed to be enabled on eligible VMA or splitting failed,try_to_unmap_onefails.For VMS that is not HGM eligible,
try_to_unmap_onestill unmaps the whole P*D.When only the raw HWPOISON subpage is unmapped but others keep mapped, the old way in
memory_failureto check if unmapping successful doesn't work. So introduceis_unmapping_successful()to cover both existing and new unmapping behavior.
Quoted 5 lines
For the new unmapping behavior, store how many times a raw HWPOISON page is expected to be unmapped, and how many times it is actually unmapped in
try_to_unmap_one(). A HWPOISON raw page is expected to be unmapped from a VMA if splitting succeeded intry_to_split_huge_mapping(), sounmap_success= (nr_expected_unamps==nr_actual_unmaps).
s/nr_expected_unamps/nr_expected_unmaps/
Quoted 92 lines
Old
folio_set_hugetlb_hwpoisonreturns-EHWPOISONif a folio has any raw HWPOISON subpage, andtry_memory_failure_hugetlbwon't attempt recovery actions again because recovery used to be done on the entire hugepage. With the new unmapping behavior, this doesn't hold. More subpages in the hugepage can become corrupted, and needs to be recovered (i.e. unmapped) individually. Newfolio_set_hugetlb_hwpoisonreturns 0 after adding a new raw subpage toraw_hwp_list.Unmapping raw HWPOISON page requires allocating
raw_hwp_pagesuccessfully infolio_set_hugetlb_hwpoison, sotry_memory_failure_hugetlbnow may fail due to OOM.Signed-off-by Jiaqi Yan <[email protected]>--- include/linux/mm.h | 20 ++++++- mm/memory-failure.c | 140 ++++++++++++++++++++++++++++++++++++++------ mm/rmap.c | 38 +++++++++++- 3 files changed, 175 insertions(+), 23 deletions(-)include/linux/mm.h +19
-1@@ -3522,20 +3522,38 @@ enum mf_action_page_type {*/ extern const struct attribute_group memory_failure_attr_group; -#ifdef CONFIG_HUGETLB_PAGE /* * Struct raw_hwp_page represents information about "raw error page", * constructing singly linked list from ->_hugetlb_hwpoison field of folio. +* @node: the node in folio->_hugetlb_hwpoison list. +* @page: the raw HWPOISON page struct. +* @nr_vmas_mapped: the number of VMAs that map @page when detected. +* @nr_expected_unmaps: if a VMA that maps @page when detected is eligible +* for high granularity mapping, @page is expected to be unmapped. +* @nr_actual_unmaps: how many times the raw page is actually unmapped. */ struct raw_hwp_page {struct llist_node node;struct page *page; +int nr_vmas_mapped; +int nr_expected_unmaps; +int nr_actual_unmaps; }; +#ifdef CONFIG_HUGETLB_PAGE static inline struct llist_head *raw_hwp_list_head (struct folio *folio) {return (struct llist_head *)&folio->_hugetlb_hwpoison; } + +struct raw_hwp_page *find_in_raw_hwp_list (struct folio *folio, +struct page *subpage); +#else +static inline struct raw_hwp_page *find_in_raw_hwp_list (struct folio *folio, +struct page *subpage) +{ +return NULL ; +}#endif #if defined(CONFIG_TRANSPARENT_HUGEPAGE ) || defined(CONFIG_HUGETLBFS )mm/memory-failure.c +5
-4@@ -1120,10 +1120,10 @@ static int me_swapcache_clean(struct page_state *ps, struct page *p) }/* -* Huge pages. Needs work. -* Issues: -* - Error on hugepage is contained in hugepage unit (not in raw page unit.) -* To narrow down kill region to one page, we need to break up pmd. +* Huge pages. +* - Without HGM: Error on hugepage is contained in hugepage unit (not in +* raw page unit). +* - With HGM: Kill region is narrowed down to just one page. */ static int me_huge_page (struct page_state *ps,struct page *p) {@@ -1131,6 +1131,7 @@ static int me_huge_page(struct page_state *ps, struct page *p)struct page *hpage =compound_head (p);struct address_space *mapping;bool extra_pins = false; +struct raw_hwp_page *hwp_page =find_in_raw_hwp_list (page_folio (p), p);if (!PageHuge (hpage))return MF_DELAYED ;
mm/memory-failure.c static int me_huge_page(struct page_state *ps, struct page *p) · line 1161
Quoted 5 lines
mm/memory-failure.c +1
-1@@ -1157,7 +1158,8 @@ static int me_huge_page(struct page_state *ps, struct page *p) } } -if (has_extra_refcount (ps, p, extra_pins)) + if (hwp_page->nr_expected_unmaps == 0 &&
This additional check seems unclear to me, so could you add some comment about the intention?
I think that "hwp_page->nr_expected_unmaps == 0" means no mapping is split. This is OK because it just keeps the current behavior (where no HGM happens). But thinking about the "partial" case where some vmas use HGM and some not, has_extra_refcount() check seems to be meaningful, so maybe more resticted check like "hwp_page->nr_expected_unmaps < hwp_page->nr_vmas_mapped" could be better?
Quoted 24 lines
mm/memory-failure.c +7
-0+ has_extra_refcount(ps, p, extra_pins)) res =MF_FAILED ;return res;@@ -1497,24 +1499,30 @@ static void try_to_split_huge_mapping(struct folio *folio,unsigned long poisoned_addr;unsigned long head_addr;struct hugetlb_pte hpte; +struct raw_hwp_page *hwp_page =NULL ;if (WARN_ON (!mapping))return ;VM_BUG_ON_FOLIO (!folio_test_locked (folio), folio); + hwp_page =find_in_raw_hwp_list (folio, poisoned_page); +VM_BUG_ON_PAGE (!hwp_page, poisoned_page); + pgoff_start =folio_pgoff (folio); pgoff_end = pgoff_start +folio_nr_pages (folio) -1 ; vma_interval_tree_foreach(vma, &mapping->i_mmap, pgoff_start, pgoff_end) { + ++hwp_page->nr_vmas_mapped; +
mm/memory-failure.c static void try_to_split_huge_mapping(struct folio *folio, · line 1525
Quoted 10 lines
mm/memory-failure.c +2
-2/* Enable HGM on HGM-eligible VMAs. */ if (!hugetlb_hgm_eligible (vma))continue ;i_mmap_assert_locked (vma->vm_file->f_mapping);if (hugetlb_enable_hgm_vma (vma)) { -pr_err ("Failed to enable HGM on eligible VMA=[%#lx, %#lx)\n" , - vma->vm_start, vma->vm_end); +pr_err ("%#lx: failed to enable HGM on eligible VMA=[%#lx, %#lx)\n" , +page_to_pfn (poisoned_page), vma->vm_start, vma->vm_end);
This page_to_pfn(poisoned_page) can be called multiple times in this function, so this can be stored in a local variable.
Quoted 28 lines
mm/memory-failure.c +8
-2continue ; }@@ -1528,15 +1536,21 @@ static void try_to_split_huge_mapping(struct folio *folio,* lock on vma->vm_file->f_mapping, which caller * (e.g. hwpoison_user_mappings) should already acquired. */ -if (hugetlb_full_walk (&hpte, vma, head_addr)) +if (hugetlb_full_walk (&hpte, vma, head_addr)) { +pr_err ("%#lx: failed to PT-walk with HGM on eligible VMA=[%#lx, %#lx)\n" , +page_to_pfn (poisoned_page), vma->vm_start, vma->vm_end);continue ; + }if (hugetlb_split_to_shift (vma->vm_mm, vma, &hpte, poisoned_addr,PAGE_SHIFT )) { -pr_err ("Failed to split huge mapping: pfn=%#lx, vaddr=%#lx in VMA=[%#lx, %#lx)\n" , +pr_err ("%#lx: Failed to split huge mapping: vaddr=%#lx in VMA=[%#lx, %#lx)\n" ,page_to_pfn (poisoned_page), poisoned_addr, vma->vm_start, vma->vm_end); +continue ; } + + ++hwp_page->nr_expected_unmaps; } }#else
mm/memory-failure.c static void try_to_split_huge_mapping(struct folio *folio, · line 1567
Quoted 8 lines
mm/memory-failure.c +5
-0@@ -1546,6 +1560,47 @@ static void try_to_split_huge_mapping(struct folio *folio, }#endif /* CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING */ +static bool is_unmapping_successful (struct folio *folio, +struct page *poisoned_page) +{ +bool unmap_success = false; +struct raw_hwp_page *hwp_page =find_in_raw_hwp_list (folio, poisoned_page);
As a small optimization, this find_in_raw_hwp_list() can be called after the if-block below, because hwp_page is used only after it.
mm/memory-failure.c static void try_to_split_huge_mapping(struct folio *folio, · line 1572
Quoted 5 lines
mm/memory-failure.c +5
-0+ +if (!folio_test_hugetlb (folio) || +folio_test_anon (folio) || + !IS_ENABLED (CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING )) { + unmap_success =folio_mapped (folio);
Maybe you mean "unmap_success = !folio_mapped(folio)"n ?
Quoted 5 lines
mm/memory-failure.c +5
-0+if (!unmap_success) +pr_err ("%#lx: failed to unmap page (mapcount=%d)\n" , +page_to_pfn (poisoned_page), +page_mapcount (folio_page (folio,0 ))); +
mm/memory-failure.c static void try_to_split_huge_mapping(struct folio *folio, · line 1587
Quoted 10 lines
mm/memory-failure.c +10
-0+return unmap_success; + } + +VM_BUG_ON_PAGE (!hwp_page, poisoned_page); + +/* +* Unmapping may not happen for some VMA: +* - HGM-eligible VMA but @poisoned_page is not faulted yet: nothing +* needs to be done at this point yet until page fault handling. +* - HGM-non-eliggible VMA: mapcount decreases by nr_subpages for each VMA,
s/eliggible/eligible/
Quoted 85 lines
mm/memory-failure.c +47
-5+* but not tracked so cannot tell if successfully unmapped from such VMA. +*/ +if (hwp_page->nr_vmas_mapped != hwp_page->nr_expected_unmaps) +pr_info ("%#lx: mapped by %d VMAs but %d unmappings are expected\n" , +page_to_pfn (poisoned_page), hwp_page->nr_vmas_mapped, + hwp_page->nr_expected_unmaps); + + unmap_success = hwp_page->nr_expected_unmaps == hwp_page->nr_actual_unmaps; + +if (!unmap_success) +pr_err ("%#lx: failed to unmap page (folio_mapcount=%d)\n" , +page_to_pfn (poisoned_page),folio_mapcount (folio)); + +return unmap_success; +} +/* * Do all that is necessary to remove user space mappings. Unmap * the pages and send SIGBUS to the processes if the data was dirty. @@ -1631,10 +1686,7 @@ static bool hwpoison_user_mappings(struct page *p, unsigned long pfn,try_to_unmap (folio, ttu); } - unmap_success = !page_mapped (hpage); -if (!unmap_success) -pr_err ("%#lx: failed to unmap page (mapcount=%d)\n" , - pfn,page_mapcount (hpage)); + unmap_success =is_unmapping_successful (folio, p);/* * try_to_unmap() might put mlocked page in lru cache, so call @@ -1827,6 +1879,31 @@ EXPORT_SYMBOL_GPL(mf_dax_kill_procs);#ifdef CONFIG_HUGETLB_PAGE +/* +* Given a HWPOISON @subpage as raw page, find its location in @folio's +* _hugetlb_hwpoison. Return NULL if @subpage is not in the list. +*/ +struct raw_hwp_page *find_in_raw_hwp_list (struct folio *folio, +struct page *subpage) +{ +struct llist_node *t, *tnode; +struct llist_head *raw_hwp_head =raw_hwp_list_head (folio); +struct raw_hwp_page *hwp_page =NULL ; +struct raw_hwp_page *p; + +VM_BUG_ON_PAGE (PageHWPoison (subpage), subpage); + +llist_for_each_safe (tnode, t, raw_hwp_head->first) { + p = container_of(tnode, struct raw_hwp_page, node); +if (subpage == p->page) { + hwp_page = p; +break ; + } + } + +return hwp_page; +} +static unsigned long __folio_free_raw_hwp (struct folio *folio,bool move_flag) {struct llist_head *head;@@ -1837,6 +1914,9 @@ static unsigned long __folio_free_raw_hwp(struct folio *folio, bool move_flag)llist_for_each_safe (tnode, t, head->first) { struct raw_hwp_page *p = container_of(tnode, struct raw_hwp_page, node); +/* Ideally raw HWPoison pages are fully unmapped if possible. */ +WARN_ON (p->nr_expected_unmaps != p->nr_actual_unmaps); +if (move_flag)SetPageHWPoison (p->page);else @@ -1853,7 +1933,8 @@ static int folio_set_hugetlb_hwpoison(struct folio *folio, struct page *page)struct llist_head *head;struct raw_hwp_page *raw_hwp;struct llist_node *t, *tnode; -int ret =folio_test_set_hwpoison (folio) ? -EHWPOISON :0 ; +bool has_hwpoison =folio_test_set_hwpoison (folio); +bool hgm_enabled =IS_ENABLED (CONFIG_HUGETLB_HIGH_GRANULARITY_MAPPING );/* * Once the hwpoison hugepage has lost reliable raw error info, @@ -1873,9 +1954,20 @@ static int folio_set_hugetlb_hwpoison(struct folio *folio, struct page *page) raw_hwp =kmalloc (sizeof (struct raw_hwp_page ),GFP_ATOMIC );
mm/memory-failure.c static int folio_set_hugetlb_hwpoison(struct folio *folio, struct page *page) · line 1964
Quoted 10 lines
mm/memory-failure.c +7
-0if (raw_hwp) { raw_hwp->page = page; + raw_hwp->nr_vmas_mapped =0 ; + raw_hwp->nr_expected_unmaps =0 ; + raw_hwp->nr_actual_unmaps =0 ;llist_add (&raw_hwp->node, head); +if (hgm_enabled) +/* +* A new raw poisoned page. Don't return +* HWPOISON. Error event will be counted
(Sorry nitpicking ...)
"Don't return -EHWPOISON" ?
Quoted 37 lines
mm/memory-failure.c +15
-2+* in action_result(). +*/ +return 0 ; +/* the first error event will be counted in action_result(). */ -if (ret) +if (has_hwpoison)num_poisoned_pages_inc (page_to_pfn (page)); }else {/* @@ -1889,8 +1981,16 @@ static int folio_set_hugetlb_hwpoison(struct folio *folio, struct page *page)* used any more, so free it. */ __folio_free_raw_hwp (folio, false); + +/* +* HGM relies on raw_hwp allocated and inserted to raw_hwp_list. +*/ +if (hgm_enabled) +return -ENOMEM ; } -return ret; + +BUG_ON (hgm_enabled); +return has_hwpoison ? -EHWPOISON :0 ; }static unsigned long folio_free_raw_hwp (struct folio *folio,bool move_flag)@@ -1936,6 +2036,7 @@ int __get_huge_page_for_hwpoison(unsigned long pfn, int flags,struct page *page =pfn_to_page (pfn);struct folio *folio =page_folio (page);int ret =2 ;/* fallback to normal page handling */ +int set_page_hwpoison =0 ;bool count_increased = false;if (!folio_test_hugetlb (folio))
mm/memory-failure.c int __get_huge_page_for_hwpoison(unsigned long pfn, int flags, · line 2062
Quoted 8 lines
mm/memory-failure.c +3
-2@@ -1956,8 +2057,9 @@ int __get_huge_page_for_hwpoison(unsigned long pfn, int flags, goto out; } -if (folio_set_hugetlb_hwpoison (folio, page)) { - ret = -EHWPOISON ; + set_page_hwpoison =folio_set_hugetlb_hwpoison (folio, page); +if (set_page_hwpoison) { + ret = set_page_hwpoison;
This adds a new error return code, so could you update comment on this function?
Thanks,
Naoya Horiguchi
Quoted 83 lines
mm/memory-failure.c +1
-1goto out; }@@ -2004,7 +2106,7 @@ static int try_memory_failure_hugetlb(unsigned long pfn, int flags, int *hugetlb res =kill_accessing_process (current,folio_pfn (folio), flags); }return res; - }else if (res == -EBUSY ) { + }else if (res == -EBUSY || res == -ENOMEM ) {if (!(flags &MF_NO_RETRY )) { flags |=MF_NO_RETRY ; goto retry;mm/rmap.c +35
-3@@ -1453,6 +1453,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,struct mmu_notifier_range range;enum ttu_flags flags = (enum ttu_flags )(long )arg;bool page_poisoned; +bool hgm_eligible =hugetlb_hgm_eligible (vma); +struct raw_hwp_page *hwp_page;/* * When racing against e.g. zap_pte_range() on another cpu, @@ -1525,6 +1527,29 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,* in the case where the hugetlb page is poisoned. */ VM_BUG_ON_FOLIO (!page_poisoned, folio); + +/* +* When VMA is not HGM eligible, unmap at hugepage's +* original P*D. +* +* When HGM is eligible: +* - if original P*D is split to smaller P*Ds and +* PTEs, we skip subpage if it is not raw HWPoison +* page, or it was but was already unmapped. +* - if original P*D is not split, skip unmapping +* and memory_failure result will be MF_IGNORED. +*/ +if (hgm_eligible) { +if (pvmw.pte_order >0 ) +continue ; + hwp_page =find_in_raw_hwp_list (folio, subpage); +if (hwp_page ==NULL ) +continue ; +if (hwp_page->nr_expected_unmaps == + hwp_page->nr_actual_unmaps) +continue ; + } +/* * huge_pmd_unshare may unmap an entire PMD page. * There is no way of knowing exactly which PMDs may @@ -1760,12 +1785,19 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,* * See Documentation/mm/mmu_notifier.rst */ -if (folio_test_hugetlb (folio)) +if (!folio_test_hugetlb (folio)) +page_remove_rmap (subpage, vma, false); +else {hugetlb_remove_rmap (subpage, pvmw.pte_order +PAGE_SHIFT ,hstate_vma (vma), vma); -else -page_remove_rmap (subpage, vma, false); +if (hgm_eligible) { +VM_BUG_ON_FOLIO (pvmw.pte_order >0 , folio); +VM_BUG_ON_FOLIO (!hwp_page, folio); +VM_BUG_ON_FOLIO (subpage != hwp_page->page, folio); + ++hwp_page->nr_actual_unmaps; + } + }if (vma->vm_flags &VM_LOCKED )mlock_drain_local ();Signature · 1 line
2.40.1.495.gc816e09b53d-goog
RFC PATCH 5/7 hugetlb: only VM_FAULT_HWPOISON_LARGE raw page
In reply to Jiaqi YanMemory raw pages can become HWPOISON between when userspace maps a hugepage and when userspace faults in the hugepage.
Today when hugetlb faults somewhere in a hugepage containing HWPOISON raw pages, the result is a VM_FAULT_HWPOISON_LARGE.
This commit teaches hugetlb page fault handler to only VM_FAULT_HWPOISON_LARGE if the faulting address is within HWPOISON raw page; otherwise, fault handler can continue to fault in healthy raw pages.
--- include/linux/mm.h | 2 + mm/hugetlb.c | 129 ++++++++++++++++++++++++++++++++++++++++++-- mm/memory-failure.c | 1 + 3 files changed, 127 insertions(+), 5 deletions(-)
include/linux/mm.h +2 -0
@@ -3531,6 +3531,7 @@ extern const struct attribute_group memory_failure_attr_group;* @nr_expected_unmaps: if a VMA that maps @page when detected is eligible * for high granularity mapping, @page is expected to be unmapped. * @nr_actual_unmaps: how many times the raw page is actually unmapped. +* @index: index of the poisoned subpage in the folio. */ struct raw_hwp_page {struct llist_node node;
@@ -3538,6 +3539,7 @@ struct raw_hwp_page {int nr_vmas_mapped;int nr_expected_unmaps;int nr_actual_unmaps; +unsigned long index; };#ifdef CONFIG_HUGETLB_PAGE
mm/hugetlb.c +124 -5
155 lines not on this page · raw message
mm/memory-failure.c +1 -0
@@ -1957,6 +1957,7 @@ static int folio_set_hugetlb_hwpoison(struct folio *folio, struct page *page) raw_hwp->nr_vmas_mapped =0 ; raw_hwp->nr_expected_unmaps =0 ; raw_hwp->nr_actual_unmaps =0 ; + raw_hwp->index =folio_page_idx (folio, page);llist_add (&raw_hwp->node, head);if (hgm_enabled)/*
Signature · 1 line
2.40.1.495.gc816e09b53d-goog
RFC PATCH 6/7 selftest/mm: test PAGESIZE unmapping HWPOISON pages
In reply to Jiaqi YanAfter injecting memory errors to byte addresses inside HugeTLB page, the updated test checks
1. only a raw page is unmapped, and userspace gets correct SIGBUS from kernel.
2. other subpages in the same hugepage are still mapped and data not corrupted.
--- tools/testing/selftests/mm/hugetlb-hgm.c | 194 +++++++++++++++++++---- 1 file changed, 167 insertions(+), 27 deletions(-)
tools/testing/selftests/mm/hugetlb-hgm.c +167 -27
@@ -39,6 +39,10 @@#define MADV_SPLIT 26#endif +#ifndef NUM_HWPOISON_PAGES +#define NUM_HWPOISON_PAGES 3UL +#endif +#define PREFIX " ... "#define ERROR_PREFIX " !!! "
@@ -241,6 +245,9 @@ static int test_sigbus(char *addr, bool poison) sigbus_addr, addr); else if (poison && !was_mceerr) printf(ERROR_PREFIX "didn't get an MCEERR?\n"); + else if (!poison && was_mceerr) + printf(ERROR_PREFIX "got BUS_MCEERR_AR sigbus on expected healthy address: %p\n", + sigbus_addr); else ret =0 ; out:
@@ -272,43 +279,176 @@ static int read_event_from_uffd(int *uffd, pthread_t *pthread)return 0 ; } -static int test_sigbus_range (char *primary_map,size_t len,bool hwpoison) +struct range_exclude_pages { +/* Starting address of the buffer. */ +char *mapping; +/* Length of the buffer in bytes. */ +size_t length; +/* The value that each byte in buffer should equal to. */ +char value; +/* +* PAGESIZE aligned addresses excluded from the checking, +* e.g. if PAGE_SIZE=4k, for each addr in excludes, +* skips checking on [addr, addr + 4096). +*/ +unsigned long excluded[NUM_HWPOISON_PAGES ]; +}; + +static int check_range_exclude_pages (struct range_exclude_pages *range) +{ +const unsigned long pagesize =getpagesize (); +unsigned long excluded_index; +unsigned long page_index; +bool should_skip; +size_t i =0 ; +size_t j =0 ; + +while (i < range->length) { + page_index = ((unsigned long )(range->mapping + i)) / pagesize; + should_skip = false; +for (j =0 ; j <NUM_HWPOISON_PAGES ; ++j) { + excluded_index = range->excluded[j] / pagesize; +if (page_index == excluded_index) { + should_skip = true; +break ; + } + } +if (should_skip) { +printf (PREFIX "skip excluded addr range [%#lx, %#lx)\n" , + (unsigned long )(range->mapping + i), + (unsigned long )(range->mapping + i + pagesize)); + i += pagesize; +continue ; + } +if (range->mapping[i] != range->value) { +printf (ERROR_PREFIX "mismatch at %p (%d != %d)\n" , + &range->mapping[i], range->mapping[i], range->value); +return -1 ; + } + ++i; + } + +return 0 ; +} + +enum test_status verify_raw_pages (char *map,size_t len, +unsigned long excluded[NUM_HWPOISON_PAGES ]) {const unsigned long pagesize =getpagesize (); -const int num_checks =512 ; -unsigned long bytes_per_check = len/num_checks; -int i; +unsigned long size, offset, value; +size_t j =0 ; + +for (size = len /2 , offset =0 , value =1 ; size > pagesize; + offset += size, size /=2 , ++value) { +struct range_exclude_pages range = { + .mapping = map + offset, + .length = size, + .value = value, + }; +for (j =0 ; j <NUM_HWPOISON_PAGES ; ++j) + range.excluded[j] = excluded[j]; + +printf (PREFIX "checking non-poisoned range [%p, %p) " +"(len=%#lx) per-byte value=%lu\n" , + range.mapping, range.mapping + range.length, + range.length, value); +if (check_range_exclude_pages (&range)) +return TEST_FAILED ; + + printf(PREFIX PREFIX "good\n"); + } -printf (PREFIX "checking that we can't access " -"(%d addresses within %p -> %p)\n" , - num_checks, primary_map, primary_map + len); +return TEST_PASSED ; +} -if (pagesize > bytes_per_check) - bytes_per_check = pagesize; +static int read_hwpoison_pages (unsigned long *nr_hwp_pages) +{ +const unsigned long pagesize =getpagesize (); +char buffer[256 ] = {0 }; +char *cmd ="cat /proc/meminfo | grep -i HardwareCorrupted | grep -o '[0-9]*'" ; +FILE *cmdfile =popen (cmd,"r" ); -for (i =0 ; i < len; i += bytes_per_check) -if (test_sigbus (primary_map + i, hwpoison) <0 ) -return 1 ; -/* check very last byte, because we left it unmapped */ -if (test_sigbus (primary_map + len -1 , hwpoison)) -return 1 ; +if (!(fgets (buffer,sizeof (buffer), cmdfile))) { +perror ("failed to read HardwareCorrupted from /proc/meminfo\n" ); +return -1 ; + } +pclose (cmdfile); + *nr_hwp_pages =atoll (buffer) *1024 / pagesize;return 0 ; } -static enum test_status test_hwpoison (char *primary_map,size_t len) +static enum test_status test_hwpoison_one_raw_page (char *hwpoison_addr) { -printf (PREFIX "poisoning %p -> %p\n" , primary_map, primary_map + len); -if (madvise (primary_map, len,MADV_HWPOISON ) <0 ) { +const unsigned long pagesize =getpagesize (); + +printf (PREFIX "poisoning [%p, %p) (len=%#lx)\n" , + hwpoison_addr, hwpoison_addr + pagesize, pagesize); +if (madvise (hwpoison_addr, pagesize,MADV_HWPOISON ) <0 ) {perror (ERROR_PREFIX "MADV_HWPOISON failed" );return TEST_SKIPPED ; } -return test_sigbus_range (primary_map, len, true) - ?TEST_FAILED :TEST_PASSED ; +printf (PREFIX "checking poisoned range [%p, %p) (len=%#lx)\n" , + hwpoison_addr, hwpoison_addr + pagesize, pagesize); +if (test_sigbus (hwpoison_addr, true) <0 ) +return TEST_FAILED ; + +return TEST_PASSED ; } -static int test_fork (int uffd,char *primary_map,size_t len) +static enum test_status test_hwpoison_present (char *map,size_t len, +bool already_injected) +{ +const unsigned long pagesize =getpagesize (); +const unsigned long hwpoison_next =128 ; +unsigned long nr_hwpoison_pages_before, nr_hwpoison_pages_after; +enum test_status ret; +size_t i; +char *hwpoison_addr = map; +unsigned long hwpoison_addrs[NUM_HWPOISON_PAGES ]; + +if (hwpoison_next * (NUM_HWPOISON_PAGES -1 ) >= (len / pagesize)) { +printf (ERROR_PREFIX "max hwpoison_addr out of range" ); +return TEST_SKIPPED ; + } + +for (i =0 ; i <NUM_HWPOISON_PAGES ; ++i) { + hwpoison_addrs[i] = (unsigned long )hwpoison_addr; + hwpoison_addr += hwpoison_next * pagesize; + } + +if (already_injected) +return verify_raw_pages (map, len, hwpoison_addrs); + +if (read_hwpoison_pages (&nr_hwpoison_pages_before)) { +printf (ERROR_PREFIX "check #HWPOISON pages\n" ); +return TEST_SKIPPED ; + } +printf (PREFIX "Before injections, #HWPOISON pages = %ld\n" , nr_hwpoison_pages_before); + +for (i =0 ; i <NUM_HWPOISON_PAGES ; ++i) { + ret =test_hwpoison_one_raw_page ((char *)hwpoison_addrs[i]); +if (ret !=TEST_PASSED ) +return ret; + } + +if (read_hwpoison_pages (&nr_hwpoison_pages_after)) { +printf (ERROR_PREFIX "check #HWPOISON pages\n" ); +return TEST_SKIPPED ; + } +printf (PREFIX "After injections, #HWPOISON pages = %ld\n" , nr_hwpoison_pages_after); + +if (nr_hwpoison_pages_after - nr_hwpoison_pages_before !=NUM_HWPOISON_PAGES ) { +printf (ERROR_PREFIX "delta #HWPOISON pages != %ld" , +NUM_HWPOISON_PAGES ); +return TEST_FAILED ; + } + +return verify_raw_pages (map, len, hwpoison_addrs); +} + +int test_fork (int uffd,char *primary_map,size_t len) {int status;int ret =0 ;
@@ -360,7 +500,6 @@ static int test_fork(int uffd, char *primary_map, size_t len)pthread_join (uffd_thd,NULL );return ret; - } static int uffd_register(int uffd, char *primary_map, unsigned long len,
@@ -394,6 +533,7 @@ test_hgm(int fd, size_t hugepagesize, size_t len, enum test_type type)bool uffd_wp = type ==TEST_UFFDWP ;bool verify = type ==TEST_DEFAULT ;int register_args; +enum test_status hwp_status =TEST_SKIPPED ;if (ftruncate (fd, len) <0 ) {perror (ERROR_PREFIX "ftruncate failed" );
@@ -489,10 +629,10 @@ test_hgm(int fd, size_t hugepagesize, size_t len, enum test_type type)* mapping. */ if (hwpoison) { -enum test_status new_status =test_hwpoison (primary_map, len); - -if (new_status !=TEST_PASSED ) { - status = new_status; +/* test_hwpoison can fail with TEST_SKIPPED. */ + hwp_status =test_hwpoison_present (primary_map, len, false); +if (hwp_status !=TEST_PASSED ) { + status = hwp_status; goto done; } }
@@ -539,7 +679,7 @@ test_hgm(int fd, size_t hugepagesize, size_t len, enum test_type type)/* * Verify that memory is still poisoned. */ -if (hwpoison &&test_sigbus_range (primary_map, len, true)) +if (hwpoison &&test_hwpoison_present (primary_map, len, true)) goto done; status =TEST_PASSED ;
Signature · 1 line
2.40.1.495.gc816e09b53d-goog
RFC PATCH 7/7 selftest/mm: test PAGESIZE unmapping UFFD WP marker HWPOISON pages
In reply to Jiaqi YanFor not-yet-faulted hugepage containing HWPOISON raw page, test
1. only HWPOISON raw page will not be faulted, and a BUS_MCEERR_AR SIGBUS will be sent to userspace.
2. healthy raw pages are faulted in as normal. Since the hugepage has been writeprotect by UFFD, non BUS_MCEERR_AR SIGBUS will be sent to userspace.
--- tools/testing/selftests/mm/hugetlb-hgm.c | 170 +++++++++++++++++++++++ 1 file changed, 170 insertions(+)
tools/testing/selftests/mm/hugetlb-hgm.c +170 -0
@@ -515,6 +515,169 @@ static int uffd_register(int uffd, char *primary_map, unsigned long len,return ioctl (uffd,UFFDIO_REGISTER , ®); } +static int setup_present_map (char *present_map,size_t len) +{ +size_t offset =0 ; +unsigned char iter =0 ; +unsigned long pagesize =getpagesize (); +uint64_t size; + +for (size = len/2 ; size >= pagesize; + offset += size, size /=2 ) { + iter++; +memset (present_map + offset, iter, size); + } +return 0 ; +} + +static enum test_status test_hwpoison_absent_uffd_wp (int fd,size_t hugepagesize,size_t len) +{ +int uffd; +char *absent_map, *present_map; +struct uffdio_api api; +int register_args; +struct sigaction new, old; +enum test_status status =TEST_SKIPPED ; +const unsigned long pagesize =getpagesize (); +const unsigned long hwpoison_index =128 ; +char *hwpoison_addr; + +if (hwpoison_index >= (len / pagesize)) { +printf (ERROR_PREFIX "hwpoison_index out of range" ); +return TEST_FAILED ; + } + +if (ftruncate (fd, len) <0 ) { +perror (ERROR_PREFIX "ftruncate failed" ); +return TEST_FAILED ; + } + + uffd =userfaultfd (O_CLOEXEC ); +if (uffd <0 ) { +perror (ERROR_PREFIX "uffd not created" ); +return TEST_FAILED ; + } + + absent_map =mmap (NULL , len,PROT_READ |PROT_WRITE ,MAP_SHARED , fd,0 ); +if (absent_map ==MAP_FAILED ) { +perror (ERROR_PREFIX "mmap for ABSENT mapping failed" ); + goto close_uffd; + } +printf (PREFIX "ABSENT mapping: %p\n" , absent_map); + + api.api =UFFD_API ; + api.features =UFFD_FEATURE_SIGBUS |UFFD_FEATURE_EXACT_ADDRESS | +UFFD_FEATURE_EVENT_FORK ; +if (ioctl (uffd,UFFDIO_API , &api) ==-1 ) { +perror (ERROR_PREFIX "UFFDIO_API failed" ); + goto unmap_absent; + } + +/* +* Register with UFFDIO_REGISTER_MODE_WP to have UFFD WP bit on +* the HugeTLB page table entry. +*/ + register_args =UFFDIO_REGISTER_MODE_MISSING |UFFDIO_REGISTER_MODE_WP ; +if (uffd_register (uffd, absent_map, len, register_args)) { +perror (ERROR_PREFIX "UFFDIO_REGISTER failed" ); + goto unmap_absent; + } + + new.sa_sigaction = &sigbus_handler; + new.sa_flags =SA_SIGINFO ; +if (sigaction (SIGBUS , &new, &old) <0 ) { +perror (ERROR_PREFIX "could not setup SIGBUS handler" ); + goto unmap_absent; + } + +/* +* Set WP markers to the absent huge mapping. With HGM enabled in +* kernel CONFIG, memory_failure will enabled HGM in kernel, +* so no need to enable HGM from userspace. +*/ +if (userfaultfd_writeprotect (uffd, absent_map, len, true) <0 ) { + status =TEST_FAILED ; + goto unmap_absent; + } + + status =TEST_PASSED ; + +/* +* With MAP_SHARED hugetlb memory, we cna inject memory error to +* not-yet-faulted mapping (absent_map) by injecting memory error +* to a already faulted mapping (present_map). +*/ + present_map =mmap (NULL , len,PROT_READ |PROT_WRITE ,MAP_SHARED , fd,0 ); +if (present_map ==MAP_FAILED ) { +perror (ERROR_PREFIX "mmap for non present mapping failed" ); + goto close_uffd; + } +printf (PREFIX "PRESENT mapping: %p\n" , present_map); +setup_present_map (present_map, len); + + hwpoison_addr = present_map + hwpoison_index * pagesize; +if (madvise (hwpoison_addr, pagesize,MADV_HWPOISON )) { +perror (PREFIX "MADV_HWPOISON a page in PRESENT mapping failed" ); + status =TEST_FAILED ; + goto unmap_present; + } + +printf (PREFIX "checking poisoned range [%p, %p) (len=%#lx) in PRESENT mapping\n" , + hwpoison_addr, hwpoison_addr + pagesize, pagesize); +if (test_sigbus (hwpoison_addr, true) <0 ) { + status =TEST_FAILED ; + goto done; + } +printf (PREFIX "checking healthy pages in PRESENT mapping\n" ); +unsigned long hwpoison_addrs[] = { + (unsigned long )hwpoison_addr, + (unsigned long )hwpoison_addr, + (unsigned long )hwpoison_addr + }; + status =verify_raw_pages (present_map, len, hwpoison_addrs); +if (status !=TEST_PASSED ) { +printf (ERROR_PREFIX "checking healthy pages failed\n" ); + goto done; + } + +for (int i =0 ; i < len; i += pagesize) { +if (i == hwpoison_index * pagesize) { +printf (PREFIX "checking poisoned range [%p, %p) (len=%#lx) in ABSENT mapping\n" , + absent_map + i, absent_map + i + pagesize, pagesize); +if (test_sigbus (absent_map + i, true) <0 ) { + status =TEST_FAILED ; +break ; + } + }else { +/* +* With UFFD_FEATURE_SIGBUS, we should get a SIGBUS for +* every not faulted (non present) page/byte. +*/ +if (test_sigbus (absent_map + i, false) <0 ) { +printf (PREFIX "checking healthy range [%p, %p) (len=%#lx) in ABSENT mapping failed\n" , + absent_map + i, absent_map + i + pagesize, pagesize); + status =TEST_FAILED ; +break ; + } + } + } +done: +if (ftruncate (fd,0 ) <0 ) { +perror (ERROR_PREFIX "ftruncate back to 0 failed" ); + status =TEST_FAILED ; + } +unmap_present: +printf (PREFIX "Unmap PRESENT mapping=%p\n" , absent_map); +munmap (present_map, len); +unmap_absent: +printf (PREFIX "Unmap ABSENT mapping=%p\n" , absent_map); +munmap (absent_map, len); +close_uffd: +printf (PREFIX "Close UFFD\n" ); +close (uffd); +return status; +} +enum test_type {TEST_DEFAULT ,TEST_UFFDWP ,
@@ -744,6 +907,13 @@ int main(void)printf ("HGM hwpoison test: %s\n" ,status_to_str (status));if (status ==TEST_FAILED ) ret =-1 ; + +printf ("HGM hwpoison UFFD-WP marker test...\n" ); + status =test_hwpoison_absent_uffd_wp (fd, hugepagesize, len); +printf ("HGM hwpoison UFFD-WP marker test: %s\n" , +status_to_str (status)); +if (status ==TEST_FAILED ) + ret =-1 ; close:close (fd);
Signature · 1 line
2.40.1.495.gc816e09b53d-goog