Improve hugetlbfs read on HWPOISON hugepages
Subsystems (6)
v3 cover Improve hugetlbfs read on HWPOISON hugepages
Today when hardware memory is corrupted in a hugetlb hugepage, kernel leaves the hugepage in pagecache [1]; otherwise future mmap or read will suject to silent data corruption. This is implemented by returning -EIO from hugetlb_read_iter immediately if the hugepage has HWPOISON flag set.
Since memory_failure already tracks the raw HWPOISON subpages in a hugepage, a natural improvement is possible: if userspace only asks for healthy subpages in the pagecache, kernel can return these data.
This patchset implements this improvement. It consist of three parts. The 1st commit exports the functionality to tell if a subpage inside a hugetlb hugepage is a raw HWPOISON page. The 2nd commit teaches hugetlbfs_read_iter to return as many healthy bytes as possible. The 3rd commit properly tests this new feature.
[1] commit 8625147cafaa ("hugetlbfs: don't delete error page from pagecache")
Changelog
v2 => v3
* Updates commit messages for future reader to know background and code details
* v3 is based on commit 5bb367dca2b9 ("Merge branch 'master' into mm-stable")
v1 => v2
* __folio_free_raw_hwp deletes all entries in raw_hwp_list before it traverses and frees raw_hwp_page.
* find_raw_hwp_page => __is_raw_hwp_subpage and __is_raw_hwp_subpage only returns bool instead of a raw_hwp_page entry.
* is_raw_hwp_subpage holds hugetlb_lock while checking __is_raw_hwp_subpage.
* No need to do folio_lock in adjust_range_hwpoison.
* v2 is based on commit a6e79df92e4a ("mm/gup: disallow FOLL_LONGTERM GUP-fast writing to file-backed mappings")
Jiaqi Yan (4):
mm/hwpoison: delete all entries before traversal in
__folio_free_raw_hwp
mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
hugetlbfs: improve read HWPOISON hugepage
selftests/mm: add tests for HWPOISON hugetlbfs read
fs/hugetlbfs/inode.c | 58 +++- include/linux/hugetlb.h | 19 ++ include/linux/mm.h | 7 + mm/hugetlb.c | 10 + mm/memory-failure.c | 42 ++- tools/testing/selftests/mm/.gitignore | 1 + tools/testing/selftests/mm/Makefile | 1 + .../selftests/mm/hugetlb-read-hwpoison.c | 322 ++++++++++++++++++ 8 files changed, 439 insertions(+), 21 deletions(-) create mode 100644 tools/testing/selftests/mm/hugetlb-read-hwpoison.c
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
PATCH v3 1/4 mm/hwpoison: delete all entries before traversal in __folio_free_raw_hwp
Traversal on llist (e.g. llist_for_each_safe) is only safe AFTER entries are deleted from the llist. Correct the way __folio_free_raw_hwp deletes and frees raw_hwp_page entries in raw_hwp_list: first llist_del_all, then kfree within llist_for_each_safe.
As of today, concurrent adding, deleting, and traversal on raw_hwp_list from hugetlb.c and/or memory-failure.c are fine with each other. Note this is guaranteed partly by the lock-free nature of llist, and partly by holding hugetlb_lock and/or mf_mutex. For example, as llist_del_all is lock-free with itself, folio_clear_hugetlb_hwpoison()s from __update_and_free_hugetlb_folio and memory_failure won't need explicit locking when freeing the raw_hwp_list. New code that manipulates raw_hwp_list must be careful to ensure the concurrency correctness.
--- mm/memory-failure.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
mm/memory-failure.c +3 -5
@@ -1829,12 +1829,11 @@ static inline struct llist_head *raw_hwp_list_head(struct folio *folio)static unsigned long __folio_free_raw_hwp (struct folio *folio,bool move_flag) { -struct llist_head *head; -struct llist_node *t, *tnode; +struct llist_node *t, *tnode, *head;unsigned long count =0 ; - head =raw_hwp_list_head (folio); -llist_for_each_safe (tnode, t, head->first) { + head =llist_del_all (raw_hwp_list_head (folio)); +llist_for_each_safe (tnode, t, head) { struct raw_hwp_page *p = container_of(tnode, struct raw_hwp_page, node);if (move_flag)
@@ -1844,7 +1843,6 @@ static unsigned long __folio_free_raw_hwp(struct folio *folio, bool move_flag)kfree (p); count++; } -llist_del_all (head);return count; }
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
Re: [PATCH v3 1/4] mm/hwpoison: delete all entries before traversal in __folio_free_raw_hwp
Quoted 7 lines · Jiaqi Yan
Traversal on llist (e.g.
llist_for_each_safe) is only safe AFTER entries are deleted from the llist. Correct the way__folio_free_raw_hwpdeletes and freesraw_hwp_pageentries inraw_hwp_list: firstllist_del_all, then kfree withinllist_for_each_safe.As of today, concurrent adding, deleting, and traversal on
raw_hwp_listfromhugetlb.cand/ormemory-failure.care fine with each other. Note
I think there's a race on freeing the raw_hwp_list between unpoison_memory and __update_and_free_hugetlb_folio:
unpoison_memory __update_and_free_hugetlb_folio
if (folio_test_hwpoison)
folio_clear_hugetlb_hwpoison
folio_free_raw_hwp folio_free_raw_hwp
folio_test_clear_hwpoison
unpoison_memory and __update_and_free_hugetlb_folio can traverse and free the raw_hwp_list at the same time. And I believe your patch will fix the problem. Thanks.
Quoted 10 lines
this is guaranteed partly by the lock-free nature of llist, and partly by holding
hugetlb_lockand/ormf_mutex. For example, asllist_del_allis lock-free with itself, folio_clear_hugetlb_hwpoison()s from__update_and_free_hugetlb_folioandmemory_failurewon't need explicit locking when freeing theraw_hwp_list. New code that manipulatesraw_hwp_listmust be careful to ensure the concurrency correctness.Acked-by Mike Kravetz <[email protected]>Acked-by Naoya Horiguchi <[email protected]>Signed-off-by Jiaqi Yan <[email protected]>
Anyway, this patch looks good to me.
Thanks.
PATCH v3 2/4 mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
In reply to Jiaqi YanAdd the functionality, is_raw_hwp_subpage, to tell if a subpage of a hugetlb folio is a raw HWPOISON page. This functionality relies on RawHwpUnreliable to be not set; otherwise hugepage's raw HWPOISON list becomes meaningless.
is_raw_hwp_subpage needs to hold hugetlb_lock in order to synchronize with __get_huge_page_for_hwpoison, who iterates and inserts an entry to raw_hwp_list. llist itself doesn't ensure insertion is synchornized with the iterating used by __is_raw_hwp_list. Caller can minimize the overhead of lock cycles by first checking if folio / head page's HWPOISON flag is set.
Exports this functionality to be immediately used in the read operation for hugetlbfs.
--- include/linux/hugetlb.h | 19 +++++++++++++++++++ include/linux/mm.h | 7 +++++++ mm/hugetlb.c | 10 ++++++++++ mm/memory-failure.c | 34 ++++++++++++++++++++++++---------- 4 files changed, 60 insertions(+), 10 deletions(-)
include/linux/hugetlb.h +19 -0
@@ -1007,6 +1007,25 @@ void hugetlb_register_node(struct node *node);void hugetlb_unregister_node (struct node *node);#endif +/* +* Struct raw_hwp_page represents information about "raw error page", +* constructing singly linked list from ->_hugetlb_hwpoison field of folio. +*/ +struct raw_hwp_page { +struct llist_node node; +struct page *page; +}; + +static inline struct llist_head *raw_hwp_list_head (struct folio *folio) +{ +return (struct llist_head *)&folio->_hugetlb_hwpoison; +} + +/* +* Check if a given raw @subpage in a hugepage @folio is HWPOISON. +*/ +bool is_raw_hwp_subpage (struct folio *folio,struct page *subpage); +#else /* CONFIG_HUGETLB_PAGE */ struct hstate {};
include/linux/mm.h +7 -0
@@ -3680,6 +3680,7 @@ extern const struct attribute_group memory_failure_attr_group;extern void memory_failure_queue (unsigned long pfn,int flags);extern int __get_huge_page_for_hwpoison (unsigned long pfn,int flags,bool *migratable_cleared); +extern bool __is_raw_hwp_subpage (struct folio *folio,struct page *subpage);void num_poisoned_pages_inc (unsigned long pfn);void num_poisoned_pages_sub (unsigned long pfn,long i);struct task_struct *task_early_kill (struct task_struct *tsk,int force_early);
@@ -3694,6 +3695,12 @@ static inline int __get_huge_page_for_hwpoison(unsigned long pfn, int flags,return 0 ; } +static inline bool __is_raw_hwp_subpage (struct folio *folio, +struct page *subpage) +{ +return false; +} +static inline void num_poisoned_pages_inc (unsigned long pfn) { }
mm/hugetlb.c +10 -0
@@ -7373,6 +7373,16 @@ int get_huge_page_for_hwpoison(unsigned long pfn, int flags,return ret; } +bool is_raw_hwp_subpage (struct folio *folio,struct page *subpage) +{ +bool ret; + +spin_lock_irq (&hugetlb_lock); + ret =__is_raw_hwp_subpage (folio, subpage); +spin_unlock_irq (&hugetlb_lock); +return ret; +} +void folio_putback_active_hugetlb (struct folio *folio) {spin_lock_irq (&hugetlb_lock);
mm/memory-failure.c +24 -10
@@ -1813,18 +1813,32 @@ EXPORT_SYMBOL_GPL(mf_dax_kill_procs);#endif /* CONFIG_FS_DAX */ #ifdef CONFIG_HUGETLB_PAGE -/* -* Struct raw_hwp_page represents information about "raw error page", -* constructing singly linked list from ->_hugetlb_hwpoison field of folio. -*/ -struct raw_hwp_page { -struct llist_node node; -struct page *page; -}; -static inline struct llist_head *raw_hwp_list_head (struct folio *folio) +bool __is_raw_hwp_subpage (struct folio *folio,struct page *subpage) { -return (struct llist_head *)&folio->_hugetlb_hwpoison; +struct llist_head *raw_hwp_head; +struct raw_hwp_page *p, *tmp; +bool ret = false; + +if (!folio_test_hwpoison (folio)) +return false; + +/* +* When RawHwpUnreliable is set, kernel lost track of which subpages +* are HWPOISON. So return as if ALL subpages are HWPOISONed. +*/ +if (folio_test_hugetlb_raw_hwp_unreliable (folio)) +return true; + + raw_hwp_head =raw_hwp_list_head (folio); +llist_for_each_entry_safe (p, tmp, raw_hwp_head->first, node) { +if (subpage == p->page) { + ret = true; +break ; + } + } + +return ret; }static unsigned long __folio_free_raw_hwp (struct folio *folio,bool move_flag)
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
Quoted 1 line · Jiaqi Yan
Add the functionality,
is_raw_hwp_subpage, to tell if a subpage of a
This is incorrect naming. "subpage" was needed before we had the folio concept, but it should not be used any more. We have folios and pages now.
Also, abbreviating "hwpoison" as "hwp" seems like a bad idea to me. hwp is already used as an acronym by acpi, intel_pstate, some clock drivers, an ethernet driver, and a scsi driver.
Andrew Morton
maintainerRe: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
Quoted 10 lines · On Fri, 7 Jul 2023 21:31:39 +0100 Matthew Wilcox <[email protected]> wrote:
Quoted 1 line · Jiaqi Yan
Add the functionality,
is_raw_hwp_subpage, to tell if a subpage of aThis is incorrect naming. "subpage" was needed before we had the folio concept, but it should not be used any more. We have folios and pages now.
Also, abbreviating "hwpoison" as "hwp" seems like a bad idea to me. hwp is already used as an acronym by acpi,
intel_pstate, some clock drivers, an ethernet driver, and a scsi driver.
Thanks, Matthew. I'll merge v3 so we hit next -next (which actually won't be until Monday), and add a note that v4 is expected.
Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
In reply to Matthew WilcoxQuoted 6 lines · Matthew Wilcox
Quoted 1 line · Jiaqi Yan
Add the functionality,
is_raw_hwp_subpage, to tell if a subpage of aThis is incorrect naming. "subpage" was needed before we had the folio concept, but it should not be used any more. We have folios and pages now.
I think we can address the raw hwpoison page by the offset in folio/hugepage to eliminate the concept of "subpage".
Quoted 4 lines
Also, abbreviating "hwpoison" as "hwp" seems like a bad idea to me. hwp is already used as an acronym by acpi,
intel_pstate, some clock drivers, an ethernet driver, and a scsi driver.
I originally introduced the abbreviation "hwp" to avoid using a lengthy function name such as "folio_test_hugetlb_raw_hwpoison_unreliable()." Therefore, I prefer using "rawhwp" instead of a longer form like "raw_hwpoison," although I don't expect any confusion between "hwp" and "rawhwp."
As for "hwp_walk", another user of "hwp" in in mm/memory-failure.c, we can easily substitute it with "hwpoison_walk."
Thanks,
Naoya Horiguchi
Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
Quoted 9 lines · Naoya Horiguchi
Quoted 6 lines · Matthew Wilcox
Quoted 1 line · Jiaqi Yan
Add the functionality,
is_raw_hwp_subpage, to tell if a subpage of aThis is incorrect naming. "subpage" was needed before we had the folio concept, but it should not be used any more. We have folios and pages now.
Thanks for your comment, Matthew.
Quoted 8 lines
I think we can address the raw hwpoison page by the offset in folio/hugepage to eliminate the concept of "subpage".
Quoted 4 lines
Also, abbreviating "hwpoison" as "hwp" seems like a bad idea to me. hwp is already used as an acronym by acpi,
intel_pstate, some clock drivers, an ethernet driver, and a scsi driver.
Quoted 5 lines
I originally introduced the abbreviation "hwp" to avoid using a lengthy function name such as "
folio_test_hugetlb_raw_hwpoison_unreliable()." Therefore, I prefer using "rawhwp" instead of a longer form like "raw_hwpoison," although I don't expect any confusion between "hwp" and "rawhwp."
These are names in my mind, what do you think?
* is_rawhwp_page_in_hugepage
* is_raw_hwpoison_page_in_hugepage // I prefer this one
* folio_test_hugetlb_raw_hwpoison_page
Quoted 2 lines
As for "
hwp_walk", another user of "hwp" in inmm/memory-failure.c, we can easily substitute it with "hwpoison_walk."
In this "hwp_walk" case, I also prefer "hwpoison" than "hwp". I can create a separate renaming patch.
Quoted 3 lines
Thanks,
Naoya Horiguchi
Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
Quoted 28 lines · Jiaqi Yan
Quoted 9 lines · Naoya Horiguchi
Quoted 6 lines · Matthew Wilcox
Quoted 1 line · Jiaqi Yan
Add the functionality,
is_raw_hwp_subpage, to tell if a subpage of aThis is incorrect naming. "subpage" was needed before we had the folio concept, but it should not be used any more. We have folios and pages now.
Thanks for your comment, Matthew.
Quoted 8 lines
I think we can address the raw hwpoison page by the offset in folio/hugepage to eliminate the concept of "subpage".
Quoted 4 lines
Also, abbreviating "hwpoison" as "hwp" seems like a bad idea to me. hwp is already used as an acronym by acpi,
intel_pstate, some clock drivers, an ethernet driver, and a scsi driver.Quoted 5 lines
I originally introduced the abbreviation "hwp" to avoid using a lengthy function name such as "
folio_test_hugetlb_raw_hwpoison_unreliable()." Therefore, I prefer using "rawhwp" instead of a longer form like "raw_hwpoison," although I don't expect any confusion between "hwp" and "rawhwp."
Quoted 3 lines
These are names in my mind, what do you think?
*is_rawhwp_page_in_hugepage
*is_raw_hwpoison_page_in_hugepage// I prefer this one
This one is fine to me.
Quoted 7 lines
*
folio_test_hugetlb_raw_hwpoison_pageQuoted 2 lines
As for "
hwp_walk", another user of "hwp" in inmm/memory-failure.c, we can easily substitute it with "hwpoison_walk."In this "
hwp_walk" case, I also prefer "hwpoison" than "hwp". I can create a separate renaming patch.
Great, thank you.
- Naoya Horiguchi
Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
In reply to Jiaqi YanQuoted 35 lines · Jiaqi Yan
Add the functionality,
is_raw_hwp_subpage, to tell if a subpage of a hugetlb folio is a raw HWPOISON page. This functionality relies on RawHwpUnreliable to be not set; otherwise hugepage's raw HWPOISON list becomes meaningless.
is_raw_hwp_subpageneeds to holdhugetlb_lockin order to synchronize with__get_huge_page_for_hwpoison, who iterates and inserts an entry toraw_hwp_list. llist itself doesn't ensure insertion is synchornized with the iterating used by__is_raw_hwp_list. Caller can minimize the overhead of lock cycles by first checking if folio / head page's HWPOISON flag is set.Exports this functionality to be immediately used in the read operation for hugetlbfs.
Reviewed-by Mike Kravetz <[email protected]>Reviewed-by Naoya Horiguchi <[email protected]>Signed-off-by Jiaqi Yan <[email protected]>--- include/linux/hugetlb.h | 19 +++++++++++++++++++ include/linux/mm.h | 7 +++++++ mm/hugetlb.c | 10 ++++++++++ mm/memory-failure.c | 34 ++++++++++++++++++++++++---------- 4 files changed, 60 insertions(+), 10 deletions(-)...
mm/memory-failure.c +7
-2-static inline struct llist_head *raw_hwp_list_head (struct folio *folio) +bool __is_raw_hwp_subpage (struct folio *folio,struct page *subpage) { -return (struct llist_head *)&folio->_hugetlb_hwpoison; +struct llist_head *raw_hwp_head; +struct raw_hwp_page *p, *tmp; +bool ret = false; + +if (!folio_test_hwpoison (folio)) +return false;
mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1834
Quoted 10 lines
mm/memory-failure.c +10
-0+ +/* +* When RawHwpUnreliable is set, kernel lost track of which subpages +* are HWPOISON. So return as if ALL subpages are HWPOISONed. +*/ +if (folio_test_hugetlb_raw_hwp_unreliable (folio)) +return true; + + raw_hwp_head =raw_hwp_list_head (folio); +llist_for_each_entry_safe (p, tmp, raw_hwp_head->first, node) {
Since we don't free the raw_hwp_list, does llist_for_each_entry works same as llist_for_each_entry_safe?
mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1842
Quoted 8 lines
mm/memory-failure.c +7
-0+if (subpage == p->page) { + ret = true; +break ; + } + } + +return ret; }
It seems there's a race between __is_raw_hwp_subpage and unpoison_memory:
unpoison_memory __is_raw_hwp_subpage
if (!folio_test_hwpoison(folio)) -- hwpoison is set
folio_free_raw_hwp llist_for_each_entry_safe raw_hwp_list
llist_del_all ..
folio_test_clear_hwpoison
But __is_raw_hwp_subpage is used in hugetlbfs, unpoison_memory couldn't reach here because there's a folio_mapping == NULL check before folio_free_raw_hwp.
Anyway, this patch looks good to me.
Thanks.
Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
Quoted 59 lines · Miaohe Lin
Quoted 35 lines · Jiaqi Yan
Add the functionality,
is_raw_hwp_subpage, to tell if a subpage of a hugetlb folio is a raw HWPOISON page. This functionality relies on RawHwpUnreliable to be not set; otherwise hugepage's raw HWPOISON list becomes meaningless.
is_raw_hwp_subpageneeds to holdhugetlb_lockin order to synchronize with__get_huge_page_for_hwpoison, who iterates and inserts an entry toraw_hwp_list. llist itself doesn't ensure insertion is synchornized with the iterating used by__is_raw_hwp_list. Caller can minimize the overhead of lock cycles by first checking if folio / head page's HWPOISON flag is set.Exports this functionality to be immediately used in the read operation for hugetlbfs.
Reviewed-by Mike Kravetz <[email protected]>Reviewed-by Naoya Horiguchi <[email protected]>Signed-off-by Jiaqi Yan <[email protected]>--- include/linux/hugetlb.h | 19 +++++++++++++++++++ include/linux/mm.h | 7 +++++++ mm/hugetlb.c | 10 ++++++++++ mm/memory-failure.c | 34 ++++++++++++++++++++++++---------- 4 files changed, 60 insertions(+), 10 deletions(-)...
mm/memory-failure.c +7
-2-static inline struct llist_head *raw_hwp_list_head (struct folio *folio) +bool __is_raw_hwp_subpage (struct folio *folio,struct page *subpage) { -return (struct llist_head *)&folio->_hugetlb_hwpoison; +struct llist_head *raw_hwp_head; +struct raw_hwp_page *p, *tmp; +bool ret = false; + +if (!folio_test_hwpoison (folio)) +return false;mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1834
Quoted 10 lines
mm/memory-failure.c +10
-0+ +/* +* When RawHwpUnreliable is set, kernel lost track of which subpages +* are HWPOISON. So return as if ALL subpages are HWPOISONed. +*/ +if (folio_test_hugetlb_raw_hwp_unreliable (folio)) +return true; + + raw_hwp_head =raw_hwp_list_head (folio); +llist_for_each_entry_safe (p, tmp, raw_hwp_head->first, node) {Since we don't free the
raw_hwp_list, doesllist_for_each_entryworks same asllist_for_each_entry_safe?mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1842
Quoted 8 lines
mm/memory-failure.c +7
-0+if (subpage == p->page) { + ret = true; +break ; + } + } + +return ret; }
Quoted 7 lines
It seems there's a race between
__is_raw_hwp_subpageandunpoison_memory:
unpoison_memory__is_raw_hwp_subpage
if (!folio_test_hwpoison(folio)) -- hwpoison is set
folio_free_raw_hwpllist_for_each_entry_saferaw_hwp_list
llist_del_all..
folio_test_clear_hwpoison
Thanks Miaohe for raising this concern.
Quoted 2 lines
But
__is_raw_hwp_subpageis used in hugetlbfs,unpoison_memorycouldn't reach here because there's afolio_mapping== NULL check beforefolio_free_raw_hwp.
I agree. But in near future I do want to make __is_raw_hwp_subpage work for shared-mapping hugetlb, so it would be nice to work with unpoison_memory. It doesn't seem to me that holding mf_mutex in __is_raw_hwp_subpage is nice or even absolutely correct. Let me think if I can come up with something in v4.
Quoted 6 lines
Anyway, this patch looks good to me.
Reviewed-by Miaohe Lin <[email protected]>Thanks.
Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
Quoted 43 lines · Jiaqi Yan
Quoted 41 lines · Miaohe Lin
Quoted 39 lines · Jiaqi Yan
Add the functionality,
is_raw_hwp_subpage, to tell if a subpage of a hugetlb folio is a raw HWPOISON page. This functionality relies on RawHwpUnreliable to be not set; otherwise hugepage's raw HWPOISON list becomes meaningless.
is_raw_hwp_subpageneeds to holdhugetlb_lockin order to synchronize with__get_huge_page_for_hwpoison, who iterates and inserts an entry toraw_hwp_list. llist itself doesn't ensure insertion is synchornized with the iterating used by__is_raw_hwp_list. Caller can minimize the overhead of lock cycles by first checking if folio / head page's HWPOISON flag is set.Exports this functionality to be immediately used in the read operation for hugetlbfs.
Reviewed-by Mike Kravetz <[email protected]>Reviewed-by Naoya Horiguchi <[email protected]>Signed-off-by Jiaqi Yan <[email protected]>--- include/linux/hugetlb.h | 19 +++++++++++++++++++ include/linux/mm.h | 7 +++++++ mm/hugetlb.c | 10 ++++++++++ mm/memory-failure.c | 34 ++++++++++++++++++++++++---------- 4 files changed, 60 insertions(+), 10 deletions(-)...
mm/memory-failure.c +11
-2-static inline struct llist_head *raw_hwp_list_head (struct folio *folio) +bool __is_raw_hwp_subpage (struct folio *folio,struct page *subpage) { -return (struct llist_head *)&folio->_hugetlb_hwpoison; +struct llist_head *raw_hwp_head; +struct raw_hwp_page *p, *tmp; +bool ret = false; + +if (!folio_test_hwpoison (folio)) +return false; + +/* +* When RawHwpUnreliable is set, kernel lost track of which subpages +* are HWPOISON. So return as if ALL subpages are HWPOISONed.
Quoted 8 lines
Quoted 8 lines
mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1834
Quoted 6 lines
mm/memory-failure.c +6
-0+*/ +if (folio_test_hugetlb_raw_hwp_unreliable (folio)) +return true; + + raw_hwp_head =raw_hwp_list_head (folio); +llist_for_each_entry_safe (p, tmp, raw_hwp_head->first, node) {Since we don't free the
raw_hwp_list, doesllist_for_each_entryworks same asllist_for_each_entry_safe?
Sorry I missed this comment. Yes they are the same but llist_for_each_entry doesn't need "tmp". I will switch to llist_for_each_entry in v4.
Quoted 21 lines
Quoted 11 lines
mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1842
Quoted 10 lines
mm/memory-failure.c +7
-0+if (subpage == p->page) { + ret = true; +break ; + } + } + +return ret; }Quoted 7 lines
It seems there's a race between
__is_raw_hwp_subpageandunpoison_memory:
unpoison_memory__is_raw_hwp_subpage
if (!folio_test_hwpoison(folio)) -- hwpoison is set
folio_free_raw_hwpllist_for_each_entry_saferaw_hwp_list
llist_del_all..
folio_test_clear_hwpoisonThanks Miaohe for raising this concern.
Quoted 8 lines
Quoted 2 lines
But
__is_raw_hwp_subpageis used in hugetlbfs,unpoison_memorycouldn't reach here because there's afolio_mapping== NULL check beforefolio_free_raw_hwp.I agree. But in near future I do want to make
__is_raw_hwp_subpagework for shared-mapping hugetlb, so it would be nice to work withunpoison_memory. It doesn't seem to me that holdingmf_mutexin__is_raw_hwp_subpageis nice or even absolutely correct. Let me think if I can come up with something in v4.
At my 2nd thought, if __is_raw_hwp_subpage simply takes mf_mutex before llist_for_each_entry, it will introduce a deadlock:
unpoison_memory __is_raw_hwp_subpage
held mf_mutex held hugetlb_lock
get_hwpoison_hugetlb_folio attempts mf_mutex
attempts hugetlb lock
Not for this patch series, but for future, is it a good idea to make mf_mutex available to hugetlb code? Then enforce the order of locking to be mf_mutex first, hugetlb_lock second? I believe this is the current locking pattern / order for try_memory_failure_hugetlb.
Quoted 7 lines
Quoted 7 lines
Anyway, this patch looks good to me.
Reviewed-by Miaohe Lin <[email protected]>Thanks.
Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
Quoted 40 lines · Jiaqi Yan
Quoted 22 lines · Jiaqi Yan
Quoted 11 lines · Miaohe Lin
mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1842
Quoted 8 lines · Jiaqi Yan
mm/memory-failure.c +7
-0+if (subpage == p->page) { + ret = true; +break ; + } + } + +return ret; }Quoted 7 lines
It seems there's a race between
__is_raw_hwp_subpageandunpoison_memory:
unpoison_memory__is_raw_hwp_subpage
if (!folio_test_hwpoison(folio)) -- hwpoison is set
folio_free_raw_hwpllist_for_each_entry_saferaw_hwp_list
llist_del_all..
folio_test_clear_hwpoisonThanks Miaohe for raising this concern.
Quoted 8 lines
Quoted 2 lines
But
__is_raw_hwp_subpageis used in hugetlbfs,unpoison_memorycouldn't reach here because there's afolio_mapping== NULL check beforefolio_free_raw_hwp.I agree. But in near future I do want to make
__is_raw_hwp_subpagework for shared-mapping hugetlb, so it would be nice to work withunpoison_memory. It doesn't seem to me that holdingmf_mutexin__is_raw_hwp_subpageis nice or even absolutely correct. Let me think if I can come up with something in v4.At my 2nd thought, if
__is_raw_hwp_subpagesimply takesmf_mutexbeforellist_for_each_entry, it will introduce a deadlock:
unpoison_memory__is_raw_hwp_subpage
heldmf_mutexheldhugetlb_lock
get_hwpoison_hugetlb_folioattemptsmf_mutex
attempts hugetlb lock
Quoted 4 lines
Not for this patch series, but for future, is it a good idea to make
mf_mutexavailable to hugetlb code? Then enforce the order of locking to bemf_mutexfirst,hugetlb_locksecond? I believe this is the current locking pattern / order fortry_memory_failure_hugetlb.
I think only holding mf_mutex in __is_raw_hwp_subpage would be sufficient to prevent races with unpoison_memory. memory failure code needs to take both mf_mutex and hugetlb_lock. The hugetlb lock is to prevent hugetlb page state changes. IIUC, __is_raw_hwp_subpage is only taking hugetlb_lock to prevent races with memory failure code.
Of course, I could be missing something as there are subtle issues with locking in the memory failure code.
Signature · 1 line
Mike Kravetz
Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
Quoted 47 lines · Mike Kravetz
Quoted 40 lines · Jiaqi Yan
Quoted 22 lines · Jiaqi Yan
Quoted 11 lines · Miaohe Lin
mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1842
Quoted 8 lines · Jiaqi Yan
mm/memory-failure.c +7
-0+if (subpage == p->page) { + ret = true; +break ; + } + } + +return ret; }Quoted 7 lines
It seems there's a race between
__is_raw_hwp_subpageandunpoison_memory:
unpoison_memory__is_raw_hwp_subpage
if (!folio_test_hwpoison(folio)) -- hwpoison is set
folio_free_raw_hwpllist_for_each_entry_saferaw_hwp_list
llist_del_all..
folio_test_clear_hwpoisonThanks Miaohe for raising this concern.
Quoted 8 lines
Quoted 2 lines
But
__is_raw_hwp_subpageis used in hugetlbfs,unpoison_memorycouldn't reach here because there's afolio_mapping== NULL check beforefolio_free_raw_hwp.I agree. But in near future I do want to make
__is_raw_hwp_subpagework for shared-mapping hugetlb, so it would be nice to work withunpoison_memory. It doesn't seem to me that holdingmf_mutexin__is_raw_hwp_subpageis nice or even absolutely correct. Let me think if I can come up with something in v4.At my 2nd thought, if
__is_raw_hwp_subpagesimply takesmf_mutexbeforellist_for_each_entry, it will introduce a deadlock:
unpoison_memory__is_raw_hwp_subpage
heldmf_mutexheldhugetlb_lock
get_hwpoison_hugetlb_folioattemptsmf_mutex
attempts hugetlb lockQuoted 4 lines
Not for this patch series, but for future, is it a good idea to make
mf_mutexavailable to hugetlb code? Then enforce the order of locking to bemf_mutexfirst,hugetlb_locksecond? I believe this is the current locking pattern / order fortry_memory_failure_hugetlb.
Quoted 5 lines
I think only holding
mf_mutexin__is_raw_hwp_subpagewould be sufficient to prevent races withunpoison_memory. memory failure code needs to take bothmf_mutexandhugetlb_lock. The hugetlb lock is to prevent hugetlb page state changes. IIUC,__is_raw_hwp_subpageis only takinghugetlb_lockto prevent races with memory failure code.
Thanks Mike, I think mf_mutex is a simple and correct solution. I will drop hugetlb_lock from __is_raw_hwp_subpage in v4.
Quoted 5 lines
Of course, I could be missing something as there are subtle issues with locking in the memory failure code.
--
Mike Kravetz
Miaohe Lin
maintainerRe: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
In reply to Mike KravetzQuoted 41 lines · Mike Kravetz
Quoted 40 lines · Jiaqi Yan
Quoted 22 lines · Jiaqi Yan
Quoted 11 lines · Miaohe Lin
mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1842
Quoted 8 lines · Jiaqi Yan
mm/memory-failure.c +7
-0+if (subpage == p->page) { + ret = true; +break ; + } + } + +return ret; }Quoted 7 lines
It seems there's a race between
__is_raw_hwp_subpageandunpoison_memory:
unpoison_memory__is_raw_hwp_subpage
if (!folio_test_hwpoison(folio)) -- hwpoison is set
folio_free_raw_hwpllist_for_each_entry_saferaw_hwp_list
llist_del_all..
folio_test_clear_hwpoisonThanks Miaohe for raising this concern.
Quoted 8 lines
Quoted 2 lines
But
__is_raw_hwp_subpageis used in hugetlbfs,unpoison_memorycouldn't reach here because there's afolio_mapping== NULL check beforefolio_free_raw_hwp.I agree. But in near future I do want to make
__is_raw_hwp_subpagework for shared-mapping hugetlb, so it would be nice to work withunpoison_memory. It doesn't seem to me that holdingmf_mutexin__is_raw_hwp_subpageis nice or even absolutely correct. Let me think if I can come up with something in v4.At my 2nd thought, if
__is_raw_hwp_subpagesimply takesmf_mutexbeforellist_for_each_entry, it will introduce a deadlock:
unpoison_memory__is_raw_hwp_subpage
heldmf_mutexheldhugetlb_lock
get_hwpoison_hugetlb_folioattemptsmf_mutex
attempts hugetlb lock
Quoted 7 lines
Quoted 4 lines
Not for this patch series, but for future, is it a good idea to make
mf_mutexavailable to hugetlb code? Then enforce the order of locking to bemf_mutexfirst,hugetlb_locksecond? I believe this is the current locking pattern / order fortry_memory_failure_hugetlb.I think only holding
mf_mutexin__is_raw_hwp_subpagewould be sufficient to prevent races withunpoison_memory. memory failure code needs to take
Since soft_offline_page, memory_failure and unpoison_memory both holds mf_mutex, I think this should be enough to prevent races between them too.
Thanks.
PATCH v3 3/4 hugetlbfs: improve read HWPOISON hugepage
In reply to Jiaqi YanWhen a hugepage contains HWPOISON pages, read() fails to read any byte of the hugepage and returns -EIO, although many bytes in the HWPOISON hugepage are readable.
Improve this by allowing hugetlbfs_read_iter returns as many bytes as possible. For a requested range [offset, offset + len) that contains HWPOISON page, return [offset, first HWPOISON page addr); the next read attempt will fail and return -EIO.
--- fs/hugetlbfs/inode.c | 58 +++++++++++++++++++++++++++++++++++++++----- 1 file changed, 52 insertions(+), 6 deletions(-)
fs/hugetlbfs/inode.c +52 -6
@@ -282,6 +282,42 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr, }#endif +/* +* Someone wants to read @bytes from a HWPOISON hugetlb @page from @offset. +* Returns the maximum number of bytes one can read without touching the 1st raw +* HWPOISON subpage. +* +* The implementation borrows the iteration logic from copy_page_to_iter*. +*/ +static size_t adjust_range_hwpoison (struct page *page,size_t offset,size_t bytes) +{ +size_t n =0 ; +size_t res =0 ; +struct folio *folio =page_folio (page); + +/* First subpage to start the loop. */ + page += offset /PAGE_SIZE ; + offset %=PAGE_SIZE ; +while (1 ) { +if (is_raw_hwp_subpage (folio, page)) +break ; + +/* Safe to read n bytes without touching HWPOISON subpage. */ + n =min (bytes, (size_t )PAGE_SIZE - offset); + res += n; + bytes -= n; +if (!bytes || !n) +break ; + offset += n; +if (offset ==PAGE_SIZE ) { + page++; + offset =0 ; + } + } + +return res; +} +/* * Support for read() - Find the page attached to f_mapping and copy out the * data. This provides functionality similar to filemap_read().
@@ -300,7 +336,7 @@ static ssize_t hugetlbfs_read_iter(struct kiocb *iocb, struct iov_iter *to)while (iov_iter_count (to)) {struct page *page; -size_t nr, copied; +size_t nr, copied, want;/* nr is the maximum number of bytes to copy from this page */ nr =huge_page_size (h);
@@ -328,16 +364,26 @@ static ssize_t hugetlbfs_read_iter(struct kiocb *iocb, struct iov_iter *to) }else {unlock_page (page); -if (PageHWPoison (page)) { -put_page (page); - retval = -EIO ; -break ; +if (!PageHWPoison (page)) + want = nr; +else { +/* +* Adjust how many bytes safe to read without +* touching the 1st raw HWPOISON subpage after +* offset. +*/ + want =adjust_range_hwpoison (page, offset, nr); +if (want ==0 ) { +put_page (page); + retval = -EIO ; +break ; + } }/* * We have the page, copy it to user space buffer. */ - copied =copy_page_to_iter (page, offset, nr, to); + copied =copy_page_to_iter (page, offset, want, to);put_page (page); } offset += copied;
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
PATCH v3 4/4 selftests/mm: add tests for HWPOISON hugetlbfs read
In reply to Jiaqi YanAdd tests for the improvement made to read operation on HWPOISON hugetlb page with different read granularities. For each chunk size, three read scenarios are tested:
1. Simple regression test on read without HWPOISON.
2. Sequential read page by page should succeed until encounters the 1st raw HWPOISON subpage.
3. After skip a raw HWPOISON subpage by lseek, read()s always succeed.
--- tools/testing/selftests/mm/.gitignore | 1 + tools/testing/selftests/mm/Makefile | 1 + .../selftests/mm/hugetlb-read-hwpoison.c | 322 ++++++++++++++++++ 3 files changed, 324 insertions(+) create mode 100644 tools/testing/selftests/mm/hugetlb-read-hwpoison.c
tools/testing/selftests/mm/.gitignore +1 -0
@@ -5,6 +5,7 @@ hugepage-mremap hugepage-shm hugepage-vmemmap hugetlb-madvise +hugetlb-read-hwpoison khugepaged map_hugetlb map_populate
tools/testing/selftests/mm/Makefile +1 -0
@@ -41,6 +41,7 @@ TEST_GEN_PROGS += gup_longterm TEST_GEN_PROGS += gup_test TEST_GEN_PROGS += hmm-tests TEST_GEN_PROGS += hugetlb-madvise +TEST_GEN_PROGS += hugetlb-read-hwpoison TEST_GEN_PROGS += hugepage-mmap TEST_GEN_PROGS += hugepage-mremap TEST_GEN_PROGS += hugepage-shm
tools/testing/selftests/mm/hugetlb-read-hwpoison.c new file +322 -0
322 lines not on this page · raw message
Signature · 1 line
2.41.0.255.g8b1d071c50-goog