Improve hugetlbfs read on HWPOISON hugepages
Subsystems (6)
v2 cover Improve hugetlbfs read on HWPOISON hugepages
Today when hardware memory is corrupted in a hugetlb hugepage, kernel leaves the hugepage in pagecache [1]; otherwise future mmap or read will suject to silent data corruption. This is implemented by returning -EIO from hugetlb_read_iter immediately if the hugepage has HWPOISON flag set.
Since memory_failure already tracks the raw HWPOISON subpages in a hugepage, a natural improvement is possible: if userspace only asks for healthy subpages in the pagecache, kernel can return these data.
This patchset implements this improvement. The 1st commit fixes an issue in __folio_free_raw_hwp. The 2nd commit exports the functionality to tell if a subpage inside a hugetlb hugepage is a raw HWPOISON page. The 3rd commit teaches hugetlbfs_read_iter to return as many healthy bytes as possible. The last commit properly tests this new feature.
[1] commit 8625147cafaa ("hugetlbfs: don't delete error page from pagecache")
Changelog
v1 => v2
* __folio_free_raw_hwp deletes all entries in raw_hwp_list before it traverses and frees raw_hwp_page.
* find_raw_hwp_page => __is_raw_hwp_subpage and __is_raw_hwp_subpage only returns bool instead of a raw_hwp_page entry.
* is_raw_hwp_subpage holds hugetlb_lock while checking __is_raw_hwp_subpage.
* No need to do folio_lock in adjust_range_hwpoison.
* v2 is based on commit a6e79df92e4a ("mm/gup: disallow FOLL_LONGTERM GUP-fast writing to file-backed mappings")
Jiaqi Yan (4):
mm/hwpoison: delete all entries before traversal in
__folio_free_raw_hwp
mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
hugetlbfs: improve read HWPOISON hugepage
selftests/mm: add tests for HWPOISON hugetlbfs read
fs/hugetlbfs/inode.c | 58 +++- include/linux/hugetlb.h | 19 ++ include/linux/mm.h | 7 + mm/hugetlb.c | 10 + mm/memory-failure.c | 42 ++- tools/testing/selftests/mm/.gitignore | 1 + tools/testing/selftests/mm/Makefile | 1 + .../selftests/mm/hugetlb-read-hwpoison.c | 322 ++++++++++++++++++ 8 files changed, 439 insertions(+), 21 deletions(-) create mode 100644 tools/testing/selftests/mm/hugetlb-read-hwpoison.c
Signature · 1 line
2.41.0.162.gfafddb0af9-goog
PATCH v2 1/4 mm/hwpoison: delete all entries before traversal in __folio_free_raw_hwp
Traversal on llist (e.g. llist_for_each_safe) is only safe AFTER entries are deleted from the llist.
llist_del_all are lock free with itself. folio_clear_hugetlb_hwpoison()s from __update_and_free_hugetlb_folio and memory_failure won't need explicit locking when freeing the raw_hwp_list.
--- mm/memory-failure.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
mm/memory-failure.c +3 -5
@@ -1825,12 +1825,11 @@ static inline struct llist_head *raw_hwp_list_head(struct folio *folio)static unsigned long __folio_free_raw_hwp (struct folio *folio,bool move_flag) { -struct llist_head *head; -struct llist_node *t, *tnode; +struct llist_node *t, *tnode, *head;unsigned long count =0 ; - head =raw_hwp_list_head (folio); -llist_for_each_safe (tnode, t, head->first) { + head =llist_del_all (raw_hwp_list_head (folio)); +llist_for_each_safe (tnode, t, head) { struct raw_hwp_page *p = container_of(tnode, struct raw_hwp_page, node);if (move_flag)
@@ -1840,7 +1839,6 @@ static unsigned long __folio_free_raw_hwp(struct folio *folio, bool move_flag)kfree (p); count++; } -llist_del_all (head);return count; }
Signature · 1 line
2.41.0.162.gfafddb0af9-goog
Re: [PATCH v2 1/4] mm/hwpoison: delete all entries before traversal in __folio_free_raw_hwp
Quoted 8 lines · Jiaqi Yan
Traversal on llist (e.g.
llist_for_each_safe) is only safe AFTER entries are deleted from the llist.
llist_del_allare lock free with itself. folio_clear_hugetlb_hwpoison()s from__update_and_free_hugetlb_folioandmemory_failurewon't need explicit locking when freeing theraw_hwp_list.Signed-off-by Jiaqi Yan <[email protected]>
(Sorry if stupid question...) folio_set_hugetlb_hwpoison() also calls llist_for_each_safe() but it still traverses the list without calling llist_del_all(). This convention applies only when removing item(s)?
Thanks,
Naoya Horiguchi
Quoted 37 lines
--- mm/memory-failure.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)mm/memory-failure.c +3
-5@@ -1825,12 +1825,11 @@ static inline struct llist_head *raw_hwp_list_head(struct folio *folio)static unsigned long __folio_free_raw_hwp (struct folio *folio,bool move_flag) { -struct llist_head *head; -struct llist_node *t, *tnode; +struct llist_node *t, *tnode, *head;unsigned long count =0 ; - head =raw_hwp_list_head (folio); -llist_for_each_safe (tnode, t, head->first) { + head =llist_del_all (raw_hwp_list_head (folio)); +llist_for_each_safe (tnode, t, head) { struct raw_hwp_page *p = container_of(tnode, struct raw_hwp_page, node);if (move_flag)@@ -1840,7 +1839,6 @@ static unsigned long __folio_free_raw_hwp(struct folio *folio, bool move_flag)kfree (p); count++; } -llist_del_all (head);return count; }Signature · 1 line
2.41.0.162.gfafddb0af9-goog
Re: [PATCH v2 1/4] mm/hwpoison: delete all entries before traversal in __folio_free_raw_hwp
Quoted 9 lines · Naoya Horiguchi
Quoted 7 lines · Jiaqi Yan
Traversal on llist (e.g.
llist_for_each_safe) is only safe AFTER entries are deleted from the llist.
llist_del_allare lock free with itself. folio_clear_hugetlb_hwpoison()s from__update_and_free_hugetlb_folioandmemory_failurewon't need explicit locking when freeing theraw_hwp_list.
Quoted 5 lines
Quoted 1 line
Signed-off-by Jiaqi Yan <[email protected]>(Sorry if stupid question...)
folio_set_hugetlb_hwpoison()also callsllist_for_each_safe()but it still traverses the list without callingllist_del_all(). This convention applies only when removing item(s)?
I think in our previous discussion, Mike and I agree as of today's code in hugetlb.c and memory-failure.c, concurrent adding, deleting, traversing are fine with each other and with themselves [1], but new code need to be careful wrt ops on raw_hwp_list.
This patch is a low-hanging fruit to ensure any caller of __folio_free_raw_hwp won't introduce any problem by correcting one thing in __folio_free_raw_hwp: since it wants to delete raw_hwp_page entries in the list, it should do it by first llist_del_all, and then kfree with a llist_for_each_safe.
As for folio_set_hugetlb_hwpoison, I am not very comfortable fixing it. I imagine a way to fix it is llist_del_all() => llist_for_each_safe{...} => llist_add_batch(), or llist_add() within llist_for_each_safe{...}. I haven't really thought through if this is a correct fix.
[1] https://lore.kernel.org/lkml/CACw3F51o1ZFSYZa+XLnk4Wwjy2w_q=Kn+aOQs0=qpfG-ZYDFKg@mail.gmail.com/#t
Quoted 41 lines
Thanks,
Naoya HoriguchiQuoted 37 lines
--- mm/memory-failure.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)mm/memory-failure.c +3
-6@@ -1825,12 +1825,11 @@ static inline struct llist_head *raw_hwp_list_head(struct folio *folio)static unsigned long __folio_free_raw_hwp (struct folio *folio,bool move_flag) { -struct llist_head *head; -struct llist_node *t, *tnode; +struct llist_node *t, *tnode, *head;unsigned long count =0 ; - head =raw_hwp_list_head (folio); -llist_for_each_safe (tnode, t, head->first) { + head =llist_del_all (raw_hwp_list_head (folio)); +llist_for_each_safe (tnode, t, head) { struct raw_hwp_page *p = container_of(tnode, struct raw_hwp_page, node);if (move_flag)@@ -1840,7 +1839,6 @@ static unsigned long __folio_free_raw_hwp(struct folio *folio, bool move_flag)kfree (p); count++; } -llist_del_all (head);return count; } --2.41.0.162.gfafddb0af9-goog
Naoya Horiguchi
Acked-byRe: [PATCH v2 1/4] mm/hwpoison: delete all entries before traversal in __folio_free_raw_hwp
Quoted 22 lines · Jiaqi Yan
Quoted 9 lines · Naoya Horiguchi
Quoted 7 lines · Jiaqi Yan
Traversal on llist (e.g.
llist_for_each_safe) is only safe AFTER entries are deleted from the llist.
llist_del_allare lock free with itself. folio_clear_hugetlb_hwpoison()s from__update_and_free_hugetlb_folioandmemory_failurewon't need explicit locking when freeing theraw_hwp_list.Quoted 5 lines
Quoted 1 line
Signed-off-by Jiaqi Yan <[email protected]>(Sorry if stupid question...)
folio_set_hugetlb_hwpoison()also callsllist_for_each_safe()but it still traverses the list without callingllist_del_all(). This convention applies only when removing item(s)?I think in our previous discussion, Mike and I agree as of today's code in
hugetlb.candmemory-failure.c, concurrent adding, deleting, traversing are fine with each other and with themselves [1], but new code need to be careful wrt ops onraw_hwp_list.
Quoted 5 lines
This patch is a low-hanging fruit to ensure any caller of
__folio_free_raw_hwpwon't introduce any problem by correcting one thing in__folio_free_raw_hwp: since it wants to deleteraw_hwp_pageentries in the list, it should do it by firstllist_del_all, and then kfree with allist_for_each_safe.
Thanks for the explanation, this is worth adding to the patch description for future developers to understand the background.
Quoted 6 lines
As for
folio_set_hugetlb_hwpoison, I am not very comfortable fixing it. I imagine a way to fix it isllist_del_all()=> llist_for_each_safe{...} =>llist_add_batch(), orllist_add()within llist_for_each_safe{...}. I haven't really thought through if this is a correct fix.
I see. Changing folio_set_hugetlb_hwpoison() like this is a little too complex considering that this fix is for precaution.
So no change on this for now is fine to me.
Anyway this patch looks fine to me.
Quoted 45 lines
[1] https://lore.kernel.org/lkml/CACw3F51o1ZFSYZa+XLnk4Wwjy2w_q=Kn+aOQs0=qpfG-ZYDFKg@mail.gmail.com/#t
Quoted 41 lines
Thanks,
Naoya HoriguchiQuoted 37 lines
--- mm/memory-failure.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)mm/memory-failure.c +3
-6@@ -1825,12 +1825,11 @@ static inline struct llist_head *raw_hwp_list_head(struct folio *folio)static unsigned long __folio_free_raw_hwp (struct folio *folio,bool move_flag) { -struct llist_head *head; -struct llist_node *t, *tnode; +struct llist_node *t, *tnode, *head;unsigned long count =0 ; - head =raw_hwp_list_head (folio); -llist_for_each_safe (tnode, t, head->first) { + head =llist_del_all (raw_hwp_list_head (folio)); +llist_for_each_safe (tnode, t, head) { struct raw_hwp_page *p = container_of(tnode, struct raw_hwp_page, node);if (move_flag)@@ -1840,7 +1839,6 @@ static unsigned long __folio_free_raw_hwp(struct folio *folio, bool move_flag)kfree (p); count++; } -llist_del_all (head);return count; } --2.41.0.162.gfafddb0af9-goog
Re: [PATCH v2 1/4] mm/hwpoison: delete all entries before traversal in __folio_free_raw_hwp
In reply to Jiaqi YanQuoted 11 lines · Jiaqi Yan
Traversal on llist (e.g.
llist_for_each_safe) is only safe AFTER entries are deleted from the llist.
llist_del_allare lock free with itself. folio_clear_hugetlb_hwpoison()s from__update_and_free_hugetlb_folioandmemory_failurewon't need explicit locking when freeing theraw_hwp_list.Signed-off-by Jiaqi Yan <[email protected]>--- mm/memory-failure.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
After updating the reason for patch in commit message as suggested by Naoya,
Signature · 1 line
Mike Kravetz
Quoted 32 lines
mm/memory-failure.c +3
-5@@ -1825,12 +1825,11 @@ static inline struct llist_head *raw_hwp_list_head(struct folio *folio)static unsigned long __folio_free_raw_hwp (struct folio *folio,bool move_flag) { -struct llist_head *head; -struct llist_node *t, *tnode; +struct llist_node *t, *tnode, *head;unsigned long count =0 ; - head =raw_hwp_list_head (folio); -llist_for_each_safe (tnode, t, head->first) { + head =llist_del_all (raw_hwp_list_head (folio)); +llist_for_each_safe (tnode, t, head) { struct raw_hwp_page *p = container_of(tnode, struct raw_hwp_page, node);if (move_flag)@@ -1840,7 +1839,6 @@ static unsigned long __folio_free_raw_hwp(struct folio *folio, bool move_flag)kfree (p); count++; } -llist_del_all (head);return count; }Signature · 1 line
2.41.0.162.gfafddb0af9-goog
Re: [PATCH v2 1/4] mm/hwpoison: delete all entries before traversal in __folio_free_raw_hwp
Quoted 9 lines · Mike Kravetz
Quoted 7 lines · Jiaqi Yan
Traversal on llist (e.g.
llist_for_each_safe) is only safe AFTER entries are deleted from the llist.
llist_del_allare lock free with itself. folio_clear_hugetlb_hwpoison()s from__update_and_free_hugetlb_folioandmemory_failurewon't need explicit locking when freeing theraw_hwp_list.
Quoted 6 lines
Quoted 4 lines
Signed-off-by Jiaqi Yan <[email protected]>--- mm/memory-failure.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)After updating the reason for patch in commit message as suggested by Naoya,
Thank you both Mike and Naoya! I will add the explanation in the next version.
Quoted 38 lines
Acked-by Mike Kravetz <[email protected]>--
Mike KravetzQuoted 32 lines
mm/memory-failure.c +3
-6@@ -1825,12 +1825,11 @@ static inline struct llist_head *raw_hwp_list_head(struct folio *folio)static unsigned long __folio_free_raw_hwp (struct folio *folio,bool move_flag) { -struct llist_head *head; -struct llist_node *t, *tnode; +struct llist_node *t, *tnode, *head;unsigned long count =0 ; - head =raw_hwp_list_head (folio); -llist_for_each_safe (tnode, t, head->first) { + head =llist_del_all (raw_hwp_list_head (folio)); +llist_for_each_safe (tnode, t, head) { struct raw_hwp_page *p = container_of(tnode, struct raw_hwp_page, node);if (move_flag)@@ -1840,7 +1839,6 @@ static unsigned long __folio_free_raw_hwp(struct folio *folio, bool move_flag)kfree (p); count++; } -llist_del_all (head);return count; } --2.41.0.162.gfafddb0af9-goog
PATCH v2 2/4 mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
In reply to Jiaqi YanAdds the functionality to tell if a subpage of a hugetlb folio is a raw HWPOISON page. This functionality relies on RawHwpUnreliable to be not set; otherwise hugepage's HWPOISON list becomes meaningless.
Exports this functionality to be immediately used in the read operation for hugetlbfs.
--- include/linux/hugetlb.h | 19 +++++++++++++++++++ include/linux/mm.h | 7 +++++++ mm/hugetlb.c | 10 ++++++++++ mm/memory-failure.c | 34 ++++++++++++++++++++++++---------- 4 files changed, 60 insertions(+), 10 deletions(-)
include/linux/hugetlb.h +19 -0
@@ -1013,6 +1013,25 @@ void hugetlb_register_node(struct node *node);void hugetlb_unregister_node (struct node *node);#endif +/* +* Struct raw_hwp_page represents information about "raw error page", +* constructing singly linked list from ->_hugetlb_hwpoison field of folio. +*/ +struct raw_hwp_page { +struct llist_node node; +struct page *page; +}; + +static inline struct llist_head *raw_hwp_list_head (struct folio *folio) +{ +return (struct llist_head *)&folio->_hugetlb_hwpoison; +} + +/* +* Check if a given raw @subpage in a hugepage @folio is HWPOISON. +*/ +bool is_raw_hwp_subpage (struct folio *folio,struct page *subpage); +#else /* CONFIG_HUGETLB_PAGE */ struct hstate {};
include/linux/mm.h +7 -0
@@ -3671,6 +3671,7 @@ extern const struct attribute_group memory_failure_attr_group;extern void memory_failure_queue (unsigned long pfn,int flags);extern int __get_huge_page_for_hwpoison (unsigned long pfn,int flags,bool *migratable_cleared); +extern bool __is_raw_hwp_subpage (struct folio *folio,struct page *subpage);void num_poisoned_pages_inc (unsigned long pfn);void num_poisoned_pages_sub (unsigned long pfn,long i);struct task_struct *task_early_kill (struct task_struct *tsk,int force_early);
@@ -3685,6 +3686,12 @@ static inline int __get_huge_page_for_hwpoison(unsigned long pfn, int flags,return 0 ; } +static inline bool __is_raw_hwp_subpage (struct folio *folio, +struct page *subpage) +{ +return false; +} +static inline void num_poisoned_pages_inc (unsigned long pfn) { }
mm/hugetlb.c +10 -0
@@ -7377,6 +7377,16 @@ int get_huge_page_for_hwpoison(unsigned long pfn, int flags,return ret; } +bool is_raw_hwp_subpage (struct folio *folio,struct page *subpage) +{ +bool ret; + +spin_lock_irq (&hugetlb_lock); + ret =__is_raw_hwp_subpage (folio, subpage); +spin_unlock_irq (&hugetlb_lock); +return ret; +} +void folio_putback_active_hugetlb (struct folio *folio) {spin_lock_irq (&hugetlb_lock);
mm/memory-failure.c +24 -10
@@ -1809,18 +1809,32 @@ EXPORT_SYMBOL_GPL(mf_dax_kill_procs);#endif /* CONFIG_FS_DAX */ #ifdef CONFIG_HUGETLB_PAGE -/* -* Struct raw_hwp_page represents information about "raw error page", -* constructing singly linked list from ->_hugetlb_hwpoison field of folio. -*/ -struct raw_hwp_page { -struct llist_node node; -struct page *page; -}; -static inline struct llist_head *raw_hwp_list_head (struct folio *folio) +bool __is_raw_hwp_subpage (struct folio *folio,struct page *subpage) { -return (struct llist_head *)&folio->_hugetlb_hwpoison; +struct llist_head *raw_hwp_head; +struct raw_hwp_page *p, *tmp; +bool ret = false; + +if (!folio_test_hwpoison (folio)) +return false; + +/* +* When RawHwpUnreliable is set, kernel lost track of which subpages +* are HWPOISON. So return as if ALL subpages are HWPOISONed. +*/ +if (folio_test_hugetlb_raw_hwp_unreliable (folio)) +return true; + + raw_hwp_head =raw_hwp_list_head (folio); +llist_for_each_entry_safe (p, tmp, raw_hwp_head->first, node) { +if (subpage == p->page) { + ret = true; +break ; + } + } + +return ret; }static unsigned long __folio_free_raw_hwp (struct folio *folio,bool move_flag)
Signature · 1 line
2.41.0.162.gfafddb0af9-goog
Re: [PATCH v2 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
Quoted 75 lines · Jiaqi Yan
Adds the functionality to tell if a subpage of a hugetlb folio is a raw HWPOISON page. This functionality relies on RawHwpUnreliable to be not set; otherwise hugepage's HWPOISON list becomes meaningless.
Exports this functionality to be immediately used in the read operation for hugetlbfs.
Signed-off-by Jiaqi Yan <[email protected]>--- include/linux/hugetlb.h | 19 +++++++++++++++++++ include/linux/mm.h | 7 +++++++ mm/hugetlb.c | 10 ++++++++++ mm/memory-failure.c | 34 ++++++++++++++++++++++++---------- 4 files changed, 60 insertions(+), 10 deletions(-)include/linux/hugetlb.h +19
-0@@ -1013,6 +1013,25 @@ void hugetlb_register_node(struct node *node);void hugetlb_unregister_node (struct node *node);#endif +/* +* Struct raw_hwp_page represents information about "raw error page", +* constructing singly linked list from ->_hugetlb_hwpoison field of folio. +*/ +struct raw_hwp_page { +struct llist_node node; +struct page *page; +}; + +static inline struct llist_head *raw_hwp_list_head (struct folio *folio) +{ +return (struct llist_head *)&folio->_hugetlb_hwpoison; +} + +/* +* Check if a given raw @subpage in a hugepage @folio is HWPOISON. +*/ +bool is_raw_hwp_subpage (struct folio *folio,struct page *subpage); +#else /* CONFIG_HUGETLB_PAGE */ struct hstate {};include/linux/mm.h +7
-0@@ -3671,6 +3671,7 @@ extern const struct attribute_group memory_failure_attr_group;extern void memory_failure_queue (unsigned long pfn,int flags);extern int __get_huge_page_for_hwpoison (unsigned long pfn,int flags,bool *migratable_cleared); +extern bool __is_raw_hwp_subpage (struct folio *folio,struct page *subpage);void num_poisoned_pages_inc (unsigned long pfn);void num_poisoned_pages_sub (unsigned long pfn,long i);struct task_struct *task_early_kill (struct task_struct *tsk,int force_early);@@ -3685,6 +3686,12 @@ static inline int __get_huge_page_for_hwpoison(unsigned long pfn, int flags,return 0 ; } +static inline bool __is_raw_hwp_subpage (struct folio *folio, +struct page *subpage) +{ +return false; +} +static inline void num_poisoned_pages_inc (unsigned long pfn) { }mm/hugetlb.c +0
-0
mm/hugetlb.c int get_huge_page_for_hwpoison(unsigned long pfn, int flags, · line 7386
Quoted 10 lines
mm/hugetlb.c +7
-0@@ -7377,6 +7377,16 @@ int get_huge_page_for_hwpoison(unsigned long pfn, int flags,return ret; } +bool is_raw_hwp_subpage (struct folio *folio,struct page *subpage) +{ +bool ret; + +spin_lock_irq (&hugetlb_lock); + ret =__is_raw_hwp_subpage (folio, subpage); +spin_unlock_irq (&hugetlb_lock);
Can you describe what races the hugetlb_lock prevents here?
Signature · 1 line
Mike Kravetz
Quoted 56 lines
mm/hugetlb.c +3
-0+return ret; +} +void folio_putback_active_hugetlb (struct folio *folio) {spin_lock_irq (&hugetlb_lock);mm/memory-failure.c +24
-10@@ -1809,18 +1809,32 @@ EXPORT_SYMBOL_GPL(mf_dax_kill_procs);#endif /* CONFIG_FS_DAX */ #ifdef CONFIG_HUGETLB_PAGE -/* -* Struct raw_hwp_page represents information about "raw error page", -* constructing singly linked list from ->_hugetlb_hwpoison field of folio. -*/ -struct raw_hwp_page { -struct llist_node node; -struct page *page; -}; -static inline struct llist_head *raw_hwp_list_head (struct folio *folio) +bool __is_raw_hwp_subpage (struct folio *folio,struct page *subpage) { -return (struct llist_head *)&folio->_hugetlb_hwpoison; +struct llist_head *raw_hwp_head; +struct raw_hwp_page *p, *tmp; +bool ret = false; + +if (!folio_test_hwpoison (folio)) +return false; + +/* +* When RawHwpUnreliable is set, kernel lost track of which subpages +* are HWPOISON. So return as if ALL subpages are HWPOISONed. +*/ +if (folio_test_hugetlb_raw_hwp_unreliable (folio)) +return true; + + raw_hwp_head =raw_hwp_list_head (folio); +llist_for_each_entry_safe (p, tmp, raw_hwp_head->first, node) { +if (subpage == p->page) { + ret = true; +break ; + } + } + +return ret; }static unsigned long __folio_free_raw_hwp (struct folio *folio,bool move_flag)Signature · 1 line
2.41.0.162.gfafddb0af9-goog
Re: [PATCH v2 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
Quoted 80 lines · Mike Kravetz
Quoted 78 lines · Jiaqi Yan
Adds the functionality to tell if a subpage of a hugetlb folio is a raw HWPOISON page. This functionality relies on RawHwpUnreliable to be not set; otherwise hugepage's HWPOISON list becomes meaningless.
Exports this functionality to be immediately used in the read operation for hugetlbfs.
Signed-off-by Jiaqi Yan <[email protected]>--- include/linux/hugetlb.h | 19 +++++++++++++++++++ include/linux/mm.h | 7 +++++++ mm/hugetlb.c | 10 ++++++++++ mm/memory-failure.c | 34 ++++++++++++++++++++++++---------- 4 files changed, 60 insertions(+), 10 deletions(-)include/linux/hugetlb.h +19
-0@@ -1013,6 +1013,25 @@ void hugetlb_register_node(struct node *node);void hugetlb_unregister_node (struct node *node);#endif +/* +* Struct raw_hwp_page represents information about "raw error page", +* constructing singly linked list from ->_hugetlb_hwpoison field of folio. +*/ +struct raw_hwp_page { +struct llist_node node; +struct page *page; +}; + +static inline struct llist_head *raw_hwp_list_head (struct folio *folio) +{ +return (struct llist_head *)&folio->_hugetlb_hwpoison; +} + +/* +* Check if a given raw @subpage in a hugepage @folio is HWPOISON. +*/ +bool is_raw_hwp_subpage (struct folio *folio,struct page *subpage); +#else /* CONFIG_HUGETLB_PAGE */ struct hstate {};include/linux/mm.h +7
-0@@ -3671,6 +3671,7 @@ extern const struct attribute_group memory_failure_attr_group;extern void memory_failure_queue (unsigned long pfn,int flags);extern int __get_huge_page_for_hwpoison (unsigned long pfn,int flags,bool *migratable_cleared); +extern bool __is_raw_hwp_subpage (struct folio *folio,struct page *subpage);void num_poisoned_pages_inc (unsigned long pfn);void num_poisoned_pages_sub (unsigned long pfn,long i);struct task_struct *task_early_kill (struct task_struct *tsk,int force_early);@@ -3685,6 +3686,12 @@ static inline int __get_huge_page_for_hwpoison(unsigned long pfn, int flags,return 0 ; } +static inline bool __is_raw_hwp_subpage (struct folio *folio, +struct page *subpage) +{ +return false; +} +static inline void num_poisoned_pages_inc (unsigned long pfn) { }mm/hugetlb.c +0
-0@@ -7377,6 +7377,16 @@ int get_huge_page_for_hwpoison(unsigned long pfn, int flags,return ret; }
Quoted 9 lines
mm/hugetlb.c int get_huge_page_for_hwpoison(unsigned long pfn, int flags, · line 7386
Quoted 7 lines
mm/hugetlb.c +7
-0+bool is_raw_hwp_subpage (struct folio *folio,struct page *subpage) +{ +bool ret; + +spin_lock_irq (&hugetlb_lock); + ret =__is_raw_hwp_subpage (folio, subpage); +spin_unlock_irq (&hugetlb_lock);Can you describe what races the
hugetlb_lockprevents here?
I think we should sync here with __get_huge_page_for_hwpoison, who iterates and inserts an entry to raw_hwp_list. llist itself doesn't ensure insertion is synchronized with iterating from __is_raw_hwp_subpage.
Quoted 59 lines
--
Mike KravetzQuoted 56 lines
mm/hugetlb.c +3
-0+return ret; +} +void folio_putback_active_hugetlb (struct folio *folio) {spin_lock_irq (&hugetlb_lock);mm/memory-failure.c +24
-11@@ -1809,18 +1809,32 @@ EXPORT_SYMBOL_GPL(mf_dax_kill_procs);#endif /* CONFIG_FS_DAX */ #ifdef CONFIG_HUGETLB_PAGE -/* -* Struct raw_hwp_page represents information about "raw error page", -* constructing singly linked list from ->_hugetlb_hwpoison field of folio. -*/ -struct raw_hwp_page { -struct llist_node node; -struct page *page; -}; -static inline struct llist_head *raw_hwp_list_head (struct folio *folio) +bool __is_raw_hwp_subpage (struct folio *folio,struct page *subpage) { -return (struct llist_head *)&folio->_hugetlb_hwpoison; +struct llist_head *raw_hwp_head; +struct raw_hwp_page *p, *tmp; +bool ret = false; + +if (!folio_test_hwpoison (folio)) +return false; + +/* +* When RawHwpUnreliable is set, kernel lost track of which subpages +* are HWPOISON. So return as if ALL subpages are HWPOISONed. +*/ +if (folio_test_hugetlb_raw_hwp_unreliable (folio)) +return true; + + raw_hwp_head =raw_hwp_list_head (folio); +llist_for_each_entry_safe (p, tmp, raw_hwp_head->first, node) { +if (subpage == p->page) { + ret = true; +break ; + } + } + +return ret; }static unsigned long __folio_free_raw_hwp (struct folio *folio,bool move_flag) --2.41.0.162.gfafddb0af9-goog
Mike Kravetz
Reviewed-byRe: [PATCH v2 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
Quoted 13 lines · Jiaqi Yan
Quoted 11 lines · Mike Kravetz
mm/hugetlb.c int get_huge_page_for_hwpoison(unsigned long pfn, int flags, · line 7386
Quoted 8 lines · Jiaqi Yan
mm/hugetlb.c +7
-0+bool is_raw_hwp_subpage (struct folio *folio,struct page *subpage) +{ +bool ret; + +spin_lock_irq (&hugetlb_lock); + ret =__is_raw_hwp_subpage (folio, subpage); +spin_unlock_irq (&hugetlb_lock);Can you describe what races the
hugetlb_lockprevents here?
Quoted 5 lines
I think we should sync here with
__get_huge_page_for_hwpoison, who iterates and inserts an entry toraw_hwp_list. llist itself doesn't ensure insertion is synchronized with iterating from__is_raw_hwp_subpage.
Ok, makes sense. And, since this is only called in the file read patch when we encounter a PageHWPoison(page), the overhead of the lock cycles is not of concern.
You can add,
Signature · 1 line
Mike Kravetz
Re: [PATCH v2 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
Quoted 21 lines · Mike Kravetz
Quoted 13 lines · Jiaqi Yan
Quoted 11 lines · Mike Kravetz
mm/hugetlb.c int get_huge_page_for_hwpoison(unsigned long pfn, int flags, · line 7386
Quoted 8 lines · Jiaqi Yan
mm/hugetlb.c +7
-0+bool is_raw_hwp_subpage (struct folio *folio,struct page *subpage) +{ +bool ret; + +spin_lock_irq (&hugetlb_lock); + ret =__is_raw_hwp_subpage (folio, subpage); +spin_unlock_irq (&hugetlb_lock);Can you describe what races the
hugetlb_lockprevents here?Quoted 5 lines
I think we should sync here with
__get_huge_page_for_hwpoison, who iterates and inserts an entry toraw_hwp_list. llist itself doesn't ensure insertion is synchronized with iterating from__is_raw_hwp_subpage.
Quoted 3 lines
Ok, makes sense. And, since this is only called in the file read patch when we encounter a PageHWPoison(page), the overhead of the lock cycles is not of concern.
Yes, thanks for pointing this out, (which I forgot), as is_raw_hwp_subpage() is after PageHWPoison check in hugetlbfs_read_iter. I think both this and the reason why holding the lock is worth mentioning in the commit msg.
Quoted 6 lines
You can add,
Reviewed-by Mike Kravetz <[email protected]>--
Mike Kravetz
Re: [PATCH v2 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
In reply to Jiaqi YanQuoted 8 lines · Jiaqi Yan
Adds the functionality to tell if a subpage of a hugetlb folio is a raw HWPOISON page. This functionality relies on RawHwpUnreliable to be not set; otherwise hugepage's HWPOISON list becomes meaningless.
Exports this functionality to be immediately used in the read operation for hugetlbfs.
Signed-off-by Jiaqi Yan <[email protected]>
Looks good to me, thank you.
PATCH v2 3/4 hugetlbfs: improve read HWPOISON hugepage
In reply to Jiaqi YanWhen a hugepage contains HWPOISON pages, read() fails to read any byte of the hugepage and returns -EIO, although many bytes in the HWPOISON hugepage are readable.
Improve this by allowing hugetlbfs_read_iter returns as many bytes as possible. For a requested range [offset, offset + len) that contains HWPOISON page, return [offset, first HWPOISON page addr); the next read attempt will fail and return -EIO.
--- fs/hugetlbfs/inode.c | 58 +++++++++++++++++++++++++++++++++++++++----- 1 file changed, 52 insertions(+), 6 deletions(-)
fs/hugetlbfs/inode.c +52 -6
@@ -282,6 +282,42 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr, }#endif +/* +* Someone wants to read @bytes from a HWPOISON hugetlb @page from @offset. +* Returns the maximum number of bytes one can read without touching the 1st raw +* HWPOISON subpage. +* +* The implementation borrows the iteration logic from copy_page_to_iter*. +*/ +static size_t adjust_range_hwpoison (struct page *page,size_t offset,size_t bytes) +{ +size_t n =0 ; +size_t res =0 ; +struct folio *folio =page_folio (page); + +/* First subpage to start the loop. */ + page += offset /PAGE_SIZE ; + offset %=PAGE_SIZE ; +while (1 ) { +if (is_raw_hwp_subpage (folio, page)) +break ; + +/* Safe to read n bytes without touching HWPOISON subpage. */ + n =min (bytes, (size_t )PAGE_SIZE - offset); + res += n; + bytes -= n; +if (!bytes || !n) +break ; + offset += n; +if (offset ==PAGE_SIZE ) { + page++; + offset =0 ; + } + } + +return res; +} +/* * Support for read() - Find the page attached to f_mapping and copy out the * data. This provides functionality similar to filemap_read().
@@ -300,7 +336,7 @@ static ssize_t hugetlbfs_read_iter(struct kiocb *iocb, struct iov_iter *to)while (iov_iter_count (to)) {struct page *page; -size_t nr, copied; +size_t nr, copied, want;/* nr is the maximum number of bytes to copy from this page */ nr =huge_page_size (h);
@@ -328,16 +364,26 @@ static ssize_t hugetlbfs_read_iter(struct kiocb *iocb, struct iov_iter *to) }else {unlock_page (page); -if (PageHWPoison (page)) { -put_page (page); - retval = -EIO ; -break ; +if (!PageHWPoison (page)) + want = nr; +else { +/* +* Adjust how many bytes safe to read without +* touching the 1st raw HWPOISON subpage after +* offset. +*/ + want =adjust_range_hwpoison (page, offset, nr); +if (want ==0 ) { +put_page (page); + retval = -EIO ; +break ; + } }/* * We have the page, copy it to user space buffer. */ - copied =copy_page_to_iter (page, offset, nr, to); + copied =copy_page_to_iter (page, offset, want, to);put_page (page); } offset += copied;
Signature · 1 line
2.41.0.162.gfafddb0af9-goog
Re: [PATCH v2 3/4] hugetlbfs: improve read HWPOISON hugepage
Quoted 9 lines · Jiaqi Yan
When a hugepage contains HWPOISON pages,
read()fails to read any byte of the hugepage and returns-EIO, although many bytes in the HWPOISON hugepage are readable.Improve this by allowing
hugetlbfs_read_iterreturns as many bytes as possible. For a requested range [offset, offset + len) that contains HWPOISON page, return [offset, first HWPOISON page addr); the next read attempt will fail and return-EIO.
Quoted 4 lines
Signed-off-by Jiaqi Yan <[email protected]>--- fs/hugetlbfs/inode.c | 58 +++++++++++++++++++++++++++++++++++++++----- 1 file changed, 52 insertions(+), 6 deletions(-)
I went through the code and could not find any problems.
However, code like this is where I often make mistakes. So, it would be great if someone else could take a look as well.
Signature · 1 line
Mike Kravetz
Quoted 92 lines
fs/hugetlbfs/inode.c +52
-6@@ -282,6 +282,42 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr, }#endif +/* +* Someone wants to read @bytes from a HWPOISON hugetlb @page from @offset. +* Returns the maximum number of bytes one can read without touching the 1st raw +* HWPOISON subpage. +* +* The implementation borrows the iteration logic from copy_page_to_iter*. +*/ +static size_t adjust_range_hwpoison (struct page *page,size_t offset,size_t bytes) +{ +size_t n =0 ; +size_t res =0 ; +struct folio *folio =page_folio (page); + +/* First subpage to start the loop. */ + page += offset /PAGE_SIZE ; + offset %=PAGE_SIZE ; +while (1 ) { +if (is_raw_hwp_subpage (folio, page)) +break ; + +/* Safe to read n bytes without touching HWPOISON subpage. */ + n =min (bytes, (size_t )PAGE_SIZE - offset); + res += n; + bytes -= n; +if (!bytes || !n) +break ; + offset += n; +if (offset ==PAGE_SIZE ) { + page++; + offset =0 ; + } + } + +return res; +} +/* * Support for read() - Find the page attached to f_mapping and copy out the * data. This provides functionality similar to filemap_read(). @@ -300,7 +336,7 @@ static ssize_t hugetlbfs_read_iter(struct kiocb *iocb, struct iov_iter *to)while (iov_iter_count (to)) {struct page *page; -size_t nr, copied; +size_t nr, copied, want;/* nr is the maximum number of bytes to copy from this page */ nr =huge_page_size (h);@@ -328,16 +364,26 @@ static ssize_t hugetlbfs_read_iter(struct kiocb *iocb, struct iov_iter *to) }else {unlock_page (page); -if (PageHWPoison (page)) { -put_page (page); - retval = -EIO ; -break ; +if (!PageHWPoison (page)) + want = nr; +else { +/* +* Adjust how many bytes safe to read without +* touching the 1st raw HWPOISON subpage after +* offset. +*/ + want =adjust_range_hwpoison (page, offset, nr); +if (want ==0 ) { +put_page (page); + retval = -EIO ; +break ; + } }/* * We have the page, copy it to user space buffer. */ - copied =copy_page_to_iter (page, offset, nr, to); + copied =copy_page_to_iter (page, offset, want, to);put_page (page); } offset += copied;Signature · 1 line
2.41.0.162.gfafddb0af9-goog
Re: [PATCH v2 3/4] hugetlbfs: improve read HWPOISON hugepage
In reply to Jiaqi YanQuoted 10 lines · Jiaqi Yan
When a hugepage contains HWPOISON pages,
read()fails to read any byte of the hugepage and returns-EIO, although many bytes in the HWPOISON hugepage are readable.Improve this by allowing
hugetlbfs_read_iterreturns as many bytes as possible. For a requested range [offset, offset + len) that contains HWPOISON page, return [offset, first HWPOISON page addr); the next read attempt will fail and return-EIO.Signed-off-by Jiaqi Yan <[email protected]>
Looks good to me.
PATCH v2 4/4 selftests/mm: add tests for HWPOISON hugetlbfs read
In reply to Jiaqi YanAdd tests for the improvement made to read operation on HWPOISON hugetlb page with different read granularities. For each chunk size, three read scenarios are tested:
1. Simple regression test on read without HWPOISON.
2. Sequential read page by page should succeed until encounters the 1st raw HWPOISON subpage.
3. After skip raw HWPOISON subpage by lseek, read()s always succeed.
--- tools/testing/selftests/mm/.gitignore | 1 + tools/testing/selftests/mm/Makefile | 1 + .../selftests/mm/hugetlb-read-hwpoison.c | 322 ++++++++++++++++++ 3 files changed, 324 insertions(+) create mode 100644 tools/testing/selftests/mm/hugetlb-read-hwpoison.c
tools/testing/selftests/mm/.gitignore +1 -0
@@ -5,6 +5,7 @@ hugepage-mremap hugepage-shm hugepage-vmemmap hugetlb-madvise +hugetlb-read-hwpoison khugepaged map_hugetlb map_populate
tools/testing/selftests/mm/Makefile +1 -0
@@ -38,6 +38,7 @@ TEST_GEN_PROGS += gup_longterm TEST_GEN_PROGS += gup_test TEST_GEN_PROGS += hmm-tests TEST_GEN_PROGS += hugetlb-madvise +TEST_GEN_PROGS += hugetlb-read-hwpoison TEST_GEN_PROGS += hugepage-mmap TEST_GEN_PROGS += hugepage-mremap TEST_GEN_PROGS += hugepage-shm
tools/testing/selftests/mm/hugetlb-read-hwpoison.c new file +322 -0
322 lines not on this page · raw message
Signature · 1 line
2.41.0.162.gfafddb0af9-goog
Re: [PATCH v2 4/4] selftests/mm: add tests for HWPOISON hugetlbfs read
Quoted 8 lines · Jiaqi Yan
Add tests for the improvement made to read operation on HWPOISON hugetlb page with different read granularities. For each chunk size, three read scenarios are tested:
1. Simple regression test on read without HWPOISON.
2. Sequential read page by page should succeed until encounters the 1st raw HWPOISON subpage.
3. After skip raw HWPOISON subpage by lseek, read()s always succeed.
Quoted 7 lines
Signed-off-by Jiaqi Yan <[email protected]>--- tools/testing/selftests/mm/.gitignore | 1 + tools/testing/selftests/mm/Makefile | 1 + .../selftests/mm/hugetlb-read-hwpoison.c | 322 ++++++++++++++++++ 3 files changed, 324 insertions(+) create mode 100644 tools/testing/selftests/mm/hugetlb-read-hwpoison.c
Thank you for adding a selftest!
Signature · 1 line
Mike Kravetz
Re: [PATCH v2 4/4] selftests/mm: add tests for HWPOISON hugetlbfs read
In reply to Jiaqi YanQuoted 9 lines · Jiaqi Yan
Add tests for the improvement made to read operation on HWPOISON hugetlb page with different read granularities. For each chunk size, three read scenarios are tested:
1. Simple regression test on read without HWPOISON.
2. Sequential read page by page should succeed until encounters the 1st raw HWPOISON subpage.
3. After skip raw HWPOISON subpage by lseek, read()s always succeed.Signed-off-by Jiaqi Yan <[email protected]>