Skip to contentSkip to conversation

Improve hugetlbfs read on HWPOISON hugepages

PATCH Superseded Jiaqi Yan 6 participants 17 messages Jul 7, 2023 – Jul 12, 2023

Subsystems (6)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Jiaqi Yan

author
#0 lore raw

v3 cover Improve hugetlbfs read on HWPOISON hugepages

Today when hardware memory is corrupted in a hugetlb hugepage, kernel leaves the hugepage in pagecache [1]; otherwise future mmap or read will suject to silent data corruption. This is implemented by returning -EIO from hugetlb_read_iter immediately if the hugepage has HWPOISON flag set.

Since memory_failure already tracks the raw HWPOISON subpages in a hugepage, a natural improvement is possible: if userspace only asks for healthy subpages in the pagecache, kernel can return these data.

This patchset implements this improvement. It consist of three parts. The 1st commit exports the functionality to tell if a subpage inside a hugetlb hugepage is a raw HWPOISON page. The 2nd commit teaches hugetlbfs_read_iter to return as many healthy bytes as possible. The 3rd commit properly tests this new feature.

[1] commit 8625147cafaa ("hugetlbfs: don't delete error page from pagecache")

Changelog

v2 => v3
* Updates commit messages for future reader to know background and code details
* v3 is based on commit 5bb367dca2b9 ("Merge branch 'master' into mm-stable")

v1 => v2
* __folio_free_raw_hwp deletes all entries in raw_hwp_list before it traverses and frees raw_hwp_page.
* find_raw_hwp_page => __is_raw_hwp_subpage and __is_raw_hwp_subpage only returns bool instead of a raw_hwp_page entry.
* is_raw_hwp_subpage holds hugetlb_lock while checking __is_raw_hwp_subpage.
* No need to do folio_lock in adjust_range_hwpoison.
* v2 is based on commit a6e79df92e4a ("mm/gup: disallow FOLL_LONGTERM GUP-fast writing to file-backed mappings")

Jiaqi Yan (4):
mm/hwpoison: delete all entries before traversal in
__folio_free_raw_hwp
mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON
hugetlbfs: improve read HWPOISON hugepage
selftests/mm: add tests for HWPOISON hugetlbfs read

 fs/hugetlbfs/inode.c                          |  58 +++-
 include/linux/hugetlb.h                       |  19 ++
 include/linux/mm.h                            |   7 +
 mm/hugetlb.c                                  |  10 +
 mm/memory-failure.c                           |  42 ++-
 tools/testing/selftests/mm/.gitignore         |   1 +
 tools/testing/selftests/mm/Makefile           |   1 +
 .../selftests/mm/hugetlb-read-hwpoison.c      | 322 ++++++++++++++++++
 8 files changed, 439 insertions(+), 21 deletions(-)
 create mode 100644 tools/testing/selftests/mm/hugetlb-read-hwpoison.c
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
author
#1 lore raw

PATCH v3 1/4 mm/hwpoison: delete all entries before traversal in __folio_free_raw_hwp

Traversal on llist (e.g. llist_for_each_safe) is only safe AFTER entries are deleted from the llist. Correct the way __folio_free_raw_hwp deletes and frees raw_hwp_page entries in raw_hwp_list: first llist_del_all, then kfree within llist_for_each_safe.

As of today, concurrent adding, deleting, and traversal on raw_hwp_list from hugetlb.c and/or memory-failure.c are fine with each other. Note this is guaranteed partly by the lock-free nature of llist, and partly by holding hugetlb_lock and/or mf_mutex. For example, as llist_del_all is lock-free with itself, folio_clear_hugetlb_hwpoison()s from __update_and_free_hugetlb_folio and memory_failure won't need explicit locking when freeing the raw_hwp_list. New code that manipulates raw_hwp_list must be careful to ensure the concurrency correctness.

---
 mm/memory-failure.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
mm/memory-failure.c +3 -5
index e245191e6b04..a08677dcf953 100644
@@ -1829,12 +1829,11 @@ static inline struct llist_head *raw_hwp_list_head(struct folio *folio)
 
 static unsigned long __folio_free_raw_hwp(struct folio *folio, bool move_flag)
 {
-	struct llist_head *head;
-	struct llist_node *t, *tnode;
+	struct llist_node *t, *tnode, *head;
 	unsigned long count = 0;
 
-	head = raw_hwp_list_head(folio);
-	llist_for_each_safe(tnode, t, head->first) {
+	head = llist_del_all(raw_hwp_list_head(folio));
+	llist_for_each_safe(tnode, t, head) {
 		struct raw_hwp_page *p = container_of(tnode, struct raw_hwp_page, node);
 
 		if (move_flag)
@@ -1844,7 +1843,6 @@ static unsigned long __folio_free_raw_hwp(struct folio *folio, bool move_flag)
 		kfree(p);
 		count++;
 	}
-	llist_del_all(head);
 	return count;
 }
 
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
maintainer Reviewed-by on v3 1/4
#2 lore raw

Re: [PATCH v3 1/4] mm/hwpoison: delete all entries before traversal in __folio_free_raw_hwp

Quoted 7 lines · Jiaqi Yan

Traversal on llist (e.g. llist_for_each_safe) is only safe AFTER entries are deleted from the llist. Correct the way __folio_free_raw_hwp deletes and frees raw_hwp_page entries in raw_hwp_list: first llist_del_all, then kfree within llist_for_each_safe.

As of today, concurrent adding, deleting, and traversal on raw_hwp_list from hugetlb.c and/or memory-failure.c are fine with each other. Note

I think there's a race on freeing the raw_hwp_list between unpoison_memory and __update_and_free_hugetlb_folio:

unpoison_memory __update_and_free_hugetlb_folio
if (folio_test_hwpoison)
folio_clear_hugetlb_hwpoison
folio_free_raw_hwp folio_free_raw_hwp
folio_test_clear_hwpoison

unpoison_memory and __update_and_free_hugetlb_folio can traverse and free the raw_hwp_list at the same time. And I believe your patch will fix the problem. Thanks.

Quoted 10 lines

this is guaranteed partly by the lock-free nature of llist, and partly by holding hugetlb_lock and/or mf_mutex. For example, as llist_del_all is lock-free with itself, folio_clear_hugetlb_hwpoison()s from __update_and_free_hugetlb_folio and memory_failure won't need explicit locking when freeing the raw_hwp_list. New code that manipulates raw_hwp_list must be careful to ensure the concurrency correctness.

Anyway, this patch looks good to me.

Thanks.

author
#3 lore raw

PATCH v3 2/4 mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON

In reply to Jiaqi Yan

Add the functionality, is_raw_hwp_subpage, to tell if a subpage of a hugetlb folio is a raw HWPOISON page. This functionality relies on RawHwpUnreliable to be not set; otherwise hugepage's raw HWPOISON list becomes meaningless.

is_raw_hwp_subpage needs to hold hugetlb_lock in order to synchronize with __get_huge_page_for_hwpoison, who iterates and inserts an entry to raw_hwp_list. llist itself doesn't ensure insertion is synchornized with the iterating used by __is_raw_hwp_list. Caller can minimize the overhead of lock cycles by first checking if folio / head page's HWPOISON flag is set.

Exports this functionality to be immediately used in the read operation for hugetlbfs.

---
 include/linux/hugetlb.h | 19 +++++++++++++++++++
 include/linux/mm.h      |  7 +++++++
 mm/hugetlb.c            | 10 ++++++++++
 mm/memory-failure.c     | 34 ++++++++++++++++++++++++----------
 4 files changed, 60 insertions(+), 10 deletions(-)
include/linux/hugetlb.h +19 -0
index ca3c8e10f24a..4a745af98525 100644
@@ -1007,6 +1007,25 @@ void hugetlb_register_node(struct node *node);
 void hugetlb_unregister_node(struct node *node);
 #endif
 
+/*
+ * Struct raw_hwp_page represents information about "raw error page",
+ * constructing singly linked list from ->_hugetlb_hwpoison field of folio.
+ */
+struct raw_hwp_page {
+	struct llist_node node;
+	struct page *page;
+};
+
+static inline struct llist_head *raw_hwp_list_head(struct folio *folio)
+{
+	return (struct llist_head *)&folio->_hugetlb_hwpoison;
+}
+
+/*
+ * Check if a given raw @subpage in a hugepage @folio is HWPOISON.
+ */
+bool is_raw_hwp_subpage(struct folio *folio, struct page *subpage);
+
 #else	/* CONFIG_HUGETLB_PAGE */
 struct hstate {};
 
include/linux/mm.h +7 -0
index 74f1be743ba2..edaa18b6f731 100644
@@ -3680,6 +3680,7 @@ extern const struct attribute_group memory_failure_attr_group;
 extern void memory_failure_queue(unsigned long pfn, int flags);
 extern int __get_huge_page_for_hwpoison(unsigned long pfn, int flags,
 					bool *migratable_cleared);
+extern bool __is_raw_hwp_subpage(struct folio *folio, struct page *subpage);
 void num_poisoned_pages_inc(unsigned long pfn);
 void num_poisoned_pages_sub(unsigned long pfn, long i);
 struct task_struct *task_early_kill(struct task_struct *tsk, int force_early);
@@ -3694,6 +3695,12 @@ static inline int __get_huge_page_for_hwpoison(unsigned long pfn, int flags,
 	return 0;
 }
 
+static inline bool __is_raw_hwp_subpage(struct folio *folio,
+					struct page *subpage)
+{
+	return false;
+}
+
 static inline void num_poisoned_pages_inc(unsigned long pfn)
 {
 }
mm/hugetlb.c +10 -0
index bce28cca73a1..9c608d2f6630 100644
@@ -7373,6 +7373,16 @@ int get_huge_page_for_hwpoison(unsigned long pfn, int flags,
 	return ret;
 }
 
+bool is_raw_hwp_subpage(struct folio *folio, struct page *subpage)
+{
+	bool ret;
+
+	spin_lock_irq(&hugetlb_lock);
+	ret = __is_raw_hwp_subpage(folio, subpage);
+	spin_unlock_irq(&hugetlb_lock);
+	return ret;
+}
+
 void folio_putback_active_hugetlb(struct folio *folio)
 {
 	spin_lock_irq(&hugetlb_lock);
mm/memory-failure.c +24 -10
index a08677dcf953..5b6c8ceb13c0 100644
@@ -1813,18 +1813,32 @@ EXPORT_SYMBOL_GPL(mf_dax_kill_procs);
 #endif /* CONFIG_FS_DAX */
 
 #ifdef CONFIG_HUGETLB_PAGE
-/*
- * Struct raw_hwp_page represents information about "raw error page",
- * constructing singly linked list from ->_hugetlb_hwpoison field of folio.
- */
-struct raw_hwp_page {
-	struct llist_node node;
-	struct page *page;
-};
 
-static inline struct llist_head *raw_hwp_list_head(struct folio *folio)
+bool __is_raw_hwp_subpage(struct folio *folio, struct page *subpage)
 {
-	return (struct llist_head *)&folio->_hugetlb_hwpoison;
+	struct llist_head *raw_hwp_head;
+	struct raw_hwp_page *p, *tmp;
+	bool ret = false;
+
+	if (!folio_test_hwpoison(folio))
+		return false;
+
+	/*
+	 * When RawHwpUnreliable is set, kernel lost track of which subpages
+	 * are HWPOISON. So return as if ALL subpages are HWPOISONed.
+	 */
+	if (folio_test_hugetlb_raw_hwp_unreliable(folio))
+		return true;
+
+	raw_hwp_head = raw_hwp_list_head(folio);
+	llist_for_each_entry_safe(p, tmp, raw_hwp_head->first, node) {
+		if (subpage == p->page) {
+			ret = true;
+			break;
+		}
+	}
+
+	return ret;
 }
 
 static unsigned long __folio_free_raw_hwp(struct folio *folio, bool move_flag)
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
#4 lore raw

Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON

Quoted 1 line · Jiaqi Yan

Add the functionality, is_raw_hwp_subpage, to tell if a subpage of a

This is incorrect naming. "subpage" was needed before we had the folio concept, but it should not be used any more. We have folios and pages now.

Also, abbreviating "hwpoison" as "hwp" seems like a bad idea to me. hwp is already used as an acronym by acpi, intel_pstate, some clock drivers, an ethernet driver, and a scsi driver.

maintainer
#5 lore raw

Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON

Quoted 10 lines · On Fri, 7 Jul 2023 21:31:39 +0100 Matthew Wilcox <[email protected]> wrote:
Quoted 1 line · Jiaqi Yan

Add the functionality, is_raw_hwp_subpage, to tell if a subpage of a

This is incorrect naming. "subpage" was needed before we had the folio concept, but it should not be used any more. We have folios and pages now.

Also, abbreviating "hwpoison" as "hwp" seems like a bad idea to me. hwp is already used as an acronym by acpi, intel_pstate, some clock drivers, an ethernet driver, and a scsi driver.

Thanks, Matthew. I'll merge v3 so we hit next -next (which actually won't be until Monday), and add a note that v4 is expected.

#6 lore raw

Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON

In reply to Matthew Wilcox
Quoted 6 lines · Matthew Wilcox
Quoted 1 line · Jiaqi Yan

Add the functionality, is_raw_hwp_subpage, to tell if a subpage of a

This is incorrect naming. "subpage" was needed before we had the folio concept, but it should not be used any more. We have folios and pages now.

I think we can address the raw hwpoison page by the offset in folio/hugepage to eliminate the concept of "subpage".

Quoted 4 lines

Also, abbreviating "hwpoison" as "hwp" seems like a bad idea to me. hwp is already used as an acronym by acpi, intel_pstate, some clock drivers, an ethernet driver, and a scsi driver.

I originally introduced the abbreviation "hwp" to avoid using a lengthy function name such as "folio_test_hugetlb_raw_hwpoison_unreliable()." Therefore, I prefer using "rawhwp" instead of a longer form like "raw_hwpoison," although I don't expect any confusion between "hwp" and "rawhwp."
As for "hwp_walk", another user of "hwp" in in mm/memory-failure.c, we can easily substitute it with "hwpoison_walk."

Thanks,
Naoya Horiguchi

author
#7 lore raw

Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON

Quoted 9 lines · Naoya Horiguchi
Quoted 6 lines · Matthew Wilcox
Quoted 1 line · Jiaqi Yan

Add the functionality, is_raw_hwp_subpage, to tell if a subpage of a

This is incorrect naming. "subpage" was needed before we had the folio concept, but it should not be used any more. We have folios and pages now.

Thanks for your comment, Matthew.

Quoted 8 lines

I think we can address the raw hwpoison page by the offset in folio/hugepage to eliminate the concept of "subpage".

Quoted 4 lines

Also, abbreviating "hwpoison" as "hwp" seems like a bad idea to me. hwp is already used as an acronym by acpi, intel_pstate, some clock drivers, an ethernet driver, and a scsi driver.

Quoted 5 lines

I originally introduced the abbreviation "hwp" to avoid using a lengthy function name such as "folio_test_hugetlb_raw_hwpoison_unreliable()." Therefore, I prefer using "rawhwp" instead of a longer form like "raw_hwpoison," although I don't expect any confusion between "hwp" and "rawhwp."

These are names in my mind, what do you think?
* is_rawhwp_page_in_hugepage
* is_raw_hwpoison_page_in_hugepage // I prefer this one
* folio_test_hugetlb_raw_hwpoison_page

Quoted 2 lines

As for "hwp_walk", another user of "hwp" in in mm/memory-failure.c, we can easily substitute it with "hwpoison_walk."

In this "hwp_walk" case, I also prefer "hwpoison" than "hwp". I can create a separate renaming patch.

Quoted 3 lines

Thanks,
Naoya Horiguchi

#8 lore raw

Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON

Quoted 28 lines · Jiaqi Yan
Quoted 9 lines · Naoya Horiguchi
Quoted 6 lines · Matthew Wilcox
Quoted 1 line · Jiaqi Yan

Add the functionality, is_raw_hwp_subpage, to tell if a subpage of a

This is incorrect naming. "subpage" was needed before we had the folio concept, but it should not be used any more. We have folios and pages now.

Thanks for your comment, Matthew.

Quoted 8 lines

I think we can address the raw hwpoison page by the offset in folio/hugepage to eliminate the concept of "subpage".

Quoted 4 lines

Also, abbreviating "hwpoison" as "hwp" seems like a bad idea to me. hwp is already used as an acronym by acpi, intel_pstate, some clock drivers, an ethernet driver, and a scsi driver.

Quoted 5 lines

I originally introduced the abbreviation "hwp" to avoid using a lengthy function name such as "folio_test_hugetlb_raw_hwpoison_unreliable()." Therefore, I prefer using "rawhwp" instead of a longer form like "raw_hwpoison," although I don't expect any confusion between "hwp" and "rawhwp."

Quoted 3 lines

These are names in my mind, what do you think?
* is_rawhwp_page_in_hugepage
* is_raw_hwpoison_page_in_hugepage // I prefer this one

This one is fine to me.

Quoted 7 lines

* folio_test_hugetlb_raw_hwpoison_page

Quoted 2 lines

As for "hwp_walk", another user of "hwp" in in mm/memory-failure.c, we can easily substitute it with "hwpoison_walk."

In this "hwp_walk" case, I also prefer "hwpoison" than "hwp". I can create a separate renaming patch.

Great, thank you.

- Naoya Horiguchi

maintainer Reviewed-by on v3 2/4
#9 lore raw

Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON

In reply to Jiaqi Yan
Quoted 35 lines · Jiaqi Yan

Add the functionality, is_raw_hwp_subpage, to tell if a subpage of a hugetlb folio is a raw HWPOISON page. This functionality relies on RawHwpUnreliable to be not set; otherwise hugepage's raw HWPOISON list becomes meaningless.

is_raw_hwp_subpage needs to hold hugetlb_lock in order to synchronize with __get_huge_page_for_hwpoison, who iterates and inserts an entry to raw_hwp_list. llist itself doesn't ensure insertion is synchornized with the iterating used by __is_raw_hwp_list. Caller can minimize the overhead of lock cycles by first checking if folio / head page's HWPOISON flag is set.

Exports this functionality to be immediately used in the read operation for hugetlbfs.

---
 include/linux/hugetlb.h | 19 +++++++++++++++++++
 include/linux/mm.h      |  7 +++++++
 mm/hugetlb.c            | 10 ++++++++++
 mm/memory-failure.c     | 34 ++++++++++++++++++++++++----------
 4 files changed, 60 insertions(+), 10 deletions(-)

...

mm/memory-failure.c +7 -2
-static inline struct llist_head *raw_hwp_list_head(struct folio *folio)
+bool __is_raw_hwp_subpage(struct folio *folio, struct page *subpage)
 {
-	return (struct llist_head *)&folio->_hugetlb_hwpoison;
+	struct llist_head *raw_hwp_head;
+	struct raw_hwp_page *p, *tmp;
+	bool ret = false;
+
+	if (!folio_test_hwpoison(folio))
+		return false;

mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1834

Quoted 10 lines
mm/memory-failure.c +10 -0
+
+	/*
+	 * When RawHwpUnreliable is set, kernel lost track of which subpages
+	 * are HWPOISON. So return as if ALL subpages are HWPOISONed.
+	 */
+	if (folio_test_hugetlb_raw_hwp_unreliable(folio))
+		return true;
+
+	raw_hwp_head = raw_hwp_list_head(folio);
+	llist_for_each_entry_safe(p, tmp, raw_hwp_head->first, node) {

Since we don't free the raw_hwp_list, does llist_for_each_entry works same as llist_for_each_entry_safe?

mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1842

Quoted 8 lines
mm/memory-failure.c +7 -0
+		if (subpage == p->page) {
+			ret = true;
+			break;
+		}
+	}
+
+	return ret;
 }

It seems there's a race between __is_raw_hwp_subpage and unpoison_memory:
unpoison_memory __is_raw_hwp_subpage
if (!folio_test_hwpoison(folio)) -- hwpoison is set
folio_free_raw_hwp llist_for_each_entry_safe raw_hwp_list
llist_del_all ..
folio_test_clear_hwpoison

But __is_raw_hwp_subpage is used in hugetlbfs, unpoison_memory couldn't reach here because there's a folio_mapping == NULL check before folio_free_raw_hwp.

Anyway, this patch looks good to me.

Thanks.

author
#10 lore raw

Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON

Quoted 59 lines · Miaohe Lin
Quoted 35 lines · Jiaqi Yan

Add the functionality, is_raw_hwp_subpage, to tell if a subpage of a hugetlb folio is a raw HWPOISON page. This functionality relies on RawHwpUnreliable to be not set; otherwise hugepage's raw HWPOISON list becomes meaningless.

is_raw_hwp_subpage needs to hold hugetlb_lock in order to synchronize with __get_huge_page_for_hwpoison, who iterates and inserts an entry to raw_hwp_list. llist itself doesn't ensure insertion is synchornized with the iterating used by __is_raw_hwp_list. Caller can minimize the overhead of lock cycles by first checking if folio / head page's HWPOISON flag is set.

Exports this functionality to be immediately used in the read operation for hugetlbfs.

---
 include/linux/hugetlb.h | 19 +++++++++++++++++++
 include/linux/mm.h      |  7 +++++++
 mm/hugetlb.c            | 10 ++++++++++
 mm/memory-failure.c     | 34 ++++++++++++++++++++++++----------
 4 files changed, 60 insertions(+), 10 deletions(-)

...

mm/memory-failure.c +7 -2
-static inline struct llist_head *raw_hwp_list_head(struct folio *folio)
+bool __is_raw_hwp_subpage(struct folio *folio, struct page *subpage)
 {
-     return (struct llist_head *)&folio->_hugetlb_hwpoison;
+     struct llist_head *raw_hwp_head;
+     struct raw_hwp_page *p, *tmp;
+     bool ret = false;
+
+     if (!folio_test_hwpoison(folio))
+             return false;

mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1834

Quoted 10 lines
mm/memory-failure.c +10 -0
+
+     /*
+      * When RawHwpUnreliable is set, kernel lost track of which subpages
+      * are HWPOISON. So return as if ALL subpages are HWPOISONed.
+      */
+     if (folio_test_hugetlb_raw_hwp_unreliable(folio))
+             return true;
+
+     raw_hwp_head = raw_hwp_list_head(folio);
+     llist_for_each_entry_safe(p, tmp, raw_hwp_head->first, node) {

Since we don't free the raw_hwp_list, does llist_for_each_entry works same as llist_for_each_entry_safe?

mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1842

Quoted 8 lines
mm/memory-failure.c +7 -0
+             if (subpage == p->page) {
+                     ret = true;
+                     break;
+             }
+     }
+
+     return ret;
 }
Quoted 7 lines

It seems there's a race between __is_raw_hwp_subpage and unpoison_memory:
unpoison_memory __is_raw_hwp_subpage
if (!folio_test_hwpoison(folio)) -- hwpoison is set
folio_free_raw_hwp llist_for_each_entry_safe raw_hwp_list
llist_del_all ..
folio_test_clear_hwpoison

Thanks Miaohe for raising this concern.

Quoted 2 lines

But __is_raw_hwp_subpage is used in hugetlbfs, unpoison_memory couldn't reach here because there's a folio_mapping == NULL check before folio_free_raw_hwp.

I agree. But in near future I do want to make __is_raw_hwp_subpage work for shared-mapping hugetlb, so it would be nice to work with unpoison_memory. It doesn't seem to me that holding mf_mutex in __is_raw_hwp_subpage is nice or even absolutely correct. Let me think if I can come up with something in v4.

Quoted 6 lines

Anyway, this patch looks good to me.

Thanks.

author
#11 lore raw

Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON

Quoted 43 lines · Jiaqi Yan
Quoted 41 lines · Miaohe Lin
Quoted 39 lines · Jiaqi Yan

Add the functionality, is_raw_hwp_subpage, to tell if a subpage of a hugetlb folio is a raw HWPOISON page. This functionality relies on RawHwpUnreliable to be not set; otherwise hugepage's raw HWPOISON list becomes meaningless.

is_raw_hwp_subpage needs to hold hugetlb_lock in order to synchronize with __get_huge_page_for_hwpoison, who iterates and inserts an entry to raw_hwp_list. llist itself doesn't ensure insertion is synchornized with the iterating used by __is_raw_hwp_list. Caller can minimize the overhead of lock cycles by first checking if folio / head page's HWPOISON flag is set.

Exports this functionality to be immediately used in the read operation for hugetlbfs.

---
 include/linux/hugetlb.h | 19 +++++++++++++++++++
 include/linux/mm.h      |  7 +++++++
 mm/hugetlb.c            | 10 ++++++++++
 mm/memory-failure.c     | 34 ++++++++++++++++++++++++----------
 4 files changed, 60 insertions(+), 10 deletions(-)

...

mm/memory-failure.c +11 -2
-static inline struct llist_head *raw_hwp_list_head(struct folio *folio)
+bool __is_raw_hwp_subpage(struct folio *folio, struct page *subpage)
 {
-     return (struct llist_head *)&folio->_hugetlb_hwpoison;
+     struct llist_head *raw_hwp_head;
+     struct raw_hwp_page *p, *tmp;
+     bool ret = false;
+
+     if (!folio_test_hwpoison(folio))
+             return false;
+
+     /*
+      * When RawHwpUnreliable is set, kernel lost track of which subpages
+      * are HWPOISON. So return as if ALL subpages are HWPOISONed.
Quoted 8 lines
Quoted 8 lines

mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1834

Quoted 6 lines
mm/memory-failure.c +6 -0
+      */
+     if (folio_test_hugetlb_raw_hwp_unreliable(folio))
+             return true;
+
+     raw_hwp_head = raw_hwp_list_head(folio);
+     llist_for_each_entry_safe(p, tmp, raw_hwp_head->first, node) {

Since we don't free the raw_hwp_list, does llist_for_each_entry works same as llist_for_each_entry_safe?

Sorry I missed this comment. Yes they are the same but llist_for_each_entry doesn't need "tmp". I will switch to llist_for_each_entry in v4.

Quoted 21 lines
Quoted 11 lines

mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1842

Quoted 10 lines
mm/memory-failure.c +7 -0
+             if (subpage == p->page) {
+                     ret = true;
+                     break;
+             }
+     }
+
+     return ret;
 }
Quoted 7 lines

It seems there's a race between __is_raw_hwp_subpage and unpoison_memory:
unpoison_memory __is_raw_hwp_subpage
if (!folio_test_hwpoison(folio)) -- hwpoison is set
folio_free_raw_hwp llist_for_each_entry_safe raw_hwp_list
llist_del_all ..
folio_test_clear_hwpoison

Thanks Miaohe for raising this concern.

Quoted 8 lines
Quoted 2 lines

But __is_raw_hwp_subpage is used in hugetlbfs, unpoison_memory couldn't reach here because there's a folio_mapping == NULL check before folio_free_raw_hwp.

I agree. But in near future I do want to make __is_raw_hwp_subpage work for shared-mapping hugetlb, so it would be nice to work with unpoison_memory. It doesn't seem to me that holding mf_mutex in __is_raw_hwp_subpage is nice or even absolutely correct. Let me think if I can come up with something in v4.

At my 2nd thought, if __is_raw_hwp_subpage simply takes mf_mutex before llist_for_each_entry, it will introduce a deadlock:

unpoison_memory __is_raw_hwp_subpage
held mf_mutex held hugetlb_lock
get_hwpoison_hugetlb_folio attempts mf_mutex
attempts hugetlb lock

Not for this patch series, but for future, is it a good idea to make mf_mutex available to hugetlb code? Then enforce the order of locking to be mf_mutex first, hugetlb_lock second? I believe this is the current locking pattern / order for try_memory_failure_hugetlb.

Quoted 7 lines
Quoted 7 lines

Anyway, this patch looks good to me.

Thanks.

#12 lore raw

Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON

Quoted 40 lines · Jiaqi Yan
Quoted 22 lines · Jiaqi Yan
Quoted 11 lines · Miaohe Lin

mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1842

Quoted 8 lines · Jiaqi Yan
mm/memory-failure.c +7 -0
+             if (subpage == p->page) {
+                     ret = true;
+                     break;
+             }
+     }
+
+     return ret;
 }
Quoted 7 lines

It seems there's a race between __is_raw_hwp_subpage and unpoison_memory:
unpoison_memory __is_raw_hwp_subpage
if (!folio_test_hwpoison(folio)) -- hwpoison is set
folio_free_raw_hwp llist_for_each_entry_safe raw_hwp_list
llist_del_all ..
folio_test_clear_hwpoison

Thanks Miaohe for raising this concern.

Quoted 8 lines
Quoted 2 lines

But __is_raw_hwp_subpage is used in hugetlbfs, unpoison_memory couldn't reach here because there's a folio_mapping == NULL check before folio_free_raw_hwp.

I agree. But in near future I do want to make __is_raw_hwp_subpage work for shared-mapping hugetlb, so it would be nice to work with unpoison_memory. It doesn't seem to me that holding mf_mutex in __is_raw_hwp_subpage is nice or even absolutely correct. Let me think if I can come up with something in v4.

At my 2nd thought, if __is_raw_hwp_subpage simply takes mf_mutex before llist_for_each_entry, it will introduce a deadlock:

unpoison_memory __is_raw_hwp_subpage
held mf_mutex held hugetlb_lock
get_hwpoison_hugetlb_folio attempts mf_mutex
attempts hugetlb lock

Quoted 4 lines

Not for this patch series, but for future, is it a good idea to make mf_mutex available to hugetlb code? Then enforce the order of locking to be mf_mutex first, hugetlb_lock second? I believe this is the current locking pattern / order for try_memory_failure_hugetlb.

I think only holding mf_mutex in __is_raw_hwp_subpage would be sufficient to prevent races with unpoison_memory. memory failure code needs to take both mf_mutex and hugetlb_lock. The hugetlb lock is to prevent hugetlb page state changes. IIUC, __is_raw_hwp_subpage is only taking hugetlb_lock to prevent races with memory failure code.

Of course, I could be missing something as there are subtle issues with locking in the memory failure code.

Signature · 1 line
Mike Kravetz
author
#13 lore raw

Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON

Quoted 47 lines · Mike Kravetz
Quoted 40 lines · Jiaqi Yan
Quoted 22 lines · Jiaqi Yan
Quoted 11 lines · Miaohe Lin

mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1842

Quoted 8 lines · Jiaqi Yan
mm/memory-failure.c +7 -0
+             if (subpage == p->page) {
+                     ret = true;
+                     break;
+             }
+     }
+
+     return ret;
 }
Quoted 7 lines

It seems there's a race between __is_raw_hwp_subpage and unpoison_memory:
unpoison_memory __is_raw_hwp_subpage
if (!folio_test_hwpoison(folio)) -- hwpoison is set
folio_free_raw_hwp llist_for_each_entry_safe raw_hwp_list
llist_del_all ..
folio_test_clear_hwpoison

Thanks Miaohe for raising this concern.

Quoted 8 lines
Quoted 2 lines

But __is_raw_hwp_subpage is used in hugetlbfs, unpoison_memory couldn't reach here because there's a folio_mapping == NULL check before folio_free_raw_hwp.

I agree. But in near future I do want to make __is_raw_hwp_subpage work for shared-mapping hugetlb, so it would be nice to work with unpoison_memory. It doesn't seem to me that holding mf_mutex in __is_raw_hwp_subpage is nice or even absolutely correct. Let me think if I can come up with something in v4.

At my 2nd thought, if __is_raw_hwp_subpage simply takes mf_mutex before llist_for_each_entry, it will introduce a deadlock:

unpoison_memory __is_raw_hwp_subpage
held mf_mutex held hugetlb_lock
get_hwpoison_hugetlb_folio attempts mf_mutex
attempts hugetlb lock

Quoted 4 lines

Not for this patch series, but for future, is it a good idea to make mf_mutex available to hugetlb code? Then enforce the order of locking to be mf_mutex first, hugetlb_lock second? I believe this is the current locking pattern / order for try_memory_failure_hugetlb.

Quoted 5 lines

I think only holding mf_mutex in __is_raw_hwp_subpage would be sufficient to prevent races with unpoison_memory. memory failure code needs to take both mf_mutex and hugetlb_lock. The hugetlb lock is to prevent hugetlb page state changes. IIUC, __is_raw_hwp_subpage is only taking hugetlb_lock to prevent races with memory failure code.

Thanks Mike, I think mf_mutex is a simple and correct solution. I will drop hugetlb_lock from __is_raw_hwp_subpage in v4.

Quoted 5 lines

Of course, I could be missing something as there are subtle issues with locking in the memory failure code.
--
Mike Kravetz

maintainer
#14 lore raw

Re: [PATCH v3 2/4] mm/hwpoison: check if a subpage of a hugetlb folio is raw HWPOISON

In reply to Mike Kravetz
Quoted 41 lines · Mike Kravetz
Quoted 40 lines · Jiaqi Yan
Quoted 22 lines · Jiaqi Yan
Quoted 11 lines · Miaohe Lin

mm/memory-failure.c EXPORT_SYMBOL_GPL(mf_dax_kill_procs); · line 1842

Quoted 8 lines · Jiaqi Yan
mm/memory-failure.c +7 -0
+             if (subpage == p->page) {
+                     ret = true;
+                     break;
+             }
+     }
+
+     return ret;
 }
Quoted 7 lines

It seems there's a race between __is_raw_hwp_subpage and unpoison_memory:
unpoison_memory __is_raw_hwp_subpage
if (!folio_test_hwpoison(folio)) -- hwpoison is set
folio_free_raw_hwp llist_for_each_entry_safe raw_hwp_list
llist_del_all ..
folio_test_clear_hwpoison

Thanks Miaohe for raising this concern.

Quoted 8 lines
Quoted 2 lines

But __is_raw_hwp_subpage is used in hugetlbfs, unpoison_memory couldn't reach here because there's a folio_mapping == NULL check before folio_free_raw_hwp.

I agree. But in near future I do want to make __is_raw_hwp_subpage work for shared-mapping hugetlb, so it would be nice to work with unpoison_memory. It doesn't seem to me that holding mf_mutex in __is_raw_hwp_subpage is nice or even absolutely correct. Let me think if I can come up with something in v4.

At my 2nd thought, if __is_raw_hwp_subpage simply takes mf_mutex before llist_for_each_entry, it will introduce a deadlock:

unpoison_memory __is_raw_hwp_subpage
held mf_mutex held hugetlb_lock
get_hwpoison_hugetlb_folio attempts mf_mutex
attempts hugetlb lock

Quoted 7 lines
Quoted 4 lines

Not for this patch series, but for future, is it a good idea to make mf_mutex available to hugetlb code? Then enforce the order of locking to be mf_mutex first, hugetlb_lock second? I believe this is the current locking pattern / order for try_memory_failure_hugetlb.

I think only holding mf_mutex in __is_raw_hwp_subpage would be sufficient to prevent races with unpoison_memory. memory failure code needs to take

Since soft_offline_page, memory_failure and unpoison_memory both holds mf_mutex, I think this should be enough to prevent races between them too.

Thanks.

author
#15 lore raw

PATCH v3 3/4 hugetlbfs: improve read HWPOISON hugepage

In reply to Jiaqi Yan

When a hugepage contains HWPOISON pages, read() fails to read any byte of the hugepage and returns -EIO, although many bytes in the HWPOISON hugepage are readable.

Improve this by allowing hugetlbfs_read_iter returns as many bytes as possible. For a requested range [offset, offset + len) that contains HWPOISON page, return [offset, first HWPOISON page addr); the next read attempt will fail and return -EIO.

---
 fs/hugetlbfs/inode.c | 58 +++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 52 insertions(+), 6 deletions(-)
fs/hugetlbfs/inode.c +52 -6
index 7b17ccfa039d..c2b807d37f85 100644
@@ -282,6 +282,42 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr,
 }
 #endif
 
+/*
+ * Someone wants to read @bytes from a HWPOISON hugetlb @page from @offset.
+ * Returns the maximum number of bytes one can read without touching the 1st raw
+ * HWPOISON subpage.
+ *
+ * The implementation borrows the iteration logic from copy_page_to_iter*.
+ */
+static size_t adjust_range_hwpoison(struct page *page, size_t offset, size_t bytes)
+{
+	size_t n = 0;
+	size_t res = 0;
+	struct folio *folio = page_folio(page);
+
+	/* First subpage to start the loop. */
+	page += offset / PAGE_SIZE;
+	offset %= PAGE_SIZE;
+	while (1) {
+		if (is_raw_hwp_subpage(folio, page))
+			break;
+
+		/* Safe to read n bytes without touching HWPOISON subpage. */
+		n = min(bytes, (size_t)PAGE_SIZE - offset);
+		res += n;
+		bytes -= n;
+		if (!bytes || !n)
+			break;
+		offset += n;
+		if (offset == PAGE_SIZE) {
+			page++;
+			offset = 0;
+		}
+	}
+
+	return res;
+}
+
 /*
  * Support for read() - Find the page attached to f_mapping and copy out the
  * data. This provides functionality similar to filemap_read().
@@ -300,7 +336,7 @@ static ssize_t hugetlbfs_read_iter(struct kiocb *iocb, struct iov_iter *to)
 
 	while (iov_iter_count(to)) {
 		struct page *page;
-		size_t nr, copied;
+		size_t nr, copied, want;
 
 		/* nr is the maximum number of bytes to copy from this page */
 		nr = huge_page_size(h);
@@ -328,16 +364,26 @@ static ssize_t hugetlbfs_read_iter(struct kiocb *iocb, struct iov_iter *to)
 		} else {
 			unlock_page(page);
 
-			if (PageHWPoison(page)) {
-				put_page(page);
-				retval = -EIO;
-				break;
+			if (!PageHWPoison(page))
+				want = nr;
+			else {
+				/*
+				 * Adjust how many bytes safe to read without
+				 * touching the 1st raw HWPOISON subpage after
+				 * offset.
+				 */
+				want = adjust_range_hwpoison(page, offset, nr);
+				if (want == 0) {
+					put_page(page);
+					retval = -EIO;
+					break;
+				}
 			}
 
 			/*
 			 * We have the page, copy it to user space buffer.
 			 */
-			copied = copy_page_to_iter(page, offset, nr, to);
+			copied = copy_page_to_iter(page, offset, want, to);
 			put_page(page);
 		}
 		offset += copied;
Signature · 1 line
2.41.0.255.g8b1d071c50-goog
author
#16 lore raw

PATCH v3 4/4 selftests/mm: add tests for HWPOISON hugetlbfs read

In reply to Jiaqi Yan

Add tests for the improvement made to read operation on HWPOISON hugetlb page with different read granularities. For each chunk size, three read scenarios are tested:
1. Simple regression test on read without HWPOISON.
2. Sequential read page by page should succeed until encounters the 1st raw HWPOISON subpage.
3. After skip a raw HWPOISON subpage by lseek, read()s always succeed.

---
 tools/testing/selftests/mm/.gitignore         |   1 +
 tools/testing/selftests/mm/Makefile           |   1 +
 .../selftests/mm/hugetlb-read-hwpoison.c      | 322 ++++++++++++++++++
 3 files changed, 324 insertions(+)
 create mode 100644 tools/testing/selftests/mm/hugetlb-read-hwpoison.c
tools/testing/selftests/mm/.gitignore +1 -0
index 7e2a982383c0..cdc9ce4426b9 100644
@@ -5,6 +5,7 @@ hugepage-mremap
 hugepage-shm
 hugepage-vmemmap
 hugetlb-madvise
+hugetlb-read-hwpoison
 khugepaged
 map_hugetlb
 map_populate
tools/testing/selftests/mm/Makefile +1 -0
index 66d7c07dc177..b7fce9073279 100644
@@ -41,6 +41,7 @@ TEST_GEN_PROGS += gup_longterm
 TEST_GEN_PROGS += gup_test
 TEST_GEN_PROGS += hmm-tests
 TEST_GEN_PROGS += hugetlb-madvise
+TEST_GEN_PROGS += hugetlb-read-hwpoison
 TEST_GEN_PROGS += hugepage-mmap
 TEST_GEN_PROGS += hugepage-mremap
 TEST_GEN_PROGS += hugepage-shm
tools/testing/selftests/mm/hugetlb-read-hwpoison.c new file +322 -0

322 lines not on this page · raw message

Signature · 1 line
2.41.0.255.g8b1d071c50-goog