Skip to contentSkip to conversation

mm: hugetlb: add hwcrp_hugepages to record memory failure on hugetlbfs

PATCH In review wangbin 3 participants 5 messages Jun 23, 2021 – Jul 11, 2021

Subsystems (2)
lore
Share & tools
lore

wangbin

author
#0 lore raw

PATCH v2 1/1 mm: hugetlb: add hwcrp_hugepages to record memory failure on hugetlbfs

From: Bin Wang <[email protected]>

In the current hugetlbfs memory failure handler, reserved huge page counts are used to record the number of huge pages with hwposion. There are two problems:

1. We call hugetlb_fix_reserve_counts() to change reserved counts in hugetlbfs_error_remove_page(). But this function is only called if hugetlb_unreserve_pages() fails, and hugetlb_unreserve_pages() fails only if kmalloc in region_del() fails, which is almost impossible. As a result, the reserved count is not corrected as expected when a memory failure occurs.

2. Reserved counts is designed to display the number of hugepages reserved at mmap() time. This means that even if we fix the first issue, reserved counts will be confusing because we can't tell if it's hwposion or reserved hugepage.

This patch adds hardware corrput huge pages counts to record memory failure on hugetlbfs instead of reserved counts.

---
 fs/hugetlbfs/inode.c    |  3 +--
 include/linux/hugetlb.h |  3 +++
 mm/hugetlb.c            | 30 ++++++++++++++++++++++++++++++
 3 files changed, 34 insertions(+), 2 deletions(-)
fs/hugetlbfs/inode.c +1 -2
index 926eeb9bf4eb..ffb6e7b6756b 100644
@@ -986,8 +986,7 @@ static int hugetlbfs_error_remove_page(struct address_space *mapping,
 	pgoff_t index = page->index;
 
 	remove_huge_page(page);
-	if (unlikely(hugetlb_unreserve_pages(inode, index, index + 1, 1)))
-		hugetlb_fix_reserve_counts(inode);
+	hugetlb_fix_hwcrp_counts(page);
 
 	return 0;
 }
include/linux/hugetlb.h +3 -0
index f7ca1a3870ea..1d5bada80aa5 100644
@@ -171,6 +171,7 @@ void putback_active_hugepage(struct page *page);
 void move_hugetlb_state(struct page *oldpage, struct page *newpage, int reason);
 void free_huge_page(struct page *page);
 void hugetlb_fix_reserve_counts(struct inode *inode);
+void hugetlb_fix_hwcrp_counts(struct page *page);
 extern struct mutex *hugetlb_fault_mutex_table;
 u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx);
 
@@ -602,12 +603,14 @@ struct hstate {
 	unsigned long free_huge_pages;
 	unsigned long resv_huge_pages;
 	unsigned long surplus_huge_pages;
+	unsigned long hwcrp_huge_pages;
 	unsigned long nr_overcommit_huge_pages;
 	struct list_head hugepage_activelist;
 	struct list_head hugepage_freelists[MAX_NUMNODES];
 	unsigned int nr_huge_pages_node[MAX_NUMNODES];
 	unsigned int free_huge_pages_node[MAX_NUMNODES];
 	unsigned int surplus_huge_pages_node[MAX_NUMNODES];
+	unsigned int hwcrp_huge_pages_node[MAX_NUMNODES];
 #ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP
 	unsigned int nr_free_vmemmap_pages;
 #endif
mm/hugetlb.c +30 -0
index 760b5fb836b8..3e6385381db7 100644
@@ -763,6 +763,15 @@ void hugetlb_fix_reserve_counts(struct inode *inode)
 		pr_warn("hugetlb: Huge Page Reserved count may go negative.\n");
 }
 
+void hugetlb_fix_hwcrp_counts(struct page *page)
+{
+	struct hstate *h = &default_hstate;
+	int nid = page_to_nid(page);
+
+	h->hwcrp_huge_pages++;
+	h->hwcrp_huge_pages_node[nid]++;
+}
+
 /*
  * Count and return the number of huge pages in the reserve map
  * that intersect with the range [f, t).
@@ -3293,12 +3302,30 @@ static ssize_t surplus_hugepages_show(struct kobject *kobj,
 }
 HSTATE_ATTR_RO(surplus_hugepages);
 
+static ssize_t hwcrp_hugepages_show(struct kobject *kobj,
+					struct kobj_attribute *attr, char *buf)
+{
+	struct hstate *h;
+	unsigned long hwcrp_huge_pages;
+	int nid;
+
+	h = kobj_to_hstate(kobj, &nid);
+	if (nid == NUMA_NO_NODE)
+		hwcrp_huge_pages = h->hwcrp_huge_pages;
+	else
+		hwcrp_huge_pages = h->hwcrp_huge_pages_node[nid];
+
+	return sysfs_emit(buf, "%lu\n", hwcrp_huge_pages);
+}
+HSTATE_ATTR_RO(hwcrp_hugepages);
+
 static struct attribute *hstate_attrs[] = {
 	&nr_hugepages_attr.attr,
 	&nr_overcommit_hugepages_attr.attr,
 	&free_hugepages_attr.attr,
 	&resv_hugepages_attr.attr,
 	&surplus_hugepages_attr.attr,
+	&hwcrp_hugepages_attr.attr,
 #ifdef CONFIG_NUMA
 	&nr_hugepages_mempolicy_attr.attr,
 #endif
@@ -3368,6 +3395,7 @@ static struct attribute *per_node_hstate_attrs[] = {
 	&nr_hugepages_attr.attr,
 	&free_hugepages_attr.attr,
 	&surplus_hugepages_attr.attr,
+	&hwcrp_hugepages_attr.attr,
 	NULL,
 };
 
@@ -3862,11 +3890,13 @@ void hugetlb_report_meminfo(struct seq_file *m)
 				   "HugePages_Free:    %5lu\n"
 				   "HugePages_Rsvd:    %5lu\n"
 				   "HugePages_Surp:    %5lu\n"
+				   "HugePages_Hwcrp:   %5lu\n"
 				   "Hugepagesize:   %8lu kB\n",
 				   count,
 				   h->free_huge_pages,
 				   h->resv_huge_pages,
 				   h->surplus_huge_pages,
+				   h->hwcrp_huge_pages,
 				   huge_page_size(h) / SZ_1K);
 	}
 
Signature · 1 line
2.23.0
#1 lore raw
Quoted 5 lines · wangbin

From: Bin Wang <[email protected]>

In the current hugetlbfs memory failure handler, reserved huge page counts are used to record the number of huge pages with hwposion. There are two problems:

Plese review the comments from the first patch. Reserved huge page counts are NOT used to record the number of huge pages with hwposion.

Quoted 28 lines

1. We call hugetlb_fix_reserve_counts() to change reserved counts in hugetlbfs_error_remove_page(). But this function is only called if hugetlb_unreserve_pages() fails, and hugetlb_unreserve_pages() fails only if kmalloc in region_del() fails, which is almost impossible. As a result, the reserved count is not corrected as expected when a memory failure occurs.

2. Reserved counts is designed to display the number of hugepages reserved at mmap() time. This means that even if we fix the first issue, reserved counts will be confusing because we can't tell if it's hwposion or reserved hugepage.

This patch adds hardware corrput huge pages counts to record memory failure on hugetlbfs instead of reserved counts.

---
 fs/hugetlbfs/inode.c    |  3 +--
 include/linux/hugetlb.h |  3 +++
 mm/hugetlb.c            | 30 ++++++++++++++++++++++++++++++
 3 files changed, 34 insertions(+), 2 deletions(-)
fs/hugetlbfs/inode.c +0 -0
index 926eeb9bf4eb..ffb6e7b6756b 100644

fs/hugetlbfs/inode.c static int hugetlbfs_error_remove_page(struct address_space *mapping, · line 990

Quoted 5 lines
fs/hugetlbfs/inode.c +0 -2
@@ -986,8 +986,7 @@ static int hugetlbfs_error_remove_page(struct address_space *mapping,
 	pgoff_t index = page->index;
 
 	remove_huge_page(page);
-	if (unlikely(hugetlb_unreserve_pages(inode, index, index + 1, 1)))
-		hugetlb_fix_reserve_counts(inode);

As mentioned, huge page reserve counts are not used to record number of poisioned pages. The calls to hugetlb_unreserve_pages and possibly hugetlb_fix_reserve_counts are necessary for reserve accounting. They can not be removed.

fs/hugetlbfs/inode.c static int hugetlbfs_error_remove_page(struct address_space *mapping, · line 989

Quoted 1 line
fs/hugetlbfs/inode.c +1 -0
+	hugetlb_fix_hwcrp_counts(page);

This new routine just counts memory errors on 'in use' huge pages. I do not see a call anywhere to count memory errors on huge pages not in use.

Quoted 17 lines
fs/hugetlbfs/inode.c +0 -0
 	return 0;
 }
include/linux/hugetlb.h +1 -0
index f7ca1a3870ea..1d5bada80aa5 100644
@@ -171,6 +171,7 @@ void putback_active_hugepage(struct page *page);
 void move_hugetlb_state(struct page *oldpage, struct page *newpage, int reason);
 void free_huge_page(struct page *page);
 void hugetlb_fix_reserve_counts(struct inode *inode);
+void hugetlb_fix_hwcrp_counts(struct page *page);
 extern struct mutex *hugetlb_fault_mutex_table;
 u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx);
 
@@ -602,12 +603,14 @@ struct hstate {
 	unsigned long free_huge_pages;

include/linux/hugetlb.h struct hstate { · line 613

Quoted 10 lines
include/linux/hugetlb.h +2 -0
 	unsigned long resv_huge_pages;
 	unsigned long surplus_huge_pages;
+	unsigned long hwcrp_huge_pages;
 	unsigned long nr_overcommit_huge_pages;
 	struct list_head hugepage_activelist;
 	struct list_head hugepage_freelists[MAX_NUMNODES];
 	unsigned int nr_huge_pages_node[MAX_NUMNODES];
 	unsigned int free_huge_pages_node[MAX_NUMNODES];
 	unsigned int surplus_huge_pages_node[MAX_NUMNODES];
+	unsigned int hwcrp_huge_pages_node[MAX_NUMNODES];

I understand your requirement to count the number of memory errors on hugetlb pages. However, we need to think carefully about we represent that count.

Noaya, do you have opinions on where would be the best place to store this information? The hugetlb memory error code has the comment 'needs work'. Ideally, we could isolate memory errors to a single base (4K for x86) page and free the remaining base pages to buddy. We could also potentially allocate a 'replacement' hugetlb page doing something like alloc_and_dissolve_huge_page.

If we get an error on a hugetlb page and can isolate it to a base page and replace the huge page, is it still a huge page memory error?

IMO, we should work on isolating memory errors to a base page and replacing the huge page. Then, the existing count of base pages with memory errors would be sufficient?

This is something I would like to work, but I have higher priorities right now.

Signature · 1 line
Mike Kravetz
author
#2 lore raw
Quoted 7 lines
Quoted 7 lines
fs/hugetlbfs/inode.c +0 -0
index 926eeb9bf4eb..ffb6e7b6756b 100644
@@ -986,8 +986,7 @@ static int hugetlbfs_error_remove_page(struct address_space *mapping,
 	pgoff_t index = page->index;
 
Quoted 8 lines

fs/hugetlbfs/inode.c static int hugetlbfs_error_remove_page(struct address_space *mapping, · line 990

Quoted 3 lines
fs/hugetlbfs/inode.c +0 -2
 	remove_huge_page(page);
-	if (unlikely(hugetlb_unreserve_pages(inode, index, index + 1, 1)))
-		hugetlb_fix_reserve_counts(inode);

As mentioned, huge page reserve counts are not used to record number of poisioned pages. The calls to hugetlb_unreserve_pages and possibly hugetlb_fix_reserve_counts are necessary for reserve accounting. They can not be removed.

Thanks for your explanation very much. I didn't get the point of the comments from the first patch. hugetlb_fix_reserve_counts() shouldn't be removed and I will fix this.

Quoted 5 lines

fs/hugetlbfs/inode.c static int hugetlbfs_error_remove_page(struct address_space *mapping, · line 989

Quoted 1 line
fs/hugetlbfs/inode.c +1 -0
+	hugetlb_fix_hwcrp_counts(page);

This new routine just counts memory errors on 'in use' huge pages. I do not see a call anywhere to count memory errors on huge pages not in use.

It's my oversight. I should have considered this situation. I tested it with hwcrp_hugepages count, and this is the result:

# cat /proc/meminfo |grep -E 'HugePages_|Hard'
HardwareCorrupted:     0 kB
HugePages_Total:      64
HugePages_Free:       64
HugePages_Rsvd:        0
HugePages_Surp:        0
HugePages_Hwcrp:       0
No count changes, even the HardwareCorrupted. I'm not sure if this is
normal. This is what happens in kernel(stable master branch):
static int memory_failure_hugetlb(unsigned long pfn, int flags)
{
	...
	/* TestSetPageHWPoison return 0 */
	if (TestSetPageHWPoison(head)) {
		...
	}
	num_poisoned_pages_inc(); /* HardwareCorrupted += PAGE_SIZE */

	/* get_hwpoison_page() return 0 */
	if (!(flags & MF_COUNT_INCREASED) && !get_hwpoison_page(p, flags, 0)) {
		/*
		 * Check "filter hit" and "race with other subpage."
		 */
		lock_page(head);
		/* PageHWPoison() return 1 */
		if (PageHWPoison(head)) {
			/* (p != head && TestSetPageHWPoison(head)) is hit */
			if ((hwpoison_filter(p) && TestClearPageHWPoison(p))
			    || (p != head && TestSetPageHWPoison(head))) {
				/* HardwareCorrupted -= PAGE_SIZE */
				num_poisoned_pages_dec();
				unlock_page(head);
				return 0;
			}
		}
		...
	}
	...
}

It seems like that memory errors on huge pages not in use hit the "race with other subpage". I think I shouldn't add hwcrp_hugepages in this routine. Maybe we need more conditions to distinguish this.

Quoted 42 lines
Quoted 17 lines
fs/hugetlbfs/inode.c +0 -0
 	return 0;
 }
include/linux/hugetlb.h +1 -0
index f7ca1a3870ea..1d5bada80aa5 100644
@@ -171,6 +171,7 @@ void putback_active_hugepage(struct page *page);
 void move_hugetlb_state(struct page *oldpage, struct page *newpage, int reason);
 void free_huge_page(struct page *page);
 void hugetlb_fix_reserve_counts(struct inode *inode);
+void hugetlb_fix_hwcrp_counts(struct page *page);
 extern struct mutex *hugetlb_fault_mutex_table;
 u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx);
 
@@ -602,12 +603,14 @@ struct hstate {
 	unsigned long free_huge_pages;

include/linux/hugetlb.h struct hstate { · line 613

Quoted 10 lines
include/linux/hugetlb.h +2 -0
 	unsigned long resv_huge_pages;
 	unsigned long surplus_huge_pages;
+	unsigned long hwcrp_huge_pages;
 	unsigned long nr_overcommit_huge_pages;
 	struct list_head hugepage_activelist;
 	struct list_head hugepage_freelists[MAX_NUMNODES];
 	unsigned int nr_huge_pages_node[MAX_NUMNODES];
 	unsigned int free_huge_pages_node[MAX_NUMNODES];
 	unsigned int surplus_huge_pages_node[MAX_NUMNODES];
+	unsigned int hwcrp_huge_pages_node[MAX_NUMNODES];

I understand your requirement to count the number of memory errors on hugetlb pages. However, we need to think carefully about we represent that count.

Noaya, do you have opinions on where would be the best place to store this information? The hugetlb memory error code has the comment 'needs work'. Ideally, we could isolate memory errors to a single base (4K for x86) page and free the remaining base pages to buddy. We could also potentially allocate a 'replacement' hugetlb page doing something like alloc_and_dissolve_huge_page.

If we get an error on a hugetlb page and can isolate it to a base page and replace the huge page, is it still a huge page memory error?

Quoted 6 lines

IMO, we should work on isolating memory errors to a base page and replacing the huge page. Then, the existing count of base pages with memory errors would be sufficient?

This is something I would like to work, but I have higher priorities right now.

Yes, splitting the huge pages and isolating a base page is ideal. And we do this with dissolve_free_huge_page() when page_mapping() return NULL. I think there is a reason(but I do not get it) why we don't split huge pags in hugetlbfs_error_remove_page() or after. So I choose to add a new count.

--
Bin Wang

#3 lore raw

Hi Bin, Mike,

Quoted 68 lines · Bin Wang
Quoted 7 lines
Quoted 7 lines
fs/hugetlbfs/inode.c +0 -0
index 926eeb9bf4eb..ffb6e7b6756b 100644
@@ -986,8 +986,7 @@ static int hugetlbfs_error_remove_page(struct address_space *mapping,
 	pgoff_t index = page->index;
 
Quoted 8 lines

fs/hugetlbfs/inode.c static int hugetlbfs_error_remove_page(struct address_space *mapping, · line 990

Quoted 3 lines
fs/hugetlbfs/inode.c +0 -2
 	remove_huge_page(page);
-	if (unlikely(hugetlb_unreserve_pages(inode, index, index + 1, 1)))
-		hugetlb_fix_reserve_counts(inode);

As mentioned, huge page reserve counts are not used to record number of poisioned pages. The calls to hugetlb_unreserve_pages and possibly hugetlb_fix_reserve_counts are necessary for reserve accounting. They can not be removed.

Thanks for your explanation very much. I didn't get the point of the comments from the first patch. hugetlb_fix_reserve_counts() shouldn't be removed and I will fix this.

Quoted 5 lines

fs/hugetlbfs/inode.c static int hugetlbfs_error_remove_page(struct address_space *mapping, · line 989

Quoted 1 line
fs/hugetlbfs/inode.c +1 -0
+	hugetlb_fix_hwcrp_counts(page);

This new routine just counts memory errors on 'in use' huge pages. I do not see a call anywhere to count memory errors on huge pages not in use.

It's my oversight. I should have considered this situation. I tested it with hwcrp_hugepages count, and this is the result:

# cat /proc/meminfo |grep -E 'HugePages_|Hard'
HardwareCorrupted:     0 kB
HugePages_Total:      64
HugePages_Free:       64
HugePages_Rsvd:        0
HugePages_Surp:        0
HugePages_Hwcrp:       0
No count changes, even the HardwareCorrupted. I'm not sure if this is
normal. This is what happens in kernel(stable master branch):
static int memory_failure_hugetlb(unsigned long pfn, int flags)
{
	...
	/* TestSetPageHWPoison return 0 */
	if (TestSetPageHWPoison(head)) {
		...
	}
	num_poisoned_pages_inc(); /* HardwareCorrupted += PAGE_SIZE */

	/* get_hwpoison_page() return 0 */
	if (!(flags & MF_COUNT_INCREASED) && !get_hwpoison_page(p, flags, 0)) {
		/*
		 * Check "filter hit" and "race with other subpage."
		 */
		lock_page(head);
		/* PageHWPoison() return 1 */
		if (PageHWPoison(head)) {
			/* (p != head && TestSetPageHWPoison(head)) is hit */
			if ((hwpoison_filter(p) && TestClearPageHWPoison(p))
			    || (p != head && TestSetPageHWPoison(head))) {
				/* HardwareCorrupted -= PAGE_SIZE */
				num_poisoned_pages_dec();
				unlock_page(head);
				return 0;
			}
		}
		...
	}
	...
}
Quoted 3 lines

It seems like that memory errors on huge pages not in use hit the "race with other subpage". I think I shouldn't add hwcrp_hugepages in this routine. Maybe we need more conditions to distinguish this.

BTW, I think that the race handled by this if-block will be no longer necessary if mf_mutex is extended to cover unpoison_memory() by patch "mm/hwpoison: mf_mutex for soft offline and unpoison" [1]. I'm thinking of updating patch 2/6 [2] to remove this if-block.

[1]: https://lore.kernel.org/linux-mm/[email protected]/
[2]: https://lore.kernel.org/linux-mm/[email protected]/

# .. so you don't have to care for the detail of this part.
Quoted 33 lines
Quoted 33 lines
Quoted 18 lines
fs/hugetlbfs/inode.c +0 -0
 	return 0;
 }
include/linux/hugetlb.h +1 -0
index f7ca1a3870ea..1d5bada80aa5 100644
@@ -171,6 +171,7 @@ void putback_active_hugepage(struct page *page);
 void move_hugetlb_state(struct page *oldpage, struct page *newpage, int reason);
 void free_huge_page(struct page *page);
 void hugetlb_fix_reserve_counts(struct inode *inode);
+void hugetlb_fix_hwcrp_counts(struct page *page);
 extern struct mutex *hugetlb_fault_mutex_table;
 u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx);
 
@@ -602,12 +603,14 @@ struct hstate {
 	unsigned long free_huge_pages;

include/linux/hugetlb.h struct hstate { · line 613

Quoted 10 lines
include/linux/hugetlb.h +2 -0
 	unsigned long resv_huge_pages;
 	unsigned long surplus_huge_pages;
+	unsigned long hwcrp_huge_pages;
 	unsigned long nr_overcommit_huge_pages;
 	struct list_head hugepage_activelist;
 	struct list_head hugepage_freelists[MAX_NUMNODES];
 	unsigned int nr_huge_pages_node[MAX_NUMNODES];
 	unsigned int free_huge_pages_node[MAX_NUMNODES];
 	unsigned int surplus_huge_pages_node[MAX_NUMNODES];
+	unsigned int hwcrp_huge_pages_node[MAX_NUMNODES];

I understand your requirement to count the number of memory errors on hugetlb pages. However, we need to think carefully about we represent that count.

Quoted 6 lines
Quoted 6 lines

Noaya, do you have opinions on where would be the best place to store this information? The hugetlb memory error code has the comment 'needs work'. Ideally, we could isolate memory errors to a single base (4K for x86) page and free the remaining base pages to buddy. We could also potentially allocate a 'replacement' hugetlb page doing something like alloc_and_dissolve_huge_page.

Yes, these two are the important goals of hugetlb support.

Quoted 3 lines
Quoted 3 lines

If we get an error on a hugetlb page and can isolate it to a base page and replace the huge page, is it still a huge page memory error?

That's not counted as a huge page error, but just a base page error.

Quoted 4 lines
Quoted 4 lines

IMO, we should work on isolating memory errors to a base page and replacing the huge page. Then, the existing count of base pages with memory errors would be sufficient?

Yes, I agree with Mike. Adding new fields dedicated to the (maybe temporary) counters in struct hstate seems to me an overkill.

Quoted 9 lines
Quoted 3 lines

This is something I would like to work, but I have higher priorities right now.

Yes, splitting the huge pages and isolating a base page is ideal. And we do this with dissolve_free_huge_page() when page_mapping() return NULL. I think there is a reason(but I do not get it) why we don't split huge pags in hugetlbfs_error_remove_page() or after. So I choose to add a new count.

Maybe the resource is the main reason of this incompleteness, I noticed this for years and continued to say "this is in my todo list", but still don't make it (really sorry about that...). Anyway, if you can (I hope) solve your problem with "/proc/kpageflag" approach, which is a recommended solution.

Thanks,
Naoya Horiguchi

author
#4 lore raw

Hi Naoya,

Quoted 10 lines
Quoted 5 lines

Yes, splitting the huge pages and isolating a base page is ideal. And we do this with dissolve_free_huge_page() when page_mapping() return NULL. I think there is a reason(but I do not get it) why we don't split huge pags in hugetlbfs_error_remove_page() or after. So I choose to add a new count.

Maybe the resource is the main reason of this incompleteness, I noticed this for years and continued to say "this is in my todo list", but still don't make it (really sorry about that...). Anyway, if you can (I hope) solve your problem with "/proc/kpageflag" approach, which is a recommended solution.

I do not understand the exact meaning of the "resource". I have tried to call dissolve_free_huge_page() after hugetlbfs_error_remove_page() and it worked. In my opinion, the error huge page has been truncated from the hugetlbfs. It cannot be accessed and allocated again. I think it is safe to split it.

I would appreciate it if you could point out what I overlooked. And I will try to solve it.

Thanks,
Bin Wang