Skip to contentSkip to conversation

process accidentally killed by mce because of huge page migration

3.10.x-stable Xishi Qiu 2 participants 2 messages Feb 13, 2014

Subsystems (2)
Memory management HugeTLB
lore
Share & tools
lore

Xishi Qiu

author
#0 lore raw

Hi Naoya or Greg,

We found a bug in 3.10.x.
1) use sysfs interface soft_offline_page() to migrate a huge page.
2) the hpage become free after migrate_huge_page().
3) the free hpage is alloced and used by process A.
4) hwpoison flag is set by set_page_hwpoison_huge_page()
5) mce find this poisoned page.
6) process A was killed.
7) other processes which use this page will be killed too.

I tested this bug, one process keeps allocating huge page, and I use sysfs interface to soft offline a huge page, then received:
"MCE: Killing UCP:2717 due to hardware memory corruption fault at 8200034"

Upstream kernel is free from this bug because of these two commits:

f15bdfa802bfa5eb6b4b5a241b97ec9fa1204a35 mm/memory-failure.c: fix memory leak in successful soft offlining

c8721bbbdd36382de51cd6b7a56322e0acca2414
mm: memory-hotplug: enable memory hotplug to handle hugepage

The latter is not a bug fix and it's too big, the following patch can fix this bug.

What do you think? Use the simple fix or backport the big patch?

Thanks,
Xishi Qiu

Signed-off-by Xishi Qiu <[email protected]>
---
 mm/hugetlb.c |   11 +++++++++--
 1 files changed, 9 insertions(+), 2 deletions(-)
mm/hugetlb.c +9 -2
index 7c5eb85..6cb5b3b 100644
@@ -21,6 +21,7 @@
 #include <linux/rmap.h>
 #include <linux/swap.h>
 #include <linux/swapops.h>
+#include <linux/page-isolation.h>
 
 #include <asm/page.h>
 #include <asm/pgtable.h>

@@ -517,9 +518,15 @@ static struct page *dequeue_huge_page_node(struct hstate *h, int nid)
 {
 	struct page *page;
 
-	if (list_empty(&h->hugepage_freelists[nid]))
+	list_for_each_entry(page, &h->hugepage_freelists[nid], lru)
+		if (!is_migrate_isolate_page(page))
+			break;
+	/*
+	 * if 'non-isolated free hugepage' not found on the list,
+	 * the allocation fails.
+	 */
+	if (&h->hugepage_freelists[nid] == &page->lru)
 		return NULL;
-	page = list_entry(h->hugepage_freelists[nid].next, struct page, lru);
 	list_move(&page->lru, &h->hugepage_activelist);
 	set_page_refcounted(page);
 	h->free_huge_pages--;
Signature · 7 lines
1.7.1 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Reviewed-by
#1 lore raw

Hi Xishi,

Quoted 11 lines · Xishi Qiu

Hi Naoya or Greg,

We found a bug in 3.10.x.
1) use sysfs interface soft_offline_page() to migrate a huge page.
2) the hpage become free after migrate_huge_page().
3) the free hpage is alloced and used by process A.
4) hwpoison flag is set by set_page_hwpoison_huge_page()
5) mce find this poisoned page.
6) process A was killed.
7) other processes which use this page will be killed too.

Quoted 8 lines

I tested this bug, one process keeps allocating huge page, and I use sysfs interface to soft offline a huge page, then received:
"MCE: Killing UCP:2717 due to hardware memory corruption fault at 8200034"

Upstream kernel is free from this bug because of these two commits:

f15bdfa802bfa5eb6b4b5a241b97ec9fa1204a35 mm/memory-failure.c: fix memory leak in successful soft offlining

Correct. Although this problem is not about memory leak, this patch moves unset_migratetype_isolate(), which is important to avoid the race.

Quoted 2 lines

c8721bbbdd36382de51cd6b7a56322e0acca2414
mm: memory-hotplug: enable memory hotplug to handle hugepage

The problem is that we accidentally have a hwpoisoned hugepage in free hugepage list. It could happend in the the following scenario:

process A process B

migrate_huge_page
put_page (old hugepage)
linked to free hugepage list
hugetlb_fault
hugetlb_no_page
alloc_huge_page
dequeue_huge_page_vma
dequeue_huge_page_node
(steal hwpoisoned hugepage)
set_page_hwpoison_huge_page
dequeue_hwpoisoned_huge_page
(fail to dequeue)

In upstream, we avoid the race by making dequeue_huge_page_node refuse to use hugepages with MIGRATE_ISOLATE set. But this depends on the fact that we set MIGRATE_ISOLATE on hugepage under migration. This is not the case in current stable-3.10.

Quoted 4 lines

The latter is not a bug fix and it's too big, the following patch can fix this bug.

What do you think? Use the simple fix or backport the big patch?

I definitely agree to your suggestion. Please feel free to add the above explanation in the patch description with

Reviewed-by Naoya Horiguchi <[email protected]>

Thanks,
Naoya

Quoted 42 lines

Thanks,
Xishi Qiu

Signed-off-by Xishi Qiu <[email protected]>
---
 mm/hugetlb.c |   11 +++++++++--
 1 files changed, 9 insertions(+), 2 deletions(-)
mm/hugetlb.c +9 -2
index 7c5eb85..6cb5b3b 100644
@@ -21,6 +21,7 @@
 #include <linux/rmap.h>
 #include <linux/swap.h>
 #include <linux/swapops.h>
+#include <linux/page-isolation.h>
 
 #include <asm/page.h>
 #include <asm/pgtable.h>

@@ -517,9 +518,15 @@ static struct page *dequeue_huge_page_node(struct hstate *h, int nid)
 {
 	struct page *page;
 
-	if (list_empty(&h->hugepage_freelists[nid]))
+	list_for_each_entry(page, &h->hugepage_freelists[nid], lru)
+		if (!is_migrate_isolate_page(page))
+			break;
+	/*
+	 * if 'non-isolated free hugepage' not found on the list,
+	 * the allocation fails.
+	 */
+	if (&h->hugepage_freelists[nid] == &page->lru)
 		return NULL;
-	page = list_entry(h->hugepage_freelists[nid].next, struct page, lru);
 	list_move(&page->lru, &h->hugepage_activelist);
 	set_page_refcounted(page);
 	h->free_huge_pages--;
Signature · 1 line
1.7.1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>