Skip to contentSkip to conversation

mm: Fix a huge pud insertion race during faulting

RFC PATCH In review Thomas Hellström (VMware) 3 participants 5 messages Oct 8, 2019 – Oct 16, 2019

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 mm: Fix a huge pud insertion race during faulting

From: Thomas Hellstrom <[email protected]>

A huge pud page can theoretically be faulted in racing with pmd_alloc() in __handle_mm_fault(). That will lead to pmd_alloc() returning an invalid pmd pointer. Fix this by adding a pud_trans_unstable() function similar to pmd_trans_unstable() and check whether the pud is really stable before using the pmd pointer.

Race:
Thread 1: Thread 2: Comment
create_huge_pud() Fallback - not taken.
create_huge_pud() Taken.
pmd_alloc() Returns an invalid pointer.

Cc Matthew Wilcox <[email protected]>
Fixes a00cc7d9dd93 ("mm, x86: add support for PUD-sized transparent hugepages")

---
RFC: We include pud_devmap() as an unstable PUD flag. Is this correct?
Do the same for pmds?

---
 include/asm-generic/pgtable.h | 25 +++++++++++++++++++++++++
 mm/memory.c                   |  6 ++++++
 2 files changed, 31 insertions(+)
include/asm-generic/pgtable.h +25 -0
index 818691846c90..70c2058230ba 100644
@@ -912,6 +912,31 @@ static inline int pud_trans_huge(pud_t pud)
 }
 #endif
 
+/* See pmd_none_or_trans_huge_or_clear_bad for discussion. */
+static inline int pud_none_or_trans_huge_or_dev_or_clear_bad(pud_t *pud)
+{
+	pud_t pudval = READ_ONCE(*pud);
+
+	if (pud_none(pudval) || pud_trans_huge(pudval) || pud_devmap(pudval))
+		return 1;
+	if (unlikely(pud_bad(pudval))) {
+		pud_clear_bad(pud);
+		return 1;
+	}
+	return 0;
+}
+
+/* See pmd_trans_unstable for discussion. */
+static inline int pud_trans_unstable(pud_t *pud)
+{
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) &&			\
+	defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD)
+	return pud_none_or_trans_huge_or_dev_or_clear_bad(pud);
+#else
+	return 0;
+#endif
+}
+
 #ifndef pmd_read_atomic
 static inline pmd_t pmd_read_atomic(pmd_t *pmdp)
 {
mm/memory.c +6 -0
index b1ca51a079f2..43ff372f4f07 100644
@@ -3914,6 +3914,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,
 	vmf.pud = pud_alloc(mm, p4d, address);
 	if (!vmf.pud)
 		return VM_FAULT_OOM;
+retry_pud:
 	if (pud_none(*vmf.pud) && __transparent_hugepage_enabled(vma)) {
 		ret = create_huge_pud(&vmf);
 		if (!(ret & VM_FAULT_FALLBACK))
@@ -3940,6 +3941,11 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,
 	vmf.pmd = pmd_alloc(mm, vmf.pud, address);
 	if (!vmf.pmd)
 		return VM_FAULT_OOM;
+
+	/* Huge pud page fault raced with pmd_alloc? */
+	if (pud_trans_unstable(vmf.pud))
+		goto retry_pud;
+
 	if (pmd_none(*vmf.pmd) && __transparent_hugepage_enabled(vma)) {
 		ret = create_huge_pmd(&vmf);
 		if (!(ret & VM_FAULT_FALLBACK))
Signature · 1 line
2.20.1
#1 lore raw
Quoted 14 lines · Thomas Hellström (VMware)

From: Thomas Hellstrom <[email protected]>

A huge pud page can theoretically be faulted in racing with pmd_alloc() in __handle_mm_fault(). That will lead to pmd_alloc() returning an invalid pmd pointer. Fix this by adding a pud_trans_unstable() function similar to pmd_trans_unstable() and check whether the pud is really stable before using the pmd pointer.

Race:
Thread 1: Thread 2: Comment
create_huge_pud() Fallback - not taken.
create_huge_pud() Taken.
pmd_alloc() Returns an invalid pointer.

Quoted 6 lines
Cc Matthew Wilcox <[email protected]>
Fixes a00cc7d9dd93 ("mm, x86: add support for PUD-sized transparent hugepages")

---
RFC: We include pud_devmap() as an unstable PUD flag. Is this correct?
Do the same for pmds?

I *think* it is correct and we should do the same for PMD, but I may be wrong.

Dan, Matthew, could you comment on this?

Quoted 68 lines
---
 include/asm-generic/pgtable.h | 25 +++++++++++++++++++++++++
 mm/memory.c                   |  6 ++++++
 2 files changed, 31 insertions(+)
include/asm-generic/pgtable.h +25 -0
index 818691846c90..70c2058230ba 100644
@@ -912,6 +912,31 @@ static inline int pud_trans_huge(pud_t pud)
 }
 #endif
 
+/* See pmd_none_or_trans_huge_or_clear_bad for discussion. */
+static inline int pud_none_or_trans_huge_or_dev_or_clear_bad(pud_t *pud)
+{
+	pud_t pudval = READ_ONCE(*pud);
+
+	if (pud_none(pudval) || pud_trans_huge(pudval) || pud_devmap(pudval))
+		return 1;
+	if (unlikely(pud_bad(pudval))) {
+		pud_clear_bad(pud);
+		return 1;
+	}
+	return 0;
+}
+
+/* See pmd_trans_unstable for discussion. */
+static inline int pud_trans_unstable(pud_t *pud)
+{
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) &&			\
+	defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD)
+	return pud_none_or_trans_huge_or_dev_or_clear_bad(pud);
+#else
+	return 0;
+#endif
+}
+
 #ifndef pmd_read_atomic
 static inline pmd_t pmd_read_atomic(pmd_t *pmdp)
 {
mm/memory.c +6 -0
index b1ca51a079f2..43ff372f4f07 100644
@@ -3914,6 +3914,7 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,
 	vmf.pud = pud_alloc(mm, p4d, address);
 	if (!vmf.pud)
 		return VM_FAULT_OOM;
+retry_pud:
 	if (pud_none(*vmf.pud) && __transparent_hugepage_enabled(vma)) {
 		ret = create_huge_pud(&vmf);
 		if (!(ret & VM_FAULT_FALLBACK))
@@ -3940,6 +3941,11 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma,
 	vmf.pmd = pmd_alloc(mm, vmf.pud, address);
 	if (!vmf.pmd)
 		return VM_FAULT_OOM;
+
+	/* Huge pud page fault raced with pmd_alloc? */
+	if (pud_trans_unstable(vmf.pud))
+		goto retry_pud;
+
 	if (pmd_none(*vmf.pmd) && __transparent_hugepage_enabled(vma)) {
 		ret = create_huge_pmd(&vmf);
 		if (!(ret & VM_FAULT_FALLBACK))
Signature · 1 line
2.20.1
Signature · 1 line
 Kirill A. Shutemov
#2 lore raw
Quoted 23 lines · Kirill A. Shutemov
Quoted 14 lines · Thomas Hellström (VMware)

From: Thomas Hellstrom <[email protected]>

A huge pud page can theoretically be faulted in racing with pmd_alloc() in __handle_mm_fault(). That will lead to pmd_alloc() returning an invalid pmd pointer. Fix this by adding a pud_trans_unstable() function similar to pmd_trans_unstable() and check whether the pud is really stable before using the pmd pointer.

Race:
Thread 1: Thread 2: Comment
create_huge_pud() Fallback - not taken.
create_huge_pud() Taken.
pmd_alloc() Returns an invalid pointer.

Quoted 6 lines
Cc Matthew Wilcox <[email protected]>
Fixes a00cc7d9dd93 ("mm, x86: add support for PUD-sized transparent hugepages")

---
RFC: We include pud_devmap() as an unstable PUD flag. Is this correct?
Do the same for pmds?

Quoted 4 lines

I *think* it is correct and we should do the same for PMD, but I may be wrong.

Dan, Matthew, could you comment on this?

The _devmap() check in these paths near _trans_unstable() has always been about avoiding assumptions that the corresponding page might be page cache or anonymous which for dax it's neither and does not behave like a typical page.

#3 lore raw

Hi, Dan,

Quoted 22 lines · Dan Williams
Quoted 21 lines · Kirill A. Shutemov
Quoted 20 lines · Thomas Hellström (VMware)

From: Thomas Hellstrom <[email protected]>

A huge pud page can theoretically be faulted in racing with pmd_alloc() in __handle_mm_fault(). That will lead to pmd_alloc() returning an invalid pmd pointer. Fix this by adding a pud_trans_unstable() function similar to pmd_trans_unstable() and check whether the pud is really stable before using the pmd pointer.

Race:
Thread 1: Thread 2: Comment
create_huge_pud() Fallback - not taken.
create_huge_pud() Taken.
pmd_alloc() Returns an invalid pointer.

Cc Matthew Wilcox <[email protected]>
Fixes a00cc7d9dd93 ("mm, x86: add support for PUD-sized transparent hugepages")

---
RFC: We include pud_devmap() as an unstable PUD flag. Is this correct?
Do the same for pmds?

Quoted 8 lines
Quoted 4 lines

I *think* it is correct and we should do the same for PMD, but I may be wrong.

Dan, Matthew, could you comment on this?

The _devmap() check in these paths near _trans_unstable() has always been about avoiding assumptions that the corresponding page might be page cache or anonymous which for dax it's neither and does not behave like a typical page.

The concern here is that _trans_huge() returns false for _devmap() pages, which means that also _trans_unstable() returns false.

Still, I figure someone could zap the entry at any time using madvise(), so AFAICT the entry is indeed unstable, and it's a bug not to include _devmap() in the _trans_unstable() functions?

Thanks,

Thomas

#4 lore raw
Quoted 35 lines · Thomas Hellström (VMware)

Hi, Dan,

Quoted 22 lines · Dan Williams
Quoted 21 lines · Kirill A. Shutemov
Quoted 20 lines · Thomas Hellström (VMware)

From: Thomas Hellstrom <[email protected]>

A huge pud page can theoretically be faulted in racing with pmd_alloc() in __handle_mm_fault(). That will lead to pmd_alloc() returning an invalid pmd pointer. Fix this by adding a pud_trans_unstable() function similar to pmd_trans_unstable() and check whether the pud is really stable before using the pmd pointer.

Race:
Thread 1: Thread 2: Comment
create_huge_pud() Fallback - not taken.
create_huge_pud() Taken.
pmd_alloc() Returns an invalid pointer.

Cc Matthew Wilcox <[email protected]>
Fixes a00cc7d9dd93 ("mm, x86: add support for PUD-sized transparent hugepages")

---
RFC: We include pud_devmap() as an unstable PUD flag. Is this correct?
Do the same for pmds?

Quoted 8 lines
Quoted 4 lines

I *think* it is correct and we should do the same for PMD, but I may be wrong.

Dan, Matthew, could you comment on this?

The _devmap() check in these paths near _trans_unstable() has always been about avoiding assumptions that the corresponding page might be page cache or anonymous which for dax it's neither and does not behave like a typical page.

Quoted 6 lines

The concern here is that _trans_huge() returns false for _devmap() pages, which means that also _trans_unstable() returns false.

Still, I figure someone could zap the entry at any time using madvise(), so AFAICT the entry is indeed unstable, and it's a bug not to include _devmap() in the _trans_unstable() functions?

Yes, I can't think a case where it is wrong to include _devmap() in a _trans_unstable(). It may be unnecessary if the given path can't reasonably ever encounter a file-backed dax mapping, but it's otherwise ok to always consider _devmap().