Skip to contentSkip to conversation

x86/mm/64: Do not dereference non-present PGD entries

PATCH In review Joerg Roedel 5 participants 6 messages Aug 7, 2020 – Aug 13, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/mm/64: Do not dereference non-present PGD entries

From: Joerg Roedel <[email protected]>

The code for preallocate_vmalloc_pages() was written under the assumption that the p4d_offset() and pud_offset() functions will perform present checks before dereferencing the parent entries.

This assumption is wrong an leads to a bug in the code which causes the physical address found in the PGD be used as a page-table page, even if the PGD is not present.

So the code flow currently is:

	pgd = pgd_offset_k(addr);
	p4d = p4d_offset(pgd, addr);
	if (p4d_none(*p4d))
		p4d = p4d_alloc(&init_mm, pgd, addr);

This lacks a check for pgd_none() at least, the correct flow would be:

	pgd = pgd_offset_k(addr);
	if (pgd_none(*pgd))
		p4d = p4d_alloc(&init_mm, pgd, addr);
	else
		p4d = p4d_offset(pgd, addr);

But this is the same flow that the p4d_alloc() and the pud_alloc() functions use internally, so there is no need to duplicate them.

Remove the p?d_none() checks from the function and just call into p4d_alloc() and pud_alloc() to correctly pre-allocate the PGD entries.

Fixes 6eb82f994026 ("x86/mm: Pre-allocate P4D/PUD pages for vmalloc area")
---
 arch/x86/mm/init_64.c | 31 +++++++++++++------------------
 1 file changed, 13 insertions(+), 18 deletions(-)
arch/x86/mm/init_64.c +13 -18
index 3f4e29a78f2b..449e071240e1 100644
@@ -1253,28 +1253,23 @@ static void __init preallocate_vmalloc_pages(void)
 		p4d_t *p4d;
 		pud_t *pud;
 
-		p4d = p4d_offset(pgd, addr);
-		if (p4d_none(*p4d)) {
-			/* Can only happen with 5-level paging */
-			p4d = p4d_alloc(&init_mm, pgd, addr);
-			if (!p4d) {
-				lvl = "p4d";
-				goto failed;
-			}
-		}
+		lvl = "p4d";
+		p4d = p4d_alloc(&init_mm, pgd, addr);
+		if (!p4d)
+			goto failed;
 
+		/*
+		 * With 5-level paging the P4D level is not folded. So the PGDs
+		 * are now populated and there is no need to walk down to the
+		 * PUD level.
+		 */
 		if (pgtable_l5_enabled())
 			continue;
 
-		pud = pud_offset(p4d, addr);
-		if (pud_none(*pud)) {
-			/* Ends up here only with 4-level paging */
-			pud = pud_alloc(&init_mm, p4d, addr);
-			if (!pud) {
-				lvl = "pud";
-				goto failed;
-			}
-		}
+		lvl = "pud";
+		pud = pud_alloc(&init_mm, p4d, addr);
+		if (!pud)
+			goto failed;
 	}
 
 	return;
Signature · 1 line
2.26.2
Tested-by on v1 1/1
#1 lore raw
Quoted 86 lines · Joerg Roedel

From: Joerg Roedel <[email protected]>

The code for preallocate_vmalloc_pages() was written under the assumption that the p4d_offset() and pud_offset() functions will perform present checks before dereferencing the parent entries.

This assumption is wrong an leads to a bug in the code which causes the physical address found in the PGD be used as a page-table page, even if the PGD is not present.

So the code flow currently is:

        pgd = pgd_offset_k(addr);
        p4d = p4d_offset(pgd, addr);
        if (p4d_none(*p4d))
                p4d = p4d_alloc(&init_mm, pgd, addr);

This lacks a check for pgd_none() at least, the correct flow would be:

        pgd = pgd_offset_k(addr);
        if (pgd_none(*pgd))
                p4d = p4d_alloc(&init_mm, pgd, addr);
        else
                p4d = p4d_offset(pgd, addr);

But this is the same flow that the p4d_alloc() and the pud_alloc() functions use internally, so there is no need to duplicate them.

Remove the p?d_none() checks from the function and just call into p4d_alloc() and pud_alloc() to correctly pre-allocate the PGD entries.

Fixes 6eb82f994026 ("x86/mm: Pre-allocate P4D/PUD pages for vmalloc area")
---
 arch/x86/mm/init_64.c | 31 +++++++++++++------------------
 1 file changed, 13 insertions(+), 18 deletions(-)
arch/x86/mm/init_64.c +13 -19
index 3f4e29a78f2b..449e071240e1 100644
@@ -1253,28 +1253,23 @@ static void __init preallocate_vmalloc_pages(void)
                p4d_t *p4d;
                pud_t *pud;
 
-               p4d = p4d_offset(pgd, addr);
-               if (p4d_none(*p4d)) {
-                       /* Can only happen with 5-level paging */
-                       p4d = p4d_alloc(&init_mm, pgd, addr);
-                       if (!p4d) {
-                               lvl = "p4d";
-                               goto failed;
-                       }
-               }
+               lvl = "p4d";
+               p4d = p4d_alloc(&init_mm, pgd, addr);
+               if (!p4d)
+                       goto failed;
 
+               /*
+                * With 5-level paging the P4D level is not folded. So the PGDs
+                * are now populated and there is no need to walk down to the
+                * PUD level.
+                */
                if (pgtable_l5_enabled())
                        continue;
 
-               pud = pud_offset(p4d, addr);
-               if (pud_none(*pud)) {
-                       /* Ends up here only with 4-level paging */
-                       pud = pud_alloc(&init_mm, p4d, addr);
-                       if (!pud) {
-                               lvl = "pud";
-                               goto failed;
-                       }
-               }
+               lvl = "pud";
+               pud = pud_alloc(&init_mm, p4d, addr);
+               if (!pud)
+                       goto failed;
        }
 
        return;
--
Quoted 1 line

2.26.2

This appears to fix the issue, so:

Reviewed-by on v1 1/1
#2 lore raw
In reply to Joerg Roedel
Quoted 28 lines · Joerg Roedel

From: Joerg Roedel <[email protected]>

The code for preallocate_vmalloc_pages() was written under the assumption that the p4d_offset() and pud_offset() functions will perform present checks before dereferencing the parent entries.

This assumption is wrong an leads to a bug in the code which causes the physical address found in the PGD be used as a page-table page, even if the PGD is not present.

So the code flow currently is:

	pgd = pgd_offset_k(addr);
	p4d = p4d_offset(pgd, addr);
	if (p4d_none(*p4d))
		p4d = p4d_alloc(&init_mm, pgd, addr);

This lacks a check for pgd_none() at least, the correct flow would be:

	pgd = pgd_offset_k(addr);
	if (pgd_none(*pgd))
		p4d = p4d_alloc(&init_mm, pgd, addr);
	else
		p4d = p4d_offset(pgd, addr);

But this is the same flow that the p4d_alloc() and the pud_alloc() functions use internally, so there is no need to duplicate them.

Quoted 6 lines

Remove the p?d_none() checks from the function and just call into p4d_alloc() and pud_alloc() to correctly pre-allocate the PGD entries.

Fixes 6eb82f994026 ("x86/mm: Pre-allocate P4D/PUD pages for vmalloc area")

LGTM,

Quoted 53 lines
---
 arch/x86/mm/init_64.c | 31 +++++++++++++------------------
 1 file changed, 13 insertions(+), 18 deletions(-)
arch/x86/mm/init_64.c +13 -18
index 3f4e29a78f2b..449e071240e1 100644
@@ -1253,28 +1253,23 @@ static void __init preallocate_vmalloc_pages(void)
 		p4d_t *p4d;
 		pud_t *pud;
 
-		p4d = p4d_offset(pgd, addr);
-		if (p4d_none(*p4d)) {
-			/* Can only happen with 5-level paging */
-			p4d = p4d_alloc(&init_mm, pgd, addr);
-			if (!p4d) {
-				lvl = "p4d";
-				goto failed;
-			}
-		}
+		lvl = "p4d";
+		p4d = p4d_alloc(&init_mm, pgd, addr);
+		if (!p4d)
+			goto failed;
 
+		/*
+		 * With 5-level paging the P4D level is not folded. So the PGDs
+		 * are now populated and there is no need to walk down to the
+		 * PUD level.
+		 */
 		if (pgtable_l5_enabled())
 			continue;
 
-		pud = pud_offset(p4d, addr);
-		if (pud_none(*pud)) {
-			/* Ends up here only with 4-level paging */
-			pud = pud_alloc(&init_mm, p4d, addr);
-			if (!pud) {
-				lvl = "pud";
-				goto failed;
-			}
-		}
+		lvl = "pud";
+		pud = pud_alloc(&init_mm, p4d, addr);
+		if (!pud)
+			goto failed;
 	}
 
 	return;
Signature · 1 line
2.26.2
Signature · 2 lines
Sincerely yours,
Mike.
#3 lore raw
In reply to Joerg Roedel

... adding Kirill

arch/x86/mm/init_64.c static void __init preallocate_vmalloc_pages(void) · line 1267

Quoted 12 lines · Joerg Roedel
arch/x86/mm/init_64.c +9 -0
+		lvl = "p4d";
+		p4d = p4d_alloc(&init_mm, pgd, addr);
+		if (!p4d)
+			goto failed;
 
+		/*
+		 * With 5-level paging the P4D level is not folded. So the PGDs
+		 * are now populated and there is no need to walk down to the
+		 * PUD level.
+		 */
 		if (pgtable_l5_enabled())
 			continue;

It's early and I'm a coffee or two short of awake, but I had to stare at the comment for a but to make sense of it.

It feels wrong, I think, because the 5-level code usually ends up doing *more* allocations and in this case, it is _appearing_ to do fewer. Would something like this make sense?

/*
* The goal here is to allocate all possibly required
* hardware page tables pointed to by the top hardware
* level.
*
* On 4-level systems, the p4d layer is folded away and
* the above code does no preallocation. Below, go down
* to the pud _software_ level to ensure the second
* hardware level is allocated.
*/

Quoted 14 lines
arch/x86/mm/init_64.c +4 -9
-		pud = pud_offset(p4d, addr);
-		if (pud_none(*pud)) {
-			/* Ends up here only with 4-level paging */
-			pud = pud_alloc(&init_mm, p4d, addr);
-			if (!pud) {
-				lvl = "pud";
-				goto failed;
-			}
-		}
+		lvl = "pud";
+		pud = pud_alloc(&init_mm, p4d, addr);
+		if (!pud)
+			goto failed;
 	}
#4 lore raw
Quoted 19 lines · Dave Hansen

... adding Kirill

arch/x86/mm/init_64.c static void __init preallocate_vmalloc_pages(void) · line 1267

Quoted 12 lines · Joerg Roedel
arch/x86/mm/init_64.c +9 -0
+		lvl = "p4d";
+		p4d = p4d_alloc(&init_mm, pgd, addr);
+		if (!p4d)
+			goto failed;
 
+		/*
+		 * With 5-level paging the P4D level is not folded. So the PGDs
+		 * are now populated and there is no need to walk down to the
+		 * PUD level.
+		 */
 		if (pgtable_l5_enabled())
 			continue;

It's early and I'm a coffee or two short of awake, but I had to stare at the comment for a but to make sense of it.

Quoted 3 lines

It feels wrong, I think, because the 5-level code usually ends up doing *more* allocations and in this case, it is _appearing_ to do fewer. Would something like this make sense?

Unless I miss something, with 5 levels vmalloc mappings are shared at p4d level, so allocating a p4d page would be enough. With 4 levels, p4d_alloc() is a nop and pud is the first actually populated level below pgd.

Quoted 26 lines

/*
* The goal here is to allocate all possibly required
* hardware page tables pointed to by the top hardware
* level.
*
* On 4-level systems, the p4d layer is folded away and
* the above code does no preallocation. Below, go down
* to the pud _software_ level to ensure the second
* hardware level is allocated.
*/

Quoted 14 lines
arch/x86/mm/init_64.c +4 -9
-		pud = pud_offset(p4d, addr);
-		if (pud_none(*pud)) {
-			/* Ends up here only with 4-level paging */
-			pud = pud_alloc(&init_mm, p4d, addr);
-			if (!pud) {
-				lvl = "pud";
-				goto failed;
-			}
-		}
+		lvl = "pud";
+		pud = pud_alloc(&init_mm, p4d, addr);
+		if (!pud)
+			goto failed;
 	}
Signature · 2 lines
Sincerely yours,
Mike.
#5 lore raw
Quoted 20 lines · Mike Rapoport
Quoted 19 lines · Dave Hansen

... adding Kirill

arch/x86/mm/init_64.c static void __init preallocate_vmalloc_pages(void) · line 1267

Quoted 12 lines · Joerg Roedel
arch/x86/mm/init_64.c +9 -0
+		lvl = "p4d";
+		p4d = p4d_alloc(&init_mm, pgd, addr);
+		if (!p4d)
+			goto failed;
 
+		/*
+		 * With 5-level paging the P4D level is not folded. So the PGDs
+		 * are now populated and there is no need to walk down to the
+		 * PUD level.
+		 */
 		if (pgtable_l5_enabled())
 			continue;

It's early and I'm a coffee or two short of awake, but I had to stare at the comment for a but to make sense of it.

Quoted 19 lines
Quoted 3 lines

It feels wrong, I think, because the 5-level code usually ends up doing *more* allocations and in this case, it is _appearing_ to do fewer. Would something like this make sense?

Unless I miss something, with 5 levels vmalloc mappings are shared at p4d level, so allocating a p4d page would be enough. With 4 levels, p4d_alloc() is a nop and pud is the first actually populated level below pgd.

Quoted 10 lines

/*
* The goal here is to allocate all possibly required
* hardware page tables pointed to by the top hardware
* level.
*
* On 4-level systems, the p4d layer is folded away and
* the above code does no preallocation. Below, go down
* to the pud _software_ level to ensure the second
* hardware level is allocated.
*/

Would be nice to integrate all these explanations into the comment itself?

Thanks,

Ingo