Skip to contentSkip to conversation

x86: Fix kexec 5-level to 4-level paging transition

PATCH Applied Usama Arif 3 participants 7 messages Nov 3, 2025 – Nov 6, 2025

Subsystems (2)
lore
Share & tools
lore

Usama Arif

author
#0 lore raw

v3 cover x86: Fix kexec 5-level to 4-level paging transition

This series addresses critical bugs in the kexec path when transitioning from a kernel using 5-level page tables to one using 4-level page tables.

The root cause is improper handling of PGD entry value during the page level transition. Specifically PGD entry value is masked with PAGE_MASK instead of PTE_PFN_MASK, failing to account for high-order software bits like _PAGE_BIT_NOPTISHADOW (bit 58).

When bit 58 (_PAGE_BIT_NOPTISHADOW) is set in the source kernel, the target 4-level kernel doesn't recognize it and fails to mask it properly, leading to kexec failure.

Patch 1: Fixes the x86 boot compressed code path by replacing direct CR3 dereferencing with read_cr3_pa() and using PTE_PFN_MASK instead of PAGE_MASK.

Patch 2: Applies the same fix to the EFI stub code path. (Done in a separate patch as Fixes tag is different).

The patches are based on aaa9c3550b60d6259d6ea8b1175ade8d1242444e (next-20251022)

v2 -> v3:
- Use native_pgd_val instead of pgd_val to fix broken build with allmodconfig. I wanted to keep the code between pgtable_64.c and x86-5lvl.c consistent so changed it in both patches
(Borislav Petkov and Ard Biesheuvel)
- Commit message improvements (Dave Hansen)

v1 -> v2:
- Remove patch 3 to fix kexec for source kernel in 5-level to 4-level transition where the 4 level kernel doesnt have patch 1 and 2 (Dave Hansen)
- Add include for asm/pgtable.h to fix build for x86_64-allnoconfig (kernel test bot)
- Use native_read_cr3_pa and for both paths (Ard Biesheuvel)

Usama Arif (2):
x86/boot: Fix page table access in 5-level to 4-level paging
transition
efi/libstub: Fix page table access in 5-level to 4-level paging
transition

 arch/x86/boot/compressed/pgtable_64.c   | 11 +++++++----
 drivers/firmware/efi/libstub/x86-5lvl.c |  4 ++--
 2 files changed, 9 insertions(+), 6 deletions(-)
Signature · 1 line
2.47.3
author
#1 lore raw

PATCH v3 1/2 x86/boot: Fix page table access in 5-level to 4-level paging transition

When transitioning from 5-level to 4-level paging, the existing code incorrectly accesses page table entries by directly dereferencing CR3 and applying PAGE_MASK. This approach has several issues:

- __native_read_cr3() returns the raw CR3 register value, which on x86_64 includes not just the physical address but also flags. Bits above the physical address width of the system i.e. above __PHYSICAL_MASK_SHIFT) are also not masked.
- The PGD entry is masked by PAGE_SIZE which doesn't take into account the higher bits such as _PAGE_BIT_NOPTISHADOW.

Replace this with proper accessor functions:
- native_read_cr3_pa(): Uses CR3_ADDR_MASK to additionally mask metadata out of CR3 (like SME or LAM bits). All remaining bits are real address bits or reserved and must be 0.
- mask pgd value with PTE_PFN_MASK instead of PAGE_MASK, accounting for flags above bit 51 (_PAGE_BIT_NOPTISHADOW in particular). Bits below 51, but above the max physical address are reserved and must be 0.

Fixes e9d0e6330eb8 ("x86/boot/compressed/64: Prepare new top-level page table for trampoline")
---
 arch/x86/boot/compressed/pgtable_64.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
arch/x86/boot/compressed/pgtable_64.c +7 -4
index bdd26050dff77..0e89e197e1126 100644
@@ -3,6 +3,7 @@
 #include <asm/bootparam.h>
 #include <asm/bootparam_utils.h>
 #include <asm/e820/types.h>
+#include <asm/pgtable.h>
 #include <asm/processor.h>
 #include "../string.h"
 #include "efi.h"
@@ -168,9 +169,10 @@ asmlinkage void configure_5level_paging(struct boot_params *bp, void *pgtable)
 		 * For 4- to 5-level paging transition, set up current CR3 as
 		 * the first and the only entry in a new top-level page table.
 		 */
-		*trampoline_32bit = __native_read_cr3() | _PAGE_TABLE_NOENC;
+		*trampoline_32bit = native_read_cr3_pa() | _PAGE_TABLE_NOENC;
 	} else {
-		unsigned long src;
+		u64 *new_cr3;
+		pgd_t *pgdp;
 
 		/*
 		 * For 5- to 4-level paging transition, copy page table pointed
@@ -180,8 +182,9 @@ asmlinkage void configure_5level_paging(struct boot_params *bp, void *pgtable)
 		 * We cannot just point to the page table from trampoline as it
 		 * may be above 4G.
 		 */
-		src = *(unsigned long *)__native_read_cr3() & PAGE_MASK;
-		memcpy(trampoline_32bit, (void *)src, PAGE_SIZE);
+		pgdp = (pgd_t *)native_read_cr3_pa();
+		new_cr3 = (u64 *)(native_pgd_val(pgdp[0]) & PTE_PFN_MASK);
+		memcpy(trampoline_32bit, new_cr3, PAGE_SIZE);
 	}
 
 	toggle_la57(trampoline_32bit);
Signature · 1 line
2.47.3
author
#2 lore raw

PATCH v3 2/2 efi/libstub: Fix page table access in 5-level to 4-level paging transition

In reply to Usama Arif

When transitioning from 5-level to 4-level paging, the existing code incorrectly accesses page table entries by directly dereferencing CR3 and applying PAGE_MASK. This approach has several issues:

- __native_read_cr3() returns the raw CR3 register value, which on x86_64 includes not just the physical address but also flags Bits above the physical address width of the system (i.e. above __PHYSICAL_MASK_SHIFT) are also not masked.
- The pgd value is masked by PAGE_SIZE which doesn't take into account the higher bits such as _PAGE_BIT_NOPTISHADOW.

Replace this with proper accessor functions:
- native_read_cr3_pa(): Uses CR3_ADDR_MASK to additionally mask metadata out of CR3 (like SME or LAM bits). All remaining bits are real address bits or reserved and must be 0.
- mask pgd value with PTE_PFN_MASK instead of PAGE_MASK, accounting for flags above bit 51 (_PAGE_BIT_NOPTISHADOW in particular). Bits below 51, but above the max physical address are reserved and must be 0.

Fixes cb1c9e02b0c1 ("x86/efistub: Perform 4/5 level paging switch from the stub")
---
 drivers/firmware/efi/libstub/x86-5lvl.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/firmware/efi/libstub/x86-5lvl.c +2 -2
index f1c5fb45d5f7c..c00d0ae7ed5d5 100644
@@ -66,7 +66,7 @@ void efi_5level_switch(void)
 	bool have_la57 = native_read_cr4() & X86_CR4_LA57;
 	bool need_toggle = want_la57 ^ have_la57;
 	u64 *pgt = (void *)la57_toggle + PAGE_SIZE;
-	u64 *cr3 = (u64 *)__native_read_cr3();
+	pgd_t *cr3 = (pgd_t *)native_read_cr3_pa();
 	u64 *new_cr3;
 
 	if (!la57_toggle || !need_toggle)
@@ -82,7 +82,7 @@ void efi_5level_switch(void)
 		new_cr3[0] = (u64)cr3 | _PAGE_TABLE_NOENC;
 	} else {
 		/* take the new root table pointer from the current entry #0 */
-		new_cr3 = (u64 *)(cr3[0] & PAGE_MASK);
+		new_cr3 = (u64 *)(native_pgd_val(cr3[0]) & PTE_PFN_MASK);
 
 		/* copy the new root table if it is not 32-bit addressable */
 		if ((u64)new_cr3 > U32_MAX)
Signature · 1 line
2.47.3
merge bot Applied
#3 lore raw

[tip: x86/mm] efi/libstub: Fix page table access in 5-level to 4-level paging transition

The following commit has been merged into the x86/mm branch of tip:

The following commit has been merged into the x86/mm branch of tip:

Commit-ID:     84361123413efc84b06f3441c6c827b95d902732
Gitweb:        https://git.kernel.org/tip/84361123413efc84b06f3441c6c827b95d902732
Author:        Usama Arif <[email protected]>
AuthorDate:    Mon, 03 Nov 2025 14:09:23 
Committer:     Borislav Petkov (AMD) <[email protected]>
CommitterDate: Wed, 05 Nov 2025 17:31:32 +01:00

efi/libstub: Fix page table access in 5-level to 4-level paging transition

When transitioning from 5-level to 4-level paging, the existing code incorrectly accesses page table entries by directly dereferencing CR3 and applying PAGE_MASK. This approach has several issues:

- __native_read_cr3() returns the raw CR3 register value, which on x86_64 includes not just the physical address but also flags Bits above the physical address width of the system (i.e. above __PHYSICAL_MASK_SHIFT) are also not masked.

- The pgd value is masked by PAGE_SIZE which doesn't take into account the higher bits such as _PAGE_BIT_NOPTISHADOW.

Replace this with proper accessor functions:

- native_read_cr3_pa(): Uses CR3_ADDR_MASK to additionally mask metadata out of CR3 (like SME or LAM bits). All remaining bits are real address bits or reserved and must be 0.

- mask pgd value with PTE_PFN_MASK instead of PAGE_MASK, accounting for flags above bit 51 (_PAGE_BIT_NOPTISHADOW in particular). Bits below 51, but above the max physical address are reserved and must be 0.

---
 drivers/firmware/efi/libstub/x86-5lvl.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/firmware/efi/libstub/x86-5lvl.c +2 -2
index f1c5fb4..c00d0ae 100644
@@ -66,7 +66,7 @@ void efi_5level_switch(void)
 	bool have_la57 = native_read_cr4() & X86_CR4_LA57;
 	bool need_toggle = want_la57 ^ have_la57;
 	u64 *pgt = (void *)la57_toggle + PAGE_SIZE;
-	u64 *cr3 = (u64 *)__native_read_cr3();
+	pgd_t *cr3 = (pgd_t *)native_read_cr3_pa();
 	u64 *new_cr3;
 
 	if (!la57_toggle || !need_toggle)
@@ -82,7 +82,7 @@ void efi_5level_switch(void)
 		new_cr3[0] = (u64)cr3 | _PAGE_TABLE_NOENC;
 	} else {
 		/* take the new root table pointer from the current entry #0 */
-		new_cr3 = (u64 *)(cr3[0] & PAGE_MASK);
+		new_cr3 = (u64 *)(native_pgd_val(cr3[0]) & PTE_PFN_MASK);
 
 		/* copy the new root table if it is not 32-bit addressable */
 		if ((u64)new_cr3 > U32_MAX)
maintainer
#4 lore raw
In reply to Usama Arif
Quoted 6 lines · Usama Arif

v2 -> v3:
- Use native_pgd_val instead of pgd_val to fix broken build with allmodconfig. I wanted to keep the code between pgtable_64.c and x86-5lvl.c consistent so changed it in both patches
(Borislav Petkov and Ard Biesheuvel)
- Commit message improvements (Dave Hansen)

Did you run the build tests I suggested?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author
#5 lore raw
Quoted 10 lines · Borislav Petkov
Quoted 6 lines · Usama Arif

v2 -> v3:
- Use native_pgd_val instead of pgd_val to fix broken build with allmodconfig. I wanted to keep the code between pgtable_64.c and x86-5lvl.c consistent so changed it in both patches
(Borislav Petkov and Ard Biesheuvel)
- Commit message improvements (Dave Hansen)

Did you run the build tests I suggested?

Yes, I did the below build tests:

make LLVM=1 allnoconfig; make LLVM=1 bzImage
make LLVM=1 defconfig; make LLVM=1 bzImage
make LLVM=1 allmodconfig; make LLVM=1 bzImage
make LLVM=1 allyesconfig; make LLVM=1 bzImage

make LLVM=1 ARCH=i386 allnoconfig; make LLVM=1 ARCH=i386 bzImage make LLVM=1 ARCH=i386 defconfig; make LLVM=1 ARCH=i386 bzImage make LLVM=1 ARCH=i386 allmodconfig; make LLVM=1 ARCH=i386 bzImage make LLVM=1 ARCH=i386 allyesconfig; make LLVM=1 ARCH=i386 bzImage

The i386 ones had a failure in lib/math/test_mul_u64_u64_div_u64.c:156:9 for linux-next/master so I rebased my patches on v6.17 and tested and they all built successfully.

maintainer
#6 lore raw
Quoted 1 line · Usama Arif

Yes, I did the below build tests:

Thanks!

Quoted 9 lines

make LLVM=1 allnoconfig; make LLVM=1 bzImage
make LLVM=1 defconfig; make LLVM=1 bzImage
make LLVM=1 allmodconfig; make LLVM=1 bzImage
make LLVM=1 allyesconfig; make LLVM=1 bzImage

make LLVM=1 ARCH=i386 allnoconfig; make LLVM=1 ARCH=i386 bzImage make LLVM=1 ARCH=i386 defconfig; make LLVM=1 ARCH=i386 bzImage make LLVM=1 ARCH=i386 allmodconfig; make LLVM=1 ARCH=i386 bzImage make LLVM=1 ARCH=i386 allyesconfig; make LLVM=1 ARCH=i386 bzImage

Next time try gcc too pls. :-) That's the first compiler we ever supported.

Quoted 3 lines

The i386 ones had a failure in lib/math/test_mul_u64_u64_div_u64.c:156:9 for linux-next/master so I rebased my patches on v6.17 and tested and they all built successfully.

Yeah, that was pointless.

You can simply say that the 32-bit build fails because of an unrelated reason.

But backporting it to another kernel doesn't have any bearing on the code this is going to be applied ontop of so...

But not a problem, I'll do the rest of the testing here.

Thanks again.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette