Skip to contentSkip to conversation

x86: Fix kexec 5-level to 4-level paging transition

PATCH Superseded Usama Arif 5 participants 11 messages Oct 28, 2025 – Oct 31, 2025

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Usama Arif

author
#0 lore raw

v2 cover x86: Fix kexec 5-level to 4-level paging transition

This series addresses critical bugs in the kexec path when transitioning from a kernel using 5-level page tables to one using 4-level page tables.

The root cause is improper handling of PGD entry value during the page level transition. Specifically PGD entry value is masked with PAGE_MASK instead of PTE_PFN_MASK, failing to account for high-order software bits like _PAGE_BIT_NOPTISHADOW (bit 58).

When bit 58 (_PAGE_BIT_NOPTISHADOW) is set in the source kernel, the target 4-level kernel doesn't recognize it and fails to mask it properly, leading to kexec failure.

Patch 1: Fixes the x86 boot compressed code path by replacing direct CR3 dereferencing with read_cr3_pa() and using PTE_PFN_MASK instead of PAGE_MASK.

Patch 2: Applies the same fix to the EFI stub code path. (Done in a separate patch as Fixes tag is different).

The patches are based on aaa9c3550b60d6259d6ea8b1175ade8d1242444e (next-20251022)

v1 -> v2:
- Remove patch 3 from v1 to fix kexec for source kernel in 5-level to 4-level transition where the 4 level kernel doesnt have patch 1 and 2 (Dave Hansen)
- Add include for asm/pgtable.h to fix build for x86_64-allnoconfig (kernel test bot)
- Use native_read_cr3_pa and for both paths (Ard Biesheuvel)

Usama Arif (2):
x86/boot: Fix page table access in 5-level to 4-level paging
transition
efi/libstub: Fix page table access in 5-level to 4-level paging
transition

 arch/x86/boot/compressed/pgtable_64.c   | 11 +++++++----
 drivers/firmware/efi/libstub/x86-5lvl.c |  4 ++--
 2 files changed, 9 insertions(+), 6 deletions(-)
Signature · 1 line
2.47.3
author
#1 lore raw

PATCH v2 1/2 x86/boot: Fix page table access in 5-level to 4-level paging transition

When transitioning from 5-level to 4-level paging, the existing code incorrectly accesses page table entries by directly dereferencing CR3 and applying PAGE_MASK. This approach has several issues:

- __native_read_cr3() returns the raw CR3 register value, which on x86_64 includes not just the physical address but also flags. Bits above the physical address width of the system i.e. above __PHYSICAL_MASK_SHIFT) are also not masked.
- The PGD entry is masked by PAGE_SIZE which doesn't take into account the higher bits such as _PAGE_BIT_NOPTISHADOW.

Replace this with proper accessor functions:
- native_read_cr3_pa(): Uses CR3_ADDR_MASK properly clearing SME encryption bit and extracting only the physical address portion.
- mask pgd value with PTE_PFN_MASK instead of PAGE_MASK, accounting for flags above physical address (_PAGE_BIT_NOPTISHADOW in particular).

Fixes e9d0e6330eb8 ("x86/boot/compressed/64: Prepare new top-level page table for trampoline")
---
 arch/x86/boot/compressed/pgtable_64.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
arch/x86/boot/compressed/pgtable_64.c +7 -4
index bdd26050dff77..f812b81a538c2 100644
@@ -3,6 +3,7 @@
 #include <asm/bootparam.h>
 #include <asm/bootparam_utils.h>
 #include <asm/e820/types.h>
+#include <asm/pgtable.h>
 #include <asm/processor.h>
 #include "../string.h"
 #include "efi.h"
@@ -168,9 +169,10 @@ asmlinkage void configure_5level_paging(struct boot_params *bp, void *pgtable)
 		 * For 4- to 5-level paging transition, set up current CR3 as
 		 * the first and the only entry in a new top-level page table.
 		 */
-		*trampoline_32bit = __native_read_cr3() | _PAGE_TABLE_NOENC;
+		*trampoline_32bit = native_read_cr3_pa() | _PAGE_TABLE_NOENC;
 	} else {
-		unsigned long src;
+		u64 *new_cr3;
+		pgd_t *pgdp;
 
 		/*
 		 * For 5- to 4-level paging transition, copy page table pointed
@@ -180,8 +182,9 @@ asmlinkage void configure_5level_paging(struct boot_params *bp, void *pgtable)
 		 * We cannot just point to the page table from trampoline as it
 		 * may be above 4G.
 		 */
-		src = *(unsigned long *)__native_read_cr3() & PAGE_MASK;
-		memcpy(trampoline_32bit, (void *)src, PAGE_SIZE);
+		pgdp = (pgd_t *)native_read_cr3_pa();
+		new_cr3 = (u64 *)(pgd_val(pgdp[0]) & PTE_PFN_MASK);
+		memcpy(trampoline_32bit, new_cr3, PAGE_SIZE);
 	}
 
 	toggle_la57(trampoline_32bit);
Signature · 1 line
2.47.3
Acked-by on v2 1/2
#2 lore raw

Re: [PATCH v2 1/2] x86/boot: Fix page table access in 5-level to 4-level paging transition

Quoted 2 lines · Usama Arif

- native_read_cr3_pa(): Uses CR3_ADDR_MASK properly clearing SME encryption bit and extracting only the physical address portion.

I guess we can apply these as-is. They do fix a bug.

But I find these descriptions a bit unsatisfying. CR3_ADDR_MASK happens to work here on 64-bit. Interestingly enough, it wouldn't have been as good of a fix on PAE paging because it ignores those upper bits instead of reserving them.

But CR3_ADDR_MASK doesn't "extract... only the physical address portion". It also extracts reserved bits.

It also doesn't mention the LAM bits. It's not just SME.

This would be better:

- native_read_cr3_pa(): Uses CR3_ADDR_MASK to additionally mask metadata out of CR3 (like SME or LAM bits). All remaining bits are real address bits or reserved and must be 0.

Quoted 2 lines

- mask pgd value with PTE_PFN_MASK instead of PAGE_MASK, accounting for flags above physical address (_PAGE_BIT_NOPTISHADOW in particular).

This also isn't _quite_ right. The "flags above physical" address are dynamic. They move because the max physical address (MAXPHYADDR) is enumerated and changes from CPU to CPU.

It's OK in this case because moving MAXPHYADDR down just changes bits from address bits to reserved (must be 0).

In a perfect world, we would construct a kexec CR3 with the dynamic MAXPHYADDR (plus masking out the lower 12 bits). That would be obviously correct for *all* 32-bit and 64-bit cases and wouldn't even rely on knowing where the boundary is between ignored and reserved. The approach in these patches is a fine improvement We don't need to be perfect.

Ideally this second bullet would be:

- mask pgd value with PTE_PFN_MASK instead of PAGE_MASK, accounting for flags above bit 51 (_PAGE_BIT_NOPTISHADOW in particular). Bits below 51, but above the max physical address are reserved and must be 0.

But it's fine-ish as-is.

author
#3 lore raw

PATCH v2 2/2 efi/libstub: Fix page table access in 5-level to 4-level paging transition

In reply to Usama Arif

When transitioning from 5-level to 4-level paging, the existing code incorrectly accesses page table entries by directly dereferencing CR3 and applying PAGE_MASK. This approach has several issues:

- __native_read_cr3() returns the raw CR3 register value, which on x86_64 includes not just the physical address but also flags Bits above the physical address width of the system (i.e. above __PHYSICAL_MASK_SHIFT) are also not masked.
- The pgd value is masked by PAGE_SIZE which doesn't take into account the higher bits such as _PAGE_BIT_NOPTISHADOW.

Replace this with proper accessor functions:
- native_read_cr3_pa(): Uses CR3_ADDR_MASK properly clearing SME encryption bit and extracting only the physical address portion.
- mask pgd value with PTE_PFN_MASK instead of PAGE_MASK, accounting for flags above physical address (_PAGE_BIT_NOPTISHADOW in particular).

Fixes cb1c9e02b0c1 ("x86/efistub: Perform 4/5 level paging switch from the stub")
---
 drivers/firmware/efi/libstub/x86-5lvl.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/firmware/efi/libstub/x86-5lvl.c +2 -2
index f1c5fb45d5f7c..36b4a611f6adf 100644
@@ -66,7 +66,7 @@ void efi_5level_switch(void)
 	bool have_la57 = native_read_cr4() & X86_CR4_LA57;
 	bool need_toggle = want_la57 ^ have_la57;
 	u64 *pgt = (void *)la57_toggle + PAGE_SIZE;
-	u64 *cr3 = (u64 *)__native_read_cr3();
+	pgd_t *cr3 = (pgd_t *)native_read_cr3_pa();
 	u64 *new_cr3;
 
 	if (!la57_toggle || !need_toggle)
@@ -82,7 +82,7 @@ void efi_5level_switch(void)
 		new_cr3[0] = (u64)cr3 | _PAGE_TABLE_NOENC;
 	} else {
 		/* take the new root table pointer from the current entry #0 */
-		new_cr3 = (u64 *)(cr3[0] & PAGE_MASK);
+		new_cr3 = (u64 *)(pgd_val(cr3[0]) & PTE_PFN_MASK);
 
 		/* copy the new root table if it is not 32-bit addressable */
 		if ((u64)new_cr3 > U32_MAX)
Signature · 1 line
2.47.3
maintainer
#4 lore raw

Re: [PATCH v2 2/2] efi/libstub: Fix page table access in 5-level to 4-level paging transition

Quoted 23 lines · Usama Arif

When transitioning from 5-level to 4-level paging, the existing code incorrectly accesses page table entries by directly dereferencing CR3 and applying PAGE_MASK. This approach has several issues:

- __native_read_cr3() returns the raw CR3 register value, which on x86_64 includes not just the physical address but also flags Bits above the physical address width of the system (i.e. above __PHYSICAL_MASK_SHIFT) are also not masked.
- The pgd value is masked by PAGE_SIZE which doesn't take into account the higher bits such as _PAGE_BIT_NOPTISHADOW.

Replace this with proper accessor functions:
- native_read_cr3_pa(): Uses CR3_ADDR_MASK properly clearing SME encryption bit and extracting only the physical address portion.
- mask pgd value with PTE_PFN_MASK instead of PAGE_MASK, accounting for flags above physical address (_PAGE_BIT_NOPTISHADOW in particular).

Fixes cb1c9e02b0c1 ("x86/efistub: Perform 4/5 level paging switch from the stub")
Quoted 3 lines
---
 drivers/firmware/efi/libstub/x86-5lvl.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

allmodconfig build:

ld: error: unplaced orphan section __bug_table' from drivers/firmware/efi/libstub/x86-5lvl.stub.o'
ld: error: unplaced orphan section .altinstructions' from drivers/firmware/efi/libstub/x86-5lvl.stub.o'
ld: error: unplaced orphan section .altinstr_replacement' from drivers/firmware/efi/libstub/x86-5lvl.stub.o'
ld: Unexpected GOT/PLT entries detected!
ld: Unexpected run-time procedure linkages detected!
ld: Unexpected run-time relocations (.rela) detected!
ld: drivers/firmware/efi/libstub/x86-5lvl.stub.o: in function `efi_5level_switch':
x86-5lvl.c:(.text+0x13e): undefined reference to `pv_ops'
ld: x86-5lvl.c:(.text+0x14d): undefined reference to `pv_ops'
ld: drivers/firmware/efi/libstub/x86-5lvl.stub.o:(.altinstr_replacement+0x1): undefined reference to `BUG_func'
ld: arch/x86/boot/compressed/vmlinux: hidden symbol `pv_ops' isn't defined
ld: final link failed: bad value
make[3]: *** [arch/x86/boot/compressed/Makefile:116: arch/x86/boot/compressed/vmlinux] Error 1 make[2]: *** [arch/x86/boot/Makefile:96: arch/x86/boot/compressed/vmlinux] Error 2
make[1]: *** [arch/x86/Makefile:308: bzImage] Error 2
make: *** [Makefile:248: __sub-make] Error 2

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
maintainer
#5 lore raw

Re: [PATCH v2 2/2] efi/libstub: Fix page table access in 5-level to 4-level paging transition

Quoted 31 lines · Borislav Petkov
Quoted 23 lines · Usama Arif

When transitioning from 5-level to 4-level paging, the existing code incorrectly accesses page table entries by directly dereferencing CR3 and applying PAGE_MASK. This approach has several issues:

- __native_read_cr3() returns the raw CR3 register value, which on x86_64 includes not just the physical address but also flags Bits above the physical address width of the system (i.e. above __PHYSICAL_MASK_SHIFT) are also not masked.
- The pgd value is masked by PAGE_SIZE which doesn't take into account the higher bits such as _PAGE_BIT_NOPTISHADOW.

Replace this with proper accessor functions:
- native_read_cr3_pa(): Uses CR3_ADDR_MASK properly clearing SME encryption bit and extracting only the physical address portion.
- mask pgd value with PTE_PFN_MASK instead of PAGE_MASK, accounting for flags above physical address (_PAGE_BIT_NOPTISHADOW in particular).

Fixes cb1c9e02b0c1 ("x86/efistub: Perform 4/5 level paging switch from the stub")
Quoted 3 lines
---
 drivers/firmware/efi/libstub/x86-5lvl.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

allmodconfig build:

Quoted 17 lines

ld: error: unplaced orphan section __bug_table' from drivers/firmware/efi/libstub/x86-5lvl.stub.o'
ld: error: unplaced orphan section .altinstructions' from drivers/firmware/efi/libstub/x86-5lvl.stub.o'
ld: error: unplaced orphan section .altinstr_replacement' from drivers/firmware/efi/libstub/x86-5lvl.stub.o'
ld: Unexpected GOT/PLT entries detected!
ld: Unexpected run-time procedure linkages detected!
ld: Unexpected run-time relocations (.rela) detected!
ld: drivers/firmware/efi/libstub/x86-5lvl.stub.o: in function `efi_5level_switch':
x86-5lvl.c:(.text+0x13e): undefined reference to `pv_ops'
ld: x86-5lvl.c:(.text+0x14d): undefined reference to `pv_ops'
ld: drivers/firmware/efi/libstub/x86-5lvl.stub.o:(.altinstr_replacement+0x1): undefined reference to `BUG_func'
ld: arch/x86/boot/compressed/vmlinux: hidden symbol `pv_ops' isn't defined
ld: final link failed: bad value
make[3]: *** [arch/x86/boot/compressed/Makefile:116: arch/x86/boot/compressed/vmlinux] Error 1 make[2]: *** [arch/x86/boot/Makefile:96: arch/x86/boot/compressed/vmlinux] Error 2
make[1]: *** [arch/x86/Makefile:308: bzImage] Error 2
make: *** [Makefile:248: __sub-make] Error 2

This code should be using native_pgd_val() not pgd_val().

maintainer
#6 lore raw

Re: [PATCH v2 2/2] efi/libstub: Fix page table access in 5-level to 4-level paging transition

Quoted 1 line · Ard Biesheuvel

This code should be using native_pgd_val() not pgd_val().

Seems to fix it, thanks.

I'll let Usama do more testing along with the usual build smoke tests - all permutations of the below:

ARCHES=('x86_64' 'i386')
SMOKE_CONFIGS=("allnoconfig" "defconfig" "allmodconfig" "allyesconfig")

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
maintainer Reviewed-by on the v2 series
#7 lore raw
In reply to Usama Arif
Quoted 35 lines · Usama Arif

This series addresses critical bugs in the kexec path when transitioning from a kernel using 5-level page tables to one using 4-level page tables.

The root cause is improper handling of PGD entry value during the page level transition. Specifically PGD entry value is masked with PAGE_MASK instead of PTE_PFN_MASK, failing to account for high-order software bits like _PAGE_BIT_NOPTISHADOW (bit 58).

When bit 58 (_PAGE_BIT_NOPTISHADOW) is set in the source kernel, the target 4-level kernel doesn't recognize it and fails to mask it properly, leading to kexec failure.

Patch 1: Fixes the x86 boot compressed code path by replacing direct CR3 dereferencing with read_cr3_pa() and using PTE_PFN_MASK instead of PAGE_MASK.

Patch 2: Applies the same fix to the EFI stub code path. (Done in a separate patch as Fixes tag is different).

The patches are based on aaa9c3550b60d6259d6ea8b1175ade8d1242444e (next-20251022)

v1 -> v2:
- Remove patch 3 from v1 to fix kexec for source kernel in 5-level to 4-level transition where the 4 level kernel doesnt have patch 1 and 2 (Dave Hansen)
- Add include for asm/pgtable.h to fix build for x86_64-allnoconfig (kernel test bot)
- Use native_read_cr3_pa and for both paths (Ard Biesheuvel)

Quoted 6 lines

Usama Arif (2):
x86/boot: Fix page table access in 5-level to 4-level paging
transition
efi/libstub: Fix page table access in 5-level to 4-level paging
transition

maintainer
#8 lore raw
In reply to Usama Arif
Quoted 2 lines · Usama Arif

This series addresses critical bugs in the kexec path when transitioning from a kernel using 5-level page tables to one using 4-level page tables.

Out of curiosity: what is the real-life use case for this?

Judging by the Reported-by's I guess Meta is doing some kexec-ing into default kernels which are 4-level so that they can work on any machine ...

Close?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#9 lore raw
Quoted 10 lines · Borislav Petkov
Quoted 2 lines · Usama Arif

This series addresses critical bugs in the kexec path when transitioning from a kernel using 5-level page tables to one using 4-level page tables.

Out of curiosity: what is the real-life use case for this?

Judging by the Reported-by's I guess Meta is doing some kexec-ing into default kernels which are 4-level so that they can work on any machine ...

Close?

Older kernels in our fleet run with 5-level paging disabled. The newer one enables it. Machines need to switch between kernel version from time to time for different reasons. Switching from the newer kernel to an older one triggered the issue.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
maintainer
#10 lore raw
Quoted 4 lines · Kiryl Shutsemau

Older kernels in our fleet run with 5-level paging disabled. The newer one enables it. Machines need to switch between kernel version from time to time for different reasons. Switching from the newer kernel to an older one triggered the issue.

Thx, makes sense.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette