x86/mm: Fix some issues with using trampoline_pgd

From: Joerg Roedel <[email protected]>

Hi,

here are a couple of fixes and documentation improvements for the use of the trampoline_pgd in the kernel. Most importantly it fixes the issue that switching to the trampoline_pgd will unmap the kernel stack and real_mode_header, making crashes likely before the code can actually jump to real mode.

The first patch adds a comment to document that the trampoline_pgd aliases kernel page-tables in the user address range, establishing global TLB entries for these addresses. The next two patches add global TLB flushes when switching to and from the trampoline_pgd.

The last patch extends the trampoline_pgd to cover the whole kernel address range. This is needed to make sure the stack and the real_mode_header are still mapped after the switch and that the code flow can safely reach real-mode.

Please review.

Thanks,

Joerg

Changes v2->v3:

- Addressed review comments from Dave Hansen

Link to v2: https://lore.kernel.org/lkml/[email protected]/

Joerg Roedel (4):
x86/realmode: Add comment for Global bit usage in trampline_pgd x86/mm/64: Flush global TLB on boot and AP bringup x86/mm: Flush global TLB when switching to trampoline page-table x86/64/mm: Map all kernel memory into trampoline_pgd

 arch/x86/include/asm/realmode.h |  1 +
 arch/x86/kernel/head64.c        | 15 ++++++++++++++
 arch/x86/kernel/head_64.S       | 19 +++++++++++++++++-
 arch/x86/kernel/reboot.c        | 12 ++---------
 arch/x86/mm/init.c              |  5 +++++
 arch/x86/realmode/init.c        | 35 ++++++++++++++++++++++++++++++++-
 6 files changed, 75 insertions(+), 12 deletions(-)

base-commit: 5816b3e6577eaa676ceb00a848f0fd65fe2adc29

2.33.0

From: Joerg Roedel <[email protected]>

Document the fact that using the trampoline_pgd will result in the creation of global TLB entries in the user range of the address space.

---
 arch/x86/mm/init.c | 5 +++++
 1 file changed, 5 insertions(+)

index 23a14d82e783..accd702d4253 100644
@@ -714,6 +714,11 @@ static void __init memory_map_bottom_up(unsigned long map_start,
 static void __init init_trampoline(void)
 {
 #ifdef CONFIG_X86_64
+	/*
+	 * The code below will alias kernel page-tables in the user-range of the
+	 * address space, including the Global bit. So global TLB entries will
+	 * be created when using the trampoline page-table.
+	 */
 	if (!kaslr_memory_enabled())
 		trampoline_pgd_entry = init_top_pgt[pgd_index(__PAGE_OFFSET)];
 	else

2.33.0

From: Joerg Roedel <[email protected]>

The AP bringup code uses the trampoline_pgd page-table, which establishes global mappings in the user range of the address space. Flush the global TLB entries after the indentity mappings are removed so no stale entries remain in the TLB.

---
 arch/x86/kernel/head64.c  | 15 +++++++++++++++
 arch/x86/kernel/head_64.S | 19 ++++++++++++++++++-
 2 files changed, 33 insertions(+), 1 deletion(-)

index de01903c3735..cae21afe0922 100644
@@ -457,6 +457,19 @@ static void __init copy_bootdata(char *real_mode_data)
 	sme_unmap_bootdata(real_mode_data);
 }
 
+/*
+ * The __flush_tlb_all() function uses all kinds of state which is not
+ * initialized that early and can not be used here. So the helper below is used
+ * to flush global TLB entries.
+ */
+static void __init early_flush_tlb_global(void)
+{
+	unsigned long cr4 = native_read_cr4();
+
+	native_write_cr4(cr4 ^ X86_CR4_PGE);
+	native_write_cr4(cr4);
+}
+
 asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
 {
 	/*

@@ -478,6 +491,8 @@ asmlinkage __visible void __init x86_64_start_kernel(char * real_mode_data)
 	/* Kill off the identity-map trampoline */
 	reset_early_page_tables();
 
+	early_flush_tlb_global();
+
 	clear_bss();
 
 	clear_page(init_top_pgt);

index d8b3ebd2bb85..bd4b6ebafdc3 100644
@@ -166,9 +166,26 @@ SYM_INNER_LABEL(secondary_startup_64_no_verify, SYM_L_GLOBAL)
 	call	sev_verify_cbit
 	popq	%rsi
 
-	/* Switch to new page-table */
+	/*
+	 * Switch to new page-table
+	 *
+	 * For the boot CPU this switches to early_top_pgt which still has the
+	 * indentity mappings present. The secondary CPUs will switch to the
+	 * init_top_pgt here, away from the trampoline_pgd and unmapping the
+	 * indentity mapped ranges.
+	 *
+	 * Do a global TLB flush after the CR3 switch to make sure the TLB
+	 * entries from the identity mapping are flushed.
+	 */
 	movq	%rax, %cr3
 
+	/* Flush global TLB entries - only needed for secondary CPUs */
+	movq	%cr4, %rcx
+	movq	%rcx, %rax
+	xorq	$X86_CR4_PGE, %rcx
+	movq	%rcx, %cr4
+	movq	%rax, %cr4
+
 	/* Ensure I am executing from virtual addresses */
 	movq	$1f, %rax
 	ANNOTATE_RETPOLINE_SAFE

2.33.0

Also, I'm wondering if you could compact this even more by defining a function toggling the PGE bit and calling it from everywhere, even from asm.

Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Yeah, that would make sense, but is probably worth its own patch-set. Unifying this across arch/x86/ needs to touch a couple more places and needs special care so that the function is safe to call from early asm.

Regards,

Jörg Rödel
[email protected]

SUSE Software Solutions Germany GmbH
Maxfeldstr. 5
90409 Nürnberg
Germany
 
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev

I'd gladly review a preparatory patchset doing that. The usual strategy is, cleanup and refactoring first, new features later.

Thx.

Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Is that also true for fixes? Because this patch-set actually tries to fix an issue present in current code and not adding any new feature.

Thanks,

Jörg Rödel
[email protected]

SUSE Software Solutions Germany GmbH
Maxfeldstr. 5
90409 Nürnberg
Germany
 
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev

From: Joerg Roedel <[email protected]>

Move the switching code into a function so that it can be re-used and add a global TLB flush. This makes sure that usage of memory which is not mapped in the trampoline page-table is reliably caught.

---
 arch/x86/include/asm/realmode.h |  1 +
 arch/x86/kernel/reboot.c        | 12 ++----------
 arch/x86/realmode/init.c        | 23 +++++++++++++++++++++++
 3 files changed, 26 insertions(+), 10 deletions(-)

index 5db5d083c873..331474b150f1 100644
@@ -89,6 +89,7 @@ static inline void set_real_mode_mem(phys_addr_t mem)
 }
 
 void reserve_real_mode(void);
+void load_trampoline_pgtable(void);
 
 #endif /* __ASSEMBLY__ */
 

index 0a40df66a40d..fa700b46588e 100644
@@ -113,17 +113,9 @@ void __noreturn machine_real_restart(unsigned int type)
 	spin_unlock(&rtc_lock);
 
 	/*
-	 * Switch back to the initial page table.
+	 * Switch to the trampoline page table.
 	 */
-#ifdef CONFIG_X86_32
-	load_cr3(initial_page_table);
-#else
-	write_cr3(real_mode_header->trampoline_pgd);
-
-	/* Exiting long mode will fail if CR4.PCIDE is set. */
-	if (boot_cpu_has(X86_FEATURE_PCID))
-		cr4_clear_bits(X86_CR4_PCIDE);
-#endif
+	load_trampoline_pgtable();
 
 	/* Jump to the identity-mapped low memory code */
 #ifdef CONFIG_X86_32

index 31b5856010cb..b9802b18f504 100644
@@ -17,6 +17,29 @@ u32 *trampoline_cr4_features;
 /* Hold the pgd entry used on booting additional CPUs */
 pgd_t trampoline_pgd_entry;
 
+void load_trampoline_pgtable(void)
+{
+#ifdef CONFIG_X86_32
+	load_cr3(initial_page_table);
+#else
+	/* Exiting long mode will fail if CR4.PCIDE is set. */
+	if (boot_cpu_has(X86_FEATURE_PCID))
+		cr4_clear_bits(X86_CR4_PCIDE);
+
+	write_cr3(real_mode_header->trampoline_pgd);
+#endif
+
+	/*
+	 * The CR3 write above will not flush global TLB entries.
+	 * Stale, global entries from previous sets of page tables may
+	 * still be present.  Flush those stale entries.
+	 *
+	 * This ensures that memory accessed while running with
+	 * trampoline_pgd is *actually* mapped into trampoline_pgd.
+	 */
+	__flush_tlb_all();
+}
+
 void __init reserve_real_mode(void)
 {
 	phys_addr_t mem;

2.33.0

This looks fine. But, just to be clear, nothing in this series reuses the code, right?

Correct, but my SEV-ES kexec series will re-use it, as it also uses the trampoline_pgd and switches to real-mode to park the APs.

Regards,

Jörg

Okay, but in the caller it is not visible the CR4.PCID is disabled in this function. I'd rather update the comment to tell that the function is called before transitioning to real mode?

Yes, the call to cr4_clear_bits() is not safe anymore on the trampoline page-table, because the per-cpu areas are not fully mapped anymore.

This changes with the next patch, but its nevertheless more robust to minimize the code running on the trampoline page-table.

I will add that to the commit message.

Regards,

Jörg Rödel
[email protected]

SUSE Software Solutions Germany GmbH
Maxfeldstr. 5
90409 Nürnberg
Germany
 
(HRB 36809, AG Nürnberg)
Geschäftsführer: Ivo Totev

Well, if something calls load_trampoline_pgtable(), it kinda assumes that if it wants that that function will do all the necessary steps to load it, including clearing PCIDE.

Why does the caller even need to know that that function clears CR4.PCIDE?

Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

From: Joerg Roedel <[email protected]>

The trampoline_pgd only maps the 0xfffffff000000000-0xffffffffffffffff range of kernel memory (with 4-level paging). This range contains the kernels text+data+bss mappings and the module mapping space, but not the direct mapping and the vmalloc area.

This is enough to get an application processors out of real-mode, but for code that switches back to real-mode the trampoline_pgd is missing important parts of the address space. For example, consider this code from arch/x86/kernel/reboot.c, function machine_real_restart() for a 64-bit kernel:

	#ifdef CONFIG_X86_32
		load_cr3(initial_page_table);
	#else
		write_cr3(real_mode_header->trampoline_pgd);

		/* Exiting long mode will fail if CR4.PCIDE is set. */
		if (boot_cpu_has(X86_FEATURE_PCID))
			cr4_clear_bits(X86_CR4_PCIDE);
	#endif

		/* Jump to the identity-mapped low memory code */
	#ifdef CONFIG_X86_32
		asm volatile("jmpl *%0" : :
			     "rm" (real_mode_header->machine_real_restart_asm),
			     "a" (type));
	#else
		asm volatile("ljmpl *%0" : :
			     "m" (real_mode_header->machine_real_restart_asm),
			     "D" (type));
	#endif

The code switches to the trampoline_pgd, which unmaps the direct mapping and also the kernel stack. The call to cr4_clear_bits() will find no stack and crash the machine. The real_mode_header pointer below points into the direct mapping, and dereferencing it also causes a crash.

The reason this does not crash always is only that kernel mappings are global and the CR3 switch does not flush those mappings. But if theses mappings are not in the TLB already, the above code will crash before it can jump to the real-mode stub.

Extend the trampoline_pgd to contain all kernel mappings to prevent these crashes and to make code which runs on this page-table more robust.

---
 arch/x86/realmode/init.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

index b9802b18f504..77617cd624fe 100644
@@ -95,6 +95,7 @@ static void __init setup_real_mode(void)
 #ifdef CONFIG_X86_64
 	u64 *trampoline_pgd;
 	u64 efer;
+	int i;
 #endif
 
 	base = (unsigned char *)real_mode_header;

@@ -151,8 +152,17 @@ static void __init setup_real_mode(void)
 	trampoline_header->flags = 0;
 
 	trampoline_pgd = (u64 *) __va(real_mode_header->trampoline_pgd);
+
+	/* Map the real mode stub as virtual == physical */
 	trampoline_pgd[0] = trampoline_pgd_entry.pgd;
-	trampoline_pgd[511] = init_top_pgt[511].pgd;
+
+	/*
+	 * Include the entirety of the kernel mapping into the trampoline
+	 * PGD.  This way, all mappings present in the normal kernel page
+	 * tables are usable while running on trampoline_pgd.
+	 */
+	for (i = pgd_index(__PAGE_OFFSET); i < PTRS_PER_PGD; i++)
+		trampoline_pgd[i] = init_top_pgt[i].pgd;
 #endif
 
 	sme_sev_setup_real_mode(trampoline_header);

2.33.0