Misc GDT fixes and a cleanup

The x86 smpboot trampoline expects initial_page_table to have the GDT mapped. If the GDT ends up in a virtually mapped per-cpu page, then it won't be in the page tables at all until perc-pu areas are set up. The result will be a triple fault the first time that the CPU attempts to access the GDT after LGDT loads the perc-pu GDT.

This appears to be an old bug, but somehow the GDT fixmap rework is triggering it. This seems to have something to do with the memory layout.

Signed-off-by Andy Lutomirski <[email protected]>

Cc Ard Biesheuvel <[email protected]>

Cc Boris Ostrovsky <[email protected]>

Cc Borislav Petkov <[email protected]>

Cc Brian Gerst <[email protected]>

Cc Denys Vlasenko <[email protected]>

Cc H. Peter Anvin <[email protected]>

Cc Josh Poimboeuf <[email protected]>

Cc Juergen Gross <jgross-IBi9RG/[email protected]>

Cc Linus Torvalds <torvalds-de/[email protected]>

Cc Matt Fleming <matt-mF/unelCI9GS6iBeEJttW/[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Thomas Garnier <[email protected]>

Cc Thomas Gleixner <[email protected]>

Cc [email protected]

Link http://lkml.kernel.org/r/a553264a5972c6a86f9b5caac237470a0c74a720.1490218061.git.luto-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org open here

Signed-off-by Ingo Molnar <[email protected]>

---
 arch/x86/kernel/setup.c        | 15 ---------------
 arch/x86/kernel/setup_percpu.c | 21 +++++++++++++++++++++
 2 files changed, 21 insertions(+), 15 deletions(-)

arch/x86/kernel/setup.c +0 -15

index 4bf0c89..56b1177 100644
@@ -1226,21 +1226,6 @@ void __init setup_arch(char **cmdline_p)
 
 	kasan_init();
 
-#ifdef CONFIG_X86_32
-	/* sync back kernel address range */
-	clone_pgd_range(initial_page_table + KERNEL_PGD_BOUNDARY,
-			swapper_pg_dir     + KERNEL_PGD_BOUNDARY,
-			KERNEL_PGD_PTRS);
-
-	/*
-	 * sync back low identity map too.  It is used for example
-	 * in the 32-bit EFI stub.
-	 */
-	clone_pgd_range(initial_page_table,
-			swapper_pg_dir     + KERNEL_PGD_BOUNDARY,
-			min(KERNEL_PGD_PTRS, KERNEL_PGD_BOUNDARY));
-#endif
-
 	tboot_probe();
 
 	map_vsyscall();

arch/x86/kernel/setup_percpu.c +12 -0

index 11338b0..bb1e8cc 100644
@@ -288,4 +288,25 @@ void __init setup_per_cpu_areas(void)
 
 	/* Setup cpu initialized, callin, callout masks */
 	setup_cpu_local_masks();
+
+#ifdef CONFIG_X86_32
+	/*
+	 * Sync back kernel address range.  We want to make sure that
+	 * all kernel mappings, including percpu mappings, are available
+	 * in the smpboot asm.  We can't reliably pick up percpu
+	 * mappings using vmalloc_fault(), because exception dispatch
+	 * needs percpu data.
+	 */
+	clone_pgd_range(initial_page_table + KERNEL_PGD_BOUNDARY,
+			swapper_pg_dir     + KERNEL_PGD_BOUNDARY,
+			KERNEL_PGD_PTRS);

Quoted 6 lines · tip-bot for Andy Lutomirski

The x86 smpboot trampoline expects initial_page_table to have the GDT mapped. If the GDT ends up in a virtually mapped per-cpu page, then it won't be in the page tables at all until perc-pu areas are set up. The result will be a triple fault the first time that the CPU attempts to access the GDT after LGDT loads the perc-pu GDT.

Quoted 3 lines

This appears to be an old bug, but somehow the GDT fixmap rework is triggering it. This seems to have something to do with the memory layout.

This breaks the boot on our Intel Quark platform (IOT2000, similar to Galileo Gen2). Reverting it over master makes it work again. Any idea what goes wrong? Let me know how I can help debugging this.

Quoted 10 lines · Jan Kiszka

Quoted 9 lines · tip-bot for Andy Lutomirski

The x86 smpboot trampoline expects initial_page_table to have the GDT mapped. If the GDT ends up in a virtually mapped per-cpu page, then it won't be in the page tables at all until perc-pu areas are set up. The result will be a triple fault the first time that the CPU attempts to access the GDT after LGDT loads the perc-pu GDT.

This appears to be an old bug, but somehow the GDT fixmap rework is triggering it. This seems to have something to do with the memory layout.

Quoted 3 lines

This breaks the boot on our Intel Quark platform (IOT2000, similar to Galileo Gen2). Reverting it over master makes it work again. Any idea what goes wrong? Let me know how I can help debugging this.

JFYI: As of today linux-next when _kexec:ed_ works fine to me

Perhaps I can test this later with direct boot from SD card.

Quoted 12 lines · Andy Shevchenko

Quoted 10 lines · Jan Kiszka

Quoted 9 lines · tip-bot for Andy Lutomirski

The x86 smpboot trampoline expects initial_page_table to have the GDT mapped. If the GDT ends up in a virtually mapped per-cpu page, then it won't be in the page tables at all until perc-pu areas are set up. The result will be a triple fault the first time that the CPU attempts to access the GDT after LGDT loads the perc-pu GDT.

This appears to be an old bug, but somehow the GDT fixmap rework is triggering it. This seems to have something to do with the memory layout.

Quoted 8 lines

Quoted 3 lines

This breaks the boot on our Intel Quark platform (IOT2000, similar to Galileo Gen2). Reverting it over master makes it work again. Any idea what goes wrong? Let me know how I can help debugging this.

JFYI: As of today linux-next when _kexec:ed_ works fine to me

Perhaps I can test this later with direct boot from SD card.

The most likely explanation is that there's some code that needs the page table synced and runs before setup_per_cpu_areas(). The relevant init code is:

    setup_arch(&command_line);
    mm_init_cpumask(&init_mm);
    setup_command_line(command_line);
    setup_nr_cpu_ids();
    setup_per_cpu_areas();

so I didn't move it very far. It would be awesome if we could get a backtrace when the failure happens, but it's likely to be a triple fault. Is this an EFI boot? I bet the failure is in efi_init().

Could you try reverting just the deletions in the patch? I.e. try a kernel with both the old and the new copies of the code I moved.

On 2017-05-08 13:21, Andy Lutomirski wrote:

Yes, it's an EFI thing. Unfortunately, I didn't make earlycon/earlyprintk work yet.

Quoted 4 lines · Jan Kiszka

On 2017-05-08 13:21, Andy Lutomirski wrote:

Yes, it's an EFI thing. Unfortunately, I didn't make earlycon/earlyprintk work yet.

Quark does use HS UART for the console and we have no support for such in x86/earlyprintk. earlycon should work if you add proper parameters to the command line (including port base address, no ttySx), though I don't remember if it makes really "early".

Quoted 33 lines · Andy Lutomirski

Quoted 12 lines · Andy Shevchenko

Quoted 10 lines · Jan Kiszka

Quoted 9 lines · tip-bot for Andy Lutomirski

The x86 smpboot trampoline expects initial_page_table to have the GDT mapped. If the GDT ends up in a virtually mapped per-cpu page, then it won't be in the page tables at all until perc-pu areas are set up. The result will be a triple fault the first time that the CPU attempts to access the GDT after LGDT loads the perc-pu GDT.

This appears to be an old bug, but somehow the GDT fixmap rework is triggering it. This seems to have something to do with the memory layout.

Quoted 8 lines

Quoted 3 lines

This breaks the boot on our Intel Quark platform (IOT2000, similar to Galileo Gen2). Reverting it over master makes it work again. Any idea what goes wrong? Let me know how I can help debugging this.

JFYI: As of today linux-next when _kexec:ed_ works fine to me

Perhaps I can test this later with direct boot from SD card.

The most likely explanation is that there's some code that needs the page table synced and runs before setup_per_cpu_areas(). The relevant init code is:

    setup_arch(&command_line);
    mm_init_cpumask(&init_mm);
    setup_command_line(command_line);
    setup_nr_cpu_ids();
    setup_per_cpu_areas();

Quoted 3 lines

so I didn't move it very far. It would be awesome if we could get a backtrace when the failure happens, but it's likely to be a triple fault. Is this an EFI boot? I bet the failure is in efi_init().

Yes, it's an EFI thing. Unfortunately, I didn't make earlycon/earlyprintk work yet.

Quoted 1 line

Quoted 2 lines

Could you try reverting just the deletions in the patch? I.e. try a kernel with both the old and the new copies of the code I moved.

Let me try that later. I can also move the new code around to nail down the dependency.

Quoted 42 lines · Jan Kiszka

Quoted 33 lines · Andy Lutomirski

Quoted 12 lines · Andy Shevchenko

Quoted 10 lines · Jan Kiszka

Quoted 9 lines · tip-bot for Andy Lutomirski

The x86 smpboot trampoline expects initial_page_table to have the GDT mapped. If the GDT ends up in a virtually mapped per-cpu page, then it won't be in the page tables at all until perc-pu areas are set up. The result will be a triple fault the first time that the CPU attempts to access the GDT after LGDT loads the perc-pu GDT.

This appears to be an old bug, but somehow the GDT fixmap rework is triggering it. This seems to have something to do with the memory layout.

Quoted 8 lines

Quoted 3 lines

This breaks the boot on our Intel Quark platform (IOT2000, similar to Galileo Gen2). Reverting it over master makes it work again. Any idea what goes wrong? Let me know how I can help debugging this.

JFYI: As of today linux-next when _kexec:ed_ works fine to me

Perhaps I can test this later with direct boot from SD card.

The most likely explanation is that there's some code that needs the page table synced and runs before setup_per_cpu_areas(). The relevant init code is:

    setup_arch(&command_line);
    mm_init_cpumask(&init_mm);
    setup_command_line(command_line);
    setup_nr_cpu_ids();
    setup_per_cpu_areas();

Quoted 3 lines

so I didn't move it very far. It would be awesome if we could get a backtrace when the failure happens, but it's likely to be a triple fault. Is this an EFI boot? I bet the failure is in efi_init().

Yes, it's an EFI thing. Unfortunately, I didn't make earlycon/earlyprintk work yet.

Quoted 1 line

Quoted 6 lines

Quoted 2 lines

Could you try reverting just the deletions in the patch? I.e. try a kernel with both the old and the new copies of the code I moved.

Let me try that later. I can also move the new code around to nail down the dependency.

I found the reason: your patch is very discriminating! Not the whole world is multicore yet. ;)

setup_per_cpu_areas() is taken from mm/percpu.c in case of !CONFIG_SMP. So the new home for the resync is not even built.

Any suggestions how to refactor things instead?

Jan

--
Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux

Hi all-

This applies to tip:x86/mm. For ease of testing, the series is here, too:

https://git.kernel.org/pub/scm/linux/kernel/git/luto/linux.git/tag/?h=review_20170322_gdt_and_wp

This fixes a few issues, most of which appear to be rather old. For whatever reason, Thomas' GDT series unearthed them. (And one is a genuine bug in Thomas' code but, in his defense, he might have cut-and-pasted it verbatim from the identical bug in the EFI code.)

The last three patches are cleanups I did while tracking these down.

Boris, any chance you could test this series on Xen? The 64-bit case works for me, but I'm having issues testing on 32-bit right now.

Ingo, the first patch should address your concerns from the earlier version.

Andy Lutomirski (7):
selftests/x86/ldt_gdt_32: Work around a glibc sigaction bug x86/gdt: Fix setup_fixmap_gdt() to use the correct PA x86/efi/32: Fix EFI on systems where the percpu GDT is virtually
mapped
x86/boot/32: Defer resyncing initial_page_table until percpu is set up x86/gdt: Get rid of the get_*_gdt_*_vaddr() helpers x86/xen/gdt: Use X86_FEATURE_XENPV instead of globals for the GDT
fixup
x86/boot/32: Rewrite test_wp_bit()

Hi all-

This applies to tip:x86/mm. For ease of testing, the series is here, too:

https://git.kernel.org/pub/scm/linux/kernel/git/luto/linux.git/tag/?h=review_20170322_gdt_and_wp

This fixes a few issues, most of which appear to be rather old. For whatever reason, Thomas' GDT series unearthed them. (And one is a genuine bug in Thomas' code but, in his defense, he might have cut-and-pasted it verbatim from the identical bug in the EFI code.)

The last three patches are cleanups I did while tracking these down.

Boris, any chance you could test this series on Xen? The 64-bit case works for me, but I'm having issues testing on 32-bit right now.