Skip to contentSkip to conversation

Fix VMI crash on boot in 2.6.27+ kernels

PATCH In review Zachary Amsden 4 participants 16 messages Dec 9, 2008 – Dec 12, 2008 x86

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Fix VMI crash on boot in 2.6.27+ kernels

Patches backported into 2.6.27.4 caused a regression with VMI kernels running on VMware which ends in a page fault during boot. I have a fix which still allows DMI checks to be done early.

VMI initialiation can relocate the fixmap, causing early_ioremap to malfunction if it is initialized before the relocation. The ioremap area is low enough in virtual address space that no actual collision occurs, however, because the pagetables for it were not allocated under VMI mode, the pagetable updates are dropped by the hypervisor as irrelevant, resulting in a crash on boot.

The best fix is perhaps to move early_ioremap_init() after vmi_init(). The only things done before VMI init are basic memory access, things like collating the memory map, collecting boot CPUID capabilities, and parsing the early command line options... which vmi_init needs.

Since this went back into 2.6.27, it needs to go to both 2.6.28 and eventually to stable. I didn't add any comments or anything as there could be some debate what the proper ordering should be. In case that becomes an interesting discussion, there are two relevant facts in git today:

1) no clients of early_ioremap occur before DMI.
2) VMI requires access to early boot params.

If any can suggest a better ordering, I am certainly open to that as well.

Thanks,

Zach

VMI initialiation can relocate the fixmap, causing early_ioremap to malfunction if it is initialized before the relocation. The ioremap area is low enough in virtual address space that no actual collision occurs, however, because the pagetables for it were not allocated under VMI mode, the pagetable updates are dropped by the hypervisor as irrelevant, resulting in a crash on boot.

arch/x86/kernel/setup.c +2 -1
index 9d5674f..9627753 100644
@@ -795,7 +795,6 @@ void __init setup_arch(char **cmdline_p)
 #endif
 
 	early_cpu_init();
-	early_ioremap_init();
 
 	ROOT_DEV = old_decode_dev(boot_params.hdr.root_dev);
 	screen_info = boot_params.screen_info;
@@ -888,6 +887,8 @@ void __init setup_arch(char **cmdline_p)
 	vmi_init();
 #endif
 
+	early_ioremap_init();
+
 	/* after early param, so could get panic from serial */
 	reserve_early_setup_data();
 
#1 lore raw
Quoted 3 lines · Zachary Amsden

Patches backported into 2.6.27.4 caused a regression with VMI kernels running on VMware which ends in a page fault during boot. I have a fix which still allows DMI checks to be done early.

Is this regression also present on 2.6.28-rc kernels as well?

thanks,

greg k-h

#2 lore raw
Quoted 6 lines · Greg KH
Quoted 3 lines · Zachary Amsden

Patches backported into 2.6.27.4 caused a regression with VMI kernels running on VMware which ends in a page fault during boot. I have a fix which still allows DMI checks to be done early.

Is this regression also present on 2.6.28-rc kernels as well?

Yes, I reproduced and fixed this on a git tree synced to mainline today.

Thanks,

Zach

#3 lore raw
In reply to Zachary Amsden
Quoted 15 lines · Zachary Amsden

Patches backported into 2.6.27.4 caused a regression with VMI kernels running on VMware which ends in a page fault during boot. I have a fix which still allows DMI checks to be done early.

The best fix is perhaps to move early_ioremap_init() after vmi_init(). The only things done before VMI init are basic memory access, things like collating the memory map, collecting boot CPUID capabilities, and parsing the early command line options... which vmi_init needs.

Since this went back into 2.6.27, it needs to go to both 2.6.28 and eventually to stable. I didn't add any comments or anything as there could be some debate what the proper ordering should be. In case that becomes an interesting discussion, there are two relevant facts in git today:

Quoted 5 lines

1) no clients of early_ioremap occur before DMI.
2) VMI requires access to early boot params.

If any can suggest a better ordering, I am certainly open to that as well.

VMI initialiation can relocate the fixmap, causing early_ioremap to malfunction if it is initialized before the relocation. The ioremap area is low enough in virtual address space that no actual collision occurs, however, because the pagetables for it were not allocated under VMI mode, the pagetable updates are dropped by the hypervisor as irrelevant, resulting in a crash on boot.

arch/x86/kernel/setup.c +2 -1
index 9d5674f..9627753 100644
@@ -795,7 +795,6 @@ void __init setup_arch(char **cmdline_p)
 #endif
 
 	early_cpu_init();
-	early_ioremap_init();
 
 	ROOT_DEV = old_decode_dev(boot_params.hdr.root_dev);
 	screen_info = boot_params.screen_info;
@@ -888,6 +887,8 @@ void __init setup_arch(char **cmdline_p)
 	vmi_init();
 #endif
 
+	early_ioremap_init();
+
 	/* after early param, so could get panic from serial */
 	reserve_early_setup_data();
 

you can not move that late,

parse_setup_data==>early_memremap==>__early_ioremap

YH

#4 lore raw
Quoted 8 lines · Yinghai Lu

VMI initialiation can relocate the fixmap, causing early_ioremap to malfunction if it is initialized before the relocation. The ioremap area is low enough in virtual address space that no actual collision occurs, however, because the pagetables for it were not allocated under VMI mode, the pagetable updates are dropped by the hypervisor as irrelevant, resulting in a crash on boot.

I have mentioned in the past that I think the very concept of relocating the fixmap to be utterly braindead. Instead, I believe we should locate it low in kernel space so it doesn't have to be relocated. It's unfortunately a relatively large change.

-hpa

#5 lore raw
     A. On Tue, 2008-12-09 at 17:48 -0800, H. Peter Anvin wrote:
Quoted 10 lines
Quoted 8 lines · Yinghai Lu

VMI initialiation can relocate the fixmap, causing early_ioremap to malfunction if it is initialized before the relocation. The ioremap area is low enough in virtual address space that no actual collision occurs, however, because the pagetables for it were not allocated under VMI mode, the pagetable updates are dropped by the hypervisor as irrelevant, resulting in a crash on boot.

Quoted 4 lines

I have mentioned in the past that I think the very concept of relocating the fixmap to be utterly braindead. Instead, I believe we should locate it low in kernel space so it doesn't have to be relocated. It's unfortunately a relatively large change.

I agree, but a bit too late for 2.6.27 stable and 2.6.28-rc.

#6 lore raw
Quoted 8 lines · Zachary Amsden
Quoted 5 lines
Quoted 1 line

I have mentioned in the past that I think the very concept of relocating the fixmap to be utterly braindead. Instead, I believe we should locate it low in kernel space so it doesn't have to be relocated. It's unfortunately a relatively large change.

I agree, but a bit too late for 2.6.27 stable and 2.6.28-rc.

Of course, no argument there.

-hpa

#7 lore raw
In reply to Yinghai Lu
Quoted 3 lines · Yinghai Lu

you can not move that late,

parse_setup_data==>early_memremap==>__early_ioremap

Eww. Let me find a way to fix that. It may be as simple as detecting the required fixmap relocation first, then initializing VMI later, when we can parse options, but it will need testing...

Thanks for the quick response, I realize it is late in the 2.6.28-rc phase here and I will send an updated fix tomorrow.

Zach

#8 lore raw
In reply to Yinghai Lu
Quoted 3 lines · Yinghai Lu

you can not move that late,

parse_setup_data==>early_memremap==>__early_ioremap

How does this look?

VMI initialiation can relocate the fixmap, causing early_ioremap to malfunction if it is initialized before the relocation. To fix this, VMI activation is split into two phases; the detection, which must happen before setting up ioremap, and the activation, which must happen after parsing early boot parameters.

This fixes a crash on boot when VMI is enabled under VMware.

arch/x86/include/asm/vmi.h +1 -0
index b7c0dea..31dd52d 100644
@@ -224,6 +224,7 @@ struct pci_header {
 
 /* Function prototypes for bootstrapping */
 extern void vmi_init(void);
+extern void vmi_activate(void);
 extern void vmi_bringup(void);
 extern void vmi_apply_boot_page_allocations(void);
 
arch/x86/kernel/setup.c +8 -6
index 9d5674f..4c381cb 100644
@@ -794,6 +794,11 @@ void __init setup_arch(char **cmdline_p)
 	printk(KERN_INFO "Command line: %s\n", boot_command_line);
 #endif
 
+#ifdef CONFIG_VMI
+	/* VMI may relocate the fixmap; do this before touching ioremap area */
+	vmi_init();
+#endif
+
 	early_cpu_init();
 	early_ioremap_init();
 
@@ -880,12 +885,9 @@ void __init setup_arch(char **cmdline_p)
 	check_efer();
 #endif
 
-#if defined(CONFIG_VMI) && defined(CONFIG_X86_32)
-	/*
-	 * Must be before kernel pagetables are setup
-	 * or fixmap area is touched.
-	 */
-	vmi_init();
+#if defined(CONFIG_VMI)
+	/* Must be before kernel pagetables are setup */
+	vmi_activate();
 #endif
 
 	/* after early param, so could get panic from serial */
arch/x86/kernel/vmi_32.c +11 -5
index 8b6c393..22fd657 100644
@@ -960,8 +960,6 @@ static inline int __init activate_vmi(void)
 
 void __init vmi_init(void)
 {
-	unsigned long flags;
-
 	if (!vmi_rom)
 		probe_vmi_rom();
 	else
@@ -973,13 +971,21 @@ void __init vmi_init(void)
 
 	reserve_top_address(-vmi_rom->virtual_top);
 
-	local_irq_save(flags);
-	activate_vmi();
-
 #ifdef CONFIG_X86_IO_APIC
 	/* This is virtual hardware; timer routing is wired correctly */
 	no_timer_check = 1;
 #endif
+}
+
+void vmi_activate(void)
+{
+	unsigned long flags;
+
+	if (!vmi_rom)
+		return;
+
+	local_irq_save(flags);
+	activate_vmi();
 	local_irq_restore(flags & X86_EFLAGS_IF);
 }
 
#9 lore raw
Quoted 8 lines · Zachary Amsden
Quoted 3 lines · Yinghai Lu

you can not move that late,

parse_setup_data==>early_memremap==>__early_ioremap

How does this look?

arch/x86/kernel/setup.c +8 -6
index 9d5674f..4c381cb 100644
@@ -794,6 +794,11 @@ void __init setup_arch(char **cmdline_p)
 	printk(KERN_INFO "Command line: %s\n", boot_command_line);
 #endif
 
+#ifdef CONFIG_VMI
+	/* VMI may relocate the fixmap; do this before touching ioremap area */
+	vmi_init();
+#endif
+
 	early_cpu_init();
 	early_ioremap_init();
 
@@ -880,12 +885,9 @@ void __init setup_arch(char **cmdline_p)
 	check_efer();
 #endif
 
-#if defined(CONFIG_VMI) && defined(CONFIG_X86_32)
-	/*
-	 * Must be before kernel pagetables are setup
-	 * or fixmap area is touched.
-	 */
-	vmi_init();
+#if defined(CONFIG_VMI)
+	/* Must be before kernel pagetables are setup */
+	vmi_activate();
 #endif
 
 	/* after early param, so could get panic from serial */
arch/x86/kernel/vmi_32.c +0 -2
index 8b6c393..22fd657 100644
@@ -960,8 +960,6 @@ static inline int __init activate_vmi(void)
 
 void __init vmi_init(void)
 {
-	unsigned long flags;
-
 	if (!vmi_rom)
 		probe_vmi_rom();
 	else
@@ -973,13 +971,21 @@ void __init vmi_init(void)
 
 	reserve_top_address(-vmi_rom->virtual_top);
 
 

it seems still have some problem.
you moved reserve_top_address before parse_parameter...

so
void __init reserve_top_address(unsigned long reserve)

{
        BUG_ON(fixmaps_set > 0);
        printk(KERN_INFO "Reserving virtual address space above 0x%08x\n",
               (int)-reserve);
        __FIXADDR_TOP = -reserve - PAGE_SIZE;
        __VMALLOC_RESERVE += reserve;
}

/*
* vmalloc=size forces the vmalloc area to be exactly 'size'
* bytes. This can be used to increase (or decrease) the
* vmalloc area - the default is 128m.
*/
static int __init parse_vmalloc(char *arg)

{
        if (!arg)
                return -EINVAL;

        /* Add VMALLOC_OFFSET to the parsed value due to vm area guard hole*/
        __VMALLOC_RESERVE = memparse(arg, &arg) + VMALLOC_OFFSET;
        return 0;
}

early_param("vmalloc", parse_vmalloc);

__VMALLOC_RESERVE will be overwriten by vmalloc=...

you may need to split reserve_top_address() to two functions...

YH

#10 lore raw
Quoted 15 lines · Yinghai Lu
Quoted 8 lines · Zachary Amsden
Quoted 3 lines · Yinghai Lu

you can not move that late,

parse_setup_data==>early_memremap==>__early_ioremap

How does this look?

arch/x86/kernel/setup.c +0 -0
index 9d5674f..4c381cb 100644

arch/x86/kernel/setup.c void __init setup_arch(char **cmdline_p) · line 800

Quoted 7 lines
arch/x86/kernel/setup.c +4 -0
@@ -794,6 +794,11 @@ void __init setup_arch(char **cmdline_p)
 	printk(KERN_INFO "Command line: %s\n", boot_command_line);
 #endif
 
+#ifdef CONFIG_VMI
+	/* VMI may relocate the fixmap; do this before touching ioremap area */
+	vmi_init();
+#endif

Shouldn't the #ifdef not be needed here if the .h files are set up properly for the vmi_init prototype? Please try to keep them out of .c files wherever possible.

thanks,

greg k-h

#11 lore raw

Re: [PATCH] Fix VMI crash on boot in 2.6.27, 2.6.28 kernels

Quoted 8 lines · Greg KH

Review

Quoted 4 lines
(quoted diff) +4 -0
+#ifdef CONFIG_VMI
+	/* VMI may relocate the fixmap; do this before touching ioremap area */
+	vmi_init();
+#endif

Shouldn't the #ifdef not be needed here if the .h files are set up properly for the vmi_init prototype? Please try to keep them out of .c files wherever possible.

Yes, they should. Judging by setup.c though, you would think the opposite... in any case I fixed it. Please apply - and yes, I tested compile both ways.

VMI initialiation can relocate the fixmap, causing early_ioremap to malfunction if it is initialized before the relocation. To fix this, VMI activation is split into two phases; the detection, which must happen before setting up ioremap, and the activation, which must happen after parsing early boot parameters.

This fixes a crash on boot when VMI is enabled under VMware.

arch/x86/include/asm/vmi.h +7 -1
index b7c0dea..128958a 100644
@@ -223,9 +223,15 @@ struct pci_header {
 } __attribute__((packed));
 
 /* Function prototypes for bootstrapping */
+#ifdef CONFIG_VMI
 extern void vmi_init(void);
+extern void vmi_activate(void);
 extern void vmi_bringup(void);
-extern void vmi_apply_boot_page_allocations(void);
+#else
+#define vmi_init()
+#define vmi_activate()
+#define vmi_bringup()
+#endif
 
 /* State needed to start an application processor in an SMP system. */
 struct vmi_ap_state {
arch/x86/kernel/setup.c +5 -7
index 9d5674f..bdec76e 100644
@@ -794,6 +794,9 @@ void __init setup_arch(char **cmdline_p)
 	printk(KERN_INFO "Command line: %s\n", boot_command_line);
 #endif
 
+	/* VMI may relocate the fixmap; do this before touching ioremap area */
+	vmi_init();
+
 	early_cpu_init();
 	early_ioremap_init();
 
@@ -880,13 +883,8 @@ void __init setup_arch(char **cmdline_p)
 	check_efer();
 #endif
 
-#if defined(CONFIG_VMI) && defined(CONFIG_X86_32)
-	/*
-	 * Must be before kernel pagetables are setup
-	 * or fixmap area is touched.
-	 */
-	vmi_init();
-#endif
+	/* Must be before kernel pagetables are setup */
+	vmi_activate();
 
 	/* after early param, so could get panic from serial */
 	reserve_early_setup_data();
arch/x86/kernel/smpboot.c +0 -2
index 7b10933..f71f96f 100644
@@ -294,9 +294,7 @@ static void __cpuinit start_secondary(void *unused)
 	 * fragile that we want to limit the things done here to the
 	 * most necessary things.
 	 */
-#ifdef CONFIG_VMI
 	vmi_bringup();
-#endif
 	cpu_init();
 	preempt_disable();
 	smp_callin();
arch/x86/kernel/vmi_32.c +11 -5
index 8b6c393..22fd657 100644
@@ -960,8 +960,6 @@ static inline int __init activate_vmi(void)
 
 void __init vmi_init(void)
 {
-	unsigned long flags;
-
 	if (!vmi_rom)
 		probe_vmi_rom();
 	else
@@ -973,13 +971,21 @@ void __init vmi_init(void)
 
 	reserve_top_address(-vmi_rom->virtual_top);
 
-	local_irq_save(flags);
-	activate_vmi();
-
 #ifdef CONFIG_X86_IO_APIC
 	/* This is virtual hardware; timer routing is wired correctly */
 	no_timer_check = 1;
 #endif
+}
+
+void vmi_activate(void)
+{
+	unsigned long flags;
+
+	if (!vmi_rom)
+		return;
+
+	local_irq_save(flags);
+	activate_vmi();
 	local_irq_restore(flags & X86_EFLAGS_IF);
 }
 
#12 lore raw

Re: [PATCH] Fix VMI crash on boot in 2.6.27, 2.6.28 kernels

Quoted 33 lines · Zachary Amsden
Quoted 8 lines · Greg KH

arch/x86/include/asm/vmi.h struct pci_header { · line 234

Quoted 4 lines
arch/x86/include/asm/vmi.h +4 -0
+#ifdef CONFIG_VMI
+	/* VMI may relocate the fixmap; do this before touching ioremap area */
+	vmi_init();
+#endif

Shouldn't the #ifdef not be needed here if the .h files are set up properly for the vmi_init prototype? Please try to keep them out of .c files wherever possible.

Yes, they should. Judging by setup.c though, you would think the opposite... in any case I fixed it. Please apply - and yes, I tested compile both ways.

VMI initialiation can relocate the fixmap, causing early_ioremap to malfunction if it is initialized before the relocation. To fix this, VMI activation is split into two phases; the detection, which must happen before setting up ioremap, and the activation, which must happen after parsing early boot parameters.

This fixes a crash on boot when VMI is enabled under VMware.

arch/x86/include/asm/vmi.h +0 -0
index b7c0dea..128958a 100644
@@ -223,9 +223,15 @@ struct pci_header {
 } __attribute__((packed));
 
 /* Function prototypes for bootstrapping */

arch/x86/include/asm/vmi.h struct pci_header { · line 234

Quoted 10 lines
arch/x86/include/asm/vmi.h +7 -1
+#ifdef CONFIG_VMI
 extern void vmi_init(void);
+extern void vmi_activate(void);
 extern void vmi_bringup(void);
-extern void vmi_apply_boot_page_allocations(void);
+#else
+#define vmi_init()
+#define vmi_activate()
+#define vmi_bringup()
+#endif

static inline please, don't use #defines for function prototypes, it's not nice. See Andrew's previous rants about this for details :)

thanks,

greg k-h

#13 lore raw

Re: [PATCH] Fix VMI crash on boot in 2.6.27, 2.6.28 kernels

Quoted 8 lines · Greg KH

arch/x86/include/asm/vmi.h struct pci_header { · line 234

Quoted 5 lines
arch/x86/include/asm/vmi.h +5 -0
+#else
+#define vmi_init()
+#define vmi_activate()
+#define vmi_bringup()
+#endif

static inline please, don't use #defines for function prototypes, it's not nice. See Andrew's previous rants about this for details :)

And if it is not possible, technically, for whatever reason, the proper forms look like:

#define foo() ((void)0)
#define bar(x) ((void)(x))
#define baz(x,y) ((void)((x),(y)))

... which preserve side effects, even if they don't guarantee type safety.

-hpa

#14 lore raw

Re: [PATCH] Fix VMI crash on boot in 2.6.27, 2.6.28 kernels

In reply to Greg KH
Quoted 3 lines · Greg KH

static inline please, don't use #defines for function prototypes, it's not nice. See Andrew's previous rants about this for details :)

Fixed, thanks.

VMI initialiation can relocate the fixmap, causing early_ioremap to malfunction if it is initialized before the relocation. To fix this, VMI activation is split into two phases; the detection, which must happen before setting up ioremap, and the activation, which must happen after parsing early boot parameters.

This fixes a crash on boot when VMI is enabled under VMware.

arch/x86/include/asm/vmi.h +7 -1
index b7c0dea..61e08c0 100644
@@ -223,9 +223,15 @@ struct pci_header {
 } __attribute__((packed));
 
 /* Function prototypes for bootstrapping */
+#ifdef CONFIG_VMI
 extern void vmi_init(void);
+extern void vmi_activate(void);
 extern void vmi_bringup(void);
-extern void vmi_apply_boot_page_allocations(void);
+#else
+static inline void vmi_init(void) {}
+static inline void vmi_activate(void) {}
+static inline void vmi_bringup(void) {}
+#endif
 
 /* State needed to start an application processor in an SMP system. */
 struct vmi_ap_state {
arch/x86/kernel/setup.c +5 -7
index 9d5674f..bdec76e 100644
@@ -794,6 +794,9 @@ void __init setup_arch(char **cmdline_p)
 	printk(KERN_INFO "Command line: %s\n", boot_command_line);
 #endif
 
+	/* VMI may relocate the fixmap; do this before touching ioremap area */
+	vmi_init();
+
 	early_cpu_init();
 	early_ioremap_init();
 
@@ -880,13 +883,8 @@ void __init setup_arch(char **cmdline_p)
 	check_efer();
 #endif
 
-#if defined(CONFIG_VMI) && defined(CONFIG_X86_32)
-	/*
-	 * Must be before kernel pagetables are setup
-	 * or fixmap area is touched.
-	 */
-	vmi_init();
-#endif
+	/* Must be before kernel pagetables are setup */
+	vmi_activate();
 
 	/* after early param, so could get panic from serial */
 	reserve_early_setup_data();
arch/x86/kernel/smpboot.c +0 -2
index 7b10933..f71f96f 100644
@@ -294,9 +294,7 @@ static void __cpuinit start_secondary(void *unused)
 	 * fragile that we want to limit the things done here to the
 	 * most necessary things.
 	 */
-#ifdef CONFIG_VMI
 	vmi_bringup();
-#endif
 	cpu_init();
 	preempt_disable();
 	smp_callin();
arch/x86/kernel/vmi_32.c +11 -5
index 8b6c393..22fd657 100644
@@ -960,8 +960,6 @@ static inline int __init activate_vmi(void)
 
 void __init vmi_init(void)
 {
-	unsigned long flags;
-
 	if (!vmi_rom)
 		probe_vmi_rom();
 	else
@@ -973,13 +971,21 @@ void __init vmi_init(void)
 
 	reserve_top_address(-vmi_rom->virtual_top);
 
-	local_irq_save(flags);
-	activate_vmi();
-
 #ifdef CONFIG_X86_IO_APIC
 	/* This is virtual hardware; timer routing is wired correctly */
 	no_timer_check = 1;
 #endif
+}
+
+void vmi_activate(void)
+{
+	unsigned long flags;
+
+	if (!vmi_rom)
+		return;
+
+	local_irq_save(flags);
+	activate_vmi();
 	local_irq_restore(flags & X86_EFLAGS_IF);
 }
 
#15 lore raw
In reply to Yinghai Lu
Quoted 2 lines · Yinghai Lu

it seems still have some problem.
you moved reserve_top_address before parse_parameter...

I really only care about resetting fixmap_top. Adding more vmalloc space to accomodate what we stole is just being nice... in practice this should not be a problem.

Quoted 3 lines

__VMALLOC_RESERVE will be overwriten by vmalloc=...

you may need to split reserve_top_address() to two functions...

For now, misuse vmalloc=XXX at your own risk - why the need for it anyway?

I agree, it should be cleaned up, as HPA suggests, we should move the fixmap to a fixed place anyways, and have vmalloc area from the top of linear memory down to the end of the physical memory mapping. But there really isn't time to do anything better for 2.6.28.

Zach