Skip to contentSkip to conversation

x86: disable virt on kdump and emergency_restart

PATCH In review Eduardo Habkost 5 participants 37 messages Nov 4, 2008 – Nov 9, 2008

Subsystems (5)
x86 KVM KVM x86 Live update +1 more subsystems
lore
Share & tools
lore
#0 lore raw

cover x86: disable virt on kdump and emergency_restart

Hi,

This is a new version of the series to disabling virtualization on kdump, now extended to do the same tricks on emergency_restart() if needed.

This series is against linux-next-20081103.

Patches 01-07 simply move the non-kdump-specific parts
of nmi_shootdown_cpus() to reboot.c, so it can be used by emergency_restart(). They should be a no-op in relation to existing code.

Patches 08 adds the virt_disable function registering interface, like the previous series.

Patch 09 hooks emergency_virt_disable() into kdump crash_shutdown code.

Patch 10 hooks emergency_virt_disable() into emergency_restart() using nmi_shootdown_cpus().

Patches 11-15 change KVM so that it registers a virt_disable function when loading.

Finally, patch 16 restore the previous reboot=kbd default.

Signature · 6 lines
Eduardo

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#1 lore raw

PATCH 1/16 x86 kdump: Extract kdump-specific code from crash_nmi_callback()

The NMI CPU-halting code will be used on non-kdump cases, also (e.g. emergency_reboot when virtualization is enabled).

---
 arch/x86/kernel/crash.c |   31 ++++++++++++++++++++-----------
 1 files changed, 20 insertions(+), 11 deletions(-)
arch/x86/kernel/crash.c +20 -11
index 81e01f7..e8c58aa 100644
@@ -35,19 +35,34 @@ static int crashing_cpu;
 #if defined(CONFIG_SMP) && defined(CONFIG_X86_LOCAL_APIC)
 static atomic_t waiting_for_crash_ipi;
 
-static int crash_nmi_callback(struct notifier_block *self,
-			unsigned long val, void *data)
+static void kdump_nmi_callback(int cpu, struct die_args *args)
 {
 	struct pt_regs *regs;
 #ifdef CONFIG_X86_32
 	struct pt_regs fixed_regs;
 #endif
+
+	regs = args->regs;
+
+#ifdef CONFIG_X86_32
+	if (!user_mode_vm(regs)) {
+		crash_fixup_ss_esp(&fixed_regs, regs);
+		regs = &fixed_regs;
+	}
+#endif
+	crash_save_cpu(regs, cpu);
+
+	disable_local_APIC();
+}
+
+static int crash_nmi_callback(struct notifier_block *self,
+			unsigned long val, void *data)
+{
 	int cpu;
 
 	if (val != DIE_NMI_IPI)
 		return NOTIFY_OK;
 
-	regs = ((struct die_args *)data)->regs;
 	cpu = raw_smp_processor_id();
 
 	/* Don't do anything if this handler is invoked on crashing cpu.
@@ -58,14 +73,8 @@ static int crash_nmi_callback(struct notifier_block *self,
 		return NOTIFY_STOP;
 	local_irq_disable();
 
-#ifdef CONFIG_X86_32
-	if (!user_mode_vm(regs)) {
-		crash_fixup_ss_esp(&fixed_regs, regs);
-		regs = &fixed_regs;
-	}
-#endif
-	crash_save_cpu(regs, cpu);
-	disable_local_APIC();
+	kdump_nmi_callback(cpu, (struct die_args*)data);
+
 	atomic_dec(&waiting_for_crash_ipi);
 	/* Assume hlt works */
 	halt();
Signature · 7 lines
1.5.5.GIT


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#2 lore raw

PATCH 2/16 x86 kdump: Move crashing_cpu assignment to nmi_shootdown_cpus()

In reply to Eduardo Habkost

This variable will be moved to non-kdump-specific code.

---
 arch/x86/kernel/crash.c |    8 +++++---
 1 files changed, 5 insertions(+), 3 deletions(-)
arch/x86/kernel/crash.c +5 -3
index e8c58aa..0e77fe9 100644
@@ -29,10 +29,11 @@
 
 #include <mach_ipi.h>
 
+#if defined(CONFIG_SMP) && defined(CONFIG_X86_LOCAL_APIC)
+
 /* This keeps a track of which one is crashing cpu. */
 static int crashing_cpu;
 
-#if defined(CONFIG_SMP) && defined(CONFIG_X86_LOCAL_APIC)
 static atomic_t waiting_for_crash_ipi;
 
 static void kdump_nmi_callback(int cpu, struct die_args *args)
@@ -97,6 +98,9 @@ static void nmi_shootdown_cpus(void)
 {
 	unsigned long msecs;
 
+	/* Make a note of crashing cpu. Will be used in NMI callback.*/
+	crashing_cpu = safe_smp_processor_id();
+
 	atomic_set(&waiting_for_crash_ipi, num_online_cpus() - 1);
 	/* Would it be better to replace the trap vector here? */
 	if (register_die_notifier(&crash_nmi_nb))
@@ -137,8 +141,6 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
 	/* The kernel is broken so disable interrupts */
 	local_irq_disable();
 
-	/* Make a note of crashing cpu. Will be used in NMI callback.*/
-	crashing_cpu = safe_smp_processor_id();
 	nmi_shootdown_cpus();
 	lapic_shutdown();
 #if defined(CONFIG_X86_IO_APIC)
Signature · 7 lines
1.5.5.GIT


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#3 lore raw

PATCH 3/16 x86 kdump: Create kdump_nmi_shootdown_cpus()

In reply to Eduardo Habkost

For the kdump-specific code that was living on nmi_shootdown_cpus().

---
 arch/x86/kernel/crash.c |   11 +++++++++--
 1 files changed, 9 insertions(+), 2 deletions(-)
arch/x86/kernel/crash.c +9 -2
index 0e77fe9..7f6e2b2 100644
@@ -119,10 +119,17 @@ static void nmi_shootdown_cpus(void)
 	}
 
 	/* Leave the nmi callback set */
+}
+
+static void kdump_nmi_shootdown_cpus(void)
+{
+	nmi_shootdown_cpus();
+
 	disable_local_APIC();
 }
+
 #else
-static void nmi_shootdown_cpus(void)
+static void kdump_nmi_shootdown_cpus(void)
 {
 	/* There are no cpus to shootdown */
 }
@@ -141,7 +148,7 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
 	/* The kernel is broken so disable interrupts */
 	local_irq_disable();
 
-	nmi_shootdown_cpus();
+	kdump_nmi_shootdown_cpus();
 	lapic_shutdown();
 #if defined(CONFIG_X86_IO_APIC)
 	disable_IO_APIC();
Signature · 7 lines
1.5.5.GIT


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#4 lore raw

PATCH 4/16 x86 kdump: Make kdump_nmi_callback() a function ptr on crash_nmi_callback()

In reply to Eduardo Habkost

The reboot code will use a different function on crash_nmi_callback(). Adding a function pointer parameter to nmi_shootdown_cpus() for that.

---
 arch/x86/kernel/crash.c |   11 ++++++++---
 1 files changed, 8 insertions(+), 3 deletions(-)
arch/x86/kernel/crash.c +8 -3
index 7f6e2b2..ffbe54b 100644
@@ -29,10 +29,13 @@
 
 #include <mach_ipi.h>
 
+typedef void (*nmi_shootdown_cb)(int, struct die_args*);
+
 #if defined(CONFIG_SMP) && defined(CONFIG_X86_LOCAL_APIC)
 
 /* This keeps a track of which one is crashing cpu. */
 static int crashing_cpu;
+static nmi_shootdown_cb shootdown_callback;
 
 static atomic_t waiting_for_crash_ipi;
 
@@ -74,7 +77,7 @@ static int crash_nmi_callback(struct notifier_block *self,
 		return NOTIFY_STOP;
 	local_irq_disable();
 
-	kdump_nmi_callback(cpu, (struct die_args*)data);
+	shootdown_callback(cpu, (struct die_args*)data);
 
 	atomic_dec(&waiting_for_crash_ipi);
 	/* Assume hlt works */
@@ -94,13 +97,15 @@ static struct notifier_block crash_nmi_nb = {
 	.notifier_call = crash_nmi_callback,
 };
 
-static void nmi_shootdown_cpus(void)
+static void nmi_shootdown_cpus(nmi_shootdown_cb callback)
 {
 	unsigned long msecs;
 
 	/* Make a note of crashing cpu. Will be used in NMI callback.*/
 	crashing_cpu = safe_smp_processor_id();
 
+	shootdown_callback = callback;
+
 	atomic_set(&waiting_for_crash_ipi, num_online_cpus() - 1);
 	/* Would it be better to replace the trap vector here? */
 	if (register_die_notifier(&crash_nmi_nb))
@@ -123,7 +128,7 @@ static void nmi_shootdown_cpus(void)
 
 static void kdump_nmi_shootdown_cpus(void)
 {
-	nmi_shootdown_cpus();
+	nmi_shootdown_cpus(kdump_nmi_callback);
 
 	disable_local_APIC();
 }
Signature · 7 lines
1.5.5.GIT


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#5 lore raw

PATCH 5/16 x86 kdump: Make nmi_shootdown_cpus() non-static

In reply to Eduardo Habkost

Add prototype to asm/reboot.h.

---
 arch/x86/include/asm/reboot.h |    5 +++++
 arch/x86/kernel/crash.c       |    3 +--
 2 files changed, 6 insertions(+), 2 deletions(-)
arch/x86/include/asm/reboot.h +5 -0
index df77103..562d4fd 100644
@@ -1,6 +1,8 @@
 #ifndef _ASM_X86_REBOOT_H
 #define _ASM_X86_REBOOT_H
 
+#include <linux/kdebug.h>
+
 struct pt_regs;
 
 struct machine_ops {
@@ -18,4 +20,7 @@ void native_machine_crash_shutdown(struct pt_regs *regs);
 void native_machine_shutdown(void);
 void machine_real_restart(const unsigned char *code, int length);
 
+typedef void (*nmi_shootdown_cb)(int, struct die_args*);
+void nmi_shootdown_cpus(nmi_shootdown_cb callback);
+
 #endif /* _ASM_X86_REBOOT_H */
arch/x86/kernel/crash.c +1 -2
index ffbe54b..84d6338 100644
@@ -29,7 +29,6 @@
 
 #include <mach_ipi.h>
 
-typedef void (*nmi_shootdown_cb)(int, struct die_args*);
 
 #if defined(CONFIG_SMP) && defined(CONFIG_X86_LOCAL_APIC)
 
@@ -97,7 +96,7 @@ static struct notifier_block crash_nmi_nb = {
 	.notifier_call = crash_nmi_callback,
 };
 
-static void nmi_shootdown_cpus(nmi_shootdown_cb callback)
+void nmi_shootdown_cpus(nmi_shootdown_cb callback)
 {
 	unsigned long msecs;
 
Signature · 7 lines
1.5.5.GIT


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#6 lore raw

PATCH 6/16 x86: Move nmi_shootdown_cpus() to reboot.c

In reply to Eduardo Habkost

Now nmi_shootdown_cpus() is ready to be used by non-kdump code also. Move it to reboot.c.

---
 arch/x86/kernel/crash.c  |   73 ------------------------------------------
 arch/x86/kernel/reboot.c |   79 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 79 insertions(+), 73 deletions(-)
arch/x86/kernel/crash.c +0 -73
index 84d6338..d84a852 100644
@@ -32,12 +32,6 @@
 
 #if defined(CONFIG_SMP) && defined(CONFIG_X86_LOCAL_APIC)
 
-/* This keeps a track of which one is crashing cpu. */
-static int crashing_cpu;
-static nmi_shootdown_cb shootdown_callback;
-
-static atomic_t waiting_for_crash_ipi;
-
 static void kdump_nmi_callback(int cpu, struct die_args *args)
 {
 	struct pt_regs *regs;
@@ -58,73 +52,6 @@ static void kdump_nmi_callback(int cpu, struct die_args *args)
 	disable_local_APIC();
 }
 
-static int crash_nmi_callback(struct notifier_block *self,
-			unsigned long val, void *data)
-{
-	int cpu;
-
-	if (val != DIE_NMI_IPI)
-		return NOTIFY_OK;
-
-	cpu = raw_smp_processor_id();
-
-	/* Don't do anything if this handler is invoked on crashing cpu.
-	 * Otherwise, system will completely hang. Crashing cpu can get
-	 * an NMI if system was initially booted with nmi_watchdog parameter.
-	 */
-	if (cpu == crashing_cpu)
-		return NOTIFY_STOP;
-	local_irq_disable();
-
-	shootdown_callback(cpu, (struct die_args*)data);
-
-	atomic_dec(&waiting_for_crash_ipi);
-	/* Assume hlt works */
-	halt();
-	for (;;)
-		cpu_relax();
-
-	return 1;
-}
-
-static void smp_send_nmi_allbutself(void)
-{
-	send_IPI_allbutself(NMI_VECTOR);
-}
-
-static struct notifier_block crash_nmi_nb = {
-	.notifier_call = crash_nmi_callback,
-};
-
-void nmi_shootdown_cpus(nmi_shootdown_cb callback)
-{
-	unsigned long msecs;
-
-	/* Make a note of crashing cpu. Will be used in NMI callback.*/
-	crashing_cpu = safe_smp_processor_id();
-
-	shootdown_callback = callback;
-
-	atomic_set(&waiting_for_crash_ipi, num_online_cpus() - 1);
-	/* Would it be better to replace the trap vector here? */
-	if (register_die_notifier(&crash_nmi_nb))
-		return;		/* return what? */
-	/* Ensure the new callback function is set before sending
-	 * out the NMI
-	 */
-	wmb();
-
-	smp_send_nmi_allbutself();
-
-	msecs = 1000; /* Wait at most a second for the other cpus to stop */
-	while ((atomic_read(&waiting_for_crash_ipi) > 0) && msecs) {
-		mdelay(1);
-		msecs--;
-	}
-
-	/* Leave the nmi callback set */
-}
-
 static void kdump_nmi_shootdown_cpus(void)
 {
 	nmi_shootdown_cpus(kdump_nmi_callback);
arch/x86/kernel/reboot.c +79 -0

88 lines not on this page · raw message

Signature · 7 lines
1.5.5.GIT


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#7 lore raw

PATCH 7/16 x86: Make nmi_shootdown_cpus() available on !SMP and !X86_LOCAL_APIC

In reply to Eduardo Habkost

The X86_LOCAL_APIC #ifdef was for kdump. For !SMP, the function simply does nothing.

---
 arch/x86/kernel/reboot.c |   13 ++++++++++++-
 1 files changed, 12 insertions(+), 1 deletions(-)
arch/x86/kernel/reboot.c +12 -1
index b65fc86..80d5e9c 100644
@@ -522,7 +522,7 @@ void machine_crash_shutdown(struct pt_regs *regs)
 #endif
 
 
-#if defined(CONFIG_SMP) && defined(CONFIG_X86_LOCAL_APIC)
+#if defined(CONFIG_SMP)
 
 /* This keeps a track of which one is crashing cpu. */
 static int crashing_cpu;
@@ -568,6 +568,12 @@ static struct notifier_block crash_nmi_nb = {
 	.notifier_call = crash_nmi_callback,
 };
 
+/* Halt all other CPUs, calling the specified function on each of them
+ *
+ * This function can be used to halt all other CPUs on crash
+ * or emergency reboot time. The function passed as parameter
+ * will be called inside a NMI handler on all CPUs.
+ */
 void nmi_shootdown_cpus(nmi_shootdown_cb callback)
 {
 	unsigned long msecs;
@@ -596,4 +602,9 @@ void nmi_shootdown_cpus(nmi_shootdown_cb callback)
 
 	/* Leave the nmi callback set */
 }
+#else /* !CONFIG_SMP */
+void nmi_shootdown_cpus(nmi_shootdown_cb callback)
+{
+	/* No other CPUs to shoot down */
+}
 #endif
Signature · 7 lines
1.5.5.GIT


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#8 lore raw

Re: [PATCH 07/16] x86: Make nmi_shootdown_cpus() available on !SMP and !X86_LOCAL_APIC

Quoted 3 lines · Eduardo Habkost

The X86_LOCAL_APIC #ifdef was for kdump. For !SMP, the function simply does nothing.

The bit below is also needed. It can be added to the series right before patch 07/16.

After getting some review, I will resubmit the series with it, and with a few style fixes suggested by checkpatch.

---
From: Eduardo Habkost <[email protected]>
Subject: [PATCH 06.1/16] x86: Disable IRQs before doing anything on nmi_shootdown_cpus()

We need to know on which CPU we are running on, and we don't want to be preempted while doing this.

---
 arch/x86/kernel/reboot.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
arch/x86/kernel/reboot.c +1 -0
index 23a9d78..407106e 100644
@@ -577,6 +577,7 @@ static struct notifier_block crash_nmi_nb = {
 void nmi_shootdown_cpus(nmi_shootdown_cb callback)
 {
 	unsigned long msecs;
+	local_irq_disable();
 
 	/* Make a note of crashing cpu. Will be used in NMI callback.*/
 	crashing_cpu = safe_smp_processor_id();
Signature · 10 lines
1.5.5.GIT


-- 
Eduardo

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#9 lore raw

PATCH 8/16 x86: Emergency virtualization disable function

In reply to Eduardo Habkost

This patch adds an interface to set a function that can be used to disable virtualization extensions on the CPU on emergency cases, such as on kdump or emergency reboot.

The function will be set by code that enables virtualization extensions on the CPUs (i.e. KVM), and should do the very least to disable virt extensions on the CPU where it is being called.

The functions that set the function pointer uses RCU synchronization, just in case the crash NMI is triggered while KVM is unloading.

We can't just use the same notifiers used at reboot time (that are used by non-crash-dump kexec), because at crash time some CPUs may have IRQs disabled, so we can't use IPIs. The crash shutdown code use NMIs to tell the other CPUs to be halted, so the notifier call will be hooked into the CPU halting code that is on the crash shutdown NMI handler.

[v2: drop 'unsigned int cpu' arg from function]
[v3: make emergency_virt_disable() non-static]
[v4: add a config option for it: CPU_VIRT_EXTENSIONS]
[v5: add has_virt_extensions() function]
[v6: don't define the registering functions if CPU_VIRT_EXTENSIONS is
not enabled]

---
 arch/x86/include/asm/virtext.h |   29 +++++++++++++
 arch/x86/kernel/Makefile       |    1 +
 arch/x86/kernel/virtext.c      |   89 ++++++++++++++++++++++++++++++++++++++++
 arch/x86/kvm/Kconfig           |    5 ++
 4 files changed, 124 insertions(+), 0 deletions(-)
 create mode 100644 arch/x86/include/asm/virtext.h
 create mode 100644 arch/x86/kernel/virtext.c
arch/x86/include/asm/virtext.h new file +29 -0
new file mode 100644
index 0000000..72b7caa
@@ -0,0 +1,29 @@
+/* CPU virtualization extensions handling
+ */
+#ifndef _ASM_X86_VIRTEX_H
+#define _ASM_X86_VIRTEX_H
+
+#ifdef CONFIG_CPU_VIRT_EXTENSIONS
+
+int set_virt_disable_func(void (*fn)(void));
+void clear_virt_disable_func(void);
+void emergency_virt_disable(void);
+int has_virt_extensions(void);
+
+
+#else /* !CONFIG_CPU_VIRT_EXTENSIONS */
+
+static inline
+void emergency_virt_disable(void)
+{
+}
+
+static inline
+int has_virt_extensions(void)
+{
+	return 0;
+}
+
+#endif /* CONFIG_CPU_VIRT_EXTENSIONS */
+
+#endif /* _ASM_X86_VIRTEX_H */
arch/x86/kernel/Makefile +1 -0
index 58814cc..84a0e23 100644
@@ -68,6 +68,7 @@ obj-$(CONFIG_X86_REBOOTFIXUPS)	+= reboot_fixups_32.o
 obj-$(CONFIG_DYNAMIC_FTRACE)	+= ftrace.o
 obj-$(CONFIG_KEXEC)		+= machine_kexec_$(BITS).o
 obj-$(CONFIG_KEXEC)		+= relocate_kernel_$(BITS).o crash.o
+obj-$(CONFIG_CPU_VIRT_EXTENSIONS) += virtext.o
 obj-$(CONFIG_CRASH_DUMP)	+= crash_dump_$(BITS).o
 obj-$(CONFIG_X86_NUMAQ)		+= numaq_32.o
 obj-$(CONFIG_X86_ES7000)	+= es7000_32.o
arch/x86/kernel/virtext.c new file +89 -0

89 lines not on this page · raw message

arch/x86/kvm/Kconfig +5 -0
index ce3251c..69373cc 100644
@@ -15,6 +15,10 @@ menuconfig VIRTUALIZATION
 
 	  If you say N, all options in this submenu will be skipped and disabled.
 
+# Core code for handling CPU virt extensions
+config CPU_VIRT_EXTENSIONS
+	bool
+
 if VIRTUALIZATION
 
 config KVM
@@ -23,6 +27,7 @@ config KVM
 	select PREEMPT_NOTIFIERS
 	select MMU_NOTIFIER
 	select ANON_INODES
+	select CPU_VIRT_EXTENSIONS
 	---help---
 	  Support hosting fully virtualized guest machines using hardware
 	  virtualization extensions.  You will need a fairly recent
Signature · 7 lines
1.5.5.GIT


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#10 lore raw

Re: [PATCH 08/16] x86: Emergency virtualization disable function

Quoted 4 lines · Eduardo Habkost
arch/x86/kernel/virtext.c new file +4 -0
+int set_virt_disable_func(void (*fn)(void))
+{
+	int r = 0;
+

arch/x86/kernel/virtext.c line 45

Quoted 10 lines
arch/x86/kernel/virtext.c new file +10 -0
+	spin_lock(&virt_disable_lock);
+	if (!virt_disable_fn)
+		rcu_assign_pointer(virt_disable_fn, fn);
+	else
+		r = -EEXIST;
+	spin_unlock(&virt_disable_lock);
+
+	return r;
+}
+EXPORT_SYMBOL(set_virt_disable_func);

EXPORT_SYMBOL_GPL?

Quoted 1 line

+EXPORT_SYMBOL(clear_virt_disable_func);

EXPORT_SYMBOL_GPL?

We are talking a core internal api that should not even be exported if KVM is compiled into the kernel.

I have had to tell people NO too many times by that wanted to shove code on the kexec on panic path that had no business there. I do not want to give the least little impression that this is an ok hook for the to use. The very specific name helps in that regard thank you for that. Having the symbol exported GPL would help even more.

Overall I think the code is just barely ok.

I don't like the fact that to run 2-3 instructions per cpu we are two function pointers deep. It feels like we have an excess of abstraction here on the kvm side.

Is it possible to have the individual kvm modules call set_virt_disable_func and clear_virt_disable_func? Instead of going through the x86_kvm_ops?

It really feels like we have an excess of abstraction here.

Eric

#11 lore raw

Re: [PATCH 08/16] x86: Emergency virtualization disable function

Quoted 25 lines · Eric W. Biederman
Quoted 4 lines · Eduardo Habkost
arch/x86/kernel/virtext.c new file +4 -0
+int set_virt_disable_func(void (*fn)(void))
+{
+	int r = 0;
+

arch/x86/kernel/virtext.c line 45

Quoted 10 lines
arch/x86/kernel/virtext.c new file +10 -0
+	spin_lock(&virt_disable_lock);
+	if (!virt_disable_fn)
+		rcu_assign_pointer(virt_disable_fn, fn);
+	else
+		r = -EEXIST;
+	spin_unlock(&virt_disable_lock);
+
+	return r;
+}
+EXPORT_SYMBOL(set_virt_disable_func);

EXPORT_SYMBOL_GPL?

Quoted 1 line

+EXPORT_SYMBOL(clear_virt_disable_func);

EXPORT_SYMBOL_GPL?

We are talking a core internal api that should not even be exported if KVM is compiled into the kernel.

Quoted 7 lines

I have had to tell people NO too many times by that wanted to shove code on the kexec on panic path that had no business there. I do not want to give the least little impression that this is an ok hook for the to use. The very specific name helps in that regard thank you for that. Having the symbol exported GPL would help even more.

Agreed. I will change that if nobody else objects.

Quoted 7 lines

Overall I think the code is just barely ok.

I don't like the fact that to run 2-3 instructions per cpu we are two function pointers deep. It feels like we have an excess of abstraction here on the kvm side.

Quoted 5 lines

Is it possible to have the individual kvm modules call set_virt_disable_func and clear_virt_disable_func? Instead of going through the x86_kvm_ops?

It really feels like we have an excess of abstraction here.

We could move the set_virt_disable_func() calls to vmx.c and svm.c (on hardware_setup/hardware_unsetup). One could argue that it is sort of a coincidence that we need the code for both vmx and svm.

Avi, what do you think?

Signature · 1 line
Eduardo
#12 lore raw

Re: [PATCH 08/16] x86: Emergency virtualization disable function

Quoted 4 lines · Eduardo Habkost

We could move the set_virt_disable_func() calls to vmx.c and svm.c (on hardware_setup/hardware_unsetup). One could argue that it is sort of a coincidence that we need the code for both vmx and svm.

I don't share this fear of function calls, but perhaps that's due to lack of experience with kdump.

If you want to be extra simple and safe, remove kvm from the equation. Make the disabling code part of kdump/emergency_restart and only rely on the convention that cr3.vmxe == vmxon.

That has the advantage of working with other virtualization systems as well.

Signature · 7 lines
error compiling committee.c: too many arguments to function


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#13 lore raw

Re: [PATCH 08/16] x86: Emergency virtualization disable function

Quoted 8 lines · Avi Kivity
Quoted 4 lines · Eduardo Habkost

We could move the set_virt_disable_func() calls to vmx.c and svm.c (on hardware_setup/hardware_unsetup). One could argue that it is sort of a coincidence that we need the code for both vmx and svm.

I don't share this fear of function calls, but perhaps that's due to lack of experience with kdump.

Mostly I'm just afraid of extra complexity.

Quoted 3 lines

If you want to be extra simple and safe, remove kvm from the equation. Make the disabling code part of kdump/emergency_restart and only rely on the convention that cr3.vmxe == vmxon.

Convention?

Quoted 1 line

That has the advantage of working with other virtualization systems as well.

Sounds good.

Eric

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#14 lore raw

Re: [PATCH 08/16] x86: Emergency virtualization disable function

Quoted 7 lines · Eric W. Biederman
Quoted 5 lines

If you want to be extra simple and safe, remove kvm from the equation. Make the disabling code part of kdump/emergency_restart and only rely on the convention that cr3.vmxe == vmxon.

Convention?

There is a de-facto convention supported by at least vmware and kvm. If cr4.vmxe is 1, then we are in vmx operation. If cr4.vmxe is 0, then we are not in vmx operation. This allows us to determine whether we need to execute vmxoff without any APIs.

Signature · 1 line
error compiling committee.c: too many arguments to function
#15 lore raw

Re: [PATCH 08/16] x86: Emergency virtualization disable function

Quoted 6 lines · Avi Kivity
Quoted 5 lines · Eric W. Biederman
Quoted 5 lines

If you want to be extra simple and safe, remove kvm from the equation. Make the disabling code part of kdump/emergency_restart and only rely on the convention that cr3.vmxe == vmxon.

Quoted 7 lines
Quoted 2 lines

Convention?

There is a de-facto convention supported by at least vmware and kvm. If cr4.vmxe is 1, then we are in vmx operation. If cr4.vmxe is 0, then we are not in vmx operation. This allows us to determine whether we need to execute vmxoff without any APIs.

I am just worried about the probing needed to make sure CR4 is available (and that the CR4.VMXE bit means what we expect it to mean), before we try to read it and check VMXE. The same for SVM and the MSRs we need to touch to disable SVM.

I prefer to reuse code that already exists on KVM and is working than adding new probing code that I won't be able to test on all hardware configurations.

Signature · 1 line
Eduardo
#16 lore raw

Re: [PATCH 08/16] x86: Emergency virtualization disable function

Quoted 12 lines · Eduardo Habkost
Quoted 10 lines · Avi Kivity
Quoted 8 lines · Eric W. Biederman
Quoted 5 lines

If you want to be extra simple and safe, remove kvm from the equation. Make the disabling code part of kdump/emergency_restart and only rely on the convention that cr3.vmxe == vmxon.

Convention?

Quoted 8 lines
Quoted 5 lines

There is a de-facto convention supported by at least vmware and kvm. If cr4.vmxe is 1, then we are in vmx operation. If cr4.vmxe is 0, then we are not in vmx operation. This allows us to determine whether we need to execute vmxoff without any APIs.

I am just worried about the probing needed to make sure CR4 is available (and that the CR4.VMXE bit means what we expect it to mean),

That's cpu_has_kvm_support() in vmx.c. Should compile to three instructions.

Quoted 4 lines

before we
try to read it and check VMXE. The same for SVM and the MSRs we need to touch to disable SVM.

That's has_svm() in svm.c; slightly longer but not any more dangerous.

Quoted 4 lines

I prefer to reuse code that already exists on KVM and is working than adding new probing code that I won't be able to test on all hardware configurations.

We could move the code to a header file, and so compile it both for kvm and for emergency_restart/kdump.

Signature · 1 line
error compiling committee.c: too many arguments to function
#17 lore raw

PATCH 9/16 kdump: Hook emergency_virt_disable() on crash shutdown code

In reply to Eduardo Habkost
---
 arch/x86/kernel/crash.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
arch/x86/kernel/crash.c +3 -0
index d84a852..87780ba 100644
@@ -26,6 +26,7 @@
 #include <linux/kdebug.h>
 #include <asm/smp.h>
 #include <asm/reboot.h>
+#include <asm/virtext.h>
 
 #include <mach_ipi.h>
 
@@ -49,6 +50,7 @@ static void kdump_nmi_callback(int cpu, struct die_args *args)
 #endif
 	crash_save_cpu(regs, cpu);
 
+	emergency_virt_disable();
 	disable_local_APIC();
 }
 
@@ -80,6 +82,7 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
 	local_irq_disable();
 
 	kdump_nmi_shootdown_cpus();
+	emergency_virt_disable();
 	lapic_shutdown();
 #if defined(CONFIG_X86_IO_APIC)
 	disable_IO_APIC();
Signature · 7 lines
1.5.5.GIT


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#18 lore raw

Re: [PATCH 09/16] kdump: Hook emergency_virt_disable() on crash shutdown code

Quoted 26 lines · Eduardo Habkost
---
 arch/x86/kernel/crash.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
arch/x86/kernel/crash.c +2 -0
index d84a852..87780ba 100644
@@ -26,6 +26,7 @@
 #include <linux/kdebug.h>
 #include <asm/smp.h>
 #include <asm/reboot.h>
+#include <asm/virtext.h>
 
 #include <mach_ipi.h>
 
@@ -49,6 +50,7 @@ static void kdump_nmi_callback(int cpu, struct die_args *args)
 #endif
 	crash_save_cpu(regs, cpu);
 
+	emergency_virt_disable();
 	disable_local_APIC();
 }
 

arch/x86/kernel/crash.c void native_machine_crash_shutdown(struct pt_regs *regs) · line 85

Quoted 4 lines
arch/x86/kernel/crash.c +1 -0
@@ -80,6 +82,7 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
 	local_irq_disable();
 
 	kdump_nmi_shootdown_cpus();
+	emergency_virt_disable();

Hi Eduardo,

It might be a good idea to put a 2-3 line comment saying why it is imporatant to disable virtualization here before we continue.

Thanks
Vivek

#19 lore raw

Re: [PATCH 09/16] kdump: Hook emergency_virt_disable() on crash shutdown code

Quoted 29 lines · Vivek Goyal
Quoted 28 lines · Eduardo Habkost
---
 arch/x86/kernel/crash.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
arch/x86/kernel/crash.c +2 -0
index d84a852..87780ba 100644
@@ -26,6 +26,7 @@
 #include <linux/kdebug.h>
 #include <asm/smp.h>
 #include <asm/reboot.h>
+#include <asm/virtext.h>
 
 #include <mach_ipi.h>
 
@@ -49,6 +50,7 @@ static void kdump_nmi_callback(int cpu, struct die_args *args)
 #endif
 	crash_save_cpu(regs, cpu);
 
+	emergency_virt_disable();
 	disable_local_APIC();
 }
 
@@ -80,6 +82,7 @@ void native_machine_crash_shutdown(struct pt_regs *regs)
 	local_irq_disable();
 
Quoted 7 lines

arch/x86/kernel/crash.c void native_machine_crash_shutdown(struct pt_regs *regs) · line 85

Quoted 2 lines
arch/x86/kernel/crash.c +1 -0
 	kdump_nmi_shootdown_cpus();
+	emergency_virt_disable();

Hi Eduardo,

It might be a good idea to put a 2-3 line comment saying why it is imporatant to disable virtualization here before we continue.

True. Sometimes I forget that detailed commit messages don't replace good documentation on source code. I will do that.

Signature · 1 line
Eduardo
#20 lore raw

PATCH 10/16 x86: disable virtualization on all CPUs if needed, on emergency_restart

In reply to Eduardo Habkost

On emergency_restart, we may need to use NMIs to disable virtualization on all CPUs. We do that by using nmi_shootdown_cpus(), but only if a virt_disable function was set by KVM.

Finding a proper point to hook the nmi_shootdown_cpus() call wasn't really easy, as the non-emergency machine_restart() (that doesn't need the NMI tricks) uses machine_emergency_restart() directly.

The solution to make this work without adding a new function or argument to machine_ops was setting a 'reboot_emergency' flag that tells if native_machine_emergency_restart() needs to do the virt cleanup or not.

---
 arch/x86/kernel/reboot.c |   48 ++++++++++++++++++++++++++++++++++++++++++++-
 1 files changed, 46 insertions(+), 2 deletions(-)
arch/x86/kernel/reboot.c +46 -2
index 80d5e9c..932763f 100644
@@ -12,6 +12,7 @@
 #include <asm/proto.h>
 #include <asm/reboot_fixups.h>
 #include <asm/reboot.h>
+#include <asm/virtext.h>
 
 #ifdef CONFIG_X86_32
 # include <linux/dmi.h>
@@ -42,6 +43,12 @@ int reboot_force;
 static int reboot_cpu = -1;
 #endif
 
+/* This is set if we need to go through the 'emergency' path.
+ * When machine_emergency_restart() is called, we may be on
+ * an inconsistent state and won't be able to do a clean cleanup
+ */
+static int reboot_emergency;
+
 /* reboot=b[ios] | s[mp] | t[riple] | k[bd] | e[fi] [, [w]arm | [c]old]
    warm   Don't set the cold reboot flag
    cold   Set the cold reboot flag
@@ -357,6 +364,33 @@ static inline void kb_wait(void)
 	}
 }
 
+static void disable_virt_nmi(int cpu, struct die_args *args)
+{
+	emergency_virt_disable();
+}
+
+/* Use NMIs as IPIs to tell all CPUs to disable virtualization
+ */
+static void emergency_virt_disable_all(void)
+{
+	/* We will do this NMI stuff only when we may have virtualization
+	 * enabled.
+	 */
+	if (has_virt_extensions()) {
+		/* Kill the other CPUs */
+		nmi_shootdown_cpus(disable_virt_nmi);
+		/* Disable virt on this CPU */
+		emergency_virt_disable();
+	}
+
+	/* If another CPU call set_virt_disable_func()
+	 * here, we will be lost. Unless we want to add the NMI
+	 * stuff to the reboot path for non-virt users, we will
+	 * have to live with this (unlikely) possibility.
+	 */
+
+}
+
 void __attribute__((weak)) mach_reboot_fixups(void)
 {
 }
@@ -365,9 +399,13 @@ static void native_machine_emergency_restart(void)
 {
 	int i;
 
+	if (reboot_emergency)
+		emergency_virt_disable_all();
+
 	/* Tell the BIOS if we want cold or warm reboot */
 	*((unsigned short *)__va(0x472)) = reboot_mode;
 
+
 	for (;;) {
 		/* Could also try the reset bit in the Hammer NB */
 		switch (reboot_type) {
@@ -456,13 +494,19 @@ void native_machine_shutdown(void)
 #endif
 }
 
+static void __machine_emergency_restart(int emergency)
+{
+	reboot_emergency = emergency;
+	machine_ops.emergency_restart();
+}
+
 static void native_machine_restart(char *__unused)
 {
 	printk("machine restart\n");
 
 	if (!reboot_force)
 		machine_shutdown();
-	machine_emergency_restart();
+	__machine_emergency_restart(0);
 }
 
 static void native_machine_halt(void)
@@ -501,7 +545,7 @@ void machine_shutdown(void)
 
 void machine_emergency_restart(void)
 {
-	machine_ops.emergency_restart();
+	__machine_emergency_restart(1);
 }
 
 void machine_restart(char *cmd)
Signature · 7 lines
1.5.5.GIT


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#21 lore raw

PATCH 11/16 kvm_x86_ops: crash_hardware_disable() operation

In reply to Eduardo Habkost

The vmx hardware_disable() function does too much for the virt_disable crash handler, so we will have a new operation for the crash-time virt_disable case.

---
 arch/x86/include/asm/kvm_host.h |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
arch/x86/include/asm/kvm_host.h +1 -0
index 09e6c56..002023b 100644
@@ -426,6 +426,7 @@ struct kvm_x86_ops {
 	int (*disabled_by_bios)(void);             /* __init */
 	void (*hardware_enable)(void *dummy);      /* __init */
 	void (*hardware_disable)(void *dummy);
+	void (*crash_hardware_disable)(void *dummy);
 	void (*check_processor_compatibility)(void *rtn);
 	int (*hardware_setup)(void);               /* __init */
 	void (*hardware_unsetup)(void);            /* __exit */
Signature · 7 lines
1.5.5.GIT


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#22 lore raw

PATCH 12/16 kvm: svm: set crash_hardware_disable to svm_hardware_disable

In reply to Eduardo Habkost

We can use it as is, as it is a simple function.

---
 arch/x86/kvm/svm.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
arch/x86/kvm/svm.c +1 -0
index f0ad4d4..6b3a660 100644
@@ -1942,6 +1942,7 @@ static struct kvm_x86_ops svm_x86_ops = {
 	.check_processor_compatibility = svm_check_processor_compat,
 	.hardware_enable = svm_hardware_enable,
 	.hardware_disable = svm_hardware_disable,
+	.crash_hardware_disable = svm_hardware_disable,
 	.cpu_has_accelerated_tpr = svm_cpu_has_accelerated_tpr,
 
 	.vcpu_create = svm_create_vcpu,
Signature · 7 lines
1.5.5.GIT


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#23 lore raw

PATCH 13/16 kvm: vmx: crash_hardware_disable function

In reply to Eduardo Habkost

We need to first check if virtualization was enabled. We do this by checking CR4.VMXE. If it is set, run vmxoff and clear CR4.VMXE.

---
 arch/x86/kvm/vmx.c |   16 ++++++++++++++--
 1 files changed, 14 insertions(+), 2 deletions(-)
arch/x86/kvm/vmx.c +14 -2
index ece9cb7..0f4e191 100644
@@ -1091,13 +1091,24 @@ static void vmclear_local_vcpus(void)
 		__vcpu_clear(vmx);
 }
 
-static void hardware_disable(void *garbage)
+static void __hardware_disable(void)
 {
-	vmclear_local_vcpus();
 	asm volatile (__ex(ASM_VMX_VMXOFF) : : : "cc");
 	write_cr4(read_cr4() & ~X86_CR4_VMXE);
 }
 
+static void hardware_disable(void *garbage)
+{
+	vmclear_local_vcpus();
+	__hardware_disable();
+}
+
+static void crash_hardware_disable(void *garbage)
+{
+	if (read_cr4() & X86_CR4_VMXE)
+		__hardware_disable();
+}
+
 static __init int adjust_vmx_controls(u32 ctl_min, u32 ctl_opt,
 				      u32 msr, u32 *result)
 {
@@ -3589,6 +3600,7 @@ static struct kvm_x86_ops vmx_x86_ops = {
 	.check_processor_compatibility = vmx_check_processor_compat,
 	.hardware_enable = hardware_enable,
 	.hardware_disable = hardware_disable,
+	.crash_hardware_disable = crash_hardware_disable,
 	.cpu_has_accelerated_tpr = cpu_has_vmx_virtualize_apic_accesses,
 
 	.vcpu_create = vmx_create_vcpu,
Signature · 7 lines
1.5.5.GIT


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#24 lore raw

PATCH 14/16 kvmx: x86: set kvm_x86_ops earlier on kvm_arch_init()

In reply to Eduardo Habkost

Small change that will be needed when we use set_virt_disable_func() on kvm_arch_init().

---
 arch/x86/kvm/x86.c |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)
arch/x86/kvm/x86.c +5 -2
index 9a4a39c..049c6a0 100644
@@ -2603,19 +2603,22 @@ int kvm_arch_init(void *opaque)
 		goto out;
 	}
 
+	kvm_x86_ops = ops;
+
 	r = kvm_mmu_module_init();
 	if (r)
-		goto out;
+		goto out_clear_ops;
 
 	kvm_init_msr_list();
 
-	kvm_x86_ops = ops;
 	kvm_mmu_set_nonpresent_ptes(0ull, 0ull);
 	kvm_mmu_set_base_ptes(PT_PRESENT_MASK);
 	kvm_mmu_set_mask_ptes(PT_USER_MASK, PT_ACCESSED_MASK,
 			PT_DIRTY_MASK, PT64_NX_MASK, 0, 0);
 	return 0;
 
+out_clear_ops:
+	kvm_x86_ops = NULL;
 out:
 	return r;
 }
Signature · 7 lines
1.5.5.GIT


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#25 lore raw

PATCH 15/16 kvm: x86: set kdump virt_disable function on initialization

In reply to Eduardo Habkost

Finally implement the virt_disable function for kdump. It will call kvm_x86_ops->crash_hardware_disable(), that will disable virtualization extensions on the CPU if it is not disabled yet.

---
 arch/x86/kvm/x86.c |   19 ++++++++++++++++++-
 1 files changed, 18 insertions(+), 1 deletions(-)
arch/x86/kvm/x86.c +18 -1
index 049c6a0..9e61baf 100644
@@ -40,6 +40,7 @@
 #include <asm/msr.h>
 #include <asm/desc.h>
 #include <asm/mtrr.h>
+#include <asm/virtext.h>
 
 #define MAX_IO_MSRS 256
 #define CR0_RESERVED_BITS						\
@@ -2581,6 +2582,13 @@ int kvm_emulate_pio_string(struct kvm_vcpu *vcpu, struct kvm_run *run, int in,
 }
 EXPORT_SYMBOL_GPL(kvm_emulate_pio_string);
 
+/* Called at crash time, so we can disable virtualization if needed
+ */
+static void crash_hardware_disable(void)
+{
+	kvm_x86_ops->crash_hardware_disable(NULL);
+}
+
 int kvm_arch_init(void *opaque)
 {
 	int r;
@@ -2605,9 +2613,15 @@ int kvm_arch_init(void *opaque)
 
 	kvm_x86_ops = ops;
 
+	r = set_virt_disable_func(crash_hardware_disable);
+	if (r) {
+		printk(KERN_ERR "kvm: virt_disable function already set?\n");
+		goto out_clear_ops;
+	}
+
 	r = kvm_mmu_module_init();
 	if (r)
-		goto out_clear_ops;
+		goto out_clear_crash;
 
 	kvm_init_msr_list();
 
@@ -2617,6 +2631,8 @@ int kvm_arch_init(void *opaque)
 			PT_DIRTY_MASK, PT64_NX_MASK, 0, 0);
 	return 0;
 
+out_clear_crash:
+	clear_virt_disable_func();
 out_clear_ops:
 	kvm_x86_ops = NULL;
 out:
@@ -2625,6 +2641,7 @@ out:
 
 void kvm_arch_exit(void)
 {
+	clear_virt_disable_func();
 	kvm_x86_ops = NULL;
 	kvm_mmu_module_exit();
 }
Signature · 7 lines
1.5.5.GIT


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#26 lore raw

Re: [PATCH 15/16] kvm: x86: set kdump virt_disable function on initialization

Quoted 37 lines · Eduardo Habkost

Finally implement the virt_disable function for kdump. It will call kvm_x86_ops->crash_hardware_disable(), that will disable virtualization extensions on the CPU if it is not disabled yet.

---
 arch/x86/kvm/x86.c |   19 ++++++++++++++++++-
 1 files changed, 18 insertions(+), 1 deletions(-)
arch/x86/kvm/x86.c +8 -0
index 049c6a0..9e61baf 100644
@@ -40,6 +40,7 @@
 #include <asm/msr.h>
 #include <asm/desc.h>
 #include <asm/mtrr.h>
+#include <asm/virtext.h>
 
 #define MAX_IO_MSRS 256
 #define CR0_RESERVED_BITS						\
@@ -2581,6 +2582,13 @@ int kvm_emulate_pio_string(struct kvm_vcpu *vcpu, struct
 kvm_run *run, int in,
 }
 EXPORT_SYMBOL_GPL(kvm_emulate_pio_string);
 
+/* Called at crash time, so we can disable virtualization if needed
+ */
+static void crash_hardware_disable(void)
+{
+	kvm_x86_ops->crash_hardware_disable(NULL);
+}
+
 int kvm_arch_init(void *opaque)
 {
arch/x86/kvm/x86.c +0 -0
 	int r;

arch/x86/kvm/x86.c int kvm_arch_init(void *opaque) · line 2616

Quoted 4 lines
arch/x86/kvm/x86.c +1 -0
@@ -2605,9 +2613,15 @@ int kvm_arch_init(void *opaque)
 
 	kvm_x86_ops = ops;
 
+	r = set_virt_disable_func(crash_hardware_disable);

Can we make this say:
set_virt_disable_func(kvm_x86_ops->crash_hardware_disable);

So we can avoid going through 2 levels of function pointers? I find that a little scary in code that might be running at the edge of stack overflow.

Quoted 31 lines
arch/x86/kvm/x86.c +9 -1
+	if (r) {
+		printk(KERN_ERR "kvm: virt_disable function already set?\n");
+		goto out_clear_ops;
+	}
+
 	r = kvm_mmu_module_init();
 	if (r)
-		goto out_clear_ops;
+		goto out_clear_crash;
 
 	kvm_init_msr_list();
 
@@ -2617,6 +2631,8 @@ int kvm_arch_init(void *opaque)
 			PT_DIRTY_MASK, PT64_NX_MASK, 0, 0);
 	return 0;
 
+out_clear_crash:
+	clear_virt_disable_func();
 out_clear_ops:
 	kvm_x86_ops = NULL;
 out:
@@ -2625,6 +2641,7 @@ out:
 
 void kvm_arch_exit(void)
 {
+	clear_virt_disable_func();
 	kvm_x86_ops = NULL;
 	kvm_mmu_module_exit();
 }
Signature · 1 line
1.5.5.GIT
#27 lore raw

Re: [PATCH 15/16] kvm: x86: set kdump virt_disable function on initialization

Quoted 42 lines · Eric W. Biederman
Quoted 40 lines · Eduardo Habkost

Finally implement the virt_disable function for kdump. It will call kvm_x86_ops->crash_hardware_disable(), that will disable virtualization extensions on the CPU if it is not disabled yet.

---
 arch/x86/kvm/x86.c |   19 ++++++++++++++++++-
 1 files changed, 18 insertions(+), 1 deletions(-)
arch/x86/kvm/x86.c +8 -0
index 049c6a0..9e61baf 100644
@@ -40,6 +40,7 @@
 #include <asm/msr.h>
 #include <asm/desc.h>
 #include <asm/mtrr.h>
+#include <asm/virtext.h>
 
 #define MAX_IO_MSRS 256
 #define CR0_RESERVED_BITS						\
@@ -2581,6 +2582,13 @@ int kvm_emulate_pio_string(struct kvm_vcpu *vcpu, struct
 kvm_run *run, int in,
 }
 EXPORT_SYMBOL_GPL(kvm_emulate_pio_string);
 
+/* Called at crash time, so we can disable virtualization if needed
+ */
+static void crash_hardware_disable(void)
+{
+	kvm_x86_ops->crash_hardware_disable(NULL);
+}
+
 int kvm_arch_init(void *opaque)
 {
arch/x86/kvm/x86.c +0 -0
 	int r;
@@ -2605,9 +2613,15 @@ int kvm_arch_init(void *opaque)
 
 	kvm_x86_ops = ops;
 
Quoted 8 lines

arch/x86/kvm/x86.c int kvm_arch_init(void *opaque) · line 2616

Quoted 1 line
arch/x86/kvm/x86.c +1 -0
+	r = set_virt_disable_func(crash_hardware_disable);

Can we make this say:
set_virt_disable_func(kvm_x86_ops->crash_hardware_disable);

So we can avoid going through 2 levels of function pointers? I find that a little scary in code that might be running at the edge of stack overflow.

When I've checked this (on x86_64), gcc did tail recursion optimization and the call was just a jump to the function, so stack usage shouldn't be a problem.

But I am inclined to agree with you about the excess of abstraction.

Signature · 1 line
Eduardo
#28 lore raw

Re: [PATCH 15/16] kvm: x86: set kdump virt_disable function on initialization

In reply to Eric W. Biederman
Quoted 11 lines · Eric W. Biederman

arch/x86/kvm/x86.c int kvm_arch_init(void *opaque) · line 2616

Quoted 3 lines
arch/x86/kvm/x86.c +1 -0
+	r = set_virt_disable_func(crash_hardware_disable);

Can we make this say:
set_virt_disable_func(kvm_x86_ops->crash_hardware_disable);

So we can avoid going through 2 levels of function pointers? I find that a little scary in code that might be running at the edge of stack overflow.

Actually, with scheduling disabled we can overflow the stack as much as we like. It will reduce the quality of the dump, but everything ought to work.

Signature · 1 line
error compiling committee.c: too many arguments to function
#29 lore raw

PATCH 16/16 Revert "x86: default to reboot via ACPI"

In reply to Eduardo Habkost

This reverts commit c7ffa6c26277b403920e2255d10df849bd613380.

Now that we have the hooks to disable virtualization on emergency_restart(), we can get back to the BOOT_KBD reboot_type default.

---
 arch/x86/kernel/reboot.c |    6 +-----
 1 files changed, 1 insertions(+), 5 deletions(-)
arch/x86/kernel/reboot.c +1 -5
index 932763f..f05e819 100644
@@ -32,11 +32,7 @@ EXPORT_SYMBOL(pm_power_off);
 
 static const struct desc_ptr no_idt = {};
 static int reboot_mode;
-/*
- * Keyboard reset and triple fault may result in INIT, not RESET, which
- * doesn't work when we're in vmx root mode.  Try ACPI first.
- */
-enum reboot_type reboot_type = BOOT_ACPI;
+enum reboot_type reboot_type = BOOT_KBD;
 int reboot_force;
 
 #if defined(CONFIG_X86_32) && defined(CONFIG_SMP)
Signature · 7 lines
1.5.5.GIT


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#30 lore raw
In reply to Eduardo Habkost
Quoted 5 lines · Eduardo Habkost

Hi,

This is a new version of the series to disabling virtualization on kdump, now extended to do the same tricks on emergency_restart() if needed.

Looks good. If you me to push it upstream, I'll need kexec/kdump acks. Otherwise, ack for the kvm bits.

Signature · 1 line
error compiling committee.c: too many arguments to function
#31 lore raw
Quoted 9 lines · Avi Kivity
Quoted 5 lines · Eduardo Habkost

Hi,

This is a new version of the series to disabling virtualization on kdump, now extended to do the same tricks on emergency_restart() if needed.

Looks good. If you me to push it upstream, I'll need kexec/kdump acks. Otherwise, ack for the kvm bits.

Looks good to me from kdump perspective.

Is there a way we can prevent any other module from using virt disable callback incase kvm is not loaded?

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#32 lore raw
Quoted 8 lines · Vivek Goyal
Quoted 7 lines · Avi Kivity
Quoted 5 lines · Eduardo Habkost

Hi,

This is a new version of the series to disabling virtualization on kdump, now extended to do the same tricks on emergency_restart() if needed.

Quoted 7 lines
Quoted 2 lines

Looks good. If you me to push it upstream, I'll need kexec/kdump acks. Otherwise, ack for the kvm bits.

Looks good to me from kdump perspective.

Is there a way we can prevent any other module from using virt disable callback incase kvm is not loaded?

I think we can't prevent other modules from using the API. But 3rd-party modules deal with CPU virtualization extensions today won't live with KVM today, anyway. And between not living together with KVM and breaking kdump/reboot, and not living together with KVM and making kdump and reboot working, I the the latter looks better.

But I see this as an orthogonal issue. If we include a "CPU virtualization extensions core API" in the future, it could simply keep the same emergency_virt_disable() API available for emergency_restart and kdump.

Signature · 6 lines
Eduardo

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#33 lore raw
In reply to Vivek Goyal
Quoted 3 lines · Vivek Goyal

Is there a way we can prevent any other module from using virt disable callback incase kvm is not loaded?

We can inline the kvm specific code (which will make it useful for other hypervisors, and remove the hook).

Signature · 7 lines
error compiling committee.c: too many arguments to function


_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#34 lore raw
In reply to Avi Kivity
Quoted 9 lines · Avi Kivity
Quoted 5 lines · Eduardo Habkost

Hi,

This is a new version of the series to disabling virtualization on kdump, now extended to do the same tricks on emergency_restart() if needed.

Looks good. If you me to push it upstream, I'll need kexec/kdump acks. Otherwise, ack for the kvm bits.

general ack for the x86 bits, but i'm not sure whether we should be pushing this upstream so late in the cycle. If we do it in the next cycle then it's best we do it in the x86 tree, the KVM impact seems much smaller than the general x86 impact.

Ingo

#35 lore raw
Quoted 5 lines · Ingo Molnar

general ack for the x86 bits, but i'm not sure whether we should be pushing this upstream so late in the cycle. If we do it in the next cycle then it's best we do it in the x86 tree, the KVM impact seems much smaller than the general x86 impact.

It certainly doesn't fall under the recent regression rule, and there's a simple workaround (rmmod -r kvm) so I agree it's best to defer for the next cycle.

Signature · 1 line
error compiling committee.c: too many arguments to function
#36 lore raw
Quoted 10 lines · Avi Kivity
Quoted 5 lines · Ingo Molnar

general ack for the x86 bits, but i'm not sure whether we should be pushing this upstream so late in the cycle. If we do it in the next cycle then it's best we do it in the x86 tree, the KVM impact seems much smaller than the general x86 impact.

It certainly doesn't fall under the recent regression rule, and there's a simple workaround (rmmod -r kvm) so I agree it's best to defer for the next cycle.

Shouldn't reboot=kbd be the default again still during this cycle, then?

The cases where the new reboot=acpi default broke didn't necessarily involve KVM.

Signature · 1 line
Eduardo