Skip to contentSkip to conversation

xenpv: don't BUG when failing to setup NMI callback

PATCH In review Vitaly Kuznetsov 3 participants 5 messages Jun 16, 2014

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 xenpv: don't BUG when failing to setup NMI callback

some old Xen hypervisors (prior to 3.2) forbid DomUs to register NMI callbacks. E.g. we have the following code in xen-3.1:

    if ( (d->domain_id != 0) || (v->vcpu_id != 0) )
        return -EINVAL;

Commit 6efa20e49b9cb1db1ab66870cc37323474a75a13 introduced kernel crash in case PV guest fails to register NMI callback. All x86_64 PV guests will fail to boot on top of such hypervisors (RHEL5 example):

(XEN) traps.c:405:d7 Unhandled invalid opcode fault/trap [#6] in domain 7 on VCPU 0 [ec=0000] (XEN) domain_crash_sync called from entry.S
(XEN) Domain 7 (vcpu#0) crashed on cpu#3:
(XEN) ----[ Xen-3.1.2-389.el5 x86_64 debug=n Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: e033:[<ffffffff81004d96>]
(XEN) RFLAGS: 0000000000000282 CONTEXT: guest

(XEN) rax: ffffffffffffffea   rbx: 0000000000000000   rcx: 0000000000000002
(XEN) rdx: 0000000000000001   rsi: ffffffff81b0fe28   rdi: 0000000000000000
(XEN) rbp: ffffffff81b0fe40   rsp: ffffffff81b0fde8   r8:  0000000000000000
(XEN) r9:  ffffffff81b0fdd0   r10: 0000000000007ff0   r11: 00000000ffffffff
(XEN) r12: ffffffff81d65900   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 00000000000026b0
(XEN) cr3: 000000013a263000   cr2: 0000000000000000

(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033
...

However it is possible to proceed without NMI callback registered.

Changes in v2:
- skip nmi in cvt_gate_to_trap() for Xen < 3.2
- do not call xen_enable_nmi() when running under Xen < 3.2
- never BUG() in xen_enable_nmi()

---
 arch/x86/xen/enlighten.c |  9 +++++----
 arch/x86/xen/setup.c     | 17 ++++++++++++++---
 2 files changed, 19 insertions(+), 7 deletions(-)
arch/x86/xen/enlighten.c +5 -4
index f17b292..60ec5ef 100644
@@ -746,12 +746,13 @@ static int cvt_gate_to_trap(int vector, const gate_desc *val,
 		 */
 		;
 #endif
-	} else if (addr == (unsigned long)nmi)
+	} else if (addr == (unsigned long)nmi) {
 		/*
-		 * Use the native version as well.
+		 * Use the native version as well but require Xen >= 3.2
 		 */
-		;
-	else {
+		if (!xen_running_on_version_or_later(3, 2))
+			return 0;
+	} else {
 		/* Some other trap using IST? */
 		if (WARN_ON(val->ist != 0))
 			return 0;
arch/x86/xen/setup.c +14 -3
index 821a11a..fdc73d3 100644
@@ -593,8 +593,14 @@ void xen_enable_syscall(void)
 void xen_enable_nmi(void)
 {
 #ifdef CONFIG_X86_64
-	if (register_callback(CALLBACKTYPE_nmi, (char *)nmi))
-		BUG();
+	int ret;
+
+	ret = register_callback(CALLBACKTYPE_nmi, (char *)nmi);
+	if (ret != 0) {
+		/* Hypervisor probably forbids us to register NMI callback or
+		   some other error happened */
+		pr_warn("xen: failed to register NMI callback: %d\n", ret);
+	}
 #endif
 }
 void __init xen_pvmmu_arch_setup(void)
@@ -611,7 +617,12 @@ void __init xen_pvmmu_arch_setup(void)
 
 	xen_enable_sysenter();
 	xen_enable_syscall();
-	xen_enable_nmi();
+
+	/* Xen versions prior to 3.2 forbid DomUs to register NMI callbacks */
+	if (xen_running_on_version_or_later(3, 2))
+		xen_enable_nmi();
+	else
+		pr_warn("xen: skipping NMI callback registration for Xen < 3.2");
 }
 
 /* This function is not called for HVM domains */
Signature · 1 line
1.9.3
Reviewed-by on v2 1/1
#1 lore raw
Quoted 28 lines · Vitaly Kuznetsov

some old Xen hypervisors (prior to 3.2) forbid DomUs to register NMI callbacks. E.g. we have the following code in xen-3.1:

    if ( (d->domain_id != 0) || (v->vcpu_id != 0) )
        return -EINVAL;

Commit 6efa20e49b9cb1db1ab66870cc37323474a75a13 introduced kernel crash in case PV guest fails to register NMI callback. All x86_64 PV guests will fail to boot on top of such hypervisors (RHEL5 example):

(XEN) traps.c:405:d7 Unhandled invalid opcode fault/trap [#6] in domain 7 on VCPU 0 [ec=0000] (XEN) domain_crash_sync called from entry.S
(XEN) Domain 7 (vcpu#0) crashed on cpu#3:
(XEN) ----[ Xen-3.1.2-389.el5 x86_64 debug=n Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: e033:[<ffffffff81004d96>]
(XEN) RFLAGS: 0000000000000282 CONTEXT: guest

(XEN) rax: ffffffffffffffea   rbx: 0000000000000000   rcx: 0000000000000002
(XEN) rdx: 0000000000000001   rsi: ffffffff81b0fe28   rdi: 0000000000000000
(XEN) rbp: ffffffff81b0fe40   rsp: ffffffff81b0fde8   r8:  0000000000000000
(XEN) r9:  ffffffff81b0fdd0   r10: 0000000000007ff0   r11: 00000000ffffffff
(XEN) r12: ffffffff81d65900   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 00000000000026b0
(XEN) cr3: 000000013a263000   cr2: 0000000000000000

(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033
...

Quoted 8 lines

However it is possible to proceed without NMI callback registered.

Changes in v2:
- skip nmi in cvt_gate_to_trap() for Xen < 3.2
- do not call xen_enable_nmi() when running under Xen < 3.2
- never BUG() in xen_enable_nmi()

Awesome!

David, if you don't get to stash this patch in the queue - I can do it on Friday.

Quoted 65 lines
---
 arch/x86/xen/enlighten.c |  9 +++++----
 arch/x86/xen/setup.c     | 17 ++++++++++++++---
 2 files changed, 19 insertions(+), 7 deletions(-)
arch/x86/xen/enlighten.c +5 -4
index f17b292..60ec5ef 100644
@@ -746,12 +746,13 @@ static int cvt_gate_to_trap(int vector, const gate_desc *val,
 		 */
 		;
 #endif
-	} else if (addr == (unsigned long)nmi)
+	} else if (addr == (unsigned long)nmi) {
 		/*
-		 * Use the native version as well.
+		 * Use the native version as well but require Xen >= 3.2
 		 */
-		;
-	else {
+		if (!xen_running_on_version_or_later(3, 2))
+			return 0;
+	} else {
 		/* Some other trap using IST? */
 		if (WARN_ON(val->ist != 0))
 			return 0;
arch/x86/xen/setup.c +14 -3
index 821a11a..fdc73d3 100644
@@ -593,8 +593,14 @@ void xen_enable_syscall(void)
 void xen_enable_nmi(void)
 {
 #ifdef CONFIG_X86_64
-	if (register_callback(CALLBACKTYPE_nmi, (char *)nmi))
-		BUG();
+	int ret;
+
+	ret = register_callback(CALLBACKTYPE_nmi, (char *)nmi);
+	if (ret != 0) {
+		/* Hypervisor probably forbids us to register NMI callback or
+		   some other error happened */
+		pr_warn("xen: failed to register NMI callback: %d\n", ret);
+	}
 #endif
 }
 void __init xen_pvmmu_arch_setup(void)
@@ -611,7 +617,12 @@ void __init xen_pvmmu_arch_setup(void)
 
 	xen_enable_sysenter();
 	xen_enable_syscall();
-	xen_enable_nmi();
+
+	/* Xen versions prior to 3.2 forbid DomUs to register NMI callbacks */
+	if (xen_running_on_version_or_later(3, 2))
+		xen_enable_nmi();
+	else
+		pr_warn("xen: skipping NMI callback registration for Xen < 3.2");
 }
 
 /* This function is not called for HVM domains */
Signature · 1 line
1.9.3
Reviewed-by on v2 1/1
#2 lore raw
In reply to Vitaly Kuznetsov
Quoted 28 lines · Vitaly Kuznetsov

some old Xen hypervisors (prior to 3.2) forbid DomUs to register NMI callbacks. E.g. we have the following code in xen-3.1:

    if ( (d->domain_id != 0) || (v->vcpu_id != 0) )
        return -EINVAL;

Commit 6efa20e49b9cb1db1ab66870cc37323474a75a13 introduced kernel crash in case PV guest fails to register NMI callback. All x86_64 PV guests will fail to boot on top of such hypervisors (RHEL5 example):

(XEN) traps.c:405:d7 Unhandled invalid opcode fault/trap [#6] in domain 7 on VCPU 0 [ec=0000] (XEN) domain_crash_sync called from entry.S
(XEN) Domain 7 (vcpu#0) crashed on cpu#3:
(XEN) ----[ Xen-3.1.2-389.el5 x86_64 debug=n Not tainted ]----
(XEN) CPU: 3
(XEN) RIP: e033:[<ffffffff81004d96>]
(XEN) RFLAGS: 0000000000000282 CONTEXT: guest

(XEN) rax: ffffffffffffffea   rbx: 0000000000000000   rcx: 0000000000000002
(XEN) rdx: 0000000000000001   rsi: ffffffff81b0fe28   rdi: 0000000000000000
(XEN) rbp: ffffffff81b0fe40   rsp: ffffffff81b0fde8   r8:  0000000000000000
(XEN) r9:  ffffffff81b0fdd0   r10: 0000000000007ff0   r11: 00000000ffffffff
(XEN) r12: ffffffff81d65900   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080050033   cr4: 00000000000026b0
(XEN) cr3: 000000013a263000   cr2: 0000000000000000

(XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e02b cs: e033
...

Quoted 8 lines

However it is possible to proceed without NMI callback registered.

Changes in v2:
- skip nmi in cvt_gate_to_trap() for Xen < 3.2
- do not call xen_enable_nmi() when running under Xen < 3.2
- never BUG() in xen_enable_nmi()

Awesome!

David, if you don't get to stash this patch in the queue - I can do it on Friday.

Quoted 65 lines
---
 arch/x86/xen/enlighten.c |  9 +++++----
 arch/x86/xen/setup.c     | 17 ++++++++++++++---
 2 files changed, 19 insertions(+), 7 deletions(-)
arch/x86/xen/enlighten.c +5 -4
index f17b292..60ec5ef 100644
@@ -746,12 +746,13 @@ static int cvt_gate_to_trap(int vector, const gate_desc *val,
 		 */
 		;
 #endif
-	} else if (addr == (unsigned long)nmi)
+	} else if (addr == (unsigned long)nmi) {
 		/*
-		 * Use the native version as well.
+		 * Use the native version as well but require Xen >= 3.2
 		 */
-		;
-	else {
+		if (!xen_running_on_version_or_later(3, 2))
+			return 0;
+	} else {
 		/* Some other trap using IST? */
 		if (WARN_ON(val->ist != 0))
 			return 0;
arch/x86/xen/setup.c +14 -3
index 821a11a..fdc73d3 100644
@@ -593,8 +593,14 @@ void xen_enable_syscall(void)
 void xen_enable_nmi(void)
 {
 #ifdef CONFIG_X86_64
-	if (register_callback(CALLBACKTYPE_nmi, (char *)nmi))
-		BUG();
+	int ret;
+
+	ret = register_callback(CALLBACKTYPE_nmi, (char *)nmi);
+	if (ret != 0) {
+		/* Hypervisor probably forbids us to register NMI callback or
+		   some other error happened */
+		pr_warn("xen: failed to register NMI callback: %d\n", ret);
+	}
 #endif
 }
 void __init xen_pvmmu_arch_setup(void)
@@ -611,7 +617,12 @@ void __init xen_pvmmu_arch_setup(void)
 
 	xen_enable_sysenter();
 	xen_enable_syscall();
-	xen_enable_nmi();
+
+	/* Xen versions prior to 3.2 forbid DomUs to register NMI callbacks */
+	if (xen_running_on_version_or_later(3, 2))
+		xen_enable_nmi();
+	else
+		pr_warn("xen: skipping NMI callback registration for Xen < 3.2");
 }
 
 /* This function is not called for HVM domains */
Signature · 1 line
1.9.3
#3 lore raw
In reply to Vitaly Kuznetsov
Quoted 8 lines · Vitaly Kuznetsov
arch/x86/xen/enlighten.c +0 -1
@@ -746,12 +746,13 @@ static int cvt_gate_to_trap(int vector, const gate_desc *val,
 		 */
 		;
 #endif
-	} else if (addr == (unsigned long)nmi)

arch/x86/xen/enlighten.c static int cvt_gate_to_trap(int vector, const gate_desc *val, · line 755

Quoted 10 lines
arch/x86/xen/enlighten.c +5 -3
+	} else if (addr == (unsigned long)nmi) {
 		/*
-		 * Use the native version as well.
+		 * Use the native version as well but require Xen >= 3.2
 		 */
-		;
-	else {
+		if (!xen_running_on_version_or_later(3, 2))
+			return 0;
+	} else {

The end result of this looked odd so I played around with it a bit and it turns out that the register_callback for the NMI is entirely unneeded (since we set the NMI callback via the write_idt pv-op and it's also broken since it only sets the callback on the current VCPU.

I'm preparing an alternate patch, but I do not have a Xen 3.1 install I can test with.

David

#4 lore raw
In reply to Vitaly Kuznetsov
Quoted 8 lines · Vitaly Kuznetsov
arch/x86/xen/enlighten.c +0 -1
@@ -746,12 +746,13 @@ static int cvt_gate_to_trap(int vector, const gate_desc *val,
 		 */
 		;
 #endif
-	} else if (addr == (unsigned long)nmi)

arch/x86/xen/enlighten.c static int cvt_gate_to_trap(int vector, const gate_desc *val, · line 755

Quoted 10 lines
arch/x86/xen/enlighten.c +5 -3
+	} else if (addr == (unsigned long)nmi) {
 		/*
-		 * Use the native version as well.
+		 * Use the native version as well but require Xen >= 3.2
 		 */
-		;
-	else {
+		if (!xen_running_on_version_or_later(3, 2))
+			return 0;
+	} else {

The end result of this looked odd so I played around with it a bit and it turns out that the register_callback for the NMI is entirely unneeded (since we set the NMI callback via the write_idt pv-op and it's also broken since it only sets the callback on the current VCPU.

I'm preparing an alternate patch, but I do not have a Xen 3.1 install I can test with.

David