Skip to contentSkip to conversation

xen/x86: implement NMI continuation

PATCH Superseded Juergen Gross 2 participants 16 messages Nov 9, 2020 – Nov 12, 2020

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v4 cover xen/x86: implement NMI continuation

Move sending of a virq event for oprofile to the local vcpu from NMI to normal interrupt context.

This has been tested with a small test patch using the continuation framework of patch 1 for all NMIs and doing a print to console in the continuation handler.

Version 1 of this small series was sent to the security list before.

Changes in V3:
- switched to self-IPI instead of softirq
- added patch 3

Changes in V4:
- use less generic approach

Juergen Gross (3):
xen/x86: add nmi continuation framework xen/oprofile: use NMI continuation for sending virq to guest xen/x86: issue pci_serr error message via NMI continuation

 xen/arch/x86/apic.c             | 13 +++++++---
 xen/arch/x86/oprofile/nmi_int.c | 20 +++++++++++++--
 xen/arch/x86/traps.c            | 44 ++++++++++++++++++++++++++++-----
 xen/include/asm-x86/nmi.h       | 11 ++++++++-
 xen/include/asm-x86/softirq.h   |  5 ++--
 xen/include/asm-x86/xenoprof.h  |  7 ++++++
 6 files changed, 85 insertions(+), 15 deletions(-)
Signature · 1 line
2.26.2
#1 lore raw

PATCH v4 1/3 xen/x86: add nmi continuation framework

Actions in NMI context are rather limited as e.g. locking is rather fragile.

Add a framework to continue processing in normal interrupt context after leaving NMI processing.

This is done by a high priority interrupt vector triggered via a self IPI from NMI context, which will then call the continuation function specified during NMI handling.

---
V4:
- make the framework less generic

V2:
- add prototype for continuation function (Roger Pau Monné)
- switch from softirq to explicit self-IPI (Jan Beulich)

---
 xen/arch/x86/apic.c       | 13 ++++++++++---
 xen/arch/x86/traps.c      | 19 +++++++++++++++++++
 xen/include/asm-x86/nmi.h | 11 ++++++++++-
 3 files changed, 39 insertions(+), 4 deletions(-)
xen/arch/x86/apic.c +10 -3
index 60627fd6e6..7497ddb5da 100644
@@ -40,6 +40,7 @@
 #include <irq_vectors.h>
 #include <xen/kexec.h>
 #include <asm/guest.h>
+#include <asm/nmi.h>
 #include <asm/time.h>
 
 static bool __read_mostly tdt_enabled;
@@ -1376,16 +1377,22 @@ void spurious_interrupt(struct cpu_user_regs *regs)
 {
     /*
      * Check if this is a vectored interrupt (most likely, as this is probably
-     * a request to dump local CPU state). Vectored interrupts are ACKed;
-     * spurious interrupts are not.
+     * a request to dump local CPU state or to continue NMI handling).
+     * Vectored interrupts are ACKed; spurious interrupts are not.
      */
     if (apic_isr_read(SPURIOUS_APIC_VECTOR)) {
+        bool is_spurious;
+
         ack_APIC_irq();
+        is_spurious = !nmi_check_continuation();
         if (this_cpu(state_dump_pending)) {
             this_cpu(state_dump_pending) = false;
             dump_execstate(regs);
-            return;
+            is_spurious = false;
         }
+
+        if ( !is_spurious )
+            return;
     }
 
     /* see sw-dev-man vol 3, chapter 7.4.13.5 */
xen/arch/x86/traps.c +19 -0
index bc5b8f8ea3..5005ac6e6e 100644
@@ -79,6 +79,7 @@
 #include <public/hvm/params.h>
 #include <asm/cpuid.h>
 #include <xsm/xsm.h>
+#include <asm/mach-default/irq_vectors.h>
 #include <asm/pv/traps.h>
 #include <asm/pv/mm.h>
 
@@ -1799,6 +1800,24 @@ void unset_nmi_callback(void)
     nmi_callback = dummy_nmi_callback;
 }
 
+bool nmi_check_continuation(void)
+{
+    bool ret = false;
+
+    return ret;
+}
+
+void trigger_nmi_continuation(void)
+{
+    /*
+     * Issue a self-IPI. Handling is done in spurious_interrupt().
+     * NMI could have happened in IPI sequence, so wait for ICR being idle
+     * again before leaving NMI handler.
+     */
+    send_IPI_self(SPURIOUS_APIC_VECTOR);
+    apic_wait_icr_idle();
+}
+
 void do_device_not_available(struct cpu_user_regs *regs)
 {
 #ifdef CONFIG_PV
xen/include/asm-x86/nmi.h +10 -1
index a288f02a50..9a5da14162 100644
@@ -33,5 +33,14 @@ nmi_callback_t *set_nmi_callback(nmi_callback_t *callback);
 void unset_nmi_callback(void);
 
 DECLARE_PER_CPU(unsigned int, nmi_count);
- 
+
+/**
+ * trigger_nmi_continuation
+ *
+ * Schedule continuation to be started in interrupt context after NMI handling.
+ */
+void trigger_nmi_continuation(void);
+
+/* Check for NMI continuation pending. */
+bool nmi_check_continuation(void);
 #endif /* ASM_NMI_H */
Signature · 1 line
2.26.2
Reviewed-by on v4 1/3
#2 lore raw

Re: [PATCH v4 1/3] xen/x86: add nmi continuation framework

Quoted 11 lines · Juergen Gross

Actions in NMI context are rather limited as e.g. locking is rather fragile.

Add a framework to continue processing in normal interrupt context after leaving NMI processing.

This is done by a high priority interrupt vector triggered via a self IPI from NMI context, which will then call the continuation function specified during NMI handling.

with one further adjustment request:

Quoted 11 lines
xen/arch/x86/traps.c +7 -0
@@ -1799,6 +1800,24 @@ void unset_nmi_callback(void)
     nmi_callback = dummy_nmi_callback;
 }
 
+bool nmi_check_continuation(void)
+{
+    bool ret = false;
+
+    return ret;
+}
+

xen/arch/x86/traps.c void unset_nmi_callback(void) · line 1819

Quoted 10 lines
xen/arch/x86/traps.c +10 -0
+void trigger_nmi_continuation(void)
+{
+    /*
+     * Issue a self-IPI. Handling is done in spurious_interrupt().
+     * NMI could have happened in IPI sequence, so wait for ICR being idle
+     * again before leaving NMI handler.
+     */
+    send_IPI_self(SPURIOUS_APIC_VECTOR);
+    apic_wait_icr_idle();
+}

This additionally relies on send_IPI_self_legacy() calling send_IPI_shortcut(), rather than e.g. resolving the local CPU number to a destination ID. I think this wants saying maybe here, but more importantly in that function.

Jan

#3 lore raw

Re: [PATCH v4 1/3] xen/x86: add nmi continuation framework

Quoted 34 lines · Jan Beulich
Quoted 11 lines · Juergen Gross

Actions in NMI context are rather limited as e.g. locking is rather fragile.

Add a framework to continue processing in normal interrupt context after leaving NMI processing.

This is done by a high priority interrupt vector triggered via a self IPI from NMI context, which will then call the continuation function specified during NMI handling.

with one further adjustment request:

Quoted 18 lines
xen/arch/x86/traps.c +14 -0
@@ -1799,6 +1800,24 @@ void unset_nmi_callback(void)
      nmi_callback = dummy_nmi_callback;
  }
 
+bool nmi_check_continuation(void)
+{
+    bool ret = false;
+
+    return ret;
+}
+
+void trigger_nmi_continuation(void)
+{
+    /*
+     * Issue a self-IPI. Handling is done in spurious_interrupt().
+     * NMI could have happened in IPI sequence, so wait for ICR being idle
+     * again before leaving NMI handler.
+     */
Quoted 8 lines

xen/arch/x86/traps.c void unset_nmi_callback(void) · line 1819

Quoted 3 lines
xen/arch/x86/traps.c +3 -0
+    send_IPI_self(SPURIOUS_APIC_VECTOR);
+    apic_wait_icr_idle();
+}

This additionally relies on send_IPI_self_legacy() calling send_IPI_shortcut(), rather than e.g. resolving the local CPU number to a destination ID. I think this wants saying maybe here, but more importantly in that function.

Okay.

Juergen

#4 lore raw

PATCH v4 2/3 xen/oprofile: use NMI continuation for sending virq to guest

In reply to Juergen Gross

Instead of calling send_guest_vcpu_virq() from NMI context use the NMI continuation framework for that purpose. This avoids taking locks in NMI mode.

---
V4:
- rework to less generic approach

---
 xen/arch/x86/oprofile/nmi_int.c | 20 ++++++++++++++++++--
 xen/arch/x86/traps.c            |  4 ++++
 xen/include/asm-x86/xenoprof.h  |  7 +++++++
 3 files changed, 29 insertions(+), 2 deletions(-)
xen/arch/x86/oprofile/nmi_int.c +18 -2
index 0f103d80a6..5f17cba0b2 100644
@@ -38,6 +38,8 @@ static unsigned long saved_lvtpc[NR_CPUS];
 
 static char *cpu_type;
 
+static DEFINE_PER_CPU(struct vcpu *, nmi_cont_vcpu);
+
 static int passive_domain_msr_op_checks(unsigned int msr, int *typep, int *indexp)
 {
 	struct vpmu_struct *vpmu = vcpu_vpmu(current);
@@ -83,14 +85,28 @@ void passive_domain_destroy(struct vcpu *v)
 		model->free_msr(v);
 }
 
+bool nmi_oprofile_send_virq(void)
+{
+	struct vcpu *v = this_cpu(nmi_cont_vcpu);
+
+	if ( v )
+		send_guest_vcpu_virq(v, VIRQ_XENOPROF);
+
+	this_cpu(nmi_cont_vcpu) = NULL;
+
+	return v;
+}
+
 static int nmi_callback(const struct cpu_user_regs *regs, int cpu)
 {
 	int xen_mode, ovf;
 
 	ovf = model->check_ctrs(cpu, &cpu_msrs[cpu], regs);
 	xen_mode = ring_0(regs);
-	if ( ovf && is_active(current->domain) && !xen_mode )
-		send_guest_vcpu_virq(current, VIRQ_XENOPROF);
+	if ( ovf && is_active(current->domain) && !xen_mode ) {
+		this_cpu(nmi_cont_vcpu) = current;
+		trigger_nmi_continuation();
+	}
 
 	if ( ovf == 2 )
 		current->arch.nmi_pending = true;
xen/arch/x86/traps.c +4 -0
index 5005ac6e6e..7cb7d7e09c 100644
@@ -65,6 +65,7 @@
 #include <asm/debugger.h>
 #include <asm/msr.h>
 #include <asm/nmi.h>
+#include <asm/xenoprof.h>
 #include <asm/shared.h>
 #include <asm/x86_emulate.h>
 #include <asm/traps.h>
@@ -1804,6 +1805,9 @@ bool nmi_check_continuation(void)
 {
     bool ret = false;
 
+    if ( nmi_oprofile_send_virq() )
+        ret = true;
+
     return ret;
 }
 
xen/include/asm-x86/xenoprof.h +7 -0
index 1026ba2e1f..cf6af8c5df 100644
@@ -69,6 +69,8 @@ int passive_domain_do_rdmsr(unsigned int msr, uint64_t *msr_content);
 int passive_domain_do_wrmsr(unsigned int msr, uint64_t msr_content);
 void passive_domain_destroy(struct vcpu *v);
 
+bool nmi_oprofile_send_virq(void);
+
 #else
 
 static inline int passive_domain_do_rdmsr(unsigned int msr,
@@ -85,6 +87,11 @@ static inline int passive_domain_do_wrmsr(unsigned int msr,
 
 static inline void passive_domain_destroy(struct vcpu *v) {}
 
+static inline bool nmi_oprofile_send_virq(void)
+{
+    return false;
+}
+
 #endif /* CONFIG_XENOPROF */
 
 #endif /* __ASM_X86_XENOPROF_H__ */
Signature · 1 line
2.26.2
#5 lore raw

Re: [PATCH v4 2/3] xen/oprofile: use NMI continuation for sending virq to guest

xen/arch/x86/oprofile/nmi_int.c void passive_domain_destroy(struct vcpu *v) · line 95

Quoted 12 lines · Juergen Gross
xen/arch/x86/oprofile/nmi_int.c +8 -0
@@ -83,14 +85,28 @@ void passive_domain_destroy(struct vcpu *v)
 		model->free_msr(v);
 }
 
+bool nmi_oprofile_send_virq(void)
+{
+	struct vcpu *v = this_cpu(nmi_cont_vcpu);
+
+	if ( v )
+		send_guest_vcpu_virq(v, VIRQ_XENOPROF);
+
+	this_cpu(nmi_cont_vcpu) = NULL;

What if, by the time we make it here, a 2nd NMI has arrived? I agree the next overflow interrupt shouldn't arrive this quickly, but I also think you want to zap the per-CPU variable first here, and ...

Quoted 4 lines
xen/arch/x86/oprofile/nmi_int.c +4 -0
+
+	return v;
+}
+

xen/arch/x86/oprofile/nmi_int.c void passive_domain_destroy(struct vcpu *v) · line 107

Quoted 10 lines
xen/arch/x86/oprofile/nmi_int.c +2 -2
 static int nmi_callback(const struct cpu_user_regs *regs, int cpu)
 {
 	int xen_mode, ovf;
 
 	ovf = model->check_ctrs(cpu, &cpu_msrs[cpu], regs);
 	xen_mode = ring_0(regs);
-	if ( ovf && is_active(current->domain) && !xen_mode )
-		send_guest_vcpu_virq(current, VIRQ_XENOPROF);
+	if ( ovf && is_active(current->domain) && !xen_mode ) {
+		this_cpu(nmi_cont_vcpu) = current;

... avoid overwriting any non-NULL value here. That's then of course still not closing the window, but has (imo) overall better behavior.

Also, style-wise, going through the file it looks to be mainly Linux style, so may I suggest your additions / changes to be done that way, rather than extending use of this funny mixed style?

Jan

#6 lore raw

Re: [PATCH v4 2/3] xen/oprofile: use NMI continuation for sending virq to guest

Quoted 5 lines · Jan Beulich
Quoted 4 lines · Juergen Gross
xen/arch/x86/oprofile/nmi_int.c +0 -0
@@ -83,14 +85,28 @@ void passive_domain_destroy(struct vcpu *v)
  		model->free_msr(v);
  }
 
Quoted 13 lines

xen/arch/x86/oprofile/nmi_int.c void passive_domain_destroy(struct vcpu *v) · line 95

Quoted 8 lines
xen/arch/x86/oprofile/nmi_int.c +8 -0
+bool nmi_oprofile_send_virq(void)
+{
+	struct vcpu *v = this_cpu(nmi_cont_vcpu);
+
+	if ( v )
+		send_guest_vcpu_virq(v, VIRQ_XENOPROF);
+
+	this_cpu(nmi_cont_vcpu) = NULL;

What if, by the time we make it here, a 2nd NMI has arrived? I agree the next overflow interrupt shouldn't arrive this quickly, but I also think you want to zap the per-CPU variable first here, and ...

How could that happen? This function is activated only from NMI context in case the NMI happened in guest mode. And it will be executed with higher priority than any guest, so there is a zero chance another NMI in guest mode can happen in between.

I can add a comment in this regard if you want.

Quoted 20 lines
Quoted 5 lines
xen/arch/x86/oprofile/nmi_int.c +4 -0
+
+	return v;
+}
+

xen/arch/x86/oprofile/nmi_int.c void passive_domain_destroy(struct vcpu *v) · line 107

Quoted 10 lines
xen/arch/x86/oprofile/nmi_int.c +2 -2
  static int nmi_callback(const struct cpu_user_regs *regs, int cpu)
  {
  	int xen_mode, ovf;
 
  	ovf = model->check_ctrs(cpu, &cpu_msrs[cpu], regs);
  	xen_mode = ring_0(regs);
-	if ( ovf && is_active(current->domain) && !xen_mode )
-		send_guest_vcpu_virq(current, VIRQ_XENOPROF);
+	if ( ovf && is_active(current->domain) && !xen_mode ) {
+		this_cpu(nmi_cont_vcpu) = current;

... avoid overwriting any non-NULL value here. That's then of course still not closing the window, but has (imo) overall better behavior.

Quoted 4 lines

Also, style-wise, going through the file it looks to be mainly Linux style, so may I suggest your additions / changes to be done that way, rather than extending use of this funny mixed style?

Works for me.

Juergen

#7 lore raw

Re: [PATCH v4 2/3] xen/oprofile: use NMI continuation for sending virq to guest

Quoted 20 lines · Jürgen Groß
Quoted 5 lines · Jan Beulich
Quoted 4 lines · Juergen Gross
xen/arch/x86/oprofile/nmi_int.c +0 -0
@@ -83,14 +85,28 @@ void passive_domain_destroy(struct vcpu *v)
  		model->free_msr(v);
  }
 
Quoted 13 lines

xen/arch/x86/oprofile/nmi_int.c void passive_domain_destroy(struct vcpu *v) · line 95

Quoted 8 lines
xen/arch/x86/oprofile/nmi_int.c +8 -0
+bool nmi_oprofile_send_virq(void)
+{
+	struct vcpu *v = this_cpu(nmi_cont_vcpu);
+
+	if ( v )
+		send_guest_vcpu_virq(v, VIRQ_XENOPROF);
+
+	this_cpu(nmi_cont_vcpu) = NULL;

What if, by the time we make it here, a 2nd NMI has arrived? I agree the next overflow interrupt shouldn't arrive this quickly, but I also think you want to zap the per-CPU variable first here, and ...

Quoted 4 lines

How could that happen? This function is activated only from NMI context in case the NMI happened in guest mode. And it will be executed with higher priority than any guest, so there is a zero chance another NMI in guest mode can happen in between.

While I'll admit I didn't pay attention to the bogus (as far as HVM is concerned) xen_mode check, my understanding is that the self-IPI will be delivered once we're back in guest mode, as that's the first time IRQs would be on again (even event checking gets deferred by sending a self-IPI). If another NMI was latched by that time, it would take precedence over the IRQ and would also be delivered on the guest mode insn that the IRET returned to.

I agree though that this is benign, as the vCPU wouldn't have been context switched out yet, i.e. current is still the same and there'll then merely be two NMI instances folded into one.

However, I still think the ordering would better be changed, to set a good precedent.

Quoted 6 lines
Quoted 6 lines

xen/arch/x86/oprofile/nmi_int.c void passive_domain_destroy(struct vcpu *v) · line 105

Quoted 6 lines
xen/arch/x86/oprofile/nmi_int.c +0 -0
  static int nmi_callback(const struct cpu_user_regs *regs, int cpu)
  {
  	int xen_mode, ovf;
 
  	ovf = model->check_ctrs(cpu, &cpu_msrs[cpu], regs);
  	xen_mode = ring_0(regs);

Unrelated to the patch here (i.e. just as an observation), this use of ring_0() looks bogus when the NMI occurred in HVM guest mode.

Jan

#8 lore raw

Re: [PATCH v4 2/3] xen/oprofile: use NMI continuation for sending virq to guest

Quoted 35 lines · Jan Beulich
Quoted 20 lines · Jürgen Groß
Quoted 5 lines · Jan Beulich
Quoted 4 lines · Juergen Gross
xen/arch/x86/oprofile/nmi_int.c +0 -0
@@ -83,14 +85,28 @@ void passive_domain_destroy(struct vcpu *v)
   		model->free_msr(v);
   }
 
Quoted 13 lines

xen/arch/x86/oprofile/nmi_int.c void passive_domain_destroy(struct vcpu *v) · line 95

Quoted 8 lines
xen/arch/x86/oprofile/nmi_int.c +8 -0
+bool nmi_oprofile_send_virq(void)
+{
+	struct vcpu *v = this_cpu(nmi_cont_vcpu);
+
+	if ( v )
+		send_guest_vcpu_virq(v, VIRQ_XENOPROF);
+
+	this_cpu(nmi_cont_vcpu) = NULL;

What if, by the time we make it here, a 2nd NMI has arrived? I agree the next overflow interrupt shouldn't arrive this quickly, but I also think you want to zap the per-CPU variable first here, and ...

Quoted 4 lines

How could that happen? This function is activated only from NMI context in case the NMI happened in guest mode. And it will be executed with higher priority than any guest, so there is a zero chance another NMI in guest mode can happen in between.

While I'll admit I didn't pay attention to the bogus (as far as HVM is concerned) xen_mode check, my understanding is that the self-IPI will be delivered once we're back in guest mode, as that's the first time IRQs would be on again (even event checking gets deferred by sending a self-IPI). If another NMI was latched by that time, it would take precedence over the IRQ and would also be delivered on the guest mode insn that the IRET returned to.

Quoted 3 lines

I agree though that this is benign, as the vCPU wouldn't have been context switched out yet, i.e. current is still the same and there'll then merely be two NMI instances folded into one.

Correct.

Quoted 3 lines

However, I still think the ordering would better be changed, to set a good precedent.

Okay, if you want that.

Quoted 11 lines
Quoted 7 lines
Quoted 7 lines

xen/arch/x86/oprofile/nmi_int.c void passive_domain_destroy(struct vcpu *v) · line 105

Quoted 7 lines
xen/arch/x86/oprofile/nmi_int.c +0 -0
   static int nmi_callback(const struct cpu_user_regs *regs, int cpu)
   {
   	int xen_mode, ovf;
 
   	ovf = model->check_ctrs(cpu, &cpu_msrs[cpu], regs);
   	xen_mode = ring_0(regs);

Unrelated to the patch here (i.e. just as an observation), this use of ring_0() looks bogus when the NMI occurred in HVM guest mode.

An NMI in an HVM guest due to oprofile would be a VMEXIT with NMI reason, or just be handled completely inside the guest, right?

I don't see how this test should ever result in xen_mode being false for an HVM guest.

Juergen

#9 lore raw

Re: [PATCH v4 2/3] xen/oprofile: use NMI continuation for sending virq to guest

Quoted 11 lines · Jürgen Groß
Quoted 10 lines · Jan Beulich
Quoted 8 lines · Jürgen Groß
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Juergen Gross
   static int nmi_callback(const struct cpu_user_regs *regs, int cpu)
   {
   	int xen_mode, ovf;
   
   	ovf = model->check_ctrs(cpu, &cpu_msrs[cpu], regs);
   	xen_mode = ring_0(regs);
Quoted 6 lines
Quoted 3 lines

Unrelated to the patch here (i.e. just as an observation), this use of ring_0() looks bogus when the NMI occurred in HVM guest mode.

An NMI in an HVM guest due to oprofile would be a VMEXIT with NMI reason, or just be handled completely inside the guest, right?

Yes, and in the former case for VMX it would be handed on to do_nmi(), with the guest register state. For SVM it would get handled on the next STGI, i.e. would indeed never surface from HVM guest mode.

Quoted 2 lines

I don't see how this test should ever result in xen_mode being false for an HVM guest.

I think, because of hvm_invalidate_regs_fields(), on VMX it would be consistently true in release builds and consistently false in debug ones.

Jan

#10 lore raw

Re: [PATCH v4 2/3] xen/oprofile: use NMI continuation for sending virq to guest

Quoted 23 lines · Jan Beulich
Quoted 11 lines · Jürgen Groß
Quoted 10 lines · Jan Beulich
Quoted 8 lines · Jürgen Groß
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Juergen Gross
    static int nmi_callback(const struct cpu_user_regs *regs, int cpu)
    {
    	int xen_mode, ovf;
    
    	ovf = model->check_ctrs(cpu, &cpu_msrs[cpu], regs);
    	xen_mode = ring_0(regs);
Quoted 6 lines
Quoted 3 lines

Unrelated to the patch here (i.e. just as an observation), this use of ring_0() looks bogus when the NMI occurred in HVM guest mode.

An NMI in an HVM guest due to oprofile would be a VMEXIT with NMI reason, or just be handled completely inside the guest, right?

Yes, and in the former case for VMX it would be handed on to do_nmi(), with the guest register state. For SVM it would get handled on the next STGI, i.e. would indeed never surface from HVM guest mode.

Quoted 6 lines
Quoted 2 lines

I don't see how this test should ever result in xen_mode being false for an HVM guest.

I think, because of hvm_invalidate_regs_fields(), on VMX it would be consistently true in release builds and consistently false in debug ones.

Ah, okay. I searched for do_nmi(), but the vmx code uses the exception table instead.

So I guess this should be:

xen_mode = !guest_mode(regs);

Juergen

#11 lore raw

Re: [PATCH v4 2/3] xen/oprofile: use NMI continuation for sending virq to guest

Quoted 31 lines · Jürgen Groß
Quoted 23 lines · Jan Beulich
Quoted 11 lines · Jürgen Groß
Quoted 10 lines · Jan Beulich
Quoted 8 lines · Jürgen Groß
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Juergen Gross
    static int nmi_callback(const struct cpu_user_regs *regs, int cpu)
    {
    	int xen_mode, ovf;
    
    	ovf = model->check_ctrs(cpu, &cpu_msrs[cpu], regs);
    	xen_mode = ring_0(regs);
Quoted 6 lines
Quoted 3 lines

Unrelated to the patch here (i.e. just as an observation), this use of ring_0() looks bogus when the NMI occurred in HVM guest mode.

An NMI in an HVM guest due to oprofile would be a VMEXIT with NMI reason, or just be handled completely inside the guest, right?

Yes, and in the former case for VMX it would be handed on to do_nmi(), with the guest register state. For SVM it would get handled on the next STGI, i.e. would indeed never surface from HVM guest mode.

Quoted 6 lines
Quoted 2 lines

I don't see how this test should ever result in xen_mode being false for an HVM guest.

I think, because of hvm_invalidate_regs_fields(), on VMX it would be consistently true in release builds and consistently false in debug ones.

Quoted 6 lines

Ah, okay. I searched for do_nmi(), but the vmx code uses the exception table instead.

So I guess this should be:

xen_mode = !guest_mode(regs);

Yes, I think so. Just that guest_mode() also has its issues (my patch "x86: refine guest_mode()" improving it at least some is still pending Andrew's go / no-go / improvement suggestions), so whether it's suitable to use here may need some careful evaluation.

Jan

#12 lore raw

Re: [PATCH v4 2/3] xen/oprofile: use NMI continuation for sending virq to guest

Quoted 35 lines · Jan Beulich
Quoted 34 lines · Jürgen Groß
Quoted 23 lines · Jan Beulich
Quoted 11 lines · Jürgen Groß
Quoted 10 lines · Jan Beulich
Quoted 8 lines · Jürgen Groß
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Juergen Gross
     static int nmi_callback(const struct cpu_user_regs *regs, int cpu)
     {
     	int xen_mode, ovf;
     
     	ovf = model->check_ctrs(cpu, &cpu_msrs[cpu], regs);
     	xen_mode = ring_0(regs);
Quoted 6 lines
Quoted 3 lines

Unrelated to the patch here (i.e. just as an observation), this use of ring_0() looks bogus when the NMI occurred in HVM guest mode.

An NMI in an HVM guest due to oprofile would be a VMEXIT with NMI reason, or just be handled completely inside the guest, right?

Yes, and in the former case for VMX it would be handed on to do_nmi(), with the guest register state. For SVM it would get handled on the next STGI, i.e. would indeed never surface from HVM guest mode.

Quoted 6 lines
Quoted 2 lines

I don't see how this test should ever result in xen_mode being false for an HVM guest.

I think, because of hvm_invalidate_regs_fields(), on VMX it would be consistently true in release builds and consistently false in debug ones.

Ah, okay. I searched for do_nmi(), but the vmx code uses the exception table instead.

Quoted 8 lines
Quoted 3 lines

So I guess this should be:

xen_mode = !guest_mode(regs);

Yes, I think so. Just that guest_mode() also has its issues (my patch "x86: refine guest_mode()" improving it at least some is still pending Andrew's go / no-go / improvement suggestions), so whether it's suitable to use here may need some careful evaluation.

I'll leave the test as is for now.

We can revisit it when your patch has been committed.

Juergen

#13 lore raw

PATCH v4 3/3 xen/x86: issue pci_serr error message via NMI continuation

In reply to Juergen Gross

Instead of using a softirq pci_serr_error() can use NMI continuation for issuing an error message.

---
V4:
- rework to less generic approach

---
 xen/arch/x86/traps.c          | 21 +++++++++++++++------
 xen/include/asm-x86/softirq.h |  5 ++---
 2 files changed, 17 insertions(+), 9 deletions(-)
xen/arch/x86/traps.c +15 -6
index 7cb7d7e09c..6aeccef32d 100644
@@ -1660,10 +1660,18 @@ void do_general_protection(struct cpu_user_regs *regs)
     panic("GENERAL PROTECTION FAULT\n[error_code=%04x]\n", regs->error_code);
 }
 
-static void pci_serr_softirq(void)
+static bool pci_serr_cont;
+
+static bool pci_serr_nmicont(void)
 {
+    if ( !pci_serr_cont )
+        return false;
+
+    pci_serr_cont = false;
     printk("\n\nNMI - PCI system error (SERR)\n");
     outb(inb(0x61) & 0x0b, 0x61); /* re-enable the PCI SERR error line. */
+
+    return true;
 }
 
 static void nmi_hwdom_report(unsigned int reason_idx)
@@ -1688,9 +1696,9 @@ static void pci_serr_error(const struct cpu_user_regs *regs)
         nmi_hwdom_report(_XEN_NMIREASON_pci_serr);
         /* fallthrough */
     case 'i': /* 'ignore' */
-        /* Would like to print a diagnostic here but can't call printk()
-           from NMI context -- raise a softirq instead. */
-        raise_softirq(PCI_SERR_SOFTIRQ);
+        /* Issue error message in NMI continuation. */
+        pci_serr_cont = true;
+        trigger_nmi_continuation();
         break;
     default:  /* 'fatal' */
         console_force_unlock();
@@ -1808,6 +1816,9 @@ bool nmi_check_continuation(void)
     if ( nmi_oprofile_send_virq() )
         ret = true;
 
+    if ( pci_serr_nmicont() )
+        ret = true;
+
     return ret;
 }
 
@@ -2154,8 +2165,6 @@ void __init trap_init(void)
     percpu_traps_init();
 
     cpu_init();
-
-    open_softirq(PCI_SERR_SOFTIRQ, pci_serr_softirq);
 }
 
 void activate_debugregs(const struct vcpu *curr)
xen/include/asm-x86/softirq.h +2 -3
index 0b7a77f11f..415ee866c7 100644
@@ -6,9 +6,8 @@
 #define VCPU_KICK_SOFTIRQ      (NR_COMMON_SOFTIRQS + 2)
 
 #define MACHINE_CHECK_SOFTIRQ  (NR_COMMON_SOFTIRQS + 3)
-#define PCI_SERR_SOFTIRQ       (NR_COMMON_SOFTIRQS + 4)
-#define HVM_DPCI_SOFTIRQ       (NR_COMMON_SOFTIRQS + 5)
-#define NR_ARCH_SOFTIRQS       6
+#define HVM_DPCI_SOFTIRQ       (NR_COMMON_SOFTIRQS + 4)
+#define NR_ARCH_SOFTIRQS       5
 
 bool arch_skip_send_event_check(unsigned int cpu);
 
Signature · 1 line
2.26.2
Reviewed-by on v4 3/3
#14 lore raw

Re: [PATCH v4 3/3] xen/x86: issue pci_serr error message via NMI continuation

Quoted 4 lines · Juergen Gross

Instead of using a softirq pci_serr_error() can use NMI continuation for issuing an error message.

with one minor change to be considered:

xen/arch/x86/traps.c bool nmi_check_continuation(void) · line 1823

Quoted 9 lines
xen/arch/x86/traps.c +3 -0
@@ -1808,6 +1816,9 @@ bool nmi_check_continuation(void)
     if ( nmi_oprofile_send_virq() )
         ret = true;
 
+    if ( pci_serr_nmicont() )
+        ret = true;
+
     return ret;
 }

As the likely more important part, wouldn't it be better to insert this ahead of the oprofile check?

Jan

#15 lore raw

Re: [PATCH v4 3/3] xen/x86: issue pci_serr error message via NMI continuation

Quoted 13 lines · Jan Beulich
Quoted 4 lines · Juergen Gross

Instead of using a softirq pci_serr_error() can use NMI continuation for issuing an error message.

with one minor change to be considered:

Quoted 4 lines
xen/arch/x86/traps.c +0 -0
@@ -1808,6 +1816,9 @@ bool nmi_check_continuation(void)
      if ( nmi_oprofile_send_virq() )
          ret = true;
 
Quoted 8 lines

xen/arch/x86/traps.c bool nmi_check_continuation(void) · line 1823

Quoted 5 lines
xen/arch/x86/traps.c +3 -0
+    if ( pci_serr_nmicont() )
+        ret = true;
+
      return ret;
  }

As the likely more important part, wouldn't it be better to insert this ahead of the oprofile check?

Fine with me.

Juergen