Skip to contentSkip to conversation

vm_event: fix race-condition when disabling monitor events

PATCH Superseded Tamas K Lengyel 2 participants 11 messages May 15, 2020 – May 21, 2020

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover vm_event: fix race-condition when disabling monitor events

For the last couple years we have received numerous reports from users of monitor vm_events of spurious guest crashes when using events. In particular, it has observed that the problem occurs when vm_events are being disabled. The nature of the guest crash varied widely and has only occured occasionally. This made debugging the issue particularly hard. We had discussions about this issue even here on the xen-devel mailinglist with no luck figuring it out.

The bug has now been identified as a race-condition between register event handling and disabling the vm_event interface.

Patch 96760e2fba100d694300a81baddb5740e0f8c0ee, "vm_event: deny register writes if refused by vm_event reply" is the patch that introduced the error. In this patch emulation of register write events can be postponed until the corresponding vm_event handler decides whether to allow such write to take place. Unfortunately this can only be implemented by performing the deny/allow step when the vCPU gets scheduled. Due to that postponed emulation of the event if the user decides to pause the VM in the vm_event handler and then disable events, the entire emulation step is skipped the next time the vCPU is resumed. Even if the user doesn't pause during the vm_event handling but exits immediately and disables vm_event, the situation becomes racey as disabling vm_event may succeed before the guest's vCPUs get scheduled with the pending emulation task. This has been particularly the case with VMs that have several vCPUs as after the VM is unpaused it may actually take a long time before all vCPUs get scheduled.

The only solution currently is to poll each vCPU before vm_events are disabled to verify they had been scheduled. The following patches resolve this issue in a much nicer way.

Patch 1 adds an option to the monitor_op domctl that needs to be specified if
the user wants to actually use the postponed register-write handling mechanism. If that option is not specified then handling is performed the same way as before patch 96760e2fba100d694300a81baddb5740e0f8c0ee.

Patch 2 performs sanity checking when disabling vm_events to determine whether
its safe to free all vm_event structures. The vCPUs still get unpaused to allow them to get scheduled and perform any of their pending operations, but otherwise an -EAGAIN error is returned signaling to the user that they need to wait and try again disabling the interface.

Patch 3 adds a vm_event specifically to signal to the user when it is safe to
continue disabling the interface.

Shout out to our friends at CERT.pl for stumbling upon a crucial piece of information that lead to finally squashing this nasty bug.

Tamas K Lengyel (3):
xen/monitor: Control register values
xen/vm_event: add vm_event_check_pending_op
xen/vm_event: Add safe to disable vm_event

 xen/arch/x86/hvm/hvm.c            | 69 +++++++++++++++++++++++--------
 xen/arch/x86/hvm/monitor.c        | 14 +++++++
 xen/arch/x86/monitor.c            | 23 ++++++++++-
 xen/arch/x86/vm_event.c           | 23 +++++++++++
 xen/common/vm_event.c             | 17 ++++++--
 xen/include/asm-arm/vm_event.h    |  7 ++++
 xen/include/asm-x86/domain.h      |  2 +
 xen/include/asm-x86/hvm/monitor.h |  1 +
 xen/include/asm-x86/vm_event.h    |  2 +
 xen/include/public/domctl.h       |  3 ++
 xen/include/public/vm_event.h     |  8 ++++
 11 files changed, 147 insertions(+), 22 deletions(-)
Signature · 1 line
2.26.1
#1 lore raw

PATCH 1/3 xen/monitor: Control register values

Extend the monitor_op domctl to include option that enables controlling what values certain registers are permitted to hold by a monitor subscriber.

---
 xen/arch/x86/hvm/hvm.c       | 31 +++++++++++++++++++------------
 xen/arch/x86/monitor.c       | 10 +++++++++-
 xen/include/asm-x86/domain.h |  1 +
 xen/include/public/domctl.h  |  1 +
 4 files changed, 30 insertions(+), 13 deletions(-)
xen/arch/x86/hvm/hvm.c +19 -12
index 814b7020d8..063f8ddc18 100644
@@ -2263,9 +2263,10 @@ int hvm_set_cr0(unsigned long value, bool may_defer)
     {
         ASSERT(v->arch.vm_event);
 
-        if ( hvm_monitor_crX(CR0, value, old_value) )
+        if ( hvm_monitor_crX(CR0, value, old_value) &&
+             v->domain->arch.monitor.control_register_values )
         {
-            /* The actual write will occur in hvm_do_resume(), if permitted. */
+            /* The actual write will occur in hvm_do_resume, if permitted. */
             v->arch.vm_event->write_data.do_write.cr0 = 1;
             v->arch.vm_event->write_data.cr0 = value;
 
@@ -2362,9 +2363,10 @@ int hvm_set_cr3(unsigned long value, bool may_defer)
     {
         ASSERT(v->arch.vm_event);
 
-        if ( hvm_monitor_crX(CR3, value, old) )
+        if ( hvm_monitor_crX(CR3, value, old) &&
+             v->domain->arch.monitor.control_register_values )
         {
-            /* The actual write will occur in hvm_do_resume(), if permitted. */
+            /* The actual write will occur in hvm_do_resume, if permitted. */
             v->arch.vm_event->write_data.do_write.cr3 = 1;
             v->arch.vm_event->write_data.cr3 = value;
 
@@ -2443,9 +2445,10 @@ int hvm_set_cr4(unsigned long value, bool may_defer)
     {
         ASSERT(v->arch.vm_event);
 
-        if ( hvm_monitor_crX(CR4, value, old_cr) )
+        if ( hvm_monitor_crX(CR4, value, old_cr) &&
+             v->domain->arch.monitor.control_register_values )
         {
-            /* The actual write will occur in hvm_do_resume(), if permitted. */
+            /* The actual write will occur in hvm_do_resume, if permitted. */
             v->arch.vm_event->write_data.do_write.cr4 = 1;
             v->arch.vm_event->write_data.cr4 = value;
 
@@ -3587,13 +3590,17 @@ int hvm_msr_write_intercept(unsigned int msr, uint64_t msr_content,
 
         ASSERT(v->arch.vm_event);
 
-        /* The actual write will occur in hvm_do_resume() (if permitted). */
-        v->arch.vm_event->write_data.do_write.msr = 1;
-        v->arch.vm_event->write_data.msr = msr;
-        v->arch.vm_event->write_data.value = msr_content;
-
         hvm_monitor_msr(msr, msr_content, msr_old_content);
-        return X86EMUL_OKAY;
+
+        if ( v->domain->arch.monitor.control_register_values )
+        {
+            /* The actual write will occur in hvm_do_resume, if permitted. */
+            v->arch.vm_event->write_data.do_write.msr = 1;
+            v->arch.vm_event->write_data.msr = msr;
+            v->arch.vm_event->write_data.value = msr_content;
+
+            return X86EMUL_OKAY;
+        }
     }
 
     if ( (ret = guest_wrmsr(v, msr, msr_content)) != X86EMUL_UNHANDLEABLE )
xen/arch/x86/monitor.c +9 -1
index bbcb7536c7..1517a97f50 100644
@@ -144,7 +144,15 @@ int arch_monitor_domctl_event(struct domain *d,
                               struct xen_domctl_monitor_op *mop)
 {
     struct arch_domain *ad = &d->arch;
-    bool requested_status = (XEN_DOMCTL_MONITOR_OP_ENABLE == mop->op);
+    bool requested_status;
+
+    if ( XEN_DOMCTL_MONITOR_OP_CONTROL_REGISTERS == mop->op )
+    {
+        ad->monitor.control_register_values = true;
+        return 0;
+    }
+
+    requested_status = (XEN_DOMCTL_MONITOR_OP_ENABLE == mop->op);
 
     switch ( mop->event )
     {
xen/include/asm-x86/domain.h +1 -0
index 5b6d909266..d890ab7a22 100644
@@ -416,6 +416,7 @@ struct arch_domain
          * This is used to filter out pagefaults.
          */
         unsigned int inguest_pagefault_disabled                            : 1;
+        unsigned int control_register_values                               : 1;
         struct monitor_msr_bitmap *msr_bitmap;
         uint64_t write_ctrlreg_mask[4];
     } monitor;
xen/include/public/domctl.h +1 -0
index 1ad34c35eb..cbcd25f12c 100644
@@ -1025,6 +1025,7 @@ struct xen_domctl_psr_cmt_op {
 #define XEN_DOMCTL_MONITOR_OP_DISABLE           1
 #define XEN_DOMCTL_MONITOR_OP_GET_CAPABILITIES  2
 #define XEN_DOMCTL_MONITOR_OP_EMULATE_EACH_REP  3
+#define XEN_DOMCTL_MONITOR_OP_CONTROL_REGISTERS 4
 
 #define XEN_DOMCTL_MONITOR_EVENT_WRITE_CTRLREG         0
 #define XEN_DOMCTL_MONITOR_EVENT_MOV_TO_MSR            1
Signature · 1 line
2.26.1
#2 lore raw

Re: [PATCH 1/3] xen/monitor: Control register values

Quoted 3 lines · Tamas K Lengyel

Extend the monitor_op domctl to include option that enables controlling what values certain registers are permitted to hold by a monitor subscriber.

This needs a bit more explanation, especially for those of us who aren't that introspection savvy. For example, from the text here I didn't expect a simple bool control, but something where actual (register) values get passed back and forth.

xen/arch/x86/hvm/hvm.c int hvm_set_cr0(unsigned long value, bool may_defer) · line 2269

Quoted 12 lines
xen/arch/x86/hvm/hvm.c +3 -2
@@ -2263,9 +2263,10 @@ int hvm_set_cr0(unsigned long value, bool may_defer)
     {
         ASSERT(v->arch.vm_event);
 
-        if ( hvm_monitor_crX(CR0, value, old_value) )
+        if ( hvm_monitor_crX(CR0, value, old_value) &&
+             v->domain->arch.monitor.control_register_values )
         {
-            /* The actual write will occur in hvm_do_resume(), if permitted. */
+            /* The actual write will occur in hvm_do_resume, if permitted. */

Please can you leave alone this and the similar comments below. And for consistency _add_ parentheses to the one new instance you add?

xen/arch/x86/monitor.c int arch_monitor_domctl_event(struct domain *d, · line 151

Quoted 12 lines
xen/arch/x86/monitor.c +5 -1
@@ -144,7 +144,15 @@ int arch_monitor_domctl_event(struct domain *d,
                               struct xen_domctl_monitor_op *mop)
 {
     struct arch_domain *ad = &d->arch;
-    bool requested_status = (XEN_DOMCTL_MONITOR_OP_ENABLE == mop->op);
+    bool requested_status;
+
+    if ( XEN_DOMCTL_MONITOR_OP_CONTROL_REGISTERS == mop->op )
+    {
+        ad->monitor.control_register_values = true;

And there's no way to clear this flag again?

Jan

#3 lore raw

Re: [PATCH 1/3] xen/monitor: Control register values

Quoted 17 lines · Jan Beulich
Quoted 3 lines · Tamas K Lengyel

Extend the monitor_op domctl to include option that enables controlling what values certain registers are permitted to hold by a monitor subscriber.

This needs a bit more explanation, especially for those of us who aren't that introspection savvy. For example, from the text here I didn't expect a simple bool control, but something where actual (register) values get passed back and forth.

Quoted 6 lines
xen/arch/x86/hvm/hvm.c +0 -0
@@ -2263,9 +2263,10 @@ int hvm_set_cr0(unsigned long value, bool may_defer)
     {
         ASSERT(v->arch.vm_event);
 
Quoted 10 lines

xen/arch/x86/hvm/hvm.c int hvm_set_cr0(unsigned long value, bool may_defer) · line 2269

Quoted 6 lines
xen/arch/x86/hvm/hvm.c +3 -2
-        if ( hvm_monitor_crX(CR0, value, old_value) )
+        if ( hvm_monitor_crX(CR0, value, old_value) &&
+             v->domain->arch.monitor.control_register_values )
         {
-            /* The actual write will occur in hvm_do_resume(), if permitted. */
+            /* The actual write will occur in hvm_do_resume, if permitted. */

Please can you leave alone this and the similar comments below. And for consistency _add_ parentheses to the one new instance you add?

I changed to because now it doesn't fit into the 80-line limit below, and then changed it everywhere _for_ consistency.

Quoted 15 lines

xen/arch/x86/monitor.c int arch_monitor_domctl_event(struct domain *d, · line 151

Quoted 13 lines
xen/arch/x86/monitor.c +5 -1
@@ -144,7 +144,15 @@ int arch_monitor_domctl_event(struct domain *d,
                               struct xen_domctl_monitor_op *mop)
 {
     struct arch_domain *ad = &d->arch;
-    bool requested_status = (XEN_DOMCTL_MONITOR_OP_ENABLE == mop->op);
+    bool requested_status;
+
+    if ( XEN_DOMCTL_MONITOR_OP_CONTROL_REGISTERS == mop->op )
+    {
+        ad->monitor.control_register_values = true;

And there's no way to clear this flag again?

There is. Disable the monitor vm_event interface and reinitialize.

Tamas

#4 lore raw

Re: [PATCH 1/3] xen/monitor: Control register values

Quoted 29 lines · Tamas K Lengyel
Quoted 17 lines · Jan Beulich
Quoted 3 lines · Tamas K Lengyel

Extend the monitor_op domctl to include option that enables controlling what values certain registers are permitted to hold by a monitor subscriber.

This needs a bit more explanation, especially for those of us who aren't that introspection savvy. For example, from the text here I didn't expect a simple bool control, but something where actual (register) values get passed back and forth.

Quoted 6 lines
xen/arch/x86/hvm/hvm.c +0 -0
@@ -2263,9 +2263,10 @@ int hvm_set_cr0(unsigned long value, bool may_defer)
     {
         ASSERT(v->arch.vm_event);
 
Quoted 10 lines

xen/arch/x86/hvm/hvm.c int hvm_set_cr0(unsigned long value, bool may_defer) · line 2269

Quoted 6 lines
xen/arch/x86/hvm/hvm.c +3 -2
-        if ( hvm_monitor_crX(CR0, value, old_value) )
+        if ( hvm_monitor_crX(CR0, value, old_value) &&
+             v->domain->arch.monitor.control_register_values )
         {
-            /* The actual write will occur in hvm_do_resume(), if permitted. */
+            /* The actual write will occur in hvm_do_resume, if permitted. */

Please can you leave alone this and the similar comments below. And for consistency _add_ parentheses to the one new instance you add?

Quoted 2 lines

I changed to because now it doesn't fit into the 80-line limit below, and then changed it everywhere _for_ consistency.

The 80-char limit is easy to deal with - wrap the line.

Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
xen/arch/x86/monitor.c +0 -0
@@ -144,7 +144,15 @@ int arch_monitor_domctl_event(struct domain *d,
                               struct xen_domctl_monitor_op *mop)
 {
     struct arch_domain *ad = &d->arch;
Quoted 8 lines

xen/arch/x86/monitor.c int arch_monitor_domctl_event(struct domain *d, · line 151

Quoted 6 lines
xen/arch/x86/monitor.c +5 -1
-    bool requested_status = (XEN_DOMCTL_MONITOR_OP_ENABLE == mop->op);
+    bool requested_status;
+
+    if ( XEN_DOMCTL_MONITOR_OP_CONTROL_REGISTERS == mop->op )
+    {
+        ad->monitor.control_register_values = true;

And there's no way to clear this flag again?

Quoted 1 line

There is. Disable the monitor vm_event interface and reinitialize.

Quite heavy handed, isn't it?

Jan

#5 lore raw

Re: [PATCH 1/3] xen/monitor: Control register values

Quoted 51 lines · Jan Beulich
Quoted 29 lines · Tamas K Lengyel
Quoted 17 lines · Jan Beulich
Quoted 3 lines · Tamas K Lengyel

Extend the monitor_op domctl to include option that enables controlling what values certain registers are permitted to hold by a monitor subscriber.

This needs a bit more explanation, especially for those of us who aren't that introspection savvy. For example, from the text here I didn't expect a simple bool control, but something where actual (register) values get passed back and forth.

Quoted 6 lines
xen/arch/x86/hvm/hvm.c +0 -0
@@ -2263,9 +2263,10 @@ int hvm_set_cr0(unsigned long value, bool may_defer)
     {
         ASSERT(v->arch.vm_event);
 
Quoted 10 lines

xen/arch/x86/hvm/hvm.c int hvm_set_cr0(unsigned long value, bool may_defer) · line 2269

Quoted 6 lines
xen/arch/x86/hvm/hvm.c +3 -2
-        if ( hvm_monitor_crX(CR0, value, old_value) )
+        if ( hvm_monitor_crX(CR0, value, old_value) &&
+             v->domain->arch.monitor.control_register_values )
         {
-            /* The actual write will occur in hvm_do_resume(), if permitted. */
+            /* The actual write will occur in hvm_do_resume, if permitted. */

Please can you leave alone this and the similar comments below. And for consistency _add_ parentheses to the one new instance you add?

Quoted 2 lines

I changed to because now it doesn't fit into the 80-line limit below, and then changed it everywhere _for_ consistency.

The 80-char limit is easy to deal with - wrap the line.

Quoted 15 lines
Quoted 14 lines

xen/arch/x86/monitor.c int arch_monitor_domctl_event(struct domain *d, · line 151

Quoted 12 lines
xen/arch/x86/monitor.c +5 -1
@@ -144,7 +144,15 @@ int arch_monitor_domctl_event(struct domain *d,
                               struct xen_domctl_monitor_op *mop)
 {
     struct arch_domain *ad = &d->arch;
-    bool requested_status = (XEN_DOMCTL_MONITOR_OP_ENABLE == mop->op);
+    bool requested_status;
+
+    if ( XEN_DOMCTL_MONITOR_OP_CONTROL_REGISTERS == mop->op )
+    {
+        ad->monitor.control_register_values = true;

And there's no way to clear this flag again?

Quoted 3 lines
Quoted 1 line

There is. Disable the monitor vm_event interface and reinitialize.

Quite heavy handed, isn't it?

Not really. It's perfectly suitable for what its used for. You either need this feature for the duration of your monitoring or you don't. There is no in-between.

Tamas

#6 lore raw

PATCH 2/3 xen/vm_event: add vm_event_check_pending_op

In reply to Tamas K Lengyel

Perform sanity checking when shutting vm_event down to determine whether it is safe to do so. Error out with -EAGAIN in case pending operations have been found for the domain.

---
 xen/arch/x86/vm_event.c        | 23 +++++++++++++++++++++++
 xen/common/vm_event.c          | 17 ++++++++++++++---
 xen/include/asm-arm/vm_event.h |  7 +++++++
 xen/include/asm-x86/vm_event.h |  2 ++
 4 files changed, 46 insertions(+), 3 deletions(-)
xen/arch/x86/vm_event.c +23 -0
index 848d69c1b0..558b7da4b1 100644
@@ -297,6 +297,29 @@ void vm_event_emulate_check(struct vcpu *v, vm_event_response_t *rsp)
     };
 }
 
+bool vm_event_check_pending_op(struct vcpu *v)
+{
+    struct monitor_write_data *w = &v->arch.vm_event->write_data;
+
+    if ( !v->arch.vm_event->sync_event )
+        return false;
+
+    if ( w->do_write.cr0 )
+        return true;
+    if ( w->do_write.cr3 )
+        return true;
+    if ( w->do_write.cr4 )
+        return true;
+    if ( w->do_write.msr )
+        return true;
+    if ( v->arch.vm_event->set_gprs )
+        return true;
+    if ( v->arch.vm_event->emulate_flags )
+        return true;
+
+    return false;
+}
+
 /*
  * Local variables:
  * mode: C
xen/common/vm_event.c +14 -3
index 127f2d58f1..2df327a42c 100644
@@ -183,6 +183,7 @@ static int vm_event_disable(struct domain *d, struct vm_event_domain **p_ved)
     if ( vm_event_check_ring(ved) )
     {
         struct vcpu *v;
+        bool pending_op = false;
 
         spin_lock(&ved->lock);
 
@@ -192,9 +193,6 @@ static int vm_event_disable(struct domain *d, struct vm_event_domain **p_ved)
             return -EBUSY;
         }
 
-        /* Free domU's event channel and leave the other one unbound */
-        free_xen_event_channel(d, ved->xen_port);
-
         /* Unblock all vCPUs */
         for_each_vcpu ( d, v )
         {
@@ -203,8 +201,21 @@ static int vm_event_disable(struct domain *d, struct vm_event_domain **p_ved)
                 vcpu_unpause(v);
                 ved->blocked--;
             }
+
+            if ( vm_event_check_pending_op(v) )
+                pending_op = true;
         }
 
+        /* vm_event ops are still pending until vCPUs get scheduled */
+        if ( pending_op )
+        {
+            spin_unlock(&ved->lock);
+            return -EAGAIN;
+        }
+
+        /* Free domU's event channel and leave the other one unbound */
+        free_xen_event_channel(d, ved->xen_port);
+
         destroy_ring_for_helper(&ved->ring_page, ved->ring_pg_struct);
 
         vm_event_cleanup_domain(d);
xen/include/asm-arm/vm_event.h +7 -0
index 14d1d341cc..5cbc9c6dc2 100644
@@ -58,4 +58,11 @@ void vm_event_sync_event(struct vcpu *v, bool value)
     /* Not supported on ARM. */
 }
 
+static inline
+bool vm_event_check_pending_op(struct vcpu *v)
+{
+    /* Not supported on ARM. */
+    return false;
+}
+
 #endif /* __ASM_ARM_VM_EVENT_H__ */
xen/include/asm-x86/vm_event.h +2 -0
index 785e741fba..9c5ce3129c 100644
@@ -54,4 +54,6 @@ void vm_event_emulate_check(struct vcpu *v, vm_event_response_t *rsp);
 
 void vm_event_sync_event(struct vcpu *v, bool value);
 
+bool vm_event_check_pending_op(struct vcpu *v);
+
 #endif /* __ASM_X86_VM_EVENT_H__ */
Signature · 1 line
2.26.1
#7 lore raw

PATCH 3/3 xen/vm_event: Add safe to disable vm_event

In reply to Tamas K Lengyel

Instead of having to repeatedly try to disable vm_events, request a specific vm_event to be sent when the domain is safe to continue with shutting down the vm_event interface.

---
 xen/arch/x86/hvm/hvm.c            | 38 ++++++++++++++++++++++++++-----
 xen/arch/x86/hvm/monitor.c        | 14 ++++++++++++
 xen/arch/x86/monitor.c            | 13 +++++++++++
 xen/include/asm-x86/domain.h      |  1 +
 xen/include/asm-x86/hvm/monitor.h |  1 +
 xen/include/public/domctl.h       |  2 ++
 xen/include/public/vm_event.h     |  8 +++++++
 7 files changed, 71 insertions(+), 6 deletions(-)
xen/arch/x86/hvm/hvm.c +32 -6
index 063f8ddc18..50c67e7b8e 100644
@@ -563,15 +563,41 @@ void hvm_do_resume(struct vcpu *v)
         v->arch.hvm.inject_event.vector = HVM_EVENT_VECTOR_UNSET;
     }
 
-    if ( unlikely(v->arch.vm_event) && v->arch.monitor.next_interrupt_enabled )
+    if ( unlikely(v->arch.vm_event) )
     {
-        struct x86_event info;
+        struct domain *d = v->domain;
+
+        if ( v->arch.monitor.next_interrupt_enabled )
+        {
+            struct x86_event info;
+
+            if ( hvm_get_pending_event(v, &info) )
+            {
+                hvm_monitor_interrupt(info.vector, info.type, info.error_code,
+                                      info.cr2);
+                v->arch.monitor.next_interrupt_enabled = false;
+            }
+        }
 
-        if ( hvm_get_pending_event(v, &info) )
+        if ( d->arch.monitor.safe_to_disable )
         {
-            hvm_monitor_interrupt(info.vector, info.type, info.error_code,
-                                  info.cr2);
-            v->arch.monitor.next_interrupt_enabled = false;
+            struct vcpu *check_vcpu;
+            bool pending_op = false;
+
+            for_each_vcpu ( d, check_vcpu )
+            {
+                if ( vm_event_check_pending_op(check_vcpu) )
+                {
+                    pending_op = true;
+                    break;
+                }
+            }
+
+            if ( !pending_op )
+            {
+                hvm_monitor_safe_to_disable();
+                d->arch.monitor.safe_to_disable = false;
+            }
         }
     }
 }
xen/arch/x86/hvm/monitor.c +14 -0
index f5d89e71d1..8e67dd1a0b 100644
@@ -300,6 +300,20 @@ bool hvm_monitor_check_p2m(unsigned long gla, gfn_t gfn, uint32_t pfec,
     return monitor_traps(curr, true, &req) >= 0;
 }
 
+bool hvm_monitor_safe_to_disable(void)
+{
+    struct vcpu *curr = current;
+    struct arch_domain *ad = &curr->domain->arch;
+    vm_event_request_t req = {};
+
+    if ( !ad->monitor.safe_to_disable )
+        return 0;
+
+    req.reason = VM_EVENT_REASON_SAFE_TO_DISABLE;
+
+    return monitor_traps(curr, 0, &req);
+}
+
 /*
  * Local variables:
  * mode: C
xen/arch/x86/monitor.c +13 -0
index 1517a97f50..86e0ba2fbc 100644
@@ -339,6 +339,19 @@ int arch_monitor_domctl_event(struct domain *d,
         break;
     }
 
+    case XEN_DOMCTL_MONITOR_EVENT_SAFE_TO_DISABLE:
+    {
+        bool old_status = ad->monitor.safe_to_disable;
+
+        if ( unlikely(old_status == requested_status) )
+            return -EEXIST;
+
+        domain_pause(d);
+        ad->monitor.safe_to_disable = requested_status;
+        domain_unpause(d);
+        break;
+    }
+
     default:
         /*
          * Should not be reached unless arch_monitor_get_capabilities() is
xen/include/asm-x86/domain.h +1 -0
index d890ab7a22..948b750c71 100644
@@ -417,6 +417,7 @@ struct arch_domain
          */
         unsigned int inguest_pagefault_disabled                            : 1;
         unsigned int control_register_values                               : 1;
+        unsigned int safe_to_disable                                       : 1;
         struct monitor_msr_bitmap *msr_bitmap;
         uint64_t write_ctrlreg_mask[4];
     } monitor;
xen/include/asm-x86/hvm/monitor.h +1 -0
index 66de24cb75..194e2f857e 100644
@@ -52,6 +52,7 @@ bool hvm_monitor_emul_unimplemented(void);
 
 bool hvm_monitor_check_p2m(unsigned long gla, gfn_t gfn, uint32_t pfec,
                            uint16_t kind);
+bool hvm_monitor_safe_to_disable(void);
 
 #endif /* __ASM_X86_HVM_MONITOR_H__ */
 
xen/include/public/domctl.h +2 -0
index cbcd25f12c..247e809a6c 100644
@@ -1040,6 +1040,8 @@ struct xen_domctl_psr_cmt_op {
 #define XEN_DOMCTL_MONITOR_EVENT_EMUL_UNIMPLEMENTED    10
 /* Enabled by default */
 #define XEN_DOMCTL_MONITOR_EVENT_INGUEST_PAGEFAULT     11
+/* Always async, disables automaticaly on first event */
+#define XEN_DOMCTL_MONITOR_EVENT_SAFE_TO_DISABLE       12
 
 struct xen_domctl_monitor_op {
     uint32_t op; /* XEN_DOMCTL_MONITOR_OP_* */
xen/include/public/vm_event.h +8 -0
index fdd3ad8a30..b66d2a8634 100644
@@ -159,6 +159,14 @@
 #define VM_EVENT_REASON_DESCRIPTOR_ACCESS       13
 /* Current instruction is not implemented by the emulator */
 #define VM_EVENT_REASON_EMUL_UNIMPLEMENTED      14
+/*
+ * When shutting down vm_event it may not be immediately safe to complete the
+ * process as some vCPUs may be pending synchronization. This async event
+ * type can be used to receive a notification when its safe to finish disabling
+ * the vm_event interface. All other event types need to be disabled before
+ * registering to this one.
+ */
+#define VM_EVENT_REASON_SAFE_TO_DISABLE         15
 
 /* Supported values for the vm_event_write_ctrlreg index. */
 #define VM_EVENT_X86_CR0    0
Signature · 1 line
2.26.1
#8 lore raw

Re: [PATCH 3/3] xen/vm_event: Add safe to disable vm_event

xen/arch/x86/hvm/hvm.c void hvm_do_resume(struct vcpu *v) · line 568

Quoted 11 lines · Tamas K Lengyel
xen/arch/x86/hvm/hvm.c +2 -2
@@ -563,15 +563,41 @@ void hvm_do_resume(struct vcpu *v)
         v->arch.hvm.inject_event.vector = HVM_EVENT_VECTOR_UNSET;
     }
 
-    if ( unlikely(v->arch.vm_event) && v->arch.monitor.next_interrupt_enabled )
+    if ( unlikely(v->arch.vm_event) )
     {
-        struct x86_event info;
+        struct domain *d = v->domain;

const

Quoted 9 lines
xen/arch/x86/hvm/hvm.c +9 -0
+        if ( v->arch.monitor.next_interrupt_enabled )
+        {
+            struct x86_event info;
+
+            if ( hvm_get_pending_event(v, &info) )
+            {
+                hvm_monitor_interrupt(info.vector, info.type, info.error_code,
+                                      info.cr2);
+                v->arch.monitor.next_interrupt_enabled = false;

xen/arch/x86/hvm/hvm.c void hvm_do_resume(struct vcpu *v) · line 584

Quoted 10 lines
xen/arch/x86/hvm/hvm.c +4 -4
+            }
+        }
 
-        if ( hvm_get_pending_event(v, &info) )
+        if ( d->arch.monitor.safe_to_disable )
         {
-            hvm_monitor_interrupt(info.vector, info.type, info.error_code,
-                                  info.cr2);
-            v->arch.monitor.next_interrupt_enabled = false;
+            struct vcpu *check_vcpu;

const again, requiring a respective adjustment to patch 2.

Quoted 4 lines
xen/arch/x86/hvm/hvm.c +4 -0
+            bool pending_op = false;
+
+            for_each_vcpu ( d, check_vcpu )
+            {

xen/arch/x86/hvm/hvm.c void hvm_do_resume(struct vcpu *v) · line 598

Quoted 10 lines
xen/arch/x86/hvm/hvm.c +10 -0
+                if ( vm_event_check_pending_op(check_vcpu) )
+                {
+                    pending_op = true;
+                    break;
+                }
+            }
+
+            if ( !pending_op )
+            {
+                hvm_monitor_safe_to_disable();

This new function returns bool without the caller caring about the return value.

Jan

#9 lore raw

Re: [PATCH 3/3] xen/vm_event: Add safe to disable vm_event

Quoted 47 lines · Jan Beulich

xen/arch/x86/hvm/hvm.c void hvm_do_resume(struct vcpu *v) · line 568

Quoted 11 lines · Tamas K Lengyel
xen/arch/x86/hvm/hvm.c +2 -2
@@ -563,15 +563,41 @@ void hvm_do_resume(struct vcpu *v)
         v->arch.hvm.inject_event.vector = HVM_EVENT_VECTOR_UNSET;
     }
 
-    if ( unlikely(v->arch.vm_event) && v->arch.monitor.next_interrupt_enabled )
+    if ( unlikely(v->arch.vm_event) )
     {
-        struct x86_event info;
+        struct domain *d = v->domain;

const

Quoted 9 lines
xen/arch/x86/hvm/hvm.c +9 -0
+        if ( v->arch.monitor.next_interrupt_enabled )
+        {
+            struct x86_event info;
+
+            if ( hvm_get_pending_event(v, &info) )
+            {
+                hvm_monitor_interrupt(info.vector, info.type, info.error_code,
+                                      info.cr2);
+                v->arch.monitor.next_interrupt_enabled = false;

xen/arch/x86/hvm/hvm.c void hvm_do_resume(struct vcpu *v) · line 584

Quoted 10 lines
xen/arch/x86/hvm/hvm.c +4 -4
+            }
+        }
 
-        if ( hvm_get_pending_event(v, &info) )
+        if ( d->arch.monitor.safe_to_disable )
         {
-            hvm_monitor_interrupt(info.vector, info.type, info.error_code,
-                                  info.cr2);
-            v->arch.monitor.next_interrupt_enabled = false;
+            struct vcpu *check_vcpu;

const again, requiring a respective adjustment to patch 2.

Quoted 9 lines
xen/arch/x86/hvm/hvm.c +9 -0
+            bool pending_op = false;
+
+            for_each_vcpu ( d, check_vcpu )
+            {
+                if ( vm_event_check_pending_op(check_vcpu) )
+                {
+                    pending_op = true;
+                    break;
+                }
Quoted 8 lines

xen/arch/x86/hvm/hvm.c void hvm_do_resume(struct vcpu *v) · line 598

Quoted 5 lines
xen/arch/x86/hvm/hvm.c +5 -0
+            }
+
+            if ( !pending_op )
+            {
+                hvm_monitor_safe_to_disable();

This new function returns bool without the caller caring about the return value.

Yea, there is actually nothing to be done if the event can't be sent for whatever reason, I guess I'll just turn it to void.

Tamas

#10 lore raw

Re: [PATCH 3/3] xen/vm_event: Add safe to disable vm_event

In reply to Jan Beulich
Quoted 8 lines · Jan Beulich
Quoted 6 lines · Tamas K Lengyel
xen/arch/x86/hvm/hvm.c +0 -0
@@ -563,15 +563,41 @@ void hvm_do_resume(struct vcpu *v)
         v->arch.hvm.inject_event.vector = HVM_EVENT_VECTOR_UNSET;
     }
 
Quoted 7 lines

xen/arch/x86/hvm/hvm.c void hvm_do_resume(struct vcpu *v) · line 568

Quoted 5 lines
xen/arch/x86/hvm/hvm.c +2 -2
-    if ( unlikely(v->arch.vm_event) && v->arch.monitor.next_interrupt_enabled )
+    if ( unlikely(v->arch.vm_event) )
     {
-        struct x86_event info;
+        struct domain *d = v->domain;

const

This can't be const, we disable the safe_to_disable option below after sending the one-shot async event.

Tamas