Skip to contentSkip to conversation

xen: fix/enhance temporary vcpu pinning

PATCH Superseded Juergen Gross 4 participants 27 messages Jul 23, 2019

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover xen: fix/enhance temporary vcpu pinning

While trying to handle temporary vcpu pinnings in a sane way in my core scheduling series I stumbled over a bug and found a nice way to simplify the temporary pinning cases.

I'm sending the two patches independently from my core scheduling series as they should be considered even without core scheduling.

Juergen Gross (2):
xen/sched: fix locking in restore_vcpu_affinity()
xen: merge temporary vcpu pinning scenarios

 xen/arch/x86/pv/traps.c | 20 +-------------------
 xen/arch/x86/traps.c    |  8 ++------
 xen/common/domain.c     |  4 +---
 xen/common/schedule.c   | 40 ++++++++++++++++++++++++++++------------
 xen/common/wait.c       | 26 ++++++++------------------
 xen/include/xen/sched.h |  8 +++++---
 6 files changed, 45 insertions(+), 61 deletions(-)
Signature · 7 lines
2.16.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH 1/2 xen/sched: fix locking in restore_vcpu_affinity()

Commit 0763cd2687897b55e7 ("xen/sched: don't disable scheduler on cpus during suspend") removed a lock in restore_vcpu_affinity() which needs to stay: cpumask_scratch_cpu() must be protected by the scheduler lock. restore_vcpu_affinity() is being called by thaw_domains(), so with multiple domains in the system another domain might already be running and the scheduler might make use of cpumask_scratch_cpu() already.

---
 xen/common/schedule.c | 5 +++++
 1 file changed, 5 insertions(+)
xen/common/schedule.c +5 -0
index 25f6ab388d..89bc259ae4 100644
@@ -708,6 +708,8 @@ void restore_vcpu_affinity(struct domain *d)
          * set v->processor of each of their vCPUs to something that will
          * make sense for the scheduler of the cpupool in which they are in.
          */
+        lock = vcpu_schedule_lock_irq(v);
+
         cpumask_and(cpumask_scratch_cpu(cpu), v->cpu_hard_affinity,
                     cpupool_domain_cpumask(d));
         if ( cpumask_empty(cpumask_scratch_cpu(cpu)) )
@@ -731,6 +733,9 @@ void restore_vcpu_affinity(struct domain *d)
 
         v->processor = cpumask_any(cpumask_scratch_cpu(cpu));
 
+        spin_unlock_irq(lock);
+
+	/* v->processor might have changed, so reacquire the lock. */
         lock = vcpu_schedule_lock_irq(v);
         v->processor = sched_pick_cpu(vcpu_scheduler(v), v);
         spin_unlock_irq(lock);
Signature · 7 lines
2.16.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v1 1/2
#2 lore raw

Re: [Xen-devel] [PATCH 1/2] xen/sched: fix locking in restore_vcpu_affinity()

Quoted 7 lines · Juergen Gross

Commit 0763cd2687897b55e7 ("xen/sched: don't disable scheduler on cpus
during suspend") removed a lock in restore_vcpu_affinity() which needs
to stay: cpumask_scratch_cpu() must be protected by the scheduler lock.

And indeed I recall the thing looked weird, and bringing it up... but then I also managed to convince myself that it wasn't a problem, which apparently is. :-(

Sorry for having overlooked that.

Quoted 7 lines

restore_vcpu_affinity() is being called by thaw_domains(), so with multiple domains in the system another domain might already be running and the scheduler might make use of cpumask_scratch_cpu() already.

Thanks and Regards

Signature · 6 lines
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw

Re: [Xen-devel] [PATCH 1/2] xen/sched: fix locking in restore_vcpu_affinity()

In reply to Juergen Gross
Quoted 12 lines · Juergen Gross
xen/common/schedule.c +2 -0
@@ -708,6 +708,8 @@ void restore_vcpu_affinity(struct domain *d)
           * set v->processor of each of their vCPUs to something that will
           * make sense for the scheduler of the cpupool in which they are in.
           */
+        lock = vcpu_schedule_lock_irq(v);
+
          cpumask_and(cpumask_scratch_cpu(cpu), v->cpu_hard_affinity,
                      cpupool_domain_cpumask(d));
          if ( cpumask_empty(cpumask_scratch_cpu(cpu)) )

xen/common/schedule.c void restore_vcpu_affinity(struct domain *d) · line 738

Quoted 6 lines
xen/common/schedule.c +3 -0
@@ -731,6 +733,9 @@ void restore_vcpu_affinity(struct domain *d)
 
          v->processor = cpumask_any(cpumask_scratch_cpu(cpu));
 
+        spin_unlock_irq(lock);
+
+	/* v->processor might have changed, so reacquire the lock. */

Whoever commits this (perhaps me, later today) will want to replace the hard tab here.

Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw

Re: [Xen-devel] [PATCH 1/2] xen/sched: fix locking in restore_vcpu_affinity()

Quoted 14 lines · Jan Beulich
Quoted 13 lines · Juergen Gross
xen/common/schedule.c +2 -0
@@ -708,6 +708,8 @@ void restore_vcpu_affinity(struct domain *d)
            * set v->processor of each of their vCPUs to something that will
            * make sense for the scheduler of the cpupool in which they are in.
            */
+        lock = vcpu_schedule_lock_irq(v);
+
           cpumask_and(cpumask_scratch_cpu(cpu), v->cpu_hard_affinity,
                       cpupool_domain_cpumask(d));
           if ( cpumask_empty(cpumask_scratch_cpu(cpu)) )
@@ -731,6 +733,9 @@ void restore_vcpu_affinity(struct domain *d)
 
Quoted 8 lines

xen/common/schedule.c void restore_vcpu_affinity(struct domain *d) · line 738

Quoted 5 lines
xen/common/schedule.c +3 -0
           v->processor = cpumask_any(cpumask_scratch_cpu(cpu));
 
+        spin_unlock_irq(lock);
+
+	/* v->processor might have changed, so reacquire the lock. */

Whoever commits this (perhaps me, later today) will want to replace the hard tab here.

Oh, I'm sorry for that!

We really want checkpatch in Xen!

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw

Re: [Xen-devel] [PATCH 1/2] xen/sched: fix locking in restore_vcpu_affinity()

In reply to Jan Beulich
Quoted 14 lines · Jan Beulich
Quoted 13 lines · Juergen Gross
xen/common/schedule.c +2 -0
@@ -708,6 +708,8 @@ void restore_vcpu_affinity(struct domain *d)
           * set v->processor of each of their vCPUs to something that will
           * make sense for the scheduler of the cpupool in which they are in.
           */
+        lock = vcpu_schedule_lock_irq(v);
+
          cpumask_and(cpumask_scratch_cpu(cpu), v->cpu_hard_affinity,
                      cpupool_domain_cpumask(d));
          if ( cpumask_empty(cpumask_scratch_cpu(cpu)) )
@@ -731,6 +733,9 @@ void restore_vcpu_affinity(struct domain *d)
 
Quoted 7 lines

xen/common/schedule.c void restore_vcpu_affinity(struct domain *d) · line 738

Quoted 5 lines
xen/common/schedule.c +3 -0
          v->processor = cpumask_any(cpumask_scratch_cpu(cpu));
 
+        spin_unlock_irq(lock);
+
+	/* v->processor might have changed, so reacquire the lock. */

Whoever commits this (perhaps me, later today) will want to replace the hard tab here.

I've already committed this, and did fix up the tab.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw

PATCH 2/2 xen: merge temporary vcpu pinning scenarios

In reply to Juergen Gross

Today there are three scenarios which are pinning vcpus temporarily to a single physical cpu:

- NMI/MCE injection into PV domains
- wait_event() handling
- vcpu_pin_override() handling

Each of those cases are handled independently today using their own temporary cpumask to save the old affinity settings.

The three cases can be combined as the two latter cases will only pin a vcpu to the physical cpu it is already running on, while vcpu_pin_override() is allowed to fail.

So merge the three temporary pinning scenarios by only using one cpumask and a per-vcpu bitmask for specifying which of the three scenarios is currently active (they are allowed to nest).

Note that we don't need to call domain_update_node_affinity() as we are only pinning for a brief period of time.

---
 xen/arch/x86/pv/traps.c | 20 +-------------------
 xen/arch/x86/traps.c    |  8 ++------
 xen/common/domain.c     |  4 +---
 xen/common/schedule.c   | 35 +++++++++++++++++++++++------------
 xen/common/wait.c       | 26 ++++++++------------------
 xen/include/xen/sched.h |  8 +++++---
 6 files changed, 40 insertions(+), 61 deletions(-)
xen/arch/x86/pv/traps.c +1 -19
index 1740784ff2..37dac300ba 100644
@@ -151,25 +151,7 @@ static void nmi_mce_softirq(void)
 
     BUG_ON(st->vcpu == NULL);
 
-    /*
-     * Set the tmp value unconditionally, so that the check in the iret
-     * hypercall works.
-     */
-    cpumask_copy(st->vcpu->cpu_hard_affinity_tmp,
-                 st->vcpu->cpu_hard_affinity);
-
-    if ( (cpu != st->processor) ||
-         (st->processor != st->vcpu->processor) )
-    {
-
-        /*
-         * We are on a different physical cpu.  Make sure to wakeup the vcpu on
-         * the specified processor.
-         */
-        vcpu_set_hard_affinity(st->vcpu, cpumask_of(st->processor));
-
-        /* Affinity is restored in the iret hypercall. */
-    }
+    vcpu_set_tmp_affinity(st->vcpu, st->processor, VCPU_AFFINITY_NMI);
 
     /*
      * Only used to defer wakeup of domain/vcpu to a safe (non-NMI/MCE)
xen/arch/x86/traps.c +2 -6
index 25b4b47e5e..22bb790858 100644
@@ -1601,12 +1601,8 @@ void async_exception_cleanup(struct vcpu *curr)
         return;
 
     /* Restore affinity.  */
-    if ( !cpumask_empty(curr->cpu_hard_affinity_tmp) &&
-         !cpumask_equal(curr->cpu_hard_affinity_tmp, curr->cpu_hard_affinity) )
-    {
-        vcpu_set_hard_affinity(curr, curr->cpu_hard_affinity_tmp);
-        cpumask_clear(curr->cpu_hard_affinity_tmp);
-    }
+    if ( curr->affinity_broken & VCPU_AFFINITY_NMI )
+        vcpu_set_tmp_affinity(curr, -1, VCPU_AFFINITY_NMI);
 
     if ( !(curr->async_exception_mask & (curr->async_exception_mask - 1)) )
         trap = __scanbit(curr->async_exception_mask, VCPU_TRAP_NONE);
xen/common/domain.c +1 -3
index 55aa759b75..e8e850796e 100644
@@ -133,7 +133,6 @@ static void vcpu_info_reset(struct vcpu *v)
 static void vcpu_destroy(struct vcpu *v)
 {
     free_cpumask_var(v->cpu_hard_affinity);
-    free_cpumask_var(v->cpu_hard_affinity_tmp);
     free_cpumask_var(v->cpu_hard_affinity_saved);
     free_cpumask_var(v->cpu_soft_affinity);
 
@@ -161,7 +160,6 @@ struct vcpu *vcpu_create(
     grant_table_init_vcpu(v);
 
     if ( !zalloc_cpumask_var(&v->cpu_hard_affinity) ||
-         !zalloc_cpumask_var(&v->cpu_hard_affinity_tmp) ||
          !zalloc_cpumask_var(&v->cpu_hard_affinity_saved) ||
          !zalloc_cpumask_var(&v->cpu_soft_affinity) )
         goto fail;
@@ -1269,7 +1267,7 @@ int vcpu_reset(struct vcpu *v)
     v->async_exception_mask = 0;
     memset(v->async_exception_state, 0, sizeof(v->async_exception_state));
 #endif
-    cpumask_clear(v->cpu_hard_affinity_tmp);
+    v->affinity_broken = 0;
     clear_bit(_VPF_blocked, &v->pause_flags);
     clear_bit(_VPF_in_reset, &v->pause_flags);
 
xen/common/schedule.c +23 -12
index 89bc259ae4..d4de74f9c8 100644
@@ -1106,47 +1106,58 @@ void watchdog_domain_destroy(struct domain *d)
         kill_timer(&d->watchdog_timer[i]);
 }
 
-int vcpu_pin_override(struct vcpu *v, int cpu)
+int vcpu_set_tmp_affinity(struct vcpu *v, int cpu, uint8_t reason)
 {
     spinlock_t *lock;
     int ret = -EINVAL;
+    bool migrate;
 
     lock = vcpu_schedule_lock_irq(v);
 
     if ( cpu < 0 )
     {
-        if ( v->affinity_broken )
+        if ( v->affinity_broken & reason )
         {
-            sched_set_affinity(v, v->cpu_hard_affinity_saved, NULL);
-            v->affinity_broken = 0;
             ret = 0;
+            v->affinity_broken &= ~reason;
         }
+        if ( !ret && !v->affinity_broken )
+            sched_set_affinity(v, v->cpu_hard_affinity_saved, NULL);
     }
     else if ( cpu < nr_cpu_ids )
     {
-        if ( v->affinity_broken )
+        if ( (v->affinity_broken & reason) ||
+             (v->affinity_broken && v->processor != cpu) )
             ret = -EBUSY;
         else if ( cpumask_test_cpu(cpu, VCPU2ONLINE(v)) )
         {
-            cpumask_copy(v->cpu_hard_affinity_saved, v->cpu_hard_affinity);
-            v->affinity_broken = 1;
-            sched_set_affinity(v, cpumask_of(cpu), NULL);
+            if ( !v->affinity_broken )
+            {
+                cpumask_copy(v->cpu_hard_affinity_saved, v->cpu_hard_affinity);
+                sched_set_affinity(v, cpumask_of(cpu), NULL);
+            }
+            v->affinity_broken |= reason;
             ret = 0;
         }
     }
 
-    if ( ret == 0 )
+    migrate = !ret && !cpumask_test_cpu(v->processor, v->cpu_hard_affinity);
+    if ( migrate )
         vcpu_migrate_start(v);
 
     vcpu_schedule_unlock_irq(lock, v);
 
-    domain_update_node_affinity(v->domain);
-
-    vcpu_migrate_finish(v);
+    if ( migrate )
+        vcpu_migrate_finish(v);
 
     return ret;
 }
 
+int vcpu_pin_override(struct vcpu *v, int cpu)
+{
+    return vcpu_set_tmp_affinity(v, cpu, VCPU_AFFINITY_OVERRIDE);
+}
+
 typedef long ret_t;
 
 #endif /* !COMPAT */
xen/common/wait.c +8 -18
index 4f830a14e8..9f9ad033b3 100644
@@ -34,8 +34,6 @@ struct waitqueue_vcpu {
      */
     void *esp;
     char *stack;
-    cpumask_t saved_affinity;
-    unsigned int wakeup_cpu;
 #endif
 };
 
@@ -131,9 +129,7 @@ static void __prepare_to_wait(struct waitqueue_vcpu *wqv)
     ASSERT(wqv->esp == 0);
 
     /* Save current VCPU affinity; force wakeup on *this* CPU only. */
-    wqv->wakeup_cpu = smp_processor_id();
-    cpumask_copy(&wqv->saved_affinity, curr->cpu_hard_affinity);
-    if ( vcpu_set_hard_affinity(curr, cpumask_of(wqv->wakeup_cpu)) )
+    if ( vcpu_set_tmp_affinity(curr, smp_processor_id(), VCPU_AFFINITY_WAIT) )
     {
         gdprintk(XENLOG_ERR, "Unable to set vcpu affinity\n");
         domain_crash(current->domain);
@@ -182,30 +178,24 @@ static void __prepare_to_wait(struct waitqueue_vcpu *wqv)
 static void __finish_wait(struct waitqueue_vcpu *wqv)
 {
     wqv->esp = NULL;
-    (void)vcpu_set_hard_affinity(current, &wqv->saved_affinity);
+    vcpu_set_tmp_affinity(current, -1, VCPU_AFFINITY_WAIT);
 }
 
 void check_wakeup_from_wait(void)
 {
-    struct waitqueue_vcpu *wqv = current->waitqueue_vcpu;
+    struct vcpu *curr = current;
+    struct waitqueue_vcpu *wqv = curr->waitqueue_vcpu;
 
     ASSERT(list_empty(&wqv->list));
 
     if ( likely(wqv->esp == NULL) )
         return;
 
-    /* Check if we woke up on the wrong CPU. */
-    if ( unlikely(smp_processor_id() != wqv->wakeup_cpu) )
+    /* Check if we are still pinned. */
+    if ( unlikely(!(curr->affinity_broken & VCPU_AFFINITY_WAIT)) )
     {
-        /* Re-set VCPU affinity and re-enter the scheduler. */
-        struct vcpu *curr = current;
-        cpumask_copy(&wqv->saved_affinity, curr->cpu_hard_affinity);
-        if ( vcpu_set_hard_affinity(curr, cpumask_of(wqv->wakeup_cpu)) )
-        {
-            gdprintk(XENLOG_ERR, "Unable to set vcpu affinity\n");
-            domain_crash(current->domain);
-        }
-        wait(); /* takes us back into the scheduler */
+        gdprintk(XENLOG_ERR, "vcpu affinity lost\n");
+        domain_crash(current->domain);
     }
 
     /*
xen/include/xen/sched.h +5 -3
index b40c8fd138..721c429454 100644
@@ -200,7 +200,10 @@ struct vcpu
     /* VCPU is paused following shutdown request (d->is_shutting_down)? */
     bool             paused_for_shutdown;
     /* VCPU need affinity restored */
-    bool             affinity_broken;
+    uint8_t          affinity_broken;
+#define VCPU_AFFINITY_OVERRIDE    0x01
+#define VCPU_AFFINITY_NMI         0x02
+#define VCPU_AFFINITY_WAIT        0x04
 
     /* A hypercall has been preempted. */
     bool             hcall_preempted;
@@ -246,8 +249,6 @@ struct vcpu
     /* Bitmask of CPUs on which this VCPU may run. */
     cpumask_var_t    cpu_hard_affinity;
     /* Used to change affinity temporarily. */
-    cpumask_var_t    cpu_hard_affinity_tmp;
-    /* Used to restore affinity across S3. */
     cpumask_var_t    cpu_hard_affinity_saved;
 
     /* Bitmask of CPUs on which this VCPU prefers to run. */
@@ -875,6 +876,7 @@ int cpu_disable_scheduler(unsigned int cpu);
 /* We need it in dom0_setup_vcpu */
 void sched_set_affinity(struct vcpu *v, const cpumask_t *hard,
                         const cpumask_t *soft);
+int vcpu_set_tmp_affinity(struct vcpu *v, int cpu, uint8_t reason);
 int vcpu_set_hard_affinity(struct vcpu *v, const cpumask_t *affinity);
 int vcpu_set_soft_affinity(struct vcpu *v, const cpumask_t *affinity);
 void restore_vcpu_affinity(struct domain *d);
Signature · 7 lines
2.16.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#7 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

Quoted 49 lines · Juergen Gross

Today there are three scenarios which are pinning vcpus temporarily to a single physical cpu:

- NMI/MCE injection into PV domains
- wait_event() handling
- vcpu_pin_override() handling

Each of those cases are handled independently today using their own temporary cpumask to save the old affinity settings.

The three cases can be combined as the two latter cases will only pin a vcpu to the physical cpu it is already running on, while vcpu_pin_override() is allowed to fail.

So merge the three temporary pinning scenarios by only using one cpumask and a per-vcpu bitmask for specifying which of the three scenarios is currently active (they are allowed to nest).

Note that we don't need to call domain_update_node_affinity() as we are only pinning for a brief period of time.

---
 xen/arch/x86/pv/traps.c | 20 +-------------------
 xen/arch/x86/traps.c    |  8 ++------
 xen/common/domain.c     |  4 +---
 xen/common/schedule.c   | 35 +++++++++++++++++++++++------------
 xen/common/wait.c       | 26 ++++++++------------------
 xen/include/xen/sched.h |  8 +++++---
 6 files changed, 40 insertions(+), 61 deletions(-)
xen/arch/x86/pv/traps.c +0 -10
index 1740784ff2..37dac300ba 100644
@@ -151,25 +151,7 @@ static void nmi_mce_softirq(void)
 
     BUG_ON(st->vcpu == NULL);
 
-    /*
-     * Set the tmp value unconditionally, so that the check in the iret
-     * hypercall works.
-     */
-    cpumask_copy(st->vcpu->cpu_hard_affinity_tmp,
-                 st->vcpu->cpu_hard_affinity);
-
-    if ( (cpu != st->processor) ||
-         (st->processor != st->vcpu->processor) )
-    {

xen/arch/x86/pv/traps.c static void nmi_mce_softirq(void) · line 154

Quoted 10 lines
xen/arch/x86/pv/traps.c +1 -9
-
-        /*
-         * We are on a different physical cpu.  Make sure to wakeup the vcpu on
-         * the specified processor.
-         */
-        vcpu_set_hard_affinity(st->vcpu, cpumask_of(st->processor));
-
-        /* Affinity is restored in the iret hypercall. */
-    }
+    vcpu_set_tmp_affinity(st->vcpu, st->processor, VCPU_AFFINITY_NMI);

Please can we keep the comment explaining where the affinity is restored, which is a disguised explanation of why it is PV-only.

Quoted 58 lines
xen/common/schedule.c +16 -9
index 89bc259ae4..d4de74f9c8 100644
@@ -1106,47 +1106,58 @@ void watchdog_domain_destroy(struct domain *d)
         kill_timer(&d->watchdog_timer[i]);
 }
 
-int vcpu_pin_override(struct vcpu *v, int cpu)
+int vcpu_set_tmp_affinity(struct vcpu *v, int cpu, uint8_t reason)
 {
     spinlock_t *lock;
     int ret = -EINVAL;
+    bool migrate;
 
     lock = vcpu_schedule_lock_irq(v);
 
     if ( cpu < 0 )
     {
-        if ( v->affinity_broken )
+        if ( v->affinity_broken & reason )
         {
-            sched_set_affinity(v, v->cpu_hard_affinity_saved, NULL);
-            v->affinity_broken = 0;
             ret = 0;
+            v->affinity_broken &= ~reason;
         }
+        if ( !ret && !v->affinity_broken )
+            sched_set_affinity(v, v->cpu_hard_affinity_saved, NULL);
     }
     else if ( cpu < nr_cpu_ids )
     {
-        if ( v->affinity_broken )
+        if ( (v->affinity_broken & reason) ||
+             (v->affinity_broken && v->processor != cpu) )
             ret = -EBUSY;
         else if ( cpumask_test_cpu(cpu, VCPU2ONLINE(v)) )
         {
-            cpumask_copy(v->cpu_hard_affinity_saved, v->cpu_hard_affinity);
-            v->affinity_broken = 1;
-            sched_set_affinity(v, cpumask_of(cpu), NULL);
+            if ( !v->affinity_broken )
+            {
+                cpumask_copy(v->cpu_hard_affinity_saved, v->cpu_hard_affinity);
+                sched_set_affinity(v, cpumask_of(cpu), NULL);
+            }
+            v->affinity_broken |= reason;
             ret = 0;
         }
     }
 
-    if ( ret == 0 )
+    migrate = !ret && !cpumask_test_cpu(v->processor, v->cpu_hard_affinity);
+    if ( migrate )
         vcpu_migrate_start(v);
 
     vcpu_schedule_unlock_irq(lock, v);
 

xen/common/schedule.c void watchdog_domain_destroy(struct domain *d) · line 1156

Quoted 10 lines
xen/common/schedule.c +3 -3
-    domain_update_node_affinity(v->domain);
-
-    vcpu_migrate_finish(v);
+    if ( migrate )
+        vcpu_migrate_finish(v);
 
     return ret;
 }
 
+int vcpu_pin_override(struct vcpu *v, int cpu)

There are exactly two callers of vcpu_pin_override().  I'd take the opportunity to make vcpu_set_tmp_affinity() the single API call for adjusting affinity.

Quoted 36 lines
xen/common/schedule.c +4 -0
+{
+    return vcpu_set_tmp_affinity(v, cpu, VCPU_AFFINITY_OVERRIDE);
+}
+
 typedef long ret_t;
 
 #endif /* !COMPAT */
xen/common/wait.c +5 -5
index 4f830a14e8..9f9ad033b3 100644
@@ -182,30 +178,24 @@ static void __prepare_to_wait(struct waitqueue_vcpu *wqv)
 static void __finish_wait(struct waitqueue_vcpu *wqv)
 {
     wqv->esp = NULL;
-    (void)vcpu_set_hard_affinity(current, &wqv->saved_affinity);
+    vcpu_set_tmp_affinity(current, -1, VCPU_AFFINITY_WAIT);
 }
 
 void check_wakeup_from_wait(void)
 {
-    struct waitqueue_vcpu *wqv = current->waitqueue_vcpu;
+    struct vcpu *curr = current;
+    struct waitqueue_vcpu *wqv = curr->waitqueue_vcpu;
 
     ASSERT(list_empty(&wqv->list));
 
     if ( likely(wqv->esp == NULL) )
         return;
 
-    /* Check if we woke up on the wrong CPU. */
-    if ( unlikely(smp_processor_id() != wqv->wakeup_cpu) )
+    /* Check if we are still pinned. */
+    if ( unlikely(!(curr->affinity_broken & VCPU_AFFINITY_WAIT)) )
     {
-        /* Re-set VCPU affinity and re-enter the scheduler. */

xen/common/wait.c static void __prepare_to_wait(struct waitqueue_vcpu *wqv) · line 198

Quoted 10 lines
xen/common/wait.c +2 -8
-        struct vcpu *curr = current;
-        cpumask_copy(&wqv->saved_affinity, curr->cpu_hard_affinity);
-        if ( vcpu_set_hard_affinity(curr, cpumask_of(wqv->wakeup_cpu)) )
-        {
-            gdprintk(XENLOG_ERR, "Unable to set vcpu affinity\n");
-            domain_crash(current->domain);
-        }
-        wait(); /* takes us back into the scheduler */
+        gdprintk(XENLOG_ERR, "vcpu affinity lost\n");
+        domain_crash(current->domain);

curr

Quoted 8 lines
xen/common/wait.c +0 -0
     }
 
     /*
xen/include/xen/sched.h +0 -0
index b40c8fd138..721c429454 100644

xen/include/xen/sched.h struct vcpu · line 205

Quoted 7 lines
xen/include/xen/sched.h +3 -1
@@ -200,7 +200,10 @@ struct vcpu
     /* VCPU is paused following shutdown request (d->is_shutting_down)? */
     bool             paused_for_shutdown;
     /* VCPU need affinity restored */
-    bool             affinity_broken;
+    uint8_t          affinity_broken;
+#define VCPU_AFFINITY_OVERRIDE    0x01
+#define VCPU_AFFINITY_NMI         0x02

VCPU_AFFINITY_NMI_MCE ?  It is used for more than just NMIs.

~Andrew

xen/include/xen/sched.h struct vcpu · line 209

Quoted 5 lines
xen/include/xen/sched.h +1 -0
+#define VCPU_AFFINITY_WAIT        0x04
 
     /* A hypercall has been preempted. */
     bool             hcall_preempted;

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#8 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

Quoted 55 lines · Andrew Cooper
Quoted 54 lines · Juergen Gross

Today there are three scenarios which are pinning vcpus temporarily to a single physical cpu:

- NMI/MCE injection into PV domains
- wait_event() handling
- vcpu_pin_override() handling

Each of those cases are handled independently today using their own temporary cpumask to save the old affinity settings.

The three cases can be combined as the two latter cases will only pin a vcpu to the physical cpu it is already running on, while vcpu_pin_override() is allowed to fail.

So merge the three temporary pinning scenarios by only using one cpumask and a per-vcpu bitmask for specifying which of the three scenarios is currently active (they are allowed to nest).

Note that we don't need to call domain_update_node_affinity() as we are only pinning for a brief period of time.

---
  xen/arch/x86/pv/traps.c | 20 +-------------------
  xen/arch/x86/traps.c    |  8 ++------
  xen/common/domain.c     |  4 +---
  xen/common/schedule.c   | 35 +++++++++++++++++++++++------------
  xen/common/wait.c       | 26 ++++++++------------------
  xen/include/xen/sched.h |  8 +++++---
  6 files changed, 40 insertions(+), 61 deletions(-)
xen/arch/x86/pv/traps.c +0 -15
index 1740784ff2..37dac300ba 100644
@@ -151,25 +151,7 @@ static void nmi_mce_softirq(void)
 
      BUG_ON(st->vcpu == NULL);
 
-    /*
-     * Set the tmp value unconditionally, so that the check in the iret
-     * hypercall works.
-     */
-    cpumask_copy(st->vcpu->cpu_hard_affinity_tmp,
-                 st->vcpu->cpu_hard_affinity);
-
-    if ( (cpu != st->processor) ||
-         (st->processor != st->vcpu->processor) )
-    {
-
-        /*
-         * We are on a different physical cpu.  Make sure to wakeup the vcpu on
-         * the specified processor.
-         */
Quoted 8 lines

xen/arch/x86/pv/traps.c static void nmi_mce_softirq(void) · line 154

Quoted 5 lines
xen/arch/x86/pv/traps.c +1 -4
-        vcpu_set_hard_affinity(st->vcpu, cpumask_of(st->processor));
-
-        /* Affinity is restored in the iret hypercall. */
-    }
+    vcpu_set_tmp_affinity(st->vcpu, st->processor, VCPU_AFFINITY_NMI);

Please can we keep the comment explaining where the affinity is restored, which is a disguised explanation of why it is PV-only.

Okay.

Quoted 65 lines
Quoted 65 lines
xen/common/schedule.c +18 -12
index 89bc259ae4..d4de74f9c8 100644
@@ -1106,47 +1106,58 @@ void watchdog_domain_destroy(struct domain *d)
          kill_timer(&d->watchdog_timer[i]);
  }
 
-int vcpu_pin_override(struct vcpu *v, int cpu)
+int vcpu_set_tmp_affinity(struct vcpu *v, int cpu, uint8_t reason)
  {
      spinlock_t *lock;
      int ret = -EINVAL;
+    bool migrate;
 
      lock = vcpu_schedule_lock_irq(v);
 
      if ( cpu < 0 )
      {
-        if ( v->affinity_broken )
+        if ( v->affinity_broken & reason )
          {
-            sched_set_affinity(v, v->cpu_hard_affinity_saved, NULL);
-            v->affinity_broken = 0;
              ret = 0;
+            v->affinity_broken &= ~reason;
          }
+        if ( !ret && !v->affinity_broken )
+            sched_set_affinity(v, v->cpu_hard_affinity_saved, NULL);
      }
      else if ( cpu < nr_cpu_ids )
      {
-        if ( v->affinity_broken )
+        if ( (v->affinity_broken & reason) ||
+             (v->affinity_broken && v->processor != cpu) )
              ret = -EBUSY;
          else if ( cpumask_test_cpu(cpu, VCPU2ONLINE(v)) )
          {
-            cpumask_copy(v->cpu_hard_affinity_saved, v->cpu_hard_affinity);
-            v->affinity_broken = 1;
-            sched_set_affinity(v, cpumask_of(cpu), NULL);
+            if ( !v->affinity_broken )
+            {
+                cpumask_copy(v->cpu_hard_affinity_saved, v->cpu_hard_affinity);
+                sched_set_affinity(v, cpumask_of(cpu), NULL);
+            }
+            v->affinity_broken |= reason;
              ret = 0;
          }
      }
 
-    if ( ret == 0 )
+    migrate = !ret && !cpumask_test_cpu(v->processor, v->cpu_hard_affinity);
+    if ( migrate )
          vcpu_migrate_start(v);
 
      vcpu_schedule_unlock_irq(lock, v);
 
-    domain_update_node_affinity(v->domain);
-
-    vcpu_migrate_finish(v);
+    if ( migrate )
+        vcpu_migrate_finish(v);
 
Quoted 8 lines

xen/common/schedule.c void watchdog_domain_destroy(struct domain *d) · line 1156

Quoted 4 lines
xen/common/schedule.c +1 -0
      return ret;
  }
 
+int vcpu_pin_override(struct vcpu *v, int cpu)

There are exactly two callers of vcpu_pin_override().  I'd take the opportunity to make vcpu_set_tmp_affinity() the single API call for adjusting affinity.

Fine with me.

Quoted 41 lines
Quoted 41 lines
xen/common/schedule.c +4 -0
+{
+    return vcpu_set_tmp_affinity(v, cpu, VCPU_AFFINITY_OVERRIDE);
+}
+
  typedef long ret_t;
 
  #endif /* !COMPAT */
xen/common/wait.c +5 -9
index 4f830a14e8..9f9ad033b3 100644
@@ -182,30 +178,24 @@ static void __prepare_to_wait(struct waitqueue_vcpu *wqv)
  static void __finish_wait(struct waitqueue_vcpu *wqv)
  {
      wqv->esp = NULL;
-    (void)vcpu_set_hard_affinity(current, &wqv->saved_affinity);
+    vcpu_set_tmp_affinity(current, -1, VCPU_AFFINITY_WAIT);
  }
 
  void check_wakeup_from_wait(void)
  {
-    struct waitqueue_vcpu *wqv = current->waitqueue_vcpu;
+    struct vcpu *curr = current;
+    struct waitqueue_vcpu *wqv = curr->waitqueue_vcpu;
 
      ASSERT(list_empty(&wqv->list));
 
      if ( likely(wqv->esp == NULL) )
          return;
 
-    /* Check if we woke up on the wrong CPU. */
-    if ( unlikely(smp_processor_id() != wqv->wakeup_cpu) )
+    /* Check if we are still pinned. */
+    if ( unlikely(!(curr->affinity_broken & VCPU_AFFINITY_WAIT)) )
      {
-        /* Re-set VCPU affinity and re-enter the scheduler. */
-        struct vcpu *curr = current;
-        cpumask_copy(&wqv->saved_affinity, curr->cpu_hard_affinity);
-        if ( vcpu_set_hard_affinity(curr, cpumask_of(wqv->wakeup_cpu)) )
-        {
Quoted 8 lines

xen/common/wait.c static void __prepare_to_wait(struct waitqueue_vcpu *wqv) · line 198

Quoted 6 lines
xen/common/wait.c +2 -4
-            gdprintk(XENLOG_ERR, "Unable to set vcpu affinity\n");
-            domain_crash(current->domain);
-        }
-        wait(); /* takes us back into the scheduler */
+        gdprintk(XENLOG_ERR, "vcpu affinity lost\n");
+        domain_crash(current->domain);

curr

Okay.

Quoted 18 lines
Quoted 9 lines
xen/common/wait.c +0 -0
      }
 
      /*
xen/include/xen/sched.h +0 -0
index b40c8fd138..721c429454 100644

xen/include/xen/sched.h struct vcpu · line 205

Quoted 7 lines
xen/include/xen/sched.h +3 -1
@@ -200,7 +200,10 @@ struct vcpu
      /* VCPU is paused following shutdown request (d->is_shutting_down)? */
      bool             paused_for_shutdown;
      /* VCPU need affinity restored */
-    bool             affinity_broken;
+    uint8_t          affinity_broken;
+#define VCPU_AFFINITY_OVERRIDE    0x01
+#define VCPU_AFFINITY_NMI         0x02

VCPU_AFFINITY_NMI_MCE ?  It is used for more than just NMIs.

Okay.

BTW: The MCE case is never triggered (there is no caller of pv_raise_interrupt() with TRAP_machine_check). Is this code left in place on purpose or could it be removed?

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#9 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

Quoted 21 lines · Juergen Gross
Quoted 11 lines · Andrew Cooper
Quoted 9 lines · Juergen Gross

      }
        /*

xen/include/xen/sched.h +0 -0
index b40c8fd138..721c429454 100644
@@ -200,7 +200,10 @@ struct vcpu
       /* VCPU is paused following shutdown request

(d->is_shutting_down)? */

Quoted 8 lines

xen/include/xen/sched.h struct vcpu · line 205

Quoted 6 lines

      bool             paused_for_shutdown;
      /* VCPU need affinity restored */
-    bool             affinity_broken;
+    uint8_t          affinity_broken;

xen/include/xen/sched.h +2 -0
+#define VCPU_AFFINITY_OVERRIDE    0x01
+#define VCPU_AFFINITY_NMI         0x02

VCPU_AFFINITY_NMI_MCE ?  It is used for more than just NMIs.

Quoted 5 lines

Okay.

BTW: The MCE case is never triggered (there is no caller of pv_raise_interrupt() with TRAP_machine_check). Is this code left in place on purpose or could it be removed?

It come from the restore_all_guest path in assembly, via process_mce.

This code is horribly tangled and in severe need of rewriting in C.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#10 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

Quoted 22 lines · Andrew Cooper
Quoted 21 lines · Juergen Gross
Quoted 19 lines · Andrew Cooper
Quoted 13 lines · Juergen Gross

      }
        /*

xen/include/xen/sched.h +0 -0
index b40c8fd138..721c429454 100644
@@ -200,7 +200,10 @@ struct vcpu
       /* VCPU is paused following shutdown request

(d->is_shutting_down)? */
      bool             paused_for_shutdown;
      /* VCPU need affinity restored */
-    bool             affinity_broken;
+    uint8_t          affinity_broken;

xen/include/xen/sched.h struct vcpu · line 205

Quoted 2 lines
xen/include/xen/sched.h +2 -0
+#define VCPU_AFFINITY_OVERRIDE    0x01
+#define VCPU_AFFINITY_NMI         0x02

VCPU_AFFINITY_NMI_MCE ?  It is used for more than just NMIs.

Quoted 7 lines
Quoted 5 lines

Okay.

BTW: The MCE case is never triggered (there is no caller of pv_raise_interrupt() with TRAP_machine_check). Is this code left in place on purpose or could it be removed?

It come from the restore_all_guest path in assembly, via process_mce.

Are you sure? I can see that it would call set_guest_machinecheck_trapbounce(), but I don't see how NMI_MCE_SOFTIRQ would be triggered on this path, which would be required for hitting nmi_mce_softirq().

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#11 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

Quoted 29 lines · Juergen Gross
Quoted 28 lines · Andrew Cooper
Quoted 26 lines · Juergen Gross
Quoted 11 lines · Andrew Cooper
Quoted 9 lines · Juergen Gross

       }
         /*

xen/include/xen/sched.h +0 -0
index b40c8fd138..721c429454 100644
@@ -200,7 +200,10 @@ struct vcpu
        /* VCPU is paused following shutdown request

(d->is_shutting_down)? */

Quoted 8 lines

xen/include/xen/sched.h struct vcpu · line 205

Quoted 6 lines

       bool             paused_for_shutdown;
       /* VCPU need affinity restored */
-    bool             affinity_broken;
+    uint8_t          affinity_broken;

xen/include/xen/sched.h +2 -0
+#define VCPU_AFFINITY_OVERRIDE    0x01
+#define VCPU_AFFINITY_NMI         0x02

VCPU_AFFINITY_NMI_MCE ?  It is used for more than just NMIs.

Okay.

BTW: The MCE case is never triggered (there is no caller of pv_raise_interrupt() with TRAP_machine_check). Is this code left in place on purpose or could it be removed?

Quoted 6 lines
Quoted 1 line

It come from the restore_all_guest path in assembly, via process_mce.

Are you sure? I can see that it would call set_guest_machinecheck_trapbounce(), but I don't see how NMI_MCE_SOFTIRQ would be triggered on this path, which would be required for hitting nmi_mce_softirq().

Looking at it, I'm not so sure...

There is certainly room for further clean-up here, for anyone with enough time to disentangle this rats nest.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#12 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

In reply to Juergen Gross
Quoted 9 lines · Juergen Gross

Today there are three scenarios which are pinning vcpus temporarily to a single physical cpu:

- NMI/MCE injection into PV domains
- wait_event() handling
- vcpu_pin_override() handling

Each of those cases are handled independently today using their own temporary cpumask to save the old affinity settings.

And what guarantees that no two of them will be in use at the same time? You don't even mention ...

Quoted 3 lines

The three cases can be combined as the two latter cases will only pin a vcpu to the physical cpu it is already running on, while vcpu_pin_override() is allowed to fail.

.. the NMI/#MC injection case here (despite the use of "the two" and "while" giving the impression). Or (looking at the actual code) did you mean "former" instead of "latter"? But if so - id that true? v->processor gets latched into st->processor before raising the softirq, but can't the vCPU be moved elsewhere by the time the softirq handler actually gains control? If that's not possible (and if it's not obvious why, and as you can see it's not obvious to me), then I think a code comment wants to be added there.

Independent of that introducing new failure cases for vcpu_pin_override() isn't really nice. Then again any two racing/conflicting pinning attempts can't result in anything good.

Nevertheless - nice idea, so a few minor comments on the code as well, in the hope that my point above can be addressed.

xen/common/schedule.c void watchdog_domain_destroy(struct domain *d) · line 1109

Quoted 8 lines
xen/common/schedule.c +1 -1
@@ -1106,47 +1106,58 @@ void watchdog_domain_destroy(struct domain *d)
          kill_timer(&d->watchdog_timer[i]);
  }
 
-int vcpu_pin_override(struct vcpu *v, int cpu)
+int vcpu_set_tmp_affinity(struct vcpu *v, int cpu, uint8_t reason)

I'd find it pretty nice if at this occasion the type of "cpu" was changed to "unsigned int", using UINT_MAX or NR_CPUS instead of -1 for the "remove override" case.

I'd also prefer if you didn't use "tmp" as an infix here. Make it "temporary", "transient", or some such. Perhaps even omit "set", the more that the function may also clear it.

Quoted 26 lines
xen/common/schedule.c +5 -6
@@ -182,30 +178,24 @@ static void __prepare_to_wait(struct waitqueue_vcpu *wqv)
  static void __finish_wait(struct waitqueue_vcpu *wqv)
  {
      wqv->esp = NULL;
-    (void)vcpu_set_hard_affinity(current, &wqv->saved_affinity);
+    vcpu_set_tmp_affinity(current, -1, VCPU_AFFINITY_WAIT);
  }
 
  void check_wakeup_from_wait(void)
  {
-    struct waitqueue_vcpu *wqv = current->waitqueue_vcpu;
+    struct vcpu *curr = current;
+    struct waitqueue_vcpu *wqv = curr->waitqueue_vcpu;
 
      ASSERT(list_empty(&wqv->list));
 
      if ( likely(wqv->esp == NULL) )
          return;
 
-    /* Check if we woke up on the wrong CPU. */
-    if ( unlikely(smp_processor_id() != wqv->wakeup_cpu) )
+    /* Check if we are still pinned. */
+    if ( unlikely(!(curr->affinity_broken & VCPU_AFFINITY_WAIT)) )
      {
-        /* Re-set VCPU affinity and re-enter the scheduler. */
-        struct vcpu *curr = current;

xen/common/schedule.c static void __prepare_to_wait(struct waitqueue_vcpu *wqv) · line 199

Quoted 10 lines
xen/common/schedule.c +2 -7
-        cpumask_copy(&wqv->saved_affinity, curr->cpu_hard_affinity);
-        if ( vcpu_set_hard_affinity(curr, cpumask_of(wqv->wakeup_cpu)) )
-        {
-            gdprintk(XENLOG_ERR, "Unable to set vcpu affinity\n");
-            domain_crash(current->domain);
-        }
-        wait(); /* takes us back into the scheduler */
+        gdprintk(XENLOG_ERR, "vcpu affinity lost\n");
+        domain_crash(current->domain);
      }

Please use curr in favor of current.

Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#13 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

Quoted 9 lines · Jan Beulich
Quoted 8 lines · Juergen Gross

Today there are three scenarios which are pinning vcpus temporarily to
a single physical cpu:

- NMI/MCE injection into PV domains
- wait_event() handling
- vcpu_pin_override() handling

Quoted 7 lines
Quoted 2 lines

Each of those cases are handled independently today using their own temporary cpumask to save the old affinity settings.

And what guarantees that no two of them will be in use at the same time?
You don't even mention ...

AFAIUI, the case of any two of the above being in use at the same time is handled. Basically, it is fine, as far as they (temporarily) set the affinity to the same CPU, which is said to always be the case, here...

Quoted 5 lines
Quoted 4 lines

The three cases can be combined as the two latter cases will only pin
a vcpu to the physical cpu it is already running on, while vcpu_pin_override() is allowed to fail.

...for the first two. For vcpu_pin_override(), if it is called for pinning the vcpu to CPU X, with any of the other twos in effect, and having temporarily pinned the vcpu to CPU X already, then things are fine again. If vcpu_pin_override() wants to pin to CPU Y, it fails.

All that being said, I agree it'd be good to have a bit more comments (and I asked the same, although about a different path, in my own reply).

Regards

Signature · 6 lines
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#14 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

In reply to Jan Beulich
Quoted 14 lines · Jan Beulich
Quoted 9 lines · Juergen Gross

Today there are three scenarios which are pinning vcpus temporarily to a single physical cpu:

- NMI/MCE injection into PV domains
- wait_event() handling
- vcpu_pin_override() handling

Each of those cases are handled independently today using their own temporary cpumask to save the old affinity settings.

And what guarantees that no two of them will be in use at the same time? You don't even mention ...

Quoted 7 lines
Quoted 3 lines

The three cases can be combined as the two latter cases will only pin a vcpu to the physical cpu it is already running on, while vcpu_pin_override() is allowed to fail.

.. the NMI/#MC injection case here (despite the use of "the two" and "while" giving the impression). Or (looking at the actual code) did you mean "former" instead of "latter"? But if so - id that true?

Yes, I meant former.

Quoted 5 lines

v->processor gets latched into st->processor before raising the softirq, but can't the vCPU be moved elsewhere by the time the softirq handler actually gains control? If that's not possible (and if it's not obvious why, and as you can see it's not obvious to me), then I think a code comment wants to be added there.

You are right, it might be possible for the vcpu to move around.

OTOH is it really important to run the target vcpu exactly on the cpu it is executing (or has last executed) at the time the NMI/MCE is being queued? This is in no way related to the cpu the MCE or NMI has been happening on. It is just a random cpu, and so it would be if we'd do the cpu selection when the softirq handler is running.

One question to understand the idea nehind all that: _why_ is the vcpu pinned until it does an iret? I could understand if it would be pinned to the cpu where the NMI/MCE was happening, but this is not the case.

Quoted 3 lines

Independent of that introducing new failure cases for vcpu_pin_override() isn't really nice. Then again any two racing/conflicting pinning attempts can't result in anything good.

Right.

Quoted 9 lines

Nevertheless - nice idea, so a few minor comments on the code as well, in the hope that my point above can be addressed.

Quoted 6 lines
xen/common/schedule.c +0 -0
@@ -1106,47 +1106,58 @@ void watchdog_domain_destroy(struct domain *d)
           kill_timer(&d->watchdog_timer[i]);
   }
 
Quoted 6 lines

xen/common/schedule.c void watchdog_domain_destroy(struct domain *d) · line 1109

Quoted 2 lines
xen/common/schedule.c +1 -1
-int vcpu_pin_override(struct vcpu *v, int cpu)
+int vcpu_set_tmp_affinity(struct vcpu *v, int cpu, uint8_t reason)

I'd find it pretty nice if at this occasion the type of "cpu" was changed to "unsigned int", using UINT_MAX or NR_CPUS instead of -1 for the "remove override" case.

Fine with me.

Quoted 3 lines

I'd also prefer if you didn't use "tmp" as an infix here. Make it "temporary", "transient", or some such. Perhaps even omit "set", the more that the function may also clear it.

Okay.

Quoted 31 lines
Quoted 31 lines
xen/common/schedule.c +5 -10
@@ -182,30 +178,24 @@ static void __prepare_to_wait(struct waitqueue_vcpu *wqv)
   static void __finish_wait(struct waitqueue_vcpu *wqv)
   {
       wqv->esp = NULL;
-    (void)vcpu_set_hard_affinity(current, &wqv->saved_affinity);
+    vcpu_set_tmp_affinity(current, -1, VCPU_AFFINITY_WAIT);
   }
 
   void check_wakeup_from_wait(void)
   {
-    struct waitqueue_vcpu *wqv = current->waitqueue_vcpu;
+    struct vcpu *curr = current;
+    struct waitqueue_vcpu *wqv = curr->waitqueue_vcpu;
 
       ASSERT(list_empty(&wqv->list));
 
       if ( likely(wqv->esp == NULL) )
           return;
 
-    /* Check if we woke up on the wrong CPU. */
-    if ( unlikely(smp_processor_id() != wqv->wakeup_cpu) )
+    /* Check if we are still pinned. */
+    if ( unlikely(!(curr->affinity_broken & VCPU_AFFINITY_WAIT)) )
       {
-        /* Re-set VCPU affinity and re-enter the scheduler. */
-        struct vcpu *curr = current;
-        cpumask_copy(&wqv->saved_affinity, curr->cpu_hard_affinity);
-        if ( vcpu_set_hard_affinity(curr, cpumask_of(wqv->wakeup_cpu)) )
-        {
-            gdprintk(XENLOG_ERR, "Unable to set vcpu affinity\n");
Quoted 8 lines

xen/common/schedule.c static void __prepare_to_wait(struct waitqueue_vcpu *wqv) · line 199

Quoted 6 lines
xen/common/schedule.c +2 -3
-            domain_crash(current->domain);
-        }
-        wait(); /* takes us back into the scheduler */
+        gdprintk(XENLOG_ERR, "vcpu affinity lost\n");
+        domain_crash(current->domain);
       }

Please use curr in favor of current.

Yes.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#15 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

Quoted 15 lines · Juergen Gross
Quoted 5 lines · Jan Beulich

v->processor gets latched into st->processor before raising the softirq, but can't the vCPU be moved elsewhere by the time the softirq handler actually gains control? If that's not possible (and if it's not obvious why, and as you can see it's not obvious to me), then I think a code comment wants to be added there.

You are right, it might be possible for the vcpu to move around.

OTOH is it really important to run the target vcpu exactly on the cpu it is executing (or has last executed) at the time the NMI/MCE is being queued? This is in no way related to the cpu the MCE or NMI has been happening on. It is just a random cpu, and so it would be if we'd do the cpu selection when the softirq handler is running.

Quoted 3 lines

One question to understand the idea nehind all that: _why_ is the vcpu pinned until it does an iret? I could understand if it would be pinned to the cpu where the NMI/MCE was happening, but this is not the case.

Then it was never finished or got broken, I would guess. Especially for #MC the idea is/was for the domain to be able to access the MSRs in question (). Iirc, that is, and prior to vMCE appearing.

Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#16 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

Quoted 16 lines · Jan Beulich
Quoted 15 lines · Juergen Gross
Quoted 5 lines · Jan Beulich

v->processor gets latched into st->processor before raising the softirq, but can't the vCPU be moved elsewhere by the time the softirq handler actually gains control? If that's not possible (and if it's not obvious why, and as you can see it's not obvious to me), then I think a code comment wants to be added there.

You are right, it might be possible for the vcpu to move around.

OTOH is it really important to run the target vcpu exactly on the cpu it is executing (or has last executed) at the time the NMI/MCE is being queued? This is in no way related to the cpu the MCE or NMI has been happening on. It is just a random cpu, and so it would be if we'd do the cpu selection when the softirq handler is running.

Quoted 5 lines
Quoted 3 lines

One question to understand the idea nehind all that: _why_ is the vcpu pinned until it does an iret? I could understand if it would be pinned to the cpu where the NMI/MCE was happening, but this is not the case.

Then it was never finished or got broken, I would guess.

Oh, no. The #MC side use has gone away in 3a91769d6e, without cleaning up other code. So there doesn't seem to be any such requirement anymore.

Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#17 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

Quoted 17 lines · Jan Beulich
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Juergen Gross
Quoted 5 lines · Jan Beulich

v->processor gets latched into st->processor before raising the softirq, but can't the vCPU be moved elsewhere by the time the softirq handler actually gains control? If that's not possible (and if it's not obvious why, and as you can see it's not obvious to me), then I think a code comment wants to be added there.

You are right, it might be possible for the vcpu to move around.

OTOH is it really important to run the target vcpu exactly on the cpu it is executing (or has last executed) at the time the NMI/MCE is being queued? This is in no way related to the cpu the MCE or NMI has been happening on. It is just a random cpu, and so it would be if we'd do the cpu selection when the softirq handler is running.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

One question to understand the idea nehind all that: _why_ is the vcpu pinned until it does an iret? I could understand if it would be pinned to the cpu where the NMI/MCE was happening, but this is not the case.

Then it was never finished or got broken, I would guess.

Oh, no. The #MC side use has gone away in 3a91769d6e, without cleaning up other code. So there doesn't seem to be any such requirement anymore.

Ah, okay, so no need any longer to rename VCPU_AFFINITY_NMI. :-)

I'll add a patch removing the MCE cruft.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#18 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

In reply to Jan Beulich
Quoted 17 lines · Jan Beulich
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Juergen Gross
Quoted 5 lines · Jan Beulich

v->processor gets latched into st->processor before raising the softirq, but can't the vCPU be moved elsewhere by the time the softirq handler actually gains control? If that's not possible (and if it's not obvious why, and as you can see it's not obvious to me), then I think a code comment wants to be added there.

You are right, it might be possible for the vcpu to move around.

OTOH is it really important to run the target vcpu exactly on the cpu it is executing (or has last executed) at the time the NMI/MCE is being queued? This is in no way related to the cpu the MCE or NMI has been happening on. It is just a random cpu, and so it would be if we'd do the cpu selection when the softirq handler is running.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

One question to understand the idea nehind all that: _why_ is the vcpu pinned until it does an iret? I could understand if it would be pinned to the cpu where the NMI/MCE was happening, but this is not the case.

Then it was never finished or got broken, I would guess.

Oh, no. The #MC side use has gone away in 3a91769d6e, without cleaning up other code. So there doesn't seem to be any such requirement anymore.

So just to be sure: you are fine for me removing the pinning for NMIs?

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#19 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

Quoted 22 lines · Juergen Gross
Quoted 21 lines · Jan Beulich
Quoted 20 lines · Jan Beulich
Quoted 18 lines · Juergen Gross
Quoted 5 lines · Jan Beulich

v->processor gets latched into st->processor before raising the softirq, but can't the vCPU be moved elsewhere by the time the softirq handler actually gains control? If that's not possible (and if it's not obvious why, and as you can see it's not obvious to me), then I think a code comment wants to be added there.

You are right, it might be possible for the vcpu to move around.

OTOH is it really important to run the target vcpu exactly on the cpu it is executing (or has last executed) at the time the NMI/MCE is being queued? This is in no way related to the cpu the MCE or NMI has been happening on. It is just a random cpu, and so it would be if we'd do the cpu selection when the softirq handler is running.

One question to understand the idea nehind all that: _why_ is the vcpu pinned until it does an iret? I could understand if it would be pinned to the cpu where the NMI/MCE was happening, but this is not the case.

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Then it was never finished or got broken, I would guess.

Oh, no. The #MC side use has gone away in 3a91769d6e, without cleaning up other code. So there doesn't seem to be any such requirement anymore.

So just to be sure: you are fine for me removing the pinning for NMIs?

No, not the pinning as a whole. The forced CPU0 affinity should still remain. It's just that there's no correlation anymore between the CPU a vCPU was running on and the CPU it is to be pinned to (temporarily).

What can go away if the #MC part of the logic.

Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#20 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

Quoted 25 lines · Jan Beulich
Quoted 24 lines · Juergen Gross
Quoted 23 lines · Jan Beulich
Quoted 21 lines · Jan Beulich
Quoted 15 lines · Juergen Gross
Quoted 5 lines · Jan Beulich

v->processor gets latched into st->processor before raising the softirq, but can't the vCPU be moved elsewhere by the time the softirq handler actually gains control? If that's not possible (and if it's not obvious why, and as you can see it's not obvious to me), then I think a code comment wants to be added there.

You are right, it might be possible for the vcpu to move around.

OTOH is it really important to run the target vcpu exactly on the cpu it is executing (or has last executed) at the time the NMI/MCE is being queued? This is in no way related to the cpu the MCE or NMI has been happening on. It is just a random cpu, and so it would be if we'd do the cpu selection when the softirq handler is running.

Quoted 3 lines

One question to understand the idea nehind all that: _why_ is the vcpu pinned until it does an iret? I could understand if it would be pinned to the cpu where the NMI/MCE was happening, but this is not the case.

Then it was never finished or got broken, I would guess.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Oh, no. The #MC side use has gone away in 3a91769d6e, without cleaning up other code. So there doesn't seem to be any such requirement anymore.

So just to be sure: you are fine for me removing the pinning for NMIs?

No, not the pinning as a whole. The forced CPU0 affinity should still remain. It's just that there's no correlation anymore between the CPU a vCPU was running on and the CPU it is to be pinned to (temporarily).

I don't get it. Today vcpu0 of the hardware domain is pinned to the cpu it was last running on when the NMI happened. Why is that important? Or do you want to change the logic and pin vcpu0 for NMI handling always to CPU0?

Quoted 1 line

What can go away if the #MC part of the logic.

Okay.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#21 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

Quoted 42 lines · Juergen Gross
Quoted 41 lines · Jan Beulich
Quoted 39 lines · Juergen Gross
Quoted 31 lines · Jan Beulich
Quoted 30 lines · Jan Beulich
Quoted 28 lines · Juergen Gross
Quoted 9 lines · Jan Beulich

v->processor gets latched into st->processor before raising the softirq,
but can't the vCPU be moved elsewhere by the time the softirq handler
actually gains control? If that's not possible (and if it's not obvious
why, and as you can see it's not obvious to me), then I think a code
comment wants to be added there.

You are right, it might be possible for the vcpu to move around.

OTOH is it really important to run the target vcpu exactly on the cpu
it is executing (or has last executed) at the time the NMI/MCE is being
queued? This is in no way related to the cpu the MCE or NMI has been happening on. It is just a random cpu, and so it would be if we'd do the
cpu selection when the softirq handler is running.

One question to understand the idea nehind all that: _why_ is the vcpu
pinned until it does an iret? I could understand if it would be pinned
to the cpu where the NMI/MCE was happening, but this is not the case.

Quoted 5 lines
Quoted 1 line

Then it was never finished or got broken, I would guess.

Oh, no. The #MC side use has gone away in 3a91769d6e, without cleaning up other code. So there doesn't seem to be any such requirement anymore.

So just to be sure: you are fine for me removing the pinning for NMIs?

Quoted 8 lines
Quoted 3 lines

No, not the pinning as a whole. The forced CPU0 affinity should still remain. It's just that there's no correlation anymore between the CPU a vCPU was running on and the CPU it is to be pinned to (temporarily).

I don't get it. Today vcpu0 of the hardware domain is pinned to the cpu it was last running on when the NMI happened. Why is that important? Or do you want to change the logic and pin vcpu0 for NMI handling always to CPU0?

Its (allegedly) for when dom0 knows some system-specific way of getting extra information out of the platform, that happens to be core-specific.

There are rare cases where SMI's need to be executed on CPU0, and I wouldn't put it past hardware designers to have similar aspects for NMIs.

That said, as soon as the gaping security hole which is the default-readibility of all MSRs, I bet the utility of this pinning mechanism will be 0.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#22 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

Quoted 52 lines · Andrew Cooper
Quoted 42 lines · Juergen Gross
Quoted 41 lines · Jan Beulich
Quoted 39 lines · Juergen Gross
Quoted 31 lines · Jan Beulich
Quoted 30 lines · Jan Beulich
Quoted 28 lines · Juergen Gross
Quoted 9 lines · Jan Beulich

v->processor gets latched into st->processor before raising the softirq,
but can't the vCPU be moved elsewhere by the time the softirq handler
actually gains control? If that's not possible (and if it's not obvious
why, and as you can see it's not obvious to me), then I think a code
comment wants to be added there.

You are right, it might be possible for the vcpu to move around.

OTOH is it really important to run the target vcpu exactly on the cpu
it is executing (or has last executed) at the time the NMI/MCE is being
queued? This is in no way related to the cpu the MCE or NMI has been happening on. It is just a random cpu, and so it would be if we'd do the
cpu selection when the softirq handler is running.

One question to understand the idea nehind all that: _why_ is the vcpu
pinned until it does an iret? I could understand if it would be pinned
to the cpu where the NMI/MCE was happening, but this is not the case.

Quoted 5 lines
Quoted 1 line

Then it was never finished or got broken, I would guess.

Oh, no. The #MC side use has gone away in 3a91769d6e, without cleaning up other code. So there doesn't seem to be any such requirement anymore.

So just to be sure: you are fine for me removing the pinning for NMIs?

Quoted 8 lines
Quoted 3 lines

No, not the pinning as a whole. The forced CPU0 affinity should still remain. It's just that there's no correlation anymore between the CPU a vCPU was running on and the CPU it is to be pinned to (temporarily).

I don't get it. Today vcpu0 of the hardware domain is pinned to the cpu it was last running on when the NMI happened. Why is that important? Or do you want to change the logic and pin vcpu0 for NMI handling always to CPU0?

Quoted 5 lines

Its (allegedly) for when dom0 knows some system-specific way of getting extra information out of the platform, that happens to be core-specific.

There are rare cases where SMI's need to be executed on CPU0, and I wouldn't put it past hardware designers to have similar aspects for NMIs.

Understood. But today vcpu0 is _not_ bound to CPU0, but to any cpu it happened to run on.

Quoted 4 lines

That said, as soon as the gaping security hole which is the default-readibility of all MSRs, I bet the utility of this pinning mechanism will be 0.

And my reasoning is that this is the case today already, as there is no pinning to CPU0 done, at least not on purpose.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#23 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

Quoted 68 lines · Juergen Gross
Quoted 56 lines · Andrew Cooper
Quoted 49 lines · Juergen Gross
Quoted 41 lines · Jan Beulich
Quoted 40 lines · Juergen Gross
Quoted 38 lines · Jan Beulich
Quoted 24 lines · Jan Beulich
Quoted 23 lines · Juergen Gross
Quoted 9 lines · Jan Beulich

v->processor gets latched into st->processor before raising the softirq,
but can't the vCPU be moved elsewhere by the time the softirq handler
actually gains control? If that's not possible (and if it's not obvious
why, and as you can see it's not obvious to me), then I think a code
comment wants to be added there.

You are right, it might be possible for the vcpu to move around.

OTOH is it really important to run the target vcpu exactly on the cpu
it is executing (or has last executed) at the time the NMI/MCE is being
queued? This is in no way related to the cpu the MCE or NMI has been
happening on. It is just a random cpu, and so it would be if we'd do the
cpu selection when the softirq handler is running.

Quoted 8 lines
Quoted 6 lines

One question to understand the idea nehind all that: _why_ is the vcpu
pinned until it does an iret? I could understand if it would be pinned
to the cpu where the NMI/MCE was happening, but this is not the case.

Then it was never finished or got broken, I would guess.

Oh, no. The #MC side use has gone away in 3a91769d6e, without cleaning
up other code. So there doesn't seem to be any such requirement anymore.

Quoted 6 lines
Quoted 2 lines

So just to be sure: you are fine for me removing the pinning for NMIs?

No, not the pinning as a whole. The forced CPU0 affinity should still remain. It's just that there's no correlation anymore between the CPU a vCPU was running on and the CPU it is to be pinned to (temporarily).

Quoted 5 lines

I don't get it. Today vcpu0 of the hardware domain is pinned to the cpu it was last running on when the NMI happened. Why is that important? Or do you want to change the logic and pin vcpu0 for NMI handling always
to CPU0?

Quoted 6 lines

Its (allegedly) for when dom0 knows some system-specific way of getting extra information out of the platform, that happens to be core-specific.

There are rare cases where SMI's need to be executed on CPU0, and I wouldn't put it past hardware designers to have similar aspects for NMIs.

Understood. But today vcpu0 is _not_ bound to CPU0, but to any cpu it happened to run on.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

That said, as soon as the gaping security hole which is the default-readibility of all MSRs, I bet the utility of this pinning mechanism will be 0.

And my reasoning is that this is the case today already, as there is no pinning to CPU0 done, at least not on purpose.

Based on this analysis, I'd be tempted to drop the pinning completely. It clearly isn't working in a rational way.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#24 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

In reply to Juergen Gross
Quoted 8 lines · Juergen Gross
Quoted 7 lines · Jan Beulich
Quoted 5 lines · Juergen Gross
Quoted 2 lines · Jan Beulich

Oh, no. The #MC side use has gone away in 3a91769d6e, without cleaning up other code. So there doesn't seem to be any such requirement anymore.

So just to be sure: you are fine for me removing the pinning for NMIs?

Quoted 8 lines
Quoted 3 lines

No, not the pinning as a whole. The forced CPU0 affinity should still remain. It's just that there's no correlation anymore between the CPU a vCPU was running on and the CPU it is to be pinned to (temporarily).

I don't get it. Today vcpu0 of the hardware domain is pinned to the cpu it was last running on when the NMI happened. Why is that important? Or do you want to change the logic and pin vcpu0 for NMI handling always to CPU0?

Hmm, I've been confused. To me it doesn't make sense to force affinity other than to CPU0 for an NMI. Hence I was mis-reading the original send_guest_trap() invocation (in particular its middle argument). Looking at commit 355b0469a8 ("x86: MCA support") I would guess the affinity change for NMI delivery was done by mistake, or at best just to not distinguish NMI from #MC.

As a result, considering that before that there was no affinity change at all, and considering further than changing to other than CPU0 doesn't really make sense, but we got away with not doing so for so many years, I think I agree that it can be ripped out altogether. Personally I'd prefer a fix to make it use CPU0, but I'm in no way going to insist.

Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#25 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

In reply to Juergen Gross
Quoted 11 lines · Juergen Gross

Today there are three scenarios which are pinning vcpus temporarily to
a single physical cpu:

- NMI/MCE injection into PV domains
- wait_event() handling
- vcpu_pin_override() handling

Each of those cases are handled independently today using their own temporary cpumask to save the old affinity settings.

Quoted 8 lines

The three cases can be combined as the two latter cases will only pin a vcpu to the physical cpu it is already running on, while vcpu_pin_override() is allowed to fail.

So merge the three temporary pinning scenarios by only using one cpumask and a per-vcpu bitmask for specifying which of the three scenarios is currently active (they are allowed to nest).

Right. And, after this patch, all the three cases above and suspend/resume, all use cpu_hard_affinity_saved.

Can you add a paragraph, either here in the changelog, on in a comment (e.g., at the top or inside of vcpu_set_tmp_affinity()), about why this is all fine?

Regards

Signature · 6 lines
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#26 lore raw

Re: [Xen-devel] [PATCH 2/2] xen: merge temporary vcpu pinning scenarios

Quoted 20 lines · Dario Faggioli
Quoted 11 lines · Juergen Gross

Today there are three scenarios which are pinning vcpus temporarily to
a single physical cpu:

- NMI/MCE injection into PV domains
- wait_event() handling
- vcpu_pin_override() handling

Each of those cases are handled independently today using their own temporary cpumask to save the old affinity settings.

Quoted 8 lines

The three cases can be combined as the two latter cases will only pin a vcpu to the physical cpu it is already running on, while vcpu_pin_override() is allowed to fail.

So merge the three temporary pinning scenarios by only using one cpumask and a per-vcpu bitmask for specifying which of the three scenarios is currently active (they are allowed to nest).

Quoted 6 lines

Right. And, after this patch, all the three cases above and suspend/resume, all use cpu_hard_affinity_saved.

Can you add a paragraph, either here in the changelog, on in a comment (e.g., at the top or inside of vcpu_set_tmp_affinity()), about why this is all fine?

Yes.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel