Skip to contentSkip to conversation

xen: some scheduler related simplifications

PATCH In review Juergen Gross 5 participants 13 messages Mar 14, 2019 – Mar 25, 2019 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

cover xen: some scheduler related simplifications

While working on the scheduler I needed to modify some bits which are worth to be considered as a stand-alone improvement IMO.

Juergen Gross (2):
xen: introduce a cpumask with all bits set xen/sched: don't disable scheduler on cpus during suspend

 xen/arch/x86/io_apic.c    |   4 +-
 xen/common/cpu.c          |   4 ++
 xen/common/cpupool.c      |   2 +-
 xen/common/schedule.c     | 163 +++++++++++++---------------------------------
 xen/include/xen/cpumask.h |   2 +
 5 files changed, 53 insertions(+), 122 deletions(-)
Signature · 7 lines
2.16.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH 1/2 xen: introduce a cpumask with all bits set

There are several places in Xen allocating a cpumask on the stack and setting all bits in it just to use it as an initial mask for allowing all cpus.

Save the stack space and omit the need for runtime initialization by defining a globally accessible cpumask_all variable.

---
 xen/arch/x86/io_apic.c    |  4 +---
 xen/common/cpu.c          |  4 ++++
 xen/common/schedule.c     | 17 ++++-------------
 xen/include/xen/cpumask.h |  2 ++
 4 files changed, 11 insertions(+), 16 deletions(-)
xen/arch/x86/io_apic.c +1 -3
index daa5e9e5ff..a5344ed727 100644
@@ -1881,7 +1881,6 @@ static void __init check_timer(void)
     int apic1, pin1, apic2, pin2;
     int vector, ret;
     unsigned long flags;
-    cpumask_t mask_all;
 
     local_irq_save(flags);
 
@@ -1892,8 +1891,7 @@ static void __init check_timer(void)
     vector = IRQ0_VECTOR;
     clear_irq_vector(0);
 
-    cpumask_setall(&mask_all);
-    if ((ret = bind_irq_vector(0, vector, &mask_all)))
+    if ((ret = bind_irq_vector(0, vector, &cpumask_all)))
         printk(KERN_ERR"..IRQ0 is not set correctly with ioapic!!!, err:%d\n", ret);
     
     irq_desc[0].status &= ~IRQ_DISABLED;
xen/common/cpu.c +4 -0
index 653a56b840..836c62f97f 100644
@@ -11,6 +11,10 @@ unsigned int __read_mostly nr_cpumask_bits
     = BITS_TO_LONGS(NR_CPUS) * BITS_PER_LONG;
 #endif
 
+const cpumask_t cpumask_all = {
+    .bits[0 ... (BITS_TO_LONGS(NR_CPUS) - 1)] = ~0UL
+};
+
 /*
  * cpu_bit_bitmap[] is a special, "compressed" data structure that
  * represents all NR_CPUS bits binary values of 1<<nr.
xen/common/schedule.c +4 -13
index fd587622f4..60755a631e 100644
@@ -256,9 +256,6 @@ static void sched_spin_unlock_double(spinlock_t *lock1, spinlock_t *lock2,
 int sched_init_vcpu(struct vcpu *v, unsigned int processor)
 {
     struct domain *d = v->domain;
-    cpumask_t allcpus;
-
-    cpumask_setall(&allcpus);
 
     v->processor = processor;
 
@@ -280,9 +277,9 @@ int sched_init_vcpu(struct vcpu *v, unsigned int processor)
      * domain-0 VCPUs, are pinned onto their respective physical CPUs.
      */
     if ( is_idle_domain(d) || d->is_pinned )
-        sched_set_affinity(v, cpumask_of(processor), &allcpus);
+        sched_set_affinity(v, cpumask_of(processor), &cpumask_all);
     else
-        sched_set_affinity(v, &allcpus, &allcpus);
+        sched_set_affinity(v, &cpumask_all, &cpumask_all);
 
     /* Idle VCPUs are scheduled immediately, so don't put them in runqueue. */
     if ( is_idle_domain(d) )
@@ -361,7 +358,6 @@ int sched_move_domain(struct domain *d, struct cpupool *c)
     for_each_vcpu ( d, v )
     {
         spinlock_t *lock;
-        cpumask_t allcpus;
 
         vcpudata = v->sched_priv;
 
@@ -369,11 +365,9 @@ int sched_move_domain(struct domain *d, struct cpupool *c)
         migrate_timer(&v->singleshot_timer, new_p);
         migrate_timer(&v->poll_timer, new_p);
 
-        cpumask_setall(&allcpus);
-
         lock = vcpu_schedule_lock_irq(v);
 
-        sched_set_affinity(v, &allcpus, &allcpus);
+        sched_set_affinity(v, &cpumask_all, &cpumask_all);
 
         v->processor = new_p;
         /*
@@ -812,8 +806,6 @@ int cpu_disable_scheduler(unsigned int cpu)
             if ( cpumask_empty(&online_affinity) &&
                  cpumask_test_cpu(cpu, v->cpu_hard_affinity) )
             {
-                cpumask_t allcpus;
-
                 if ( v->affinity_broken )
                 {
                     /* The vcpu is temporarily pinned, can't move it. */
@@ -831,8 +823,7 @@ int cpu_disable_scheduler(unsigned int cpu)
                 else
                     printk(XENLOG_DEBUG "Breaking affinity for %pv\n", v);
 
-                cpumask_setall(&allcpus);
-                sched_set_affinity(v, &allcpus, NULL);
+                sched_set_affinity(v, &cpumask_all, NULL);
             }
 
             if ( v->processor != cpu )
xen/include/xen/cpumask.h +2 -0
index b4cc92a4f5..5a43438988 100644
@@ -75,6 +75,8 @@
 
 typedef struct cpumask{ DECLARE_BITMAP(bits, NR_CPUS); } cpumask_t;
 
+extern const cpumask_t cpumask_all;
+
 extern unsigned int nr_cpu_ids;
 
 #if NR_CPUS > 4 * BITS_PER_LONG
Signature · 7 lines
2.16.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] xen: introduce a cpumask with all bits set

Quoted 8 lines · Juergen Gross

There are several places in Xen allocating a cpumask on the stack and setting all bits in it just to use it as an initial mask for allowing all cpus.

Save the stack space and omit the need for runtime initialization by defining a globally accessible cpumask_all variable.

style suggestions which can be folded on commit.

xen/arch/x86/io_apic.c static void __init check_timer(void) · line 1894

Quoted 11 lines
xen/arch/x86/io_apic.c +1 -2
index daa5e9e5ff..a5344ed727 100644
@@ -1892,8 +1891,7 @@ static void __init check_timer(void)
     vector = IRQ0_VECTOR;
     clear_irq_vector(0);
 
-    cpumask_setall(&mask_all);
-    if ((ret = bind_irq_vector(0, vector, &mask_all)))
+    if ((ret = bind_irq_vector(0, vector, &cpumask_all)))

Spaces, seeing as we're modifying the line.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw

Re: [PATCH 1/2] xen: introduce a cpumask with all bits set

Quoted 21 lines · Andrew Cooper
Quoted 8 lines · Juergen Gross

There are several places in Xen allocating a cpumask on the stack and setting all bits in it just to use it as an initial mask for allowing all cpus.

Save the stack space and omit the need for runtime initialization by defining a globally accessible cpumask_all variable.

style suggestions which can be folded on commit.

Quoted 8 lines
xen/arch/x86/io_apic.c +0 -0
index daa5e9e5ff..a5344ed727 100644
@@ -1892,8 +1891,7 @@ static void __init check_timer(void)
     vector = IRQ0_VECTOR;
     clear_irq_vector(0);
 
Quoted 5 lines

xen/arch/x86/io_apic.c static void __init check_timer(void) · line 1894

Quoted 3 lines
xen/arch/x86/io_apic.c +1 -2
-    cpumask_setall(&mask_all);
-    if ((ret = bind_irq_vector(0, vector, &mask_all)))
+    if ((ret = bind_irq_vector(0, vector, &cpumask_all)))

Spaces, seeing as we're modifying the line.

You are aware this file originates from the Linux kernel?

I really don't mind either way, I just wanted to mention it...

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Acked-by on v1 1/2
#4 lore raw

Re: [PATCH 1/2] xen: introduce a cpumask with all bits set

In reply to Juergen Gross
Quoted 8 lines · Juergen Gross

There are several places in Xen allocating a cpumask on the stack and setting all bits in it just to use it as an initial mask for allowing all cpus.

Save the stack space and omit the need for runtime initialization by defining a globally accessible cpumask_all variable.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Reviewed-by
#5 lore raw

Re: [PATCH 1/2] xen: introduce a cpumask with all bits set

Quoted 7 lines · George Dunlap
Quoted 6 lines · Juergen Gross

There are several places in Xen allocating a cpumask on the stack and
setting all bits in it just to use it as an initial mask for allowing
all cpus.

Quoted 8 lines
Quoted 5 lines

Save the stack space and omit the need for runtime initialization by
defining a globally accessible cpumask_all variable.

Regards,
Dario

Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw

Re: [PATCH 1/2] xen: introduce a cpumask with all bits set

In reply to Juergen Gross

xen/arch/x86/io_apic.c static void __init check_timer(void) · line 1894

Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 14.03.19 at 10:59, <[email protected]> wrote:
xen/arch/x86/io_apic.c +1 -2
@@ -1892,8 +1891,7 @@ static void __init check_timer(void)
     vector = IRQ0_VECTOR;
     clear_irq_vector(0);
 
-    cpumask_setall(&mask_all);
-    if ((ret = bind_irq_vector(0, vector, &mask_all)))
+    if ((ret = bind_irq_vector(0, vector, &cpumask_all)))

Since you replace the users of cpumask_setall(), wouldn't it be better to also remove that function from the header?

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

A message that is not in the archive (sent off-list or not yet received): [email protected]

#8 lore raw

Re: [PATCH 1/2] xen: introduce a cpumask with all bits set

Quoted 11 lines · Jan Beulich

xen/arch/x86/io_apic.c static void __init check_timer(void) · line 1894

Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 14.03.19 at 10:59, <[email protected]> wrote:
xen/arch/x86/io_apic.c +1 -2
@@ -1892,8 +1891,7 @@ static void __init check_timer(void)
     vector = IRQ0_VECTOR;
     clear_irq_vector(0);
 
-    cpumask_setall(&mask_all);
-    if ((ret = bind_irq_vector(0, vector, &mask_all)))
+    if ((ret = bind_irq_vector(0, vector, &cpumask_all)))

Since you replace the users of cpumask_setall(), wouldn't it be better to also remove that function from the header?

I don't replace all of them. There are still some users left where replacing with cpumask_all would be less optimal.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#9 lore raw

Re: [PATCH 1/2] xen: introduce a cpumask with all bits set

In reply to Juergen Gross
Quoted 12 lines
Quoted 1 line
Quoted 1 line · On 14.03.19 at 10:59, <[email protected]> wrote:
xen/arch/x86/io_apic.c +0 -1
@@ -1881,7 +1881,6 @@ static void __init check_timer(void)
     int apic1, pin1, apic2, pin2;
     int vector, ret;
     unsigned long flags;
-    cpumask_t mask_all;
 
     local_irq_save(flags);
 

xen/arch/x86/io_apic.c static void __init check_timer(void) · line 1895

Quoted 7 lines
xen/arch/x86/io_apic.c +1 -2
@@ -1892,8 +1891,7 @@ static void __init check_timer(void)
     vector = IRQ0_VECTOR;
     clear_irq_vector(0);
 
-    cpumask_setall(&mask_all);
-    if ((ret = bind_irq_vector(0, vector, &mask_all)))
+    if ((ret = bind_irq_vector(0, vector, &cpumask_all)))
         printk(KERN_ERR"..IRQ0 is not set correctly with ioapic!!!, err:%d\n", ret);

I actually question this use: check_timer() gets called before any APs get onlined, so cpumask_of(0) should be equally fine here. But the uses in the scheduler perhaps still warrant the new object, and the code here can as well be cleaned up later.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#10 lore raw

PATCH 2/2 xen/sched: don't disable scheduler on cpus during suspend

In reply to Juergen Gross

Today there is special handling in cpu_disable_scheduler() for suspend by forcing all vcpus to the boot cpu. In fact there is no need for that as during resume the vcpus are put on the correct cpus again.

So we can just omit the call of cpu_disable_scheduler() when offlining a cpu due to suspend and on resuming we can omit taking the schedule lock for selecting the new processor.

In restore_vcpu_affinity() we should be careful when applying affinity as the cpu might not have come back to life. This in turn enables us to even support affinity_broken across suspend/resume.

---
 xen/common/cpupool.c  |   2 +-
 xen/common/schedule.c | 146 ++++++++++++++------------------------------------
 2 files changed, 42 insertions(+), 106 deletions(-)
xen/common/cpupool.c +1 -1
index e89bb67e71..7b5ce18426 100644
@@ -313,7 +313,7 @@ static long cpupool_unassign_cpu_helper(void *info)
      * cpu_disable_scheduler(), and at the bottom of this function.
      */
     rcu_read_lock(&domlist_read_lock);
-    ret = cpu_disable_scheduler(cpu);
+    ret = (system_state == SYS_STATE_suspend) ? 0 : cpu_disable_scheduler(cpu);
     cpumask_set_cpu(cpu, &cpupool_free_cpus);
 
     /*
xen/common/schedule.c +41 -105

186 lines not on this page · raw message

Signature · 7 lines
2.16.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#11 lore raw

Re: [PATCH 2/2] xen/sched: don't disable scheduler on cpus during suspend

Quoted 6 lines · Juergen Gross

Today there is special handling in cpu_disable_scheduler() for suspend
by forcing all vcpus to the boot cpu. In fact there is no need for that
as during resume the vcpus are put on the correct cpus again.

So, basically, you're saying that moving the vcpus to BSP / CPU0, before suspending is not necessary, right?

I thought it was, because going into suspend involves things like cpu_schedule_down(), which does, among other things, SCHED_OP(free_pdata) for freeing the scheduler's data structs, and hence we did not want vcpus "around".

Quoted 5 lines

So we can just omit the call of cpu_disable_scheduler() when offlining
a cpu due to suspend and on resuming we can omit taking the schedule lock for selecting the new processor.

Well, in theory, we should hold the lock when for using 'cpumask_scratch_cpu(cpu)'. Not sure this is an actual problem in this case, but still...

Quoted 4 lines

In restore_vcpu_affinity() we should be careful when applying affinity
as the cpu might not have come back to life.

But restore_vcpu_affinity() is done, in a loop, for all vcpus of all domains, in thaw_domains(), which in turn comes after enable_nonboot_cpus(), which I think is supposed to bring every CPU back up.... What am I missing?

xen/common/cpupool.c static long cpupool_unassign_cpu_helper(void · line 319

Quoted 13 lines
xen/common/cpupool.c +1 -1
index e89bb67e71..7b5ce18426 100644
@@ -313,7 +313,7 @@ static long cpupool_unassign_cpu_helper(void
 *info)
      * cpu_disable_scheduler(), and at the bottom of this function.
      */
     rcu_read_lock(&domlist_read_lock);
-    ret = cpu_disable_scheduler(cpu);
+    ret = (system_state == SYS_STATE_suspend) ? 0 :
 cpu_disable_scheduler(cpu);
 

Mmm... How can this function be called with state = STATE_suspend ?

Quoted 48 lines
xen/common/cpupool.c +0 -10
@@ -735,31 +708,36 @@ void restore_vcpu_affinity(struct domain *d)
 
         ASSERT(!vcpu_runnable(v));
 
-        lock = vcpu_schedule_lock_irq(v);
-
-        if ( v->affinity_broken )
-        {
-            sched_set_affinity(v, v->cpu_hard_affinity_saved, NULL);
-            v->affinity_broken = 0;
-
-        }
-
         /*
-         * During suspend (in cpu_disable_scheduler()), we moved

every vCPU
- * to BSP (which, as of now, is pCPU 0), as a temporary
measure to
- * allow the nonboot processors to have their data structure
freed
- * and go to sleep. But nothing guardantees that the BSP is
a valid

xen/common/cpupool.c +1 -1
-         * pCPU for a particular domain.
+         * Re-assign the initial processor as after resume we have

no
+ * guarantee the old processor has come back to life again.
*
* Therefore, here, before actually unpausing the domains,
we should
* set v->processor of each of their vCPUs to something that
will
* make sense for the scheduler of the cpupool in which they
are in.

          */
         cpumask_and(cpumask_scratch_cpu(cpu), v->cpu_hard_affinity,
xen/common/cpupool.c +6 -2
-                    cpupool_domain_cpumask(v->domain));
-        v->processor = cpumask_any(cpumask_scratch_cpu(cpu));
+                    cpupool_domain_cpumask(d));
+        if ( cpumask_empty(cpumask_scratch_cpu(cpu)) )
+        {
+            if ( v->affinity_broken )
+            {
+                sched_set_affinity(v, v->cpu_hard_affinity_saved,

NULL);

xen/common/cpupool.c +2 -0
+                v->affinity_broken = 0;
+                cpumask_and(cpumask_scratch_cpu(cpu), v-
Quoted 1 line

cpu_hard_affinity,

xen/common/cpupool.c

Quoted 2 lines
xen/common/cpupool.c +2 -0
+                            cpupool_domain_cpumask(d));
+            }

I'm not sure I follow this; I'll think deeper, but, in the meanwhile...

Quoted 10 lines
xen/common/cpupool.c +3 -1
-        spin_unlock_irq(lock);
+            if ( cpumask_empty(cpumask_scratch_cpu(cpu)) )
+            {
+                printk(XENLOG_DEBUG "Breaking affinity for %pv\n",

v);

xen/common/cpupool.c +2 -0
+                sched_set_affinity(v, &cpumask_all, NULL);
+                cpumask_and(cpumask_scratch_cpu(cpu), v-
Quoted 1 line

cpu_hard_affinity,

+ cpupool_domain_cpumask(d));

... what does it mean we're breaking the affinity while trying to restore it?

When/where is this vcpu getting its real affinity back in place then?

It might well be me, but I feel like there is something I'm missing about how this patch is supposed to work.

Regards,
Dario

Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#12 lore raw

Re: [PATCH 2/2] xen/sched: don't disable scheduler on cpus during suspend

Quoted 9 lines · Dario Faggioli
Quoted 6 lines · Juergen Gross

Today there is special handling in cpu_disable_scheduler() for suspend
by forcing all vcpus to the boot cpu. In fact there is no need for that
as during resume the vcpus are put on the correct cpus again.

So, basically, you're saying that moving the vcpus to BSP / CPU0, before suspending is not necessary, right?

Right.

Quoted 4 lines

I thought it was, because going into suspend involves things like cpu_schedule_down(), which does, among other things, SCHED_OP(free_pdata) for freeing the scheduler's data structs, and hence we did not want vcpus "around".

Suspend is done from a tasklet with all domains "frozen". So there is no active vcpu other than the idle ones in the system, meaning that there is no vcpu in any runqueue.

Quoted 9 lines
Quoted 6 lines

So we can just omit the call of cpu_disable_scheduler() when offlining
a cpu due to suspend and on resuming we can omit taking the schedule lock for selecting the new processor.

Well, in theory, we should hold the lock when for using 'cpumask_scratch_cpu(cpu)'. Not sure this is an actual problem in this case, but still...

I think there is no problem as there is no scheduling activity.

Quoted 8 lines
Quoted 4 lines

In restore_vcpu_affinity() we should be careful when applying affinity
as the cpu might not have come back to life.

But restore_vcpu_affinity() is done, in a loop, for all vcpus of all domains, in thaw_domains(), which in turn comes after enable_nonboot_cpus(), which I think is supposed to bring every CPU back up.... What am I missing?

There is no guarantee all cpus will be up after suspend. Some might fail coming up.

Quoted 15 lines
Quoted 6 lines
xen/common/cpupool.c +0 -0
index e89bb67e71..7b5ce18426 100644

xen/common/cpupool.c static long cpupool_unassign_cpu_helper(void · line 319

Quoted 8 lines
xen/common/cpupool.c +1 -1
@@ -313,7 +313,7 @@ static long cpupool_unassign_cpu_helper(void
 *info)
      * cpu_disable_scheduler(), and at the bottom of this function.
      */
     rcu_read_lock(&domlist_read_lock);
-    ret = cpu_disable_scheduler(cpu);
+    ret = (system_state == SYS_STATE_suspend) ? 0 :
 cpu_disable_scheduler(cpu);
 

Mmm... How can this function be called with state = STATE_suspend ?

Meh, my bad. I masked the wrong call of cpu_disable_scheduler(). Should have been that from __cpu_disable().

Quoted 60 lines
Quoted 49 lines
xen/common/cpupool.c +0 -10
@@ -735,31 +708,36 @@ void restore_vcpu_affinity(struct domain *d)
 
         ASSERT(!vcpu_runnable(v));
 
-        lock = vcpu_schedule_lock_irq(v);
-
-        if ( v->affinity_broken )
-        {
-            sched_set_affinity(v, v->cpu_hard_affinity_saved, NULL);
-            v->affinity_broken = 0;
-
-        }
-
         /*
-         * During suspend (in cpu_disable_scheduler()), we moved

every vCPU
- * to BSP (which, as of now, is pCPU 0), as a temporary
measure to
- * allow the nonboot processors to have their data structure
freed
- * and go to sleep. But nothing guardantees that the BSP is
a valid

xen/common/cpupool.c +1 -1
-         * pCPU for a particular domain.
+         * Re-assign the initial processor as after resume we have

no
+ * guarantee the old processor has come back to life again.
*
* Therefore, here, before actually unpausing the domains,
we should
* set v->processor of each of their vCPUs to something that
will
* make sense for the scheduler of the cpupool in which they
are in.

          */
         cpumask_and(cpumask_scratch_cpu(cpu), v->cpu_hard_affinity,
xen/common/cpupool.c +6 -2
-                    cpupool_domain_cpumask(v->domain));
-        v->processor = cpumask_any(cpumask_scratch_cpu(cpu));
+                    cpupool_domain_cpumask(d));
+        if ( cpumask_empty(cpumask_scratch_cpu(cpu)) )
+        {
+            if ( v->affinity_broken )
+            {
+                sched_set_affinity(v, v->cpu_hard_affinity_saved,

NULL);

xen/common/cpupool.c +2 -0
+                v->affinity_broken = 0;
+                cpumask_and(cpumask_scratch_cpu(cpu), v-
Quoted 1 line

cpu_hard_affinity,

xen/common/cpupool.c

Quoted 2 lines
xen/common/cpupool.c +2 -0
+                            cpupool_domain_cpumask(d));
+            }

I'm not sure I follow this; I'll think deeper, but, in the meanwhile...

Quoted 7 lines
xen/common/cpupool.c +3 -1
-        spin_unlock_irq(lock);
+            if ( cpumask_empty(cpumask_scratch_cpu(cpu)) )
+            {
+                printk(XENLOG_DEBUG "Breaking affinity for %pv\n",

v);

xen/common/cpupool.c +2 -0
+                sched_set_affinity(v, &cpumask_all, NULL);
+                cpumask_and(cpumask_scratch_cpu(cpu), v-
Quoted 7 lines
Quoted 3 lines
Quoted 1 line

cpu_hard_affinity,

+ cpupool_domain_cpumask(d));

... what does it mean we're breaking the affinity while trying to restore it?

When/where is this vcpu getting its real affinity back in place then?

This is only happening in case a cpu didn't come up again. There is no way to restore the real affinity in this case.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#13 lore raw

Re: [PATCH 2/2] xen/sched: don't disable scheduler on cpus during suspend

Quoted 8 lines · Juergen Gross
Quoted 6 lines · Dario Faggioli
Quoted 6 lines

So we can just omit the call of cpu_disable_scheduler() when offlining
a cpu due to suspend and on resuming we can omit taking the schedule
lock for selecting the new processor.

Quoted 7 lines
Quoted 4 lines

Well, in theory, we should hold the lock when for using 'cpumask_scratch_cpu(cpu)'. Not sure this is an actual problem in this
case, but still...

I think there is no problem as there is no scheduling activity.

I concur. I'm wondering whether it's worth adding a brief comment about it. If it were me doing this, I most likely would.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

In restore_vcpu_affinity() we should be careful when applying affinity
as the cpu might not have come back to life.

Quoted 9 lines
Quoted 5 lines

But restore_vcpu_affinity() is done, in a loop, for all vcpus of all
domains, in thaw_domains(), which in turn comes after enable_nonboot_cpus(), which I think is supposed to bring every CPU back up.... What am I missing?

There is no guarantee all cpus will be up after suspend. Some might fail coming up.

Ah, ok, that's the case we're trying to address. I didn't get it in the first place.

And this, in fact, was what I was missing, and the reason why I was finding some aspects of the patch a little weird. :-P

So, can we state this a bit better in the changelog, and also in a comment? I'd like the reason why we break the affinity in a function called "repair_affinity" to be more obvious. :-P

Quoted 16 lines
Quoted 7 lines
Quoted 7 lines
xen/common/cpupool.c +0 -0
index e89bb67e71..7b5ce18426 100644
@@ -313,7 +313,7 @@ static long cpupool_unassign_cpu_helper(void
 *info)
      * cpu_disable_scheduler(), and at the bottom of this
Quoted 8 lines
Quoted 7 lines

function.

      */
     rcu_read_lock(&domlist_read_lock);
xen/common/cpupool.c +1 -1
-    ret = cpu_disable_scheduler(cpu);
+    ret = (system_state == SYS_STATE_suspend) ? 0 :

cpu_disable_scheduler(cpu);

Mmm... How can this function be called with state = STATE_suspend ?

Quoted 4 lines

Meh, my bad. I masked the wrong call of cpu_disable_scheduler(). Should
have been that from __cpu_disable().

Yes, that should be the one.

Regards,
Dario

Signature · 4 lines
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Software Engineer @ SUSE https://www.suse.com/

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel