Skip to contentSkip to conversation

xen/sched: fix csched2_deinit_pdata()

PATCH In review Juergen Gross 3 participants 6 messages May 8, 2019 – May 17, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xen/sched: fix csched2_deinit_pdata()

Commit 753ba43d6d16e688 ("xen/sched: fix credit2 smt idle handling") introduced a regression when switching cpus between cpupools.

When assigning a cpu to a cpupool with credit2 being the default scheduler csched2_deinit_pdata() is called for the credit2 private data after the new scheduler's private data has been hooked to the per-cpu scheduler data. Unfortunately csched2_deinit_pdata() will cycle through all per-cpu scheduler areas it knows of for removing the cpu from the respective sibling masks including the area of the just moved cpu. This will (depending on the new scheduler) either clobber the data of the new scheduler or in case of sched_rt lead to a crash.

Avoid that by removing the cpu from the list of active cpus in credit2 data first.

The opposite problem is occurring when removing a cpu from a cpupool:
init_pdata() of credit2 will access the per-cpu data of the old scheduler.

---
 xen/common/sched_credit2.c | 23 +++++++++++------------
 1 file changed, 11 insertions(+), 12 deletions(-)
xen/common/sched_credit2.c +11 -12
index 6958b265fc..9c1c3b4e08 100644
@@ -3813,22 +3813,21 @@ init_pdata(struct csched2_private *prv, struct csched2_pcpu *spc,
         activate_runqueue(prv, spc->runq_id);
     }
 
-    __cpumask_set_cpu(cpu, &rqd->idle);
-    __cpumask_set_cpu(cpu, &rqd->active);
-    __cpumask_set_cpu(cpu, &prv->initialized);
-    __cpumask_set_cpu(cpu, &rqd->smt_idle);
+    __cpumask_set_cpu(cpu, &spc->sibling_mask);
 
-    /* On the boot cpu we are called before cpu_sibling_mask has been set up. */
-    if ( cpu == 0 && system_state < SYS_STATE_active )
-        __cpumask_set_cpu(cpu, &csched2_pcpu(cpu)->sibling_mask);
-    else
+    if ( cpumask_weight(&rqd->active) > 0 )
         for_each_cpu ( rcpu, per_cpu(cpu_sibling_mask, cpu) )
             if ( cpumask_test_cpu(rcpu, &rqd->active) )
             {
                 __cpumask_set_cpu(cpu, &csched2_pcpu(rcpu)->sibling_mask);
-                __cpumask_set_cpu(rcpu, &csched2_pcpu(cpu)->sibling_mask);
+                __cpumask_set_cpu(rcpu, &spc->sibling_mask);
             }
 
+    __cpumask_set_cpu(cpu, &rqd->idle);
+    __cpumask_set_cpu(cpu, &rqd->active);
+    __cpumask_set_cpu(cpu, &prv->initialized);
+    __cpumask_set_cpu(cpu, &rqd->smt_idle);
+
     if ( cpumask_weight(&rqd->active) == 1 )
         rqd->pick_bias = cpu;
 
@@ -3937,13 +3936,13 @@ csched2_deinit_pdata(const struct scheduler *ops, void *pcpu, int cpu)
 
     printk(XENLOG_INFO "Removing cpu %d from runqueue %d\n", cpu, spc->runq_id);
 
-    for_each_cpu ( rcpu, &rqd->active )
-        __cpumask_clear_cpu(cpu, &csched2_pcpu(rcpu)->sibling_mask);
-
     __cpumask_clear_cpu(cpu, &rqd->idle);
     __cpumask_clear_cpu(cpu, &rqd->smt_idle);
     __cpumask_clear_cpu(cpu, &rqd->active);
 
+    for_each_cpu ( rcpu, &rqd->active )
+        __cpumask_clear_cpu(cpu, &csched2_pcpu(rcpu)->sibling_mask);
+
     if ( cpumask_empty(&rqd->active) )
     {
         printk(XENLOG_INFO " No cpus left on runqueue, disabling\n");
Signature · 7 lines
2.16.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v1 1/1
#1 lore raw
Quoted 15 lines · Juergen Gross

Commit 753ba43d6d16e688 ("xen/sched: fix credit2 smt idle handling") introduced a regression when switching cpus between cpupools.

When assigning a cpu to a cpupool with credit2 being the default scheduler csched2_deinit_pdata() is called for the credit2 private data
after the new scheduler's private data has been hooked to the per-cpu scheduler data. Unfortunately csched2_deinit_pdata() will cycle through
all per-cpu scheduler areas it knows of for removing the cpu from the respective sibling masks including the area of the just moved cpu. This
will (depending on the new scheduler) either clobber the data of the new scheduler or in case of sched_rt lead to a crash.

These issues will drive us all crazy, at some point! :-(

I really wish I managed to put a cpupool test in OSSTest. I guess I can pickup my old draft, but I need some time to put together an OSSTest dev environment again. :-/

Anyway...

Quoted 2 lines

Thanks and Regards

Signature · 6 lines
Dario Faggioli, Ph.D
http://about.me/dario.faggioli
Virtualization Software Engineer
SUSE Labs, SUSE https://www.suse.com/
-------------------------------------------------------------------
<<This happens because _I_ choose it to happen!>> (Raistlin Majere)

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#2 lore raw

Ping: [PATCH] xen/sched: fix csched2_deinit_pdata()

In reply to Juergen Gross
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 08.05.19 at 13:31, <[email protected]> wrote:

Commit 753ba43d6d16e688 ("xen/sched: fix credit2 smt idle handling") introduced a regression when switching cpus between cpupools.

When assigning a cpu to a cpupool with credit2 being the default scheduler csched2_deinit_pdata() is called for the credit2 private data after the new scheduler's private data has been hooked to the per-cpu scheduler data. Unfortunately csched2_deinit_pdata() will cycle through all per-cpu scheduler areas it knows of for removing the cpu from the respective sibling masks including the area of the just moved cpu. This will (depending on the new scheduler) either clobber the data of the new scheduler or in case of sched_rt lead to a crash.

Quoted 8 lines

Avoid that by removing the cpu from the list of active cpus in credit2 data first.

The opposite problem is occurring when removing a cpu from a cpupool:
init_pdata() of credit2 will access the per-cpu data of the old scheduler.

May I ask what the disposition of this is? I've noticed too late that I've backported the commit being fixed here without waiting for this fix to go in. I'd prefer the stable trees, in particular 4.11 for the impending 4.11.2 release, to be able to pick this up soon.

Thanks, Jan

Quoted 51 lines
xen/common/sched_credit2.c +1 -5
@@ -3813,22 +3813,21 @@ init_pdata(struct csched2_private *prv, struct csched2_pcpu *spc,
         activate_runqueue(prv, spc->runq_id);
     }
 
-    __cpumask_set_cpu(cpu, &rqd->idle);
-    __cpumask_set_cpu(cpu, &rqd->active);
-    __cpumask_set_cpu(cpu, &prv->initialized);
-    __cpumask_set_cpu(cpu, &rqd->smt_idle);
+    __cpumask_set_cpu(cpu, &spc->sibling_mask);
 
-    /* On the boot cpu we are called before cpu_sibling_mask has been set 

up. */

xen/common/sched_credit2.c +10 -7
-    if ( cpu == 0 && system_state < SYS_STATE_active )
-        __cpumask_set_cpu(cpu, &csched2_pcpu(cpu)->sibling_mask);
-    else
+    if ( cpumask_weight(&rqd->active) > 0 )
         for_each_cpu ( rcpu, per_cpu(cpu_sibling_mask, cpu) )
             if ( cpumask_test_cpu(rcpu, &rqd->active) )
             {
                 __cpumask_set_cpu(cpu, &csched2_pcpu(rcpu)->sibling_mask);
-                __cpumask_set_cpu(rcpu, &csched2_pcpu(cpu)->sibling_mask);
+                __cpumask_set_cpu(rcpu, &spc->sibling_mask);
             }
 
+    __cpumask_set_cpu(cpu, &rqd->idle);
+    __cpumask_set_cpu(cpu, &rqd->active);
+    __cpumask_set_cpu(cpu, &prv->initialized);
+    __cpumask_set_cpu(cpu, &rqd->smt_idle);
+
     if ( cpumask_weight(&rqd->active) == 1 )
         rqd->pick_bias = cpu;
@@ -3937,13 +3936,13 @@ csched2_deinit_pdata(const struct scheduler *ops, 
 void *pcpu, int cpu)
 
     printk(XENLOG_INFO "Removing cpu %d from runqueue %d\n", cpu, 
 spc->runq_id);
 
-    for_each_cpu ( rcpu, &rqd->active )
-        __cpumask_clear_cpu(cpu, &csched2_pcpu(rcpu)->sibling_mask);
-
     __cpumask_clear_cpu(cpu, &rqd->idle);
     __cpumask_clear_cpu(cpu, &rqd->smt_idle);
     __cpumask_clear_cpu(cpu, &rqd->active);
 
+    for_each_cpu ( rcpu, &rqd->active )
+        __cpumask_clear_cpu(cpu, &csched2_pcpu(rcpu)->sibling_mask);
+
     if ( cpumask_empty(&rqd->active) )
Quoted 10 lines
     {
         printk(XENLOG_INFO " No cpus left on runqueue, disabling\n");
Signature · 7 lines
2.16.4


_______________________________________________
Xen-devel mailing list
[email protected] 
https://lists.xenproject.org/mailman/listinfo/xen-devel 

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw

Re: Ping: [PATCH] xen/sched: fix csched2_deinit_pdata()

Quoted 20 lines · Jan Beulich
Quoted 20 lines
Quoted 1 line
Quoted 1 line · On 08.05.19 at 13:31, <[email protected]> wrote:

Commit 753ba43d6d16e688 ("xen/sched: fix credit2 smt idle handling") introduced a regression when switching cpus between cpupools.

When assigning a cpu to a cpupool with credit2 being the default scheduler csched2_deinit_pdata() is called for the credit2 private data after the new scheduler's private data has been hooked to the per-cpu scheduler data. Unfortunately csched2_deinit_pdata() will cycle through all per-cpu scheduler areas it knows of for removing the cpu from the respective sibling masks including the area of the just moved cpu. This will (depending on the new scheduler) either clobber the data of the new scheduler or in case of sched_rt lead to a crash.

Avoid that by removing the cpu from the list of active cpus in credit2 data first.

The opposite problem is occurring when removing a cpu from a cpupool:
init_pdata() of credit2 will access the per-cpu data of the old scheduler.

Quoted 7 lines
Quoted 1 line

May I ask what the disposition of this is? I've noticed too late that I've backported the commit being fixed here without waiting for this fix to go in. I'd prefer the stable trees, in particular 4.11 for the impending 4.11.2 release, to be able to pick this up soon.

Dario already gave his Reviewed-by:

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw

Re: Ping: [PATCH] xen/sched: fix csched2_deinit_pdata()

Quoted 24 lines
Quoted 1 line
Quoted 1 line · On 17.05.19 at 15:24, <[email protected]> wrote:
Quoted 22 lines · Jan Beulich
Quoted 21 lines
Quoted 1 line
Quoted 1 line · On 08.05.19 at 13:31, <[email protected]> wrote:

Commit 753ba43d6d16e688 ("xen/sched: fix credit2 smt idle handling") introduced a regression when switching cpus between cpupools.

When assigning a cpu to a cpupool with credit2 being the default scheduler csched2_deinit_pdata() is called for the credit2 private data after the new scheduler's private data has been hooked to the per-cpu scheduler data. Unfortunately csched2_deinit_pdata() will cycle through all per-cpu scheduler areas it knows of for removing the cpu from the respective sibling masks including the area of the just moved cpu. This will (depending on the new scheduler) either clobber the data of the new scheduler or in case of sched_rt lead to a crash.

Avoid that by removing the cpu from the list of active cpus in credit2 data first.

The opposite problem is occurring when removing a cpu from a cpupool:
init_pdata() of credit2 will access the per-cpu data of the old scheduler.

Quoted 7 lines
Quoted 5 lines

May I ask what the disposition of this is? I've noticed too late that I've backported the commit being fixed here without waiting for this fix to go in. I'd prefer the stable trees, in particular 4.11 for the impending 4.11.2 release, to be able to pick this up soon.

Dario already gave his Reviewed-by:

Oh, my fault then - apologies.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw

Re: Ping: [PATCH] xen/sched: fix csched2_deinit_pdata()

Quoted 30 lines · Jan Beulich
Quoted 30 lines
Quoted 1 line
Quoted 1 line · On 17.05.19 at 15:24, <[email protected]> wrote:
Quoted 27 lines · Jan Beulich
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 08.05.19 at 13:31, <[email protected]> wrote:

Commit 753ba43d6d16e688 ("xen/sched: fix credit2 smt idle handling") introduced a regression when switching cpus between cpupools.

When assigning a cpu to a cpupool with credit2 being the default scheduler csched2_deinit_pdata() is called for the credit2 private data after the new scheduler's private data has been hooked to the per-cpu scheduler data. Unfortunately csched2_deinit_pdata() will cycle through all per-cpu scheduler areas it knows of for removing the cpu from the respective sibling masks including the area of the just moved cpu. This will (depending on the new scheduler) either clobber the data of the new scheduler or in case of sched_rt lead to a crash.

Quoted 8 lines

Avoid that by removing the cpu from the list of active cpus in credit2 data first.

The opposite problem is occurring when removing a cpu from a cpupool:
init_pdata() of credit2 will access the per-cpu data of the old scheduler.

May I ask what the disposition of this is? I've noticed too late that I've backported the commit being fixed here without waiting for this fix to go in. I'd prefer the stable trees, in particular 4.11 for the impending 4.11.2 release, to be able to pick this up soon.

Quoted 3 lines
Quoted 1 line

Dario already gave his Reviewed-by:

Oh, my fault then - apologies.

NP. In case you need the pointer:

https://lists.xen.org/archives/html/xen-devel/2019-05/msg00734.html

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel