xen: sched: Credit1 shouldn't boost vcpus being migrated.

PATCH New Dario Faggioli 3 participants 6 messages Feb 12, 2016

lore

Share & tools

lore

Dario Faggioli

author

2016-02-12 09:36 UTC #0 lore raw

cover xen: sched: Credit1 shouldn't boost vcpus being migrated.

Hi again,

Here it comes v2, redone following Jan's suggestion, which allowed to get rid of patch 2, and do everything in sched_credit.c.

So, in summary, because of the fact that vcpu_migrate() forces the vcpus into a sleep+wakeup cycle, vcpus being migrated to a new pcpu, were also being granted BOOST priority, inside Credit1, and that is not correct.

More info on v1's cover letter, which is here:

http://lists.xenproject.org/archives/html/xen-devel/2016-02/msg01620.html

I re-run the same set of benchmarks described in v1, and the result for this rework of the series are basically the same as there.

Thanks and Regards,
Dario
---
Dario Faggioli (2):
xen: credit1: trace vCPU boost/unboost
xen: credit1: avoid boosting vCPUs being "just" migrated

 xen/common/sched_credit.c    |   34 ++++++++++++++++++++++++++++++----
 xen/include/xen/perfc_defn.h |    1 +
 2 files changed, 31 insertions(+), 4 deletions(-)

--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-------------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Dario Faggioli

author

2016-02-12 09:37 UTC #1 lore raw

PATCH v2 1/2 xen: credit1: trace vCPU boost/unboost

Add tracepoints and a performance counter for boosting and unboosting in Credit1.

Note that they (the trace points) do not cover the case of the idle vCPU being boosted to run a tasklet, as there already is TRC_CSCHED_SCHED_TASKLET for that.

Signed-off-by Dario Faggioli <[email protected]>

---
Cc: George Dunlap <[email protected]>

---
 xen/common/sched_credit.c    |    8 ++++++++
 xen/include/xen/perfc_defn.h |    1 +
 2 files changed, 9 insertions(+)

xen/common/sched_credit.c +8 -0

index 671bbee..5708701 100644
@@ -126,6 +126,8 @@
 #define TRC_CSCHED_STOLEN_VCPU   TRC_SCHED_CLASS_EVT(CSCHED, 4)
 #define TRC_CSCHED_PICKED_CPU    TRC_SCHED_CLASS_EVT(CSCHED, 5)
 #define TRC_CSCHED_TICKLE        TRC_SCHED_CLASS_EVT(CSCHED, 6)
+#define TRC_CSCHED_BOOST_START   TRC_SCHED_CLASS_EVT(CSCHED, 7)
+#define TRC_CSCHED_BOOST_END     TRC_SCHED_CLASS_EVT(CSCHED, 8)
 
 
 /*

@@ -856,7 +858,11 @@ csched_vcpu_acct(struct csched_private *prv, unsigned int cpu)
      * amount of CPU resources and should no longer be boosted.
      */
     if ( svc->pri == CSCHED_PRI_TS_BOOST )
+    {
         svc->pri = CSCHED_PRI_TS_UNDER;
+        TRACE_2D(TRC_CSCHED_BOOST_END, svc->sdom->dom->domain_id,
+                 svc->vcpu->vcpu_id);
+    }
 
     /*
      * Update credits

@@ -1023,6 +1029,8 @@ csched_vcpu_wake(const struct scheduler *ops, struct vcpu *vc)
     if ( svc->pri == CSCHED_PRI_TS_UNDER &&
          !test_bit(CSCHED_FLAG_VCPU_PARKED, &svc->flags) )
     {
+        TRACE_2D(TRC_CSCHED_BOOST_START, vc->domain->domain_id, vc->vcpu_id);
+        SCHED_STAT_CRANK(vcpu_boost);
         svc->pri = CSCHED_PRI_TS_BOOST;
     }

xen/include/xen/perfc_defn.h +1 -0

index 76ee803..21c1e0b 100644
@@ -40,6 +40,7 @@ PERFCOUNTER(acct_reorder,           "csched: acct_reorder")
 PERFCOUNTER(acct_min_credit,        "csched: acct_min_credit")
 PERFCOUNTER(acct_vcpu_active,       "csched: acct_vcpu_active")
 PERFCOUNTER(acct_vcpu_idle,         "csched: acct_vcpu_idle")
+PERFCOUNTER(vcpu_boost,             "csched: vcpu_boost")
 PERFCOUNTER(vcpu_park,              "csched: vcpu_park")
 PERFCOUNTER(vcpu_unpark,            "csched: vcpu_unpark")
 PERFCOUNTER(load_balance_idle,      "csched: load_balance_idle")

Dario Faggioli

author

2016-02-12 09:37 UTC #2 lore raw

PATCH v2 2/2 xen: credit1: avoid boosting vCPUs being "just" migrated

In reply to Dario Faggioli

Moving a vCPU to a different pCPU means offlining it and then waking it up, on the new pCPU. Credit1 grants BOOST priority to vCPUs that wakes up, with the aim of improving I/O latency. The net effect of this all is that vCPUs get boosted when migrating, which shouldn't happen.

For instance, this causes scheduling anomalies and, potentially, performance problems, as reported here:
http://lists.xen.org/archives/html/xen-devel/2015-10/msg02851.html

This patch fixes this by noting down (by means of a flag) the fact that the vCPU is about to undergo a migration. This way we can tell, later, during a wakeup, whether the vCPU is migrating or unblocking, and decide whether or not to apply the boosting.

Signed-off-by Dario Faggioli <[email protected]>

---
Cc: George Dunlap <[email protected]>
Cc: Jan Beulich <[email protected]>
---
Changes from v1:
* rewritten, following suggestion got during review: there are no wakeup flags any longer, and all is done in sched_credit.c by setting a flag in csched_cpu_pick() and testing (and cleating) it in csched_vcpu_wake().

---
 xen/common/sched_credit.c |   26 ++++++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)

xen/common/sched_credit.c +22 -4

index 5708701..756e884 100644
@@ -66,6 +66,7 @@
  */
 #define CSCHED_FLAG_VCPU_PARKED    0x0  /* VCPU over capped credits */
 #define CSCHED_FLAG_VCPU_YIELD     0x1  /* VCPU yielding */
+#define CSCHED_FLAG_VCPU_MIGRATING 0x2  /* VCPU may have moved to a new pcpu */
 
 
 /*

@@ -787,6 +788,16 @@ _csched_cpu_pick(const struct scheduler *ops, struct vcpu *vc, bool_t commit)
 static int
 csched_cpu_pick(const struct scheduler *ops, struct vcpu *vc)
 {
+    struct csched_vcpu *svc = CSCHED_VCPU(vc);
+
+    /*
+     * We have been called by vcpu_migrate() (in schedule.c), as part
+     * of the process of seeing if vc can be migrated to another pcpu.
+     * We make a note about this in svc->flags so that later, in
+     * csched_vcpu_wake() (still called from vcpu_migrate()) we won't
+     * get boosted, which we don't deserve as we are "only" migrating.
+     */
+    set_bit(CSCHED_FLAG_VCPU_MIGRATING, &svc->flags);
     return _csched_cpu_pick(ops, vc, 1);
 }

@@ -1022,11 +1033,18 @@ csched_vcpu_wake(const struct scheduler *ops, struct vcpu *vc)
      * more CPU resource intensive VCPUs without impacting overall 
      * system fairness.
      *
-     * The one exception is for VCPUs of capped domains unpausing
-     * after earning credits they had overspent. We don't boost
-     * those.
+     * There are two cases, when we don't want to boost:
+     *  - VCPUs that are waking up after a migration, rather than
+     *    after having block;
+     *  - VCPUs of capped domains unpausing after earning credits
+     *    they had overspent.
+     *
+     * Note that checking whether we are "only" migrating must be
+     * done up front, as we do not want the clearing of the bit we
+     * set in csched_cpu_pick() to be short-circuited away.
      */
-    if ( svc->pri == CSCHED_PRI_TS_UNDER &&
+    if ( !__test_and_clear_bit(CSCHED_FLAG_VCPU_MIGRATING, &svc->flags)  &&
+         svc->pri == CSCHED_PRI_TS_UNDER &&
          !test_bit(CSCHED_FLAG_VCPU_PARKED, &svc->flags) )
     {
         TRACE_2D(TRC_CSCHED_BOOST_START, vc->domain->domain_id, vc->vcpu_id);

Jan Beulich

2016-02-12 09:51 UTC #3 lore raw

Re: [PATCH v2 2/2] xen: credit1: avoid boosting vCPUs being "just" migrated

Quoted 7 lines

Quoted 1 line

Quoted 1 line · On 12.02.16 at 10:37, <[email protected]> wrote:

xen/common/sched_credit.c +2 -0

@@ -787,6 +788,16 @@ _csched_cpu_pick(const struct scheduler *ops, struct vcpu *vc, bool_t commit)
 static int
 csched_cpu_pick(const struct scheduler *ops, struct vcpu *vc)
 {
+    struct csched_vcpu *svc = CSCHED_VCPU(vc);
+

xen/common/sched_credit.c _csched_cpu_pick(const struct scheduler *ops, struct vcpu *vc, bool_t commit) · line 802

Quoted 10 lines

xen/common/sched_credit.c +8 -0

+    /*
+     * We have been called by vcpu_migrate() (in schedule.c), as part
+     * of the process of seeing if vc can be migrated to another pcpu.
+     * We make a note about this in svc->flags so that later, in
+     * csched_vcpu_wake() (still called from vcpu_migrate()) we won't
+     * get boosted, which we don't deserve as we are "only" migrating.
+     */
+    set_bit(CSCHED_FLAG_VCPU_MIGRATING, &svc->flags);
     return _csched_cpu_pick(ops, vc, 1);
 }

I think you either want __set_bit() here or ...

Quoted 12 lines

xen/common/sched_credit.c +5 -3

@@ -1022,11 +1033,18 @@ csched_vcpu_wake(const struct scheduler *ops, struct vcpu *vc)
      * more CPU resource intensive VCPUs without impacting overall 
      * system fairness.
      *
-     * The one exception is for VCPUs of capped domains unpausing
-     * after earning credits they had overspent. We don't boost
-     * those.
+     * There are two cases, when we don't want to boost:
+     *  - VCPUs that are waking up after a migration, rather than
+     *    after having block;
+     *  - VCPUs of capped domains unpausing after earning credits
+     *    they had overspent.

xen/common/sched_credit.c csched_vcpu_wake(const struct scheduler *ops, struct vcpu *vc) · line 1049

Quoted 10 lines

xen/common/sched_credit.c +6 -1

+     *
+     * Note that checking whether we are "only" migrating must be
+     * done up front, as we do not want the clearing of the bit we
+     * set in csched_cpu_pick() to be short-circuited away.
      */
-    if ( svc->pri == CSCHED_PRI_TS_UNDER &&
+    if ( !__test_and_clear_bit(CSCHED_FLAG_VCPU_MIGRATING, &svc->flags)  &&
+         svc->pri == CSCHED_PRI_TS_UNDER &&
          !test_bit(CSCHED_FLAG_VCPU_PARKED, &svc->flags) )
     {

... you ought to use test_and_clear_bit() here.

Jan

Dario Faggioli

author

2016-02-12 10:50 UTC #4 lore raw

Re: [PATCH v2 2/2] xen: credit1: avoid boosting vCPUs being "just" migrated

Quoted 16 lines · Jan Beulich

Quoted 16 lines
Quoted 1 line
Quoted 1 line · On 12.02.16 at 10:37, <[email protected]> wrote:
(quoted diff) +0 -0
@@ -787,6 +788,16 @@ _csched_cpu_pick(const struct scheduler *ops,
 struct vcpu *vc, bool_t commit)
static int
csched_cpu_pick(const struct scheduler *ops, struct vcpu *vc) {
+    struct csched_vcpu *svc = CSCHED_VCPU(vc);
+
+    /*
+     * We have been called by vcpu_migrate() (in schedule.c), as part
+     * of the process of seeing if vc can be migrated to another pcpu.
+     * We make a note about this in svc->flags so that later, in +     * csched_vcpu_wake() (still called from vcpu_migrate()) we won't

Quoted 9 lines

Quoted 6 lines
+     * get boosted, which we don't deserve as we are "only" migrating.
+     */
+    set_bit(CSCHED_FLAG_VCPU_MIGRATING, &svc->flags); return _csched_cpu_pick(ops, vc, 1);
}
I think you either want __set_bit() here or ...

Yes, this is completely serialized by the vcpu's scheduler lock, so I indeed want __set_bit(), sorry for the overlook.

Thanks and Regards,
Dario

Signature · 4 lines

<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

Dario Faggioli

2016-02-12 14:16 UTC #5 lore raw

Re: [PATCH v2 2/2] xen: credit1: avoid boosting vCPUs being "just" migrated

[Yes, replying to myself]

Quoted 27 lines · Dario Faggioli

Quoted 17 lines · Jan Beulich
Quoted 17 lines
Quoted 1 line
Quoted 1 line · On 12.02.16 at 10:37, <[email protected]> wrote:
(quoted diff) +0 -0
@@ -787,6 +788,16 @@ _csched_cpu_pick(const struct scheduler
 *ops,
 struct vcpu *vc, bool_t commit)
static int
csched_cpu_pick(const struct scheduler *ops, struct vcpu *vc) {
+    struct csched_vcpu *svc = CSCHED_VCPU(vc);
+
+    /*
+     * We have been called by vcpu_migrate() (in schedule.c), as part
+     * of the process of seeing if vc can be migrated to another pcpu.
+     * We make a note about this in svc->flags so that later, in +     * csched_vcpu_wake() (still called from vcpu_migrate()) we won't
Quoted 9 lines
Quoted 6 lines
+     * get boosted, which we don't deserve as we are "only" migrating.
+     */
+    set_bit(CSCHED_FLAG_VCPU_MIGRATING, &svc->flags); return _csched_cpu_pick(ops, vc, 1);
}
I think you either want __set_bit() here or ...

Quoted 3 lines

Yes, this is completely serialized by the vcpu's scheduler lock, so I indeed want __set_bit(), sorry for the overlook.

Which is indeed the case, in the case of this svc->flags, but not for other cases when svc->flags is used, for manipulating the other two existing flags (see, for instance be6507509454adf3bb5a50b9406c88504e996d5a "credit1: Use atomic bit operations for the flags structure").

So what I want is really the opposite of what I said above: set_bit() is ok, and I need the atomic test_and_clear().

-ENEEDMORECOFFEEATMORNING :-/

Thanks again and Regards,
Dario

Signature · 4 lines

<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel