Skip to content

xen: sched: improve (a lot! :-D) Credit2 runqueue handling

Dario Faggioli xen-devel 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest9 patches 38 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/9xen: sched: fix an 'off by one \t' in credit2 debug dumpxen/common/sched_credit2.c+1 -12 replies
2/9xen: sched: improve scope and placement of credit2 boot parametersxen/common/sched_credit2.c+4 -43 replies
3/9xen: sched: make locking for {insert, remove}_vcpu consistentxen/common/sched_credit.c xen/common/sched_rt.c xen/common/schedule.c+28 -1215 replies
4/9xen: sched: add .init_pdata hook to the scheduler interfacexen/common/schedule.c xen/include/xen/sched-if.h+14 -310 replies
5/9xen: sched: make implementing .alloc_pdata optionalxen/common/sched_arinc653.c xen/common/schedule.c+3 -344 replies
6/9xen: sched: implement .init_pdata in all schedulersxen/common/sched_credit.c xen/common/sched_credit2.c xen/common/sched_rt.c+27 -680 replies
7/9xen: sched: fix per-socket runqueue creation in credit2xen/common/sched_credit2.c+43 -150 replies
8/9xen: sched: allow for choosing credit2 runqueues configuration at bootdocs/misc/xen-command-line.markdown xen/common/sched_credit2.c+63 -53 replies
9/9xen: sched: per-core runqueues as default in credit2docs/misc/xen-command-line.markdown xen/common/sched_credit2.c+2 -21 reply

Reviews trailers per patch

3 of 9 patches have a Reviewed-by or Acked-by.

PatchJuergen GrossGeorge Dunlap
1/9R-b A-b
2/9R-b
3/9
4/9
5/9
6/9
7/9
8/9
9/9R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v1 thread

Cover letter [PATCH 0/9] xen: sched: improve (a lot! :-D) Credit2 runqueue handling

Hi everyone,

Code is already there, in the Credit2 scheduler, for setting up one runqueue per each socket found in the system. However, that is not functional and only one runqueue is being created, with all the pCPUs in it.

Some more details are available here:

http://lists.xen.org/archives/html/xen-devel/2014-08/msg02168.html

As well as in our bug #36 "credit2 only uses one runqueue instead of one runq per socket" (http://bugs.xenproject.org/xen/bug/36).

Also, during a previous OPW round, Uma Sharma did some investigation, together with me and George, on how we could introduce hyperthreading support in Credit2. It turned out that arranging runqueues to be 'per-core' (rather than per-socket) was the most obvious first step. She performed some experiments, and what came out was that aggregating pCPUs per-core is actually providing better performance, even better than Credit1, in some cases. There still are scenarios where more investigation and refinements are necessary, but the per-core approach looks more promising.

Some of the measurements she performed are available here:

http://lists.xen.org/archives/html/xen-devel/2015-03/msg01499.html

I made some more myself, for my presentation at the last XenDevSummit, which is available here:

http://events.linuxfoundation.org/sites/events/files/slides/Faggioli_XenSummit.pdf

Since there was some preparatory work to be done (i.e., what's in the first half of this series), Uma couldn't really upstream her patches at the time, and we agreed that I would have followed up, and re-submit her patches after the preliminary adjustments were done. Therefore, this series encompasses Uma's work, sometimes in the exact same form in which she originally submitted it, sometimes tweaked to fit on top of the updated Credit2 codebase.

So, in summary, this series does the following:

1) first of all, it splits allocation and initialization phases of scheduling related per-pCPU data. Thanks to this, all allocations can be performed during the CPU_UP_PREPARE phase of pCPU bringup, while proper initialization can happen later, during CPU_STARTING. We really want --especially for Credit2-- to do the initialization in CPU_STARTING, as before that, we won't have the necessary topology information available.

However, we can't move the alloc_pdata hook all together (and keep doing both allocation and initialization in there), as we can't really allocate memory while in CPU_STARTING (due to IRQs being disabled). Therefore, the split looks like the best solution, and it also seems to me to make a lot of sense to me, architecturally. In fact, right now, we have schedulers that does not need to allocate anything, but have to use alloc_pdata for initialization! :-/

This was discussed already in some detail, a few time ago, in this subthread:

http://lists.xen.org/archives/html/xen-devel/2015-03/msg02309.html

2) At this point, we can easily fixup Credit2 runqueue creation phase, as we now have all the necessary topology information available when we need them. We therefore do create one runqueue per socket, and this time, it works. :-)

While there, we make it possible for the user to decide how pCPUs should be aggregated in runqueues, with a Xen boot time parameter (socket and core is what is supported for now).

3) Finally, as the per-core runqueue approach looks promising, we set per-core runqueue to be the default.

The series is structured as follows:
* patch 1 and 2 are: one Credit2 debug dump output fix (patch 1), and some Credit2 boot parameter variables scope and placement improvements (patch 2);
* patch 3 fixes a locking inconsistency in the scheduling interface;
* patch 4, 5 and 6, arrange for the introduction of the new initialization hook (init_pcpu). Whe implementing it, for the schedulers that need that, code is borrowed from the allocation hook (i.e., step 1 above);
* patch 7 and 8 fix Credit2 runqueue creation, and introduce a boot time parameter for influencing the runqueue aggregation strategy (i.e., step 2 above);
* patch 9 makes per-core the default runqueue aggregation strategy. This is done in a separate patch to avoid mixing technical and (default) policy changes.

This series is available as git branch, on top of staging, here:

git://xenbits.xen.org/people/dariof/xen.git rel/sched/credit2/fix-runqueues
http://xenbits.xen.org/gitweb/?p=people/dariof/xen.git;a=shortlog;h=refs/heads/rel/sched/credit2/fix-runqueues

There is also a branch containing this series, rebased on top of the following (already sent) three patches:

xen: sched: get rid of cpupool_scheduler_cpumask()
xen: sched: adjustments to some performance counters
xen: credit1: fix tickling when it happens from a remote pCPU

just in case those go in before this series (which I'd call likely), and one wants to pull the series on top of that:

git://xenbits.xen.org/people/dariof/xen.git rel/sched/credit2/fix-runqueues_rebase
http://xenbits.xen.org/gitweb/?p=people/dariof/xen.git;a=shortlog;h=refs/heads/rel/sched/credit2/fix-runqueues_rebase

Regards,
Dario

---
Dario Faggioli (8):
xen: sched: fix an 'off by one \t' in credit2 debug dump
xen: sched: make locking for {insert,remove}_vcpu consistent
xen: sched: add .init_pdata hook to the scheduler interface
xen: sched: make implementing .alloc_pdata optional
xen: sched: implement .init_pdata in all schedulers
xen: sched: fix per-socket runqueue creation in credit2
xen: sched: allow for choosing credit2 runqueues configuration at boot
xen: sched: per-core runqueues as default in credit2

Uma Sharma (1):
xen: sched: improve scope and placement of credit2 boot parameters

 docs/misc/xen-command-line.markdown |   11 ++
 xen/common/sched_arinc653.c         |   31 ------
 xen/common/sched_credit.c           |   47 +++++++--
 xen/common/sched_credit2.c          |  186 +++++++++++++++++++----------------
 xen/common/sched_rt.c               |   19 +++-
 xen/common/schedule.c               |   28 +++--
 xen/include/xen/sched-if.h          |    1 
 7 files changed, 182 insertions(+), 141 deletions(-)

--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)