Skip to content

RFC xen: add core scheduling support

Juergen Gross xen-devel 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 RFC Latest45 patches 35 repliesIn review
  2. v1 RFC49 patches 57 repliesSuperseded

Changes in v2 from the cover letter

- ARM is building now
- HVM domains are working now
- idling will always be done with idle_vcpu active
- other small changes see individual patches

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/45xen/sched: add inline wrappers for calling per-scheduler functionsxen/common/schedule.c xen/include/xen/sched-if.h+226 -5612 replies
2/45xen/sched: use new sched_item instead of vcpu in scheduler interfacesxen/common/sched_arinc653.c xen/common/sched_credit.c xen/common/sched_credit2.c +5 more+218 -1315 replies
3/45xen/sched: alloc struct sched_item for each vcpuxen/common/schedule.c xen/include/xen/sched.h+33 -360 replies
4/45xen/sched: move per-vcpu scheduler private data pointer to sched_itemxen/common/sched_arinc653.c xen/common/sched_credit.c xen/common/sched_credit2.c +4 more+27 -270 replies
5/45xen/sched: build a linked list of struct sched_itemxen/common/schedule.c xen/include/xen/sched.h+58 -70 replies
6/45xen/sched: introduce struct sched_resourcexen/common/sched_credit.c xen/common/sched_credit2.c xen/common/sched_null.c +4 more+40 -00 replies
7/45xen/sched: let pick_cpu return a scheduler resourcexen/common/sched_arinc653.c xen/common/sched_credit.c xen/common/sched_credit2.c +5 more+57 -530 replies
8/45xen/sched: switch schedule_data.curr to point at sched_itemxen/common/sched_arinc653.c xen/common/sched_credit.c xen/common/sched_credit2.c +4 more+220 -2230 replies
9/45xen/sched: move per cpu scheduler private data into struct sched_resourcexen/common/sched_arinc653.c xen/common/sched_credit.c xen/common/sched_credit2.c +5 more+68 -740 replies
10/45xen/sched: switch vcpu_schedule_lock to item_schedule_lockxen/common/sched_credit.c xen/common/sched_credit2.c xen/common/sched_null.c +3 more+73 -720 replies
11/45xen/sched: move some per-vcpu items to struct sched_itemxen/arch/x86/pv/emul-priv-op.c xen/arch/x86/pv/traps.c xen/arch/x86/traps.c +11 more+163 -1420 replies
12/45xen/sched: add scheduler helpers hiding vcpuxen/common/sched_credit.c xen/common/schedule.c xen/include/xen/sched-if.h+62 -160 replies
13/45xen/sched: add domain pointer to struct sched_itemxen/common/schedule.c xen/include/xen/sched.h+3 -10 replies
14/45xen/sched: add id to struct sched_itemxen/common/schedule.c xen/include/xen/sched.h+3 -10 replies
15/45xen/sched: rename scheduler related perf countersxen/common/sched_credit.c xen/common/sched_credit2.c xen/common/sched_null.c +2 more+57 -570 replies
16/45xen/sched: switch struct task_slice from vcpu to sched_itemxen/common/sched_arinc653.c xen/common/sched_credit.c xen/common/sched_credit2.c +4 more+19 -190 replies
17/45xen/sched: add is_running indicator to struct sched_itemxen/common/sched_credit.c xen/common/sched_credit2.c xen/common/sched_rt.c +2 more+32 -190 replies
18/45xen/sched: make null scheduler vcpu agnostic.xen/common/sched_null.c+149 -1550 replies
19/45xen/sched: make rt scheduler vcpu agnostic.xen/common/sched_rt.c+174 -1820 replies
20/45xen/sched: make credit scheduler vcpu agnostic.xen/common/sched_credit.c+251 -2530 replies
21/45xen/sched: make credit2 scheduler vcpu agnostic.xen/common/sched_credit2.c+403 -4170 replies
22/45xen/sched: make arinc653 scheduler vcpu agnostic.xen/common/sched_arinc653.c+101 -1070 replies
23/45xen: add sched_item_pause_nosync() and sched_item_unpause()xen/common/sched_credit.c xen/include/xen/sched-if.h+13 -30 replies
24/45xen: let vcpu_create() select processorxen/arch/arm/domain_build.c xen/arch/x86/dom0_build.c xen/arch/x86/hvm/dom0_build.c +7 more+51 -483 replies
25/45xen/sched: use sched_resource cpu instead smp_processor_id in schedulersxen/common/sched_arinc653.c xen/common/sched_credit.c xen/common/sched_credit2.c +3 more+61 -520 replies
26/45xen/sched: switch schedule() from vcpus to sched_itemsxen/common/schedule.c+40 -300 replies
27/45xen/sched: switch sched_move_irqs() to take sched_item as parameterxen/common/schedule.c+13 -50 replies
28/45xen: switch from for_each_vcpu() to for_each_sched_item()xen/common/domain.c xen/common/schedule.c+59 -570 replies
29/45xen/sched: add runstate counters to struct sched_itemxen/common/schedule.c xen/include/xen/sched.h+8 -00 replies
30/45xen/sched: rework and rename vcpu_force_reschedule()xen/arch/x86/pv/shim.c xen/common/domain.c xen/common/schedule.c +1 more+17 -182 replies
31/45xen/sched: Change vcpu_migrate_*() to operate on schedule itemxen/common/schedule.c+59 -430 replies
32/45xen/sched: move struct task_slice into struct sched_itemxen/common/sched_arinc653.c xen/common/sched_credit.c xen/common/sched_credit2.c +5 more+69 -830 replies
33/45xen/sched: add code to sync scheduling of all vcpus of a sched itemxen/arch/arm/domain.c xen/arch/x86/domain.c xen/common/schedule.c +4 more+269 -990 replies
34/45xen/sched: introduce item_runnable_state()xen/common/domain.c xen/common/sched_arinc653.c xen/common/sched_credit.c +6 more+57 -300 replies
35/45xen/sched: add support for multiple vcpus per sched item where missingxen/common/schedule.c xen/include/xen/sched-if.h+73 -270 replies
36/45x86: make loading of GDT at context switch more modularxen/arch/x86/domain.c+35 -222 replies
37/45x86: optimize loading of GDT at context switchxen/arch/x86/domain.c+13 -52 replies
38/45xen/sched: modify cpupool_domain_cpumask() to be an item maskxen/common/cpupool.c xen/common/schedule.c xen/include/xen/sched-if.h+28 -120 replies
39/45xen/sched: support allocating multiple vcpus into one sched itemxen/common/schedule.c+62 -130 replies
40/45xen/sched: add a scheduler_percpu_init() functionxen/arch/arm/smpboot.c xen/arch/x86/smpboot.c xen/common/schedule.c +1 more+17 -70 replies
41/45xen/sched: add a percpu resource indexxen/common/schedule.c+9 -20 replies
42/45xen/sched: add fall back to idle vcpu when scheduling itemxen/arch/x86/domain.c xen/common/schedule.c xen/include/asm-arm/current.h +3 more+134 -553 replies
43/45xen/sched: make vcpu_wake() and vcpu_sleep() core scheduling awarexen/common/schedule.c xen/include/xen/sched-if.h xen/include/xen/sched.h+136 -190 replies
44/45xen/sched: carve out freeing sched_item memory into dedicated functionxen/common/schedule.c+20 -160 replies
45/45xen/sched: add scheduling granularity enumxen/arch/x86/setup.c xen/common/schedule.c xen/include/xen/sched-if.h +1 more+153 -95 replies

Reviews trailers per patch

3 of 45 patches have a Reviewed-by or Acked-by.

PatchAndrew CooperJan Beulich
1/45
2/45
3/45
4/45
5/45
6/45
7/45
8/45
9/45
10/45
11/45
12/45
13/45
14/45
15/45
16/45
17/45
18/45
19/45
20/45
21/45
22/45
23/45
24/45A-b
25/45
26/45
27/45
28/45
29/45
30/45
31/45
32/45
33/45
34/45
35/45
36/45A-b
37/45R-b
38/45
39/45
40/45
41/45
42/45
43/45
44/45
45/45

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v2 thread

Cover letter [PATCH RFC V2 00/45] xen: add core scheduling support

Add support for core- and socket-scheduling in the Xen hypervisor.

Via boot parameter sched-gran=core (or sched-gran=socket) it is possible to change the scheduling granularity from cpu (the default) to either whole cores or even sockets.

All logical cpus (threads) of the core or socket are always scheduled together. This means that on a core always vcpus of the same domain will be active, and those vcpus will always be scheduled at the same time.

This is achieved by switching the scheduler to no longer see vcpus as the primary object to schedule, but "schedule items". Each schedule item consists of as many vcpus as each core has threads on the current system. The vcpu->item relation is fixed.

I have done some very basic performance testing: on a 4 cpu system (2 cores with 2 threads each) I did a "make -j 4" for building the Xen hypervisor. With This test has been run on dom0, once with no other guest active and once with another guest with 4 vcpus running the same test. The results are (always elapsed time, system time, user time):

sched-gran=cpu,    no other guest: 116.10 177.65 207.84
sched-gran=core,   no other guest: 114.04 175.47 207.45
sched-gran=cpu,    other guest:    202.30 334.21 384.63
sched-gran=core,   other guest:    207.24 293.04 371.37

All tests have been performed with credit2, the other schedulers are untested up to now.

Cpupools are not yet working, as moving cpus between cpupools needs more work (this is the reason for the series still being RFC). Same applies to cpu add/remove.

Changes in RFC V2:
- ARM is building now
- HVM domains are working now
- idling will always be done with idle_vcpu active
- other small changes see individual patches

Juergen Gross (45):
xen/sched: add inline wrappers for calling per-scheduler functions xen/sched: use new sched_item instead of vcpu in scheduler interfaces xen/sched: alloc struct sched_item for each vcpu xen/sched: move per-vcpu scheduler private data pointer to sched_item xen/sched: build a linked list of struct sched_item xen/sched: introduce struct sched_resource
xen/sched: let pick_cpu return a scheduler resource xen/sched: switch schedule_data.curr to point at sched_item xen/sched: move per cpu scheduler private data into struct
sched_resource
xen/sched: switch vcpu_schedule_lock to item_schedule_lock xen/sched: move some per-vcpu items to struct sched_item xen/sched: add scheduler helpers hiding vcpu
xen/sched: add domain pointer to struct sched_item xen/sched: add id to struct sched_item
xen/sched: rename scheduler related perf counters xen/sched: switch struct task_slice from vcpu to sched_item xen/sched: add is_running indicator to struct sched_item xen/sched: make null scheduler vcpu agnostic.
xen/sched: make rt scheduler vcpu agnostic.
xen/sched: make credit scheduler vcpu agnostic.
xen/sched: make credit2 scheduler vcpu agnostic. xen/sched: make arinc653 scheduler vcpu agnostic.
xen: add sched_item_pause_nosync() and sched_item_unpause()
xen: let vcpu_create() select processor
xen/sched: use sched_resource cpu instead smp_processor_id in
schedulers
xen/sched: switch schedule() from vcpus to sched_items xen/sched: switch sched_move_irqs() to take sched_item as parameter
xen: switch from for_each_vcpu() to for_each_sched_item() xen/sched: add runstate counters to struct sched_item xen/sched: rework and rename vcpu_force_reschedule() xen/sched: Change vcpu_migrate_*() to operate on schedule item xen/sched: move struct task_slice into struct sched_item xen/sched: add code to sync scheduling of all vcpus of a sched item xen/sched: introduce item_runnable_state()
xen/sched: add support for multiple vcpus per sched item where missing
x86: make loading of GDT at context switch more modular
x86: optimize loading of GDT at context switch
xen/sched: modify cpupool_domain_cpumask() to be an item mask xen/sched: support allocating multiple vcpus into one sched item xen/sched: add a scheduler_percpu_init() function xen/sched: add a percpu resource index
xen/sched: add fall back to idle vcpu when scheduling item xen/sched: make vcpu_wake() and vcpu_sleep() core scheduling aware xen/sched: carve out freeing sched_item memory into dedicated function xen/sched: add scheduling granularity enum

 xen/arch/arm/domain.c            |    2 +-
 xen/arch/arm/domain_build.c      |   13 +-
 xen/arch/arm/smpboot.c           |    2 +
 xen/arch/x86/dom0_build.c        |   10 +-
 xen/arch/x86/domain.c            |   95 ++-
 xen/arch/x86/hvm/dom0_build.c    |    9 +-
 xen/arch/x86/pv/dom0_build.c     |   10 +-
 xen/arch/x86/pv/emul-priv-op.c   |    1 +
 xen/arch/x86/pv/shim.c           |    4 +-
 xen/arch/x86/pv/traps.c          |    5 +-
 xen/arch/x86/setup.c             |    2 +
 xen/arch/x86/smpboot.c           |    2 +
 xen/arch/x86/traps.c             |    9 +-
 xen/common/cpupool.c             |   30 +-
 xen/common/domain.c              |   34 +-
 xen/common/domctl.c              |   23 +-
 xen/common/keyhandler.c          |    4 +-
 xen/common/sched_arinc653.c      |  258 ++++----
 xen/common/sched_credit.c        |  743 ++++++++++-----------
 xen/common/sched_credit2.c       | 1119 +++++++++++++++----------------
 xen/common/sched_null.c          |  424 ++++++------
 xen/common/sched_rt.c            |  544 +++++++--------
 xen/common/schedule.c            | 1348 +++++++++++++++++++++++++++++---------
 xen/common/softirq.c             |    6 +-
 xen/common/wait.c                |    4 +-
 xen/include/asm-arm/current.h    |    1 +
 xen/include/asm-x86/cpuidle.h    |    2 +-
 xen/include/asm-x86/current.h    |    7 +-
 xen/include/asm-x86/dom0_build.h |    3 +-
 xen/include/asm-x86/smp.h        |    3 +
 xen/include/xen/domain.h         |    3 +-
 xen/include/xen/perfc_defn.h     |   32 +-
 xen/include/xen/sched-if.h       |  418 ++++++++++--
 xen/include/xen/sched.h          |   95 ++-
 xen/include/xen/softirq.h        |    1 +
 35 files changed, 3198 insertions(+), 2068 deletions(-)
Signature · 7 lines
2.16.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel