0/2 VCPU creation and allocation
PATCH 1/1 0/2 VCPU creation and allocation
I've put together two patches. The first introduces a new dom0_op, set_max_vcpus, which with an associated variable and a check in the VCPUOP handler fixes [1]bug 288. Also included is a new VCPUOP, VCPUOP_create, which handles all of the vcpu creation tasks and leaves initialization and unpausing to VCPUOP_initialize. The separation allows for build-time allocation of vcpus which becomes more important when trying to allocate memory in a NUMA-aware manner (i.e. knowing which physical cpus are being used in a domain before we allocate the memory).
The second patch adds a new domain allocmap cpumap_t which is a bitmap of physical cpus from which vcpus are to be allocated. As vcpus are created, the selection of which physical cpu is balanced across the set of physical cpus within the map. The patch lets us control vcpu allocation at a high level (start all VCPUS on hyperthreads, NODE1, CPU2).
1. http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=288
Signature · 5 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 [email protected]
Quoted 3 lines · Ryan Harper
Also included is a new VCPUOP,
VCPUOP_create, which handles all of the vcpu creation tasks and leaves initialization and unpausing toVCPUOP_initialize.
I think two new Xen operations is one too many.
Either we should take set_max_vcpus, and have that implicitly do the work of VCPUOP_create, or we should take VCPUOP_create (callable only by domain0) and have that implicitly increase max_vcpus for the subject domain.
-- Keir
Quoted 7 lines · Keir Fraser
Quoted 3 lines · Ryan Harper
Also included is a new VCPUOP,
VCPUOP_create, which handles all of the vcpu creation tasks and leaves initialization and unpausing toVCPUOP_initialize.
Quoted 6 lines
I think two new Xen operations is one too many.
Either we should take
set_max_vcpus, and have that implicitly do the work ofVCPUOP_create, or we should takeVCPUOP_create(callable only by domain0) and have that implicitly increasemax_vcpusfor the subject domain.
OK. I'll re-work VCPUOP_create to be callable only by dom0 and auto-increment max_vcpus.
Signature · 5 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 [email protected]
Quoted 7 lines · Ryan Harper
Quoted 4 lines
Either we should take
set_max_vcpus, and have that implicitly do the work ofVCPUOP_create, or we should takeVCPUOP_create(callable only by domain0) and have that implicitly increasemax_vcpusfor the subject domain.OK. I'll re-work
VCPUOP_createto be callable only by dom0 and auto-incrementmax_vcpus.
Do you even need a max_vcpus variable? Surely the appropriate check is implicit in VCPUOP_initialise detecting whether or not the relevant VCPU has been created?
I like small code and simple interfaces. :-)
-- Keir
Quoted 9 lines · Keir Fraser
Quoted 6 lines · Ryan Harper
Quoted 5 lines
Either we should take
set_max_vcpus, and have that implicitly do the work ofVCPUOP_create, or we should takeVCPUOP_create(callable only by domain0) and have that implicitly increasemax_vcpusfor the subject
domain.
Quoted 6 lines
Quoted 2 lines
OK. I'll re-work
VCPUOP_createto be callable only by dom0 and auto-incrementmax_vcpus.Do you even need a
max_vcpusvariable? Surely the appropriate check is implicit inVCPUOP_initialisedetecting whether or not the relevant VCPU has been created?
I was going to ensure ordered VCPU creation. Without something like vcpuid < max_vcpus+1, and increment on successful creation, one can create vcpus in any order, 1,5,7, 10. I don't think it *should* matter but I've not looked elsewhere through the code to see if there are any other areas assuming all struct vcpu* being valid between 0 and n in the d->vcpus[] array.
Signature · 5 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 [email protected]
Quoted 10 lines · Ryan Harper
Quoted 3 lines
Do you even need a
max_vcpusvariable? Surely the appropriate check is implicit inVCPUOP_initialisedetecting whether or not the relevant VCPU has been created?I was going to ensure ordered VCPU creation. Without something like vcpuid < max_vcpus+1, and increment on successful creation, one can create vcpus in any order, 1,5,7, 10. I don't think it *should* matter but I've not looked elsewhere through the code to see if there are any other areas assuming all struct vcpu* being valid between 0 and n in the d->vcpus[] array.
Then the vcpu parameter to VCPUOP_create is redundant -- there's only one value you will be prepared to accept! If we don't want the flexibility of a sparse vcpu map (and I think we don't) then perhaps we are better off without VCPUOP_create (which is maybe even a bit neater, leaving vcpu_op as a completely unpriv local hypercall) and stick with the set_max_vcpus dom0_op? And have that implicitly create the vcpu struct for vcpus 0...n-1?
-- Keir
Quoted 14 lines · Keir Fraser
Quoted 10 lines · Ryan Harper
Quoted 3 lines
Do you even need a
max_vcpusvariable? Surely the appropriate check is implicit inVCPUOP_initialisedetecting whether or not the relevant VCPU has been created?I was going to ensure ordered VCPU creation. Without something like vcpuid < max_vcpus+1, and increment on successful creation, one can create vcpus in any order, 1,5,7, 10. I don't think it *should* matter but I've not looked elsewhere through the code to see if there are any other areas assuming all struct vcpu* being valid between 0 and n in the d->vcpus[] array.
Quoted 7 lines
Then the vcpu parameter to
VCPUOP_createis redundant -- there's only one value you will be prepared to accept! If we don't want the flexibility of a sparse vcpu map (and I think we don't) then perhaps we are better off withoutVCPUOP_create(which is maybe even a bit neater, leavingvcpu_opas a completely unpriv local hypercall) and stick with theset_max_vcpusdom0_op? And have that implicitly create the vcpu struct for vcpus 0...n-1?
OK, that makes sense. I'll turn VCPUOP_create into set_max_vcpus(max) which will create vcpus 1-(max-1). Any preference on the hypercall name? Does set_max_vcpus() still make sense if it is also creating vcpus?
How about DOM0_CREATEVCPUS and
do_createvcpus(struct domain* d, unsigned int max_vcpus).
Signature · 5 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 [email protected]
Quoted 7 lines · Ryan Harper
OK, that makes sense. I'll turn
VCPUOP_createintoset_max_vcpus(max)which will create vcpus 1-(max-1). Any preference on the hypercall name? Doesset_max_vcpus()still make sense if it is also creating vcpus?How about
DOM0_CREATEVCPUSand
do_createvcpus(struct domain* d, unsigned int max_vcpus).
I'm not fussed. I guess I have a slight preference for the latter (createvcpus).
-- Keir
PATCH 1/1 2nd try: 0/2 VCPU creation and allocation
Quoted 8 lines · Keir Fraser
Quoted 5 lines · Ryan Harper
OK, that makes sense. I'll turn
VCPUOP_createintoset_max_vcpus(max)which will create vcpus 1-(max-1). Any preference on the hypercall name? Doesset_max_vcpus()still make sense if it is also creating vcpus?
Quoted 5 lines
Quoted 2 lines
How about
DOM0_CREATEVCPUSand
do_createvcpus(struct domain* d, unsigned int max_vcpus).I'm not fussed. I guess I have a slight preference for the latter (createvcpus).
The first patch creates a new dom0_op, DOM0_VCPUSINCREASE, which will create VCPU 1 to (limit-1) in the target domain. The max_vcpus value is gone as is the VCPUOP_create.
The second patch is the same as before.
Signature · 5 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 [email protected]
PATCH 1/1 2nd try: 1/2 VCPU creation and allocation
In reply to Keir FraserNew dom0_op DOM0_VCPUSINCREASE.
Signature · 278 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 [email protected] diffstat output: tools/libxc/xc_domain.c | 9 +++++++++ tools/libxc/xenctrl.h | 11 +++++++++++ tools/python/xen/lowlevel/xc/xc.c | 28 ++++++++++++++++++++++++++++ tools/python/xen/xend/XendDomainInfo.py | 4 ++++ xen/arch/x86/domain_build.c | 6 ++++++ xen/common/dom0_ops.c | 27 +++++++++++++++++++++++++++ xen/common/domain.c | 25 ++++++++++--------------- xen/include/public/dom0_ops.h | 8 ++++++++ xen/include/xen/domain.h | 2 ++ 9 files changed, 105 insertions(+), 15 deletions(-) Signed-off-by: Ryan Harper <[email protected]> --- diff -r 015f8ae81276 tools/libxc/xc_domain.c --- a/tools/libxc/xc_domain.c Mon Oct 10 14:38:01 2005 +++ b/tools/libxc/xc_domain.c Mon Oct 10 17:26:54 2005 @@ -329,6 +329,15 @@ return err; } +int xc_domain_vcpus_increase(int xc_handle, u32 domid, unsigned int limit) +{ + dom0_op_t op; + op.cmd = DOM0_VCPUSINCREASE; + op.u.vcpusincrease.domain = (domid_t)domid; + op.u.vcpusincrease.limit = (u8)limit; + return do_dom0_op(xc_handle, &op); +} + /* * Local variables: * mode: C diff -r 015f8ae81276 tools/libxc/xenctrl.h --- a/tools/libxc/xenctrl.h Mon Oct 10 14:38:01 2005 +++ b/tools/libxc/xenctrl.h Mon Oct 10 17:26:54 2005 @@ -152,6 +152,17 @@ u32 ssidref, u32 *pdomid); +/* + * This function increase the number vcpus in a domain to limit. + * + * @parm xc_handle a handle to an open hypervisor interface. + * @parm domid the domain id in which vcpus are to be created. + * @parm limit the total number of vcpus to run in domain. + * @return 0 on success, -1 on failure. + */ +int xc_domain_vcpus_increase(int xc_handle, + u32 domid, + unsigned int limit); int xc_domain_dumpcore(int xc_handle, u32 domid, diff -r 015f8ae81276 tools/python/xen/lowlevel/xc/xc.c --- a/tools/python/xen/lowlevel/xc/xc.c Mon Oct 10 14:38:01 2005 +++ b/tools/python/xen/lowlevel/xc/xc.c Mon Oct 10 17:26:54 2005 @@ -91,6 +91,26 @@ return PyErr_SetFromErrno(xc_error); return PyInt_FromLong(dom); +} + +static PyObject *pyxc_domain_vcpus_increase(PyObject *self, + PyObject *args, + PyObject *kwds) +{ + XcObject *xc = (XcObject *)self; + + u32 dom, limit; + + static char *kwd_list[] = { "dom", "limit", NULL }; + + if ( !PyArg_ParseTupleAndKeywords(args, kwds, "ii", kwd_list, &dom, &limit) ) + return NULL; + + if ( xc_domain_vcpus_increase(xc->xc_handle, dom, limit) != 0 ) + return PyErr_SetFromErrno(xc_error); + + Py_INCREF(zero); + return zero; } static PyObject *pyxc_domain_pause(PyObject *self, @@ -783,6 +803,14 @@ " dom [int, 0]: Domain identifier to use (allocated if zero).\n" "Returns: [int] new domain identifier; -1 on error.\n" }, + { "domain_vcpus_increase", + (PyCFunction)pyxc_domain_vcpus_increase, + METH_VARARGS | METH_KEYWORDS, "\n" + "Increase the number of VCPUs in domain.\n" + " dom [int, 0]: Domain identifier to use.\n" + " limit [int, 0]: New total number of VCPUs in domain.\n" + "Returns: [int] 0 on success; -1 on error.\n" }, + { "domain_dumpcore", (PyCFunction)pyxc_domain_dumpcore, METH_VARARGS | METH_KEYWORDS, "\n" diff -r 015f8ae81276 tools/python/xen/xend/XendDomainInfo.py --- a/tools/python/xen/xend/XendDomainInfo.py Mon Oct 10 14:38:01 2005 +++ b/tools/python/xen/xend/XendDomainInfo.py Mon Oct 10 17:26:54 2005 @@ -1018,6 +1018,10 @@ self.image.handleBootloading() xc.domain_setcpuweight(self.domid, self.info['cpu_weight']) + + # increase the total number of vcpus in domain + xc.domain_vcpus_increase(self.domid, int(self.info['vcpus'])); + # XXX Merge with configure_maxmem? m = self.image.getDomainMemory(self.info['memory_KiB']) xc.domain_setmaxmem(self.domid, m) diff -r 015f8ae81276 xen/arch/x86/domain_build.c --- a/xen/arch/x86/domain_build.c Mon Oct 10 14:38:01 2005 +++ b/xen/arch/x86/domain_build.c Mon Oct 10 17:26:54 2005 @@ -14,6 +14,7 @@ #include <xen/event.h> #include <xen/elf.h> #include <xen/kernel.h> +#include <xen/domain.h> #include <asm/regs.h> #include <asm/system.h> #include <asm/io.h> @@ -559,6 +560,11 @@ d->shared_info->vcpu_data[i].evtchn_upcall_mask = 1; d->shared_info->n_vcpu = num_online_cpus(); + /* create extra vcpus */ + if ( (d->shared_info->n_vcpu > 1) ) + for ( i = 1; i < d->shared_info->n_vcpu; i++ ) + do_createvcpu(d, i); + /* Set up monitor table */ update_pagetables(v); diff -r 015f8ae81276 xen/common/dom0_ops.c --- a/xen/common/dom0_ops.c Mon Oct 10 14:38:01 2005 +++ b/xen/common/dom0_ops.c Mon Oct 10 17:26:54 2005 @@ -227,6 +227,33 @@ } break; + case DOM0_VCPUSINCREASE: + { + struct domain *d; + unsigned int i, limit = op->u.vcpusincrease.limit; + + if ( limit >= MAX_VIRT_CPUS ) + ret = -EINVAL; + + d = find_domain_by_id(op->u.vcpusincrease.domain); + + ret = -ESRCH; + if ( d != NULL ) + { + ret = -EINVAL; + LOCK_BIGLOCK(d); + /* NB: VCPU0 is allocated on domain creation */ + for ( i=1; i<limit; i++ ) { + if ( d->vcpu[i] == NULL ) + if ( (ret = do_createvcpu(d, i)) != 0 ) + break; + } + UNLOCK_BIGLOCK(d); + put_domain(d); + } + } + break; + case DOM0_DESTROYDOMAIN: { struct domain *d = find_domain_by_id(op->u.destroydomain.domain); diff -r 015f8ae81276 xen/common/domain.c --- a/xen/common/domain.c Mon Oct 10 14:38:01 2005 +++ b/xen/common/domain.c Mon Oct 10 17:26:54 2005 @@ -368,10 +368,9 @@ return rc; } -int boot_vcpu(struct domain *d, int vcpuid, struct vcpu_guest_context *ctxt) -{ - struct vcpu *v; - int rc; +int do_createvcpu(struct domain *d, int vcpuid) +{ + struct vcpu *v; ASSERT(d->vcpu[vcpuid] == NULL); @@ -387,20 +386,11 @@ arch_do_boot_vcpu(v); - if ( (rc = arch_set_info_guest(v, ctxt)) != 0 ) - goto out; - sched_add_domain(v); set_bit(_VCPUF_down, &v->vcpu_flags); - clear_bit(_VCPUF_ctrl_pause, &v->vcpu_flags); return 0; - - out: - arch_free_vcpu_struct(d->vcpu[vcpuid]); - d->vcpu[vcpuid] = NULL; - return rc; } long do_vcpu_op(int cmd, int vcpuid, void *arg) @@ -413,7 +403,7 @@ if ( (vcpuid < 0) || (vcpuid >= MAX_VIRT_CPUS) ) return -EINVAL; - if ( ((v = d->vcpu[vcpuid]) == NULL) && (cmd != VCPUOP_initialise) ) + if ( ((v = d->vcpu[vcpuid]) == NULL) ) return -ENOENT; switch ( cmd ) @@ -433,7 +423,12 @@ } LOCK_BIGLOCK(d); - rc = (d->vcpu[vcpuid] == NULL) ? boot_vcpu(d, vcpuid, ctxt) : -EEXIST; + clear_bit(_VCPUF_ctrl_pause, &v->vcpu_flags); + if ( (rc = arch_set_info_guest(v, ctxt)) != 0 ) { + sched_rem_domain(v); + arch_free_vcpu_struct(d->vcpu[vcpuid]); + d->vcpu[vcpuid] = NULL; + } UNLOCK_BIGLOCK(d); xfree(ctxt); diff -r 015f8ae81276 xen/include/public/dom0_ops.h --- a/xen/include/public/dom0_ops.h Mon Oct 10 14:38:01 2005 +++ b/xen/include/public/dom0_ops.h Mon Oct 10 17:26:54 2005 @@ -386,6 +386,13 @@ int is_ram; } *memory_map; } dom0_physical_memory_map_t; + +#define DOM0_VCPUSINCREASE 41 +typedef struct { + domid_t domain; /* domain to be affected */ + u8 limit; /* total number of vcpus to run */ +} dom0_vcpusincrease_t; + typedef struct { u32 cmd; @@ -422,6 +429,7 @@ dom0_getdomaininfolist_t getdomaininfolist; dom0_platform_quirk_t platform_quirk; dom0_physical_memory_map_t physical_memory_map; + dom0_vcpusincrease_t vcpusincrease; } u; } dom0_op_t; diff -r 015f8ae81276 xen/include/xen/domain.h --- a/xen/include/xen/domain.h Mon Oct 10 14:38:01 2005 +++ b/xen/include/xen/domain.h Mon Oct 10 17:26:54 2005 @@ -1,6 +1,8 @@ #ifndef __XEN_DOMAIN_H__ #define __XEN_DOMAIN_H__ + +extern int do_createvcpu(struct domain *d, int vcpuid); /* * Arch-specifics.
PATCH 1/1 2nd try: 2/2 VCPU creation and allocation
In reply to Keir FraserAllocation map.
Signature · 353 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 [email protected] diffstat output: tools/examples/xmexample1 | 4 + tools/examples/xmexample2 | 3 + tools/python/xen/lowlevel/xc/xc.c | 2 tools/python/xen/xend/XendDomainInfo.py | 16 ++++- tools/python/xen/xm/create.py | 6 ++ xen/common/dom0_ops.c | 90 +++++++++++++++++--------------- xen/common/domain.c | 28 +++++++++ xen/common/schedule.c | 11 ++- xen/include/public/dom0_ops.h | 2 xen/include/xen/domain.h | 2 xen/include/xen/sched.h | 2 11 files changed, 114 insertions(+), 52 deletions(-) Signed-off-by: Ryan Harper <[email protected]> --- diff -r 12d1e93653c2 tools/examples/xmexample1 --- a/tools/examples/xmexample1 Tue Oct 11 00:13:12 2005 +++ b/tools/examples/xmexample1 Mon Oct 10 19:35:55 2005 @@ -27,6 +27,10 @@ # Number of Virtual CPUS to use, default is 1 #vcpus = 1 + +# A bitmap of which physical cpus are vcpus allowed to use. +# ex1: 0x2 <-- bit 1 set means all vcpus will be created on CPU1 +#allocmap = 0xffffffff # default value, vcpus can run on any cpu. #---------------------------------------------------------------------------- # Define network interfaces. diff -r 12d1e93653c2 tools/examples/xmexample2 --- a/tools/examples/xmexample2 Tue Oct 11 00:13:12 2005 +++ b/tools/examples/xmexample2 Mon Oct 10 19:35:55 2005 @@ -58,6 +58,9 @@ # Number of Virtual CPUS to use, default is 1 #vcpus = 1 vcpus = 4 # make your domain a 4-way + +# A bitmap of which physical cpus are vcpus allowed to use. +allocmap = 0x2 # start all of your VCPUs on CPU1 #---------------------------------------------------------------------------- # Define network interfaces. diff -r 12d1e93653c2 tools/python/xen/lowlevel/xc/xc.c --- a/tools/python/xen/lowlevel/xc/xc.c Tue Oct 11 00:13:12 2005 +++ b/tools/python/xen/lowlevel/xc/xc.c Mon Oct 10 19:35:55 2005 @@ -185,7 +185,7 @@ static char *kwd_list[] = { "dom", "vcpu", "cpumap", NULL }; - if ( !PyArg_ParseTupleAndKeywords(args, kwds, "i|ii", kwd_list, + if ( !PyArg_ParseTupleAndKeywords(args, kwds, "i|iL", kwd_list, &dom, &vcpu, &cpumap) ) return NULL; diff -r 12d1e93653c2 tools/python/xen/xend/XendDomainInfo.py --- a/tools/python/xen/xend/XendDomainInfo.py Tue Oct 11 00:13:12 2005 +++ b/tools/python/xen/xend/XendDomainInfo.py Mon Oct 10 19:35:55 2005 @@ -266,6 +266,7 @@ result['maxmem'] = get_cfg('maxmem', int) result['maxmem_kb'] = get_cfg('maxmem_kb', int) result['cpu'] = get_cfg('cpu', int) + result['allocmap'] = get_cfg('allocmap', int) result['image'] = get_cfg('image') try: @@ -438,6 +439,7 @@ defaultInfo('cpu_weight', lambda: 1.0) defaultInfo('vcpus', lambda: 1) defaultInfo('vcpu_avail', lambda: (1 << self.info['vcpus']) - 1) + defaultInfo('allocmap' , lambda: None) defaultInfo('bootloader', lambda: None) defaultInfo('backend', lambda: []) defaultInfo('device', lambda: []) @@ -1018,6 +1020,16 @@ self.image.handleBootloading() xc.domain_setcpuweight(self.domid, self.info['cpu_weight']) + cpu = self.info['cpu'] + if cpu is not None and cpu != -1: + xc.domain_pincpu(self.domid, 0, 1 << cpu) + + # set the domain alloc map for future vcpus, + # repin VCPU0 according to the alloc map + allocmap = self.info['allocmap'] + if self.domid and allocmap: + xc.domain_pincpu(self.domid, -1, allocmap) # domain allocmap + xc.domain_pincpu(self.domid, 0, allocmap) # repin VCPU0 # increase the total number of vcpus in domain xc.domain_vcpus_increase(self.domid, int(self.info['vcpus'])); @@ -1026,10 +1038,6 @@ m = self.image.getDomainMemory(self.info['memory_KiB']) xc.domain_setmaxmem(self.domid, m) xc.domain_memory_increase_reservation(self.domid, m, 0, 0) - - cpu = self.info['cpu'] - if cpu is not None and cpu != -1: - xc.domain_pincpu(self.domid, 0, 1 << cpu) self.info['start_time'] = time.time() diff -r 12d1e93653c2 tools/python/xen/xm/create.py --- a/tools/python/xen/xm/create.py Tue Oct 11 00:13:12 2005 +++ b/tools/python/xen/xm/create.py Mon Oct 10 19:35:55 2005 @@ -151,6 +151,10 @@ gopts.var('cpu', val='CPU', fn=set_int, default=None, use="CPU to run the domain on.") + +gopts.var('allocmap', val='ALLOCMAP', + fn=set_int, default=None, + use="Set default cpumap used for allocating vcpus.") gopts.var('vcpus', val='VCPUS', fn=set_int, default=1, @@ -562,6 +566,8 @@ config.append(['maxmem', vals.maxmem]) if vals.cpu is not None: config.append(['cpu', vals.cpu]) + if vals.allocmap is not None: + config.append(['allocmap', vals.allocmap]) if vals.cpu_weight is not None: config.append(['cpu_weight', vals.cpu_weight]) if vals.blkif: diff -r 12d1e93653c2 xen/common/dom0_ops.c --- a/xen/common/dom0_ops.c Tue Oct 11 00:13:12 2005 +++ b/xen/common/dom0_ops.c Mon Oct 10 19:35:55 2005 @@ -275,7 +275,7 @@ { domid_t dom = op->u.pincpudomain.domain; struct domain *d = find_domain_by_id(dom); - struct vcpu *v; + struct vcpu *v = NULL; cpumap_t cpumap; @@ -285,27 +285,33 @@ break; } - if ( (op->u.pincpudomain.vcpu >= MAX_VIRT_CPUS) || - !d->vcpu[op->u.pincpudomain.vcpu] ) - { - ret = -EINVAL; - put_domain(d); - break; - } - - v = d->vcpu[op->u.pincpudomain.vcpu]; - if ( v == NULL ) - { - ret = -ESRCH; - put_domain(d); - break; - } - - if ( v == current ) - { - ret = -EINVAL; - put_domain(d); - break; + /* don't bail on vcpu = -1 as that sets domain cpumap */ + if ( (op->u.pincpudomain.vcpu != -1) && + ((op->u.pincpudomain.vcpu >= MAX_VIRT_CPUS) || + !d->vcpu[op->u.pincpudomain.vcpu]) + ) + { + ret = -EINVAL; + put_domain(d); + break; + } + + /* don't get a struct vcpu pointer for -1 op */ + if (op->u.pincpudomain.vcpu != -1) { + v = d->vcpu[op->u.pincpudomain.vcpu]; + if ( v == NULL ) + { + ret = -ESRCH; + put_domain(d); + break; + } + + if ( v == current ) + { + ret = -EINVAL; + put_domain(d); + break; + } } if ( copy_from_user(&cpumap, op->u.pincpudomain.cpumap, @@ -316,24 +322,28 @@ break; } - /* update cpumap for this vcpu */ - v->cpumap = cpumap; - - if ( cpumap == CPUMAP_RUNANYWHERE ) - { - clear_bit(_VCPUF_cpu_pinned, &v->vcpu_flags); - } - else - { - /* pick a new cpu from the usable map */ - int new_cpu = (int)find_first_set_bit(cpumap) % num_online_cpus(); - - vcpu_pause(v); - vcpu_migrate_cpu(v, new_cpu); - set_bit(_VCPUF_cpu_pinned, &v->vcpu_flags); - vcpu_unpause(v); - } - + /* update domain vcpu alloc map */ + if ( v == NULL ) { + d->allocmap = cpumap; + } else { + /* update cpumap for this vcpu */ + v->cpumap = cpumap; + + if ( cpumap == CPUMAP_RUNANYWHERE ) + { + clear_bit(_VCPUF_cpu_pinned, &v->vcpu_flags); + } + else + { + /* pick a new cpu from the usable map */ + int new_cpu = get_next_processor(d, v, &cpumap); + + vcpu_pause(v); + vcpu_migrate_cpu(v, new_cpu); + set_bit(_VCPUF_cpu_pinned, &v->vcpu_flags); + vcpu_unpause(v); + } + } put_domain(d); } break; diff -r 12d1e93653c2 xen/common/domain.c --- a/xen/common/domain.c Tue Oct 11 00:13:12 2005 +++ b/xen/common/domain.c Mon Oct 10 19:35:55 2005 @@ -41,6 +41,7 @@ atomic_set(&d->refcnt, 1); atomic_set(&v->pausecnt, 0); + d->allocmap = CPUMAP_RUNANYWHERE; d->domain_id = dom_id; v->processor = cpu; @@ -380,7 +381,7 @@ v = d->vcpu[vcpuid]; atomic_set(&v->pausecnt, 0); - v->cpumap = CPUMAP_RUNANYWHERE; + v->cpumap = d->allocmap; memcpy(&v->arch, &idle0_vcpu.arch, sizeof(v->arch)); @@ -469,6 +470,31 @@ return -ENOSYS; } + +/* find the least loaded processor , ignorning vcpu v, in cpumap_t *map */ +int get_next_processor(struct domain* d, struct vcpu *v, cpumap_t *map) { + struct vcpu *vc = NULL; + int pro, i, cnt[NR_CPUS] = { 0 }; + + /* count the processor layout for this dom, except for vcpu v + * whose processor field may not have been set yet. */ + for_each_vcpu( d, vc ) { + if (vc->vcpu_id != v->vcpu_id) + cnt[vc->processor]++; + } + + /* start from the first allowable cpu, guard against bogus cpus */ + pro = (int)find_first_set_bit(*map) % num_online_cpus(); + + /* pick least loaded processor in the map */ + for ( i = pro; i < num_online_cpus(); i++ ) { + if ( test_bit(i, &*map) && (cnt[i] <= cnt[pro]) ) + pro = i; + } + + return pro; +} + /* * Local variables: diff -r 12d1e93653c2 xen/common/schedule.c --- a/xen/common/schedule.c Tue Oct 11 00:13:12 2005 +++ b/xen/common/schedule.c Mon Oct 10 19:35:55 2005 @@ -125,12 +125,13 @@ v->next_in_list = vc->next_in_list; vc->next_in_list = v; - if (test_bit(_VCPUF_cpu_pinned, &vc->vcpu_flags)) { - v->processor = (vc->processor + 1) % num_online_cpus(); + + /* XXX: if previous vcpu was pinned, mark new vcpu as pinned why? */ + if (test_bit(_VCPUF_cpu_pinned, &vc->vcpu_flags)) set_bit(_VCPUF_cpu_pinned, &v->vcpu_flags); - } else { - v->processor = (vc->processor + 1) % num_online_cpus(); - } + + v->processor = get_next_processor(d, v, &d->allocmap); + } return v; diff -r 12d1e93653c2 xen/include/public/dom0_ops.h --- a/xen/include/public/dom0_ops.h Tue Oct 11 00:13:12 2005 +++ b/xen/include/public/dom0_ops.h Mon Oct 10 19:35:55 2005 @@ -181,7 +181,7 @@ typedef struct { /* IN variables. */ domid_t domain; - u16 vcpu; + s16 vcpu; cpumap_t *cpumap; } dom0_pincpudomain_t; diff -r 12d1e93653c2 xen/include/xen/domain.h --- a/xen/include/xen/domain.h Tue Oct 11 00:13:12 2005 +++ b/xen/include/xen/domain.h Mon Oct 10 19:35:55 2005 @@ -27,4 +27,6 @@ extern void dump_pageframe_info(struct domain *d); +int get_next_processor(struct domain *d, struct vcpu *v, cpumap_t *map); + #endif /* __XEN_DOMAIN_H__ */ diff -r 12d1e93653c2 xen/include/xen/sched.h --- a/xen/include/xen/sched.h Tue Oct 11 00:13:12 2005 +++ b/xen/include/xen/sched.h Mon Oct 10 19:35:55 2005 @@ -134,6 +134,8 @@ /* Bitmask of CPUs which are holding onto this domain's state. */ cpumask_t cpumask; + + cpumap_t allocmap; /* vcpu allocation bitmap */ struct arch_domain arch;
Re: [PATCH] 2nd try: 2/2 VCPU creation and allocation
Quoted 1 line · Ryan Harper
Allocation map.
What does this patch buy us, given that we already support per-vcpu maps? If I understand correctly, this adds a per-domain map, which seems redundant to me.
-- Keir
Re: [PATCH] 2nd try: 2/2 VCPU creation and allocation
Quoted 8 lines · Keir Fraser
Quoted 1 line · Ryan Harper
Allocation map.
What does this patch buy us, given that we already support per-vcpu maps? If I understand correctly, this adds a per-domain map, which seems redundant to me.
I guess not as much as it did before we had build-time vcpu creation. I can easily translate the allocation map in the configuration file into a set of pincpu ops before allocating domain memory and achieve the same effect as the patch does without the changes to the domain structure and pincpu dom0_op.
I'll resend without the changes to xen.
Signature · 5 lines
Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 [email protected]