RFC: vcpu pinning at qemu start
cover RFC: vcpu pinning at qemu start
Hi guys,
Here's a first series of patch aiming at vcpu pinning support in qemu. Ideally, as vcpu as just normal threads, the usual userspace tools can be used to set cpu affinities mask.
However, It makes it very difficult to _start_ a vm with vcpus pinned, since we don't know the thread ids from qemu in advance, nor do we know when are the vcpus created.
The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus with the specified affinities.
Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead!
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
PATCH 1/8 add thread id to vcpu structure
This allow us to track which thread is currently running the cpu
--- qemu/qemu-kvm.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
qemu/qemu-kvm.c +1 -0
@@ -47,6 +47,7 @@ struct vcpu_info {int signalled;int stop;int stopped; +int thread_id; } vcpu_info[256 ];CPUState *qemu_kvm_cpu_env (int index)
Signature · 7 lines
1.5.0.6 ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
PATCH 2/8 provide a gettid function
Since glibc does not provide a gettid call directly, only through syscall, we wrap one for kvm
--- qemu/qemu-kvm.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)
qemu/qemu-kvm.c +6 -0
@@ -19,6 +19,7 @@ int kvm_irqchip = 1;#include <libkvm.h> #include <pthread.h> #include <sys/utsname.h> +#include <sys/syscall.h> extern void perror (const char *s);
@@ -50,6 +51,11 @@ struct vcpu_info {int thread_id; } vcpu_info[256 ]; +static inline unsigned long kvm_get_thread_id (void ) +{ +return syscall (SYS_gettid); +} +CPUState *qemu_kvm_cpu_env (int index) {return vcpu_info[index].env;
Signature · 7 lines
1.5.0.6 ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
PATCH 3/8 get thread id at thread's creation
store the thread id through our new call as soon as the process is created.
--- qemu/qemu-kvm.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-)
qemu/qemu-kvm.c +2 -0
@@ -335,6 +335,7 @@ static void *ap_main_loop(void *_env) vcpu = &vcpu_info[env->cpu_index]; vcpu->env = env; + vcpu->thread_id =kvm_get_thread_id ();sigfillset (&signals);//sigdelset(&signals, SIG_IPI); sigprocmask (SIG_BLOCK , &signals,NULL );
@@ -381,6 +382,7 @@ int kvm_init_ap(void) vcpu = &vcpu_info[0 ]; vcpu->env = first_cpu; + vcpu->thread_id =kvm_get_thread_id ();signal (SIG_IPI , sig_ipi_handler);for (i =1 ; i < smp_cpus; ++i) {kvm_init_new_ap (i, env);
Signature · 7 lines
1.5.0.6 ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
PATCH 4/8 store and set cpu affinities
This patch provides a cpu_set variable to vcpu_info structure. It stores the current cpu mask for a thread.
We also provide a wrapper for storing a provided affinity, and a function to set default affinities: The default affinities are current process'
--- qemu/qemu-kvm.c | 17 +++++++++++++++++ 1 files changed, 17 insertions(+), 0 deletions(-)
qemu/qemu-kvm.c +17 -0
@@ -49,8 +49,25 @@ struct vcpu_info {int stop;int stopped;int thread_id; +cpu_set_t cpu_set; } vcpu_info[256 ]; +void kvm_store_cpu_affinity (int cpu,cpu_set_t *map) +{ +memcpy (&vcpu_info[cpu].cpu_set, map,sizeof (*map)); +} + +void kvm_register_default_affinities (void ) +{ +int i; +cpu_set_t dfl; +sched_getaffinity (0 ,sizeof (dfl), &dfl); + +for (i =0 ; i <256 ; i++) { +memcpy (&vcpu_info[i].cpu_set, &dfl,sizeof (dfl)); + } +} +static inline unsigned long kvm_get_thread_id (void ) {return syscall (SYS_gettid);
Signature · 7 lines
1.5.0.6 ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
PATCH 5/8 initialize affinities
store default process affinities before we get the chance to process any options
--- qemu/vl.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
qemu/vl.c +3 -0
@@ -8717,6 +8717,9 @@ int main(int argc, char **argv) }#endif +#ifdef USE_KVM +kvm_register_default_affinities (); +#endif register_machines (); machine = first_machine; cpu_model =NULL ;
Signature · 7 lines
1.5.0.6 ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
PATCH 6/8 stabilish default affinity for newly created cpus
--- qemu/qemu-kvm.c | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-)
qemu/qemu-kvm.c +8 -0
@@ -68,6 +68,12 @@ void kvm_register_default_affinities(void) } } +static inline void kvm_default_affinity (int cpu) +{ +sched_setaffinity (vcpu_info[cpu].thread_id,sizeof (cpu_set_t), + &vcpu_info[cpu].cpu_set); +} +static inline unsigned long kvm_get_thread_id (void ) {return syscall (SYS_gettid);
@@ -353,6 +359,7 @@ static void *ap_main_loop(void *_env) vcpu = &vcpu_info[env->cpu_index]; vcpu->env = env; vcpu->thread_id =kvm_get_thread_id (); +kvm_default_affinity (env->cpu_index);sigfillset (&signals);//sigdelset(&signals, SIG_IPI); sigprocmask (SIG_BLOCK , &signals,NULL );
@@ -400,6 +407,7 @@ int kvm_init_ap(void) vcpu = &vcpu_info[0 ]; vcpu->env = first_cpu; vcpu->thread_id =kvm_get_thread_id (); +kvm_default_affinity (0 );signal (SIG_IPI , sig_ipi_handler);for (i =1 ; i < smp_cpus; ++i) {kvm_init_new_ap (i, env);
Signature · 7 lines
1.5.0.6 ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
PATCH 7/8 process a cpu affinity mask
This patch provides a function that process a cpu affinity list in the form x,y,z... into a cpu_set_t variable.
--- qemu/vl.c | 17 +++++++++++++++++ 1 files changed, 17 insertions(+), 0 deletions(-)
qemu/vl.c +17 -0
@@ -8646,6 +8646,23 @@ void *qemu_alloc_physram(unsigned long memory)return area; } +#ifdef USE_KVM +#include <sched.h> +void process_cpu_set (const char *map,cpu_set_t *set) +{ +char *ptr = map; +int c; +CPU_ZERO (set); +do { + c =atoi (ptr); +CPU_SET (c, set); + ptr =strchr (ptr,',' ); +if (!ptr) +break ; + ptr++; + }while (*ptr); +} +#endif int main (int argc,char **argv) {
Signature · 7 lines
1.5.0.6 ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
PATCH 8/8 provide -cpu-map option
this patch introduces a -cpu-map option.
It has the form C:x,y,z..., and have the effect of setting the affinity mask of vcpu C to processors x,y,z...
--- qemu/vl.c | 13 +++++++++++++ 1 files changed, 13 insertions(+), 0 deletions(-)
qemu/vl.c +13 -0
@@ -8059,6 +8059,7 @@ static void help(int exitcode)"-no-kvm disable KVM hardware virtualization\n" #endif"-no-kvm-irqchip disable KVM kernel mode PIC/IOAPIC/LAPIC\n" + "-cpu-map C:x,y,z... set cpu 'C' affinity to processors x,y,z...\n"#endif #ifdef TARGET_I386 "-std-vga simulate a standard VGA card with VESA Bochs Extensions\n"
@@ -8177,6 +8178,7 @@ enum { QEMU_OPTION_no_acpi, QEMU_OPTION_no_kvm, QEMU_OPTION_no_kvm_irqchip, + QEMU_OPTION_cpu_map, QEMU_OPTION_no_reboot, QEMU_OPTION_show_cursor, QEMU_OPTION_daemonize,
@@ -8263,6 +8265,7 @@ const QEMUOption qemu_options[] = { { "no-kvm", 0, QEMU_OPTION_no_kvm }, #endif { "no-kvm-irqchip", 0, QEMU_OPTION_no_kvm_irqchip }, + { "cpu-map", HAS_ARG, QEMU_OPTION_cpu_map }, #endif #if defined(TARGET_PPC) || defined(TARGET_SPARC) { "g", 1, QEMU_OPTION_g },
@@ -9211,6 +9214,16 @@ int main(int argc, char **argv) kvm_irqchip =0 ; break; } +case QEMU_OPTION_cpu_map: { +int c =atoi (optarg); +char *ptr =strchr (optarg,':' ); +cpu_set_t set; +if (!ptr) +fprintf (stderr,"invalid cpu mapping %s\n" , ptr); +process_cpu_set (++ptr, &set); +kvm_store_cpu_affinity (c, &set); +break ; + }#endif case QEMU_OPTION_usb: usb_enabled =1 ;
Signature · 7 lines
1.5.0.6 ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Quoted 6 lines · Glauber Costa
Hi guys,
Here's a first series of patch aiming at vcpu pinning support in qemu. Ideally, as vcpu as just normal threads, the usual userspace tools can be used to set cpu affinities mask.
Quoted 8 lines
However, It makes it very difficult to _start_ a vm with vcpus pinned, since we don't know the thread ids from qemu in advance, nor do we know when are the vcpus created.
The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus with the specified affinities.
Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead!
Cool, this goes into the same direction as I planned for KVM-NUMA support. Do you plan to extend vcpu pinning into that direction?
Joerg
Signature · 12 lines
| AMD Saxony Limited Liability Company & Co. KG
Operating | Wilschdorfer Landstr. 101, 01109 Dresden, Germany
System | Register Court Dresden: HRA 4896
Research | General Partner authorized to represent:
Center | AMD Saxony LLC (Wilmington, Delaware, US)
| General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/Quoted 11 lines · Joerg Roedel
Quoted 10 lines · Glauber Costa
Hi guys,
Here's a first series of patch aiming at vcpu pinning support in qemu. Ideally, as vcpu as just normal threads, the usual userspace tools can be used to set cpu affinities mask.
However, It makes it very difficult to _start_ a vm with vcpus pinned, since we don't know the thread ids from qemu in advance, nor do we know when are the vcpus created.
Quoted 7 lines
Quoted 4 lines
The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus with the specified affinities.
Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead!
Cool, this goes into the same direction as I planned for KVM-NUMA support. Do you plan to extend vcpu pinning into that direction?
I don't have any immediate plans, but it is surely interesting. If the patches (or something inspired in them) make it, there's something we can draw support for.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Quoted 12 lines · Glauber Costa
Quoted 11 lines · Joerg Roedel
Quoted 10 lines · Glauber Costa
Hi guys,
Here's a first series of patch aiming at vcpu pinning support in qemu. Ideally, as vcpu as just normal threads, the usual userspace tools can be used to set cpu affinities mask.
However, It makes it very difficult to _start_ a vm with vcpus pinned, since we don't know the thread ids from qemu in advance, nor do we know when are the vcpus created.
Quoted 7 lines
Quoted 6 lines
Quoted 4 lines
The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus with the specified affinities.
Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead!
Cool, this goes into the same direction as I planned for KVM-NUMA support. Do you plan to extend vcpu pinning into that direction?
I don't have any immediate plans, but it is surely interesting. If the patches (or something inspired in them) make it, there's something we can draw support for.
There are patches for HVM-NUMA support on Xen developed by André Przywara. I think they are easy to port to KVM. Maybe it is the better aproach than implementing simple vcpu pinning.
Joerg
Signature · 16 lines
| AMD Saxony Limited Liability Company & Co. KG
Operating | Wilschdorfer Landstr. 101, 01109 Dresden, Germany
System | Register Court Dresden: HRA 4896
Research | General Partner authorized to represent:
Center | AMD Saxony LLC (Wilmington, Delaware, US)
| General Manager of AMD Saxony LLC: Dr. Hans-R. Deppe, Thomas McCoy
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-develQuoted 6 lines · Glauber Costa
Hi guys,
Here's a first series of patch aiming at vcpu pinning support in qemu. Ideally, as vcpu as just normal threads, the usual userspace tools can be used to set cpu affinities mask.
Quoted 10 lines
However, It makes it very difficult to _start_ a vm with vcpus pinned, since we don't know the thread ids from qemu in advance, nor do we know when are the vcpus created.
The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus with the specified affinities.
Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead!
So why exactly is this useful? I have a hard time constructing a reasonable use-case in my mind for something like this.
Regards,
Anthony Liguori
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Quoted 18 lines · Anthony Liguori
Quoted 17 lines · Glauber Costa
Hi guys,
Here's a first series of patch aiming at vcpu pinning support in qemu. Ideally, as vcpu as just normal threads, the usual userspace tools can be used
to set cpu affinities mask.However, It makes it very difficult to _start_ a vm with vcpus pinned, since
we don't know the thread ids from qemu in advance, nor do we know when are the
vcpus created.The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus
with the specified affinities.
Quoted 7 lines
Quoted 4 lines
Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead!
So why exactly is this useful? I have a hard time constructing a reasonable use-case in my mind for something like this.
My main interest is in management tools being able to specify pinning set ups at VM creation time.
As I said, it can be done through tools like taskset, but then you'd have to know:
* when are the threads created
* which thread ids corresponds to each cpu
And of course, for an amount of time, the threads will be running in a "wrong" cpu, which may affect workloads running there. (which is a case cpu pinning usually tries to address)
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Quoted 11 lines · Glauber Costa
My main interest is in management tools being able to specify pinning set ups at VM creation time.
As I said, it can be done through tools like taskset, but then you'd have to know:
* when are the threads created
* which thread ids corresponds to each cpuAnd of course, for an amount of time, the threads will be running in a "wrong" cpu, which may affect workloads running there. (which is a case cpu pinning usually tries to address)
A management tool can start QEMU with -S to prevent any CPUs from running, query the VCPU=>thread id relationship (modifying info cpus would be a good thing to do for this), taskset, and then run 'cont' in the monitor if they desperately need this functionality. However, I don't think the vast majority of people need this particular functionality.
My feeling is that adding an interface to do this in QEMU encourages people to not use the existing Linux tools for this or worse yet, to think they can do a better job than Linux. The whole reason this exists in Xen is that Xen's schedulers were incapable of doing CPU migration historically (which is no longer true since the credit scheduler). It was necessary to specify pinning upon creation or you were stuck with round-robin placement. So libvirt has APIs for this because they were part of the Xen API because it was needed to get reasonable performance at some point in time on Xen. I don't think this behavior is useful for KVM though. Just because Xen does it doesn't imply that we should do it.
Regards,
Anthony Liguori
Quoted 2 lines
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Quoted 13 lines · Anthony Liguori
Quoted 11 lines · Glauber Costa
My main interest is in management tools being able to specify pinning set ups at VM creation time.
As I said, it can be done through tools like taskset, but then you'd have to know:
* when are the threads created
* which thread ids corresponds to each cpuAnd of course, for an amount of time, the threads will be running in a "wrong" cpu, which may affect workloads running there. (which is a case cpu pinning usually tries to address)
Quoted 5 lines
A management tool can start QEMU with -S to prevent any CPUs from running, query the VCPU=>thread id relationship (modifying info cpus would be a good thing to do for this), taskset, and then run 'cont' in the monitor if they desperately need this functionality. However, I don't think the vast majority of people need this particular functionality.
I fully expected to have to run QEMU with -S and then use cont if I were todo CPU pinning from libvirt.
The only info I'd need to get is the PID <-> vCPU mapping data. Then I can use regular Linux taskset capabilities from libvirt to assign the initial pCPU <-> vCPU mapping and finally run 'cont'.
Quoted 10 lines
My feeling is that adding an interface to do this in QEMU encourages people to not use the existing Linux tools for this or worse yet, to think they can do a better job than Linux. The whole reason this exists in Xen is that Xen's schedulers were incapable of doing CPU migration historically (which is no longer true since the credit scheduler). It was necessary to specify pinning upon creation or you were stuck with round-robin placement. So libvirt has APIs for this because they were part of the Xen API because it was needed to get reasonable performance at some point in time on Xen. I don't think this behavior is useful for KVM though. Just because Xen does it doesn't imply that we should do it.
I agree that adding QEMU commands for stuff which Linux already has APIs and tools is a bad idea. QEMU/KVM is much nicer to manage than Xen, precisely because I can already use Linux APIs & process management tools.
Dan.
Signature · 9 lines
|=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=| ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Quoted 37 lines · Daniel P. Berrange
Quoted 9 lines · Anthony Liguori
Quoted 8 lines · Glauber Costa
My main interest is in management tools being able to specify pinning set ups at VM creation time.
As I said, it can be done through tools like taskset, but then you'd have to know:
* when are the threads created
* which thread ids corresponds to each cpuQuoted 8 lines
Quoted 3 lines
And of course, for an amount of time, the threads will be running in a "wrong" cpu, which may affect workloads running there. (which is a case cpu pinning usually tries to address)
A management tool can start QEMU with -S to prevent any CPUs from running, query the VCPU=>thread id relationship (modifying info cpus would be a good thing to do for this), taskset, and then run 'cont' in the monitor if they desperately need this functionality. However, I don't think the vast majority of people need this particular functionality.
I fully expected to have to run QEMU with -S and then use cont if I were todo CPU pinning from libvirt.
The only info I'd need to get is the PID <-> vCPU mapping data. Then I can use regular Linux taskset capabilities from libvirt to assign the initial pCPU <-> vCPU mapping and finally run 'cont'.
Quoted 10 lines
My feeling is that adding an interface to do this in QEMU encourages people to not use the existing Linux tools for this or worse yet, to think they can do a better job than Linux. The whole reason this exists in Xen is that Xen's schedulers were incapable of doing CPU migration historically (which is no longer true since the credit scheduler). It was necessary to specify pinning upon creation or you were stuck with round-robin placement. So libvirt has APIs for this because they were part of the Xen API because it was needed to get reasonable performance at some point in time on Xen. I don't think this behavior is useful for KVM though. Just because Xen does it doesn't imply that we should do it.
Quoted 3 lines
I agree that adding QEMU commands for stuff which Linux already has APIs and tools is a bad idea. QEMU/KVM is much nicer to manage than Xen, precisely because I can already use Linux APIs & process management tools.
I totally agree this is ideal, and I did not start this after thinking a little bit about
this situation. The main point is that we don't know when the cpus are created, and it does not seem to me that we will without a considerable amount of work.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Quoted 13 lines · Anthony Liguori
Quoted 11 lines · Glauber Costa
My main interest is in management tools being able to specify pinning set ups at VM creation time.
As I said, it can be done through tools like taskset, but then you'd have to know:
* when are the threads created
* which thread ids corresponds to each cpuAnd of course, for an amount of time, the threads will be running in a "wrong" cpu, which may affect workloads running there. (which is a case cpu pinning usually tries to address)
Quoted 5 lines
A management tool can start QEMU with -S to prevent any CPUs from running, query the VCPU=>thread id relationship (modifying info cpus would be a good thing to do for this), taskset, and then run 'cont' in the monitor if they desperately need this functionality. However, I don't think the vast majority of people need this particular functionality.
No, it can't. Because at the time qemu starts, no vcpu -> thread id relationship exists at all. And we don't know when it will.
It would be a different story if there were some kind of api that could warn qemu
Quoted 3 lines
My feeling is that adding an interface to do this in QEMU encourages people to not use the existing Linux tools for this or worse yet, to think they can do a better job than Linux.
I agree with you that we should stick with linux tools, and that's why I didn't provide any kind of runtime setting via qemu monitor to do this (with the infrastructure, it would be trivial). taskset will do.
Quoted 8 lines
The whole reason this exists
in Xen is that Xen's schedulers were incapable of doing CPU migration historically (which is no longer true since the credit scheduler). It was necessary to specify pinning upon creation or you were stuck with round-robin placement. So libvirt has APIs for this because they were part of the Xen API because it was needed to get reasonable performance at some point in time on Xen. I don't think this behavior is useful for KVM though. Just because Xen does it doesn't imply that we should do it.
No, not just because xen does.
I do however feel it useful, since starting a vm and then let it run unchanged is definitely an useful use case. And as I tried to show you, I can't see a good way to do that for pinning.
Quoted 7 lines
Regards,
Anthony Liguori
Quoted 2 lines
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Quoted 4 lines · Glauber Costa
Anthony Liguori wrote:
No, it can't. Because at the time qemu starts, no vcpu -> thread id relationship exists at all. And we don't know when it will.
Sure we do. The vcpu -> thread id relationship is valid after kvm_init_ap() is called which is after machine init but before the select loop is entered for the first time. Therefore, if you start qemu with -S, then connect on the monitor, and do an info cpus, you could be guaranteed to be told the mapping.
The threads are *idle* at this point so there's no harm if they were started on the "wrong" CPU. You can now taskset to your hearts content and then when you're happy with placement, you can issue a 'cont' so that the VM actually starts running. I saw "wrong" because you can still taskset the initial creation guaranteeing that the threads are created on the right group of physical CPUs, you just can't specify the exact mapping until you start interacting with the monitor.
Regards,
Anthony Liguori
Quoted 8 lines
Quoted 7 lines
Regards,
Anthony Liguori
Quoted 2 lines
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Quoted 14 lines · Anthony Liguori
Quoted 5 lines · Glauber Costa
Anthony Liguori wrote:
No, it can't. Because at the time qemu starts, no vcpu -> thread id relationship exists at all. And we don't know when it will.
Sure we do. The vcpu -> thread id relationship is valid after
kvm_init_ap()is called which is after machine init but before the select loop is entered for the first time. Therefore, if you start qemu with -S, then connect on the monitor, and do an info cpus, you could be guaranteed to be told the mapping.
Quoted 9 lines
The threads are *idle* at this point so there's no harm if they were started on the "wrong" CPU. You can now taskset to your hearts content and then when you're happy with placement, you can issue a 'cont' so that the VM actually starts running. I saw "wrong" because you can still taskset the initial creation guaranteeing that the threads are created on the right group of physical CPUs, you just can't specify the exact mapping until you start interacting with the monitor.
Good points. Initially I thought we ought to abstract the implementation and not expose the vcpu thread id, but I'm beginning to thing that due the wide variety of options (affinity, page migration, priority, cpu control groups) and the relative obscurity of the feature (which as you point out, isn't needed in the common case), we can export the thread id and let the management tools deal with it directly.
Signature · 7 lines
Do not meddle in the internals of kernels, for they are subtle and quick to panic. ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Quoted 11 lines · Anthony Liguori
Quoted 4 lines · Glauber Costa
Anthony Liguori wrote:
No, it can't. Because at the time qemu starts, no vcpu -> thread id relationship exists at all. And we don't know when it will.
Sure we do. The vcpu -> thread id relationship is valid after
kvm_init_ap()is called which is after machine init but before the select loop is entered for the first time. Therefore, if you start qemu with -S, then connect on the monitor, and do an info cpus, you could be guaranteed to be told the mapping.
I missed that. This changes everything. I now completely agree with you.
I'll post patches that expose the relationship, if it's better.
Quoted 8 lines
The threads are *idle* at this point so there's no harm if they were started on the "wrong" CPU. You can now taskset to your hearts content and then when you're happy with placement, you can issue a 'cont' so that the VM actually starts running. I saw "wrong" because you can still taskset the initial creation guaranteeing that the threads are created on the right group of physical CPUs, you just can't specify the exact mapping until you start interacting with the monitor.
Quoted 13 lines
Regards,
Anthony Liguori
Quoted 8 lines
Quoted 7 lines
Regards,
Anthony Liguori
Quoted 2 lines
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Quoted 13 lines · Anthony Liguori
Quoted 11 lines · Glauber Costa
My main interest is in management tools being able to specify pinning set ups at VM creation time.
As I said, it can be done through tools like taskset, but then you'd have to know:
* when are the threads created
* which thread ids corresponds to each cpuAnd of course, for an amount of time, the threads will be running in a "wrong" cpu, which may affect workloads running there. (which is a case cpu pinning usually tries to address)
Quoted 7 lines
A management tool can start QEMU with -S to prevent any CPUs from running, query the VCPU=>thread id relationship (modifying info cpus would be a good thing to do for this), taskset, and then run 'cont' in the monitor if they desperately need this functionality. However, I don't think the vast majority of people need this particular functionality.
Affinity control is probably useful mostly for numa configurations, where you want to restrict virtual cpus to run on the cores closest to memory. However it may well be that the scheduler is already good enough to do this on its own.
Quoted 12 lines
My feeling is that adding an interface to do this in QEMU encourages people to not use the existing Linux tools for this or worse yet, to think they can do a better job than Linux. The whole reason this exists in Xen is that Xen's schedulers were incapable of doing CPU migration historically (which is no longer true since the credit scheduler). It was necessary to specify pinning upon creation or you were stuck with round-robin placement. So libvirt has APIs for this because they were part of the Xen API because it was needed to get reasonable performance at some point in time on Xen. I don't think this behavior is useful for KVM though. Just because Xen does it doesn't imply that we should do it.
In the brutal world of hypervisors, if your competitor has a feature, you must have it too. I often get asked about cpu pinning in kvm.
[I'd like to see how Xen implements swapping, though]
Signature · 7 lines
Do not meddle in the internals of kernels, for they are subtle and quick to panic. ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Quoted 4 lines · Avi Kivity
Affinity control is probably useful mostly for numa configurations, where you want to restrict virtual cpus to run on the cores closest to memory. However it may well be that the scheduler is already good enough to do this on its own.
In that case, you need to use numactl to set a NUMA policy so it's pretty natural that you would also be using taskset. Assuming you're trying to keep a VM local to a particular node (we don't expose a virtual SRAT/SLIT table so that's all we can sanely do right now), it doesn't matter what CPU each VCPU thread lands on as long as they stay within the node. So taskset is perfectly capable to address this need today.
Quoted 2 lines
In the brutal world of hypervisors, if your competitor has a feature, you must have it too. I often get asked about cpu pinning in kvm.
And the answer to give is, of course, we support it through taskset :-)
Regards,
Anthony Liguori
Quoted 2 lines
[I'd like to see how Xen implements swapping, though]
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Quoted 6 lines · Glauber Costa
Hi guys,
Here's a first series of patch aiming at vcpu pinning support in qemu. Ideally, as vcpu as just normal threads, the usual userspace tools can be used to set cpu affinities mask.
Quoted 10 lines
However, It makes it very difficult to _start_ a vm with vcpus pinned, since we don't know the thread ids from qemu in advance, nor do we know when are the vcpus created.
The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus with the specified affinities.
Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead!
A monitor interface would be more useful than a command line option, as it allows you to migrate the vcpus at runtime, and also control hotplugged cpus. For unmanaged use, taskset is probably sufficient to control affinity from the command line.
Normally I encourage splitting patches, but this is a bit extreme. 1 and 3 are pointless without each other, 4 and 5, 7 and 8. Hope that doesn't interfere with any pay-per-patch contract.
Signature · 7 lines
Do not meddle in the internals of kernels, for they are subtle and quick to panic. ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
Quoted 23 lines · Avi Kivity
Quoted 13 lines · Glauber Costa
Hi guys,
Here's a first series of patch aiming at vcpu pinning support in qemu. Ideally, as vcpu as just normal threads, the usual userspace tools can be used
to set cpu affinities mask.However, It makes it very difficult to _start_ a vm with vcpus pinned, since
we don't know the thread ids from qemu in advance, nor do we know when are the
vcpus created.Quoted 8 lines
The patches introduce a -cpu-map option, that, if specified, starts the virtual cpus
with the specified affinities.Comments? Welcome. Random rants? Not welcome, but... how can I stop you? So go ahead!
Quoted 9 lines
A monitor interface would be more useful than a command line option, as it allows you to migrate the vcpus at runtime, and also control hotplugged cpus. For unmanaged use, taskset is probably sufficient to control affinity from the command line.
Normally I encourage splitting patches, but this is a bit extreme. 1 and 3 are pointless without each other, 4 and 5, 7 and 8. Hope that doesn't interfere with any pay-per-patch contract.
I'll post a new series that just expose the thread ids, so the management tools can use plain taskset. As for the split, there's nothing in my contract, but after all the x86 integration, it became a mania for me. Ingo made me this way.
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/