Enhance perf to collect KVM guest os statistics from host side

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

diff -Nraup linux-2.6_tipmaster0315/arch/x86/kvm/vmx.c linux-2.6_tipmaster0315_perfkvm/arch/x86/kvm/vmx.c

arch/x86/kvm/vmx.c +3 -0

@@ -26,6 +26,7 @@
  #include<linux/sched.h>
  #include<linux/moduleparam.h>
  #include<linux/ftrace_event.h>
+#include<linux/perf_event.h>
  #include "kvm_cache_regs.h"
  #include "x86.h"
 

@@ -3632,6 +3633,43 @@ static void update_cr8_intercept(struct
  	vmcs_write32(TPR_THRESHOLD, irr);
  }
 
+DEFINE_PER_CPU(int, kvm_in_guest) = {0};
+

Quoted 15 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

Quoted 8 lines

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels? Perhaps a symbol server that perf can connect to (and that would connect to guests in turn)?

Quoted 18 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Quoted 6 lines

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels?

Perhaps a
symbol server that perf can connect to (and that would connect to guests in turn)?

Quoted 15 lines

diff -Nraup linux-2.6_tipmaster0315/arch/x86/kvm/vmx.c linux-2.6_tipmaster0315_perfkvm/arch/x86/kvm/vmx.c

arch/x86/kvm/vmx.c +1 -0

@@ -26,6 +26,7 @@
  #include<linux/sched.h>
  #include<linux/moduleparam.h>
  #include<linux/ftrace_event.h>
+#include<linux/perf_event.h>
  #include "kvm_cache_regs.h"
  #include "x86.h"
 

@@ -3632,6 +3633,43 @@ static void update_cr8_intercept(struct
  	vmcs_write32(TPR_THRESHOLD, irr);
  }
 

arch/x86/kvm/vmx.c static void update_cr8_intercept(struct

Quoted 12 lines

arch/x86/kvm/vmx.c +11 -0

+DEFINE_PER_CPU(int, kvm_in_guest) = {0};
+
+static void kvm_set_in_guest(void)
+{
+	percpu_write(kvm_in_guest, 1);
+}
+
+static int kvm_is_in_guest(void)
+{
+	return percpu_read(kvm_in_guest);
+}
 

There is already PF_VCPU for this.

arch/x86/kvm/vmx.c static void update_cr8_intercept(struct

Quoted 8 lines

arch/x86/kvm/vmx.c +6 -0

+static struct perf_guest_info_callbacks kvm_guest_cbs = {
+	.is_in_guest 		= kvm_is_in_guest,
+	.is_user_mode		= kvm_is_user_mode,
+	.get_guest_ip		= kvm_get_guest_ip,
+	.reset_in_guest		= kvm_reset_in_guest,
+};
 

Should be in common code, not vmx specific.

Quoted 26 lines · Avi Kivity

Quoted 15 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

Quoted 9 lines

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Quoted 7 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels? Perhaps a symbol server that perf can connect to (and that would connect to guests in turn)?

The highest quality solution would be if KVM offered a 'guest extension' to the guest kernel's /proc/kallsyms that made it easy for user-space to get this information from an authorative source.

That's the main reason why the host side /proc/kallsyms is so popular and so
useful: while in theory it's mostly redundant information which can be gleaned from the System.map and other sources of symbol information, it's easily available and is _always_ trustable to come from the host kernel.

Separate System.map's have a tendency to go out of sync (or go missing when a devel kernel gets rebuilt, or if a devel package is not installed), and server ports (be that a TCP port space server or an UDP port space mount-point) are both a configuration hassle and are not guest-transparent.

So for instrumentation infrastructure (such as perf) we have a large and well founded preference for intrinsic, built-in, kernel-provided information: i.e. a largely 'built-in' and transparent mechanism to get to guest symbols.

Quoted 19 lines · Avi Kivity

Quoted 18 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Quoted 7 lines

Quoted 6 lines

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Thanks for your kind comments.

Quoted 4 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels?

With the patch, 'perf kvm report --sort pid" could show
summary statistics for all guest os instances. Then, use
parameter --pid of 'perf kvm record' to collect single problematic instance data.

Quoted 6 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels?

With the patch, 'perf kvm report --sort pid" could show
summary statistics for all guest os instances. Then, use
parameter --pid of 'perf kvm record' to collect single problematic instance data.

Quoted 29 lines

Quoted 28 lines

diff -Nraup linux-2.6_tipmaster0315/arch/x86/kvm/vmx.c linux-2.6_tipmaster0315_perfkvm/arch/x86/kvm/vmx.c

arch/x86/kvm/vmx.c +12 -0

@@ -26,6 +26,7 @@
   #include<linux/sched.h>
   #include<linux/moduleparam.h>
   #include<linux/ftrace_event.h>
+#include<linux/perf_event.h>
   #include "kvm_cache_regs.h"
   #include "x86.h"
 

@@ -3632,6 +3633,43 @@ static void update_cr8_intercept(struct
   	vmcs_write32(TPR_THRESHOLD, irr);
   }
 
+DEFINE_PER_CPU(int, kvm_in_guest) = {0};
+
+static void kvm_set_in_guest(void)
+{
+	percpu_write(kvm_in_guest, 1);
+}
+
+static int kvm_is_in_guest(void)
+{
+	return percpu_read(kvm_in_guest);
+}
 
 

Quoted 2 lines

There is already PF_VCPU for this.

Right, but there is a scope between kvm_guest_enter and really running in guest os, where a perf event might overflow. Anyway, the scope is very narrow, I will change it to use flag PF_VCPU.

Quoted 10 lines

arch/x86/kvm/vmx.c static void update_cr8_intercept(struct

Quoted 8 lines

arch/x86/kvm/vmx.c +6 -0

+static struct perf_guest_info_callbacks kvm_guest_cbs = {
+	.is_in_guest 		= kvm_is_in_guest,
+	.is_user_mode		= kvm_is_user_mode,
+	.get_guest_ip		= kvm_get_guest_ip,
+	.reset_in_guest		= kvm_reset_in_guest,
+};
 

Should be in common code, not vmx specific.

Right. I discussed with Yangsheng. I will move above data structures and callbacks to file arch/x86/kvm/x86.c, and add get_ip, a new callback to kvm_x86_ops.

Sorry. I found currently --pid isn't process but a thread (main thread).

Ingo,

Is it possible to support a new parameter or extend --inherit, so 'perf record' and 'perf top' could collect data on all threads of a process when the process is running?

Quoted 29 lines · Yanmin

Quoted 19 lines · Avi Kivity

Quoted 18 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Quoted 7 lines

Quoted 6 lines

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Thanks for your kind comments.

Quoted 7 lines

Quoted 4 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels?

With the patch, 'perf kvm report --sort pid" could show
summary statistics for all guest os instances. Then, use
parameter --pid of 'perf kvm record' to collect single problematic instance data.

Sorry. I found currently --pid isn't process but a thread (main thread).

Ingo,

Is it possible to support a new parameter or extend --inherit, so 'perf record' and 'perf top' could collect data on all threads of a process when the process is running?

If not, I need add a new ugly parameter which is similar to --pid to filter out process data in userspace.

Quoted 46 lines · Ingo Molnar

Quoted 24 lines · Avi Kivity

Quoted 15 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

Quoted 8 lines

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Quoted 6 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels? Perhaps a symbol server that perf can connect to (and that would connect to guests in turn)?

The highest quality solution would be if KVM offered a 'guest extension' to the guest kernel's /proc/kallsyms that made it easy for user-space to get this information from an authorative source.

That's the main reason why the host side /proc/kallsyms is so popular and so
useful: while in theory it's mostly redundant information which can be gleaned from the System.map and other sources of symbol information, it's easily available and is _always_ trustable to come from the host kernel.

Separate System.map's have a tendency to go out of sync (or go missing when a devel kernel gets rebuilt, or if a devel package is not installed), and server ports (be that a TCP port space server or an UDP port space mount-point) are both a configuration hassle and are not guest-transparent.

Quoted 3 lines

So for instrumentation infrastructure (such as perf) we have a large and well founded preference for intrinsic, built-in, kernel-provided information: i.e. a largely 'built-in' and transparent mechanism to get to guest symbols.

The symbol server's client can certainly access the bits through vmchannel.

Quoted 54 lines · Avi Kivity

Quoted 52 lines · Ingo Molnar

Quoted 26 lines · Avi Kivity

Quoted 15 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

Quoted 9 lines

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Quoted 7 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels? Perhaps a symbol server that perf can connect to (and that would connect to guests in turn)?

The highest quality solution would be if KVM offered a 'guest extension' to the guest kernel's /proc/kallsyms that made it easy for user-space to get this information from an authorative source.

That's the main reason why the host side /proc/kallsyms is so popular and so
useful: while in theory it's mostly redundant information which can be gleaned

from the System.map and other sources of symbol information, it's easily

available and is _always_ trustable to come from the host kernel.

Separate System.map's have a tendency to go out of sync (or go missing when a devel kernel gets rebuilt, or if a devel package is not installed), and server ports (be that a TCP port space server or an UDP port space mount-point) are both a configuration hassle and are not guest-transparent.

Quoted 6 lines

Quoted 4 lines

So for instrumentation infrastructure (such as perf) we have a large and well founded preference for intrinsic, built-in, kernel-provided information: i.e. a largely 'built-in' and transparent mechanism to get to guest symbols.

The symbol server's client can certainly access the bits through vmchannel.

Ok, that would work i suspect.

Would be nice to have the symbol server in tools/perf/ and also make it easy to add it to the initrd via a .config switch or so.

That would have basically all of the advantages of being built into the kernel (availability, configurability, transparency, hackability), while having all the advantages of a user-space approach as well (flexibility, extensibility, robustness, ease of maintenance, etc.).

Quoted 58 lines · Ingo Molnar

Quoted 47 lines · Avi Kivity

Quoted 46 lines · Ingo Molnar

Quoted 24 lines · Avi Kivity

Quoted 15 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

Quoted 8 lines

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Quoted 6 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels? Perhaps a symbol server that perf can connect to (and that would connect to guests in turn)?

The highest quality solution would be if KVM offered a 'guest extension' to the guest kernel's /proc/kallsyms that made it easy for user-space to get this information from an authorative source.

That's the main reason why the host side /proc/kallsyms is so popular and so
useful: while in theory it's mostly redundant information which can be gleaned from the System.map and other sources of symbol information, it's easily available and is _always_ trustable to come from the host kernel.

Separate System.map's have a tendency to go out of sync (or go missing when a devel kernel gets rebuilt, or if a devel package is not installed), and server ports (be that a TCP port space server or an UDP port space mount-point) are both a configuration hassle and are not guest-transparent.

Quoted 4 lines

Quoted 3 lines

So for instrumentation infrastructure (such as perf) we have a large and well founded preference for intrinsic, built-in, kernel-provided information: i.e. a largely 'built-in' and transparent mechanism to get to guest symbols.

The symbol server's client can certainly access the bits through vmchannel.

Ok, that would work i suspect.

Would be nice to have the symbol server in tools/perf/ and also make it easy to add it to the initrd via a .config switch or so.

Quoted 4 lines

That would have basically all of the advantages of being built into the kernel (availability, configurability, transparency, hackability), while having all the advantages of a user-space approach as well (flexibility, extensibility, robustness, ease of maintenance, etc.).

Note, I am not advocating building the vmchannel client into the host kernel. [...]

Quoted 13 lines

Quoted 13 lines

Quoted 3 lines

The symbol server's client can certainly access the bits through vmchannel.

Ok, that would work i suspect.

Would be nice to have the symbol server in tools/perf/ and also make it easy to add it to the initrd via a .config switch or so.

That would have basically all of the advantages of being built into the kernel (availability, configurability, transparency, hackability), while having all the advantages of a user-space approach as well (flexibility, extensibility, robustness, ease of maintenance, etc.).

Quoted 3 lines

Note, I am not advocating building the vmchannel client into the host kernel. [...]

Neither am i. What i suggested was a user-space binary/executable built in tools/perf and put into the initrd.

Quoted 11 lines · Ingo Molnar

Quoted 11 lines

Quoted 11 lines

Quoted 2 lines

The symbol server's client can certainly access the bits through vmchannel.

Ok, that would work i suspect.

Would be nice to have the symbol server in tools/perf/ and also make it easy to add it to the initrd via a .config switch or so.

That would have basically all of the advantages of being built into the kernel (availability, configurability, transparency, hackability), while having all the advantages of a user-space approach as well (flexibility, extensibility, robustness, ease of maintenance, etc.).

Quoted 4 lines

Quoted 2 lines

Note, I am not advocating building the vmchannel client into the host kernel. [...]

Neither am i. What i suggested was a user-space binary/executable built in tools/perf and put into the initrd.

I'm confused - initrd seems to be guest-side. I was talking about the host side.

Quoted 4 lines

I'm confused - initrd seems to be guest-side. I was talking about the host side.

host side doesnt need much support - just some client capability in perf itself. I suspect vmchannels are sufficiently flexible and configuration-free for such purposes? (i.e. like a filesystem in essence)

Quoted 6 lines · Ingo Molnar

Quoted 3 lines

I'm confused - initrd seems to be guest-side. I was talking about the host side.

host side doesnt need much support - just some client capability in perf itself. I suspect vmchannels are sufficiently flexible and configuration-free for such purposes? (i.e. like a filesystem in essence)

I haven't followed vmchannel closely, but I think it is. vmchannel is terminated in qemu on the host side, not in the host kernel. So perf would need to connect to qemu.

Quoted 5 lines

I haven't followed vmchannel closely, but I think it is. vmchannel is terminated in qemu on the host side, not in the host kernel. So perf would need to connect to qemu.

Hm, that sounds rather messy if we want to use it to basically expose kernel functionality in a guest/host unified way. Is the qemu process discoverable in some secure way?

Can we trust it?

Is there some proper tooling available to do
it, or do we have to push it through 2-3 packages to get such a useful feature done?

( That is the general thought process how many cross-discipline useful
desktop/server features hit the bit bucket before having had any chance of being vetted by users, and why Linux sucks so much when it comes to feature integration and application usability. )

Quoted 7 lines · Ingo Molnar

Quoted 4 lines

I haven't followed vmchannel closely, but I think it is. vmchannel is terminated in qemu on the host side, not in the host kernel. So perf would need to connect to qemu.

Hm, that sounds rather messy if we want to use it to basically expose kernel functionality in a guest/host unified way. Is the qemu process discoverable in some secure way?

We know its pid.

Quoted 1 line

Can we trust it?

No choice, it contains the guest address space.

Quoted 2 lines

Is there some proper tooling available to do it, or do we have to push it through 2-3 packages to get such a useful feature done?

libvirt manages qemu processes, but I don't think this should go through libvirt. qemu can do this directly by opening a unix domain socket in a well-known place.

Quoted 4 lines

( That is the general thought process how many cross-discipline useful
desktop/server features hit the bit bucket before having had any chance of being vetted by users, and why Linux sucks so much when it comes to feature integration and application usability. )

You can't solve everything in the kernel, even with a well populated tools/.

Quoted 7 lines · Avi Kivity

Quoted 5 lines · Ingo Molnar

Quoted 5 lines

I haven't followed vmchannel closely, but I think it is. vmchannel is terminated in qemu on the host side, not in the host kernel. So perf would need to connect to qemu.

Quoted 6 lines

Quoted 4 lines

Hm, that sounds rather messy if we want to use it to basically expose kernel functionality in a guest/host unified way. Is the qemu process discoverable in some secure way?

We know its pid.

How do i get a list of all 'guest instance PIDs',

and what is the way to talk
to Qemu?

Quoted 4 lines

Quoted 2 lines

Can we trust it?

No choice, it contains the guest address space.

I mean, i can trust a kernel service and i can trust /proc/kallsyms.

Can perf trust a random process claiming to be Qemu? What's the trust mechanism here?

Quoted 7 lines

Quoted 3 lines

Is there some proper tooling available to do it, or do we have to push it through 2-3 packages to get such a useful feature done?

libvirt manages qemu processes, but I don't think this should go through libvirt. qemu can do this directly by opening a unix domain socket in a well-known place.

So Qemu has never run into such problems before?

( Sounds weird - i think Qemu configuration itself should be done via a
unix domain socket driven configuration protocol as well. )

Quoted 7 lines

Quoted 5 lines

( That is the general thought process how many cross-discipline useful
desktop/server features hit the bit bucket before having had any chance of being vetted by users, and why Linux sucks so much when it comes to feature integration and application usability. )

You can't solve everything in the kernel, even with a well populated tools/.

Certainly not, but this is a technical problem in the kernel's domain, so it's a fair (and natural) expectation to be able to solve this within the kernel project.

Quoted 4 lines · Ingo Molnar

I mean, i can trust a kernel service and i can trust /proc/kallsyms.

Can perf trust a random process claiming to be Qemu? What's the trust mechanism here?

Obviously you can't trust anything you get from a guest, no matter how you get it.

Quoted 6 lines

Quoted 6 lines

I mean, i can trust a kernel service and i can trust /proc/kallsyms.

Can perf trust a random process claiming to be Qemu? What's the trust mechanism here?

Quoted 3 lines

Obviously you can't trust anything you get from a guest, no matter how you get it.

I'm not talking about the symbol strings and addresses, and the object contents for allocation (or debuginfo). I'm talking about the basic protocol of establishing which guest is which.

I.e. we really want to be able users to:

1) have it all working with a single guest, without having to specify 'which' guest (qemu PID) to work with. That is the dominant usecase both for developers and for a fair portion of testers.

2) Have some reasonable symbolic identification for guests. For example a usable approach would be to have 'perf kvm list', which would list all currently active guests:

$ perf kvm list
[1] Fedora
[2] OpenSuse
[3] Windows-XP
[4] Windows-7

And from that point on 'perf kvm -g OpenSuse record' would do the obvious thing. Users will be able to just use the 'OpenSuse' symbolic name for that guest, even if the guest got restarted and switched its main PID.

Any such facility needs trusted enumeration and a protocol where i can trust that the information i got is authorative. (I.e. 'OpenSuse' truly matches to the OpenSuse session - not to some local user starting up a Qemu instance that claims to be 'OpenSuse'.)

Is such a scheme possible/available? I suspect all the KVM configuration tools (i havent used them in some time - gui and command-line tools alike) use similar methods to ease guest management?

Quoted 10 lines · Ingo Molnar

Quoted 7 lines

Quoted 5 lines

I mean, i can trust a kernel service and i can trust /proc/kallsyms.

Can perf trust a random process claiming to be Qemu? What's the trust mechanism here?

Obviously you can't trust anything you get from a guest, no matter how you get it.

I'm not talking about the symbol strings and addresses, and the object contents for allocation (or debuginfo). I'm talking about the basic protocol of establishing which guest is which.

There is none. So far, qemu only dealt with managing just its own guest, and left all multiple guest management to higher levels up the stack (like libvirt).

Quoted 5 lines

I.e. we really want to be able users to:

1) have it all working with a single guest, without having to specify 'which' guest (qemu PID) to work with. That is the dominant usecase both for developers and for a fair portion of testers.

That's reasonable if we can get it working simply.

Quoted 14 lines

2) Have some reasonable symbolic identification for guests. For example a usable approach would be to have 'perf kvm list', which would list all currently active guests:

$ perf kvm list
[1] Fedora
[2] OpenSuse
[3] Windows-XP
[4] Windows-7

And from that point on 'perf kvm -g OpenSuse record' would do the obvious thing. Users will be able to just use the 'OpenSuse' symbolic name for that guest, even if the guest got restarted and switched its main PID.

Quoted 8 lines

Any such facility needs trusted enumeration and a protocol where i can trust that the information i got is authorative. (I.e. 'OpenSuse' truly matches to the OpenSuse session - not to some local user starting up a Qemu instance that claims to be 'OpenSuse'.)

Is such a scheme possible/available? I suspect all the KVM configuration tools (i havent used them in some time - gui and command-line tools alike) use similar methods to ease guest management?

You can do that through libvirt, but that only works for guests started through libvirt. libvirt provides command-line tools to list and manage guests (for example autostarting them on startup), and tools built on top of libvirt can manage guests graphically.

Looks like we have a layer inversion here. Maybe we need a plugin system - libvirt drops a .so into perf that teaches it how to list guests and get their symbols.

Quoted 6 lines

You can do that through libvirt, but that only works for guests started through libvirt. libvirt provides command-line tools to list and manage guests (for example autostarting them on startup), and tools built on top of libvirt can manage guests graphically.

Quoted 4 lines

Looks like we have a layer inversion here. Maybe we need a plugin system - libvirt drops a .so into perf that teaches it how to list guests and get their symbols.

Is libvirt used to start up all KVM guests? If not, if it's only used on some distros while other distros have other solutions then there's apparently no good way to get to such information, and the kernel bits of KVM do not provide it.

To the user (and to me) this looks like a KVM bug / missing feature. (and the user doesnt care where the blame is) If that is true then apparently the current KVM design has no technically actionable solution for certain categories of features!

[...]

Quoted 6 lines

Quoted 5 lines

I.e. we really want to be able users to:

1) have it all working with a single guest, without having to specify 'which' guest (qemu PID) to work with. That is the dominant usecase both for developers and for a fair portion of testers.

Quoted 1 line

That's reasonable if we can get it working simply.

IMO such ease of use is reasonable and required, full stop.
If it cannot be gotten simply then that's a bug: either in the code, or in the design, or in the development process that led to the design. Bugs need fixing. [...]

Quoted 7 lines · Ingo Molnar

[...]

Quoted 6 lines

Quoted 5 lines

I.e. we really want to be able users to:

1) have it all working with a single guest, without having to specify 'which' guest (qemu PID) to work with. That is the dominant usecase both for developers and for a fair portion of testers.

Quoted 6 lines

Quoted 1 line

That's reasonable if we can get it working simply.

IMO such ease of use is reasonable and required, full stop.
If it cannot be gotten simply then that's a bug: either in the code, or in the design, or in the development process that led to the design. Bugs need fixing. [...]

Perhaps the fact that kvm happens to deal with an interesting application area (virtualization) is misleading here. As far as the host kernel or other host userspace is concerned, qemu is just some random unprivileged userspace program (with some *optional* /dev/kvm services that might happen to require temporary root).

As such, perf trying to instrument qemu is no different than perf trying to instrument any other userspace widget. Therefore, expecting 'trusted enumeration' of instances is just as sensible as using 'trusted ps' and 'trusted /var/run/FOO.pid files'.

[...]

Quoted 4 lines

Perhaps the fact that kvm happens to deal with an interesting application area (virtualization) is misleading here. As far as the host kernel or other host userspace is concerned, qemu is just some random unprivileged userspace program [...]

You are quite mistaken: KVM isnt really a 'random unprivileged application' in this context, it is clearly an extension of system/kernel services.

( Which can be seen from the simple fact that what started the discussion was 'how do we get /proc/kallsyms from the guest'. I.e. an extension of the existing host-space /proc/kallsyms was desired. )

Hi -

Quoted 5 lines · Ingo Molnar

[...]

Quoted 4 lines

Perhaps the fact that kvm happens to deal with an interesting application area (virtualization) is misleading here. As far as the host kernel or other host userspace is concerned, qemu is just some random unprivileged userspace program [...]

Quoted 3 lines

You are quite mistaken: KVM isnt really a 'random unprivileged application' in this context, it is clearly an extension of system/kernel services.

I don't know what "extension of system/kernel services" means in this context, beyond something running on the system/kernel, like every other process. [...]

Quoted 5 lines

[...]

( Which can be seen from the simple fact that what started the
discussion was 'how do we get /proc/kallsyms from the guest'. I.e. an extension of the existing host-space /proc/kallsyms was desired. )

(Sorry, that smacks of circular reasoning.)

Quoted 11 lines · Avi Kivity

Quoted 5 lines · Ingo Molnar

I mean, i can trust a kernel service and i can trust /proc/kallsyms.

Can perf trust a random process claiming to be Qemu? What's the trust mechanism here?

Obviously you can't trust anything you get from a guest, no matter how you get it.

I'm not talking about the symbol strings and addresses, and the object contents for allocation (or debuginfo). I'm talking about the basic protocol of establishing which guest is which.

I.e. we really want to be able users to:

1) have it all working with a single guest, without having to specify 'which' guest (qemu PID) to work with. That is the dominant usecase both for developers and for a fair portion of testers.

2) Have some reasonable symbolic identification for guests. For example a usable approach would be to have 'perf kvm list', which would list all currently active guests:

$ perf kvm list
[1] Fedora
[2] OpenSuse
[3] Windows-XP
[4] Windows-7

And from that point on 'perf kvm -g OpenSuse record' would do the obvious thing. Users will be able to just use the 'OpenSuse' symbolic name for that guest, even if the guest got restarted and switched its main PID.

Is such a scheme possible/available? I suspect all the KVM configuration tools (i havent used them in some time - gui and command-line tools alike) use similar methods to ease guest management?

Ingo
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

You are quite mistaken: KVM isnt really a 'random unprivileged application' in this context, it is clearly an extension of system/kernel services.

( Which can be seen from the simple fact that what started the discussion was
'how do we get /proc/kallsyms from the guest'. I.e. an extension of the existing host-space /proc/kallsyms was desired. )

In that sense the most natural 'extension' would be the solution i mentioned a week or two ago: to have a (read only) mount of all guest filesystems, plus a channel for profiling/tracing data. That would make symbol parsing easier and it's what extends the existing 'host space' abstraction in the most natural way.

( It doesnt even have to be done via the kernel - Qemu could implement that
via FUSE for example. )

Quoted 13 lines · Ingo Molnar

Quoted 7 lines · Avi Kivity

Quoted 4 lines · Ingo Molnar

I mean, i can trust a kernel service and i can trust /proc/kallsyms.

Can perf trust a random process claiming to be Qemu? What's the trust mechanism here?

Obviously you can't trust anything you get from a guest, no matter how you get it.

I'm not talking about the symbol strings and addresses, and the object contents for allocation (or debuginfo). I'm talking about the basic protocol of establishing which guest is which.

Quoted 5 lines

I.e. we really want to be able users to:

1) have it all working with a single guest, without having to specify 'which' guest (qemu PID) to work with. That is the dominant usecase both for developers and for a fair portion of testers.

You're making too many assumptions.

There is no list of guests anymore than there is a list of web browsers.

You can have a multi-tenant scenario where you have distinct groups of virtual machines running as unprivileged users.

Quoted 10 lines

2) Have some reasonable symbolic identification for guests. For example a usable approach would be to have 'perf kvm list', which would list all currently active guests:

$ perf kvm list
[1] Fedora
[2] OpenSuse
[3] Windows-XP
[4] Windows-7

Quoted 3 lines

And from that point on 'perf kvm -g OpenSuse record' would do the obvious thing. Users will be able to just use the 'OpenSuse' symbolic name for that guest, even if the guest got restarted and switched its main PID.

Does "perf kvm list" always run as root? What if two unprivileged users both have a VM named "Fedora"?

If we look at the use-case, it's going to be something like, a user is creating virtual machines and wants to get performance information about them.

Having to run a separate tool like perf is not going to be what they would expect they had to do. Instead, they would either use their existing GUI tool (like virt-manager) or they would use their management interface (either QMP or libvirt).

The complexity of interaction is due to the fact that perf shouldn't be a stand alone tool. It should be a library or something with a programmatic interface that another tool can make use of.

Quoted 6 lines · Ingo Molnar

You are quite mistaken: KVM isnt really a 'random unprivileged application' in this context, it is clearly an extension of system/kernel services.

( Which can be seen from the simple fact that what started the discussion was
'how do we get /proc/kallsyms from the guest'. I.e. an extension of the existing host-space /proc/kallsyms was desired. )

Random tools (like perf) should not be able to do what you describe. It's a security nightmare.

If it's desirable to have /proc/kallsyms available, we can expose an interface in QEMU to provide that. That can then be plumbed through libvirt and QMP.

Then a management tool can use libvirt or QMP to obtain that information and interact with the kernel appropriately.

Quoted 6 lines

In that sense the most natural 'extension' would be the solution i mentioned a week or two ago: to have a (read only) mount of all guest filesystems, plus a channel for profiling/tracing data. That would make symbol parsing easier and it's what extends the existing 'host space' abstraction in the most natural way.

Quoted 2 lines

( It doesnt even have to be done via the kernel - Qemu could implement that
via FUSE for example. )

No way. The guest has sensitive data and exposing it widely on the host is a bad thing to do. [...]

Quoted 9 lines · Anthony Liguori

Quoted 7 lines · Ingo Molnar

You are quite mistaken: KVM isnt really a 'random unprivileged application' in this context, it is clearly an extension of system/kernel services.

( Which can be seen from the simple fact that what started the discussion was
'how do we get /proc/kallsyms from the guest'. I.e. an extension of the existing host-space /proc/kallsyms was desired. )

Quoted 3 lines

Random tools (like perf) should not be able to do what you describe. It's a security nightmare.

A security nightmare exactly how? Mind to go into details as i dont understand your point.

Quoted 4 lines

No way. The guest has sensitive data and exposing it widely on the host is a bad thing to do. [...]

Firstly, you are putting words into my mouth, as i said nothing about 'exposing it widely'. I suggest exposing it under the privileges of whoever has access to the guest image.

Secondly, regarding confidentiality, and this is guest security 101: whoever can access the image on the host _already_ has access to all the guest data!

A Linux image can generally be loopback mounted straight away:

losetup -o 32256 /dev/loop0 ./guest-image.img
mount -o ro /dev/loop0 /mnt-guest

(Or, if you are an unprivileged user who cannot mount, it can be read via ext2 tools.)

There's nothing the guest can do about that. The host is in total control of guest image data for heaven's sake!

All i'm suggesting is to make what is already possible more convenient.

Ingo

Quoted 6 lines · Ingo Molnar

Quoted 4 lines · Anthony Liguori

Quoted 3 lines · Ingo Molnar

You are quite mistaken: KVM isnt really a 'random unprivileged application' in this context, it is clearly an extension of system/kernel services.

Quoted 7 lines

Quoted 5 lines

Quoted 3 lines

( Which can be seen from the simple fact that what started the discussion was
'how do we get /proc/kallsyms from the guest'. I.e. an extension of the existing host-space /proc/kallsyms was desired. )

Random tools (like perf) should not be able to do what you describe. It's a security nightmare.

A security nightmare exactly how? Mind to go into details as i dont understand your point.

Assume you're using SELinux to implement mandatory access control. How do you label this file system?

Generally speaking, we don't know the difference between /proc/kallsyms vs. /dev/mem if we do generic passthrough. While it might be safe to have a relaxed label of kallsyms (since it's read only), it's clearly not safe to do that for /dev/mem, /etc/shadow, or any file containing sensitive information.

Rather, we ought to expose a higher level interface that we have more confidence in with respect to understanding the ramifications of exposing that guest data.

Quoted 6 lines

Quoted 2 lines

No way. The guest has sensitive data and exposing it widely on the host is a bad thing to do. [...]

Firstly, you are putting words into my mouth, as i said nothing about 'exposing it widely'. I suggest exposing it under the privileges of whoever has access to the guest image.

That doesn't work as nicely with SELinux.

It's completely reasonable to have a user that can interact in a read only mode with a VM via libvirt but cannot read the guest's disk images or the guest's memory contents.

Quoted 8 lines

Secondly, regarding confidentiality, and this is guest security 101: whoever can access the image on the host _already_ has access to all the guest data!

A Linux image can generally be loopback mounted straight away:

losetup -o 32256 /dev/loop0 ./guest-image.img
mount -o ro /dev/loop0 /mnt-guest

Quoted 5 lines

(Or, if you are an unprivileged user who cannot mount, it can be read via ext2 tools.)

There's nothing the guest can do about that. The host is in total control of guest image data for heaven's sake!

It's not that simple in a MAC environment.

Hm, that sounds rather messy if we want to use it to basically expose kernel functionality in a guest/host unified way. Is the qemu process discoverable in some secure way? Can we trust it? Is there some proper tooling available to do it, or do we have to push it through 2-3 packages to get such a useful feature done?

Quoted 5 lines · Ingo Molnar

Hm, that sounds rather messy if we want to use it to basically expose kernel functionality in a guest/host unified way. Is the qemu process discoverable in some secure way? Can we trust it? Is there some proper tooling available to do it, or do we have to push it through 2-3 packages to get such a useful feature done?

Since we want to implement a pmu usable for the guest anyway why we don't just use a guests perf to get all information we want? If we get a pmu-nmi from the guest we just re-inject it to the guest and perf in the guest gives us all information we wand including kernel and userspace symbols, stack traces, and so on.

In the previous thread we discussed about a direct trace channel between guest and host kernel (which can be used for ftrace events for example). This channel could be used to transport this information to the host kernel.

The only additional feature needed is a way for the host to start a perf instance in the guest.

Opinions?

Joerg

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 23 lines · Anthony Liguori

Quoted 12 lines · Ingo Molnar

Quoted 9 lines · Anthony Liguori

Quoted 7 lines · Ingo Molnar

You are quite mistaken: KVM isnt really a 'random unprivileged application' in this context, it is clearly an extension of system/kernel services.

( Which can be seen from the simple fact that what started the discussion was
'how do we get /proc/kallsyms from the guest'. I.e. an extension of the existing host-space /proc/kallsyms was desired. )

Quoted 6 lines

Quoted 3 lines

Random tools (like perf) should not be able to do what you describe. It's a security nightmare.

A security nightmare exactly how? Mind to go into details as i dont understand your point.

Assume you're using SELinux to implement mandatory access control. How do you label this file system?

Quoted 6 lines

Generally speaking, we don't know the difference between /proc/kallsyms vs. /dev/mem if we do generic passthrough. While it might be safe to have a relaxed label of kallsyms (since it's read only), it's clearly not safe to do that for /dev/mem, /etc/shadow, or any file containing sensitive information.

What's your _point_? Please outline a threat model, a vector of attack, _anything_ that substantiates your "it's a security nightmare" claim.

Quoted 4 lines

Rather, we ought to expose a higher level interface that we have more confidence in with respect to understanding the ramifications of exposing that guest data.

Exactly, we want something that has a flexible namespace and works well with Linux tools in general. Preferably that namespace should be human readable, and it should be hierarchic, and it should have a well-known permission model.

This concept exists in Linux and is generally called a 'filesystem'.

If a user cannot read the image file then the user has no access to its contents via other namespaces either. That is, of course, a basic security aspect.

( That is perfectly true with a non-SELinux Unix permission model as well, and
is true in the SELinux case as well. )

Erm. Please explain to me, what exactly is 'not that simple' in a MAC environment?

Also, i'd like to note that the 'restrictive SELinux setups' usecases are pretty secondary.

To demonstrate that, i'd like every KVM developer on this list who reads this mail and who has their home development system where they produce their patches set up in a restrictive MAC environment, in that you cannot even read the images you are using, to chime in with a "I'm doing that" reply.

Quoted 4 lines

If we look at the use-case, it's going to be something like, a user is creating virtual machines and wants to get performance information about them.

Quoted 9 lines

Having to run a separate tool like perf is not going to be what they would expect they had to do. Instead, they would either use their existing GUI tool (like virt-manager) or they would use their management interface (either QMP or libvirt).

The complexity of interaction is due to the fact that perf shouldn't be a stand alone tool. It should be a library or something with a programmatic interface that another tool can make use of.

But ... a GUI interface/integration is of course possible too, and it's being worked on.

perf is mainly a kernel developer tool, and kernel developers generally dont use GUIs to do their stuff: which is the (sole) reason why its first ~850 commits of tools/perf/ were done without a GUI. We go where our developers are.

In any case it's not an excuse to have no proper command-line tooling. In fact if you cannot get simpler, more atomic command-line tooling right then you'll probably doubly suck at doing a GUI as well.

Ingo

[...]
The only way to really address this is to change the interaction. Instead of running perf externally to qemu, we should support a perf command in the qemu monitor that can then tie directly to the perf tooling. That gives us the best possible user experience.

Quoted 6 lines · Yanmin

Quoted 6 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels?

Quoted 4 lines

With the patch, 'perf kvm report --sort pid" could show
summary statistics for all guest os instances. Then, use
parameter --pid of 'perf kvm record' to collect single problematic instance data.

That certainly works, though automatic association of guest data with guest symbols is friendlier.

Quoted 31 lines

Quoted 30 lines

Quoted 29 lines

diff -Nraup linux-2.6_tipmaster0315/arch/x86/kvm/vmx.c linux-2.6_tipmaster0315_perfkvm/arch/x86/kvm/vmx.c

arch/x86/kvm/vmx.c +12 -0

@@ -26,6 +26,7 @@
   #include<linux/sched.h>
   #include<linux/moduleparam.h>
   #include<linux/ftrace_event.h>
+#include<linux/perf_event.h>
   #include "kvm_cache_regs.h"
   #include "x86.h"
 

@@ -3632,6 +3633,43 @@ static void update_cr8_intercept(struct
   	vmcs_write32(TPR_THRESHOLD, irr);
   }
 
+DEFINE_PER_CPU(int, kvm_in_guest) = {0};
+
+static void kvm_set_in_guest(void)
+{
+	percpu_write(kvm_in_guest, 1);
+}
+
+static int kvm_is_in_guest(void)
+{
+	return percpu_read(kvm_in_guest);
+}
 
 

Quoted 6 lines

Quoted 2 lines

There is already PF_VCPU for this.

Right, but there is a scope between kvm_guest_enter and really running in guest os, where a perf event might overflow. Anyway, the scope is very narrow, I will change it to use flag PF_VCPU.

There is also a window between setting the flag and calling 'int $2' where an NMI might happen and be accounted incorrectly.

Perhaps separate the 'int $2' into a direct call into perf and another call for the rest of NMI handling. I don't see how it would work on svm though - AFAICT the NMI is held whereas vmx swallows it.

I guess NMIs
will be disabled until the next IRET so it isn't racy, just tricky.

Quoted 11 lines

Quoted 11 lines

arch/x86/kvm/vmx.c static void update_cr8_intercept(struct

Quoted 9 lines

arch/x86/kvm/vmx.c +6 -0

+static struct perf_guest_info_callbacks kvm_guest_cbs = {
+	.is_in_guest 		= kvm_is_in_guest,
+	.is_user_mode		= kvm_is_user_mode,
+	.get_guest_ip		= kvm_get_guest_ip,
+	.reset_in_guest		= kvm_reset_in_guest,
+};
 

Should be in common code, not vmx specific.

Quoted 4 lines

Right. I discussed with Yangsheng. I will move above data structures and callbacks to file arch/x86/kvm/x86.c, and add get_ip, a new callback to kvm_x86_ops.

You will need access to the vcpu pointer (kvm_rip_read() needs it), you can put it in a percpu variable.

I guess if it's not null, you know
you're in a guest, so no need for PF_VCPU.

Hi -

Quoted 6 lines · Anthony Liguori

[...]
The only way to really address this is to change the interaction. Instead of running perf externally to qemu, we should support a perf command in the qemu monitor that can then tie directly to the perf tooling. That gives us the best possible user experience.

To what extent could this be solved with less crossing of isolation/abstraction layers, if the perfctr facilities were properly virtualized?

Quoted 5 lines · Ingo Molnar

Hm, that sounds rather messy if we want to use it to basically expose kernel functionality in a guest/host unified way. Is the qemu process discoverable in some secure way? Can we trust it? Is there some proper tooling available to do it, or do we have to push it through 2-3 packages to get such a useful feature done?

Since we want to implement a pmu usable for the guest anyway why we don't just use a guests perf to get all information we want? [...]

Quoted 15 lines · Ingo Molnar

Quoted 4 lines

If we look at the use-case, it's going to be something like, a user is creating virtual machines and wants to get performance information about them.

Quoted 8 lines

Having to run a separate tool like perf is not going to be what they would expect they had to do. Instead, they would either use their existing GUI tool (like virt-manager) or they would use their management interface (either QMP or libvirt).

The complexity of interaction is due to the fact that perf shouldn't be a stand alone tool. It should be a library or something with a programmatic interface that another tool can make use of.

But ... a GUI interface/integration is of course possible too, and it's being worked on.

Quoted 8 lines

perf is mainly a kernel developer tool, and kernel developers generally dont use GUIs to do their stuff: which is the (sole) reason why its first ~850 commits of tools/perf/ were done without a GUI. We go where our developers are.

In any case it's not an excuse to have no proper command-line tooling. In fact if you cannot get simpler, more atomic command-line tooling right then you'll probably doubly suck at doing a GUI as well.

It's about who owns the user interface.

If qemu owns the user interface, than we can satisfy this in a very simple way by adding a perf monitor command. If we have to support third party tools, then it significantly complicates things.

Hi -

Quoted 5 lines · Anthony Liguori

[...]
The only way to really address this is to change the interaction. Instead of running perf externally to qemu, we should support a perf command in the qemu monitor that can then tie directly to the perf tooling. That gives us the best possible user experience.

To what extent could this be solved with less crossing of isolation/abstraction layers, if the perfctr facilities were properly virtualized? [...]

Monitoring guests from the host is useful for kvm developers, but less so for users.

Quoted 3 lines · Avi Kivity

Monitoring guests from the host is useful for kvm developers, but less so for users.

Guest space profiling is easy, and 'perf kvm' is not about that. (plain 'perf' will work if a proper paravirt channel is opened to the host)

I think you might have misunderstood the purpose and role of the 'perf kvm' patch here? 'perf kvm' is aimed at KVM developers: it is them who improve KVM code, not guest kernel users.

If you want to use a synthetic filesystem as the management interface for qemu, that's one thing. But you suggested exposing the guest filesystem in its entirely and that's what I disagreed with.

The guest may encrypt it's disk image. It still ought to be possible to run perf against that guest, no?

Quoted 8 lines · Ingo Molnar

Quoted 2 lines · Avi Kivity

Monitoring guests from the host is useful for kvm developers, but less so for users.

Guest space profiling is easy, and 'perf kvm' is not about that. (plain 'perf' will work if a proper paravirt channel is opened to the host)

Quoted 3 lines

I think you might have misunderstood the purpose and role of the 'perf kvm' patch here? 'perf kvm' is aimed at KVM developers: it is them who improve KVM code, not guest kernel users.

Of course I understood it. My point was that 'perf kvm' serves a tiny minority of users. [...]

[...] That doesn't mean it isn't useful, just that it doesn't satisfy all needs by itself.

Quoted 45 lines · Zhang, Yanmin

Quoted 29 lines · Yanmin

Quoted 19 lines · Avi Kivity

Quoted 18 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Quoted 7 lines

Quoted 6 lines

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Thanks for your kind comments.

Quoted 7 lines

Quoted 4 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels?

With the patch, 'perf kvm report --sort pid" could show
summary statistics for all guest os instances. Then, use
parameter --pid of 'perf kvm record' to collect single problematic instance data.

Sorry. I found currently --pid isn't process but a thread (main thread).

Ingo,

Is it possible to support a new parameter or extend --inherit, so 'perf record' and 'perf top' could collect data on all threads of a process when the process is running?

Quoted 2 lines

If not, I need add a new ugly parameter which is similar to --pid to filter out process data in userspace.

Yeah. For maximum utility i'd suggest to extend --pid to include this, and introduce --tid for the previous, limited-to-a-single-task functionality.

Most users would expect --pid to work like a 'late attach' - i.e. to work like strace -f or like a gdb attach.

Quoted 12 lines · Avi Kivity

Quoted 3 lines · Yanmin

Right, but there is a scope between kvm_guest_enter and really running in guest os, where a perf event might overflow. Anyway, the scope is very narrow, I will change it to use flag PF_VCPU.

There is also a window between setting the flag and calling 'int $2' where an NMI might happen and be accounted incorrectly.

Perhaps separate the 'int $2' into a direct call into perf and another call for the rest of NMI handling. I don't see how it would work on svm though - AFAICT the NMI is held whereas vmx swallows it.

Quoted 2 lines

I guess NMIs
will be disabled until the next IRET so it isn't racy, just tricky.

I'm not sure if vmexit does break NMI context or not. Hardware NMI context isn't reentrant till a IRET. YangSheng would like to double check it.

Quoted 4 lines

I'm not sure if vmexit does break NMI context or not. Hardware NMI context isn't reentrant till a IRET. YangSheng would like to double check it.

After more check, I think VMX won't remained NMI block state for host. That's means, if NMI happened and processor is in VMX non-root mode, it would only result in VMExit, with a reason indicate that it's due to NMI happened, but no more state change in the host.

So in that meaning, there _is_ a window between VMExit and KVM handle the NMI. Moreover, I think we _can't_ stop the re-entrance of NMI handling code because "int $2" don't have effect to block following NMI.

And if the NMI sequence is not important(I think so), then we need to generate a real NMI in current vmexit-after code. Seems let APIC send a NMI IPI to itself is a good idea.

I am debugging a patch based on apic->send_IPI_self(NMI_VECTOR) to replace "int $2". Something unexpected is happening...

Quoted 9 lines · Sheng Yang

Quoted 3 lines

I'm not sure if vmexit does break NMI context or not. Hardware NMI context isn't reentrant till a IRET. YangSheng would like to double check it.

After more check, I think VMX won't remained NMI block state for host. That's means, if NMI happened and processor is in VMX non-root mode, it would only result in VMExit, with a reason indicate that it's due to NMI happened, but no more state change in the host.

Quoted 3 lines

So in that meaning, there _is_ a window between VMExit and KVM handle the NMI. Moreover, I think we _can't_ stop the re-entrance of NMI handling code because "int $2" don't have effect to block following NMI.

That's pretty bad, as NMI runs on a separate stack (via IST). So if another NMI happens while our int $2 is running, the stack will be corrupted.

Quoted 7 lines

And if the NMI sequence is not important(I think so), then we need to generate a real NMI in current vmexit-after code. Seems let APIC send a NMI IPI to itself is a good idea.

I am debugging a patch based on apic->send_IPI_self(NMI_VECTOR) to replace "int $2". Something unexpected is happening...

I think you need DM_NMI for that to work correctly.

An alternative is to call the NMI handler directly.

Quoted 5 lines

I think you need DM_NMI for that to work correctly.

An alternative is to call the NMI handler directly.

apic_send_IPI_self() already took care of APIC_DM_NMI.

And NMI handler would block the following NMI?

Quoted 15 lines · Yanmin

Quoted 13 lines · Avi Kivity

Quoted 4 lines · Yanmin

Right, but there is a scope between kvm_guest_enter and really running in guest os, where a perf event might overflow. Anyway, the scope is very narrow, I will change it to use flag PF_VCPU.

There is also a window between setting the flag and calling 'int $2' where an NMI might happen and be accounted incorrectly.

Perhaps separate the 'int $2' into a direct call into perf and another call for the rest of NMI handling. I don't see how it would work on svm though - AFAICT the NMI is held whereas vmx swallows it.

Quoted 6 lines

Quoted 3 lines

I guess NMIs
will be disabled until the next IRET so it isn't racy, just tricky.

I'm not sure if vmexit does break NMI context or not. Hardware NMI context isn't reentrant till a IRET. YangSheng would like to double check it.

After more check, I think VMX won't remained NMI block state for host. That's means, if NMI happened and processor is in VMX non-root mode, it would only result in VMExit, with a reason indicate that it's due to NMI happened, but no more state change in the host.

So in that meaning, there _is_ a window between VMExit and KVM handle the NMI. Moreover, I think we _can't_ stop the re-entrance of NMI handling code because "int $2" don't have effect to block following NMI.

And if the NMI sequence is not important(I think so), then we need to generate a real NMI in current vmexit-after code. Seems let APIC send a NMI IPI to itself is a good idea.

I am debugging a patch based on apic->send_IPI_self(NMI_VECTOR) to replace "int $2". Something unexpected is happening...

Quoted 34 lines · Sheng Yang

Quoted 13 lines · Yanmin

Quoted 12 lines · Avi Kivity

Quoted 3 lines · Yanmin

Right, but there is a scope between kvm_guest_enter and really running in guest os, where a perf event might overflow. Anyway, the scope is very narrow, I will change it to use flag PF_VCPU.

There is also a window between setting the flag and calling 'int $2' where an NMI might happen and be accounted incorrectly.

Perhaps separate the 'int $2' into a direct call into perf and another call for the rest of NMI handling. I don't see how it would work on svm though - AFAICT the NMI is held whereas vmx swallows it.

Quoted 6 lines

Quoted 2 lines

I guess NMIs
will be disabled until the next IRET so it isn't racy, just tricky.

I'm not sure if vmexit does break NMI context or not. Hardware NMI context isn't reentrant till a IRET. YangSheng would like to double check it.

After more check, I think VMX won't remained NMI block state for host. That's means, if NMI happened and processor is in VMX non-root mode, it would only result in VMExit, with a reason indicate that it's due to NMI happened, but no more state change in the host.

So in that meaning, there _is_ a window between VMExit and KVM handle the NMI. Moreover, I think we _can't_ stop the re-entrance of NMI handling code because "int $2" don't have effect to block following NMI.

And if the NMI sequence is not important(I think so), then we need to generate a real NMI in current vmexit-after code. Seems let APIC send a NMI IPI to itself is a good idea.

Quoted 2 lines

I am debugging a patch based on apic->send_IPI_self(NMI_VECTOR) to replace "int $2". Something unexpected is happening...

You can't use the APIC to send vectors 0x00-0x1f, or at least, aren't supposed to be able to.

Quoted 47 lines · Ingo Molnar

Quoted 45 lines · Zhang, Yanmin

Quoted 29 lines · Yanmin

Quoted 19 lines · Avi Kivity

Quoted 18 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Quoted 7 lines

Quoted 6 lines

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Thanks for your kind comments.

Quoted 7 lines

Quoted 4 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels?

With the patch, 'perf kvm report --sort pid" could show
summary statistics for all guest os instances. Then, use
parameter --pid of 'perf kvm record' to collect single problematic instance data.

Sorry. I found currently --pid isn't process but a thread (main thread).

Ingo,

Is it possible to support a new parameter or extend --inherit, so 'perf record' and 'perf top' could collect data on all threads of a process when the process is running?

Quoted 8 lines

Quoted 2 lines

If not, I need add a new ugly parameter which is similar to --pid to filter out process data in userspace.

Yeah. For maximum utility i'd suggest to extend --pid to include this, and introduce --tid for the previous, limited-to-a-single-task functionality.

Most users would expect --pid to work like a 'late attach' - i.e. to work like strace -f or like a gdb attach.

Thanks Ingo, Avi.

I worked out below patch against tip/master of March 15th.

Subject: [PATCH] Change perf's parameter --pid to process-wide collection
From: Zhang, Yanmin <[email protected]>

Change parameter -p (--pid) to real process pid and add -t (--tid) meaning thread id. Now, --pid means perf collects the statistics of all threads of the process, while --tid means perf just collect the statistics of that thread.

BTW, the patch fixes a bug of 'perf stat -p'. 'perf stat' always configures attr->disabled=1 if it isn't a system-wide collection. If there is a '-p' and no forks, 'perf stat -p' doesn't collect any data. In addition, the while(!done) in run_perf_stat consumes 100% single cpu time which has bad impact on running workload. I added a sleep(1) in the loop.

Signed-off-by Zhang Yanmin <[email protected]>

Quoted 42 lines · On Thursday 18 March 2010 05:14:52 Zachary Amsden wrote:

Quoted 40 lines · Sheng Yang

Quoted 15 lines · Yanmin

Quoted 13 lines · Avi Kivity

Quoted 4 lines · Yanmin

Right, but there is a scope between kvm_guest_enter and really running in guest os, where a perf event might overflow. Anyway, the scope is very narrow, I will change it to use flag PF_VCPU.

There is also a window between setting the flag and calling 'int $2' where an NMI might happen and be accounted incorrectly.

Perhaps separate the 'int $2' into a direct call into perf and another call for the rest of NMI handling. I don't see how it would work on svm though - AFAICT the NMI is held whereas vmx swallows it.

Quoted 7 lines

Quoted 3 lines

I guess NMIs
will be disabled until the next IRET so it isn't racy, just tricky.

I'm not sure if vmexit does break NMI context or not. Hardware NMI context isn't reentrant till a IRET. YangSheng would like to double check it.

After more check, I think VMX won't remained NMI block state for host. That's means, if NMI happened and processor is in VMX non-root mode, it would only result in VMExit, with a reason indicate that it's due to NMI happened, but no more state change in the host.

So in that meaning, there _is_ a window between VMExit and KVM handle the NMI. Moreover, I think we _can't_ stop the re-entrance of NMI handling code because "int $2" don't have effect to block following NMI.

And if the NMI sequence is not important(I think so), then we need to generate a real NMI in current vmexit-after code. Seems let APIC send a NMI IPI to itself is a good idea.

I am debugging a patch based on apic->send_IPI_self(NMI_VECTOR) to replace "int $2". Something unexpected is happening...

Quoted 3 lines

You can't use the APIC to send vectors 0x00-0x1f, or at least, aren't supposed to be able to.

Um? Why?

Especially kernel is already using it to deliver NMI.

Quoted 40 lines · Sheng Yang

Quoted 39 lines · On Thursday 18 March 2010 05:14:52 Zachary Amsden wrote:

Quoted 37 lines · Sheng Yang

Quoted 13 lines · Yanmin

Quoted 12 lines · Avi Kivity

Quoted 3 lines · Yanmin

Right, but there is a scope between kvm_guest_enter and really running in guest os, where a perf event might overflow. Anyway, the scope is very narrow, I will change it to use flag PF_VCPU.

There is also a window between setting the flag and calling 'int $2' where an NMI might happen and be accounted incorrectly.

Perhaps separate the 'int $2' into a direct call into perf and another call for the rest of NMI handling. I don't see how it would work on svm though - AFAICT the NMI is held whereas vmx swallows it.

Quoted 6 lines

Quoted 2 lines

I guess NMIs
will be disabled until the next IRET so it isn't racy, just tricky.

I'm not sure if vmexit does break NMI context or not. Hardware NMI context isn't reentrant till a IRET. YangSheng would like to double check it.

After more check, I think VMX won't remained NMI block state for host. That's means, if NMI happened and processor is in VMX non-root mode, it would only result in VMExit, with a reason indicate that it's due to NMI happened, but no more state change in the host.

So in that meaning, there _is_ a window between VMExit and KVM handle the NMI. Moreover, I think we _can't_ stop the re-entrance of NMI handling code because "int $2" don't have effect to block following NMI.

And if the NMI sequence is not important(I think so), then we need to generate a real NMI in current vmexit-after code. Seems let APIC send a NMI IPI to itself is a good idea.

I am debugging a patch based on apic->send_IPI_self(NMI_VECTOR) to replace "int $2". Something unexpected is happening...

Quoted 6 lines

Quoted 2 lines

You can't use the APIC to send vectors 0x00-0x1f, or at least, aren't supposed to be able to.

Um? Why?

Especially kernel is already using it to deliver NMI.

That's the only defined case, and it is defined because the vector field is ignore for DM_NMI. Vol 3A (exact section numbers may vary depending on your version).

8.5.1 / 8.6.1

'100 (NMI) Delivers an NMI interrupt to the target processor or processors. The vector information is ignored'

8.5.2 Valid Interrupt Vectors

'Local and I/O APICs support 240 of these vectors (in the range of 16 to
255) as valid interrupts.'

8.8.4 Interrupt Acceptance for Fixed Interrupts

'...; vectors 0 through 15 are reserved by the APIC (see also: Section 8.5.2, "Valid Interrupt Vectors")'

So I misremembered, apparently you can deliver interrupts 0x10-0x1f, but vectors 0x00-0x0f are not valid to send via APIC or I/O APIC.

Hi -

Quoted 6 lines · Anthony Liguori

[...]
The only way to really address this is to change the interaction. Instead of running perf externally to qemu, we should support a perf command in the qemu monitor that can then tie directly to the perf tooling. That gives us the best possible user experience.

To what extent could this be solved with less crossing of isolation/abstraction layers, if the perfctr facilities were properly virtualized?

Quoted 63 lines · On Thursday 18 March 2010 12:50:58 Zachary Amsden wrote:

Quoted 41 lines · Sheng Yang

Quoted 40 lines · On Thursday 18 March 2010 05:14:52 Zachary Amsden wrote:

Quoted 38 lines · Sheng Yang

Quoted 14 lines · Yanmin

Quoted 13 lines · Avi Kivity

Quoted 3 lines · Yanmin

Right, but there is a scope between kvm_guest_enter and really running in guest os, where a perf event might overflow. Anyway, the scope is very narrow, I will change it to use flag PF_VCPU.

There is also a window between setting the flag and calling 'int $2' where an NMI might happen and be accounted incorrectly.

Perhaps separate the 'int $2' into a direct call into perf and another call for the rest of NMI handling. I don't see how it would work on svm though - AFAICT the NMI is held whereas vmx swallows it.

Quoted 6 lines

Quoted 2 lines

I guess NMIs
will be disabled until the next IRET so it isn't racy, just tricky.

I'm not sure if vmexit does break NMI context or not. Hardware NMI context isn't reentrant till a IRET. YangSheng would like to double check it.

After more check, I think VMX won't remained NMI block state for host. That's means, if NMI happened and processor is in VMX non-root mode, it would only result in VMExit, with a reason indicate that it's due to NMI happened, but no more state change in the host.

So in that meaning, there _is_ a window between VMExit and KVM handle the NMI. Moreover, I think we _can't_ stop the re-entrance of NMI handling code because "int $2" don't have effect to block following NMI.

And if the NMI sequence is not important(I think so), then we need to generate a real NMI in current vmexit-after code. Seems let APIC send a NMI IPI to itself is a good idea.

I am debugging a patch based on apic->send_IPI_self(NMI_VECTOR) to replace "int $2". Something unexpected is happening...

Quoted 6 lines

Quoted 2 lines

You can't use the APIC to send vectors 0x00-0x1f, or at least, aren't supposed to be able to.

Um? Why?

Especially kernel is already using it to deliver NMI.

That's the only defined case, and it is defined because the vector field is ignore for DM_NMI. Vol 3A (exact section numbers may vary depending on your version).

8.5.1 / 8.6.1

'100 (NMI) Delivers an NMI interrupt to the target processor or processors. The vector information is ignored'

8.5.2 Valid Interrupt Vectors

'Local and I/O APICs support 240 of these vectors (in the range of 16 to
255) as valid interrupts.'

Quoted 7 lines

8.8.4 Interrupt Acceptance for Fixed Interrupts

'...; vectors 0 through 15 are reserved by the APIC (see also: Section 8.5.2, "Valid Interrupt Vectors")'

So I misremembered, apparently you can deliver interrupts 0x10-0x1f, but vectors 0x00-0x0f are not valid to send via APIC or I/O APIC.

As you pointed out, NMI is not "Fixed interrupt". If we want to send NMI,
it would need a specific delivery mode rather than vector number.

And if you look at code, if we specific NMI_VECTOR, the delivery mode would
be set to NMI.

So what's wrong here?

Quoted 74 lines · Yanmin

Quoted 47 lines · Ingo Molnar

Quoted 45 lines · Zhang, Yanmin

Quoted 29 lines · Yanmin

Quoted 19 lines · Avi Kivity

Quoted 18 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Quoted 7 lines

Quoted 6 lines

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Thanks for your kind comments.

Quoted 7 lines

Quoted 4 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels?

With the patch, 'perf kvm report --sort pid" could show
summary statistics for all guest os instances. Then, use
parameter --pid of 'perf kvm record' to collect single problematic instance data.

Sorry. I found currently --pid isn't process but a thread (main thread).

Ingo,

Is it possible to support a new parameter or extend --inherit, so 'perf record' and 'perf top' could collect data on all threads of a process when the process is running?

Quoted 8 lines

Quoted 2 lines

If not, I need add a new ugly parameter which is similar to --pid to filter out process data in userspace.

Yeah. For maximum utility i'd suggest to extend --pid to include this, and introduce --tid for the previous, limited-to-a-single-task functionality.

Most users would expect --pid to work like a 'late attach' - i.e. to work like strace -f or like a gdb attach.

Thanks Ingo, Avi.

I worked out below patch against tip/master of March 15th.

Subject: [PATCH] Change perf's parameter --pid to process-wide collection
From: Zhang, Yanmin <[email protected]>

Change parameter -p (--pid) to real process pid and add -t (--tid) meaning thread id. Now, --pid means perf collects the statistics of all threads of the process, while --tid means perf just collect the statistics of that thread.

BTW, the patch fixes a bug of 'perf stat -p'. 'perf stat' always configures attr->disabled=1 if it isn't a system-wide collection. If there is a '-p' and no forks, 'perf stat -p' doesn't collect any data. In addition, the while(!done) in run_perf_stat consumes 100% single cpu time which has bad impact on running workload. I added a sleep(1) in the loop.

Quoted 1 line

Signed-off-by Zhang Yanmin <[email protected]>

Ingo,

Sorry, the patch has bugs. I need do a better job and will work out 2 separate patches against the 2 issues.

I worked out 3 new patches against tip/master tree of Mar. 17th.

Quoted 7 lines

It's about who owns the user interface.

If qemu owns the user interface, than we can satisfy this in a very simple way by adding a perf monitor command. If we have to support third party tools, then it significantly complicates things.

Of course illogical modularization complicates things 'significantly'.

Fast forward to 2010. The kernel side of KVM is maximum goodness - by far the worst-quality remaining aspects of KVM are precisely in areas that you
mention: 'if we have to support third party tools, then it significantly complicates things'. You kept Qemu as an external 'third party' entity to KVM, and KVM is clearly hurting from that - just see the recent KVM usability thread for examples about suckage.

- move a clean (and minimal) version of the Qemu code base to tools/kvm/, in the upstream kernel repo, and work on that from that point on.

- co-develop new features within the same patch. Release new versions of kvm-qemu and the kvm bits at the same time (together with the upstream kernel), at well defined points in time.

- encourage kernel-space and user-space KVM developers to work on both user-space and kernel-space bits as a single unit. It's one project and a single experience to the user.

- [ and probably libvirt should go there too ]

If KVM's hypervisor and guest kernel code can enjoy the benefits of a single repository,

why cannot the rest of KVM enjoy the same developer goodness? Only fixing that will bring the break-through in quality - not more manpower really.

Yes, i've read a thousand excuses for why this is an absolutely impossible and a bad thing to do, and none of them was really convincing to me - and you also have become rather emotional about all the arguments so it's hard to argue about it on a technical basis.

We made a similar (admittedly very difficult ...) design jump from oprofile to perf, and i can tell you from that experience that it's day and night, both in terms of development and in terms of the end result!

( We recently also made another, kernel/kernel unification that had a very
positive result: we unified the 32-bit and 64-bit x86 architectures. Even within the same repo the unification of technology is generally a good thing. The KVM/Qemu situation is different - it's more similar to the perf design. )

Not having to fight artificial package boundaries and forced package separation is very refreshing experience to a developer - and very rewarding and flexible to develop on. ABI compatibility is _easier_ to maintain in such a model. It's quite similar to the jump from Xen hacking to KVM hacking (i did both). It's a bit like the jump from CVS to Git. Trust me, you _cannot_ know the difference if you havent tried a similar jump with Qemu.

Anyway, you made your position about this rather clear and you are clearly uncompromising, so i just wanted to post this note to the list: you'll waste years of your life on a visibly crappy development model that has been unable to break through a magic usability barrier for the past 2-3 years - just like the Xen mis-design has wasted so many people's time and effort in kernel space.

I wish both you and Avi looked back 3-4 years and realized what made KVM so successful back then and why the hearts and minds of virtualization developers were captured by KVM almost overnight.

Fast forward to 2010. The kernel side of KVM is maximum goodness - by far the worst-quality remaining aspects of KVM are precisely in areas that you
mention: 'if we have to support third party tools, then it significantly complicates things'. You kept Qemu as an external 'third party' entity to KVM, and KVM is clearly hurting from that - just see the recent KVM usability thread for examples about suckage.

So a similar 'complication' is the crux of the matter behind KVM quality
problems: you've not followed through with the original KVM vision and you have not applied that concept to Qemu!

If you want to jump to the next level of technological quality you need to fix this attitude and you need to go back to the design roots of KVM. Concentrate on Qemu (as that is the weakest link now), make it a first class member of the KVM repo and simplify your development model by having a single repo:

- move a clean (and minimal) version of the Qemu code base to tools/kvm/, in the upstream kernel repo, and work on that from that point on.

- encourage kernel-space and user-space KVM developers to work on both user-space and kernel-space bits as a single unit. It's one project and a single experience to the user.

- [ and probably libvirt should go there too ]

Yes, i've read a thousand excuses for why this is an absolutely impossible and a bad thing to do, and none of them was really convincing to me - and you also have become rather emotional about all the arguments so it's hard to argue about it on a technical basis.

We made a similar (admittedly very difficult ...) design jump from oprofile to perf, and i can tell you from that experience that it's day and night, both in terms of development and in terms of the end result!

Quoted 82 lines · On Thursday 18 March 2010 13:22:28 Sheng Yang wrote:

Quoted 69 lines · On Thursday 18 March 2010 12:50:58 Zachary Amsden wrote:

Quoted 46 lines · Sheng Yang

Quoted 44 lines · On Thursday 18 March 2010 05:14:52 Zachary Amsden wrote:

Quoted 42 lines · Sheng Yang

Quoted 16 lines · Yanmin

Quoted 14 lines · Avi Kivity

Quoted 4 lines · Yanmin

Right, but there is a scope between kvm_guest_enter and really running in guest os, where a perf event might overflow. Anyway, the scope is very narrow, I will change it to use flag PF_VCPU.

There is also a window between setting the flag and calling 'int $2' where an NMI might happen and be accounted incorrectly.

Perhaps separate the 'int $2' into a direct call into perf and another call for the rest of NMI handling. I don't see how it would work on svm though - AFAICT the NMI is held whereas vmx swallows it.

Quoted 7 lines

Quoted 3 lines

I guess NMIs
will be disabled until the next IRET so it isn't racy, just tricky.

I'm not sure if vmexit does break NMI context or not. Hardware NMI context isn't reentrant till a IRET. YangSheng would like to double check it.

After more check, I think VMX won't remained NMI block state for host. That's means, if NMI happened and processor is in VMX non-root mode, it would only result in VMExit, with a reason indicate that it's due to NMI happened, but no more state change in the host.

So in that meaning, there _is_ a window between VMExit and KVM handle the NMI. Moreover, I think we _can't_ stop the re-entrance of NMI handling code because "int $2" don't have effect to block following NMI.

And if the NMI sequence is not important(I think so), then we need to generate a real NMI in current vmexit-after code. Seems let APIC send a NMI IPI to itself is a good idea.

I am debugging a patch based on apic->send_IPI_self(NMI_VECTOR) to replace "int $2". Something unexpected is happening...

Quoted 7 lines

Quoted 3 lines

You can't use the APIC to send vectors 0x00-0x1f, or at least, aren't supposed to be able to.

Um? Why?

Especially kernel is already using it to deliver NMI.

That's the only defined case, and it is defined because the vector field is ignore for DM_NMI. Vol 3A (exact section numbers may vary depending on your version).

8.5.1 / 8.6.1

'100 (NMI) Delivers an NMI interrupt to the target processor or processors. The vector information is ignored'

8.5.2 Valid Interrupt Vectors

'Local and I/O APICs support 240 of these vectors (in the range of 16 to
255) as valid interrupts.'

Quoted 8 lines

8.8.4 Interrupt Acceptance for Fixed Interrupts

'...; vectors 0 through 15 are reserved by the APIC (see also: Section 8.5.2, "Valid Interrupt Vectors")'

So I misremembered, apparently you can deliver interrupts 0x10-0x1f, but vectors 0x00-0x0f are not valid to send via APIC or I/O APIC.

As you pointed out, NMI is not "Fixed interrupt". If we want to send NMI,
it would need a specific delivery mode rather than vector number.

Quoted 5 lines

And if you look at code, if we specific NMI_VECTOR, the delivery mode would
be set to NMI.

So what's wrong here?

OK, I think I understand your points now. You meant that these vectors can't be filled in vector field directly, right? But NMI is a exception due to DM_NMI. Is that your point? I think we agree on this.

Quoted 8 lines · Ingo Molnar

Quoted 6 lines

It's about who owns the user interface.

If qemu owns the user interface, than we can satisfy this in a very simple way by adding a perf monitor command. If we have to support third party tools, then it significantly complicates things.

Of course illogical modularization complicates things 'significantly'.

Who should own the user interface then?

Quoted 6 lines

Fast forward to 2010. The kernel side of KVM is maximum goodness - by far the worst-quality remaining aspects of KVM are precisely in areas that you
mention: 'if we have to support third party tools, then it significantly complicates things'. You kept Qemu as an external 'third party' entity to KVM, and KVM is clearly hurting from that - just see the recent KVM usability thread for examples about suckage.

Any qemu usability problems are because developers (or their employers) are not interested in fixing them, not because of the repository location. Most kvm developer interest is in server-side deployment (even for desktop guests), so there is limited effort in implementing a virtualbox-style GUI.

Quoted 2 lines

- move a clean (and minimal) version of the Qemu code base to tools/kvm/, in the upstream kernel repo, and work on that from that point on.

I'll ignore the repository location which should be immaterial to a serious developer and concentrate on the 'clean and minimal' aspect, since it has some merit. [...]

[...]

Do you really think the echo'n'cat school of usability wants to write a GUI? In linux-2.6.git?

Quoted 8 lines · Avi Kivity

Quoted 7 lines · Ingo Molnar

Quoted 6 lines

It's about who owns the user interface.

If qemu owns the user interface, than we can satisfy this in a very simple way by adding a perf monitor command. If we have to support third party tools, then it significantly complicates things.

Quoted 3 lines

Quoted 1 line

Of course illogical modularization complicates things 'significantly'.

Who should own the user interface then?

If qemu was in tools/kvm/ then we wouldnt have such issues. A single patch (or series of patches) could modify tools/kvm/, arch/x86/kvm/, virt/ and tools/perf/.

Quoted 3 lines · Ingo Molnar

I wish both you and Avi looked back 3-4 years and realized what made KVM so successful back then and why the hearts and minds of virtualization developers were captured by KVM almost overnight.

Ingo,

What made KVM so successful was that the core kernel of the hypervisor was designed the right way, as a kernel module where it belonged. It was obvious to anyone who had been exposed to the main competition at the time, Xen, that this was the right approach. What has ended up killing Xen in the end is the not-invented-here approach of copying everything over, reformatting it, and rewriting half of it, which made it impossible to maintain and support as a single codebase. [...]

[...]

At my previous employer we ended up dropping all Xen efforts exactly because it was like maintaining two separate operating system kernels. The key to KVM is that once you have Linux, you practically have KVM as well.

Quoted 7 lines

Fast forward to 2010. The kernel side of KVM is maximum goodness - by far the worst-quality remaining aspects of KVM are precisely in areas that you
mention: 'if we have to support third party tools, then it significantly complicates things'. You kept Qemu as an external 'third party' entity to KVM, and KVM is clearly hurting from that - just see the recent KVM usability thread for examples about suckage.

Quoted 3 lines

So a similar 'complication' is the crux of the matter behind KVM quality
problems: you've not followed through with the original KVM vision and you have not applied that concept to Qemu!

Well there are two ways to go about this. Either you base the KVM userland on top of an existing project, like QEMU, _or_ you rewrite it all from scratch. [...]

[...] However, there is far more to it than just a couple of ioctls, for example the stack of reverse device-drivers is a pretty significant code base, rewriting that and maintaining it is not a trivial task. It is certainly my belief that the benefit we get from sharing that with QEMU by far outweighs the cost of forking it and keeping our own fork in the kernel tree. In fact it would result in exactly the same problems I mentioned above wrt Xen.

Quoted 6 lines

If you want to jump to the next level of technological quality you need to fix this attitude and you need to go back to the design roots of KVM. Concentrate on Qemu (as that is the weakest link now), make it a first class member of the KVM repo and simplify your development model by having a single repo:

Quoted 2 lines

- move a clean (and minimal) version of the Qemu code base to tools/kvm/, in the upstream kernel repo, and work on that from that point on.

With this you have just thrown away all the benefits of having the QEMU repository shared with other developers who will actively fix bugs in components we do care about for KVM.

Quoted 3 lines

- encourage kernel-space and user-space KVM developers to work on both user-space and kernel-space bits as a single unit. It's one project and a single experience to the user.

This is already happening and a total non issue.

Quoted 1 line

- [ and probably libvirt should go there too ]

Now that would be interesting, next we'll have to include things like libxml in the kernel git tree as well, to make sure libvirt doesn't get out of sync with the version supplied by your distribution vendor.

Quoted 4 lines

Yes, i've read a thousand excuses for why this is an absolutely impossible and a bad thing to do, and none of them was really convincing to me - and you also have become rather emotional about all the arguments so it's hard to argue about it on a technical basis.

So far your argument would justify pulling all of gdb into the kernel git tree as well, to support the kgdb efforts, or gcc so we can get rid of the gcc version quirks in the kernel header files, e2fsprogs and equivalent for _all_ file systems included in the kernel so we can make sure our fs tools never get out of sync with whats supported in the kernel......

Quoted 3 lines

We made a similar (admittedly very difficult ...) design jump from oprofile to perf, and i can tell you from that experience that it's day and night, both in terms of development and in terms of the end result!

The user components for perf vs oprofile are _tiny_ projects compared to the portions of QEMU that are actually used by KVM.

Oh and you completely forgot SeaBIOS. KVM+QEMU rely on SeaBIOS too, so from what you're saying we should pull that into the kernel git repository as well. Never mind the fact that we share SeaBIOS with the coreboot project which is very actively adding features to it that benefit us as well.....

Sorry, but there are times when unification make sense, and there are times where having a reasonably well designed split makes sense. KVM had problems with QEMU in the past which resulted in the qemu-kvm branch of it, which proved to be a major pain to deal with, but that is fortunately improving and qemu-kvm should go away completely at some point.

Quoted 11 lines · Ingo Molnar

Quoted 8 lines · Avi Kivity

Quoted 7 lines · Ingo Molnar

Quoted 6 lines

It's about who owns the user interface.

If qemu owns the user interface, than we can satisfy this in a very simple way by adding a perf monitor command. If we have to support third party tools, then it significantly complicates things.

Quoted 7 lines

Quoted 3 lines

Quoted 1 line

Of course illogical modularization complicates things 'significantly'.

Who should own the user interface then?

If qemu was in tools/kvm/ then we wouldnt have such issues. A single patch (or series of patches) could modify tools/kvm/, arch/x86/kvm/, virt/ and tools/perf/.

It's not a 1:1 connection. There are more users of the KVM interface. To name a few I'm aware of:

- Mac-on-Linux (PPC)
- Dolphin (PPC)
- Xenner (x86)
- Kuli (s390)

Having a clear userspace interface is the only viable solution there. And if you're interested, look at my MOL enabling patch. It's less than 500 lines of code.

The kernel/userspace interface really isn't the difficult part. Getting device emulation working properly, easily and fast is.

Quoted 9 lines · Avi Kivity

Quoted 7 lines · Ingo Molnar

Quoted 7 lines

It's about who owns the user interface.

If qemu owns the user interface, than we can satisfy this in a very simple way by adding a perf monitor command. If we have to support third party tools, then it significantly complicates things.

Quoted 4 lines

Quoted 2 lines

Of course illogical modularization complicates things 'significantly'.

Who should own the user interface then?

If qemu was in tools/kvm/ then we wouldnt have such issues. A single patch (or series of patches) could modify tools/kvm/, arch/x86/kvm/, virt/ and tools/perf/.

Numerous times did we have patches to kernel/perf_event.c that fixed some detail, also accompanied by a tools/perf/ patch fixing another detail. Having a single 'culture of contribution' is a powerful way to develop.

It turns out kernel developers can be pretty good user-space developers as well and user-space developers can be pretty good kernel developers as well. Some like to do both - as long as it's all within a single project.

The moment any change (be it as trivial as fixing a GUI detail or as complex as a new feature) involves two or more packages, development speed slows down to a crawl - while the complexity of the change might be very low!

Also, there's the harmful process that people start categorizing themselves into 'I am a kernel developer' and 'I am a user space programmer' stereotypes, which limits the scope of contributions artificially.

Quoted 7 lines

Quoted 7 lines

Fast forward to 2010. The kernel side of KVM is maximum goodness - by far the worst-quality remaining aspects of KVM are precisely in areas that you
mention: 'if we have to support third party tools, then it significantly complicates things'. You kept Qemu as an external 'third party' entity to KVM, and KVM is clearly hurting from that - just see the recent KVM usability thread for examples about suckage.

Quoted 5 lines

Any qemu usability problems are because developers (or their employers) are not interested in fixing them, not because of the repository location. Most kvm developer interest is in server-side deployment (even for desktop guests), so there is limited effort in implementing a virtualbox-style GUI.

The same has been said of oprofile as well: 'it somewhat sucks because we are too server centric', 'nobody is interested in good usability and oprofile is fine for the enterprises'. Ironically, the same has been said of Xen usability as well, up to the point KVM came around.

What was the core of the problem was a bad design and a split kernel-side user-side tool landscape.

In fact i think saying that 'our developers only care about the server' is borderline dishonest, when at the same time you are making it doubly sure (by inaction) that it stays so: by leaving an artificial package wall between kernel-side KVM and user-side KVM and not integrating the two technologies.

You'll never know what heights you could achieve if you leave that wall there
...

Furthermore, what should be realized is that bad usability hurts "server features" just as much. Most of the day-to-day testing is done on the desktop by desktop oriented testers/developers. _Not_ by enterprise shops - they tend to see the code years down the line to begin with ...

Yes, a particular feature might be server oriented, but a good portion of our testing is on the desktop and everyone is hurting from bad usability and this puts limits on contribution efficiency.

As the patch posted in _this very thread demonstrates it_, it is doubly more difficult to contribute a joint KVM+Qemu feature, because it's two separate code bases, two contribution guidelines, two release schedules. While to the user it really is just one and the same thing. It should be so for the developer as well.

Put in another way: KVM's current split design is making it easy to contribute server features (because the kernel side is clean and cool), but also makes it artificially hard to contribute desktop features: because the tooling side (Qemu) is 'just another package', is separated by a package and maintenance wall

and is made somewhat uncool by a (as some KVM developers have pointed out in this thread) quirky codebase.

(the rest of your points are really a function of this fundamental disagreement)

Quoted 9 lines

It's not a 1:1 connection. There are more users of the KVM interface. To name a few I'm aware of:

- Mac-on-Linux (PPC)
- Dolphin (PPC)
- Xenner (x86)
- Kuli (s390)

There must be a misunderstanding here: tools/perf/ still has a clear userspace interface and ABI. There's external projects making use of it: sysprof and libpfm (and probably more i dont know about it). Those projects are also contributing back.

Still it's _very_ useful to have a single reference implementation under tools/perf/ where we concentrate the best of the code. That is where we make sure that each new kernel feature is appropriately implemented in user-space as well, that the combination works well together and is releasable to users. That is what keeps us all honest: the latency of features is much lower, and there's no ping-pong of blame going on between the two components in case of bugs or in case of misfeatures.

Same goes for KVM+Qemu: it would be so much nicer to have a single, well-focused reference implementation under tools/kvm/ and have improvements flow into that code base.

That way KVM developers cannot just shrug "well, GUI suckage is a user-space problem" - like the answers i got in the KVM usability thread ...

The buck will stop here.

Quoted 3 lines

The kernel/userspace interface really isn't the difficult part. Getting device emulation working properly, easily and fast is.

The kernel/userspace ABI is not difficult at all. Getting device emulation working properly, easily and fast indeed is. And my experience is that it is not working properly nor quickly at the moment, at all. (see the 'KVM usability' thread)

Getting device emulation working properly often involves putting certain pieces that are currently done in Qemu into kernel-space. That kind of 'movement of emulation technology' from user-space component into the kernel-space component [or back] would very clearly be helped if those two components were in the same repository.

And i have first-hand experience there: we had (and have) similar scenarios with tools/perf routinely. We did some aspects in user-space, then decided to do it in kernel-space. Sometimes we moved kernel bits to user-space. It was very easy and there were no package and version complications as it's a single project. Sometimes we even moved bits back and forth until we found the right balance.

Quoted 5 lines · Ingo Molnar

Quoted 3 lines · Avi Kivity

Quoted 2 lines · Ingo Molnar

Quoted 2 lines

It's about who owns the user interface.

Quoted 8 lines

Quoted 5 lines

Quoted 4 lines

Quoted 3 lines

If qemu owns the user interface, than we can satisfy this in a very simple way by adding a perf monitor command. If we have to support third party tools, then it significantly complicates things.

Of course illogical modularization complicates things 'significantly'.

Who should own the user interface then?

If qemu was in tools/kvm/ then we wouldnt have such issues. A single patch (or series of patches) could modify tools/kvm/, arch/x86/kvm/, virt/ and tools/perf/.

We would have exactly the same issues, only they would be in a single repository. The only difference is that we could ignore potential alternatives to qemu, libvirt, and RHEV-M. But that's not how kernel ABIs are developed, we try to make them general, not suited to just one consumer that happens to be close to our heart.

Quoted 7 lines · Avi Kivity

Quoted 3 lines

- move a clean (and minimal) version of the Qemu code base to tools/kvm/, in the upstream kernel repo, and work on that from that point on.

I'll ignore the repository location which should be immaterial to a serious developer and concentrate on the 'clean and minimal' aspect, since it has some merit. [...]

To the contrary, experience shows that repository location, and in particular a shared repository for closely related bits is very much material!

It matters because when there are two separate projects, even a "serious developer" is finding it double and triple difficult to contribute even trivial changes.

It becomes literally a nightmare if you have to touch 3 packages: kernel, a library and an app codebase. It takes _forever_ to get anything useful done.

Also, 'focus on a single thing' is a very basic aspect of humans, especially those who do computer programming. Working on two code bases in two repositories at once can be very challenging physically and psychically.

So what i've seen is that OSS programmers tend to pick a side, pretty much randomly, and then rationalize it in hindsight why they prefer that side ;-)

Most of them become either a kernel developer or a user-space package developer - and then they specialize on that field and shy away from changes that involve both. It's a basic human thing to avoid the hassle that comes with multi-package changes. (One really has to be outright stupid, fanatic or desperate to even attempt such changes these days - such are the difficulties for a comparatively low return.)

The solution is to tear down such artificial walls of contribution where possible. And tearing down the wall between KVM and qemu-kvm seems very much possible and the advantages would be numerous.

Unless by "serious developer" you meant: "developer willing to [or forced to] waste time and effort on illogically structured technology".

Quoted 5 lines

[...]

Do you really think the echo'n'cat school of usability wants to write a GUI? In linux-2.6.git?

Then you'll be surprised to hear that it's happening as we speak and the commits are there in linux-2.6.git. Both a TUI and GUI is in the works.

Furthermore, the numbers show that half of the usability fixes to tools/perf/ came not from regular perf contributors but from random kernel developers and testers who when they build the latest kernel and try out perf at the same time (it's very easy because you already have it in the kernel repository - no separate download, no installation, etc. necessary).

I had literally zero such contributions when (the precursor to) 'perf' was still a separate user-space project.

You could have the same effect for Qemu: the latest bits in tools/kvm/ would be built by regular kernel testers and developers. The integration benefits dont just extend to developers, a unified project is vastly easier to test as well.

* Jes Sorensen<[email protected]> wrote:

Quoted 4 lines

At my previous employer we ended up dropping all Xen efforts exactly because it was like maintaining two separate operating system kernels. The key to KVM is that once you have Linux, you practically have KVM as well.

Yes. Please realize that what is behind it is a strikingly simple argument:

"Once you have a single project to develop and maintain all is much better."

Quoted 7 lines

[...] However, there is far more to it than just a couple of ioctls, for example the stack of reverse device-drivers is a pretty significant code base, rewriting that and maintaining it is not a trivial task. It is certainly my belief that the benefit we get from sharing that with QEMU by far outweighs the cost of forking it and keeping our own fork in the kernel tree. In fact it would result in exactly the same problems I mentioned above wrt Xen.

I do not suggest forking Qemu at all, i suggest using the most natural development model for the KVM+Qemu shared project: a single repository.

Quoted 3 lines

With this you have just thrown away all the benefits of having the QEMU repository shared with other developers who will actively fix bugs in components we do care about for KVM.

Not if it's a unified project.

Quoted 5 lines

Quoted 3 lines

- encourage kernel-space and user-space KVM developers to work on both user-space and kernel-space bits as a single unit. It's one project and a single experience to the user.

This is already happening and a total non issue.

My experience as an external observer of the end result contradicts this.

Seemingly trivial usability changes to the KVM+Qemu combo are not being done often because they involve cross-discipline changes.

Quoted 5 lines

Quoted 1 line

- [ and probably libvirt should go there too ]

Now that would be interesting, next we'll have to include things like libxml in the kernel git tree as well, to make sure libvirt doesn't get out of sync with the version supplied by your distribution vendor.

The way we have gone about this in tools/perf/ is similar to the route picked by Git: we only use very lowlevel libraries available everywhere, and we provide optional wrappers to the rest.

Quoted 5 lines

So far your argument would justify pulling all of gdb into the kernel git tree as well, to support the kgdb efforts, or gcc so we can get rid of the gcc version quirks in the kernel header files, e2fsprogs and equivalent for _all_ file systems included in the kernel so we can make sure our fs tools never get out of sync with whats supported in the kernel......

gdb and gcc is clearly extrinsic to the kernel so why would we move them there?

I was talking about tools that are closely related to the kernel - where much of the development and actual use is in combination with the Linux kernel.

90%+ of the Qemu usecases are combined with Linux. (Yes, i know that you can run Qemu without KVM, and no, i dont think it matters in the grand scheme of things and most investment into Qemu comes from the KVM angle these days. In particular it for sure does not justify handicapping future KVM evolution so drastically.)

Quoted 4 lines

Oh and you completely forgot SeaBIOS. KVM+QEMU rely on SeaBIOS too, so from what you're saying we should pull that into the kernel git repository as well. Never mind the fact that we share SeaBIOS with the coreboot project which is very actively adding features to it that benefit us as well.....

SeaBIOS is in essence a firmware, so it could either be loaded as such.

Just look at the qemu source code - the BIOSes are .bin images in qemu/pc-bios/ imported externally in essence.

qemu-kvm branch is not similar to my proposal at all: it made KVM _more_ fragmented, not more unified. I.e. it was a move in the exact opposite direction and i'd expect such a move to fail.

In fact the failure of qemu-kvm supports my point rather explicitly: it demonstrates that extra packages and split development are actively harmful.

I speak about this as a person who has done successful unifications of split codebases and in my judgement this move would be significantly beneficial to KVM.

You cannot really validly reject this proposal with "It wont work" as it clearly has worked in other, comparable cases. You could only reject this with "I have tried it and it didnt work".

Think about it: a clean and hackable user-space component in tools/kvm/. It's very tempting :-)

Quoted 4 lines

The moment any change (be it as trivial as fixing a GUI detail or as complex as a new feature) involves two or more packages, development speed slows down to a crawl - while the complexity of the change might be very low!

Why is that?