Enhance perf to collect KVM guest os statistics from host side

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

diff -Nraup linux-2.6_tipmaster0315/arch/x86/kvm/vmx.c linux-2.6_tipmaster0315_perfkvm/arch/x86/kvm/vmx.c

arch/x86/kvm/vmx.c +3 -0

@@ -26,6 +26,7 @@
  #include<linux/sched.h>
  #include<linux/moduleparam.h>
  #include<linux/ftrace_event.h>
+#include<linux/perf_event.h>
  #include "kvm_cache_regs.h"
  #include "x86.h"
 

@@ -3632,6 +3633,43 @@ static void update_cr8_intercept(struct
  	vmcs_write32(TPR_THRESHOLD, irr);
  }
 
+DEFINE_PER_CPU(int, kvm_in_guest) = {0};
+

Quoted 15 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

Quoted 8 lines

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels? Perhaps a symbol server that perf can connect to (and that would connect to guests in turn)?

Quoted 26 lines · Avi Kivity

Quoted 15 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

Quoted 9 lines

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Quoted 7 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels? Perhaps a symbol server that perf can connect to (and that would connect to guests in turn)?

The highest quality solution would be if KVM offered a 'guest extension' to the guest kernel's /proc/kallsyms that made it easy for user-space to get this information from an authorative source.

That's the main reason why the host side /proc/kallsyms is so popular and so
useful: while in theory it's mostly redundant information which can be gleaned from the System.map and other sources of symbol information, it's easily available and is _always_ trustable to come from the host kernel.

Separate System.map's have a tendency to go out of sync (or go missing when a devel kernel gets rebuilt, or if a devel package is not installed), and server ports (be that a TCP port space server or an UDP port space mount-point) are both a configuration hassle and are not guest-transparent.

So for instrumentation infrastructure (such as perf) we have a large and well founded preference for intrinsic, built-in, kernel-provided information: i.e. a largely 'built-in' and transparent mechanism to get to guest symbols.

Quoted 46 lines · Ingo Molnar

Quoted 24 lines · Avi Kivity

Quoted 15 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

Quoted 8 lines

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Quoted 6 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels? Perhaps a symbol server that perf can connect to (and that would connect to guests in turn)?

The highest quality solution would be if KVM offered a 'guest extension' to the guest kernel's /proc/kallsyms that made it easy for user-space to get this information from an authorative source.

That's the main reason why the host side /proc/kallsyms is so popular and so
useful: while in theory it's mostly redundant information which can be gleaned from the System.map and other sources of symbol information, it's easily available and is _always_ trustable to come from the host kernel.

Separate System.map's have a tendency to go out of sync (or go missing when a devel kernel gets rebuilt, or if a devel package is not installed), and server ports (be that a TCP port space server or an UDP port space mount-point) are both a configuration hassle and are not guest-transparent.

Quoted 3 lines

So for instrumentation infrastructure (such as perf) we have a large and well founded preference for intrinsic, built-in, kernel-provided information: i.e. a largely 'built-in' and transparent mechanism to get to guest symbols.

The symbol server's client can certainly access the bits through vmchannel.

Quoted 54 lines · Avi Kivity

Quoted 52 lines · Ingo Molnar

Quoted 26 lines · Avi Kivity

Quoted 15 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

Quoted 9 lines

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Quoted 7 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels? Perhaps a symbol server that perf can connect to (and that would connect to guests in turn)?

The highest quality solution would be if KVM offered a 'guest extension' to the guest kernel's /proc/kallsyms that made it easy for user-space to get this information from an authorative source.

That's the main reason why the host side /proc/kallsyms is so popular and so
useful: while in theory it's mostly redundant information which can be gleaned

from the System.map and other sources of symbol information, it's easily

available and is _always_ trustable to come from the host kernel.

Separate System.map's have a tendency to go out of sync (or go missing when a devel kernel gets rebuilt, or if a devel package is not installed), and server ports (be that a TCP port space server or an UDP port space mount-point) are both a configuration hassle and are not guest-transparent.

Quoted 6 lines

Quoted 4 lines

So for instrumentation infrastructure (such as perf) we have a large and well founded preference for intrinsic, built-in, kernel-provided information: i.e. a largely 'built-in' and transparent mechanism to get to guest symbols.

The symbol server's client can certainly access the bits through vmchannel.

Ok, that would work i suspect.

Would be nice to have the symbol server in tools/perf/ and also make it easy to add it to the initrd via a .config switch or so.

That would have basically all of the advantages of being built into the kernel (availability, configurability, transparency, hackability), while having all the advantages of a user-space approach as well (flexibility, extensibility, robustness, ease of maintenance, etc.).

Quoted 58 lines · Ingo Molnar

Quoted 47 lines · Avi Kivity

Quoted 46 lines · Ingo Molnar

Quoted 24 lines · Avi Kivity

Quoted 15 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

Quoted 8 lines

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Quoted 6 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels? Perhaps a symbol server that perf can connect to (and that would connect to guests in turn)?

The highest quality solution would be if KVM offered a 'guest extension' to the guest kernel's /proc/kallsyms that made it easy for user-space to get this information from an authorative source.

That's the main reason why the host side /proc/kallsyms is so popular and so
useful: while in theory it's mostly redundant information which can be gleaned from the System.map and other sources of symbol information, it's easily available and is _always_ trustable to come from the host kernel.

Separate System.map's have a tendency to go out of sync (or go missing when a devel kernel gets rebuilt, or if a devel package is not installed), and server ports (be that a TCP port space server or an UDP port space mount-point) are both a configuration hassle and are not guest-transparent.

Quoted 4 lines

Quoted 3 lines

So for instrumentation infrastructure (such as perf) we have a large and well founded preference for intrinsic, built-in, kernel-provided information: i.e. a largely 'built-in' and transparent mechanism to get to guest symbols.

The symbol server's client can certainly access the bits through vmchannel.

Ok, that would work i suspect.

Would be nice to have the symbol server in tools/perf/ and also make it easy to add it to the initrd via a .config switch or so.

Quoted 4 lines

That would have basically all of the advantages of being built into the kernel (availability, configurability, transparency, hackability), while having all the advantages of a user-space approach as well (flexibility, extensibility, robustness, ease of maintenance, etc.).

Note, I am not advocating building the vmchannel client into the host kernel. [...]

Quoted 13 lines

Quoted 13 lines

Quoted 3 lines

The symbol server's client can certainly access the bits through vmchannel.

Ok, that would work i suspect.

Would be nice to have the symbol server in tools/perf/ and also make it easy to add it to the initrd via a .config switch or so.

That would have basically all of the advantages of being built into the kernel (availability, configurability, transparency, hackability), while having all the advantages of a user-space approach as well (flexibility, extensibility, robustness, ease of maintenance, etc.).

Quoted 3 lines

Note, I am not advocating building the vmchannel client into the host kernel. [...]

Neither am i. What i suggested was a user-space binary/executable built in tools/perf and put into the initrd.

Quoted 11 lines · Ingo Molnar

Quoted 11 lines

Quoted 11 lines

Quoted 2 lines

The symbol server's client can certainly access the bits through vmchannel.

Ok, that would work i suspect.

Would be nice to have the symbol server in tools/perf/ and also make it easy to add it to the initrd via a .config switch or so.

That would have basically all of the advantages of being built into the kernel (availability, configurability, transparency, hackability), while having all the advantages of a user-space approach as well (flexibility, extensibility, robustness, ease of maintenance, etc.).

Quoted 4 lines

Quoted 2 lines

Note, I am not advocating building the vmchannel client into the host kernel. [...]

Neither am i. What i suggested was a user-space binary/executable built in tools/perf and put into the initrd.

I'm confused - initrd seems to be guest-side. I was talking about the host side.

Quoted 4 lines

I'm confused - initrd seems to be guest-side. I was talking about the host side.

host side doesnt need much support - just some client capability in perf itself. I suspect vmchannels are sufficiently flexible and configuration-free for such purposes? (i.e. like a filesystem in essence)

Quoted 6 lines · Ingo Molnar

Quoted 3 lines

I'm confused - initrd seems to be guest-side. I was talking about the host side.

host side doesnt need much support - just some client capability in perf itself. I suspect vmchannels are sufficiently flexible and configuration-free for such purposes? (i.e. like a filesystem in essence)

I haven't followed vmchannel closely, but I think it is. vmchannel is terminated in qemu on the host side, not in the host kernel. So perf would need to connect to qemu.

Quoted 5 lines

I haven't followed vmchannel closely, but I think it is. vmchannel is terminated in qemu on the host side, not in the host kernel. So perf would need to connect to qemu.

Hm, that sounds rather messy if we want to use it to basically expose kernel functionality in a guest/host unified way. Is the qemu process discoverable in some secure way?

Can we trust it?

Is there some proper tooling available to do
it, or do we have to push it through 2-3 packages to get such a useful feature done?

( That is the general thought process how many cross-discipline useful
desktop/server features hit the bit bucket before having had any chance of being vetted by users, and why Linux sucks so much when it comes to feature integration and application usability. )

Quoted 7 lines · Ingo Molnar

Quoted 4 lines

I haven't followed vmchannel closely, but I think it is. vmchannel is terminated in qemu on the host side, not in the host kernel. So perf would need to connect to qemu.

Hm, that sounds rather messy if we want to use it to basically expose kernel functionality in a guest/host unified way. Is the qemu process discoverable in some secure way?

We know its pid.

Quoted 1 line

Can we trust it?

No choice, it contains the guest address space.

Quoted 2 lines

Is there some proper tooling available to do it, or do we have to push it through 2-3 packages to get such a useful feature done?

libvirt manages qemu processes, but I don't think this should go through libvirt. qemu can do this directly by opening a unix domain socket in a well-known place.

Quoted 4 lines

( That is the general thought process how many cross-discipline useful
desktop/server features hit the bit bucket before having had any chance of being vetted by users, and why Linux sucks so much when it comes to feature integration and application usability. )

You can't solve everything in the kernel, even with a well populated tools/.

Quoted 7 lines · Avi Kivity

Quoted 5 lines · Ingo Molnar

Quoted 5 lines

I haven't followed vmchannel closely, but I think it is. vmchannel is terminated in qemu on the host side, not in the host kernel. So perf would need to connect to qemu.

Quoted 6 lines

Quoted 4 lines

Hm, that sounds rather messy if we want to use it to basically expose kernel functionality in a guest/host unified way. Is the qemu process discoverable in some secure way?

We know its pid.

How do i get a list of all 'guest instance PIDs',

and what is the way to talk
to Qemu?

Quoted 4 lines

Quoted 2 lines

Can we trust it?

No choice, it contains the guest address space.

I mean, i can trust a kernel service and i can trust /proc/kallsyms.

Can perf trust a random process claiming to be Qemu? What's the trust mechanism here?

Quoted 7 lines

Quoted 3 lines

Is there some proper tooling available to do it, or do we have to push it through 2-3 packages to get such a useful feature done?

libvirt manages qemu processes, but I don't think this should go through libvirt. qemu can do this directly by opening a unix domain socket in a well-known place.

So Qemu has never run into such problems before?

( Sounds weird - i think Qemu configuration itself should be done via a
unix domain socket driven configuration protocol as well. )

Quoted 7 lines

Quoted 5 lines

( That is the general thought process how many cross-discipline useful
desktop/server features hit the bit bucket before having had any chance of being vetted by users, and why Linux sucks so much when it comes to feature integration and application usability. )

You can't solve everything in the kernel, even with a well populated tools/.

Certainly not, but this is a technical problem in the kernel's domain, so it's a fair (and natural) expectation to be able to solve this within the kernel project.

Quoted 4 lines · Ingo Molnar

I mean, i can trust a kernel service and i can trust /proc/kallsyms.

Can perf trust a random process claiming to be Qemu? What's the trust mechanism here?

Obviously you can't trust anything you get from a guest, no matter how you get it.

Quoted 6 lines

Quoted 6 lines

I mean, i can trust a kernel service and i can trust /proc/kallsyms.

Can perf trust a random process claiming to be Qemu? What's the trust mechanism here?

Quoted 3 lines

Obviously you can't trust anything you get from a guest, no matter how you get it.

I'm not talking about the symbol strings and addresses, and the object contents for allocation (or debuginfo). I'm talking about the basic protocol of establishing which guest is which.

I.e. we really want to be able users to:

1) have it all working with a single guest, without having to specify 'which' guest (qemu PID) to work with. That is the dominant usecase both for developers and for a fair portion of testers.

2) Have some reasonable symbolic identification for guests. For example a usable approach would be to have 'perf kvm list', which would list all currently active guests:

$ perf kvm list
[1] Fedora
[2] OpenSuse
[3] Windows-XP
[4] Windows-7

And from that point on 'perf kvm -g OpenSuse record' would do the obvious thing. Users will be able to just use the 'OpenSuse' symbolic name for that guest, even if the guest got restarted and switched its main PID.

Any such facility needs trusted enumeration and a protocol where i can trust that the information i got is authorative. (I.e. 'OpenSuse' truly matches to the OpenSuse session - not to some local user starting up a Qemu instance that claims to be 'OpenSuse'.)

Is such a scheme possible/available? I suspect all the KVM configuration tools (i havent used them in some time - gui and command-line tools alike) use similar methods to ease guest management?

Quoted 10 lines · Ingo Molnar

Quoted 7 lines

Quoted 5 lines

I mean, i can trust a kernel service and i can trust /proc/kallsyms.

Can perf trust a random process claiming to be Qemu? What's the trust mechanism here?

Obviously you can't trust anything you get from a guest, no matter how you get it.

I'm not talking about the symbol strings and addresses, and the object contents for allocation (or debuginfo). I'm talking about the basic protocol of establishing which guest is which.

There is none. So far, qemu only dealt with managing just its own guest, and left all multiple guest management to higher levels up the stack (like libvirt).

Quoted 5 lines

I.e. we really want to be able users to:

1) have it all working with a single guest, without having to specify 'which' guest (qemu PID) to work with. That is the dominant usecase both for developers and for a fair portion of testers.

That's reasonable if we can get it working simply.

Quoted 14 lines

2) Have some reasonable symbolic identification for guests. For example a usable approach would be to have 'perf kvm list', which would list all currently active guests:

$ perf kvm list
[1] Fedora
[2] OpenSuse
[3] Windows-XP
[4] Windows-7

And from that point on 'perf kvm -g OpenSuse record' would do the obvious thing. Users will be able to just use the 'OpenSuse' symbolic name for that guest, even if the guest got restarted and switched its main PID.

Quoted 8 lines

Any such facility needs trusted enumeration and a protocol where i can trust that the information i got is authorative. (I.e. 'OpenSuse' truly matches to the OpenSuse session - not to some local user starting up a Qemu instance that claims to be 'OpenSuse'.)

Is such a scheme possible/available? I suspect all the KVM configuration tools (i havent used them in some time - gui and command-line tools alike) use similar methods to ease guest management?

You can do that through libvirt, but that only works for guests started through libvirt. libvirt provides command-line tools to list and manage guests (for example autostarting them on startup), and tools built on top of libvirt can manage guests graphically.

Looks like we have a layer inversion here. Maybe we need a plugin system - libvirt drops a .so into perf that teaches it how to list guests and get their symbols.

Quoted 6 lines

You can do that through libvirt, but that only works for guests started through libvirt. libvirt provides command-line tools to list and manage guests (for example autostarting them on startup), and tools built on top of libvirt can manage guests graphically.

Quoted 4 lines

Looks like we have a layer inversion here. Maybe we need a plugin system - libvirt drops a .so into perf that teaches it how to list guests and get their symbols.

Is libvirt used to start up all KVM guests? If not, if it's only used on some distros while other distros have other solutions then there's apparently no good way to get to such information, and the kernel bits of KVM do not provide it.

To the user (and to me) this looks like a KVM bug / missing feature. (and the user doesnt care where the blame is) If that is true then apparently the current KVM design has no technically actionable solution for certain categories of features!

[...]

Quoted 6 lines

Quoted 5 lines

I.e. we really want to be able users to:

1) have it all working with a single guest, without having to specify 'which' guest (qemu PID) to work with. That is the dominant usecase both for developers and for a fair portion of testers.

Quoted 1 line

That's reasonable if we can get it working simply.

IMO such ease of use is reasonable and required, full stop.
If it cannot be gotten simply then that's a bug: either in the code, or in the design, or in the development process that led to the design. Bugs need fixing. [...]

Quoted 7 lines · Ingo Molnar

[...]

Quoted 6 lines

Quoted 5 lines

I.e. we really want to be able users to:

1) have it all working with a single guest, without having to specify 'which' guest (qemu PID) to work with. That is the dominant usecase both for developers and for a fair portion of testers.

Quoted 6 lines

Quoted 1 line

That's reasonable if we can get it working simply.

IMO such ease of use is reasonable and required, full stop.
If it cannot be gotten simply then that's a bug: either in the code, or in the design, or in the development process that led to the design. Bugs need fixing. [...]

Perhaps the fact that kvm happens to deal with an interesting application area (virtualization) is misleading here. As far as the host kernel or other host userspace is concerned, qemu is just some random unprivileged userspace program (with some *optional* /dev/kvm services that might happen to require temporary root).

As such, perf trying to instrument qemu is no different than perf trying to instrument any other userspace widget. Therefore, expecting 'trusted enumeration' of instances is just as sensible as using 'trusted ps' and 'trusted /var/run/FOO.pid files'.

[...]

Quoted 4 lines

Perhaps the fact that kvm happens to deal with an interesting application area (virtualization) is misleading here. As far as the host kernel or other host userspace is concerned, qemu is just some random unprivileged userspace program [...]

You are quite mistaken: KVM isnt really a 'random unprivileged application' in this context, it is clearly an extension of system/kernel services.

( Which can be seen from the simple fact that what started the discussion was 'how do we get /proc/kallsyms from the guest'. I.e. an extension of the existing host-space /proc/kallsyms was desired. )

Hi -

Quoted 5 lines · Ingo Molnar

[...]

Quoted 4 lines

Perhaps the fact that kvm happens to deal with an interesting application area (virtualization) is misleading here. As far as the host kernel or other host userspace is concerned, qemu is just some random unprivileged userspace program [...]

Quoted 3 lines

You are quite mistaken: KVM isnt really a 'random unprivileged application' in this context, it is clearly an extension of system/kernel services.

I don't know what "extension of system/kernel services" means in this context, beyond something running on the system/kernel, like every other process. [...]

Quoted 5 lines

[...]

( Which can be seen from the simple fact that what started the
discussion was 'how do we get /proc/kallsyms from the guest'. I.e. an extension of the existing host-space /proc/kallsyms was desired. )

(Sorry, that smacks of circular reasoning.)

You are quite mistaken: KVM isnt really a 'random unprivileged application' in this context, it is clearly an extension of system/kernel services.

( Which can be seen from the simple fact that what started the discussion was
'how do we get /proc/kallsyms from the guest'. I.e. an extension of the existing host-space /proc/kallsyms was desired. )

In that sense the most natural 'extension' would be the solution i mentioned a week or two ago: to have a (read only) mount of all guest filesystems, plus a channel for profiling/tracing data. That would make symbol parsing easier and it's what extends the existing 'host space' abstraction in the most natural way.

( It doesnt even have to be done via the kernel - Qemu could implement that
via FUSE for example. )

Quoted 6 lines · Ingo Molnar

You are quite mistaken: KVM isnt really a 'random unprivileged application' in this context, it is clearly an extension of system/kernel services.

( Which can be seen from the simple fact that what started the discussion was
'how do we get /proc/kallsyms from the guest'. I.e. an extension of the existing host-space /proc/kallsyms was desired. )

Random tools (like perf) should not be able to do what you describe. It's a security nightmare.

If it's desirable to have /proc/kallsyms available, we can expose an interface in QEMU to provide that. That can then be plumbed through libvirt and QMP.

Then a management tool can use libvirt or QMP to obtain that information and interact with the kernel appropriately.

Quoted 6 lines

In that sense the most natural 'extension' would be the solution i mentioned a week or two ago: to have a (read only) mount of all guest filesystems, plus a channel for profiling/tracing data. That would make symbol parsing easier and it's what extends the existing 'host space' abstraction in the most natural way.

Quoted 2 lines

( It doesnt even have to be done via the kernel - Qemu could implement that
via FUSE for example. )

No way. The guest has sensitive data and exposing it widely on the host is a bad thing to do. [...]

Quoted 9 lines · Anthony Liguori

Quoted 7 lines · Ingo Molnar

You are quite mistaken: KVM isnt really a 'random unprivileged application' in this context, it is clearly an extension of system/kernel services.

( Which can be seen from the simple fact that what started the discussion was
'how do we get /proc/kallsyms from the guest'. I.e. an extension of the existing host-space /proc/kallsyms was desired. )

Quoted 3 lines

Random tools (like perf) should not be able to do what you describe. It's a security nightmare.

A security nightmare exactly how? Mind to go into details as i dont understand your point.

Quoted 4 lines

No way. The guest has sensitive data and exposing it widely on the host is a bad thing to do. [...]

Firstly, you are putting words into my mouth, as i said nothing about 'exposing it widely'. I suggest exposing it under the privileges of whoever has access to the guest image.

Secondly, regarding confidentiality, and this is guest security 101: whoever can access the image on the host _already_ has access to all the guest data!

A Linux image can generally be loopback mounted straight away:

losetup -o 32256 /dev/loop0 ./guest-image.img
mount -o ro /dev/loop0 /mnt-guest

(Or, if you are an unprivileged user who cannot mount, it can be read via ext2 tools.)

There's nothing the guest can do about that. The host is in total control of guest image data for heaven's sake!

All i'm suggesting is to make what is already possible more convenient.

Ingo

Quoted 6 lines · Ingo Molnar

Quoted 4 lines · Anthony Liguori

Quoted 3 lines · Ingo Molnar

You are quite mistaken: KVM isnt really a 'random unprivileged application' in this context, it is clearly an extension of system/kernel services.

Quoted 7 lines

Quoted 5 lines

Quoted 3 lines

( Which can be seen from the simple fact that what started the discussion was
'how do we get /proc/kallsyms from the guest'. I.e. an extension of the existing host-space /proc/kallsyms was desired. )

Random tools (like perf) should not be able to do what you describe. It's a security nightmare.

A security nightmare exactly how? Mind to go into details as i dont understand your point.

Assume you're using SELinux to implement mandatory access control. How do you label this file system?

Generally speaking, we don't know the difference between /proc/kallsyms vs. /dev/mem if we do generic passthrough. While it might be safe to have a relaxed label of kallsyms (since it's read only), it's clearly not safe to do that for /dev/mem, /etc/shadow, or any file containing sensitive information.

Rather, we ought to expose a higher level interface that we have more confidence in with respect to understanding the ramifications of exposing that guest data.

Quoted 6 lines

Quoted 2 lines

No way. The guest has sensitive data and exposing it widely on the host is a bad thing to do. [...]

Firstly, you are putting words into my mouth, as i said nothing about 'exposing it widely'. I suggest exposing it under the privileges of whoever has access to the guest image.

That doesn't work as nicely with SELinux.

It's completely reasonable to have a user that can interact in a read only mode with a VM via libvirt but cannot read the guest's disk images or the guest's memory contents.

Quoted 8 lines

Secondly, regarding confidentiality, and this is guest security 101: whoever can access the image on the host _already_ has access to all the guest data!

A Linux image can generally be loopback mounted straight away:

losetup -o 32256 /dev/loop0 ./guest-image.img
mount -o ro /dev/loop0 /mnt-guest

Quoted 5 lines

(Or, if you are an unprivileged user who cannot mount, it can be read via ext2 tools.)

There's nothing the guest can do about that. The host is in total control of guest image data for heaven's sake!

It's not that simple in a MAC environment.

Quoted 23 lines · Anthony Liguori

Quoted 12 lines · Ingo Molnar

Quoted 9 lines · Anthony Liguori

Quoted 7 lines · Ingo Molnar

You are quite mistaken: KVM isnt really a 'random unprivileged application' in this context, it is clearly an extension of system/kernel services.

( Which can be seen from the simple fact that what started the discussion was
'how do we get /proc/kallsyms from the guest'. I.e. an extension of the existing host-space /proc/kallsyms was desired. )

Quoted 6 lines

Quoted 3 lines

Random tools (like perf) should not be able to do what you describe. It's a security nightmare.

A security nightmare exactly how? Mind to go into details as i dont understand your point.

Assume you're using SELinux to implement mandatory access control. How do you label this file system?

Quoted 6 lines

Generally speaking, we don't know the difference between /proc/kallsyms vs. /dev/mem if we do generic passthrough. While it might be safe to have a relaxed label of kallsyms (since it's read only), it's clearly not safe to do that for /dev/mem, /etc/shadow, or any file containing sensitive information.

What's your _point_? Please outline a threat model, a vector of attack, _anything_ that substantiates your "it's a security nightmare" claim.

Quoted 4 lines

Rather, we ought to expose a higher level interface that we have more confidence in with respect to understanding the ramifications of exposing that guest data.

Exactly, we want something that has a flexible namespace and works well with Linux tools in general. Preferably that namespace should be human readable, and it should be hierarchic, and it should have a well-known permission model.

This concept exists in Linux and is generally called a 'filesystem'.

If a user cannot read the image file then the user has no access to its contents via other namespaces either. That is, of course, a basic security aspect.

( That is perfectly true with a non-SELinux Unix permission model as well, and
is true in the SELinux case as well. )

Erm. Please explain to me, what exactly is 'not that simple' in a MAC environment?

Also, i'd like to note that the 'restrictive SELinux setups' usecases are pretty secondary.

To demonstrate that, i'd like every KVM developer on this list who reads this mail and who has their home development system where they produce their patches set up in a restrictive MAC environment, in that you cannot even read the images you are using, to chime in with a "I'm doing that" reply.

[...]
The only way to really address this is to change the interaction. Instead of running perf externally to qemu, we should support a perf command in the qemu monitor that can then tie directly to the perf tooling. That gives us the best possible user experience.

Hi -

Quoted 6 lines · Anthony Liguori

[...]
The only way to really address this is to change the interaction. Instead of running perf externally to qemu, we should support a perf command in the qemu monitor that can then tie directly to the perf tooling. That gives us the best possible user experience.

To what extent could this be solved with less crossing of isolation/abstraction layers, if the perfctr facilities were properly virtualized?

Monitoring guests from the host is useful for kvm developers, but less so for users.

Quoted 3 lines · Avi Kivity

Monitoring guests from the host is useful for kvm developers, but less so for users.

Guest space profiling is easy, and 'perf kvm' is not about that. (plain 'perf' will work if a proper paravirt channel is opened to the host)

I think you might have misunderstood the purpose and role of the 'perf kvm' patch here? 'perf kvm' is aimed at KVM developers: it is them who improve KVM code, not guest kernel users.

Quoted 8 lines · Ingo Molnar

Quoted 2 lines · Avi Kivity

Monitoring guests from the host is useful for kvm developers, but less so for users.

Guest space profiling is easy, and 'perf kvm' is not about that. (plain 'perf' will work if a proper paravirt channel is opened to the host)

Quoted 3 lines

I think you might have misunderstood the purpose and role of the 'perf kvm' patch here? 'perf kvm' is aimed at KVM developers: it is them who improve KVM code, not guest kernel users.

Of course I understood it. My point was that 'perf kvm' serves a tiny minority of users. [...]

[...] That doesn't mean it isn't useful, just that it doesn't satisfy all needs by itself.

Hi -

Quoted 6 lines · Anthony Liguori

[...]
The only way to really address this is to change the interaction. Instead of running perf externally to qemu, we should support a perf command in the qemu monitor that can then tie directly to the perf tooling. That gives us the best possible user experience.

To what extent could this be solved with less crossing of isolation/abstraction layers, if the perfctr facilities were properly virtualized?

Hi -

Quoted 5 lines · Anthony Liguori

[...]
The only way to really address this is to change the interaction. Instead of running perf externally to qemu, we should support a perf command in the qemu monitor that can then tie directly to the perf tooling. That gives us the best possible user experience.

To what extent could this be solved with less crossing of isolation/abstraction layers, if the perfctr facilities were properly virtualized? [...]

If you want to use a synthetic filesystem as the management interface for qemu, that's one thing. But you suggested exposing the guest filesystem in its entirely and that's what I disagreed with.

The guest may encrypt it's disk image. It still ought to be possible to run perf against that guest, no?

Quoted 11 lines · Avi Kivity

Quoted 5 lines · Ingo Molnar

I mean, i can trust a kernel service and i can trust /proc/kallsyms.

Can perf trust a random process claiming to be Qemu? What's the trust mechanism here?

Obviously you can't trust anything you get from a guest, no matter how you get it.

I'm not talking about the symbol strings and addresses, and the object contents for allocation (or debuginfo). I'm talking about the basic protocol of establishing which guest is which.

I.e. we really want to be able users to:

1) have it all working with a single guest, without having to specify 'which' guest (qemu PID) to work with. That is the dominant usecase both for developers and for a fair portion of testers.

2) Have some reasonable symbolic identification for guests. For example a usable approach would be to have 'perf kvm list', which would list all currently active guests:

$ perf kvm list
[1] Fedora
[2] OpenSuse
[3] Windows-XP
[4] Windows-7

And from that point on 'perf kvm -g OpenSuse record' would do the obvious thing. Users will be able to just use the 'OpenSuse' symbolic name for that guest, even if the guest got restarted and switched its main PID.

Is such a scheme possible/available? I suspect all the KVM configuration tools (i havent used them in some time - gui and command-line tools alike) use similar methods to ease guest management?

Ingo
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 13 lines · Ingo Molnar

Quoted 7 lines · Avi Kivity

Quoted 4 lines · Ingo Molnar

I mean, i can trust a kernel service and i can trust /proc/kallsyms.

Can perf trust a random process claiming to be Qemu? What's the trust mechanism here?

Obviously you can't trust anything you get from a guest, no matter how you get it.

I'm not talking about the symbol strings and addresses, and the object contents for allocation (or debuginfo). I'm talking about the basic protocol of establishing which guest is which.

Quoted 5 lines

I.e. we really want to be able users to:

1) have it all working with a single guest, without having to specify 'which' guest (qemu PID) to work with. That is the dominant usecase both for developers and for a fair portion of testers.

You're making too many assumptions.

There is no list of guests anymore than there is a list of web browsers.

You can have a multi-tenant scenario where you have distinct groups of virtual machines running as unprivileged users.

Quoted 10 lines

2) Have some reasonable symbolic identification for guests. For example a usable approach would be to have 'perf kvm list', which would list all currently active guests:

$ perf kvm list
[1] Fedora
[2] OpenSuse
[3] Windows-XP
[4] Windows-7

Quoted 3 lines

And from that point on 'perf kvm -g OpenSuse record' would do the obvious thing. Users will be able to just use the 'OpenSuse' symbolic name for that guest, even if the guest got restarted and switched its main PID.

Does "perf kvm list" always run as root? What if two unprivileged users both have a VM named "Fedora"?

If we look at the use-case, it's going to be something like, a user is creating virtual machines and wants to get performance information about them.

Having to run a separate tool like perf is not going to be what they would expect they had to do. Instead, they would either use their existing GUI tool (like virt-manager) or they would use their management interface (either QMP or libvirt).

The complexity of interaction is due to the fact that perf shouldn't be a stand alone tool. It should be a library or something with a programmatic interface that another tool can make use of.

Quoted 4 lines

If we look at the use-case, it's going to be something like, a user is creating virtual machines and wants to get performance information about them.

Quoted 9 lines

Having to run a separate tool like perf is not going to be what they would expect they had to do. Instead, they would either use their existing GUI tool (like virt-manager) or they would use their management interface (either QMP or libvirt).

The complexity of interaction is due to the fact that perf shouldn't be a stand alone tool. It should be a library or something with a programmatic interface that another tool can make use of.

But ... a GUI interface/integration is of course possible too, and it's being worked on.

perf is mainly a kernel developer tool, and kernel developers generally dont use GUIs to do their stuff: which is the (sole) reason why its first ~850 commits of tools/perf/ were done without a GUI. We go where our developers are.

In any case it's not an excuse to have no proper command-line tooling. In fact if you cannot get simpler, more atomic command-line tooling right then you'll probably doubly suck at doing a GUI as well.

Ingo

Quoted 15 lines · Ingo Molnar

Quoted 4 lines

If we look at the use-case, it's going to be something like, a user is creating virtual machines and wants to get performance information about them.

Quoted 8 lines

Having to run a separate tool like perf is not going to be what they would expect they had to do. Instead, they would either use their existing GUI tool (like virt-manager) or they would use their management interface (either QMP or libvirt).

The complexity of interaction is due to the fact that perf shouldn't be a stand alone tool. It should be a library or something with a programmatic interface that another tool can make use of.

But ... a GUI interface/integration is of course possible too, and it's being worked on.

Quoted 8 lines

perf is mainly a kernel developer tool, and kernel developers generally dont use GUIs to do their stuff: which is the (sole) reason why its first ~850 commits of tools/perf/ were done without a GUI. We go where our developers are.

In any case it's not an excuse to have no proper command-line tooling. In fact if you cannot get simpler, more atomic command-line tooling right then you'll probably doubly suck at doing a GUI as well.

It's about who owns the user interface.

If qemu owns the user interface, than we can satisfy this in a very simple way by adding a perf monitor command. If we have to support third party tools, then it significantly complicates things.

Quoted 7 lines

It's about who owns the user interface.

If qemu owns the user interface, than we can satisfy this in a very simple way by adding a perf monitor command. If we have to support third party tools, then it significantly complicates things.

Of course illogical modularization complicates things 'significantly'.

Fast forward to 2010. The kernel side of KVM is maximum goodness - by far the worst-quality remaining aspects of KVM are precisely in areas that you
mention: 'if we have to support third party tools, then it significantly complicates things'. You kept Qemu as an external 'third party' entity to KVM, and KVM is clearly hurting from that - just see the recent KVM usability thread for examples about suckage.

- move a clean (and minimal) version of the Qemu code base to tools/kvm/, in the upstream kernel repo, and work on that from that point on.

- co-develop new features within the same patch. Release new versions of kvm-qemu and the kvm bits at the same time (together with the upstream kernel), at well defined points in time.

- encourage kernel-space and user-space KVM developers to work on both user-space and kernel-space bits as a single unit. It's one project and a single experience to the user.

- [ and probably libvirt should go there too ]

If KVM's hypervisor and guest kernel code can enjoy the benefits of a single repository,

why cannot the rest of KVM enjoy the same developer goodness? Only fixing that will bring the break-through in quality - not more manpower really.

Yes, i've read a thousand excuses for why this is an absolutely impossible and a bad thing to do, and none of them was really convincing to me - and you also have become rather emotional about all the arguments so it's hard to argue about it on a technical basis.

We made a similar (admittedly very difficult ...) design jump from oprofile to perf, and i can tell you from that experience that it's day and night, both in terms of development and in terms of the end result!

( We recently also made another, kernel/kernel unification that had a very
positive result: we unified the 32-bit and 64-bit x86 architectures. Even within the same repo the unification of technology is generally a good thing. The KVM/Qemu situation is different - it's more similar to the perf design. )

Not having to fight artificial package boundaries and forced package separation is very refreshing experience to a developer - and very rewarding and flexible to develop on. ABI compatibility is _easier_ to maintain in such a model. It's quite similar to the jump from Xen hacking to KVM hacking (i did both). It's a bit like the jump from CVS to Git. Trust me, you _cannot_ know the difference if you havent tried a similar jump with Qemu.

Anyway, you made your position about this rather clear and you are clearly uncompromising, so i just wanted to post this note to the list: you'll waste years of your life on a visibly crappy development model that has been unable to break through a magic usability barrier for the past 2-3 years - just like the Xen mis-design has wasted so many people's time and effort in kernel space.

Quoted 8 lines · Ingo Molnar

Quoted 6 lines

It's about who owns the user interface.

If qemu owns the user interface, than we can satisfy this in a very simple way by adding a perf monitor command. If we have to support third party tools, then it significantly complicates things.

Of course illogical modularization complicates things 'significantly'.

Who should own the user interface then?

Quoted 6 lines

Fast forward to 2010. The kernel side of KVM is maximum goodness - by far the worst-quality remaining aspects of KVM are precisely in areas that you
mention: 'if we have to support third party tools, then it significantly complicates things'. You kept Qemu as an external 'third party' entity to KVM, and KVM is clearly hurting from that - just see the recent KVM usability thread for examples about suckage.

Any qemu usability problems are because developers (or their employers) are not interested in fixing them, not because of the repository location. Most kvm developer interest is in server-side deployment (even for desktop guests), so there is limited effort in implementing a virtualbox-style GUI.

Quoted 8 lines · Avi Kivity

Quoted 7 lines · Ingo Molnar

Quoted 6 lines

It's about who owns the user interface.

If qemu owns the user interface, than we can satisfy this in a very simple way by adding a perf monitor command. If we have to support third party tools, then it significantly complicates things.

Quoted 3 lines

Quoted 1 line

Of course illogical modularization complicates things 'significantly'.

Who should own the user interface then?

If qemu was in tools/kvm/ then we wouldnt have such issues. A single patch (or series of patches) could modify tools/kvm/, arch/x86/kvm/, virt/ and tools/perf/.

Quoted 11 lines · Ingo Molnar

Quoted 8 lines · Avi Kivity

Quoted 7 lines · Ingo Molnar

Quoted 6 lines

It's about who owns the user interface.

If qemu owns the user interface, than we can satisfy this in a very simple way by adding a perf monitor command. If we have to support third party tools, then it significantly complicates things.

Quoted 7 lines

Quoted 3 lines

Quoted 1 line

Of course illogical modularization complicates things 'significantly'.

Who should own the user interface then?

If qemu was in tools/kvm/ then we wouldnt have such issues. A single patch (or series of patches) could modify tools/kvm/, arch/x86/kvm/, virt/ and tools/perf/.

It's not a 1:1 connection. There are more users of the KVM interface. To name a few I'm aware of:

- Mac-on-Linux (PPC)
- Dolphin (PPC)
- Xenner (x86)
- Kuli (s390)

Having a clear userspace interface is the only viable solution there. And if you're interested, look at my MOL enabling patch. It's less than 500 lines of code.

The kernel/userspace interface really isn't the difficult part. Getting device emulation working properly, easily and fast is.

Quoted 9 lines

It's not a 1:1 connection. There are more users of the KVM interface. To name a few I'm aware of:

- Mac-on-Linux (PPC)
- Dolphin (PPC)
- Xenner (x86)
- Kuli (s390)

There must be a misunderstanding here: tools/perf/ still has a clear userspace interface and ABI. There's external projects making use of it: sysprof and libpfm (and probably more i dont know about it). Those projects are also contributing back.

Still it's _very_ useful to have a single reference implementation under tools/perf/ where we concentrate the best of the code. That is where we make sure that each new kernel feature is appropriately implemented in user-space as well, that the combination works well together and is releasable to users. That is what keeps us all honest: the latency of features is much lower, and there's no ping-pong of blame going on between the two components in case of bugs or in case of misfeatures.

Same goes for KVM+Qemu: it would be so much nicer to have a single, well-focused reference implementation under tools/kvm/ and have improvements flow into that code base.

That way KVM developers cannot just shrug "well, GUI suckage is a user-space problem" - like the answers i got in the KVM usability thread ...

The buck will stop here.

Quoted 3 lines

The kernel/userspace interface really isn't the difficult part. Getting device emulation working properly, easily and fast is.

The kernel/userspace ABI is not difficult at all. Getting device emulation working properly, easily and fast indeed is. And my experience is that it is not working properly nor quickly at the moment, at all. (see the 'KVM usability' thread)

Getting device emulation working properly often involves putting certain pieces that are currently done in Qemu into kernel-space. That kind of 'movement of emulation technology' from user-space component into the kernel-space component [or back] would very clearly be helped if those two components were in the same repository.

And i have first-hand experience there: we had (and have) similar scenarios with tools/perf routinely. We did some aspects in user-space, then decided to do it in kernel-space. Sometimes we moved kernel bits to user-space. It was very easy and there were no package and version complications as it's a single project. Sometimes we even moved bits back and forth until we found the right balance.

Quoted 7 lines

Still it's _very_ useful to have a single reference implementation under tools/perf/ where we concentrate the best of the code. That is where we make sure that each new kernel feature is appropriately implemented in user-space as well, that the combination works well together and is releasable to users. That is what keeps us all honest: the latency of features is much lower, and there's no ping-pong of blame going on between the two components in case of bugs or in case of misfeatures.

That would make sense for a truly minimal userspace for kvm: we once had a tool called kvmctl which was used to run tests (since folded into qemu). It didn't contain a GUI and was unable to run a general purpose guest. It was a few hundred lines of code, and indeed patches to kvmctl had a much closer correspondence to patches with kvm (though still low, as most kvm patches don't modify the ABI).

Quoted 4 lines

Same goes for KVM+Qemu: it would be so much nicer to have a single, well-focused reference implementation under tools/kvm/ and have improvements flow into that code base.

Quoted 5 lines

That way KVM developers cannot just shrug "well, GUI suckage is a user-space problem" - like the answers i got in the KVM usability thread
...

The buck will stop here.

Suppose we copy qemu tomorrow into tools/. All the problems will be copied with it. Someone still has to write patches to fix them. Who will it be?

Quoted 8 lines · Avi Kivity

Quoted 8 lines

Still it's _very_ useful to have a single reference implementation under tools/perf/ where we concentrate the best of the code. That is where we make sure that each new kernel feature is appropriately implemented in user-space as well, that the combination works well together and is releasable to users. That is what keeps us all honest: the latency of features is much lower, and there's no ping-pong of blame going on between the two components in case of bugs or in case of misfeatures.

Quoted 7 lines

That would make sense for a truly minimal userspace for kvm: we once had a tool called kvmctl which was used to run tests (since folded into qemu). It didn't contain a GUI and was unable to run a general purpose guest. It was a few hundred lines of code, and indeed patches to kvmctl had a much closer correspondence to patches with kvm (though still low, as most kvm patches don't modify the ABI).

If it's functional to the extent of at least allowing say a serial console via the console (like the UML binary allows) i'd expect the minimal user-space to quickly grow out of this minimal state. The rest will be history.

Maybe this is a better, simpler (and much cleaner and less controversial) approach than moving a 'full' copy of qemu there.

There's certainly no risk: if qemu stays dominant then nothing is lost [tools/kvm/ can be removed after some time], and if this clean base works out fine then the useful qemu technologies will move over to it gradually and without much fuss, and the developers will move with it as well.

If it's just a token effort with near zero utility to begin with it certainly wont take off.

Once it's there in tools/kvm/ and bootable i'd certainly hack up some quick xlib based VGA output capability myself - it's not that hard ;-) It would also allow me to test whether latest-KVM still boots fine in a much simpler way. (most of my testboxes dont have qemu installed)

So you have one user signed up for that already ;-)

[...]
Distributions are very eager to update kernels even in stable periods of the distro lifetime - they are much less willing to update user-space packages. [...]

Quoted 4 lines · Ingo Molnar

[...]
Distributions are very eager to update kernels even in stable periods of the distro lifetime - they are much less willing to update user-space packages. [...]

Sorry, er, what?  What distributions eagerly upgrade kernels in stable periods, were it not primarily motivated by security fixes?  What users eagerly replace their kernels?

Quoted 7 lines · Frank Ch. Eigler

Quoted 4 lines · Ingo Molnar

[...]
Distributions are very eager to update kernels even in stable periods of the distro lifetime - they are much less willing to update user-space packages. [...]

Quoted 4 lines

Sorry, er, what? ?What distributions eagerly upgrade kernels in stable periods, were it not primarily motivated by security fixes? ?What users eagerly replace their kernels?

Us guys reading and participating on the list. ;)

[...]

Quoted 1 line

Us guys reading and participating on the list. ;)

I'd like to second that - i'm actually quite happy to update the distro kernel. Also, i have rarely any problems even with bleeding edge kernels in rawhide - they are working pretty smoothly.

A large xorg update showing up in yum update gives me the cringe though ;-)

From a parochial point of view, that makes perfect sense: someone

Quoted 4 lines · Ingo Molnar

[...]
Distributions are very eager to update kernels even in stable periods of the distro lifetime - they are much less willing to update user-space packages. [...]

Sorry, er, what? What distributions eagerly upgrade kernels in stable periods, were it not primarily motivated by security fixes? [...]

[...] What users eagerly replace their kernels?

Quoted 3 lines

[...] What users eagerly replace their kernels?

Those 99% who click on the 'install 193 updates' popup.

Quoted 5 lines · Ingo Molnar

Quoted 2 lines

[...] What users eagerly replace their kernels?

Those 99% who click on the 'install 193 updates' popup.

Of which 1 is the kernel, and 192 are userspace updates (of which one may be qemu).

Quoted 9 lines · Avi Kivity

Quoted 5 lines · Ingo Molnar

Quoted 2 lines

[...] What users eagerly replace their kernels?

Those 99% who click on the 'install 193 updates' popup.

Of which 1 is the kernel, and 192 are userspace updates (of which one may be qemu).

I think you didnt understand my (tersely explained) point - which is probably my fault. What i said is:

- distros update the kernel first. Often in stable releases as well if there's a new kernel released. (They must because it provides new hardware enablement and other critical changes they generally cannot skip.)

- Qemu on the other hand is not upgraded with (nearly) that level of urgency. Completely new versions will generally have to wait for the next distro release.

Quoted 16 lines · Ingo Molnar

Quoted 9 lines · Avi Kivity

Quoted 5 lines · Ingo Molnar

Quoted 2 lines

[...] What users eagerly replace their kernels?

Those 99% who click on the 'install 193 updates' popup.

Of which 1 is the kernel, and 192 are userspace updates (of which one may be qemu).

I think you didnt understand my (tersely explained) point - which is probably my fault. What i said is:

Quoted 7 lines

- distros update the kernel first. Often in stable releases as well if there's a new kernel released. (They must because it provides new hardware enablement and other critical changes they generally cannot skip.)

- Qemu on the other hand is not upgraded with (nearly) that level of urgency. Completely new versions will generally have to wait for the next distro release.

This has nothing todo with them being in separate source repos. We could update QEMU to new major feature releaes with the same frequency in a Fedora release, but we delibrately choose not to rebase the QEMU userspace because experiance has shown the downside from new bugs / regressions outweighs the benefit of any new features.

The QEMU updates in stable Fedora trees, now just follow the minor bugfix release stream provided by QEMU & those arrive in Fedora with little noticable delay.

Quoted 31 lines · Daniel P. Berrange

Quoted 16 lines · Ingo Molnar

Quoted 9 lines · Avi Kivity

Quoted 5 lines · Ingo Molnar

Quoted 2 lines

[...] What users eagerly replace their kernels?

Those 99% who click on the 'install 193 updates' popup.

Of which 1 is the kernel, and 192 are userspace updates (of which one may be qemu).

I think you didnt understand my (tersely explained) point - which is probably my fault. What i said is:

Quoted 7 lines

 - distros update the kernel first. Often in stable releases as well if there's a new kernel released. (They must because it provides new hardware enablement and other critical changes they generally cannot skip.)

 - Qemu on the other hand is not upgraded with (nearly) that level of urgency. Completely new versions will generally have to wait for the next distro release.

This has nothing todo with them being in separate source repos. We could update QEMU to new major feature releaes with the same frequency in a Fedora release, but we delibrately choose not to rebase the QEMU userspace because experiance has shown the downside from new bugs / regressions outweighs the benefit of any new features.

Quoted 3 lines

The QEMU updates in stable Fedora trees, now just follow the minor bugfix release stream provided by QEMU & those arrive in Fedora with little noticable delay.

That is exactly what i said: Qemu and most user-space packages are on a 'slower' update track than the kernel: generally updated for minor releases.

My further point was that the kernel on the other hand gets updated more frequently and as such, any user-space tool bits hosted in the kernel repo get updated more frequently as well.

Thanks,

       Ingo

Quoted 41 lines · Ingo Molnar

Quoted 31 lines · Daniel P. Berrange

Quoted 16 lines · Ingo Molnar

Quoted 9 lines · Avi Kivity

Quoted 5 lines · Ingo Molnar

Quoted 2 lines

[...] What users eagerly replace their kernels?

Those 99% who click on the 'install 193 updates' popup.

Of which 1 is the kernel, and 192 are userspace updates (of which one may be qemu).

I think you didnt understand my (tersely explained) point - which is probably my fault. What i said is:

Quoted 7 lines

?- distros update the kernel first. Often in stable releases as well if ? ?there's a new kernel released. (They must because it provides new hardware ? ?enablement and other critical changes they generally cannot skip.)

?- Qemu on the other hand is not upgraded with (nearly) that level of urgency. ? ?Completely new versions will generally have to wait for the next distro ? ?release.

This has nothing todo with them being in separate source repos. We could update QEMU to new major feature releaes with the same frequency in a Fedora release, but we delibrately choose not to rebase the QEMU userspace because experiance has shown the downside from new bugs / regressions outweighs the benefit of any new features.

Quoted 3 lines

The QEMU updates in stable Fedora trees, now just follow the minor bugfix release stream provided by QEMU & those arrive in Fedora with little noticable delay.

That is exactly what i said: Qemu and most user-space packages are on a 'slower' update track than the kernel: generally updated for minor releases.

Quoted 7 lines

My further point was that the kernel on the other hand gets updated more frequently and as such, any user-space tool bits hosted in the kernel repo get updated more frequently as well.

Thanks,

? ? ? ?Ingo

Just to play devil's advocate, let's not mix up the development model with the distribution model. There is nothing to stop packagers and distributors from providing separate kernel "proper" packages and perf tools packages.

It might even make good sense assuming backwards compatibility for distros that have conservative policies about new kernel versions to provide newer perf tools packages with older kernels.

Quoted 11 lines · Avi Kivity

Quoted 6 lines · Ingo Molnar

Quoted 3 lines

[...] What users eagerly replace their kernels?

Those 99% who click on the 'install 193 updates' popup.

Of which 1 is the kernel, and 192 are userspace updates (of which one may be qemu).

I think you didnt understand my (tersely explained) point - which is probably my fault. What i said is:

- distros update the kernel first. Often in stable releases as well if there's a new kernel released. (They must because it provides new hardware enablement and other critical changes they generally cannot skip.)

- Qemu on the other hand is not upgraded with (nearly) that level of urgency. Completely new versions will generally have to wait for the next distro release.

With in-kernel tools the kernel and the tooling that accompanies the kernel are upgraded in the same low-latency pathway. That is a big plus if you are offering things like instrumentation (which perf does), which relates closely to the kernel.

Furthermore, many distros package up the latest -git kernel as well. They almost never do that with user-space packages.

Let me give you a specific example:

I'm running Fedora Rawhide with 2.6.34-rc1 right now on my main desktop, and that comes with perf-2.6.34-0.10.rc1.git0.fc14.noarch.

My rawhide box has qemu-kvm-0.12.3-3.fc14.x86_64 installed. That's more than a 1000 Qemu commits older than the latest Qemu development branch.

So by being part of the kernel repo there's lower latency upgrades and earlier and better testing available on most distros.

You made it very clear that you dont want that, but please dont try to claim that those advantages do not exist - they are very much real and we are making good use of it.

Quoted 8 lines · Ingo Molnar

Quoted 6 lines · Avi Kivity

Quoted 3 lines · Ingo Molnar

Quoted 1 line

[...] What users eagerly replace their kernels?

Those 99% who click on the 'install 193 updates' popup.

Of which 1 is the kernel, and 192 are userspace updates (of which one may be qemu).

Quoted 6 lines

I think you didnt understand my (tersely explained) point - which is probably my fault. What i said is:

- distros update the kernel first. Often in stable releases as well if there's a new kernel released. (They must because it provides new hardware enablement and other critical changes they generally cannot skip.)

No, they don't. [...]

[...] RHEL 5 is still on 2.6.18, for example. Users don't like their kernels updated unless absolutely necessary, with good reason.

Kernel updates = reboots.

Quoted 3 lines

- Qemu on the other hand is not upgraded with (nearly) that level of urgency. Completely new versions will generally have to wait for the next distro release.

F12 recently updated to 2.6.32. This is probably due to 2.6.31.stable dropping away, and no capacity at Fedora to maintain it on their own. So they are caught in a bind - stay on 2.6.31 and expose users to security vulnerabilities or move to 2.6.32 and cause regressions. Not a happy choice.

Quoted 7 lines

With in-kernel tools the kernel and the tooling that accompanies the kernel are upgraded in the same low-latency pathway. That is a big plus if you are offering things like instrumentation (which perf does), which relates closely to the kernel.

Furthermore, many distros package up the latest -git kernel as well. They almost never do that with user-space packages.

I'm sure if we ask the Fedora qemu maintainer to package qemu-kvm.git they'll consider it favourably. Isn't that what rawhide is for?

Quoted 11 lines

Let me give you a specific example:

I'm running Fedora Rawhide with 2.6.34-rc1 right now on my main desktop, and that comes with perf-2.6.34-0.10.rc1.git0.fc14.noarch.

My rawhide box has qemu-kvm-0.12.3-3.fc14.x86_64 installed. That's more than a 1000 Qemu commits older than the latest Qemu development branch.

So by being part of the kernel repo there's lower latency upgrades and earlier and better testing available on most distros.

Quoted 3 lines

You made it very clear that you dont want that, but please dont try to claim that those advantages do not exist - they are very much real and we are making good use of it.

I don't mind at all if rawhide users run on the latest and greatest, but release users deserve a little more stability.

Quoted 5 lines

[...] RHEL 5 is still on 2.6.18, for example. Users don't like their kernels updated unless absolutely necessary, with good reason.

Nope - RHEL 5 is on a 2.6.18 base for entirely different reasons.

Quoted 2 lines

Kernel updates = reboots.

If you check the update frequency of RHEL 5 kernels you'll see that it's comparable to that of Fedora.

Quoted 4 lines

Quoted 4 lines

- Qemu on the other hand is not upgraded with (nearly) that level of urgency. Completely new versions will generally have to wait for the next distro release.

Quoted 6 lines

F12 recently updated to 2.6.32. This is probably due to 2.6.31.stable dropping away, and no capacity at Fedora to maintain it on their own. So they are caught in a bind - stay on 2.6.31 and expose users to security vulnerabilities or move to 2.6.32 and cause regressions. Not a happy choice.

Happy choice or not, this is what i said is the distro practice these days. (i dont know all the distros that well so i'm sure there's differences)

Quoted 3 lines

I don't mind at all if rawhide users run on the latest and greatest, but release users deserve a little more stability.

What are you suggesting, that released versions of KVM are not reliable? Of course any tools/ bits are release engineered just as much as the rest of KVM
...

Quoted 3 lines

Happy choice or not, this is what i said is the distro practice these days. (i dont know all the distros that well so i'm sure there's differences)

So in addition to all the normal kernel regressions, you want to force tools/kvm/ regressions on users.

Quoted 6 lines

Quoted 2 lines

I don't mind at all if rawhide users run on the latest and greatest, but release users deserve a little more stability.

What are you suggesting, that released versions of KVM are not reliable? Of course any tools/ bits are release engineered just as much as the rest of KVM ...

No, I am suggesting qemu-kvm.git is not as stable as released versions (and won't get fixed backported). Keep in mind that unlike many userspace applications, qemu exposes an ABI to guests which we must keep compatible.

Quoted 4 lines · Avi Kivity

Quoted 4 lines

Happy choice or not, this is what i said is the distro practice these days. (i dont know all the distros that well so i'm sure there's differences)

Quoted 3 lines

So in addition to all the normal kernel regressions, you want to force tools/kvm/ regressions on users.

So instead you force a NxN compatibility matrix [all versions of qemu combined with all versions of the kernel] instead of a linear N versions matrix with a clear focus on the last version. Brilliant engineering i have to say ;-)

Also, by your argument the kernel should be split up into a micro-kernel, with different packages for KVM, scheduler, drivers, upgradeable separately.

That would be a nightmare. (i can detail many facets of that nightmare if you insist but i'll spare the electrons for now) Fortunately few kernel developers share your views about this.

Quoted 7 lines

Quoted 7 lines

Quoted 3 lines

I don't mind at all if rawhide users run on the latest and greatest, but release users deserve a little more stability.

What are you suggesting, that released versions of KVM are not reliable? Of course any tools/ bits are release engineered just as much as the rest of KVM ...

Quoted 4 lines

No, I am suggesting qemu-kvm.git is not as stable as released versions (and won't get fixed backported). Keep in mind that unlike many userspace applications, qemu exposes an ABI to guests which we must keep compatible.

I think you still dont understand it: if a tool moves to the kernel repo, then it is _released stable_ together with the next stable kernel.

Quoted 7 lines

Quoted 4 lines

[...]
Distributions are very eager to update kernels even in stable periods of the distro lifetime - they are much less willing to update user-space packages. [...]

Sorry, er, what? What distributions eagerly upgrade kernels in stable periods, were it not primarily motivated by security fixes? [...]

Please check the popular distro called 'Fedora' for example

and its kernel upgrade policies.

Quoted 1 line

[...] What users eagerly replace their kernels?

Those 99% who click on the 'install 193 updates' popup.

Hi -

Quoted 9 lines

Quoted 7 lines

Quoted 4 lines

[...]
Distributions are very eager to update kernels even in stable periods of the distro lifetime - they are much less willing to update user-space packages. [...]

Sorry, er, what? What distributions eagerly upgrade kernels in stable periods, were it not primarily motivated by security fixes? [...]

Please check the popular distro called 'Fedora' for example

I do believe I've heard of it. According to fedora bodhi, there have been 18 kernel updates issues for fedora 11 since its release, of which 12 were for purely security updates, and most of the other six also contain security fixes. None are described as 'enhancement' updates. Oh, what about fedora 12? 8 updates total, of which 5 are security only, one for drm showstoppers, others including security fixes, again 0 tagged as 'enhancement'.

So where is that "eagerness" again?? My sense is that most users are happy to leave a stable kernel running as long as possible, and distributions know this. You surely must understand that the lkml demographics are different.

Quoted 1 line

and its kernel upgrade policies.

[citation needed]

Quoted 3 lines

Quoted 1 line

[...] What users eagerly replace their kernels?

Those 99% who click on the 'install 193 updates' popup.

That's not "eager". That's "I'm exasperated from guessing what's really important; let's not have so many updates; meh".

Quoted 9 lines · Avi Kivity

Quoted 7 lines · Ingo Molnar

Quoted 7 lines

It's about who owns the user interface.

If qemu owns the user interface, than we can satisfy this in a very simple way by adding a perf monitor command. If we have to support third party tools, then it significantly complicates things.

Quoted 4 lines

Quoted 2 lines

Of course illogical modularization complicates things 'significantly'.

Who should own the user interface then?

If qemu was in tools/kvm/ then we wouldnt have such issues. A single patch (or series of patches) could modify tools/kvm/, arch/x86/kvm/, virt/ and tools/perf/.

Numerous times did we have patches to kernel/perf_event.c that fixed some detail, also accompanied by a tools/perf/ patch fixing another detail. Having a single 'culture of contribution' is a powerful way to develop.

It turns out kernel developers can be pretty good user-space developers as well and user-space developers can be pretty good kernel developers as well. Some like to do both - as long as it's all within a single project.

The moment any change (be it as trivial as fixing a GUI detail or as complex as a new feature) involves two or more packages, development speed slows down to a crawl - while the complexity of the change might be very low!

Also, there's the harmful process that people start categorizing themselves into 'I am a kernel developer' and 'I am a user space programmer' stereotypes, which limits the scope of contributions artificially.

Quoted 7 lines

Quoted 7 lines

Fast forward to 2010. The kernel side of KVM is maximum goodness - by far the worst-quality remaining aspects of KVM are precisely in areas that you
mention: 'if we have to support third party tools, then it significantly complicates things'. You kept Qemu as an external 'third party' entity to KVM, and KVM is clearly hurting from that - just see the recent KVM usability thread for examples about suckage.

Quoted 5 lines

Any qemu usability problems are because developers (or their employers) are not interested in fixing them, not because of the repository location. Most kvm developer interest is in server-side deployment (even for desktop guests), so there is limited effort in implementing a virtualbox-style GUI.

The same has been said of oprofile as well: 'it somewhat sucks because we are too server centric', 'nobody is interested in good usability and oprofile is fine for the enterprises'. Ironically, the same has been said of Xen usability as well, up to the point KVM came around.

What was the core of the problem was a bad design and a split kernel-side user-side tool landscape.

In fact i think saying that 'our developers only care about the server' is borderline dishonest, when at the same time you are making it doubly sure (by inaction) that it stays so: by leaving an artificial package wall between kernel-side KVM and user-side KVM and not integrating the two technologies.

You'll never know what heights you could achieve if you leave that wall there
...

Furthermore, what should be realized is that bad usability hurts "server features" just as much. Most of the day-to-day testing is done on the desktop by desktop oriented testers/developers. _Not_ by enterprise shops - they tend to see the code years down the line to begin with ...

Yes, a particular feature might be server oriented, but a good portion of our testing is on the desktop and everyone is hurting from bad usability and this puts limits on contribution efficiency.

As the patch posted in _this very thread demonstrates it_, it is doubly more difficult to contribute a joint KVM+Qemu feature, because it's two separate code bases, two contribution guidelines, two release schedules. While to the user it really is just one and the same thing. It should be so for the developer as well.

Put in another way: KVM's current split design is making it easy to contribute server features (because the kernel side is clean and cool), but also makes it artificially hard to contribute desktop features: because the tooling side (Qemu) is 'just another package', is separated by a package and maintenance wall

and is made somewhat uncool by a (as some KVM developers have pointed out in this thread) quirky codebase.

(the rest of your points are really a function of this fundamental disagreement)

Quoted 5 lines · Ingo Molnar

Quoted 3 lines · Avi Kivity

Quoted 2 lines · Ingo Molnar

Quoted 2 lines

It's about who owns the user interface.

Quoted 8 lines

Quoted 5 lines

Quoted 4 lines

Quoted 3 lines

If qemu owns the user interface, than we can satisfy this in a very simple way by adding a perf monitor command. If we have to support third party tools, then it significantly complicates things.

Of course illogical modularization complicates things 'significantly'.

Who should own the user interface then?

If qemu was in tools/kvm/ then we wouldnt have such issues. A single patch (or series of patches) could modify tools/kvm/, arch/x86/kvm/, virt/ and tools/perf/.

We would have exactly the same issues, only they would be in a single repository. The only difference is that we could ignore potential alternatives to qemu, libvirt, and RHEV-M. But that's not how kernel ABIs are developed, we try to make them general, not suited to just one consumer that happens to be close to our heart.

Quoted 4 lines

The moment any change (be it as trivial as fixing a GUI detail or as complex as a new feature) involves two or more packages, development speed slows down to a crawl - while the complexity of the change might be very low!

Why is that?

I the maintainers of all packages are cooperative and responsive, then the patches will get accepted quickly. If they aren't, development will be slow. [...]

[...] It isn't any different from contributing to two unrelated kernel subsystems (which are in fact in different repositories until the next merge window).

Quoted 5 lines · Avi Kivity

Quoted 5 lines

The moment any change (be it as trivial as fixing a GUI detail or as complex as a new feature) involves two or more packages, development speed slows down to a crawl - while the complexity of the change might be very low!

Quoted 2 lines

Why is that?

It's very simple: because the contribution latencies and overhead compound, almost inevitably.

If you ever tried to implement a combo GCC+glibc+kernel feature you'll know
...

Even with the best-run projects in existence it takes forever and is very painful - and here i talk about first hand experience over many years.

Quoted 4 lines

I the maintainers of all packages are cooperative and responsive, then the patches will get accepted quickly. If they aren't, development will be slow. [...]

I'm afraid practice is different from the rosy ideal you paint there. Even with assumed 'perfect projects' there's always random differences between projects, causing doubled (tripled) overhead and compounded up overhead:

- random differences in release schedules

- random differences in contribution guidelines

- random differences in coding style

Quoted 4 lines

[...] It isn't any different from contributing to two unrelated kernel subsystems (which are in fact in different repositories until the next merge window).

You mention a perfect example: contributing to multipe kernel subsystems. Even _that_ is very noticeably harder than contributing to a single subsystem - due to the inevitable buerocratic overhead, due to different development trees, due to different merge criteria.

So you are underlining my point (perhaps without intending to): treating closely related bits of technology as a single project is much better.

Obviously arch/x86/kvm/, virt/ and tools/kvm/ should live in a single development repository (perhaps micro-differentiated by a few topical branches), for exactly those reasons you mention.

Quoted 6 lines · Ingo Molnar

Quoted 4 lines · Avi Kivity

Quoted 4 lines

The moment any change (be it as trivial as fixing a GUI detail or as complex as a new feature) involves two or more packages, development speed slows down to a crawl - while the complexity of the change might be very low!

Quoted 3 lines

Quoted 1 line

Why is that?

It's very simple: because the contribution latencies and overhead compound, almost inevitably.

It's not inevitable, if the projects are badly run, you'll have high latencies, but projects don't have to be badly run.

Quoted 6 lines

If you ever tried to implement a combo GCC+glibc+kernel feature you'll know
...

Even with the best-run projects in existence it takes forever and is very painful - and here i talk about first hand experience over many years.

Try sending a patch to qemu-devel@, you may be pleasantly surprised.

Quoted 7 lines

Quoted 3 lines

I the maintainers of all packages are cooperative and responsive, then the patches will get accepted quickly. If they aren't, development will be slow. [...]

I'm afraid practice is different from the rosy ideal you paint there. Even with assumed 'perfect projects' there's always random differences between projects, causing doubled (tripled) overhead and compounded up overhead:

Quoted 5 lines

- random differences in release schedules

- random differences in contribution guidelines

- random differences in coding style

None of these matter for steady contributors.

Quoted 11 lines

Quoted 3 lines

[...] It isn't any different from contributing to two unrelated kernel subsystems (which are in fact in different repositories until the next merge window).

You mention a perfect example: contributing to multipe kernel subsystems. Even _that_ is very noticeably harder than contributing to a single subsystem - due to the inevitable buerocratic overhead, due to different development trees, due to different merge criteria.

So you are underlining my point (perhaps without intending to): treating closely related bits of technology as a single project is much better.

Quoted 3 lines

Obviously arch/x86/kvm/, virt/ and tools/kvm/ should live in a single development repository (perhaps micro-differentiated by a few topical branches), for exactly those reasons you mention.

How is a patch for the qemu GUI eject button and the kvm shadow mmu related? Should a single maintainer deal with both?

Quoted 4 lines

It's not inevitable, if the projects are badly run, you'll have high latencies, but projects don't have to be badly run.

So the 64K dollar question is, why does Qemu still suck?

Quoted 3 lines

How is a patch for the qemu GUI eject button and the kvm shadow mmu related? Should a single maintainer deal with both?

We have co-maintainers for perf that have a different focus. It works pretty well.

Look at git log tools/perf/ and how user-space and kernel-space components interact in practice. You'll patches that only impact one side, but you'll see very big overlap both in contributor identity and in patches as well.

Also, let me put similar questions in a bit different way:

- ' how is an in-kernel PIT emulation connected to Qemu's PIT emulation? '

- ' how is the in-kernel dynticks implementation related to Qemu's implementation of hardware timers? '

- ' how is an in-kernel event for a CD-ROM eject connected to an in-Qemu eject event? '

- ' how is a new hardware virtualization feature related to being able to configure and use it via Qemu? '

- ' how is the in-kernel x86 decoder/emulator related to the Qemu x86 emulator? '

- ' how is the performance of the qemu GUI related to the way VGA buffers are mapped and accelerated by KVM? '

They are obviously deeply related.

The quality of a development process is not
defined by the easy cases where no project unification is needed. The quality of a development process is defined by the _difficult_ cases.

Quoted 4 lines · Ingo Molnar

Quoted 2 lines

It's not inevitable, if the projects are badly run, you'll have high latencies, but projects don't have to be badly run.

So the 64K dollar question is, why does Qemu still suck?

Where people sent patches, it doesn't suck (or sucks less). Where they don't, it still sucks. [...]

[...] And it cost way more than $64K.

If moving things to tools/ helps, let's move Fedora to tools/.

Quoted 5 lines

Quoted 2 lines

How is a patch for the qemu GUI eject button and the kvm shadow mmu related? Should a single maintainer deal with both?

We have co-maintainers for perf that have a different focus. It works pretty well.

And it works well when I have patches that change x86 core and kvm. But that's no longer a single repository and we have to coordinate.

Quoted 7 lines

Look at git log tools/perf/ and how user-space and kernel-space components interact in practice. You'll patches that only impact one side, but you'll see very big overlap both in contributor identity and in patches as well.

Also, let me put similar questions in a bit different way:

- ' how is an in-kernel PIT emulation connected to Qemu's PIT emulation? '

Both implement the same spec. One is be a code derivative of the other (via Xen).

Quoted 2 lines

- ' how is the in-kernel dynticks implementation related to Qemu's implementation of hardware timers? '

The quality of host kernel timers directly determines the quality of qemu's timer emulation.

Quoted 2 lines

- ' how is an in-kernel event for a CD-ROM eject connected to an in-Qemu eject event? '

Both implement the same spec. The kernel of course needs to handle all implementation variants, while qemu only needs to implement it once.

Quoted 2 lines

- ' how is a new hardware virtualization feature related to being able to configure and use it via Qemu? '

Most features (example: npt) are transparent to userspace, some are not. When they are not, we introduce an ioctl() to kvm for controlling the feature, and a command-line switch to qemu for calling it.

Quoted 2 lines

- ' how is the in-kernel x86 decoder/emulator related to the Qemu x86 emulator? '

Both implement the same spec. Note qemu is not an emulator but a binary translator.

Quoted 2 lines

- ' how is the performance of the qemu GUI related to the way VGA buffers are mapped and accelerated by KVM? '

kvm needs to support direct mapping when possible and efficient data transfer when not. The latter will obviously be much slower. When direct mapping is possible, kvm needs to track pages touched by the guest to avoid full screen redraws. The rest (interfacing to X or vnc, implementing emulated hardware acceleration, full-screen mode, etc.) are unrelated.

Quoted 1 line

They are obviously deeply related.

Not at all. [...]

[...] kvm in fact knows nothing about vga, to take your last example. [...]

[...] To suggest that qemu needs to be close to the kernel to benefit from the kernel's timer implementation means we don't care about providing quality timing except to ourselves, which luckily isn't the case.

Some time ago the various desktops needed directory change notification, and people implemented inotify (or whatever it's called today). No one suggested tools/gnome/ and tools/kde/.

Quoted 3 lines

The quality of a development process is not defined by the easy cases where no project unification is needed. The quality of a development process is defined by the _difficult_ cases.

That's true, but we don't have issues at the qemu/kvm boundary. Note we do have issues at the qemu/aio interfaces and qemu/net interfaces (out of which vhost-net was born) but these wouldn't be solved by tools/qemu/.

Quoted 6 lines · Avi Kivity

Quoted 3 lines · Ingo Molnar

Quoted 3 lines

It's not inevitable, if the projects are badly run, you'll have high latencies, but projects don't have to be badly run.

Quoted 5 lines

Quoted 2 lines

So the 64K dollar question is, why does Qemu still suck?

Where people sent patches, it doesn't suck (or sucks less). Where they don't, it still sucks. [...]

So is your point that the development process and basic code structure does not matter at all, it's just a matter of people sending patches? I beg to differ ...

Quoted 4 lines

[...] And it cost way more than $64K.

If moving things to tools/ helps, let's move Fedora to tools/.

Those bits of Fedora which deeply relate to the kernel - yes. Those bits that are arguably separate - nope.

Quoted 6 lines

Quoted 6 lines

Quoted 3 lines

How is a patch for the qemu GUI eject button and the kvm shadow mmu related? Should a single maintainer deal with both?

We have co-maintainers for perf that have a different focus. It works pretty well.

Quoted 3 lines

And it works well when I have patches that change x86 core and kvm. But that's no longer a single repository and we have to coordinate.

Actually, it works much better if, contrary to your proposal it ends up in a single repo. Last i checked both of us really worked on such a project, run by some guy. (Named Linus or so.)

Quoted 54 lines

Quoted 8 lines

Look at git log tools/perf/ and how user-space and kernel-space components interact in practice. You'll patches that only impact one side, but you'll see very big overlap both in contributor identity and in patches as well.

Also, let me put similar questions in a bit different way:

- ' how is an in-kernel PIT emulation connected to Qemu's PIT emulation? '

Both implement the same spec. One is be a code derivative of the other (via Xen).

Quoted 3 lines

- ' how is the in-kernel dynticks implementation related to Qemu's implementation of hardware timers? '

The quality of host kernel timers directly determines the quality of qemu's timer emulation.

Quoted 3 lines

- ' how is an in-kernel event for a CD-ROM eject connected to an in-Qemu eject event? '

Both implement the same spec. The kernel of course needs to handle all implementation variants, while qemu only needs to implement it once.

Quoted 3 lines

- ' how is a new hardware virtualization feature related to being able to configure and use it via Qemu? '

Most features (example: npt) are transparent to userspace, some are not. When they are not, we introduce an ioctl() to kvm for controlling the feature, and a command-line switch to qemu for calling it.

Quoted 3 lines

- ' how is the in-kernel x86 decoder/emulator related to the Qemu x86 emulator? '

Both implement the same spec. Note qemu is not an emulator but a binary translator.

Quoted 3 lines

- ' how is the performance of the qemu GUI related to the way VGA buffers are mapped and accelerated by KVM? '

kvm needs to support direct mapping when possible and efficient data transfer when not. The latter will obviously be much slower. When direct mapping is possible, kvm needs to track pages touched by the guest to avoid full screen redraws. The rest (interfacing to X or vnc, implementing emulated hardware acceleration, full-screen mode, etc.) are unrelated.

Quoted 4 lines

Quoted 2 lines

They are obviously deeply related.

Not at all. [...]

You are obviously arguing for something like UML. Fortunately KVM is not that. Or i hope it isnt.

Quoted 3 lines

[...] kvm in fact knows nothing about vga, to take your last example. [...]

Look at the VGA dirty bitmap optimization a'ka the KVM_GET_DIRTY_LOG ioctl.

See qemu/kvm-all.c's kvm_physical_sync_dirty_bitmap().

It started out as a VGA optimization (also used by live migration) and even today it's mostly used by the VGA drivers - albeit a weak one.

I wish there were stronger VGA optimizations implemented, copying the dirty bitmap is not a particularly performant solution.

(although it's certainly
better than full emulation) Graphics performance is one of the more painful aspects of KVM usability today.

Quoted 4 lines

[...] To suggest that qemu needs to be close to the kernel to benefit from the kernel's timer implementation means we don't care about providing quality timing except to ourselves, which luckily isn't the case.

That is not what i said. I said they are closely related, and where technologies are closely related, project proximity turns into project unification at a certain stage.

Quoted 4 lines

Some time ago the various desktops needed directory change notification, and people implemented inotify (or whatever it's called today). No one suggested tools/gnome/ and tools/kde/.

You are misconstruing and misrepresenting my argument - i'd expect better. Gnome and KDE runs on other kernels as well and is generally not considered close to the kernel.

Do you seriously argue that Qemu has nothing to do with KVM these days?

Quoted 8 lines

Quoted 4 lines

The quality of a development process is not defined by the easy cases where no project unification is needed. The quality of a development process is defined by the _difficult_ cases.

That's true, but we don't have issues at the qemu/kvm boundary. Note we do have issues at the qemu/aio interfaces and qemu/net interfaces (out of which vhost-net was born) but these wouldn't be solved by tools/qemu/.

That was not what i suggested. They would be solved by what i proposed:
tools/kvm/, right?

Quoted 3 lines

That is not what i said. I said they are closely related, and where technologies are closely related, project proximity turns into project unification at a certain stage.

I really don't see how. So what if both qemu and kvm implement an i8254? They can't share any code since the internal APIs are so different. [...]

[...] Even worse for the x86 emulator as qemu and kvm are fundamentally different.

Quoted 7 lines · Avi Kivity

Quoted 4 lines

That is not what i said. I said they are closely related, and where technologies are closely related, project proximity turns into project unification at a certain stage.

I really don't see how. So what if both qemu and kvm implement an i8254? They can't share any code since the internal APIs are so different. [...]

I wouldnt jump to assumptions there. perf shares some facilities with the kernel on the source code level - they can be built both in the kernel and in user-space.

But my main thought wasnt even to actually share the implementation - but to actually synchronize when a piece of device emulation moves into the kernel. It is arguably bad for performance in most cases when Qemu handles a given device - so all the common devices should be kernel accelerated.

The version and testing matrix would be simplified significantly as well: as kernel and qemu goes hand in hand, they are always on the same version.

Quoted 3 lines

[...] Even worse for the x86 emulator as qemu and kvm are fundamentally different.

So is it your argument that the difference and the duplication in x86 instruction emulation is a good thing?

You said it some time ago that
the kvm x86 emulator was very messy and you wish it was cleaner.

While qemu's is indeed rather different (it's partly a translator/JIT), i'm sure the decoder logic could be shared - and qemu has a slow-path full-emulation fallback in any case, which is similar to what in-kernel emulator does (IIRC ...).

That might have changed meanwhile.

Quoted 18 lines · Ingo Molnar

Quoted 6 lines · Avi Kivity

Quoted 3 lines

That is not what i said. I said they are closely related, and where technologies are closely related, project proximity turns into project unification at a certain stage.

I really don't see how. So what if both qemu and kvm implement an i8254? They can't share any code since the internal APIs are so different. [...]

I wouldnt jump to assumptions there. perf shares some facilities with the kernel on the source code level - they can be built both in the kernel and in user-space.

But my main thought wasnt even to actually share the implementation - but to actually synchronize when a piece of device emulation moves into the kernel. It is arguably bad for performance in most cases when Qemu handles a given device - so all the common devices should be kernel accelerated.

Quoted 2 lines

The version and testing matrix would be simplified significantly as well:
as kernel and qemu goes hand in hand, they are always on the same version.

So, you propose to allow running tools/kvm/ only on the kernel it was shipped with?

Otherwise the testing matrix isn't simplified.

Quoted 3 lines

[...] kvm in fact knows nothing about vga, to take your last example. [...]

Look at the VGA dirty bitmap optimization a'ka the KVM_GET_DIRTY_LOG ioctl.

See qemu/kvm-all.c's kvm_physical_sync_dirty_bitmap().

It started out as a VGA optimization (also used by live migration) and even today it's mostly used by the VGA drivers - albeit a weak one.

I wish there were stronger VGA optimizations implemented, copying the dirty bitmap is not a particularly performant solution. (although it's certainly better than full emulation) Graphics performance is one of the more painful aspects of KVM usability today.

Quoted 9 lines · Ingo Molnar

Quoted 3 lines

[...] kvm in fact knows nothing about vga, to take your last example. [...]

Look at the VGA dirty bitmap optimization a'ka the KVM_GET_DIRTY_LOG ioctl.

See qemu/kvm-all.c's kvm_physical_sync_dirty_bitmap().

Quoted 7 lines

It started out as a VGA optimization (also used by live migration) and even today it's mostly used by the VGA drivers - albeit a weak one.

I wish there were stronger VGA optimizations implemented, copying the dirty bitmap is not a particularly performant solution. (although it's certainly better than full emulation) Graphics performance is one of the more painful aspects of KVM usability today.

We have to maintain a dirty bitmap because we don't have a paravirtual graphics driver. IOW, someone needs to write an Xorg driver.

Ideally, we could just implement a Linux framebuffer device, right?

Well, we took that approach in Xen and that sucks even worse because the Xorg framebuffer driver doesn't implement any of the optimizations that the Linux framebuffer supports and the Xorg driver does not provide use the kernel's interfaces for providing update regions.

Of course, we need to pull in X into the kernel to fix this, right?

Any sufficiently complicated piece of software is going to interact with a lot of other projects. The solution is not to pull it all into one massive repository. It's to build relationships and to find ways to efficiently work with the various communities.

And we're working on this with X. We'll have a paravirtual graphics driver very soon. There are no magic solutions. We need more developers working on the hard problems.

Quoted 18 lines · Anthony Liguori

Quoted 16 lines · Ingo Molnar

Quoted 3 lines

[...] kvm in fact knows nothing about vga, to take your last example. [...]

Look at the VGA dirty bitmap optimization a'ka the KVM_GET_DIRTY_LOG ioctl.

See qemu/kvm-all.c's kvm_physical_sync_dirty_bitmap().

It started out as a VGA optimization (also used by live migration) and even today it's mostly used by the VGA drivers - albeit a weak one.

I wish there were stronger VGA optimizations implemented, copying the dirty bitmap is not a particularly performant solution. (although it's certainly better than full emulation) Graphics performance is one of the more painful aspects of KVM usability today.

Quoted 5 lines

We have to maintain a dirty bitmap because we don't have a paravirtual graphics driver. IOW, someone needs to write an Xorg driver.

Ideally, we could just implement a Linux framebuffer device, right?

No, you'd want to interact with DRM.

Quoted 6 lines

Xorg framebuffer driver doesn't implement any of the optimizations that the Linux framebuffer supports and the Xorg driver does not provide use the kernel's interfaces for providing update regions.

Of course, we need to pull in X into the kernel to fix this, right?

FYI, this part of X has already been pulled into the kernel, it's called DRM. If then it's being expanded.

Quoted 5 lines

Any sufficiently complicated piece of software is going to interact with a lot of other projects. The solution is not to pull it all into one massive repository. It's to build relationships and to find ways to efficiently work with the various communities.

That's my whole point with this thread: the kernel side of KVM and qemu, but all practical purposes should not be two 'separate communities'. They should be one and the same thing.

The thing is, writing up a DRM connector to a guest Linux OS could be done in no time. It could be deployed to users in no time as well, with the proper development model.

That after years and years of waiting proper GX support is _still_ not implemented in KVM is really telling of the efficiency of development based on such disjoint 'communities'. Maybe put up a committee as well to increase efficiency? ;-)

Ingo

Quoted 20 lines · Ingo Molnar

Quoted 18 lines · Anthony Liguori

Quoted 7 lines · Ingo Molnar

Quoted 2 lines

[...] kvm in fact knows nothing about vga, to take your last example. [...]

Look at the VGA dirty bitmap optimization a'ka the KVM_GET_DIRTY_LOG ioctl.

See qemu/kvm-all.c's kvm_physical_sync_dirty_bitmap().

Quoted 7 lines

It started out as a VGA optimization (also used by live migration) and even today it's mostly used by the VGA drivers - albeit a weak one.

I wish there were stronger VGA optimizations implemented, copying the dirty bitmap is not a particularly performant solution. (although it's certainly better than full emulation) Graphics performance is one of the more painful aspects of KVM usability today.

We have to maintain a dirty bitmap because we don't have a paravirtual graphics driver. IOW, someone needs to write an Xorg driver.

Quoted 2 lines

Quoted 1 line

Ideally, we could just implement a Linux framebuffer device, right?

No, you'd want to interact with DRM.

Using DRM doesn't help very much. You still need an X driver and most of the operations you care about (video rendering, window movement, etc) are not operations that need to go through DRM.

Quoted 4 lines

There are all kernel space projects, going through Xorg would be a horrible waste of performance for full-screen virtualization. It's fine for the windowed or networked case (and good as a compatibility fallback), but very much not fine for local desktop use.

3D graphics virtualization is extremely difficult in the non-passthrough case. It really requires hardware support that isn't widely available today (outside a few NVIDIA chipsets).

Quoted 6 lines

Quoted 5 lines

Xorg framebuffer driver doesn't implement any of the optimizations that the Linux framebuffer supports and the Xorg driver does not provide use the kernel's interfaces for providing update regions.

Of course, we need to pull in X into the kernel to fix this, right?

Quoted 2 lines

FYI, this part of X has already been pulled into the kernel, it's called DRM. If then it's being expanded.

It doesn't provide the things we need to a good user experience. You need things like an absolute input device, host driven display resize, RGBA hardware cursors. None of these go through DRI and it's those things that really provide the graphics user experience.

Quoted 8 lines

Quoted 4 lines

Any sufficiently complicated piece of software is going to interact with a lot of other projects. The solution is not to pull it all into one massive repository. It's to build relationships and to find ways to efficiently work with the various communities.

That's my whole point with this thread: the kernel side of KVM and qemu, but all practical purposes should not be two 'separate communities'. They should be one and the same thing.

I don't know why you keep saying this. The people who are in these "separate communities" keep claiming that they don't feel this way.

I'm not just saying this to be argumentative. Many of the people in the community have thought this same thing, and tried it themselves, and we've all come to the same conclusion.

It's certainly possible that we just missed the obvious thing to do but we'll never know that unless someone shows us.

Quoted 5 lines

Quoted 5 lines

There are all kernel space projects, going through Xorg would be a horrible waste of performance for full-screen virtualization. It's fine for the windowed or networked case (and good as a compatibility fallback), but very much not fine for local desktop use.

For the full-screen case (which is a very common mode of using a guest OS on the desktop) there's not much of window management needed. You need to save/restore as you switch in/out.

Quoted 4 lines

3D graphics virtualization is extremely difficult in the non-passthrough case. It really requires hardware support that isn't widely available today (outside a few NVIDIA chipsets).

Granted it's difficult in the general case.

Quoted 9 lines

Quoted 9 lines

Quoted 6 lines

Xorg framebuffer driver doesn't implement any of the optimizations that the Linux framebuffer supports and the Xorg driver does not provide use the kernel's interfaces for providing update regions.

Of course, we need to pull in X into the kernel to fix this, right?

FYI, this part of X has already been pulled into the kernel, it's called DRM. If then it's being expanded.

Quoted 5 lines

It doesn't provide the things we need to a good user experience. You need things like an absolute input device, host driven display resize, RGBA hardware cursors. None of these go through DRI and it's those things that really provide the graphics user experience.

With KSM the display resize is in the kernel.

Cursor management is not. Yet: i
think it would be a nice feature as the cursor could move even if Xorg is blocked or busy with other things.

Quoted 9 lines

Quoted 9 lines

Quoted 5 lines

Any sufficiently complicated piece of software is going to interact with a lot of other projects. The solution is not to pull it all into one massive repository. It's to build relationships and to find ways to efficiently work with the various communities.

That's my whole point with this thread: the kernel side of KVM and qemu, but all practical purposes should not be two 'separate communities'. They should be one and the same thing.

Quoted 3 lines

I don't know why you keep saying this. The people who are in these "separate communities" keep claiming that they don't feel this way.

If you are not two separate communities but one community, then why do you go through the (somewhat masochistic) self-punishing excercise of keeping the project in two different pieces?

In a distant past Qemu was a separate project and KVM was just a newcomer who used it for fancy stuff. Today as you say(?) the two communities are one and the same. Why not bring it to its logical conclusion?

Quoted 4 lines

I'm not just saying this to be argumentative. Many of the people in the community have thought this same thing, and tried it themselves, and we've all come to the same conclusion.

Quoted 3 lines

It's certainly possible that we just missed the obvious thing to do but we'll never know that unless someone shows us.

I'm not aware of anyone in the past having attempted to move qemu to tools/kvm/ in the uptream kernel repo, and having reported on the experiences with such a contribution setup. (obviously it's not possible at all without heavy cooperation and acceptance from you and Avi, so this will probably remain a thought experiment forever)

If then you must refer to previous attempts to 'strip down' Qemu, right? Those attempts didnt really solve the fundamental problem of project code base separation.

Ingo

Quoted 5 lines

Quoted 4 lines

Quoted 4 lines

There are all kernel space projects, going through Xorg would be a horrible waste of performance for full-screen virtualization. It's fine for the windowed or networked case (and good as a compatibility fallback), but very much not fine for local desktop use.

Quoted 3 lines

For the full-screen case (which is a very common mode of using a guest OS on the desktop) there's not much of window management needed. You need to save/restore as you switch in/out.

I don't think I've ever used full-screen mode with my VMs and I use virtualization on a daily basis.

We hear very infrequently from users using full screen mode.

Quoted 5 lines · Anthony Liguori

Quoted 5 lines

Quoted 4 lines

Quoted 4 lines

There are all kernel space projects, going through Xorg would be a horrible waste of performance for full-screen virtualization. It's fine for the windowed or networked case (and good as a compatibility fallback), but very much not fine for local desktop use.

Quoted 8 lines

Quoted 3 lines

For the full-screen case (which is a very common mode of using a guest OS on the desktop) there's not much of window management needed. You need to save/restore as you switch in/out.

I don't think I've ever used full-screen mode with my VMs and I use virtualization on a daily basis.

We hear very infrequently from users using full screen mode.

Sorry for getting slightly off-topic but I find the above statement interesting.

I don't use virtualization on daily basis but a working, fully integrated full-screen model with VirtualBox was the only reason I bothered to give VMs a second chance. From my point of view, the user experience of earlier versions (e.g. Parallels) was just too painful to live with.

/me crawls back to his hole now...

I find the 'KVM mostly cares about the server, not about the desktop' attitude expressed in this thread troubling.

Quoted 3 lines · Ingo Molnar

I find the 'KVM mostly cares about the server, not about the desktop' attitude expressed in this thread troubling.

It's not kvm, just it's developers (and their employers, where applicable). If you post desktop oriented patches I'm sure they'll be welcome.

Quoted 8 lines · Avi Kivity

Quoted 3 lines · Ingo Molnar

I find the 'KVM mostly cares about the server, not about the desktop' attitude expressed in this thread troubling.

It's not kvm, just it's developers (and their employers, where applicable). If you post desktop oriented patches I'm sure they'll be welcome.

Just such a patch-set was posted in this very thread: 'perf kvm'.

There were two negative reactions immediately, both showed a fundamental server versus desktop bias:

- you did not accept that the most important usecase is when there is a single guest running.

- the reaction to the 'how do we get symbols out of the guest' sub-question was, paraphrased: 'we dont want that due to<unspecified> security threat to XYZ selinux usecase with lots of guests'.

Anyone being aware of how Linux and KVM is being used on the desktop will know how detached that attitude is from the typical desktop usecase ...

Usability _never_ sucks because of lack of patches or lack of suggestions. I bet if you made the next server feature contingent on essential usability fixes they'd happen overnight - for God's sake there's been 1000 commits in the last 3 months in the Qemu repository so there's plenty of manpower...

Usability suckage - and i'm not going to be popular for saying this out loud - almost always shows a basic maintainer disconnect with the real world. See your very first reactions to my 'KVM usability' observations. Read back your and Anthony's replies: total 'sure, patches welcome' kind of indifference. It is _your project_, not some other project down the road ...

So that is my first-hand experience about how you are welcoming these desktop issues, in this very thread. I suspect people try a few times with suggestions, then get shot down like our suggestions were shot down and then give up.

Quoted 5 lines

There were two negative reactions immediately, both showed a fundamental server versus desktop bias:

- you did not accept that the most important usecase is when there is a single guest running.

Well, it isn't.

Quoted 3 lines

- the reaction to the 'how do we get symbols out of the guest' sub-question was, paraphrased: 'we dont want that due to<unspecified> security threat to XYZ selinux usecase with lots of guests'.

When I review a patch, I try to think of the difficult cases, not just the easy case.

Quoted 8 lines

Anyone being aware of how Linux and KVM is being used on the desktop will know how detached that attitude is from the typical desktop usecase ...

Usability _never_ sucks because of lack of patches or lack of suggestions. I bet if you made the next server feature contingent on essential usability fixes they'd happen overnight - for God's sake there's been 1000 commits in the last 3 months in the Qemu repository so there's plenty of manpower...

First of all I am not a qemu maintainer. [...]

[...] Second, from my point of view all contributors are volunteers (perhaps their employer volunteered them, but there's no difference from my perspective). Asking them to repaint my apartment as a condition to get a patch applied is abuse. If a patch is good, it gets applied.

Quoted 6 lines

Usability suckage - and i'm not going to be popular for saying this out loud - almost always shows a basic maintainer disconnect with the real world. See your very first reactions to my 'KVM usability' observations. Read back your and Anthony's replies: total 'sure, patches welcome' kind of indifference. It is _your project_, not some other project down the road ...

I could drop everything and write a gtk GUI for qemu. Is that what you want?

Quoted 8 lines · Avi Kivity

Quoted 6 lines

There were two negative reactions immediately, both showed a fundamental server versus desktop bias:

- you did not accept that the most important usecase is when there is a single guest running.

Well, it isn't.

Erm, my usability points are _doubly_ true when there are multiple guests ...

The inconvenience of having to type:

   perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms \
   --guestmodules=/home/ymzhang/guest/modules top

is very obvious even with a single guest. Now multiply that by more guests ...

Quoted 15 lines · Ingo Molnar

Quoted 6 lines · Avi Kivity

Quoted 5 lines

There were two negative reactions immediately, both showed a fundamental server versus desktop bias:

- you did not accept that the most important usecase is when there is a single guest running.

Well, it isn't.

Erm, my usability points are _doubly_ true when there are multiple guests ...

The inconvenience of having to type:

  perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms \
  --guestmodules=/home/ymzhang/guest/modules top

Quoted 1 line

is very obvious even with a single guest. Now multiply that by more guests ...

If you want to improve this, you need to do the following:

1) Add a userspace daemon that uses vmchannel that runs in the guest and can fetch kallsyms and arbitrary modules. If that daemon lives in tools/perf, that's fine.

Quoted 21 lines · Anthony Liguori

Quoted 19 lines · Ingo Molnar

Quoted 8 lines · Avi Kivity

Quoted 6 lines

There were two negative reactions immediately, both showed a fundamental server versus desktop bias:

- you did not accept that the most important usecase is when there is a single guest running.

Well, it isn't.

Erm, my usability points are _doubly_ true when there are multiple guests ...

The inconvenience of having to type:

   perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms \
   --guestmodules=/home/ymzhang/guest/modules top

is very obvious even with a single guest. Now multiply that by more guests ...

Quoted 6 lines

If you want to improve this, you need to do the following:

1) Add a userspace daemon that uses vmchannel that runs in the guest and can fetch kallsyms and arbitrary modules. If that daemon lives in tools/perf, that's fine.

Adding any new daemon to an existing guest is a deployment and usability nightmare.

The basic rule of good instrumentation is to be transparent. The moment we have to modify the user-space of a guest just to monitor it, the purpose of transparent instrumentation is defeated.

That was one of the fundamental usability mistakes of Oprofile.

There is no 'perf' daemon - all the perf functionality is _built in_, and for very good reasons. It is one of the main reasons for perf's success as well.

Now Qemu is trying to repeat that stupid mistake ...

So please either suggest a different transparent solution that is technically better than the one i suggested, or you should concede the point really.

Please try think with the heads of our users and developers and dont suggest some weird ivory-tower design that is totally impractical ...

And no, you have to code none of this, we'll do all the coding. The only thing we are asking is for you to not stand in the way of good usability ...

Thanks,

Ingo
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 17 lines · Ingo Molnar

Quoted 16 lines · Anthony Liguori

Quoted 15 lines · Ingo Molnar

Quoted 6 lines · Avi Kivity

Quoted 5 lines

There were two negative reactions immediately, both showed a fundamental server versus desktop bias:

- you did not accept that the most important usecase is when there is a single guest running.

Well, it isn't.

Erm, my usability points are _doubly_ true when there are multiple guests ...

The inconvenience of having to type:

  perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms \
  --guestmodules=/home/ymzhang/guest/modules top

is very obvious even with a single guest. Now multiply that by more guests ...

Quoted 8 lines

Quoted 5 lines

If you want to improve this, you need to do the following:

1) Add a userspace daemon that uses vmchannel that runs in the guest and can fetch kallsyms and arbitrary modules. If that daemon lives in tools/perf, that's fine.

Adding any new daemon to an existing guest is a deployment and usability nightmare.

Absolutely. In most cases it is not desirable, and you'll find that in a lot of cases it is not even possible - for non-technical reasons.

One of the main benefits of virtualization is the ability to manage and see things from the outside.

Quoted 3 lines

The basic rule of good instrumentation is to be transparent. The moment we have to modify the user-space of a guest just to monitor it, the purpose of transparent instrumentation is defeated.

Not to mention Heisenbugs and interference.

Frankly, i was surprised (and taken slightly off base) by both Avi and Anthony suggesting such a clearly inferior "add a demon to the guest space" solution. It's a usability and deployment non-starter.

Furthermore, allowing a guest to integrate/mount its files into the host VFS space (which was my suggestion) has many other uses and advantages as well, beyond the instrumentation/symbol-lookup purpose.

So can we please have some resolution here and move on: the KVM maintainers should either suggest a different transparent approach, or should retract the NAK for the solution we suggested.

We very much want to make progress and want to write code, but obviously we cannot code against a maintainer NAK, nor can we code up an inferior solution either.

Quoted 4 lines · Ingo Molnar

Frankly, i was surprised (and taken slightly off base) by both Avi and Anthony suggesting such a clearly inferior "add a demon to the guest space" solution. It's a usability and deployment non-starter.

It's only clearly inferior if you ignore every consideration against it. It's definitely not a deployment non-starter, see the tons of daemons that come with any Linux system. [...]

Quoted 6 lines · Avi Kivity

Quoted 4 lines · Ingo Molnar

Frankly, i was surprised (and taken slightly off base) by both Avi and Anthony suggesting such a clearly inferior "add a demon to the guest space" solution. It's a usability and deployment non-starter.

Quoted 4 lines

It's only clearly inferior if you ignore every consideration against it. It's definitely not a deployment non-starter, see the tons of daemons that come with any Linux system. [...]

Avi, please dont put arguments into my mouth that i never made.

My (clearly expressed) argument was that:

_a new guest-side demon is a transparent instrumentation non-starter_

What is so hard to understand about that simple concept? Instrumentation is good if it's as transparent as possible.

Of course lots of other features can be done via a new user-space package ...

Quoted 10 lines · Avi Kivity

Quoted 4 lines · Ingo Molnar

Frankly, i was surprised (and taken slightly off base) by both Avi and Anthony suggesting such a clearly inferior "add a demon to the guest space" solution. It's a usability and deployment non-starter.

It's only clearly inferior if you ignore every consideration against it. It's definitely not a deployment non-starter, see the tons of daemons that come with any Linux system. [...]

Avi, please dont put arguments into my mouth that i never made.

My (clearly expressed) argument was that:

_a new guest-side demon is a transparent instrumentation non-starter_

Adding any new daemon to an existing guest is a deployment and usability nightmare.

The basic rule of good instrumentation is to be transparent. The moment we have to modify the user-space of a guest just to monitor it, the purpose of transparent instrumentation is defeated.

Please try think with the heads of our users and developers and dont suggest some weird ivory-tower design that is totally impractical ...

And no, you have to code none of this, we'll do all the coding. The only thing we are asking is for you to not stand in the way of good usability ...

Quoted 4 lines · Ingo Molnar

Adding any new daemon to an existing guest is a deployment and usability nightmare.

The logical conclusion of that is that everything should be built into the kernel. Where a failure brings the system down or worse. Where you have to bear the memory footprint whether you ever use the functionality or not. Where to update the functionality you need to deploy a new kernel (possibly introducing unrelated bugs) and reboot.

If userspace daemons are such a deployment and usability nightmare, maybe we should fix that instead.

Quoted 12 lines · Avi Kivity

Quoted 4 lines · Ingo Molnar

Adding any new daemon to an existing guest is a deployment and usability nightmare.

The logical conclusion of that is that everything should be built into the kernel. Where a failure brings the system down or worse. Where you have to bear the memory footprint whether you ever use the functionality or not. Where to update the functionality you need to deploy a new kernel (possibly introducing unrelated bugs) and reboot.

Quoted 3 lines

If userspace daemons are such a deployment and usability nightmare, maybe we should fix that instead.

Which userspace? Deploying *anything* in the guest can be a nightmare, including paravirt drivers if you don't have a natively supported in the OS virtual hardware backoff.

Deploying things in the
host OTOH is business as usual.

And you're smart enough to know that.

Quoted 12 lines · Avi Kivity

Quoted 4 lines · Ingo Molnar

Adding any new daemon to an existing guest is a deployment and usability nightmare.

The logical conclusion of that is that everything should be built into the kernel. Where a failure brings the system down or worse. Where you have to bear the memory footprint whether you ever use the functionality or not. Where to update the functionality you need to deploy a new kernel (possibly introducing unrelated bugs) and reboot.

Quoted 3 lines

If userspace daemons are such a deployment and usability nightmare, maybe we should fix that instead.

Which userspace? Deploying *anything* in the guest can be a nightmare, including paravirt drivers if you don't have a natively supported in the OS virtual hardware backoff.

Deploying things in the
host OTOH is business as usual.

And you're smart enough to know that.

Quoted 11 lines · Olivier Galibert

Quoted 10 lines · Avi Kivity

Quoted 3 lines · Ingo Molnar

Adding any new daemon to an existing guest is a deployment and usability nightmare.

The logical conclusion of that is that everything should be built into the kernel. Where a failure brings the system down or worse. Where you have to bear the memory footprint whether you ever use the functionality or not. Where to update the functionality you need to deploy a new kernel (possibly introducing unrelated bugs) and reboot.

Quoted 5 lines

Quoted 2 lines

If userspace daemons are such a deployment and usability nightmare, maybe we should fix that instead.

Which userspace? Deploying *anything* in the guest can be a nightmare, including paravirt drivers if you don't have a natively supported in the OS virtual hardware backoff.

That includes the guest kernel. If you can deploy a new kernel in the guest, presumably you can deploy a userspace package.

Quoted 2 lines

Deploying things in the
host OTOH is business as usual.

True.

Quoted 1 line

And you're smart enough to know that.

Thanks.

Quoted 1 line

That includes the guest kernel. If you can deploy a new kernel in the guest, presumably you can deploy a userspace package.

That's not always true.
The host admin can control the guest kernel via "kvm -kernel" easily enough, but he may or may not have access to the disk that is used in the guest. (think encrypted disks, service agreements, etc)

Quoted 4 lines · Antoine Martin

Quoted 1 line

That includes the guest kernel. If you can deploy a new kernel in the guest, presumably you can deploy a userspace package.

That's not always true.
The host admin can control the guest kernel via "kvm -kernel" easily enough, but he may or may not have access to the disk that is used in the guest. (think encrypted disks, service agreements, etc)

There is a matching -initrd argument that you can use to launch a daemon.

I believe that -kernel use will be rare, though. It's a lot easier
to keep everything in one filesystem.

Quoted 7 lines · Avi Kivity

Quoted 4 lines · Antoine Martin

Quoted 1 line

That includes the guest kernel. If you can deploy a new kernel in the guest, presumably you can deploy a userspace package.

That's not always true.
The host admin can control the guest kernel via "kvm -kernel" easily enough, but he may or may not have access to the disk that is used in the guest. (think encrypted disks, service agreements, etc)

There is a matching -initrd argument that you can use to launch a daemon.

I thought this discussion was about making it easy to deploy... and generating a custom initrd isn't easy by any means, and it requires access to the guest filesystem (and its mkinitrd tools).

Quoted 2 lines

I believe that -kernel use will be rare, though. It's a lot easier
to keep everything in one filesystem.

Well, for what it's worth, I rarely ever use anything else. My virtual disks are raw so I can loop mount them easily, and I can also switch my guest kernels from outside... without ever needing to mount those disks.

Quoted 3 lines

Well, for what it's worth, I rarely ever use anything else. My virtual disks are raw so I can loop mount them easily, and I can also switch my guest kernels from outside... without ever needing to mount those disks.

Curious, what do you use them for?

btw, if you build your kernel outside the guest, then you already have access to all its symbols, without needing anything further.

Quoted 9 lines · Avi Kivity

Quoted 4 lines

Well, for what it's worth, I rarely ever use anything else. My virtual disks are raw so I can loop mount them easily, and I can also switch my guest kernels from outside... without ever needing to mount those disks.

Curious, what do you use them for?

btw, if you build your kernel outside the guest, then you already have access to all its symbols, without needing anything further.

There's two errors with your argument:

1) you are assuming that it's only about kernel symbols

Look at this 'perf report' output:

# Samples: 7127509216
#
# Overhead Command Shared Object Symbol

# ........  ..........  .............................  ......
#
     19.14%         git  git                            [.] lookup_object
     15.16%        perf  git                            [.] lookup_object
      4.74%        perf  libz.so.1.2.3                  [.] inflate
      4.52%         git  libz.so.1.2.3                  [.] inflate
      4.21%        perf  libz.so.1.2.3                  [.] inflate_table
      3.94%         git  libz.so.1.2.3                  [.] inflate_table
      3.29%         git  git                            [.] find_pack_entry_one
      3.24%         git  libz.so.1.2.3                  [.] inflate_fast
      2.96%        perf  libz.so.1.2.3                  [.] inflate_fast
      2.96%         git  git                            [.] decode_tree_entry
      2.80%        perf  libc-2.11.90.so                [.] __strlen_sse42
      2.56%         git  libc-2.11.90.so                [.] __strlen_sse42
      1.98%        perf  libc-2.11.90.so                [.] __GI_memcpy
      1.71%        perf  git                            [.] decode_tree_entry
      1.53%         git  libc-2.11.90.so                [.] __GI_memcpy
      1.48%         git  git                            [.] lookup_blob
      1.30%         git  git                            [.] process_tree
      1.30%        perf  git                            [.] process_tree
      0.90%        perf  git                            [.] tree_entry
      0.82%        perf  git                            [.] lookup_blob
      0.78%         git  [kernel.kallsyms]              [k] kstat_irqs_cpu

kernel symbols are only a small portion of the symbols. (a single line in this case)

To get to those other symbols we have to read the ELF symbols of those binaries in the guest filesystem, in the post-processing/reporting phase. This is both complex to do and relatively slow so we dont want to (and cannot) do this at sample time from IRQ context or NMI context ...

Also, many aspects of reporting are interactive so it's done lazily or on-demand. So we need ready access to the guest filesystem - for those guests which decide to integrate with the host for this.

2) the 'SystemTap mistake'

You are assuming that the symbols of the kernel when it got built got saved properly and are discoverable easily. In reality those symbols can be erased by a make clean, can be modified by a new build, can be misplaced and can generally be hard to find because each distro puts them in a different installation path.

My 10+ years experience with kernel instrumentation solutions is that kernel-driven, self-sufficient, robust, trustable, well-enumerated sources of information work far better in practice.

The thing is, in this thread i'm forced to repeat the same basic facts again and again. Could you _PLEASE_, pretty please, when it comes to instrumentation details, at least _read the mails_ of the guys who actually ... write and maintain Linux instrumentation code? This is getting ridiculous really.

Quoted 3 lines

My 10+ years experience with kernel instrumentation solutions is that kernel-driven, self-sufficient, robust, trustable, well-enumerated sources of information work far better in practice.

What about line number information? And the source? Into the kernel with them as well?

Quoted 2 lines

What about line number information?  And the source?  Into the kernel with them as well?

Sigh. Please read the _very first_ suggestion i made, which solves all that. I rarely go into discussions without suggesting technical solutions - i'm not interested in flaming, i'm interested in real solutions.

Here it is, repeated for the Nth time:

Allow a guest to (optionally) integrate its VFS namespace with the host side as well. An example scheme would be:

  /guests/Fedora-G1/
  /guests/Fedora-G1/proc/
  /guests/Fedora-G1/usr/
  /guests/Fedora-G1/.../
  /guests/OpenSuse-G2/
  /guests/OpenSuse-G2/proc/
  /guests/OpenSuse-G2/usr/
  /guests/OpenSuse-G2/.../

 ( This feature would be configurable and would be default-off, to maintain the current status quo. )

Quoted 21 lines · Ingo Molnar

Quoted 2 lines

What about line number information?  And the source?  Into the kernel with them as well?

Sigh. Please read the _very first_ suggestion i made, which solves all that. I rarely go into discussions without suggesting technical solutions - i'm not interested in flaming, i'm interested in real solutions.

Here it is, repeated for the Nth time:

Allow a guest to (optionally) integrate its VFS namespace with the host side as well. An example scheme would be:

  /guests/Fedora-G1/
  /guests/Fedora-G1/proc/
  /guests/Fedora-G1/usr/
  /guests/Fedora-G1/.../
  /guests/OpenSuse-G2/
  /guests/OpenSuse-G2/proc/
  /guests/OpenSuse-G2/usr/
  /guests/OpenSuse-G2/.../

Quoted 2 lines

 ( This feature would be configurable and would be default-off, to maintain the current status quo. )

Heh, funny. That would also solve my number one gripe with virtualization these days: how to get files in and out of guests without having to install extra packages on the guest side and fiddling with mount points on every single guest image I want to play with.

Quoted 14 lines

Quoted 14 lines

Here it is, repeated for the Nth time:

Allow a guest to (optionally) integrate its VFS namespace with the host side as well. An example scheme would be:

/guests/Fedora-G1/
/guests/Fedora-G1/proc/
/guests/Fedora-G1/usr/
/guests/Fedora-G1/.../
/guests/OpenSuse-G2/
/guests/OpenSuse-G2/proc/
/guests/OpenSuse-G2/usr/
/guests/OpenSuse-G2/.../

Quoted 7 lines

Quoted 2 lines

( This feature would be configurable and would be default-off, to maintain
the current status quo. )

Heh, funny. That would also solve my number one gripe with virtualization these days: how to get files in and out of guests without having to install extra packages on the guest side and fiddling with mount points on every single guest image I want to play with.

FYI, for offline guests, you can use libguestfs[1] to access & change files inside the guest, and read-only access to running guests files. It provides access via a interactive shell, APIs in all major languages, and also has a FUSE mdule to expose it directly in the host VFS. It could probably be made to work read-write for running guests too if its agent were installed inside the guest & leverage the new Virtio-Serial channel for comms (avoiding any network setup requirements).

Quoted 22 lines · Pekka Enberg

Quoted 21 lines · Ingo Molnar

Quoted 2 lines

What about line number information? ?And the source? ?Into the kernel with them as well?

Sigh. Please read the _very first_ suggestion i made, which solves all that. I rarely go into discussions without suggesting technical solutions - i'm not interested in flaming, i'm interested in real solutions.

Here it is, repeated for the Nth time:

Allow a guest to (optionally) integrate its VFS namespace with the host side as well. An example scheme would be:

? /guests/Fedora-G1/
? /guests/Fedora-G1/proc/
? /guests/Fedora-G1/usr/
? /guests/Fedora-G1/.../
? /guests/OpenSuse-G2/
? /guests/OpenSuse-G2/proc/
? /guests/OpenSuse-G2/usr/
? /guests/OpenSuse-G2/.../

Quoted 7 lines

Quoted 2 lines

?( This feature would be configurable and would be default-off, to maintain ? ?the current status quo. )

Heh, funny. That would also solve my number one gripe with virtualization these days: how to get files in and out of guests without having to install extra packages on the guest side and fiddling with mount points on every single guest image I want to play with.

FYI, for offline guests, you can use libguestfs[1] to access & change files inside the guest, and read-only access to running guests files. It provides access via a interactive shell, APIs in all major languages, and also has a FUSE mdule to expose it directly in the host VFS. It could probably be made to work read-write for running guests too if its agent were installed inside the guest & leverage the new Virtio-Serial channel for comms (avoiding any network setup requirements).

Regards,
Daniel

[1] http://libguestfs.org/

Quoted 9 lines · Daniel P. Berrange

FYI, for offline guests, you can use libguestfs[1] to access & change files inside the guest, and read-only access to running guests files. It provides access via a interactive shell, APIs in all major languages, and also has a FUSE mdule to expose it directly in the host VFS. It could probably be made to work read-write for running guests too if its agent were installed inside the guest & leverage the new Virtio-Serial channel for comms (avoiding any network setup requirements).

Quoted 4 lines

Regards,
Daniel

[1] http://libguestfs.org/

Yes, this is the kind of functionality i'm suggesting.

I'd suggest a different implementation for live guests: to drive this from within the live guest side of KVM, i.e. basically a paravirt driver for guestfs. You'd pass file API guests to the guest directly, via the KVM ioctl or so - and get responses from the guest.

That will give true read-write access and completely coherent (and still transparent) VFS integration, with no host-side knowledge needed for the guest's low level (raw) filesystem structure. That's a big advantage.

Yes, it needs an 'aware' guest kernel - but that is a one-off transition overhead whose cost is zero in the long run. (i.e. all KVM kernels beyond a given version would have this ability - otherwise it's guest side distribution transparent)

Even 'offline' read-only access could be implemented by booting a minimal kernel via qemu -kernel and using a 'ro' boot option. That way you could eliminate all lowlevel filesystem knowledge from libguestfs. You could run ext4 or btrfs guest filesystems and FAT ones as well - with no restriction.

This would allow 'offline' access to Windows images as well: a FAT or ntfs enabled mini-kernel could be booted in read-only mode.

This is close to the way libguestfs already works. It boots QEMU/KVM pointing to a minimal stripped down appliance linux OS image, containing a small agent it talks to over some form of vmchannel/serial/virtio-serial device. Thus the kernel in the appliance it runs is the only thing that needs to know about the filesystem/lvm/dm on-disk formats - libguestfs definitely does not want to be duplicating this detailed knowledge of on disk format itself. It is doing full read-write access to the guest filesystem in offline mode - one of the major use cases is disaster recovery from a unbootable guest OS image.

Quoted 8 lines · Daniel P. Berrange

This is close to the way libguestfs already works. It boots QEMU/KVM pointing to a minimal stripped down appliance linux OS image, containing a small agent it talks to over some form of vmchannel/serial/virtio-serial device. Thus the kernel in the appliance it runs is the only thing that needs to know about the filesystem/lvm/dm on-disk formats - libguestfs definitely does not want to be duplicating this detailed knowledge of on disk format itself. It is doing full read-write access to the guest filesystem in offline mode - one of the major use cases is disaster recovery from a unbootable guest OS image.

As Dan said, the 'daemon' part is separate and could be run as a standard part of a guest install, talking over vmchannel to the host. The only real issue I can see is adding access control to the daemon (currently it doesn't need it and doesn't do any). Doing it this way you'd be leveraging the ~250,000 lines of existing libguestfs code, bindings in multiple languages, tools etc.

Quoted 8 lines · Daniel P. Berrange

This is close to the way libguestfs already works. It boots QEMU/KVM pointing to a minimal stripped down appliance linux OS image, containing a small agent it talks to over some form of vmchannel/serial/virtio-serial device. Thus the kernel in the appliance it runs is the only thing that needs to know about the filesystem/lvm/dm on-disk formats - libguestfs definitely does not want to be duplicating this detailed knowledge of on disk format itself. It is doing full read-write access to the guest filesystem in offline mode - one of the major use cases is disaster recovery from a unbootable guest OS image.

As Dan said, the 'daemon' part is separate and could be run as a standard part of a guest install, talking over vmchannel to the host. The only real issue I can see is adding access control to the daemon (currently it doesn't need it and doesn't do any). Doing it this way you'd be leveraging the ~250,000 lines of existing libguestfs code, bindings in multiple languages, tools etc.

Quoted 37 lines · Ingo Molnar

Quoted 9 lines · Daniel P. Berrange

FYI, for offline guests, you can use libguestfs[1] to access & change files inside the guest, and read-only access to running guests files. It provides access via a interactive shell, APIs in all major languages, and also has a FUSE mdule to expose it directly in the host VFS. It could probably be made to work read-write for running guests too if its agent were installed inside the guest & leverage the new Virtio-Serial channel for comms (avoiding any network setup requirements).

Quoted 4 lines

Regards,
Daniel

[1] http://libguestfs.org/

Yes, this is the kind of functionality i'm suggesting.

I'd suggest a different implementation for live guests: to drive this from within the live guest side of KVM, i.e. basically a paravirt driver for guestfs. You'd pass file API guests to the guest directly, via the KVM ioctl or so - and get responses from the guest.

That will give true read-write access and completely coherent (and still transparent) VFS integration, with no host-side knowledge needed for the guest's low level (raw) filesystem structure. That's a big advantage.

Yes, it needs an 'aware' guest kernel - but that is a one-off transition overhead whose cost is zero in the long run. (i.e. all KVM kernels beyond a given version would have this ability - otherwise it's guest side distribution transparent)

Even 'offline' read-only access could be implemented by booting a minimal kernel via qemu -kernel and using a 'ro' boot option. That way you could eliminate all lowlevel filesystem knowledge from libguestfs. You could run ext4 or btrfs guest filesystems and FAT ones as well - with no restriction.

Quoted 2 lines

This would allow 'offline' access to Windows images as well: a FAT or ntfs enabled mini-kernel could be booted in read-only mode.

This is close to the way libguestfs already works. [...]

[...] It boots QEMU/KVM pointing to a minimal stripped down appliance linux OS image, containing a small agent it talks to over some form of vmchannel/serial/virtio-serial device. Thus the kernel in the appliance it runs is the only thing that needs to know about the filesystem/lvm/dm on-disk formats - libguestfs definitely does not want to be duplicating this detailed knowledge of on disk format itself. It is doing full read-write access to the guest filesystem in offline mode - one of the major use cases is disaster recovery from a unbootable guest OS image.

Just curious: any plans to extend this to include live read/write access as well?

I.e. to have the 'agent' (guestfsd) running universally, so that tools such as perf and by users could rely on the VFS integration as well, not just disaster recovery tools?

Quoted 6 lines · Ingo Molnar

Just curious: any plans to extend this to include live read/write access as well?

I.e. to have the 'agent' (guestfsd) running universally, so that tools such as perf and by users could rely on the VFS integration as well, not just disaster recovery tools?