Enhance perf to collect KVM guest os statistics from host side

Quoted 5 lines · Avi Kivity

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

- Easy default reference to guest instances, and a way for tools to reference them symbolically as well in the multi-guest case. Preferably something trustable and kernel-provided - not some indirect information like a PID file created by libvirt-manager or so.

Quoted 8 lines

Quoted 5 lines

Quoted 3 lines

Usually 'layering violation' is trotted out at such suggestions. [...]

That's weird, how can a feature request be a 'layering violation'?

The 'something trustable and kernel-provided'. The kernel knows nothing about guest names.

The kernel certainly knows about other resources such as task names or network interface names or tracepoint names. This is kernel design 101.

Quoted 5 lines

Quoted 5 lines

If something that users find straightforward and usable is a layering violation to you (such as easily being able to access their own files on the host as well ...) then i think you need to revisit the definition of that term instead of trying to fix the user.

Quoted 14 lines

Here is the explanation, you left it quoted:

Quoted 11 lines

Quoted 11 lines

[...] I don't like using the term, because sometimes the layers are incorrect and need to be violated. But it should be done explicitly, not as a shortcut for a minor feature (and profiling is a minor feature, most users will never use it, especially guest-from-host).

The fact is we have well defined layers today, kvm virtualizes the cpu and memory, qemu emulates devices for a single guest, libvirt manages guests. We break this sometimes but there has to be a good reason. So perf needs to talk to libvirt if it wants names. Could be done via linking, or can be done using a pluging libvirt drops into perf.

This is really just the much-discredited microkernel approach for keeping global enumeration data that should be kept by the kernel ...

Lets look at the ${HOME}/.qemu/qmp/ enumeration method suggested by Anthony. There's numerous ways that this can break:

- Those special files can get corrupted, mis-setup, get out of sync, or can be hard to discover.

- The ${HOME}/.qemu/qmp/ solution suggested by Anthony has a very obvious design flaw: it is per user. When i'm root i'd like to query _all_ current guest images, not just the ones started by root. A system might not even have a notion of '${HOME}'.

- Apps might start KVM vcpu instances without adhering to the

    ${HOME}/.qemu/qmp/ access method.

- There is no guarantee for the Qemu process to reply to a request - while the kernel can always guarantee an enumeration result. I dont want 'perf kvm' to hang or misbehave just because Qemu has hung.

Really, for such reasons user-space is pretty poor at doing system-wide enumeration and resource management. Microkernels lost for a reason.

You are committing several grave design mistakes here.

Thanks,

Ingo

Quoted 44 lines · Ingo Molnar

Quoted 9 lines · Avi Kivity

Quoted 7 lines

Quoted 6 lines

Quoted 4 lines

- Easy default reference to guest instances, and a way for tools to reference them symbolically as well in the multi-guest case. Preferably something trustable and kernel-provided - not some indirect information like a PID file created by libvirt-manager or so.

Usually 'layering violation' is trotted out at such suggestions. [...]

That's weird, how can a feature request be a 'layering violation'?

The 'something trustable and kernel-provided'. The kernel knows nothing about guest names.

The kernel certainly knows about other resources such as task names or network interface names or tracepoint names. This is kernel design 101.

Quoted 4 lines

Quoted 4 lines

If something that users find straightforward and usable is a layering violation to you (such as easily being able to access their own files on the host as well ...) then i think you need to revisit the definition of that term instead of trying to fix the user.

Quoted 12 lines

Here is the explanation, you left it quoted:

Quoted 10 lines

Quoted 10 lines

[...] I don't like using the term, because sometimes the layers are incorrect and need to be violated. But it should be done explicitly, not as a shortcut for a minor feature (and profiling is a minor feature, most users will never use it, especially guest-from-host).

The fact is we have well defined layers today, kvm virtualizes the cpu and memory, qemu emulates devices for a single guest, libvirt manages guests. We break this sometimes but there has to be a good reason. So perf needs to talk to libvirt if it wants names. Could be done via linking, or can be done using a pluging libvirt drops into perf.

This is really just the much-discredited microkernel approach for keeping global enumeration data that should be kept by the kernel ...

Lets look at the ${HOME}/.qemu/qmp/ enumeration method suggested by Anthony. There's numerous ways that this can break:

- Those special files can get corrupted, mis-setup, get out of sync, or can be hard to discover.

- The ${HOME}/.qemu/qmp/ solution suggested by Anthony has a very obvious design flaw: it is per user. When i'm root i'd like to query _all_ current guest images, not just the ones started by root. A system might not even have a notion of '${HOME}'.

Quoted 2 lines

- Apps might start KVM vcpu instances without adhering to the ${HOME}/.qemu/qmp/ access method.

Not all KVM vcpus are running operating systems.

Transitive had a product that was using a KVM context to run their binary translator which allowed them full access to the host processes virtual address space range. In this case, there is no kernel and there are no devices.

Quoted 5 lines

Transitive had a product that was using a KVM context to run their binary translator which allowed them full access to the host processes virtual address space range. In this case, there is no kernel and there are no devices.

And your point is that such vcpus should be excluded from profiling just because they fall outside the Qemu/libvirt umbrella?

Ingo

Quoted 7 lines · Ingo Molnar

Quoted 4 lines

Transitive had a product that was using a KVM context to run their binary translator which allowed them full access to the host processes virtual address space range. In this case, there is no kernel and there are no devices.

And your point is that such vcpus should be excluded from profiling just because they fall outside the Qemu/libvirt umbrella?

You don't instrument it the way you'd instrument an operating system so no, you don't want it to show up in perf kvm top.

Quoted 7 lines · Anthony Liguori

Quoted 5 lines · Ingo Molnar

Quoted 5 lines

Transitive had a product that was using a KVM context to run their binary translator which allowed them full access to the host processes virtual address space range. In this case, there is no kernel and there are no devices.

Quoted 6 lines

Quoted 3 lines

And your point is that such vcpus should be excluded from profiling just because they fall outside the Qemu/libvirt umbrella?

You don't instrument it the way you'd instrument an operating system so no, you don't want it to show up in perf kvm top.

Erm, why not? It's executing a virtualized CPU, so sure it makes sense to allow the profiling of it!

Quoted 6 lines

Transitive had a product that was using a KVM context to run their binary translator which allowed them full access to the host processes virtual address space range. In this case, there is no kernel and there are no devices.

And your point is that such vcpus should be excluded from profiling just because they fall outside the Qemu/libvirt umbrella?

That is a ridiculous position.

Quoted 6 lines · Ingo Molnar

Quoted 5 lines

Transitive had a product that was using a KVM context to run their binary translator which allowed them full access to the host processes virtual address space range. In this case, there is no kernel and there are no devices.

Quoted 5 lines

And your point is that such vcpus should be excluded from profiling just because they fall outside the Qemu/libvirt umbrella?

That is a ridiculous position.

Non-guest vcpus will not be able to provide Linux-style symbols.

Quoted 9 lines

Quoted 7 lines

And your point is that such vcpus should be excluded from profiling just because they fall outside the Qemu/libvirt umbrella?

That is a ridiculous position.

Non-guest vcpus will not be able to provide Linux-style symbols.

And why do you say that it makes no sense to profile them?

Also, why do you define 'guest vcpus' to be 'Qemu started guest vcpus'? If some other KVM using project (which you encouraged just a few mails ago) starts a vcpu we still want to be able to profile them.

This is really just the much-discredited microkernel approach for keeping global enumeration data that should be kept by the kernel ...

Lets look at the ${HOME}/.qemu/qmp/ enumeration method suggested by Anthony. There's numerous ways that this can break:

- Those special files can get corrupted, mis-setup, get out of sync, or can be hard to discover.

- The ${HOME}/.qemu/qmp/ solution suggested by Anthony has a very obvious design flaw: it is per user. When i'm root i'd like to query _all_ current guest images, not just the ones started by root. A system might not even have a notion of '${HOME}'.

- Apps might start KVM vcpu instances without adhering to the

    ${HOME}/.qemu/qmp/ access method.

- There is no guarantee for the Qemu process to reply to a request - while the kernel can always guarantee an enumeration result. I dont want 'perf kvm' to hang or misbehave just because Qemu has hung.

Quoted 17 lines · Ingo Molnar

This is really just the much-discredited microkernel approach for keeping global enumeration data that should be kept by the kernel ...

Lets look at the ${HOME}/.qemu/qmp/ enumeration method suggested by Anthony. There's numerous ways that this can break:

- Those special files can get corrupted, mis-setup, get out of sync, or can be hard to discover.

- The ${HOME}/.qemu/qmp/ solution suggested by Anthony has a very obvious design flaw: it is per user. When i'm root i'd like to query _all_ current guest images, not just the ones started by root. A system might not even have a notion of '${HOME}'.

- Apps might start KVM vcpu instances without adhering to the ${HOME}/.qemu/qmp/ access method.

Quoted 3 lines

- There is no guarantee for the Qemu process to reply to a request - while the kernel can always guarantee an enumeration result. I dont want 'perf kvm' to hang or misbehave just because Qemu has hung.

If your position basically boils down to, we can't trust userspace and we can always trust the kernel, I want to eliminate any userspace path, then I can't really help you out.

Quoted 5 lines

If your position basically boils down to, we can't trust userspace and we can always trust the kernel, I want to eliminate any userspace path, then I can't really help you out.

Why would you want to 'help me out'? I can tell a good solution from a bad one just fine.

You should instead read the long list of disadvantages above, invert them and list then as advantages for the kernel-based vcpu enumeration solution, apply common sense and go admit to yourself that indeed in this situation a kernel provided enumeration of vcpu contexts is the most robust solution.

Quoted 7 lines · Ingo Molnar

Quoted 4 lines

If your position basically boils down to, we can't trust userspace and we can always trust the kernel, I want to eliminate any userspace path, then I can't really help you out.

Why would you want to 'help me out'? I can tell a good solution from a bad one just fine.

You are basically making a kernel implementation a requirement, instead of something that follows from the requirement.

Quoted 5 lines

You should instead read the long list of disadvantages above, invert them and list then as advantages for the kernel-based vcpu enumeration solution, apply common sense and go admit to yourself that indeed in this situation a kernel provided enumeration of vcpu contexts is the most robust solution.

Having qemu enumerate guests one way or another is not a good idea IMO since it is focused on one guest and doesn't have a system-wide entity. A userspace system-wide entity will work just as well as kernel implementation, without its disadvantages.

Quoted 7 lines

Quoted 7 lines

You should instead read the long list of disadvantages above, invert them and list then as advantages for the kernel-based vcpu enumeration solution, apply common sense and go admit to yourself that indeed in this situation a kernel provided enumeration of vcpu contexts is the most robust solution.

Quoted 5 lines

Having qemu enumerate guests one way or another is not a good idea IMO since it is focused on one guest and doesn't have a system-wide entity. A userspace system-wide entity will work just as well as kernel implementation, without its disadvantages.

A system-wide user-space entity only solves one problem out of the 4 i listed, still leaving the other 3:

- Those special files can get corrupted, mis-setup, get out of sync, or can be hard to discover.

- Apps might start KVM vcpu instances without adhering to the system-wide access method.

- There is no guarantee for the system-wide process to reply to a request - while the kernel can always guarantee an enumeration result. I dont want 'perf kvm' to hang or misbehave just because the system-wide entity has hung.

Really, i think i have to give up and not try to convince you guys about this anymore - i dont think you are arguing constructively anymore and i dont want yet another pointless flamewar about this.

Please consider 'perf kvm' scrapped indefinitely, due to lack of robust KVM instrumentation features: due to lack of robust+universal vcpu/guest enumeration and due to lack of robust+universal symbol access on the KVM side. It was a really promising feature IMO and i invested two days of arguments into it trying to find a workable solution, but it was not to be.

Quoted 6 lines · Ingo Molnar

Quoted 4 lines

If your position basically boils down to, we can't trust userspace and we can always trust the kernel, I want to eliminate any userspace path, then I can't really help you out.

Why would you want to 'help me out'? I can tell a good solution from a bad one just fine.

You are basically making a kernel implementation a requirement, instead of something that follows from the requirement.

Quoted 4 lines

You should instead read the long list of disadvantages above, invert them and list then as advantages for the kernel-based vcpu enumeration solution, apply common sense and go admit to yourself that indeed in this situation a kernel provided enumeration of vcpu contexts is the most robust solution.

Having qemu enumerate guests one way or another is not a good idea IMO since it is focused on one guest and doesn't have a system-wide entity.

A userspace system-wide entity will work just as well as kernel
implementation, without its disadvantages.

Quoted 1 line

Having qemu enumerate guests one way or another is not a good idea IMO since it is focused on one guest and doesn't have a system-wide entity.

There always needs to be a system wide entity. There are two ways to enumerate instances from that system wide entity. You can centralize the creation of instances and there by maintain an list of current instances. You can also allow instances to be created in a decentralized manner and provide a standard mechanism for instances to register themselves with the system wide entity.

IOW, it's the difference between asking libvirtd to exec(qemu) vs allowing a user to exec(qemu) and having qemu connect to a well known unix domain socket for libvirt to tell libvirtd that it exists.

The later approach has a number of advantages. libvirt already supports both models. The former is the '/system' uri and the later is the '/session' uri.

What I'm proposing, is to use the host file system as the system wide entity instead of libvirtd. libvirtd can monitor the host file system to participate in these activities but ultimately, moving this functionality out of libvirtd means that it becomes the standard mechanism for all qemu instances regardless of how they're launched.

Quoted 5 lines · Anthony Liguori

Quoted 1 line

Having qemu enumerate guests one way or another is not a good idea IMO since it is focused on one guest and doesn't have a system-wide entity.

There always needs to be a system wide entity. There are two ways to enumerate instances from that system wide entity. You can centralize the creation of instances and there by maintain an list of current instances. You can also allow instances to be created in a decentralized manner and provide a standard mechanism for instances to register themselves with the system wide entity.

Quoted 5 lines

IOW, it's the difference between asking libvirtd to exec(qemu) vs allowing a user to exec(qemu) and having qemu connect to a well known unix domain socket for libvirt to tell libvirtd that it exists.

The later approach has a number of advantages. libvirt already supports both models. The former is the '/system' uri and the later is the '/session' uri.

What I'm proposing, is to use the host file system as the system wide entity instead of libvirtd. libvirtd can monitor the host file system to participate in these activities but ultimately, moving this functionality out of libvirtd means that it becomes the standard mechanism for all qemu instances regardless of how they're launched.

I don't like dropping sockets into the host filesystem, especially as they won't be cleaned up on abnormal exit. I also think this breaks our 'mechanism, not policy' policy. Someone may want to do something weird with qemu that doesn't work well with this.

We could allow starting monitors from the global configuration file, so a distribution can do this if it wants, but I don't think we should do this ourselves by default.

Quoted 2 lines

Having qemu enumerate guests one way or another is not a good idea IMO since it is focused on one guest and doesn't have a system-wide entity.

There always needs to be a system wide entity. There are two ways to enumerate instances from that system wide entity. You can centralize the creation of instances and there by maintain an list of current instances. You can also allow instances to be created in a decentralized manner and provide a standard mechanism for instances to register themselves with the system wide entity.

IOW, it's the difference between asking libvirtd to exec(qemu) vs allowing a user to exec(qemu) and having qemu connect to a well known unix domain socket for libvirt to tell libvirtd that it exists.

Am 22.03.2010 23:06, schrieb Anthony Liguori:

Quoted 12 lines · Avi Kivity

Quoted 4 lines

Having qemu enumerate guests one way or another is not a good idea IMO since it is focused on one guest and doesn't have a system-wide entity.

There always needs to be a system wide entity. There are two ways to enumerate instances from that system wide entity. You can centralize the creation of instances and there by maintain an list of current instances. You can also allow instances to be created in a decentralized manner and provide a standard mechanism for instances to register themselves with the system wide entity.

Quoted 4 lines

IOW, it's the difference between asking libvirtd to exec(qemu) vs allowing a user to exec(qemu) and having qemu connect to a well known unix domain socket for libvirt to tell libvirtd that it exists.

I think the latter is exactly what I would want for myself. I do see the advantages of having a central instance, but I really don't want to bother with libvirt configuration files or even GUIs just to get an ad-hoc VM up when I can simply run "qemu -hda hd.img -m 1024". Let alone that I usually want to have full control over qemu, including monitor access and small details available as command line options.

I know that I'm not the average user with these requirements, but still I am one user and do have these requirements. If I could just install libvirt, continue using qemu as I always did and libvirt picked my VMs up for things like global enumeration, that would be more or less the optimal thing for me.

Kevin
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

There always needs to be a system wide entity. There are two ways to enumerate instances from that system wide entity. You can centralize the creation of instances and there by maintain an list of current instances. You can also allow instances to be created in a decentralized manner and provide a standard mechanism for instances to register themselves with the system wide entity.

Quoted 7 lines · Anthony Liguori

There always needs to be a system wide entity. There are two ways to enumerate instances from that system wide entity. You can centralize the creation of instances and there by maintain an list of current instances. You can also allow instances to be created in a decentralized manner and provide a standard mechanism for instances to register themselves with the system wide entity.

And this system wide entity is the kvm module. It creates instances of 'struct kvm' and destroys them. I see no problem if we just attach a name to every instance with a good default value like kvm0, kvm1 ... or guest0, guest1 ... User-space can override the name if it wants. The kvm module takes care about the names being unique.

This is very much the same as network card numbering is implemented in the kernel.
Forcing perf to talk to qemu or even libvirt produces to much overhead imho. Instrumentation only produces useful results with low overhead.

Quoted 6 lines · Joerg Roedel

And this system wide entity is the kvm module. It creates instances of 'struct kvm' and destroys them. I see no problem if we just attach a name to every instance with a good default value like kvm0, kvm1 ... or guest0, guest1 ... User-space can override the name if it wants. The kvm module takes care about the names being unique.

So, two users can't have a guest named MyGuest each? What about namespace support? There's a lot of work in virtualizing all kernel namespaces, you're adding to that.

What about notifications when guests are added or removed?

Quoted 5 lines

This is very much the same as network card numbering is implemented in the kernel.
Forcing perf to talk to qemu or even libvirt produces to much overhead imho. Instrumentation only produces useful results with low overhead.

It's a setup cost only.

Sidenote: I really think we should come to a conclusion about the
concept. KVM integration into perf is very useful feature to analyze virtualization workloads.

Quoted 3 lines · Avi Kivity

So, two users can't have a guest named MyGuest each?  What about namespace support?  There's a lot of work in virtualizing all kernel namespaces, you're adding to that.

This enumeration is a very small and non-intrusive feature. Making it aware of namespaces is easy too.

Quoted 10 lines · Avi Kivity

Quoted 7 lines · Joerg Roedel

And this system wide entity is the kvm module. It creates instances of 'struct kvm' and destroys them. I see no problem if we just attach a name to every instance with a good default value like kvm0, kvm1 ... or guest0, guest1 ... User-space can override the name if it wants. The kvm module takes care about the names being unique.

Quoted 4 lines

So, two users can't have a guest named MyGuest each? What about namespace support? There's a lot of work in virtualizing all kernel namespaces, you're adding to that.

This enumeration is a very small and non-intrusive feature. Making it aware of namespaces is easy too.

Quoted 2 lines

What about notifications when guests are added or removed?

Who would be the consumer of such notifications? A 'perf kvm list' can live without I guess. If we need them later we can still add them.

Quoted 6 lines

Quoted 6 lines

This is very much the same as network card numbering is implemented in the kernel.
Forcing perf to talk to qemu or even libvirt produces to much overhead imho. Instrumentation only produces useful results with low overhead.

Quoted 2 lines

It's a setup cost only.

My statement was not limited to enumeration, I should have been more clear about that. The guest filesystem access-channel is another affected part. The 'perf kvm top' command will access the guest filesystem regularly and going over qemu would be more overhead here.

Providing this in the KVM module directly also has the benefit that it would work out-of-the-box with different userspaces too. Or do we want to limit 'perf kvm' to the libvirt-qemu-kvm software stack?

Sidenote: I really think we should come to a conclusion about the
concept. KVM integration into perf is very useful feature to
analyze virtualization workloads.

Quoted 3 lines · Joerg Roedel

This enumeration is a very small and non-intrusive feature. Making it aware of namespaces is easy too.

It's easier (and safer and all the other boring bits) not to do it at all in the kernel.

Quoted 2 lines

Who would be the consumer of such notifications? A 'perf kvm list' can live without I guess. If we need them later we can still add them.

System-wide monitoring needs to work equally well for guests started before or after the monitor.

Even disregarding that, if you introduce an API, people will start using it and complaining if it's incomplete.

Quoted 5 lines

My statement was not limited to enumeration, I should have been more clear about that. The guest filesystem access-channel is another affected part. The 'perf kvm top' command will access the guest filesystem regularly and going over qemu would be more overhead here.

Why? Also, the real cost would be accessing the filesystem, not copying data over qemu.

Quoted 3 lines

Providing this in the KVM module directly also has the benefit that it would work out-of-the-box with different userspaces too. Or do we want to limit 'perf kvm' to the libvirt-qemu-kvm software stack?

Other userspaces can also provide this functionality, like they have to provide disk, network, and display emulation. The kernel is not a huge library.

Quoted 9 lines · Avi Kivity

Quoted 4 lines · Joerg Roedel

This enumeration is a very small and non-intrusive feature. Making it aware of namespaces is easy too.

It's easier (and safer and all the other boring bits) not to do it at all in the kernel.

For the KVM stack is doesn't matter where it is implemented. It is as easy in qemu or libvirt as in the kernel. I also don't see big risks. On the perf side and for its users it is a lot easier to have this in the kernel.
I for example always use plain qemu when running kvm guests and never used libvirt. The only central entity I have here is the kvm kernel modules. I don't want to start using it only to be able to use perf kvm.

Quoted 6 lines

Quoted 3 lines

Who would be the consumer of such notifications? A 'perf kvm list' can live without I guess. If we need them later we can still add them.

System-wide monitoring needs to work equally well for guests started before or after the monitor.

Could be easily done using notifier chains already in the kernel. Probably implemented with much less than 100 lines of additional code.

Quoted 3 lines

Even disregarding that, if you introduce an API, people will start using it and complaining if it's incomplete.

There is nothing wrong with that. We only need to define what this API should be used for to prevent rank growth. It could be an instrumentation-only API for example.

Quoted 6 lines

Quoted 6 lines

My statement was not limited to enumeration, I should have been more clear about that. The guest filesystem access-channel is another affected part. The 'perf kvm top' command will access the guest filesystem regularly and going over qemu would be more overhead here.

Quoted 3 lines

Why? Also, the real cost would be accessing the filesystem, not copying data over qemu.

When measuring cache-misses any additional (and in this case unnecessary) copy-overhead result in less appropriate results.

Quoted 4 lines

Quoted 4 lines

Providing this in the KVM module directly also has the benefit that it would work out-of-the-box with different userspaces too. Or do we want to limit 'perf kvm' to the libvirt-qemu-kvm software stack?

Quoted 4 lines

Other userspaces can also provide this functionality, like they have to provide disk, network, and display emulation. The kernel is not a huge library.

This has nothing to do with a library. It is about entity and resource management which is what os kernels are about. The virtual machine is the entity (similar to a process) and we want to add additional access channels and names to it.

On 03/24/2010 01:59 PM, Joerg Roedel wrote:

You can always provide the kernel and module paths as command line parameters. It just won't be transparently usable, but if you're using qemu from the command line, presumably you can live with that.

Quoted 2 lines

Could be easily done using notifier chains already in the kernel. Probably implemented with much less than 100 lines of additional code.

And a userspace interface for that.

If we make an API, I'd like it to be generally useful.

It's a total headache. For example, we'd need security module hooks to determine access permissions. So far we managed to avoid that since kvm doesn't allow you to access any information beyond what you provided it directly.

Copying the objects is a one time cost. If you run perf for more than a second or two, it would fetch and cache all of the data. It's really the same problem with non-guest profiling, only magnified a bit.

Quoted 3 lines

Quoted 3 lines

Other userspaces can also provide this functionality, like they have to provide disk, network, and display emulation. The kernel is not a huge library.

kvm.ko has only a small subset of the information that is used to define a guest.

Quoted 5 lines

You can always provide the kernel and module paths as command line parameters. It just won't be transparently usable, but if you're using qemu from the command line, presumably you can live with that.

I don't want the tool for myself only. A typical perf user expects that it works transparent.

Quoted 5 lines

Quoted 3 lines

Could be easily done using notifier chains already in the kernel. Probably implemented with much less than 100 lines of additional code.

And a userspace interface for that.

Not necessarily. The perf event is configured to measure systemwide kvm by userspace. The kernel side of perf takes care that it stays system-wide even with added vm instances. So in this case the consumer for the notifier would be the perf kernel part. No userspace interface required.

Quoted 2 lines

If we make an API, I'd like it to be generally useful.

Thats hard to do at this point since we don't know what people will use it for. We should keep it simple in the beginning and add new features as they are requested and make sense in this context.

Quoted 5 lines

It's a total headache. For example, we'd need security module hooks to determine access permissions. So far we managed to avoid that since kvm doesn't allow you to access any information beyond what you provided it directly.

Depends on how it is designed. A filesystem approach was already mentioned. We could create /sys/kvm/ for example to expose information about virtual machines to userspace. This would not require any new security hooks.

Quoted 4 lines

Copying the objects is a one time cost. If you run perf for more than a second or two, it would fetch and cache all of the data. It's really the same problem with non-guest profiling, only magnified a bit.

I don't think we can cache filesystem data of a running guest on the host. It is too hard to keep such a cache coherent.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

Other userspaces can also provide this functionality, like they have to provide disk, network, and display emulation. The kernel is not a huge library.

If two userspaces run in parallel what is the single instance where perf can get a list of guests from?

Quoted 3 lines

kvm.ko has only a small subset of the information that is used to define a guest.

The subset is not small. It contains all guest vcpus, the complete interrupt routing hardware emulation and manages event the guests memory.

Quoted 2 lines · Joerg Roedel

I don't want the tool for myself only. A typical perf user expects that it works transparent.

A typical kvm user uses libvirt, so we can integrate it with that.

Someone needs to know about the new guest to fetch its symbols. Or do you want that part in the kernel too?

Quoted 4 lines

Depends on how it is designed. A filesystem approach was already mentioned. We could create /sys/kvm/ for example to expose information about virtual machines to userspace. This would not require any new security hooks.

Who would set the security context on those files?

Plus, we need cgroup support so you can't see one container's guests from an unrelated container.

Integration with qemu would allow perf to tell us that the guest is hitting the interrupt status register of a virtio-blk device in pci slot 5 (the information is already available through the kvm_mmio trace event, but only qemu can decode it).

Quoted 6 lines · Avi Kivity

Quoted 3 lines · Joerg Roedel

I don't want the tool for myself only. A typical perf user expects that it works transparent.

Quoted 2 lines

A typical kvm user uses libvirt, so we can integrate it with that.

Someone who uses libvirt and virt-manager by default is probably not interested in this feature at the same level a kvm developer is. And developers tend not to use libvirt for low-level kvm development. A number of developers have stated in this thread already that they would appreciate a solution for guest enumeration that would not involve libvirt.

Quoted 3 lines

Someone needs to know about the new guest to fetch its symbols. Or do you want that part in the kernel too?

The samples will be tagged with the guest-name (and some additional information perf needs). Perf userspace can access the symbols then through /sys/kvm/guest0/fs/...

Quoted 7 lines

Quoted 5 lines

Depends on how it is designed. A filesystem approach was already mentioned. We could create /sys/kvm/ for example to expose information about virtual machines to userspace. This would not require any new security hooks.

Who would set the security context on those files?

An approach like: "The files are owned and only readable by the same user that started the vm." might be a good start. So a user can measure its own guests and root can measure all of them.

Quoted 3 lines

Plus, we need cgroup support so you can't see one container's guests from an unrelated container.

cgroup support is an issue but we can solve that too. Its in general still less complex than going through the whole libvirt-qemu-kvm stack.

Quoted 5 lines

Integration with qemu would allow perf to tell us that the guest is hitting the interrupt status register of a virtio-blk device in pci slot 5 (the information is already available through the kvm_mmio trace event, but only qemu can decode it).

Yeah that would be interesting information. But it is more related to tracing than to pmu measurements.
The information which you mentioned above are probably better captured by an extension of trace-events to userspace.

Quoted 6 lines · Joerg Roedel

Someone who uses libvirt and virt-manager by default is probably not interested in this feature at the same level a kvm developer is. And developers tend not to use libvirt for low-level kvm development. A number of developers have stated in this thread already that they would appreciate a solution for guest enumeration that would not involve libvirt.

So would I.

But when I weigh the benefit of truly transparent system-wide perf integration for users who don't use libvirt but do use perf, versus the cost of transforming kvm from a single-process API to a system-wide API with all the complications that I've listed, it comes out in favour of not adding the API.

Quoted 3 lines

The samples will be tagged with the guest-name (and some additional information perf needs). Perf userspace can access the symbols then through /sys/kvm/guest0/fs/...

I take that as a yes? So we need a virtio-serial client in the kernel (which might be exploitable by a malicious guest if buggy) and a fs-over-virtio-serial client in the kernel (also exploitable).

Quoted 3 lines

An approach like: "The files are owned and only readable by the same user that started the vm." might be a good start. So a user can measure its own guests and root can measure all of them.

That's not how sVirt works. sVirt isolates a user's VMs from each other, so if a guest breaks into qemu it can't break into other guests owned by the same user.

Quoted 4 lines

Yeah that would be interesting information. But it is more related to tracing than to pmu measurements. The information which you mentioned above are probably better captured by an extension of trace-events to userspace.

It's all related. You start with perf, see a problem with mmio, call up a histogram of mmio or interrupts or whatever, then zoom in on the misbehaving device.

Quoted 7 lines

But when I weigh the benefit of truly transparent system-wide perf integration for users who don't use libvirt but do use perf, versus the cost of transforming kvm from a single-process API to a system-wide API with all the complications that I've listed, it comes out in favour of not adding the API.

Its not a transformation, its an extension. The current per-process /dev/kvm stays mostly untouched. Its all about having something like this:

$ cd /sys/kvm/guest0
$ ls -l
-r-------- 1 root root 0 2009-08-17 12:05 name
dr-x------ 1 root root 0 2009-08-17 12:05 fs
$ cat name
guest0
$ # ...

The fs/ directory is used as the mount point for the guest root fs.

Quoted 4 lines

Quoted 4 lines

The samples will be tagged with the guest-name (and some additional information perf needs). Perf userspace can access the symbols then through /sys/kvm/guest0/fs/...

Quoted 4 lines

I take that as a yes? So we need a virtio-serial client in the kernel (which might be exploitable by a malicious guest if buggy) and a fs-over-virtio-serial client in the kernel (also exploitable).

What I meant was: perf-kernel puts the guest-name into every sample and perf-userspace accesses /sys/kvm/guest_name/fs/ later to resolve the symbols. I leave the question of how the guest-fs is exposed to the host out of this discussion. We should discuss this seperatly.

Quoted 4 lines

Quoted 4 lines

An approach like: "The files are owned and only readable by the same user that started the vm." might be a good start. So a user can measure its own guests and root can measure all of them.

Quoted 4 lines

That's not how sVirt works. sVirt isolates a user's VMs from each other, so if a guest breaks into qemu it can't break into other guests owned by the same user.

If a vm breaks into qemu it can access the host file system which is the bigger problem. In this case there is no isolation anymore. From that context it can even kill other VMs of the same user independent of a hypothetical /sys/kvm/.

Quoted 5 lines

Quoted 5 lines

Yeah that would be interesting information. But it is more related to tracing than to pmu measurements. The information which you mentioned above are probably better captured by an extension of trace-events to userspace.

Quoted 4 lines

It's all related. You start with perf, see a problem with mmio, call up a histogram of mmio or interrupts or whatever, then zoom in on the misbehaving device.

Yes, but its different from the implementation point-of-view. For the user it surely all plays together.

Quoted 8 lines · Joerg Roedel

$ cd /sys/kvm/guest0
$ ls -l
-r-------- 1 root root 0 2009-08-17 12:05 name
dr-x------ 1 root root 0 2009-08-17 12:05 fs
$ cat name
guest0
$ # ...

Quoted 1 line

The fs/ directory is used as the mount point for the guest root fs.

The problem is /sys/kvm, not /sys/kvm/fs.

Quoted 4 lines

What I meant was: perf-kernel puts the guest-name into every sample and perf-userspace accesses /sys/kvm/guest_name/fs/ later to resolve the symbols. I leave the question of how the guest-fs is exposed to the host out of this discussion. We should discuss this seperatly.

How I see it: perf-kernel puts the guest pid into every sample, and perf-userspace uses that to resolve to a mountpoint served by fuse, or to a unix domain socket that serves the files.

Quoted 4 lines

If a vm breaks into qemu it can access the host file system which is the bigger problem. In this case there is no isolation anymore. From that context it can even kill other VMs of the same user independent of a hypothetical /sys/kvm/.

It cannot. sVirt labels the disk image and other files qemu needs with the appropriate label, and everything else is off limits. Even if you run the guest as root, it won't have access to other files.

Quoted 2 lines

Yes, but its different from the implementation point-of-view. For the user it surely all plays together.

We need qemu to cooperate for mmio tracing, and we can cooperate with qemu for symbol resolution. If it prevents adding another kernel API, that's a win from my POV.

Quoted 10 lines · Avi Kivity

Quoted 8 lines · Joerg Roedel

$ cd /sys/kvm/guest0
$ ls -l
-r-------- 1 root root 0 2009-08-17 12:05 name
dr-x------ 1 root root 0 2009-08-17 12:05 fs
$ cat name
guest0
$ # ...

Quoted 4 lines

Quoted 2 lines

The fs/ directory is used as the mount point for the guest root fs.

The problem is /sys/kvm, not /sys/kvm/fs.

I am not tied to /sys/kvm. We could also use /proc/<pid>/kvm/ for example. This would keep anything in the process space (except for the global list of VMs which we should have anyway).

Quoted 14 lines · Joerg Roedel

Quoted 12 lines · Avi Kivity

Quoted 10 lines · Joerg Roedel

$ cd /sys/kvm/guest0
$ ls -l
-r-------- 1 root root 0 2009-08-17 12:05 name
dr-x------ 1 root root 0 2009-08-17 12:05 fs
$ cat name
guest0
$ # ...

The fs/ directory is used as the mount point for the guest root fs.

Quoted 6 lines

Quoted 2 lines

The problem is /sys/kvm, not /sys/kvm/fs.

I am not tied to /sys/kvm. We could also use /proc/<pid>/kvm/ for example. This would keep anything in the process space (except for the global list of VMs which we should have anyway).

How about ~/.qemu/guests/$pid?

Quoted 9 lines

Quoted 7 lines

I am not tied to /sys/kvm. We could also use /proc/<pid>/kvm/ for example. This would keep anything in the process space (except for the global list of VMs which we should have anyway).

How about ~/.qemu/guests/$pid?

That makes it hard for perf to find it and even harder to get a list of all VMs.

With /proc/<pid>/kvm/guest we could symlink all guest
directories to /proc/kvm/ and perf reads the list from there. Also perf can easily derive the directory for a guest from its pid. Last but not least its kernel-created and thus independent from the userspace part being used.

Quoted 7 lines · Joerg Roedel

Quoted 7 lines

Quoted 7 lines

I am not tied to /sys/kvm. We could also use /proc/<pid>/kvm/ for example. This would keep anything in the process space (except for the global list of VMs which we should have anyway).

Quoted 4 lines

Quoted 2 lines

How about ~/.qemu/guests/$pid?

That makes it hard for perf to find it and even harder to get a list of all VMs.

Looks trivial to find a guest, less so with enumerating (still doable).

Quoted 3 lines

Looks trivial to find a guest, less so with enumerating (still doable).

Not so trival and even more likely to break. Even it perf has the pid of the process and wants to find the directory it has to do:

1. Get the uid of the process
2. Find the username for the uid
3. Use the username to find the home-directory

Steps 2. and 3. need nsswitch and/or pam access to get this information from whatever source the admin has configured. And depending on what the source is it may be temporarily unavailable causing nasty timeouts. In short, there are many weak parts in that chain making it more likely to break.

A kernel-based approach with /proc/<pid>/kvm does not have those issues (and to repeat myself, it is independent from the userspace being used).

It's true. If the kernel provides something, there are fewer things that can break. But if your system is so broken that you can't resolve uids, fix that before running perf. Must we design perf for that case?

After all, 'ls -l' will break under the same circumstances. It's hard to imagine doing useful work when that doesn't work.

Quoted 10 lines · Joerg Roedel

Quoted 3 lines

Looks trivial to find a guest, less so with enumerating (still doable).

Not so trival and even more likely to break. Even it perf has the pid of the process and wants to find the directory it has to do:

1. Get the uid of the process
2. Find the username for the uid
3. Use the username to find the home-directory

Quoted 6 lines

Steps 2. and 3. need nsswitch and/or pam access to get this information from whatever source the admin has configured. And depending on what the source is it may be temporarily unavailable causing nasty timeouts. In short, there are many weak parts in that chain making it more likely to break.

It's true. If the kernel provides something, there are fewer things that can break. But if your system is so broken that you can't resolve uids, fix that before running perf. Must we design perf for that case?

After all, 'ls -l' will break under the same circumstances. It's hard to imagine doing useful work when that doesn't work.

Quoted 3 lines

A kernel-based approach with /proc/<pid>/kvm does not have those issues (and to repeat myself, it is independent from the userspace being used).

It has other issues, which are IMO more problematic.

Doesn't perf already has a dependency on naming conventions for finding debug information?

Em Wed, Mar 24, 2010 at 06:09:30PM +0200, Avi Kivity escreveu:

Quoted 3 lines

Doesn't perf already has a dependency on naming conventions for finding debug information?

It looks at several places, from most symbol rich (/usr/lib/debug/, aka -debuginfo packages, where we have full symtabs) to poorest (the packaged binary, where we may just have a .dynsym).

In an ideal world, it would just get the build-id (a SHA1 cookie that is in an ELF session inserted in every binary (aka DSOs), kernel module, kallsyms or vmlinux file) and use that to look first in a local cache (implemented in perf for a long time already) or in some symbol server.

For instance, for a random perf.data file I collected here in my machine I have:

[acme@doppio linux-2.6-tip]$ perf buildid-list | grep libpthread 5c68f7afeb33309c78037e374b0deee84dd441f6 /lib64/libpthread-2.10.2.so [acme@doppio linux-2.6-tip]$

So I don't have to access /lib64/libpthread-2.10.2.so directly, nor some convention to get a debuginfo in a local file like:

/usr/lib/debug/lib64/libpthread-2.10.2.so.debug

Instead the tools look at:

[acme@doppio linux-2.6-tip]$ l ~/.debug/.build-id/5c/68f7afeb33309c78037e374b0deee84dd441f6
lrwxrwxrwx 1 acme acme 73 2010-01-06 18:53 /home/acme/.debug/.build-id/5c/68f7afeb33309c78037e374b0deee84dd441f6 -> ../../lib64/libpthread-2.10.2.so/5c68f7afeb33309c78037e374b0deee84dd441f6*

To find the file for that specific build-id, not the one installed in my machine (or on the different machine, of a different architecture) that may be completely unrelated, a new one, or one for a different arch.

Quoted 31 lines

Em Wed, Mar 24, 2010 at 06:09:30PM +0200, Avi Kivity escreveu:

Quoted 3 lines

Doesn't perf already has a dependency on naming conventions for finding debug information?

It looks at several places, from most symbol rich (/usr/lib/debug/, aka -debuginfo packages, where we have full symtabs) to poorest (the packaged binary, where we may just have a .dynsym).

In an ideal world, it would just get the build-id (a SHA1 cookie that is in an ELF session inserted in every binary (aka DSOs), kernel module, kallsyms or vmlinux file) and use that to look first in a local cache (implemented in perf for a long time already) or in some symbol server.

For instance, for a random perf.data file I collected here in my machine I have:

[acme@doppio linux-2.6-tip]$ perf buildid-list | grep libpthread 5c68f7afeb33309c78037e374b0deee84dd441f6 /lib64/libpthread-2.10.2.so [acme@doppio linux-2.6-tip]$

So I don't have to access /lib64/libpthread-2.10.2.so directly, nor some convention to get a debuginfo in a local file like:

/usr/lib/debug/lib64/libpthread-2.10.2.so.debug

Instead the tools look at:

[acme@doppio linux-2.6-tip]$ l ~/.debug/.build-id/5c/68f7afeb33309c78037e374b0deee84dd441f6
lrwxrwxrwx 1 acme acme 73 2010-01-06 18:53 /home/acme/.debug/.build-id/5c/68f7afeb33309c78037e374b0deee84dd441f6 -> ../../lib64/libpthread-2.10.2.so/5c68f7afeb33309c78037e374b0deee84dd441f6*

Quoted 3 lines

To find the file for that specific build-id, not the one installed in my machine (or on the different machine, of a different architecture) that may be completely unrelated, a new one, or one for a different arch.

Thanks. I believe qemu could easily act as a symbol server for this use case.

Quoted 31 lines · Arnaldo Carvalho de Melo

Em Wed, Mar 24, 2010 at 06:09:30PM +0200, Avi Kivity escreveu:

Quoted 3 lines

Doesn't perf already has a dependency on naming conventions for finding debug information?

It looks at several places, from most symbol rich (/usr/lib/debug/, aka -debuginfo packages, where we have full symtabs) to poorest (the packaged binary, where we may just have a .dynsym).

In an ideal world, it would just get the build-id (a SHA1 cookie that is in an ELF session inserted in every binary (aka DSOs), kernel module, kallsyms or vmlinux file) and use that to look first in a local cache (implemented in perf for a long time already) or in some symbol server.

For instance, for a random perf.data file I collected here in my machine I have:

[acme@doppio linux-2.6-tip]$ perf buildid-list | grep libpthread 5c68f7afeb33309c78037e374b0deee84dd441f6 /lib64/libpthread-2.10.2.so [acme@doppio linux-2.6-tip]$

So I don't have to access /lib64/libpthread-2.10.2.so directly, nor some convention to get a debuginfo in a local file like:

/usr/lib/debug/lib64/libpthread-2.10.2.so.debug

Instead the tools look at:

[acme@doppio linux-2.6-tip]$ l ~/.debug/.build-id/5c/68f7afeb33309c78037e374b0deee84dd441f6
lrwxrwxrwx 1 acme acme 73 2010-01-06 18:53 /home/acme/.debug/.build-id/5c/68f7afeb33309c78037e374b0deee84dd441f6 -> ../../lib64/libpthread-2.10.2.so/5c68f7afeb33309c78037e374b0deee84dd441f6*

Quoted 4 lines

To find the file for that specific build-id, not the one installed in my machine (or on the different machine, of a different architecture) that may be completely unrelated, a new one, or one for a different arch.

Thanks. I believe qemu could easily act as a symbol server for this use case.

Quoted 4 lines

Quoted 3 lines

An approach like: "The files are owned and only readable by the same user that started the vm." might be a good start. So a user can measure its own guests and root can measure all of them.

Quoted 3 lines

That's not how sVirt works. sVirt isolates a user's VMs from each other, so if a guest breaks into qemu it can't break into other guests owned by the same user.

If a vm breaks into qemu it can access the host file system which is the bigger problem. In this case there is no isolation anymore. From that context it can even kill other VMs of the same user independent of a hypothetical /sys/kvm/.

Quoted 4 lines · Joerg Roedel

Quoted 4 lines

Quoted 3 lines

An approach like: "The files are owned and only readable by the same user that started the vm." might be a good start. So a user can measure its own guests and root can measure all of them.

Quoted 8 lines

Quoted 3 lines

That's not how sVirt works. sVirt isolates a user's VMs from each other, so if a guest breaks into qemu it can't break into other guests owned by the same user.

If a vm breaks into qemu it can access the host file system which is the bigger problem. In this case there is no isolation anymore. From that context it can even kill other VMs of the same user independent of a hypothetical /sys/kvm/.

No it can't. With sVirt every single VM has a custom security label and the policy only allows it access to disks / files with a matching label, and prevents it attacking any other VMs or processes on the host. THis confines the scope of any exploit in QEMU to those resources the admin has explicitly assigned to the guest.

Quoted 7 lines

No it can't. With sVirt every single VM has a custom security label and the policy only allows it access to disks / files with a matching label, and prevents it attacking any other VMs or processes on the host. THis confines the scope of any exploit in QEMU to those resources the admin has explicitly assigned to the guest.

Even better. So a guest which breaks out can't even access its own /sys/kvm/ directory. Perfect, it doesn't need that access anyway.

Quoted 2 lines · Joerg Roedel

Even better. So a guest which breaks out can't even access its own /sys/kvm/ directory. Perfect, it doesn't need that access anyway.

But what security label does that directory have? How can we make sure that whoever needs access to those files, gets them?

Automatically created objects don't work well with that model. They're simply missing information.

Quoted 5 lines · Avi Kivity

Quoted 3 lines · Joerg Roedel

Even better. So a guest which breaks out can't even access its own /sys/kvm/ directory. Perfect, it doesn't need that access anyway.

Quoted 6 lines

But what security label does that directory have? How can we make sure that whoever needs access to those files, gets them?

Automatically created objects don't work well with that model. They're simply missing information.

If we go the /proc/<pid>/kvm way then the directory should probably inherit the label from /proc/<pid>/?

Same could be applied to /sys/kvm/guest/ if we decide for it. The VM is still bound to a single process with a /proc/<pid> after all.

Quoted 2 lines · Joerg Roedel

If we go the /proc/<pid>/kvm way then the directory should probably inherit the label from /proc/<pid>/?

That's a security policy. The security people like their policies outside the kernel.

For example, they may want a label that allows a trace context to read the data, and also qemu itself for introspection.

Quoted 5 lines · Avi Kivity

Quoted 3 lines · Joerg Roedel

If we go the /proc/<pid>/kvm way then the directory should probably inherit the label from /proc/<pid>/?

Quoted 6 lines

That's a security policy. The security people like their policies outside the kernel.

For example, they may want a label that allows a trace context to read the data, and also qemu itself for introspection.

Hm, I am not a security expert.

But is this not only one entity more for
sVirt to handle? I would leave that decision to the sVirt developers. Does attaching the same label as for the VM resources mean that root could not access it anymore?

Quoted 5 lines · Joerg Roedel

But is this not only one entity more for
sVirt to handle? I would leave that decision to the sVirt developers. Does attaching the same label as for the VM resources mean that root could not access it anymore?

IIUC processes run under a context, and there's a policy somewhere that tells you which context can access which label (and with what permissions). There was a server on the Internet once that gave you root access and invited you to attack it. No idea if anyone succeeded or not (I got bored after about a minute).

So it depends on the policy. If you attach the same label, that means all files with the same label have the same access permissions. I think.

Quoted 14 lines · Avi Kivity

Quoted 6 lines · Joerg Roedel

But is this not only one entity more for
sVirt to handle? I would leave that decision to the sVirt developers. Does attaching the same label as for the VM resources mean that root could not access it anymore?

IIUC processes run under a context, and there's a policy somewhere that tells you which context can access which label (and with what permissions). There was a server on the Internet once that gave you root access and invited you to attack it. No idea if anyone succeeded or not (I got bored after about a minute).

Quoted 3 lines

So it depends on the policy. If you attach the same label, that means all files with the same label have the same access permissions. I think.

So if this is true we can introduce a 'trace' label and add all contexts that should be allowed to trace to it.
But we probably should leave the details to the security experts ;-)

On 03/24/2010 06:31 PM, Joerg Roedel wrote:

That's just what I want to do. Leave it in userspace and then they can deal with it without telling us about it.

Quoted 4 lines

That's just what I want to do. Leave it in userspace and then they can deal with it without telling us about it.

They can't do that with a directory in /proc?

What I meant was: perf-kernel puts the guest-name into every sample and perf-userspace accesses /sys/kvm/guest_name/fs/ later to resolve the symbols. I leave the question of how the guest-fs is exposed to the host out of this discussion. We should discuss this seperatly.

Quoted 5 lines · Joerg Roedel

What I meant was: perf-kernel puts the guest-name into every sample and perf-userspace accesses /sys/kvm/guest_name/fs/ later to resolve the symbols. I leave the question of how the guest-fs is exposed to the host out of this discussion. We should discuss this seperatly.

I'd much prefer a pid like suggested later, keeps the samples smaller.

But that said, we need guest kernel events like mmap and context switches too, otherwise we simply can't make sense of guest userspace addresses, we need to know the guest address space layout.

So aside from a filesystem content, we first need mmap and context switch events to find the files we need to access.

And while I appreciate all the security talk, its basically pointless anyway, the host can access it anyway, everybody agrees on that, but still you're arguing the case..

Quoted 4 lines · Joerg Roedel

What I meant was: perf-kernel puts the guest-name into every sample and perf-userspace accesses /sys/kvm/guest_name/fs/ later to resolve the symbols. I leave the question of how the guest-fs is exposed to the host out of this discussion. We should discuss this seperatly.

I'd much prefer a pid like suggested later, keeps the samples smaller.

But that said, we need guest kernel events like mmap and context switches too, otherwise we simply can't make sense of guest userspace addresses, we need to know the guest address space layout.

Quoted 9 lines · Peter Zijlstra

Quoted 4 lines · Joerg Roedel

What I meant was: perf-kernel puts the guest-name into every sample and perf-userspace accesses /sys/kvm/guest_name/fs/ later to resolve the symbols. I leave the question of how the guest-fs is exposed to the host out of this discussion. We should discuss this seperatly.

I'd much prefer a pid like suggested later, keeps the samples smaller.

Quoted 3 lines

But that said, we need guest kernel events like mmap and context switches too, otherwise we simply can't make sense of guest userspace addresses, we need to know the guest address space layout.

With the filesystem approach all we need is the pid of the guest process. Then we can access proc/<pid>/maps of the guest and read out the address space layout, no?

Quoted 8 lines · Joerg Roedel

Quoted 5 lines

You can always provide the kernel and module paths as command line parameters. It just won't be transparently usable, but if you're using qemu from the command line, presumably you can live with that.

I don't want the tool for myself only. A typical perf user expects that it works transparent.

A typical kvm user uses libvirt, so we can integrate it with that.

Quoted 11 lines

Quoted 5 lines

Quoted 3 lines

Could be easily done using notifier chains already in the kernel. Probably implemented with much less than 100 lines of additional code.

And a userspace interface for that.

Not necessarily. The perf event is configured to measure systemwide kvm by userspace. The kernel side of perf takes care that it stays system-wide even with added vm instances. So in this case the consumer for the notifier would be the perf kernel part. No userspace interface required.

Someone needs to know about the new guest to fetch its symbols. Or do you want that part in the kernel too?

Quoted 4 lines

Someone needs to know about the new guest to fetch its symbols. Or do you want that part in the kernel too?

How about we add a virtio "guest file system access" device? The guest would then expose its own file system using that device.

On the host side this would simply be a -virtioguestfs unix:/tmp/guest.fs and you'd get a unix socket that gives you full access to the guest file system by using commands. I envision something like:

SEND: GET /proc/version
RECV: Linux version 2.6.27.37-0.1-default (geeko@buildhost) (gcc version 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux) ) #1 SMP 2009-10-15 14:56:58 +0200

Now all we need is integration in perf to enumerate virtual machines based on libvirt. If you want to run qemu-kvm directly, just go with --guestfs=/tmp/guest.fs and perf could fetch all required information automatically.

This should solve all issues while staying 100% in user space, right?

Quoted 8 lines · Alexander Graf

Quoted 4 lines

Someone needs to know about the new guest to fetch its symbols. Or do you want that part in the kernel too?

How about we add a virtio "guest file system access" device? The guest would then expose its own file system using that device.

Quoted 5 lines

On the host side this would simply be a -virtioguestfs unix:/tmp/guest.fs and you'd get a unix socket that gives you full access to the guest file system by using commands. I envision something like:

The idea is to use a dedicated channel over virtio-serial. If the channel is present the file server can serve files over it.

Quoted 11 lines

SEND: GET /proc/version
RECV: Linux version 2.6.27.37-0.1-default (geeko@buildhost) (gcc version 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux) ) #1 SMP 2009-10-15 14:56:58 +0200

Now all we need is integration in perf to enumerate virtual machines based on libvirt. If you want to run qemu-kvm directly, just go with --guestfs=/tmp/guest.fs and perf could fetch all required information automatically.

Quoted 2 lines

This should solve all issues while staying 100% in user space, right?

Yeah, needs a fuse filesystem to populate the host namespace (kind of sshfs over virtio-serial).

Quoted 16 lines · Avi Kivity

Quoted 14 lines · Alexander Graf

Quoted 5 lines

Someone needs to know about the new guest to fetch its symbols. Or do you want that part in the kernel too?

How about we add a virtio "guest file system access" device? The guest would then expose its own file system using that device.

On the host side this would simply be a -virtioguestfs unix:/tmp/guest.fs and you'd get a unix socket that gives you full access to the guest file system by using commands. I envision something like:

Quoted 3 lines

The idea is to use a dedicated channel over virtio-serial. If the channel is present the file server can serve files over it.

The file server being a kernel module inside the guest? We want to be able to serve things as early and hassle free as possible, so in this case I agree with Ingo that a kernel module is superior.

Quoted 10 lines

Quoted 10 lines

SEND: GET /proc/version
RECV: Linux version 2.6.27.37-0.1-default (geeko@buildhost) (gcc version 4.3.2 [gcc-4_3-branch revision 141291] (SUSE Linux) ) #1 SMP 2009-10-15 14:56:58 +0200

Now all we need is integration in perf to enumerate virtual machines based on libvirt. If you want to run qemu-kvm directly, just go with --guestfs=/tmp/guest.fs and perf could fetch all required information automatically.

Quoted 6 lines

Quoted 3 lines

This should solve all issues while staying 100% in user space, right?

Yeah, needs a fuse filesystem to populate the host namespace (kind of sshfs over virtio-serial).

I don't see why we need a fuse filesystem. We can of course create one later on. But for now all you need is a user connecting to that socket.

Sidenote: I really think we should come to a conclusion about the
concept. KVM integration into perf is very useful feature to analyze virtualization workloads.

Quoted 4 lines · Joerg Roedel

Sidenote: I really think we should come to a conclusion about the
concept. KVM integration into perf is very useful feature to
analyze virtualization workloads.

Agreed. I especially would like to see instruction/branch tracing working this way. This would a lot of the benefits of a simulator on a real CPU.

If you're profiling a single guest it makes more sense to do this from inside the guest - you can profile userspace as well as the kernel.

Quoted 3 lines

If you're profiling a single guest it makes more sense to do this from inside the guest - you can profile userspace as well as the kernel.

I'm interested in debugging the guest without guest cooperation.

In many cases qemu's new gdb stub works for that, but in some cases I would prefer instruction/branch traces over standard gdb style debugging.

Quoted 3 lines · Andi Kleen

Quoted 3 lines

If you're profiling a single guest it makes more sense to do this from inside the guest - you can profile userspace as well as the kernel.

Quoted 6 lines

I'm interested in debugging the guest without guest cooperation.

In many cases qemu's new gdb stub works for that, but in some cases I would prefer instruction/branch traces over standard gdb style debugging.

Isn't gdb supposed to be able to use branch traces?

It makes sense to
expose them via the gdb stub then. Not to say an external tool doesn't make sense.

Quoted 6 lines · Avi Kivity

Quoted 2 lines · Ingo Molnar

This is why i consider that line of argument rather dishonest ...

I am not going to reply to any more email from you on this thread.

Because i pointed out that i consider a line of argument intellectually dishonest?

I did not say _you_ as a person are dishonest - doing that would be an ad honimen attack against your person. (In fact i dont think you are, to the contrary)

An argument can certainly be labeled dishonest in a fair discussion and it is not a personal attack against you to express my opinion about that.

Quoted 12 lines · Ingo Molnar

Quoted 3 lines · Avi Kivity

Quoted 1 line · Ingo Molnar

This is why i consider that line of argument rather dishonest ...

I am not going to reply to any more email from you on this thread.

Because i pointed out that i consider a line of argument intellectually dishonest?

I did not say _you_ as a person are dishonest - doing that would be an ad honimen attack against your person. (In fact i dont think you are, to the contrary)

Quoted 3 lines

An argument can certainly be labeled dishonest in a fair discussion and it is not a personal attack against you to express my opinion about that.

Sigh, why am I drawn into this.

A person who uses dishonest arguments is a dishonest person. [...]

Quoted 22 lines · Avi Kivity

Quoted 20 lines · Ingo Molnar

Quoted 6 lines · Avi Kivity

Quoted 2 lines · Ingo Molnar

This is why i consider that line of argument rather dishonest ...

I am not going to reply to any more email from you on this thread.

Because i pointed out that i consider a line of argument intellectually dishonest?

I did not say _you_ as a person are dishonest - doing that would be an ad honimen attack against your person. (In fact i dont think you are, to the contrary)

An argument can certainly be labeled dishonest in a fair discussion and it is not a personal attack against you to express my opinion about that.

Quoted 4 lines

Sigh, why am I drawn into this.

A person who uses dishonest arguments is a dishonest person. [...]

That's not how i understood that phrase - and i did not mean to suggest that you are dishonest and i do not think that you are dishonest (to the contrary).

(btw, why are you interested in desktop-on-desktop? one use case is developers, which don't really need fancy GUIs; a second is people who test out distributions, but that doesn't seem to be a huge population; and a third is people running Windows for some application that doesn't run on Linux - hopefully a small catergory as well.

Quoted 10 lines

Consider the _other_ examples that are a lot more clear:

' If you expose paravirt spilocks via KVM please also make sure the KVM
tooling can make use of it, has an option for it to configure it, and that it has sufficient efficiency statistics displayed in the tool for admins to monitor.'

' If you create this new paravirt driver then please also make sure it can
be configured in the tooling. '

Quoted 2 lines

' Please also add a testcase for this bug to tools/kvm/testcases/ so we dont
repeat this same mistake in the future. '

All three happen quite commonly in qemu/kvm development. Of course someone who develops a feature also develops a patch that exposes it in qemu. There are several test cases in qemu-kvm.git/kvm/user/test.

If that is the theory then it has failed to trickle through in practice. As you know i have reported a long list of usability problems with hardly a look. That list could be created by pretty much anyone spending a few minutes of getting a first impression with qemu-kvm.

If that is the theory then it has failed to trickle through in practice. As you know i have reported a long list of usability problems with hardly a look. That list could be created by pretty much anyone spending a few minutes of getting a first impression with qemu-kvm.

Quoted 13 lines · Avi Kivity

Quoted 13 lines

Consider the _other_ examples that are a lot more clear:

' If you expose paravirt spilocks via KVM please also make sure the KVM
tooling can make use of it, has an option for it to configure it, and that it has sufficient efficiency statistics displayed in the tool for admins to monitor.'

' If you create this new paravirt driver then please also make sure it can
be configured in the tooling. '

' Please also add a testcase for this bug to tools/kvm/testcases/ so we dont
repeat this same mistake in the future. '

Quoted 4 lines

All three happen quite commonly in qemu/kvm development. Of course someone who develops a feature also develops a patch that exposes it in qemu. There are several test cases in qemu-kvm.git/kvm/user/test.

If that is the theory then it has failed to trickle through in practice. As you know i have reported a long list of usability problems with hardly a look. That list could be created by pretty much anyone spending a few minutes of getting a first impression with qemu-kvm.

So something is seriously wrong in KVM land, to pretty much anyone trying it for the first time. I have explained how i see the root cause of that, while you seem to suggest that there's nothing wrong to begin with. I guess we'll have to agree to disagree on that.

Quoted 18 lines · Ingo Molnar

Quoted 17 lines · Anthony Liguori

Quoted 16 lines · Ingo Molnar

Quoted 3 lines

[...] kvm in fact knows nothing about vga, to take your last example. [...]

Look at the VGA dirty bitmap optimization a'ka the KVM_GET_DIRTY_LOG ioctl.

See qemu/kvm-all.c's kvm_physical_sync_dirty_bitmap().

It started out as a VGA optimization (also used by live migration) and even today it's mostly used by the VGA drivers - albeit a weak one.

I wish there were stronger VGA optimizations implemented, copying the dirty bitmap is not a particularly performant solution. (although it's certainly better than full emulation) Graphics performance is one of the more painful aspects of KVM usability today.

Quoted 6 lines

Quoted 5 lines

We have to maintain a dirty bitmap because we don't have a paravirtual graphics driver. IOW, someone needs to write an Xorg driver.

Ideally, we could just implement a Linux framebuffer device, right?

No, you'd want to interact with DRM.

Using DRM doesn't help very much. You still need an X driver and most of the operations you care about (video rendering, window movement, etc) are not operations that need to go through DRM.

3D graphics virtualization is extremely difficult in the non-passthrough case. It really requires hardware support that isn't widely available today (outside a few NVIDIA chipsets).

It doesn't provide the things we need to a good user experience. You need things like an absolute input device, host driven display resize, RGBA hardware cursors. None of these go through DRI and it's those things that really provide the graphics user experience.

Implementing a virtualized DRM/KMS driver would at least get you the framebuffer interface more or less for free, while allowing you to deal with the userspace side of things incrementally (ie, running a dummy xorg on top of the virtualized fbdev until the DRI side catches up). It would also enable you to focus on the 2D and 3D parts independently.

Quoted 5 lines · Paul Mundt

Implementing a virtualized DRM/KMS driver would at least get you the framebuffer interface more or less for free, while allowing you to deal with the userspace side of things incrementally (ie, running a dummy xorg on top of the virtualized fbdev until the DRI side catches up). It would also enable you to focus on the 2D and 3D parts independently.

Guys, have a look at Gallium. In many ways it's a pile of crap, but at least it's a pile of crap designed by vmware for *exactly* your problem space.

Quoted 12 lines · Avi Kivity

Quoted 10 lines · Ingo Molnar

Quoted 7 lines · Avi Kivity

Quoted 5 lines

The moment any change (be it as trivial as fixing a GUI detail or as complex as a new feature) involves two or more packages, development speed slows down to a crawl - while the complexity of the change might be very low!

Why is that?

Quoted 6 lines

Quoted 3 lines

It's very simple: because the contribution latencies and overhead compound, almost inevitably.

It's not inevitable, if the projects are badly run, you'll have high latencies, but projects don't have to be badly run.

So the 64K dollar question is, why does Qemu still suck?

Quoted 10 lines · Ingo Molnar

Quoted 8 lines · Avi Kivity

Quoted 7 lines · Ingo Molnar

Quoted 5 lines · Avi Kivity

Quoted 4 lines

The moment any change (be it as trivial as fixing a GUI detail or as complex as a new feature) involves two or more packages, development speed slows down to a crawl - while the complexity of the change might be very low!

Why is that?

Quoted 5 lines

Quoted 4 lines

Quoted 2 lines

It's very simple: because the contribution latencies and overhead compound, almost inevitably.

It's not inevitable, if the projects are badly run, you'll have high latencies, but projects don't have to be badly run.

So the 64K dollar question is, why does Qemu still suck?

Why does Linux AIO still suck? Why do we not have a proper interface in userspace for doing asynchronous file system operations?

Why don't we have an interface in userspace to do zero-copy transmit and receive of raw network packets?

The lack of a decent userspace API for asynchronous file system operations is a huge usability problem for us. Take a look at the complexity of our -drive option. It's all because the kernel gives us sucky interfaces.

Currently there's a wall between kernel developers and user-space developers, and there's somewhat of an element of fear and arrogance on both sides. For efficient technology such walls needs torn down and people need a bit more experience with each other's areas.

Quoted 4 lines · Ingo Molnar

Currently there's a wall between kernel developers and user-space developers, and there's somewhat of an element of fear and arrogance on both sides. For efficient technology such walls needs torn down and people need a bit more experience with each other's areas.

I think you're increasing the height of that wall by arguing that a userspace project cannot be successful because it's development process sucks and the only way to fix it is to put it into the kernel where people know so much better. Instead we kernel developers should listen to requirements from users, even if their code isn't in tools/.

Good that you mention it, i think it's an excellent example.
The suckage of kernel async IO is for similar reasons: there's an ugly package separation problem between the kernel and between glibc - and between the apps that would make use of it.

( With the separated libaio it was made worse: there were 3 libraries to
work with, and even less applications that could make use of it ... )

So IMO klibc is an arguably good idea - eventually hpa will get around posting it for upstream merging again. Then we could offer both new libraries much faster, and could offer things like comprehensive AIO used pervasively within existing APIs.

The suckage of kernel async IO is for similar reasons: there's an ugly package separation problem between the kernel and between glibc

Quoted 2 lines · Ingo Molnar

The suckage of kernel async IO is for similar reasons: there's an ugly package separation problem between the kernel and between glibc

Bollocks. glibc would use (and is using) everything the kernel provides.

I didnt say it's glibc's fault - if then it's more of the kernel's fault as most of the complexity is on that side. I said it's due to the fundamental distance between the app that makes use of it, the library and the kernel, and the resulting difficulties in getting a combined solution out.

Quoted 5 lines · Ingo Molnar

I didnt say it's glibc's fault - if then it's more of the kernel's fault as most of the complexity is on that side. I said it's due to the fundamental distance between the app that makes use of it, the library and the kernel, and the resulting difficulties in getting a combined solution out.

This is wrong, too. Once there is a kernel patch that has a reasonable syscall interface it's easy enough to hack up the glibc side. [...]

[...] Don't try to artificially find an argument to support your thesis.

Where 'reasonable' is defined by you, right?

As i said, the KAIO situation is mostly the kernel's fault, but you are a pretty passive and unhelpful entity in this matter too, arent you?

For example, just to state the obvious: libaio has been written 8 years ago in 2002 and has been used in apps early on. Why arent those kernel APIs, while not being a full/complete solution, supported by glibc, and wrapped to pthreads based emulation on kernels that dont support it?

Why isnt glibc itself making use of those AIO capabilities internally? (even if it's not possible to support full POSIX AIO)

I checked today's glibc repo, and there's no sign of any of that:

Getting _something_ into glibc would certainly help move the situation.

it's certainly capable enough to run DB servers. glibc
using it would create further demand (and pressure, and incentives) for improvements.

Charming argumentation style, i really missed it.

Quoted 4 lines

For example, just to state the obvious: libaio has been written 8 years ago in 2002 and has been used in apps early on. Why arent those kernel APIs, while not being a full/complete solution, supported by glibc, and wrapped to pthreads based emulation on kernels that dont support it?

You never looked at the glibc code in use and didn't read what I wrote before. We do have an implementation of libaio using those interfaces. They exist in the Fedora/RHEL glibc and are probably copied elsewhere, too. The code is not upstream because it is not general enough. It simply doesn't work in all situations.

So it's good enough to be in Fedora/RHEL but not good enough to be in upstream glibc? How is that possible? Isnt that a double standard?

kernel's fault. But glibc is certainly not being helpful in that situation either and your earlier claim that you are only waiting for the patches is rather dishonest.

Quoted 2 lines

So it's good enough to be in Fedora/RHEL but not good enough to be in upstream glibc? How is that possible? Isnt that a double standard?

Yes its a double standard

Glibc has a higher standard than Fedora/RHEL.

Just like the Ubuntu kernel ships various ugly unfit for upstream kernel drivers.

Quoted 3 lines

kernel's fault. But glibc is certainly not being helpful in that situation either and your earlier claim that you are only waiting for the patches is rather dishonest.

I am sure Ulrich is being totally honest, but send him the patches and you'll find out. Plus you will learn what the API should look like when you try and create them ...

Quoted 8 lines

Yes its a double standard

Glibc has a higher standard than Fedora/RHEL.

Just like the Ubuntu kernel ships various ugly unfit for upstream kernel drivers.

There's a world of a difference between a fugly driver and a glibc patch.

Quoted 2 lines

Why does Linux AIO still suck? Why do we not have a proper interface in userspace for doing asynchronous file system operations?

Good that you mention it, i think it's an excellent example.

The suckage of kernel async IO is for similar reasons: there's an ugly package separation problem between the kernel and between glibc - and between the apps that would make use of it.

Quoted 7 lines · Avi Kivity

Quoted 5 lines

The moment any change (be it as trivial as fixing a GUI detail or as complex as a new feature) involves two or more packages, development speed slows down to a crawl - while the complexity of the change might be very low!

Why is that?

It's very simple: because the contribution latencies and overhead compound, almost inevitably.

If you ever tried to implement a combo GCC+glibc+kernel feature you'll know
...

Even with the best-run projects in existence it takes forever and is very painful - and here i talk about first hand experience over many years.

Quoted 10 lines · Ingo Molnar

Quoted 5 lines · Avi Kivity

Quoted 4 lines

The moment any change (be it as trivial as fixing a GUI detail or as complex as a new feature) involves two or more packages, development speed slows down to a crawl - while the complexity of the change might be very low!

Why is that?

It's very simple: because the contribution latencies and overhead compound, almost inevitably.

Quoted 5 lines

If you ever tried to implement a combo GCC+glibc+kernel feature you'll know
...

Even with the best-run projects in existence it takes forever and is very painful - and here i talk about first hand experience over many years.

Ingo, what you miss is that this is not a bad thing. Fact of the matter is, it's not just painful, it downright sucks.

This is actually a Good Thing (tm). It means you have to get your feature and its interfaces well defined and able to version forwards and backwards independently from each other. And that introduces some complexity and time and testing, but in the end it's what you want. You don't introduce a requirement to have the feature, but take advantage of it if it is there.

It may take everyone else a couple years to upgrade the compilers, tools, libraries and kernel, and by that time any bugs introduced by interacting with this feature will have been ironed out and their patterns well known.

Quoted 15 lines · Zachary Amsden

Quoted 13 lines · Ingo Molnar

Quoted 7 lines · Avi Kivity

Quoted 5 lines

The moment any change (be it as trivial as fixing a GUI detail or as complex as a new feature) involves two or more packages, development speed slows down to a crawl - while the complexity of the change might be very low!

Why is that?

It's very simple: because the contribution latencies and overhead compound, almost inevitably.

Quoted 9 lines

Quoted 6 lines

If you ever tried to implement a combo GCC+glibc+kernel feature you'll know
...

Even with the best-run projects in existence it takes forever and is very painful - and here i talk about first hand experience over many years.

Ingo, what you miss is that this is not a bad thing. Fact of the matter is, it's not just painful, it downright sucks.

Our experience is the opposite, and we tried both variants and report about our experience with both models honestly.

You only have experience about one variant - the one you advocate.

See the assymetry?

Quoted 7 lines

This is actually a Good Thing (tm). It means you have to get your feature and its interfaces well defined and able to version forwards and backwards independently from each other. And that introduces some complexity and time and testing, but in the end it's what you want. You don't introduce a requirement to have the feature, but take advantage of it if it is there.

Quoted 5 lines

It may take everyone else a couple years to upgrade the compilers, tools, libraries and kernel, and by that time any bugs introduced by interacting with this feature will have been ironed out and their patterns well known.

Sorry, but this is pain not true. The 2.4->2.6 kernel cycle debacle has taught us that waiting long to 'iron out' the details has the following effects:

- developer pain
- user pain
- distro pain
- disconnect
- loss of developers, testers and users
- grave bugs discovered months (years ...) down the line
- untested features
- developer exhaustion

It didnt work, trust me - and i've been around long enough to have suffered through the whole 2.5.x misery. Some of our worst ABIs come from that cycle as well.

Sorry, but i really think you are really trying to rationalize a disadvantage here ...

Quoted 50 lines · Ingo Molnar

Quoted 11 lines · Zachary Amsden

Quoted 10 lines · Ingo Molnar

Quoted 5 lines · Avi Kivity

Quoted 4 lines

The moment any change (be it as trivial as fixing a GUI detail or as complex as a new feature) involves two or more packages, development speed slows down to a crawl - while the complexity of the change might be very low!

Why is that?

It's very simple: because the contribution latencies and overhead compound, almost inevitably.

Quoted 7 lines

Quoted 5 lines

If you ever tried to implement a combo GCC+glibc+kernel feature you'll know
...

Even with the best-run projects in existence it takes forever and is very painful - and here i talk about first hand experience over many years.

Ingo, what you miss is that this is not a bad thing. Fact of the matter is, it's not just painful, it downright sucks.

Our experience is the opposite, and we tried both variants and report about our experience with both models honestly.

You only have experience about one variant - the one you advocate.

See the assymetry?

Quoted 11 lines

This is actually a Good Thing (tm). It means you have to get your feature and its interfaces well defined and able to version forwards and backwards independently from each other. And that introduces some complexity and time and testing, but in the end it's what you want. You don't introduce a requirement to have the feature, but take advantage of it if it is there.

It may take everyone else a couple years to upgrade the compilers, tools, libraries and kernel, and by that time any bugs introduced by interacting with this feature will have been ironed out and their patterns well known.

Sorry, but this is pain not true. The 2.4->2.6 kernel cycle debacle has taught us that waiting long to 'iron out' the details has the following effects:

- developer pain
- user pain
- distro pain
- disconnect
- loss of developers, testers and users
- grave bugs discovered months (years ...) down the line
- untested features
- developer exhaustion

Quoted 3 lines

It didnt work, trust me - and i've been around long enough to have suffered through the whole 2.5.x misery. Some of our worst ABIs come from that cycle as well.

You're talking about a single project and comparing it to my argument about multiple independent projects. In that case, I see no point in the discussion. If you want to win the argument by strawman, you are welcome to do so.

Quoted 2 lines

Sorry, but i really think you are really trying to rationalize a disadvantage here ...

This could very well be true, but until someone comes forward with compelling numbers (as in, developers committed to working on the project, number of patches and total amount of code contribution), there is no point in having an argument, there really isn't anything to discuss other than opinion. My opinion is you need a really strong justification to have a successful fork and I don't see that justification.

Too bad - there was heavy initial opposition to the arch/x86 unification as well [and heavy opposition to tools/perf/ as well], still both worked out extremely well :-)

Quoted 5 lines · Ingo Molnar

Too bad - there was heavy initial opposition to the arch/x86 unification as well [and heavy opposition to tools/perf/ as well], still both worked out extremely well :-)

Did you forget that arch/x86 was a merging of a code fork that happened several years previously? Maybe that fork shouldn't have been done to begin with.

Quoted 11 lines · Avi Kivity

Quoted 5 lines · Ingo Molnar

Too bad - there was heavy initial opposition to the arch/x86 unification as well [and heavy opposition to tools/perf/ as well], still both worked out extremely well :-)

Did you forget that arch/x86 was a merging of a code fork that happened several years previously? Maybe that fork shouldn't have been done to begin with.

We discussed and probably timidly tried to share the sharable initially but we realized it was too time wasteful. In addition to having to adapt the code to 64bit we would also had to constantly solve another problem on top of it (see the various split on _32/_64, those takes time to achieve, maybe not huge time but still definitely some time and effort). Even in retrospect I am quite sure the way x86-64 happened was optimal and if we would go back we would do it again the exact same way even if the final object was to have a common arch/x86 (and thankfully Linus is flexible and smart enough to realize

that code that isn't risking to destabilize anything shouldn't be forced out just because it's not to a totally theoretical-perfect-nitpicking-clean-state yet). It's still a lot of work do the unification later as a separate task, but it's not like if we did it immediately it would have been a lot less work. It's about the same amount of effort and we were able to defer it for later and decrease the time to market which surely has contributed to the success of x86-64.

Problem of qemu is not some lack of GUI or that it's not included in the linux kernel git tree, the definitive problem is how to merge qemu-kvm/kvm and qlx into it. If you (Avi) were the qemu maintainer I am sure there wouldn't two trees so as a developer I would totally love it, and I am sure that with you as maintainer it would have a chance to move forward with qlx on desktop virtualization without proposing to extend vnc instead to achieve a "similar" result (imagine if btrfs is published on a website and people starts to discuss if it should ever be merged ever because reinventing some part of btrfs inside ext5 might achieve ""similar"" results).

About a GUI for KVM to use on desktop distributions, that is an irrelevant concern compared to the lack of protocol more efficient than rdesktop/rdp/vnc for desktop virtualization. I've people asking me to migrate hundreds of desktops to desktop virtualization on KVM in their organizations and I tell them to use spice because I believe it's the most efficient option available (at least as far as we stick to open source open protocols), there are universities using spice on thousand of student desktops, and I think we need paravirt graphics to happen ASAP in the main qemu tree too.

In short: running KVM on the desktop is irrelevant compared to running the desktop on KVM so I suggest to focus on what is more important first ;).

Quoted 2 lines

- move a clean (and minimal) version of the Qemu code base to tools/kvm/, in the upstream kernel repo, and work on that from that point on.

I'll ignore the repository location which should be immaterial to a serious developer and concentrate on the 'clean and minimal' aspect, since it has some merit. [...]

[...]

Do you really think the echo'n'cat school of usability wants to write a GUI? In linux-2.6.git?

Quoted 7 lines · Avi Kivity

Quoted 3 lines

- move a clean (and minimal) version of the Qemu code base to tools/kvm/, in the upstream kernel repo, and work on that from that point on.

I'll ignore the repository location which should be immaterial to a serious developer and concentrate on the 'clean and minimal' aspect, since it has some merit. [...]

To the contrary, experience shows that repository location, and in particular a shared repository for closely related bits is very much material!

It matters because when there are two separate projects, even a "serious developer" is finding it double and triple difficult to contribute even trivial changes.

It becomes literally a nightmare if you have to touch 3 packages: kernel, a library and an app codebase. It takes _forever_ to get anything useful done.

Also, 'focus on a single thing' is a very basic aspect of humans, especially those who do computer programming. Working on two code bases in two repositories at once can be very challenging physically and psychically.

So what i've seen is that OSS programmers tend to pick a side, pretty much randomly, and then rationalize it in hindsight why they prefer that side ;-)

Most of them become either a kernel developer or a user-space package developer - and then they specialize on that field and shy away from changes that involve both. It's a basic human thing to avoid the hassle that comes with multi-package changes. (One really has to be outright stupid, fanatic or desperate to even attempt such changes these days - such are the difficulties for a comparatively low return.)

The solution is to tear down such artificial walls of contribution where possible. And tearing down the wall between KVM and qemu-kvm seems very much possible and the advantages would be numerous.

Unless by "serious developer" you meant: "developer willing to [or forced to] waste time and effort on illogically structured technology".

Quoted 5 lines

[...]

Do you really think the echo'n'cat school of usability wants to write a GUI? In linux-2.6.git?

Then you'll be surprised to hear that it's happening as we speak and the commits are there in linux-2.6.git. Both a TUI and GUI is in the works.

Furthermore, the numbers show that half of the usability fixes to tools/perf/ came not from regular perf contributors but from random kernel developers and testers who when they build the latest kernel and try out perf at the same time (it's very easy because you already have it in the kernel repository - no separate download, no installation, etc. necessary).

I had literally zero such contributions when (the precursor to) 'perf' was still a separate user-space project.

You could have the same effect for Qemu: the latest bits in tools/kvm/ would be built by regular kernel testers and developers. The integration benefits dont just extend to developers, a unified project is vastly easier to test as well.

Quoted 16 lines · Ingo Molnar

Quoted 5 lines · Avi Kivity

Quoted 2 lines

- move a clean (and minimal) version of the Qemu code base to tools/kvm/, in the upstream kernel repo, and work on that from that point on.

I'll ignore the repository location which should be immaterial to a serious developer and concentrate on the 'clean and minimal' aspect, since it has some merit. [...]

To the contrary, experience shows that repository location, and in particular a shared repository for closely related bits is very much material!

It matters because when there are two separate projects, even a "serious developer" is finding it double and triple difficult to contribute even trivial changes.

Quoted 3 lines

It becomes literally a nightmare if you have to touch 3 packages: kernel, a library and an app codebase. It takes _forever_ to get anything useful done.

You can't be serious. I find that the difficulty in contributing a patch has mostly to do with writing the patch, and less with figuring out which email address to send it to.

By "serious developer" I mean

- someone who is interested in contributing, not in getting their name
into the kernel commits list
- someone who is willing to read the wiki page and find out where the
repository and mailing list for a project is
- someone who will spend enough time on the project so that the time to
clone two repositories will not be a factor in their contributions

Let's wait and see then. If the tools/perf/ experience has really good results, we can reconsider this at a later date.

[...] It is actually because both kernel and user side are sync in this scheme. [...]

Quoted 2 lines · Frederic Weisbecker

[...] It is actually because both kernel and user side are sync in this scheme. [...]

This argues that co-evolution of an interface is easiest on the developers if they own both sides of that interface. No quarrel.

This does not argue that that the preservation of a stable ABI is best done this way. If anything, it makes it too easy to change both the provider and the preferred user of the interface without noticing unintentional breakage to forlorn out-of-your-tree clients.

I wish both you and Avi looked back 3-4 years and realized what made KVM so successful back then and why the hearts and minds of virtualization developers were captured by KVM almost overnight.

Fast forward to 2010. The kernel side of KVM is maximum goodness - by far the worst-quality remaining aspects of KVM are precisely in areas that you
mention: 'if we have to support third party tools, then it significantly complicates things'. You kept Qemu as an external 'third party' entity to KVM, and KVM is clearly hurting from that - just see the recent KVM usability thread for examples about suckage.

So a similar 'complication' is the crux of the matter behind KVM quality
problems: you've not followed through with the original KVM vision and you have not applied that concept to Qemu!

If you want to jump to the next level of technological quality you need to fix this attitude and you need to go back to the design roots of KVM. Concentrate on Qemu (as that is the weakest link now), make it a first class member of the KVM repo and simplify your development model by having a single repo:

- move a clean (and minimal) version of the Qemu code base to tools/kvm/, in the upstream kernel repo, and work on that from that point on.

- encourage kernel-space and user-space KVM developers to work on both user-space and kernel-space bits as a single unit. It's one project and a single experience to the user.

- [ and probably libvirt should go there too ]

Yes, i've read a thousand excuses for why this is an absolutely impossible and a bad thing to do, and none of them was really convincing to me - and you also have become rather emotional about all the arguments so it's hard to argue about it on a technical basis.

We made a similar (admittedly very difficult ...) design jump from oprofile to perf, and i can tell you from that experience that it's day and night, both in terms of development and in terms of the end result!

Quoted 3 lines · Ingo Molnar

I wish both you and Avi looked back 3-4 years and realized what made KVM so successful back then and why the hearts and minds of virtualization developers were captured by KVM almost overnight.

Ingo,

What made KVM so successful was that the core kernel of the hypervisor was designed the right way, as a kernel module where it belonged. It was obvious to anyone who had been exposed to the main competition at the time, Xen, that this was the right approach. What has ended up killing Xen in the end is the not-invented-here approach of copying everything over, reformatting it, and rewriting half of it, which made it impossible to maintain and support as a single codebase. [...]

[...]

At my previous employer we ended up dropping all Xen efforts exactly because it was like maintaining two separate operating system kernels. The key to KVM is that once you have Linux, you practically have KVM as well.

Quoted 7 lines

Fast forward to 2010. The kernel side of KVM is maximum goodness - by far the worst-quality remaining aspects of KVM are precisely in areas that you
mention: 'if we have to support third party tools, then it significantly complicates things'. You kept Qemu as an external 'third party' entity to KVM, and KVM is clearly hurting from that - just see the recent KVM usability thread for examples about suckage.

Quoted 3 lines

So a similar 'complication' is the crux of the matter behind KVM quality
problems: you've not followed through with the original KVM vision and you have not applied that concept to Qemu!

Well there are two ways to go about this. Either you base the KVM userland on top of an existing project, like QEMU, _or_ you rewrite it all from scratch. [...]

[...] However, there is far more to it than just a couple of ioctls, for example the stack of reverse device-drivers is a pretty significant code base, rewriting that and maintaining it is not a trivial task. It is certainly my belief that the benefit we get from sharing that with QEMU by far outweighs the cost of forking it and keeping our own fork in the kernel tree. In fact it would result in exactly the same problems I mentioned above wrt Xen.

Quoted 6 lines

If you want to jump to the next level of technological quality you need to fix this attitude and you need to go back to the design roots of KVM. Concentrate on Qemu (as that is the weakest link now), make it a first class member of the KVM repo and simplify your development model by having a single repo:

Quoted 2 lines

- move a clean (and minimal) version of the Qemu code base to tools/kvm/, in the upstream kernel repo, and work on that from that point on.

With this you have just thrown away all the benefits of having the QEMU repository shared with other developers who will actively fix bugs in components we do care about for KVM.

Quoted 3 lines

- encourage kernel-space and user-space KVM developers to work on both user-space and kernel-space bits as a single unit. It's one project and a single experience to the user.

This is already happening and a total non issue.

Quoted 1 line

- [ and probably libvirt should go there too ]

Now that would be interesting, next we'll have to include things like libxml in the kernel git tree as well, to make sure libvirt doesn't get out of sync with the version supplied by your distribution vendor.

Quoted 4 lines

Yes, i've read a thousand excuses for why this is an absolutely impossible and a bad thing to do, and none of them was really convincing to me - and you also have become rather emotional about all the arguments so it's hard to argue about it on a technical basis.

So far your argument would justify pulling all of gdb into the kernel git tree as well, to support the kgdb efforts, or gcc so we can get rid of the gcc version quirks in the kernel header files, e2fsprogs and equivalent for _all_ file systems included in the kernel so we can make sure our fs tools never get out of sync with whats supported in the kernel......

Quoted 3 lines

We made a similar (admittedly very difficult ...) design jump from oprofile to perf, and i can tell you from that experience that it's day and night, both in terms of development and in terms of the end result!

The user components for perf vs oprofile are _tiny_ projects compared to the portions of QEMU that are actually used by KVM.

Oh and you completely forgot SeaBIOS. KVM+QEMU rely on SeaBIOS too, so from what you're saying we should pull that into the kernel git repository as well. Never mind the fact that we share SeaBIOS with the coreboot project which is very actively adding features to it that benefit us as well.....

Sorry, but there are times when unification make sense, and there are times where having a reasonably well designed split makes sense. KVM had problems with QEMU in the past which resulted in the qemu-kvm branch of it, which proved to be a major pain to deal with, but that is fortunately improving and qemu-kvm should go away completely at some point.

* Jes Sorensen<[email protected]> wrote:

Quoted 4 lines

At my previous employer we ended up dropping all Xen efforts exactly because it was like maintaining two separate operating system kernels. The key to KVM is that once you have Linux, you practically have KVM as well.

Yes. Please realize that what is behind it is a strikingly simple argument:

"Once you have a single project to develop and maintain all is much better."

Quoted 7 lines

[...] However, there is far more to it than just a couple of ioctls, for example the stack of reverse device-drivers is a pretty significant code base, rewriting that and maintaining it is not a trivial task. It is certainly my belief that the benefit we get from sharing that with QEMU by far outweighs the cost of forking it and keeping our own fork in the kernel tree. In fact it would result in exactly the same problems I mentioned above wrt Xen.

I do not suggest forking Qemu at all, i suggest using the most natural development model for the KVM+Qemu shared project: a single repository.

Quoted 3 lines

With this you have just thrown away all the benefits of having the QEMU repository shared with other developers who will actively fix bugs in components we do care about for KVM.

Not if it's a unified project.

Quoted 5 lines

Quoted 3 lines

- encourage kernel-space and user-space KVM developers to work on both user-space and kernel-space bits as a single unit. It's one project and a single experience to the user.

This is already happening and a total non issue.

My experience as an external observer of the end result contradicts this.

Seemingly trivial usability changes to the KVM+Qemu combo are not being done often because they involve cross-discipline changes.

Quoted 5 lines

Quoted 1 line

- [ and probably libvirt should go there too ]

Now that would be interesting, next we'll have to include things like libxml in the kernel git tree as well, to make sure libvirt doesn't get out of sync with the version supplied by your distribution vendor.

The way we have gone about this in tools/perf/ is similar to the route picked by Git: we only use very lowlevel libraries available everywhere, and we provide optional wrappers to the rest.

Quoted 5 lines

So far your argument would justify pulling all of gdb into the kernel git tree as well, to support the kgdb efforts, or gcc so we can get rid of the gcc version quirks in the kernel header files, e2fsprogs and equivalent for _all_ file systems included in the kernel so we can make sure our fs tools never get out of sync with whats supported in the kernel......

gdb and gcc is clearly extrinsic to the kernel so why would we move them there?

I was talking about tools that are closely related to the kernel - where much of the development and actual use is in combination with the Linux kernel.

90%+ of the Qemu usecases are combined with Linux. (Yes, i know that you can run Qemu without KVM, and no, i dont think it matters in the grand scheme of things and most investment into Qemu comes from the KVM angle these days. In particular it for sure does not justify handicapping future KVM evolution so drastically.)

Quoted 4 lines

Oh and you completely forgot SeaBIOS. KVM+QEMU rely on SeaBIOS too, so from what you're saying we should pull that into the kernel git repository as well. Never mind the fact that we share SeaBIOS with the coreboot project which is very actively adding features to it that benefit us as well.....

SeaBIOS is in essence a firmware, so it could either be loaded as such.

Just look at the qemu source code - the BIOSes are .bin images in qemu/pc-bios/ imported externally in essence.

qemu-kvm branch is not similar to my proposal at all: it made KVM _more_ fragmented, not more unified. I.e. it was a move in the exact opposite direction and i'd expect such a move to fail.

In fact the failure of qemu-kvm supports my point rather explicitly: it demonstrates that extra packages and split development are actively harmful.

I speak about this as a person who has done successful unifications of split codebases and in my judgement this move would be significantly beneficial to KVM.

You cannot really validly reject this proposal with "It wont work" as it clearly has worked in other, comparable cases. You could only reject this with "I have tried it and it didnt work".

Think about it: a clean and hackable user-space component in tools/kvm/. It's very tempting :-)

Quoted 1 line · Ingo Molnar

* Jes Sorensen<[email protected]> wrote:

[...]

Quoted 5 lines

Quoted 4 lines

At my previous employer we ended up dropping all Xen efforts exactly because it was like maintaining two separate operating system kernels. The key to KVM is that once you have Linux, you practically have KVM as well.

Quoted 3 lines

Yes. Please realize that what is behind it is a strikingly simple argument:

"Once you have a single project to develop and maintain all is much better."

Thats a very glorified statement but it's not reality, sorry. You can do that with something like perf because it's so small and development of perf is limited to a very small group of developers.

Quoted 3 lines · Jes Sorensen

Thats a very glorified statement but it's not reality, sorry. You can do that with something like perf because it's so small and development of perf is limited to a very small group of developers.

I was not talking about just perf: i am also talking about the arch/x86/ unification which is 200+ KLOC of highly non-trivial kernel code with hundreds of contributors and with 8000+ commits in the past two years.

Also, it applies to perf as well: people said exactly that a year ago: 'perf has it easy to be clean as it is small, once it gets as large as Oprofile tooling it will be in the same messy situation'.

Today perf has more features than Oprofile, has a larger and more complex code base, has more contributors, and no, it's not in the same messy situation at all.

So whatever you think of large, unified projects, you are quite clearly mistaken. I have done and maintained through two different types of unifications and the experience was very similar: both developers and users (and maintainers) are much better off.

Quoted 6 lines · Ingo Molnar

Quoted 3 lines · Jes Sorensen

Thats a very glorified statement but it's not reality, sorry. You can do that with something like perf because it's so small and development of perf is limited to a very small group of developers.

Quoted 3 lines

I was not talking about just perf: i am also talking about the arch/x86/ unification which is 200+ KLOC of highly non-trivial kernel code with hundreds of contributors and with 8000+ commits in the past two years.

Sorry but you cannot compare merging two chunks of kernel code that originated from the same base, with the efforts of mixing a large userland project with a kernel component. Apples and oranges.

Quoted 7 lines

Also, it applies to perf as well: people said exactly that a year ago:
'perf has it easy to be clean as it is small, once it gets as large as Oprofile tooling it will be in the same messy situation'.

Today perf has more features than Oprofile, has a larger and more complex code base, has more contributors, and no, it's not in the same messy situation at all.

Both perf and oprofile are still relatively small projects in comparison to QEMU.

Quoted 4 lines

So whatever you think of large, unified projects, you are quite clearly mistaken. I have done and maintained through two different types of unifications and the experience was very similar: both developers and users (and maintainers) are much better off.

You believe that I am wrong in my assessment of unified projects, and I obviously think you are mistaken and underestimating the cost and effects of trying to merge the two.

Well I think we are just going to agree to disagree on this one. I am not against merging projects where it makes sense, but in this particular case I am strongly convinced the loss would be much greater than the gain.

Quoted 2 lines · Jes Sorensen

Both perf and oprofile are still relatively small projects in comparison to QEMU.

So is your argument that the unification does not make sense due to size? Would a smaller Qemu be more appropriate for this purpose?

Quoted 3 lines

Well I think we are just going to agree to disagree on this one. I am not against merging projects where it makes sense, but in this particular case I am strongly convinced the loss would be much greater than the gain.

I wish you said that based on first hand negative experience with unifications, not based on just pure speculation.

(and yes, i speculate too, but at least with some basis)

Quoted 5 lines · Ingo Molnar

Quoted 2 lines · Jes Sorensen

Both perf and oprofile are still relatively small projects in comparison to QEMU.

Quoted 2 lines

So is your argument that the unification does not make sense due to size? Would a smaller Qemu be more appropriate for this purpose?

As I have stated repeatedly in this discussion, a unification would hurt the QEMU development process because it would alienate a large number of QEMU developers who are *not* Linux kernel users.

QEMU is a lot more complex than you let on.

Quoted 7 lines

Quoted 3 lines

Well I think we are just going to agree to disagree on this one. I am not against merging projects where it makes sense, but in this particular case I am strongly convinced the loss would be much greater than the gain.

I wish you said that based on first hand negative experience with unifications, not based on just pure speculation.

Quoted 1 line

(and yes, i speculate too, but at least with some basis)

You still haven't given us a *single* example of unification of something that wasn't purely linked to the Linux kernel. perf/ oprofile is 100% linked to the Linux kernel, QEMU is not. I wish you would actually look at what users use QEMU for. As long as you continue to purely speculate on this, to use your own words, your arguments are not holding up.

And you are not being consistent either. You have conveniently continue to ignore my questions about why the file system tools are not to be merged into the Linux kernel source tree?

I took a quick look at the qemu.git log and more than half of all recent contributions came from Linux distributors.

So without KVM Qemu would be a much, much smaller project. It would be similar to how it was 5 years ago.

What made KVM so successful was that the core kernel of the hypervisor was designed the right way, as a kernel module where it belonged. It was obvious to anyone who had been exposed to the main competition at the time, Xen, that this was the right approach. What has ended up killing Xen in the end is the not-invented-here approach of copying everything over, reformatting it, and rewriting half of it, which made it impossible to maintain and support as a single codebase. At my previous

So far your argument would justify pulling all of gdb into the kernel git tree as well, to support the kgdb efforts, or gcc so we can get rid of the gcc version quirks in the kernel header files, e2fsprogs and equivalent for _all_ file systems included in the kernel so we can make sure our fs tools never get out of sync with whats supported in the kernel......

Btw., i made similar arguments to Avi about 3 years ago when it was going upstream, that qemu should be unified with KVM. This is more true today than ever.

Quoted 27 lines · Anthony Liguori

Quoted 25 lines · Ingo Molnar

Quoted 4 lines

If we look at the use-case, it's going to be something like, a user is creating virtual machines and wants to get performance information about them.

Quoted 9 lines

Having to run a separate tool like perf is not going to be what they would expect they had to do. Instead, they would either use their existing GUI tool (like virt-manager) or they would use their management interface (either QMP or libvirt).

The complexity of interaction is due to the fact that perf shouldn't be a stand alone tool. It should be a library or something with a programmatic interface that another tool can make use of.

But ... a GUI interface/integration is of course possible too, and it's being worked on.

perf is mainly a kernel developer tool, and kernel developers generally dont use GUIs to do their stuff: which is the (sole) reason why its first ~850 commits of tools/perf/ were done without a GUI. We go where our developers are.

In any case it's not an excuse to have no proper command-line tooling. In fact if you cannot get simpler, more atomic command-line tooling right then you'll probably doubly suck at doing a GUI as well.

Quoted 6 lines

It's about who owns the user interface.

If qemu owns the user interface, than we can satisfy this in a very simple way by adding a perf monitor command. If we have to support third party tools, then it significantly complicates things.

Of course illogical modularization complicates things 'significantly'.

- move a clean (and minimal) version of the Qemu code base to tools/kvm/, in the upstream kernel repo, and work on that from that point on.

Hm, that sounds rather messy if we want to use it to basically expose kernel functionality in a guest/host unified way. Is the qemu process discoverable in some secure way? Can we trust it? Is there some proper tooling available to do it, or do we have to push it through 2-3 packages to get such a useful feature done?

Quoted 5 lines · Ingo Molnar

Hm, that sounds rather messy if we want to use it to basically expose kernel functionality in a guest/host unified way. Is the qemu process discoverable in some secure way? Can we trust it? Is there some proper tooling available to do it, or do we have to push it through 2-3 packages to get such a useful feature done?

Since we want to implement a pmu usable for the guest anyway why we don't just use a guests perf to get all information we want? If we get a pmu-nmi from the guest we just re-inject it to the guest and perf in the guest gives us all information we wand including kernel and userspace symbols, stack traces, and so on.

In the previous thread we discussed about a direct trace channel between guest and host kernel (which can be used for ftrace events for example). This channel could be used to transport this information to the host kernel.

The only additional feature needed is a way for the host to start a perf instance in the guest.

Opinions?

Joerg

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 5 lines · Ingo Molnar

Hm, that sounds rather messy if we want to use it to basically expose kernel functionality in a guest/host unified way. Is the qemu process discoverable in some secure way? Can we trust it? Is there some proper tooling available to do it, or do we have to push it through 2-3 packages to get such a useful feature done?

Since we want to implement a pmu usable for the guest anyway why we don't just use a guests perf to get all information we want? [...]

Quoted 18 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Quoted 6 lines

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels?

Perhaps a
symbol server that perf can connect to (and that would connect to guests in turn)?

Quoted 15 lines

diff -Nraup linux-2.6_tipmaster0315/arch/x86/kvm/vmx.c linux-2.6_tipmaster0315_perfkvm/arch/x86/kvm/vmx.c

arch/x86/kvm/vmx.c +1 -0

@@ -26,6 +26,7 @@
  #include<linux/sched.h>
  #include<linux/moduleparam.h>
  #include<linux/ftrace_event.h>
+#include<linux/perf_event.h>
  #include "kvm_cache_regs.h"
  #include "x86.h"
 

@@ -3632,6 +3633,43 @@ static void update_cr8_intercept(struct
  	vmcs_write32(TPR_THRESHOLD, irr);
  }
 

arch/x86/kvm/vmx.c static void update_cr8_intercept(struct

Quoted 12 lines

arch/x86/kvm/vmx.c +11 -0

+DEFINE_PER_CPU(int, kvm_in_guest) = {0};
+
+static void kvm_set_in_guest(void)
+{
+	percpu_write(kvm_in_guest, 1);
+}
+
+static int kvm_is_in_guest(void)
+{
+	return percpu_read(kvm_in_guest);
+}
 

There is already PF_VCPU for this.

arch/x86/kvm/vmx.c static void update_cr8_intercept(struct

Quoted 8 lines

arch/x86/kvm/vmx.c +6 -0

+static struct perf_guest_info_callbacks kvm_guest_cbs = {
+	.is_in_guest 		= kvm_is_in_guest,
+	.is_user_mode		= kvm_is_user_mode,
+	.get_guest_ip		= kvm_get_guest_ip,
+	.reset_in_guest		= kvm_reset_in_guest,
+};
 

Should be in common code, not vmx specific.

Quoted 19 lines · Avi Kivity

Quoted 18 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Quoted 7 lines

Quoted 6 lines

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Thanks for your kind comments.

Quoted 4 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels?

With the patch, 'perf kvm report --sort pid" could show
summary statistics for all guest os instances. Then, use
parameter --pid of 'perf kvm record' to collect single problematic instance data.

Sorry. I found currently --pid isn't process but a thread (main thread).

Ingo,

Is it possible to support a new parameter or extend --inherit, so 'perf record' and 'perf top' could collect data on all threads of a process when the process is running?

Quoted 29 lines · Yanmin

Quoted 19 lines · Avi Kivity

Quoted 18 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Quoted 7 lines

Quoted 6 lines

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Thanks for your kind comments.

Quoted 7 lines

Quoted 4 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels?

With the patch, 'perf kvm report --sort pid" could show
summary statistics for all guest os instances. Then, use
parameter --pid of 'perf kvm record' to collect single problematic instance data.

Sorry. I found currently --pid isn't process but a thread (main thread).

Ingo,

Is it possible to support a new parameter or extend --inherit, so 'perf record' and 'perf top' could collect data on all threads of a process when the process is running?

If not, I need add a new ugly parameter which is similar to --pid to filter out process data in userspace.

Quoted 45 lines · Zhang, Yanmin

Quoted 29 lines · Yanmin

Quoted 19 lines · Avi Kivity

Quoted 18 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Quoted 7 lines

Quoted 6 lines

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Thanks for your kind comments.

Quoted 7 lines

Quoted 4 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels?

With the patch, 'perf kvm report --sort pid" could show
summary statistics for all guest os instances. Then, use
parameter --pid of 'perf kvm record' to collect single problematic instance data.

Sorry. I found currently --pid isn't process but a thread (main thread).

Ingo,

Is it possible to support a new parameter or extend --inherit, so 'perf record' and 'perf top' could collect data on all threads of a process when the process is running?

Quoted 2 lines

If not, I need add a new ugly parameter which is similar to --pid to filter out process data in userspace.

Yeah. For maximum utility i'd suggest to extend --pid to include this, and introduce --tid for the previous, limited-to-a-single-task functionality.

Most users would expect --pid to work like a 'late attach' - i.e. to work like strace -f or like a gdb attach.

Quoted 47 lines · Ingo Molnar

Quoted 45 lines · Zhang, Yanmin

Quoted 29 lines · Yanmin

Quoted 19 lines · Avi Kivity

Quoted 18 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Quoted 7 lines

Quoted 6 lines

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Thanks for your kind comments.

Quoted 7 lines

Quoted 4 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels?

With the patch, 'perf kvm report --sort pid" could show
summary statistics for all guest os instances. Then, use
parameter --pid of 'perf kvm record' to collect single problematic instance data.

Sorry. I found currently --pid isn't process but a thread (main thread).

Ingo,

Is it possible to support a new parameter or extend --inherit, so 'perf record' and 'perf top' could collect data on all threads of a process when the process is running?

Quoted 8 lines

Quoted 2 lines

If not, I need add a new ugly parameter which is similar to --pid to filter out process data in userspace.

Yeah. For maximum utility i'd suggest to extend --pid to include this, and introduce --tid for the previous, limited-to-a-single-task functionality.

Most users would expect --pid to work like a 'late attach' - i.e. to work like strace -f or like a gdb attach.

Thanks Ingo, Avi.

I worked out below patch against tip/master of March 15th.

Subject: [PATCH] Change perf's parameter --pid to process-wide collection
From: Zhang, Yanmin <[email protected]>

Change parameter -p (--pid) to real process pid and add -t (--tid) meaning thread id. Now, --pid means perf collects the statistics of all threads of the process, while --tid means perf just collect the statistics of that thread.

BTW, the patch fixes a bug of 'perf stat -p'. 'perf stat' always configures attr->disabled=1 if it isn't a system-wide collection. If there is a '-p' and no forks, 'perf stat -p' doesn't collect any data. In addition, the while(!done) in run_perf_stat consumes 100% single cpu time which has bad impact on running workload. I added a sleep(1) in the loop.

Signed-off-by Zhang Yanmin <[email protected]>

Quoted 74 lines · Yanmin

Quoted 47 lines · Ingo Molnar

Quoted 45 lines · Zhang, Yanmin

Quoted 29 lines · Yanmin

Quoted 19 lines · Avi Kivity

Quoted 18 lines · Yanmin

From: Zhang, Yanmin<[email protected]>

Based on the discussion in KVM community, I worked out the patch to support perf to collect guest os statistics from host side. This patch is implemented with Ingo, Peter and some other guys' kind help. Yang Sheng pointed out a critical bug and provided good suggestions with other guys. I really appreciate their kind help.

The patch adds new subcommand kvm to perf.

perf kvm top
perf kvm record
perf kvm report
perf kvm diff

The new perf could profile guest os kernel except guest os user space, but it could summarize guest os user space utilization per guest os.

Quoted 7 lines

Quoted 6 lines

Below are some examples.
1) perf kvm top
[root@lkp-ne01 norm]# perf kvm --host --guest --guestkallsyms=/home/ymzhang/guest/kallsyms --guestmodules=/home/ymzhang/guest/modules top

Thanks for your kind comments.

Quoted 7 lines

Quoted 4 lines

Excellent, support for guest kernel != host kernel is critical (I can't remember the last time I ran same kernels).

How would we support multiple guests with different kernels?

With the patch, 'perf kvm report --sort pid" could show
summary statistics for all guest os instances. Then, use
parameter --pid of 'perf kvm record' to collect single problematic instance data.

Sorry. I found currently --pid isn't process but a thread (main thread).

Ingo,

Is it possible to support a new parameter or extend --inherit, so 'perf record' and 'perf top' could collect data on all threads of a process when the process is running?

Quoted 8 lines

Quoted 2 lines

If not, I need add a new ugly parameter which is similar to --pid to filter out process data in userspace.

Yeah. For maximum utility i'd suggest to extend --pid to include this, and introduce --tid for the previous, limited-to-a-single-task functionality.

Most users would expect --pid to work like a 'late attach' - i.e. to work like strace -f or like a gdb attach.

Thanks Ingo, Avi.

I worked out below patch against tip/master of March 15th.

Subject: [PATCH] Change perf's parameter --pid to process-wide collection
From: Zhang, Yanmin <[email protected]>

Change parameter -p (--pid) to real process pid and add -t (--tid) meaning thread id. Now, --pid means perf collects the statistics of all threads of the process, while --tid means perf just collect the statistics of that thread.

BTW, the patch fixes a bug of 'perf stat -p'. 'perf stat' always configures attr->disabled=1 if it isn't a system-wide collection. If there is a '-p' and no forks, 'perf stat -p' doesn't collect any data. In addition, the while(!done) in run_perf_stat consumes 100% single cpu time which has bad impact on running workload. I added a sleep(1) in the loop.

Quoted 1 line

Signed-off-by Zhang Yanmin <[email protected]>

Ingo,

Sorry, the patch has bugs. I need do a better job and will work out 2 separate patches against the 2 issues.