KVM PMU virtualization

Hi,

It looks like several of us have been looking at how to use the PMU for virtualization. Rather than continuing to have discussions in smaller groups, I think it is a good idea we move it to the mailing lists to see what we can share and avoid duplicate efforts.

There are really two separate things to handle:

1) Add support to perf to allow it to monitor a KVM guest from the host.

2) Allow guests access to the PMU (or an emulated PMU), making it possible to run perf on applications running within the guest.

I know some of you have been looking at 1) and I am currently working on 2). I have been looking at various approaches, including whether it is feasible to share the PMU between the host and multiple guests. For now I am going to focus on allowing one guest to take control of the PMU, then later hopefully adding support for multiplexing it between multiple guests.

Eventually we will see proper hardware PMU virtualization from Intel and AMD (admittedly I have only looked at the Intel specs so far), and by then be able to allow the host as well as the guests to share the PMU.

If anybody else is working on this, I'd love to hear about it so we can coordinate our efforts. The main purpose with this mail was really to being the discussion to the mailing list to avoid duplicated efforts.

Quoted 15 lines · Jes Sorensen

Hi,

It looks like several of us have been looking at how to use the PMU for virtualization. Rather than continuing to have discussions in smaller groups, I think it is a good idea we move it to the mailing lists to see what we can share and avoid duplicate efforts.

There are really two separate things to handle:

1) Add support to perf to allow it to monitor a KVM guest from the host.

2) Allow guests access to the PMU (or an emulated PMU), making it possible to run perf on applications running within the guest.

Quoted 6 lines

I know some of you have been looking at 1) and I am currently working on 2). I have been looking at various approaches, including whether it is feasible to share the PMU between the host and multiple guests. For now I am going to focus on allowing one guest to take control of the PMU, then later hopefully adding support for multiplexing it between multiple guests.

Given that perf can apply the PMU to individual host tasks, I don't see fundamental problems multiplexing it between individual guests (which can then internally multiplex it again).

1) Add support to perf to allow it to monitor a KVM guest from the host.

2) Allow guests access to the PMU (or an emulated PMU), making it possible to run perf on applications running within the guest.

Quoted 23 lines · Jan Kiszka

Quoted 15 lines · Jes Sorensen

Hi,

It looks like several of us have been looking at how to use the PMU for virtualization. Rather than continuing to have discussions in smaller groups, I think it is a good idea we move it to the mailing lists to see what we can share and avoid duplicate efforts.

There are really two separate things to handle:

1) Add support to perf to allow it to monitor a KVM guest from the host.

2) Allow guests access to the PMU (or an emulated PMU), making it possible to run perf on applications running within the guest.

Quoted 6 lines

I know some of you have been looking at 1) and I am currently working on 2). I have been looking at various approaches, including whether it is feasible to share the PMU between the host and multiple guests. For now I am going to focus on allowing one guest to take control of the PMU, then later hopefully adding support for multiplexing it between multiple guests.

Quoted 3 lines

Given that perf can apply the PMU to individual host tasks, I don't see fundamental problems multiplexing it between individual guests (which can then internally multiplex it again).

In terms of how to expose it to guests, a 'soft PMU' might be a usable approach. Although to Linux guests you could expose much more functionality and an non-PMU-limited number of instrumentation events, via a more intelligent interface.

But note that in terms of handling it on the host side the PMU approach is not
acceptable: instead it should map to proper perf_events, not try to muck with the PMU itself.

That, besides integrating properly with perf usage on the host, will also allow interesting 'PMU' features on guests: you could set up the host side to trace block IO requests (or VM exits) for example, and expose that as 'PMC #0' on the guest side.

Quoted 2 lines · Jes Sorensen

1) Add support to perf to allow it to monitor a KVM guest from the host.

This shouldn't be a big problem. The PMU of AMD Fam10 processors can be configured to count only when in guest mode. Perf needs to be aware of that and fetch the rip from a different place when monitoring a guest.

Quoted 3 lines

2) Allow guests access to the PMU (or an emulated PMU), making it possible to run perf on applications running within the guest.

The biggest problem I see here is teaching the guest about the available events. The available event sets are dependent on the processor family (at least on AMD).
A simple approach would be shadowing the perf msrs which is a simple thing to do. More problematic is the reinjection of performance interrupts and performance nmis.

I personally don't like a self-defined event-set as the only solution because that would probably only work with linux and perf. I think we should have a way (additionally to a soft-event interface) which allows to expose the host pmu events to the guest.

Joerg

I personally don't like a self-defined event-set as the only solution because that would probably only work with linux and perf. [...]

Quoted 37 lines · Ingo Molnar

Quoted 23 lines · Jan Kiszka

Quoted 15 lines · Jes Sorensen

Hi,

It looks like several of us have been looking at how to use the PMU for virtualization. Rather than continuing to have discussions in smaller groups, I think it is a good idea we move it to the mailing lists to see what we can share and avoid duplicate efforts.

There are really two separate things to handle:

1) Add support to perf to allow it to monitor a KVM guest from the host.

2) Allow guests access to the PMU (or an emulated PMU), making it possible to run perf on applications running within the guest.

Quoted 6 lines

I know some of you have been looking at 1) and I am currently working on 2). I have been looking at various approaches, including whether it is feasible to share the PMU between the host and multiple guests. For now I am going to focus on allowing one guest to take control of the PMU, then later hopefully adding support for multiplexing it between multiple guests.

Quoted 3 lines

Given that perf can apply the PMU to individual host tasks, I don't see fundamental problems multiplexing it between individual guests (which can then internally multiplex it again).

In terms of how to expose it to guests, a 'soft PMU' might be a usable approach. Although to Linux guests you could expose much more functionality and an non-PMU-limited number of instrumentation events, via a more intelligent interface.

But note that in terms of handling it on the host side the PMU approach is not acceptable: instead it should map to proper perf_events, not try to muck with the PMU itself.

Quoted 5 lines

That, besides integrating properly with perf usage on the host, will also allow interesting 'PMU' features on guests: you could set up the host side to trace block IO requests (or VM exits) for example, and expose that as 'PMC
#0' on the guest side.

So virtualization becomes non-transparent to guest os? I know virtio is an optimization on guest side.

Quoted 5 lines · Joerg Roedel

Quoted 2 lines · Jes Sorensen

1) Add support to perf to allow it to monitor a KVM guest from the host.

Quoted 3 lines

This shouldn't be a big problem. The PMU of AMD Fam10 processors can be configured to count only when in guest mode. Perf needs to be aware of that and fetch the rip from a different place when monitoring a guest.

The idea is we want to measure both host and guest at the same time, and compare all the hot functions fairly.

Quoted 9 lines · Yanmin

Quoted 8 lines · Joerg Roedel

Quoted 2 lines · Jes Sorensen

1) Add support to perf to allow it to monitor a KVM guest from the host.

This shouldn't be a big problem. The PMU of AMD Fam10 processors can be configured to count only when in guest mode. Perf needs to be aware of that and fetch the rip from a different place when monitoring a guest.

Quoted 2 lines

The idea is we want to measure both host and guest at the same time, and compare all the hot functions fairly.

So you want to measure while the guest vcpu is running and the vmexit path of that vcpu (including qemu userspace part) together? The challenge here is to find out if a performance event originated in guest mode or in host mode.
But we can check for that in the nmi-protected part of the vmexit path.

Quoted 3 lines · Joerg Roedel

I personally don't like a self-defined event-set as the only solution because that would probably only work with linux and perf. [...]

The 'soft-PMU' i suggested is transparent on the guest side - if you want to enable non-Linux and legacy-Linux.

It's basically a PMU interface provided to the guest by catching the right MSR accesses, implemented via perf_event_create_kernel_counter()/etc. on the host side.

Note that the 'soft PMU' still sucks from a design POV as there's no generic hw interface to the PMU. So there would have to be a 'soft AMD' and a 'soft Intel' PMU driver at minimum.

Far cleaner would be to expose it via hypercalls to guest OSs that are interested in instrumentation.

That way it could also transparently integrate
with tracing, probes, etc. It would also be wiser to first concentrate on improving Linux<->Linux guest/host combos before gutting the design just to fit Windows into the picture ...

Quoted 4 lines · Ingo Molnar

Note that the 'soft PMU' still sucks from a design POV as there's no generic hw interface to the PMU. So there would have to be a 'soft AMD' and a 'soft Intel' PMU driver at minimum.

Right, this will severely limit migration domains to hosts of the same vendor and processor generation. There is a middle ground, though, Intel has recently moved to define an "architectural pmu" which is not model specific. I don't know if AMD adopted it. We could offer both options - native host capabilities, with a loss of compatibility, and the architectural pmu, with loss of model specific counters.

My suggestion, as always, would be to start very simple and very minimal:

Enable 'perf kvm top' to show guest overhead. Use the exact same kernel image both as a host and as guest (for testing), to not have to deal with the symbol space transport problem initially. Enable 'perf kvm record' to only record guest events by default. Etc.

This alone will be a quite useful result already - and gives a basis for further work. No need to spend months to do the big grand design straight away, all of this can be done gradually and in the order of usefulness - and you'll always have something that actually works (and helps your other KVM projects) along the way.

[ And, as so often, once you walk that path, that grand scheme you are
thinking about right now might easily become last year's really bad idea ;-) ]

So please start walking the path and experience the challenges first-hand.

Right, this will severely limit migration domains to hosts of the same vendor and processor generation. There is a middle ground, though, Intel has recently moved to define an "architectural pmu" which is not model specific. I don't know if AMD adopted it. [...]

Quoted 2 lines

Far cleaner would be to expose it via hypercalls to guest OSs that are interested in instrumentation.

It's also slower - you can give the guest direct access to the various counters so no exits are taken when reading the counters (though perhaps many tools are only interested in the interrupts, not the counter values).

Quoted 4 lines

That way it could also transparently integrate with tracing, probes, etc. It would also be wiser to first concentrate on improving Linux<->Linux guest/host combos before gutting the design just to fit Windows into the picture ...

"gutting the design"?

Quoted 6 lines · Avi Kivity

Quoted 4 lines · Ingo Molnar

Note that the 'soft PMU' still sucks from a design POV as there's no generic hw interface to the PMU. So there would have to be a 'soft AMD' and a 'soft Intel' PMU driver at minimum.

Quoted 6 lines

Right, this will severely limit migration domains to hosts of the same vendor and processor generation. There is a middle ground, though, Intel has recently moved to define an "architectural pmu" which is not model specific. I don't know if AMD adopted it. We could offer both options - native host capabilities, with a loss of compatibility, and the architectural pmu, with loss of model specific counters.

I only had a quick look yet on the architectural pmu from intel but it looks like it can be emulated for a guest on amd using existing features.

Quoted 11 lines

Quoted 8 lines

Right, this will severely limit migration domains to hosts of the same vendor and processor generation. There is a middle ground, though, Intel has recently moved to define an "architectural pmu" which is not model specific. I don't know if AMD adopted it. We could offer both options - native host capabilities, with a loss of compatibility, and the architectural pmu, with loss of model specific counters.

I only had a quick look yet on the architectural pmu from intel but it looks like it can be emulated for a guest on amd using existing features.

AMD CPUs dont have enough events for that, they cannot do the 3 fixed events in addition to the 2 generic ones.

Nor do you really want to standardize on KVM guests on returning 'GenuineIntel' in CPUID, so that the various guest side OSs use the Intel PMU drivers, right?

Quoted 13 lines · Ingo Molnar

My suggestion, as always, would be to start very simple and very minimal:

Enable 'perf kvm top' to show guest overhead. Use the exact same kernel image both as a host and as guest (for testing), to not have to deal with the symbol space transport problem initially. Enable 'perf kvm record' to only record guest events by default. Etc.

This alone will be a quite useful result already - and gives a basis for further work. No need to spend months to do the big grand design straight away, all of this can be done gradually and in the order of usefulness - and you'll always have something that actually works (and helps your other KVM projects) along the way.

Quoted 4 lines

[ And, as so often, once you walk that path, that grand scheme you are
thinking about right now might easily become last year's really bad idea ;-) ]

So please start walking the path and experience the challenges first-hand.

That sounds like a good approach for the 'measure-guest-from-host' problem. It is also not very hard to implement. Where does perf fetch the rip of the nmi from, stack only or is this configurable?

Quoted 3 lines

Given that perf can apply the PMU to individual host tasks, I don't see fundamental problems multiplexing it between individual guests (which can then internally multiplex it again).

In terms of how to expose it to guests, a 'soft PMU' might be a usable approach. Although to Linux guests you could expose much more functionality and an non-PMU-limited number of instrumentation events, via a more intelligent interface.

But note that in terms of handling it on the host side the PMU approach is not
acceptable: instead it should map to proper perf_events, not try to muck with the PMU itself.

The biggest problem I see here is teaching the guest about the available events. The available event sets are dependent on the processor family (at least on AMD).
A simple approach would be shadowing the perf msrs which is a simple thing to do. More problematic is the reinjection of performance interrupts and performance nmis.

Quoted 14 lines · Joerg Roedel

Quoted 12 lines · Avi Kivity

Quoted 4 lines · Ingo Molnar

Note that the 'soft PMU' still sucks from a design POV as there's no generic hw interface to the PMU. So there would have to be a 'soft AMD' and a 'soft Intel' PMU driver at minimum.

Right, this will severely limit migration domains to hosts of the same vendor and processor generation. There is a middle ground, though, Intel has recently moved to define an "architectural pmu" which is not model specific. I don't know if AMD adopted it. We could offer both options - native host capabilities, with a loss of compatibility, and the architectural pmu, with loss of model specific counters.

Quoted 2 lines

I only had a quick look yet on the architectural pmu from intel but it looks like it can be emulated for a guest on amd using existing features.

AMD CPUs dont have enough events for that, they cannot do the 3 fixed events in addition to the 2 generic ones.

Nor do you really want to standardize on KVM guests on returning 'GenuineIntel' in CPUID, so that the various guest side OSs use the Intel PMU drivers, right?

Quoted 7 lines

Quoted 3 lines

Far cleaner would be to expose it via hypercalls to guest OSs that are interested in instrumentation.

It's also slower - you can give the guest direct access to the various counters so no exits are taken when reading the counters (though perhaps many tools are only interested in the interrupts, not the counter values).

Direct access to counters is not something that is a big issue. [ Given that i sometimes can see KVM redraw the screen of a guest OS real-time i doubt this is the biggest of performance challenges right now ;-) ]

By far the biggest instrumentation issue is:

- availability
- usability
- flexibility

Exposing the raw hw is a step backwards in many regards.

The same way we dont
want to expose chipsets to the guest to allow them to do RAS. The same way we dont want to expose most raw PCI devices to guest in general, but have all these virt driver abstractions.

Quoted 7 lines

Quoted 5 lines

That way it could also transparently integrate with tracing, probes, etc. It would also be wiser to first concentrate on improving Linux<->Linux guest/host combos before gutting the design just to fit Windows into the picture ...

"gutting the design"?

Yes, gutting the design of a sane instrumentation API and moving it back 10-20 years by squeezing it through non-standardized and incompatible PMU drivers.

When it comes to design my main interest is the Linux<->Linux combo.

Isn't there a cpuid bit indicating the availability of architectural perfmon?

Quoted 17 lines · Ingo Molnar

Quoted 14 lines · Joerg Roedel

Quoted 12 lines · Avi Kivity

Quoted 4 lines · Ingo Molnar

Note that the 'soft PMU' still sucks from a design POV as there's no generic hw interface to the PMU. So there would have to be a 'soft AMD' and a 'soft Intel' PMU driver at minimum.

Right, this will severely limit migration domains to hosts of the same vendor and processor generation. There is a middle ground, though, Intel has recently moved to define an "architectural pmu" which is not model specific. I don't know if AMD adopted it. We could offer both options - native host capabilities, with a loss of compatibility, and the architectural pmu, with loss of model specific counters.

Quoted 5 lines

Quoted 2 lines

I only had a quick look yet on the architectural pmu from intel but it looks like it can be emulated for a guest on amd using existing features.

AMD CPUs dont have enough events for that, they cannot do the 3 fixed events in addition to the 2 generic ones.

Good point. Maybe we can emulate that with some counter round-robin usage if the guest really uses all 5 counters.

Quoted 3 lines

Nor do you really want to standardize on KVM guests on returning 'GenuineIntel' in CPUID, so that the various guest side OSs use the Intel PMU drivers, right?

Isn't there a cpuid bit indicating the availability of architectural perfmon?

Quoted 2 lines · Joerg Roedel

Isn't there a cpuid bit indicating the availability of architectural perfmon?

Nope, the perfmon flag is a fake Linux flag, set based on the contents on cpuid 0x0a

Direct access to counters is not something that is a big issue. [ Given that i sometimes can see KVM redraw the screen of a guest OS real-time i doubt this is the biggest of performance challenges right now ;-) ]

By far the biggest instrumentation issue is:

- availability
- usability
- flexibility

Exposing the raw hw is a step backwards in many regards. The same way we dont want to expose chipsets to the guest to allow them to do RAS. The same way we dont want to expose most raw PCI devices to guest in general, but have all these virt driver abstractions.

There is a way to query the CPU for 'architectural perfmon' though, via CPUID alone - that is how we set the X86_FEATURE_ARCH_PERFMON shortcut. The logic is:

         if (c->cpuid_level>  9) {
                 unsigned eax = cpuid_eax(10);
                 /* Check for version and the number of counters */
                 if ((eax&  0xff)&&  (((eax>>8)&  0xff)>  1))
                         set_cpu_cap(c, X86_FEATURE_ARCH_PERFMON);
         }

But emulating that doesnt solve the problem: as OSs generally dont key their PMU drivers off the relatively new 'architectural perfmon' CPUID detail, but based on much higher level CPUID attributes. (like Intel/AMD)

Quoted 8 lines · Ingo Molnar

Quoted 5 lines

Quoted 2 lines

Far cleaner would be to expose it via hypercalls to guest OSs that are interested in instrumentation.

It's also slower - you can give the guest direct access to the various counters so no exits are taken when reading the counters (though perhaps many tools are only interested in the interrupts, not the counter values).

Direct access to counters is not something that is a big issue. [ Given that i sometimes can see KVM redraw the screen of a guest OS real-time i doubt this is the biggest of performance challenges right now ;-) ]

Outside 4-bit vga mode, this shouldn't happen. Can you describe your scenario?

Quoted 6 lines

By far the biggest instrumentation issue is:

- availability
- usability
- flexibility

Quoted 1 line

Exposing the raw hw is a step backwards in many regards.

In a way, virtualization as a whole is a step backwards. We take the nice firesystem/timer/network/scheduler APIs, and expose them as raw hardware. The pmu isn't any different.

Quoted 10 lines · Ingo Molnar

Direct access to counters is not something that is a big issue. [ Given that i sometimes can see KVM redraw the screen of a guest OS real-time i doubt this is the biggest of performance challenges right now ;-) ]

By far the biggest instrumentation issue is:

- availability
- usability
- flexibility

Quoted 4 lines

Exposing the raw hw is a step backwards in many regards. The same way we dont want to expose chipsets to the guest to allow them to do RAS. The same way we dont want to expose most raw PCI devices to guest in general, but have all these virt driver abstractions.

I have to say I disagree on that. When you run perfmon on a system, it is normally to measure a specific application. You want to see accurate numbers for cache misses, mul instructions or whatever else is selected.

Emulating the PMU rather than using the real one, makes the numbers far less useful. The most useful way to provide PMU support in a guest is to expose the real PMU and let the guest OS program it.

We can do this in a reasonable way today, if we allow to take the PMU away from the host, and only let guests access it when it's in use. [...]

Quoted 7 lines

Quoted 7 lines

By far the biggest instrumentation issue is:

- availability
- usability
- flexibility

Quoted 6 lines

Quoted 2 lines

Exposing the raw hw is a step backwards in many regards.

In a way, virtualization as a whole is a step backwards. We take the nice firesystem/timer/network/scheduler APIs, and expose them as raw hardware. The pmu isn't any different.

Uhm, it's obviously very different. A fake NE2000 will work on both Intel and AMD CPUs. Same for a fake PIT. PMU drivers are fundamentally per CPU vendor though.

So there's no "generic hardware" to emulate.

Quoted 17 lines · Jes Sorensen

Quoted 15 lines · Ingo Molnar

Direct access to counters is not something that is a big issue. [ Given that i sometimes can see KVM redraw the screen of a guest OS real-time i doubt this is the biggest of performance challenges right now ;-) ]

By far the biggest instrumentation issue is:

- availability
- usability
- flexibility

Exposing the raw hw is a step backwards in many regards. The same way we dont want to expose chipsets to the guest to allow them to do RAS. The same way we dont want to expose most raw PCI devices to guest in general, but have all these virt driver abstractions.

Quoted 4 lines

I have to say I disagree on that. When you run perfmon on a system, it is normally to measure a specific application. You want to see accurate numbers for cache misses, mul instructions or whatever else is selected.

You can still get those. You can even enable RDPMC access and avoid VM exits.

What you _cannot_ do is to 'steal' the PMU and just give it to the guest.

Quoted 4 lines

Emulating the PMU rather than using the real one, makes the numbers far less useful. The most useful way to provide PMU support in a guest is to expose the real PMU and let the guest OS program it.

Firstly, an emulated PMU was only the second-tier option i suggested. By far the best approach is native API to the host regarding performance events and good guest side integration.

Secondly, the PMU cannot be 'given' to the guest in the general case. Those are privileged registers. They can expose sensitive host execution details, etc. etc. So if you emulate a PMU you have to exit out of most PMU accesses anyway for a secure solution. (RDPMC can still be supported, but in close cooperation with the host)

Quoted 3 lines

We can do this in a reasonable way today, if we allow to take the PMU away from the host, and only let guests access it when it's in use. [...]

You get my sure-fire NAK for that kind of crap though. Interfering with the host PMU and stealing it, is not a technical approach that has acceptable quality.

A native API to the host will lock out 100% of the install base now, and a large section of any future install base.

Quoted 3 lines · Avi Kivity

A native API to the host will lock out 100% of the install base now, and a large section of any future install base.

... which is why i suggested the soft-PMU approach.

And note that _any_ solution we offer locks out 100% of the installed base right now, as no solution is in the kernel yet. The only question is what kind of upgrade effort is needed for users to make use of the feature.

Quoted 4 lines · Ingo Molnar

Quoted 2 lines · Avi Kivity

A native API to the host will lock out 100% of the install base now, and a large section of any future install base.

Quoted 1 line

... which is why i suggested the soft-PMU approach.

Not sure I understand it completely.

Do you mean to take the model specific host pmu events, and expose them to the guest via trap'n'emulate? In that case we may as well assign the host pmu to the guest if the host isn't using it, and avoid the traps.

Do you mean to choose some older pmu and emulate it using whatever pmu model the host has? I haven't checked, but aren't there mutually exclusive events in every model pair? The closest thing would be the architectural pmu thing.

Or do you mean to define a new, kvm-specific pmu model and feed it off the host pmu? In this case all the guests will need to be taught about it, which raises the compatibility problem.

Quoted 3 lines

And note that _any_ solution we offer locks out 100% of the installed base right now, as no solution is in the kernel yet. The only question is what kind of upgrade effort is needed for users to make use of the feature.

I meant the guest installed base. Hosts can be upgraded transparently to the guests (not even a shutdown/reboot).

Quoted 3 lines · Jes Sorensen

I have to say I disagree on that. When you run perfmon on a system, it is normally to measure a specific application. You want to see accurate numbers for cache misses, mul instructions or whatever else is selected.

You can still get those. You can even enable RDPMC access and avoid VM exits.

What you _cannot_ do is to 'steal' the PMU and just give it to the guest.

Firstly, an emulated PMU was only the second-tier option i suggested. By far the best approach is native API to the host regarding performance events and good guest side integration.

Secondly, the PMU cannot be 'given' to the guest in the general case. Those are privileged registers. They can expose sensitive host execution details, etc. etc. So if you emulate a PMU you have to exit out of most PMU accesses anyway for a secure solution. (RDPMC can still be supported, but in close cooperation with the host)

Quoted 2 lines

We can do this in a reasonable way today, if we allow to take the PMU away from the host, and only let guests access it when it's in use. [...]

You get my sure-fire NAK for that kind of crap though. Interfering with the host PMU and stealing it, is not a technical approach that has acceptable quality.

You need to integrate it properly so that host PMU functionality still works fine. (Within hardware constraints)

Quoted 1 line · Ingo Molnar

... which is why i suggested the soft-PMU approach.

Not sure I understand it completely.

Do you mean to take the model specific host pmu events, and expose them to the guest via trap'n'emulate? In that case we may as well assign the host pmu to the guest if the host isn't using it, and avoid the traps.

Do you mean to choose some older pmu and emulate it using whatever pmu model the host has? I haven't checked, but aren't there mutually exclusive events in every model pair? The closest thing would be the architectural pmu thing.

Or do you mean to define a new, kvm-specific pmu model and feed it off the host pmu? In this case all the guests will need to be taught about it, which raises the compatibility problem.

Quoted 10 lines · Avi Kivity

Quoted 8 lines · Ingo Molnar

Quoted 3 lines · Avi Kivity

A native API to the host will lock out 100% of the install base now, and a large section of any future install base.

... which is why i suggested the soft-PMU approach.

Quoted 6 lines

Not sure I understand it completely.

Do you mean to take the model specific host pmu events, and expose them to the guest via trap'n'emulate? In that case we may as well assign the host pmu to the guest if the host isn't using it, and avoid the traps.

You are making the incorrect assumption that the emulated PMU uses up all host PMU resources ...

Quoted 5 lines

Do you mean to choose some older pmu and emulate it using whatever pmu model the host has? I haven't checked, but aren't there mutually exclusive events in every model pair? The closest thing would be the architectural pmu thing.

Yes, something like Core2 with 2 generic events.

That would leave 2 extra generic events on Nehalem and better. (which is really the target CPU type for any new feature we are talking about right now. Plus performance analysis tends to skew towards more modern CPU types as well.)

Plus the emulation can be smart about it and only use up a given number. Most guest OSs dont use the full PMU - they use a single counter.

Ideally for Linux<->Linux there would be a PMU paravirt driver that allocates events on an as-needed basis.

Quoted 9 lines

Or do you mean to define a new, kvm-specific pmu model and feed it off the host pmu? In this case all the guests will need to be taught about it, which raises the compatibility problem.

Quoted 4 lines

And note that _any_ solution we offer locks out 100% of the installed base right now, as no solution is in the kernel yet. The only question is what kind of upgrade effort is needed for users to make use of the feature.

Quoted 3 lines

I meant the guest installed base. Hosts can be upgraded transparently to the guests (not even a shutdown/reboot).

The irony: this time guest-transparent solutions that need no configuration are good? ;-)

The very same argument holds for the file server thing: a guest transparent solution is easier wrt. the upgrade path.

Quoted 10 lines · Ingo Molnar

Quoted 4 lines · Jes Sorensen

I have to say I disagree on that. When you run perfmon on a system, it is normally to measure a specific application. You want to see accurate numbers for cache misses, mul instructions or whatever else is selected.

You can still get those. You can even enable RDPMC access and avoid VM exits.

Quoted 1 line

What you _cannot_ do is to 'steal' the PMU and just give it to the guest.

Well you cannot steal the PMU without collaborating with perf_event.c, but thats quite feasible. Sharing the PMU between the guest and the host is very costly and guarantees incorrect results in the host. Unless you completely emulate the PMU by faking it and then allocating PMU counters one by one at the host level. However that means trapping a lot of MSR access.

Quoted 4 lines

Firstly, an emulated PMU was only the second-tier option i suggested. By far the best approach is native API to the host regarding performance events and good guest side integration.

Quoted 5 lines

Secondly, the PMU cannot be 'given' to the guest in the general case. Those are privileged registers. They can expose sensitive host execution details, etc. etc. So if you emulate a PMU you have to exit out of most PMU accesses anyway for a secure solution. (RDPMC can still be supported, but in close cooperation with the host)

There is nothing secret in the host PMU, and it's easy to clear out the counters before passing them off to the guest.

Quoted 6 lines

Quoted 2 lines

We can do this in a reasonable way today, if we allow to take the PMU away from the host, and only let guests access it when it's in use. [...]

You get my sure-fire NAK for that kind of crap though. Interfering with the host PMU and stealing it, is not a technical approach that has acceptable quality.

Having an allocation scheme and sharing it with the host, is a perfectly legitimate and very clean way to do it. Once it's given to the guest, the host knows not to touch it until it's been released again.

Quoted 2 lines

You need to integrate it properly so that host PMU functionality still works fine. (Within hardware constraints)

Well with the hardware currently available, there is no such thing as clean sharing between the host and the guest. It cannot be done without messing up the host measurements, which effectively renders measuring at the host side useless while a guest is allowed access to the PMU.

Quoted 7 lines · Ingo Molnar

Yes, something like Core2 with 2 generic events.

That would leave 2 extra generic events on Nehalem and better. (which is really the target CPU type for any new feature we are talking about right now.
Plus performance analysis tends to skew towards more modern CPU types as well.)

Can you emulate the Core 2 pmu on, say, a P4? Those P4s have very different instruction caches so I imagine the events are very different as well.

Agree about favouring modern processors.

Quoted 3 lines

Plus the emulation can be smart about it and only use up a given number. Most
guest OSs dont use the full PMU - they use a single counter.

But you have to expose all of the counters, no? Unless you go with a kvm-specific pmu as described below.

Can you emulate the Core 2 pmu on, say, a P4? [...]

Quoted 1 line

Agree about favouring modern processors.

You certainly cannot emulate the Core2 on a P4. The Core2 is Perfmon v2, whereas Nehalem and Atom are v3 if I remember correctly. [...]

Quoted 5 lines

Quoted 5 lines

Firstly, an emulated PMU was only the second-tier option i suggested. By far the best approach is native API to the host regarding performance events and good guest side integration.

Quoted 9 lines

Quoted 6 lines

Secondly, the PMU cannot be 'given' to the guest in the general case. Those are privileged registers. They can expose sensitive host execution details, etc. etc. So if you emulate a PMU you have to exit out of most PMU accesses anyway for a secure solution. (RDPMC can still be supported, but in close cooperation with the host)

There is nothing secret in the host PMU, and it's easy to clear out the counters before passing them off to the guest.

That's wrong. On some CPUs the host PMU can be used to say sample aspects of another CPU, allowing statistical attacks to recover crypto keys. It can be used to sample memory access patterns of another node.

There's a good reason PMU configuration registers are privileged and there's good value in only giving a certain sub-set to less privileged entities by default.

Quoted 4 lines

Having an allocation scheme and sharing it with the host, is a perfectly legitimate and very clean way to do it. Once it's given to the guest, the host knows not to touch it until it's been released again.

'Full PMU' is not the granularity i find acceptable though: please do what i suggested, event granularity allocation and scheduling.

We are rehashing the whole 'perfmon versus perf events/counters' design arguments again here really.

Quoted 8 lines

Quoted 3 lines

You need to integrate it properly so that host PMU functionality still works fine. (Within hardware constraints)

Well with the hardware currently available, there is no such thing as clean sharing between the host and the guest. It cannot be done without messing up the host measurements, which effectively renders measuring at the host side useless while a guest is allowed access to the PMU.

That's precisely my point: the guest should obviously not get raw access to the PMU. (except where it might matter to performance, such as RDPMC)

Quoted 5 lines · Jes Sorensen

Well you cannot steal the PMU without collaborating with perf_event.c, but thats quite feasible. Sharing the PMU between the guest and the host is very costly and guarantees incorrect results in the host. Unless you completely emulate the PMU by faking it and then allocating PMU counters one by one at the host level. However that means trapping a lot of MSR access.

It's not that many MSR accesses.

Quoted 2 lines

There is nothing secret in the host PMU, and it's easy to clear out the counters before passing them off to the guest.

That's wrong. On some CPUs the host PMU can be used to say sample aspects of another CPU, allowing statistical attacks to recover crypto keys. It can be used to sample memory access patterns of another node.

There's a good reason PMU configuration registers are privileged and there's good value in only giving a certain sub-set to less privileged entities by default.

Quoted 3 lines

Having an allocation scheme and sharing it with the host, is a perfectly legitimate and very clean way to do it. Once it's given to the guest, the host knows not to touch it until it's been released again.

'Full PMU' is not the granularity i find acceptable though: please do what i suggested, event granularity allocation and scheduling.

Quoted 4 lines

Well with the hardware currently available, there is no such thing as clean sharing between the host and the guest. It cannot be done without messing up the host measurements, which effectively renders measuring at the host side useless while a guest is allowed access to the PMU.

That's precisely my point: the guest should obviously not get raw access to the PMU. (except where it might matter to performance, such as RDPMC)

Or do you mean to define a new, kvm-specific pmu model and feed it off the host pmu? In this case all the guests will need to be taught about it, which raises the compatibility problem.

For Linux<->Linux the sanest, tier-1 approach would be to map sys_perf_open() on the guest side over to the host, transparently, via a paravirt driver.

Quoted 3 lines · Ingo Molnar

That's precisely my point: the guest should obviously not get raw access to
the PMU. (except where it might matter to performance, such as RDPMC)

That's doable if all counters are steerable. IIRC some counters are fixed function, but I'm not certain about that.

It would be the other way round - the host would steal the pmu from the guest. Later we can try to time-slice and extrapolate, though that's not going to be easy.

Quoted 1 line · Avi Kivity

Can you emulate the Core 2 pmu on, say, a P4? [...]

How about the Pentium? Or the i486?

As long as there's perf events support, the CPU can be supported in a soft PMU. You can even cross-map exotic hw events if need to be - but most of the tooling (in just about any OS) uses just a handful of core events ...

You are missing two big things wrt. compatibility here:

1) The first upgrade overhead a one time overhead only.

2) Once a Linux guest has upgraded, it will work in the future, with _any_ future CPU - _without_ having to upgrade the guest!

Dont you see the advantage of that? You can instrument an old system on new hardware, without having to upgrade that guest for the new CPU support.

With the 'steal the PMU' messy approach the guest OS has to be upgraded to the new CPU type all the time. Ad infinitum.

Quoted 4 lines · Avi Kivity

Or do you mean to define a new, kvm-specific pmu model and feed it off the host pmu? In this case all the guests will need to be taught about it, which raises the compatibility problem.

You are missing two big things wrt. compatibility here:

1) The first upgrade overhead a one time overhead only.

2) Once a Linux guest has upgraded, it will work in the future, with _any_ future CPU - _without_ having to upgrade the guest!

Dont you see the advantage of that? You can instrument an old system on new hardware, without having to upgrade that guest for the new CPU support.

Quoted 16 lines · Ingo Molnar

Quoted 5 lines

Quoted 5 lines

Firstly, an emulated PMU was only the second-tier option i suggested. By far the best approach is native API to the host regarding performance events and good guest side integration.

Quoted 7 lines

Quoted 5 lines

Secondly, the PMU cannot be 'given' to the guest in the general case. Those are privileged registers. They can expose sensitive host execution details, etc. etc. So if you emulate a PMU you have to exit out of most PMU accesses anyway for a secure solution. (RDPMC can still be supported, but in close cooperation with the host)

There is nothing secret in the host PMU, and it's easy to clear out the counters before passing them off to the guest.

That's wrong. On some CPUs the host PMU can be used to say sample aspects of another CPU, allowing statistical attacks to recover crypto keys. It can be used to sample memory access patterns of another node.

Quoted 3 lines

There's a good reason PMU configuration registers are privileged and there's good value in only giving a certain sub-set to less privileged entities by default.

Even if there were no security considerations, if the guest can observe host data in the pmu, it means the pmu is inaccurate. We should expose guest data only in the guest pmu. That's not difficult to do, you stop the pmu on exit and swap the counters on context switches.

Quoted 4 lines · Avi Kivity

It would be the other way round - the host would steal the pmu from the guest. Later we can try to time-slice and extrapolate, though that's not going to be easy.

Right, so perf already does the time slicing and interpolating thing, so a soft-pmu gets that for free.

Anyway, this discussion seems somewhat in a stale-mate position.

The KVM folks basically demand a full PMU MSR shadow with PMI passthrough so that their $legacy shit works without modification.

My question with that is how $legacy muck can ever know how the current PMU works, you can't even properly emulate a core2 pmu on a nehalem because intel keeps messing with the event codes for every new model.

So basically for this to work means the guest can't run legacy stuff anyway, but needs to run very up-to-date software, so we might as well create a soft-pmu/paravirt interface now and have all up-to-date software support that for the next generation.

Quoted 3 lines · Avi Kivity

It would be the other way round - the host would steal the pmu from the guest. Later we can try to time-slice and extrapolate, though that's not going to be easy.

Right, so perf already does the time slicing and interpolating thing, so a soft-pmu gets that for free.

Anyway, this discussion seems somewhat in a stale-mate position.

The KVM folks basically demand a full PMU MSR shadow with PMI passthrough so that their $legacy shit works without modification.

My question with that is how $legacy muck can ever know how the current PMU works, you can't even properly emulate a core2 pmu on a nehalem because intel keeps messing with the event codes for every new model.

So basically for this to work means the guest can't run legacy stuff anyway, but needs to run very up-to-date software, so we might as well create a soft-pmu/paravirt interface now and have all up-to-date software support that for the next generation.

Furthermore, when KVM doesn't virtualize the physical system topology, some PMU features cannot even be sanely used from a vcpu.

So while currently a root user can already tie up all of the pmu using perf, simply using that to hand the full pmu off to the guest still leaves lots of issues.

Quoted 24 lines · Avi Kivity

Quoted 22 lines · Ingo Molnar

Quoted 5 lines

Quoted 5 lines

Firstly, an emulated PMU was only the second-tier option i suggested. By far the best approach is native API to the host regarding performance events and good guest side integration.

Quoted 9 lines

Quoted 6 lines

Secondly, the PMU cannot be 'given' to the guest in the general case. Those are privileged registers. They can expose sensitive host execution details, etc. etc. So if you emulate a PMU you have to exit out of most PMU accesses anyway for a secure solution. (RDPMC can still be supported, but in close cooperation with the host)

There is nothing secret in the host PMU, and it's easy to clear out the counters before passing them off to the guest.

That's wrong. On some CPUs the host PMU can be used to say sample aspects of another CPU, allowing statistical attacks to recover crypto keys. It can be used to sample memory access patterns of another node.

There's a good reason PMU configuration registers are privileged and there's good value in only giving a certain sub-set to less privileged entities by default.

Quoted 5 lines

Even if there were no security considerations, if the guest can observe host data in the pmu, it means the pmu is inaccurate. We should expose guest data only in the guest pmu. That's not difficult to do, you stop the pmu on exit and swap the counters on context switches.

Again you are making an incorrect assumption: that information leakage via the PMU only occurs while the host is running on that CPU. It does not - the PMU can leak general system details _while the guest is running_.

So for this and for the many other reasons we dont want to give a raw PMU to guests:

- A paravirt event driver is more compatible and more transparent in the long
run: it allows hardware upgrade and upgraded PMU functionality (for Linux) without having to upgrade the guest OS. Via that a guest OS could even be live-migrated to a different PMU, without noticing anything about it.

In contrast, a 'stolen', 'raw' PMU directly programmed by the guest OS always assumes the guest OS is upgraded to the host. Also, 'raw' PMU state cannot be live-migrated. (save/restore doesnt help)

- It's far cleaner on the host side as well: more granular, per event usage is possible. The guest can use portion of the PMU (managed by the host), and the host can use a portion too.

In contrast, a 'stolen', 'raw' PMU directly programmed by the guest OS precludes the host OS from running some different piece of instrumentation at the same time.

- It's more secure: the host can have a finegrained policy about what kinds of events it exposes to the guest. It might chose to only expose software events for example.

In contrast, a 'stolen', 'raw' PMU directly programmed by the guest OS is an all-or-nothing policy affair: either you fully allow the guest (and live with whatever consequences the piece of hardware that takes up a fair chunk on the CPU die causes), or you allow none of it.

- A proper paravirt event driver gives more features as well: it can exposes host software events and tracepoints, probes - not restricting itself to the 'hardware PMU' abstraction.

- There's proper event scheduling and event allocation. Time-slicing, etc.

The thing is, we made quite similar arguments in the past, during the perfmon vs. perfcounters discussions. There's really a big advantage to proper abstractions, both on the host and on the guest side.

Quoted 8 lines · Ingo Molnar

You are missing two big things wrt. compatibility here:

1) The first upgrade overhead a one time overhead only.

2) Once a Linux guest has upgraded, it will work in the future,
with _any_
future CPU - _without_ having to upgrade the guest!

Quoted 2 lines

Dont you see the advantage of that? You can instrument an old system on new hardware, without having to upgrade that guest for the new CPU support.

That would only work if you are guaranteed to be able to emulate old hardware on new hardware. Not going to be feasible, so then we are in a real mess.

Quoted 5 lines · Ingo Molnar

Quoted 3 lines · Avi Kivity

Or do you mean to define a new, kvm-specific pmu model and feed it off the host pmu? In this case all the guests will need to be taught about it, which raises the compatibility problem.

Quoted 3 lines

You are missing two big things wrt. compatibility here:

1) The first upgrade overhead a one time overhead only.

May be one too many, for certain guests. Of course it may be argued that if the guest wants performance monitoring that much, they will upgrade.

Certainly guests that we don't port won't be able to use this. I doubt we'll be able to make Windows work with this - the only performance tool I'm familiar with on Windows is Intel's VTune, and that's proprietary.

Quoted 5 lines

2) Once a Linux guest has upgraded, it will work in the future, with _any_ future CPU - _without_ having to upgrade the guest!

Dont you see the advantage of that? You can instrument an old system on new hardware, without having to upgrade that guest for the new CPU support.

That also works for the architectural pmu, of course that's Intel only. And there you don't need to upgrade the guest even once.

Quoted 2 lines · Jes Sorensen

You certainly cannot emulate the Core2 on a P4. The Core2 is Perfmon v2, whereas Nehalem and Atom are v3 if I remember correctly. [...]

Of course you can emulate a good portion of it, as long as there's perf support on the host side for P4.

If the guest programs a cachemiss event, you program a cachemiss perf event on the host and feed its values to the emulated MSR state. You _dont_ program the raw PMU on the host side - just use the API i outlined to get struct perf_event.

The emulation wont be perfect: not all events will count and not all events will be available in a P4 (and some Core2 events might not even make sense in a P4), but that is reality as well: often documented events dont count, and often non-documented events count.

What matters to 99.9% of people who actually use this stuff is a few core sets of events - which are available in P4s and in Core2 as well. Cycles, instructions, branches, maybe cache-misses. Sometimes FPU stuff.

For Linux<->Linux the sanest, tier-1 approach would be to map sys_perf_open() on the guest side over to the host, transparently, via a paravirt driver.

Quoted 5 lines · Ingo Molnar

Quoted 2 lines · Jes Sorensen

You certainly cannot emulate the Core2 on a P4. The Core2 is Perfmon v2, whereas Nehalem and Atom are v3 if I remember correctly. [...]

Quoted 2 lines

Of course you can emulate a good portion of it, as long as there's perf support on the host side for P4.

Actually P4 is pretty uninteresting in this discussion due to the lack of VMX support, it's the same issue for Nehalem vs Core2. The problem is the same though, we cannot tell the guest that yes P4 has this event, but no, we are going to feed you bogus data.

Quoted 25 lines · Ingo Molnar

Quoted 23 lines · Avi Kivity

Quoted 11 lines · Ingo Molnar

Quoted 11 lines

Quoted 9 lines

Firstly, an emulated PMU was only the second-tier option i suggested. By far the best approach is native API to the host regarding performance events and good guest side integration.

Secondly, the PMU cannot be 'given' to the guest in the general case. Those are privileged registers. They can expose sensitive host execution details, etc. etc. So if you emulate a PMU you have to exit out of most PMU accesses anyway for a secure solution. (RDPMC can still be supported, but in close cooperation with the host)

There is nothing secret in the host PMU, and it's easy to clear out the counters before passing them off to the guest.

Quoted 7 lines

That's wrong. On some CPUs the host PMU can be used to say sample aspects of another CPU, allowing statistical attacks to recover crypto keys. It can be used to sample memory access patterns of another node.

There's a good reason PMU configuration registers are privileged and there's good value in only giving a certain sub-set to less privileged entities by default.

Even if there were no security considerations, if the guest can observe host data in the pmu, it means the pmu is inaccurate. We should expose guest data only in the guest pmu. That's not difficult to do, you stop the pmu on exit and swap the counters on context switches.

Quoted 3 lines

Again you are making an incorrect assumption: that information leakage via the PMU only occurs while the host is running on that CPU. It does not - the PMU can leak general system details _while the guest is running_.

You mean like bus transactions on a multicore? Well, we're already exposed to cache timing attacks.

Quoted 7 lines

So for this and for the many other reasons we dont want to give a raw PMU to guests:

- A paravirt event driver is more compatible and more transparent in the long
run: it allows hardware upgrade and upgraded PMU functionality (for Linux) without having to upgrade the guest OS. Via that a guest OS could even be live-migrated to a different PMU, without noticing anything about it.

What about Windows?

Quoted 3 lines

In contrast, a 'stolen', 'raw' PMU directly programmed by the guest OS always assumes the guest OS is upgraded to the host. Also, 'raw' PMU state cannot be live-migrated. (save/restore doesnt help)

Why not? So long as the source and destination are compatible?

Quoted 7 lines

- It's far cleaner on the host side as well: more granular, per event usage is possible. The guest can use portion of the PMU (managed by the host), and the host can use a portion too.

In contrast, a 'stolen', 'raw' PMU directly programmed by the guest OS precludes the host OS from running some different piece of instrumentation at the same time.

Right, time slicing is something we want.

Quoted 4 lines

- It's more secure: the host can have a finegrained policy about what kinds of events it exposes to the guest. It might chose to only expose software events for example.

Quoted 4 lines

In contrast, a 'stolen', 'raw' PMU directly programmed by the guest OS is an all-or-nothing policy affair: either you fully allow the guest (and live with whatever consequences the piece of hardware that takes up a fair chunk on the CPU die causes), or you allow none of it.

No, we can hide insecure events with a full pmu. Trap the control register and don't pass it on to the hardware.

Quoted 3 lines

- A proper paravirt event driver gives more features as well: it can exposes host software events and tracepoints, probes - not restricting itself to the 'hardware PMU' abstraction.

But it is limited to whatever the host stack supports. At least that's our control, but things like PEBS will take a ton of work.

Quoted 6 lines

- There's proper event scheduling and event allocation. Time-slicing, etc.

The thing is, we made quite similar arguments in the past, during the perfmon vs. perfcounters discussions. There's really a big advantage to proper abstractions, both on the host and on the guest side.

We only control half of the equation. That's very different compared to tools/perf.

Quoted 9 lines · Avi Kivity

Quoted 7 lines · Ingo Molnar

Quoted 4 lines · Avi Kivity

Or do you mean to define a new, kvm-specific pmu model and feed it off the host pmu? In this case all the guests will need to be taught about it, which raises the compatibility problem.

Quoted 8 lines

Quoted 4 lines

You are missing two big things wrt. compatibility here:

1) The first upgrade overhead a one time overhead only.

May be one too many, for certain guests. Of course it may be argued that if the guest wants performance monitoring that much, they will upgrade.

Yes, that can certainly be argued.

Note another logical inconsistency: you are assuming reluctance to upgrade for a set of users who are doing _performance analysis_.

In fact those types of users are amongst the most upgrade-happy. Often they'll run modern hardware and modern software. Most of the time they are developers themselves who try to make sure their stuff works on the latest& greatest hardware _and_ software.

Quoted 4 lines

Certainly guests that we don't port won't be able to use this. I doubt we'll be able to make Windows work with this - the only performance tool I'm familiar with on Windows is Intel's VTune, and that's proprietary.

Dont you see the extreme irony of your wish to limit Linux kernel design decisions and features based on ... Windows and other proprietary software?

Quoted 9 lines

Quoted 6 lines

2) Once a Linux guest has upgraded, it will work in the future, with _any_ future CPU - _without_ having to upgrade the guest!

Dont you see the advantage of that? You can instrument an old system on new hardware, without having to upgrade that guest for the new CPU support.

That also works for the architectural pmu, of course that's Intel only. And there you don't need to upgrade the guest even once.

Besides being Intel only, it only exposes a limited sub-set of hw events. (far fewer than the generic ones offered by perf events)

That actually works on the Intel-only architectural pmu. I'm beginning to like it more and more.

Quoted 7 lines

Quoted 3 lines

Certainly guests that we don't port won't be able to use this. I doubt we'll be able to make Windows work with this - the only performance tool I'm familiar with on Windows is Intel's VTune, and that's proprietary.

Dont you see the extreme irony of your wish to limit Linux kernel design decisions and features based on ... Windows and other proprietary software?

Not at all. Virtualization is a hardware compatibility game. To see what happens if you don't play it, see Xen. Eventually they to implemented hardware support even though the pv approach is so wonderful.

Quoted 3 lines

Furthermore, when KVM doesn't virtualize the physical system topology, some PMU features cannot even be sanely used from a vcpu.

That is definitely an issue, and there is nothing we can really do about that. Having two guests running in parallel under KVM means that they are going to see more cache misses than they would if they ran barebone on the hardware.

However even with all of this, we have to keep in mind who is going to use the performance monitoring in a guest. It is going to be application writers, mostly people writing analytical/scientific applications. They rarely have control over the OS they are running on, but are given systems and told to work on what they are given. Driver upgrades and things like that don't come quickly. However they also tend to understand limitations like these and will be able to still benefit from perf on a system like that.

All that said, what we really want is for Intel+AMD to come up with proper hw PMU virtualization support that makes it easy to rotate the full PMU in and out for a guest. Then this whole discussion will become a non issue.

Even if there were no security considerations, if the guest can observe host data in the pmu, it means the pmu is inaccurate. We should expose guest data only in the guest pmu. That's not difficult to do, you stop the pmu on exit and swap the counters on context switches.

Scheduling at event granularity would be a good thing. However we need to be able to handle the guest using the full pmu.

Quoted 5 lines

Quoted 5 lines

Again you are making an incorrect assumption: that information leakage via the PMU only occurs while the host is running on that CPU. It does not - the PMU can leak general system details _while the guest is running_.

Quoted 3 lines

You mean like bus transactions on a multicore? Well, we're already exposed to cache timing attacks.

If you give a full PMU to a guest it's a whole different dimension and quality of information. Literally hundreds of different events about all sorts of aspects of the CPU and the hardware in general.

Quoted 8 lines

Quoted 8 lines

So for this and for the many other reasons we dont want to give a raw PMU to guests:

- A paravirt event driver is more compatible and more transparent in the long
run: it allows hardware upgrade and upgraded PMU functionality (for Linux) without having to upgrade the guest OS. Via that a guest OS could even be live-migrated to a different PMU, without noticing anything about it.

Quoted 2 lines

What about Windows?

What is your question? Why should i limit Linux kernel design decisions based on any aspect of Windows? You might want to support it, but _please_ dont let the design be dictated by it ...

Quoted 6 lines

Quoted 4 lines

In contrast, a 'stolen', 'raw' PMU directly programmed by the guest OS always assumes the guest OS is upgraded to the host. Also, 'raw' PMU state cannot be live-migrated. (save/restore doesnt help)

Why not? So long as the source and destination are compatible?

'As long as it works' is certainly a good enough filter for quality ;-)

Quoted 3 lines

No, we can hide insecure events with a full pmu. Trap the control register and don't pass it on to the hardware.

So you basically concede partial emulation ...

Quoted 7 lines

Quoted 7 lines

- There's proper event scheduling and event allocation. Time-slicing, etc.

The thing is, we made quite similar arguments in the past, during the perfmon vs. perfcounters discussions. There's really a big advantage to proper abstractions, both on the host and on the guest side.

Quoted 3 lines

We only control half of the equation. That's very different compared to tools/perf.

You mean Windows?

For heaven's sake, why dont you think like Linus thought 20 years ago. To the hell with Windows suckiness and lets make sure our stuff works well.

Then the
users will come, developers will come, and people will profile Linux under Linux and maybe the tools will be so good that they'll profile under Linux using Wine just to be able to use those good tools...

If you gut Linux capabilities like that to accomodate for the suckiness of Windows, without giving a technological edge to Linux, and then we are bound to fail in the long run ...

Quoted 3 lines · Avi Kivity

That actually works on the Intel-only architectural pmu. I'm beginning to like it more and more.

Only for the arch defined events, all _7_ of them.

Quoted 7 lines · Peter Zijlstra

Quoted 3 lines · Avi Kivity

That actually works on the Intel-only architectural pmu. I'm beginning to like it more and more.

Only for the arch defined events, all _7_ of them.

That's 7 more than what we support now, and 7 more than what we can guarantee without it.

Quoted 3 lines

That's 7 more than what we support now, and 7 more than what we can guarantee without it.

Again, what windows software uses only those 7? Does it pay to only have access to those 7 or does it limit the usability to exactly the same subset a paravirt interface would?

Quoted 4 lines

If you give a full PMU to a guest it's a whole different dimension and quality of information. Literally hundreds of different events about all sorts of aspects of the CPU and the hardware in general.

Well, we filter out the bad events then.

Quoted 7 lines · Peter Zijlstra

Quoted 3 lines

That's 7 more than what we support now, and 7 more than what we can guarantee without it.

Again, what windows software uses only those 7? Does it pay to only have access to those 7 or does it limit the usability to exactly the same subset a paravirt interface would?

Good question. Would be interesting to try out VTune with the non-arch pmu masked out.

From what I understood VTune uses PEBS+LBR, although I suppose they have

Quoted 7 lines · Peter Zijlstra

Quoted 3 lines

That's 7 more than what we support now, and 7 more than what we can guarantee without it.

Again, what windows software uses only those 7? Does it pay to only have access to those 7 or does it limit the usability to exactly the same subset a paravirt interface would?

Good question. Would be interesting to try out VTune with the non-arch pmu masked out.

Quoted 4 lines · Avi Kivity

Quoted 4 lines

Quoted 4 lines

Certainly guests that we don't port won't be able to use this. I doubt we'll be able to make Windows work with this - the only performance tool I'm familiar with on Windows is Intel's VTune, and that's proprietary.

Quoted 8 lines

Quoted 4 lines

Dont you see the extreme irony of your wish to limit Linux kernel design decisions and features based on ... Windows and other proprietary software?

Not at all. Virtualization is a hardware compatibility game. To see what happens if you don't play it, see Xen. Eventually they to implemented hardware support even though the pv approach is so wonderful.

That's not quite equivalent though.

KVM used to be the clean, integrate-code-with-Linux virtualization approach, designed specifically for CPUs that can be virtualized properly. (VMX support first, then SVM, etc.)

KVM virtualized ages-old concepts with relatively straightforward hardware
ABIs: x86 execution, IRQ abstractions, device abstractions, etc.

Now you are in essence turning that all around:

- the PMU is by no means properly virtualized nor really virtualizable by direct access. There's no virtual PMU that ticks independently of the host PMU.

- the PMU hardware itself is not a well standardized piece of hardware. It's very vendor dependent and very limiting.

So to some degree you are playing the role of Xen in this specific affair. You are pushing for something that shouldnt be done in that form. You want to interfere with the host PMU by going via the fast& easy short-term hack to just let the guest OS have the PMU, without any regard to how this impacts long-term feasible solutions.

I.e. you are a bit like the guy who would have told Linus in 1994:

" Dude, why dont you use the Windows APIs? It's far more compatible and
that's the only way you could run any serious apps. Besides, it requires no upgrade. Admittedly it's a bit messy and 16-bit but hey, that's our installed base after all. "

BTW, just wondering, why would a developer be running VTune in a guest anyway? I'd think that a developer that windows oriented would simply run windows on his desktop and VTune there.

- A paravirt event driver is more compatible and more transparent in the long
run: it allows hardware upgrade and upgraded PMU functionality (for Linux) without having to upgrade the guest OS. Via that a guest OS could even be live-migrated to a different PMU, without noticing anything about it.

In contrast, a 'stolen', 'raw' PMU directly programmed by the guest OS always assumes the guest OS is upgraded to the host. Also, 'raw' PMU state cannot be live-migrated. (save/restore doesnt help)

- It's more secure: the host can have a finegrained policy about what kinds of events it exposes to the guest. It might chose to only expose software events for example.

You mean Windows?

For heaven's sake, why dont you think like Linus thought 20 years ago. To the hell with Windows suckiness and lets make sure our stuff works well. Then the users will come, developers will come, and people will profile Linux under Linux and maybe the tools will be so good that they'll profile under Linux using Wine just to be able to use those good tools...

Quoted 6 lines · Avi Kivity

Quoted 4 lines

If you give a full PMU to a guest it's a whole different dimension and quality of information. Literally hundreds of different events about all sorts of aspects of the CPU and the hardware in general.

Well, we filter out the bad events then.

Which requires trapping the MSR access, at which point a soft-PMU is almost there, right?

Quoted 3 lines

- It's more secure: the host can have a finegrained policy about what kinds of events it exposes to the guest. It might chose to only expose software events for example.

What do you mean by software events?

Quoted 8 lines · Ingo Molnar

- A paravirt event driver is more compatible and more transparent in the long
run: it allows hardware upgrade and upgraded PMU functionality (for Linux) without having to upgrade the guest OS. Via that a guest OS could even be live-migrated to a different PMU, without noticing anything about it.

In contrast, a 'stolen', 'raw' PMU directly programmed by the guest OS always assumes the guest OS is upgraded to the host. Also, 'raw' PMU state cannot be live-migrated. (save/restore doesnt help)

I agree with your arguments, having this soft-pmu for the guest has some advantages over raw pmu access. It has a lot of advantages if the guest migrated between hosts with different hardware.

But I still think we should have both, a soft-pmu and a pmu-emulation (which is a more accurate term than 'raw guest pmu access') that looks to the guest as a real hardware pmu would look like. On a linux host that is dedicated to executing virtual kvm machines there is little point in sharing the pmu between guest and host because the host will probably never use it.

Quoted 1 line

What do you mean by software events?

Things like:

aldebaran:~> perf stat -a sleep 1

Performance counter stats for 'sleep 1':

   15995.719133  task-clock-msecs         #     15.981 CPUs 
           5787  context-switches         #      0.000 M/sec
            210  CPU-migrations           #      0.000 M/sec
         193909  page-faults              #      0.012 M/sec
    28704833507  cycles                   #   1794.532 M/sec  (scaled from 78.69%)
    14387445668  instructions             #      0.501 IPC    (scaled from 90.71%)
      736644616  branches                 #     46.053 M/sec  (scaled from 90.52%)
      695884659  branch-misses            #     94.467 %      (scaled from 90.70%)
      727070678  cache-references         #     45.454 M/sec  (scaled from 88.11%)
     1305560420  cache-misses             #     81.619 M/sec  (scaled from 52.00%)

    1.000942399  seconds time elapsed

These lines:

   15995.719133  task-clock-msecs         #     15.981 CPUs 
           5787  context-switches         #      0.000 M/sec
            210  CPU-migrations           #      0.000 M/sec
         193909  page-faults              #      0.012 M/sec

Are software events of the host - a subset of which could be transparently exposed to the guest. Same for tracepoints, probes, etc. Those are not exposed by the hardware PMU. So by doing a 'soft' PMU (or even better: a paravirt channel to perf events) we gain a lot more than just raw PMU functionality.

'performance events' are about a lot more than just the PMU, it's a coherent system health / system events / structured logging framework.

Quoted 33 lines · Ingo Molnar

Quoted 1 line

What do you mean by software events?

Things like:

aldebaran:~> perf stat -a sleep 1

Performance counter stats for 'sleep 1':

   15995.719133  task-clock-msecs         #     15.981 CPUs 
           5787  context-switches         #      0.000 M/sec
            210  CPU-migrations           #      0.000 M/sec
         193909  page-faults              #      0.012 M/sec
    28704833507  cycles                   #   1794.532 M/sec  (scaled from 78.69%)
    14387445668  instructions             #      0.501 IPC    (scaled from 90.71%)
      736644616  branches                 #     46.053 M/sec  (scaled from 90.52%)
      695884659  branch-misses            #     94.467 %      (scaled from 90.70%)
      727070678  cache-references         #     45.454 M/sec  (scaled from 88.11%)
     1305560420  cache-misses             #     81.619 M/sec  (scaled from 52.00%)

    1.000942399  seconds time elapsed

These lines:

   15995.719133  task-clock-msecs         #     15.981 CPUs 
           5787  context-switches         #      0.000 M/sec
            210  CPU-migrations           #      0.000 M/sec
         193909  page-faults              #      0.012 M/sec

Are software events of the host - a subset of which could be transparently exposed to the guest. Same for tracepoints, probes, etc. Those are not exposed by the hardware PMU. So by doing a 'soft' PMU (or even better: a paravirt channel to perf events) we gain a lot more than just raw PMU functionality.

Quoted 2 lines

'performance events' are about a lot more than just the PMU, it's a coherent system health / system events / structured logging framework.

Yeah I know. But these event should be available in the guest already, no? [...]

[...] They don't need any kind of hardware support from the pmu. A paravirt perf channel from the guest to the host would be definitly a win. It would be a powerful tool for kvm/linux-guest analysis (e.g. trace host-kvm and guest-events together on the host)

There's a world of a difference between "will not use in certain usecases" and "cannot use at all because we've designed it so". By doing the latter we guarantee that sane shared usage of the PMU will never occur - which is bad.

Think about it: this whole Linux thing is about 'sharing' resources. That concet really works and permeates everything we do in the kernel. Yes, it's somewhat hard for the PMU but we've done it on the host side via perf events and we really dont want to look back ...

My experience is that once the right profiling/tracing tools are there, people will use them in every which way. The bigger a box is, the more likely shared usage will occur - just statistically. Which coincides with KVM's "the bigger the box, the better for virtualization" general mantra.

Quoted 12 lines · Jes Sorensen

Quoted 10 lines · Ingo Molnar

Direct access to counters is not something that is a big issue. [ Given that i sometimes can see KVM redraw the screen of a guest OS real-time i doubt this is the biggest of performance challenges right now ;-) ]

By far the biggest instrumentation issue is:

- availability
- usability
- flexibility

Quoted 9 lines

Quoted 5 lines

Exposing the raw hw is a step backwards in many regards. The same way we dont want to expose chipsets to the guest to allow them to do RAS. The same way we dont want to expose most raw PCI devices to guest in general, but have all these virt driver abstractions.

I have to say I disagree on that. When you run perfmon on a system, it is normally to measure a specific application. You want to see accurate numbers for cache misses, mul instructions or whatever else is selected.

You can still get those. You can even enable RDPMC access and avoid VM exits.

What you _cannot_ do is to 'steal' the PMU and just give it to the guest.

Quoted 4 lines

Emulating the PMU rather than using the real one, makes the numbers far less useful. The most useful way to provide PMU support in a guest is to expose the real PMU and let the guest OS program it.

Firstly, an emulated PMU was only the second-tier option i suggested. By far the best approach is native API to the host regarding performance events and good guest side integration.

Secondly, the PMU cannot be 'given' to the guest in the general case. Those are privileged registers. They can expose sensitive host execution details, etc. etc. So if you emulate a PMU you have to exit out of most PMU accesses anyway for a secure solution. (RDPMC can still be supported, but in close cooperation with the host)

Quoted 3 lines

We can do this in a reasonable way today, if we allow to take the PMU away from the host, and only let guests access it when it's in use. [...]

You get my sure-fire NAK for that kind of crap though. Interfering with the host PMU and stealing it, is not a technical approach that has acceptable quality.

You need to integrate it properly so that host PMU functionality still works fine. (Within hardware constraints)

1. Enhance perf to count pmu events only when cpu is in guest mode.

2. For every emulated performance counter the guest activates kvm allocates a perf_event and configures it for the guest (we may allow kvm to specify the counter index, the guest would be able to use rdpmc unintercepted then). Event filtering is also done in this step.

3. Before vmrun the guest activates all its counters,

this can fail if
the host uses them or the requested pmc index is not available for some reason.

4. Some additional magic to reinject pmu events into the guest

Quoted 2 lines · Joerg Roedel

1. Enhance perf to count pmu events only when cpu is in guest mode.

No enhancements needed, only hardware support for Intel doesn't provide this iirc.

Quoted 1 line

4. Some additional magic to reinject pmu events into the guest

Right, that is needed, and might be 'interesting' since we get them from NMI context.

Quoted 10 lines · Ingo Molnar

You are missing two big things wrt. compatibility here:

1) The first upgrade overhead a one time overhead only.

2) Once a Linux guest has upgraded, it will work in the future,
with _any_
future CPU - _without_ having to upgrade the guest!

Dont you see the advantage of that? You can instrument an old system on new hardware, without having to upgrade that guest for the new CPU support.

That would only work if you are guaranteed to be able to emulate old hardware on new hardware. Not going to be feasible, so then we are in a real mess.

Quoted 2 lines

With the 'steal the PMU' messy approach the guest OS has to be upgraded to the new CPU type all the time. Ad infinitum.

The way the Perfmon architecture is specified by Intel, that is what we are stuck with. It's not going to be possible via software emulation to count cache misses, unless you run it in a micro architecture emulator.

Quoted 6 lines · Avi Kivity

Quoted 4 lines · Peter Zijlstra

Quoted 4 lines

That's 7 more than what we support now, and 7 more than what we can guarantee without it.

Quoted 8 lines

Quoted 5 lines

Again, what windows software uses only those 7? Does it pay to only have access to those 7 or does it limit the usability to exactly the same subset a paravirt interface would?

Good question. Would be interesting to try out VTune with the non-arch pmu masked out.

Also, the ANY bit is part of the intel arch pmu, but you still have to mask it out.

BTW, just wondering, why would a developer be running VTune in a guest anyway? I'd think that a developer that windows oriented would simply run windows on his desktop and VTune there.

Quoted 11 lines · Joerg Roedel

Quoted 9 lines · Yanmin

Quoted 8 lines · Joerg Roedel

Quoted 2 lines · Jes Sorensen

1) Add support to perf to allow it to monitor a KVM guest from the host.

This shouldn't be a big problem. The PMU of AMD Fam10 processors can be configured to count only when in guest mode. Perf needs to be aware of that and fetch the rip from a different place when monitoring a guest.

Quoted 8 lines

Quoted 2 lines

The idea is we want to measure both host and guest at the same time, and compare all the hot functions fairly.

So you want to measure while the guest vcpu is running and the vmexit path of that vcpu (including qemu userspace part) together? The challenge here is to find out if a performance event originated in guest mode or in host mode.
But we can check for that in the nmi-protected part of the vmexit path.

As far as instrumentation goes, virtualization is simply another 'PID dimension' of measurement.

Today we can isolate system performance measurements/events to the following domains:

- per system
- per cpu
- per task

( Note that PowerPC already supports certain sorts of 'hypervisor/kernel/user'
domain separation, and we have some ABI details for all that but it's by no means complete. Anton is using the PowerPC bits AFAIK, so it already works to a certain degree. )

When extending measurements to KVM, we want two things:

- user friendliness: instead of having to check 'ps' and figure out which Qemu thread is the KVM thread we want to profile, just give a convenience namespace to access guest profiling info. -G ought to map to the first currently running KVM guest it can find. (which would match like 90% of the cases) - etc. No ifs and when. If 'perf kvm top' doesnt show something useful by default the whole effort is for naught.

- Extend core facilities and enable the following measurement dimensions:

host-kernel-space
host-user-space
guest-kernel-space
guest-user-space

on a per guest basis. We want to be able to measure just what the guest does, and we want to be able to measure just what the host does.

Some of this the hardware helps us with (say only measuring host kernel events is possible), some has to be done by fiddling with event enable/disable at vm-exit / vm-entry time.

My suggestion, as always, would be to start very simple and very minimal:

Enable 'perf kvm top' to show guest overhead. Use the exact same kernel image both as a host and as guest (for testing), to not have to deal with the symbol space transport problem initially. Enable 'perf kvm record' to only record guest events by default. Etc.

This alone will be a quite useful result already - and gives a basis for further work. No need to spend months to do the big grand design straight away, all of this can be done gradually and in the order of usefulness - and you'll always have something that actually works (and helps your other KVM projects) along the way.

[ And, as so often, once you walk that path, that grand scheme you are
thinking about right now might easily become last year's really bad idea ;-) ]

So please start walking the path and experience the challenges first-hand.

Quoted 7 lines · Ingo Molnar

My suggestion, as always, would be to start very simple and very minimal:

Enable 'perf kvm top' to show guest overhead. Use the exact same kernel image both as a host and as guest (for testing), to not have to deal with the symbol space transport problem initially. Enable 'perf kvm record' to only record guest events by default. Etc.

Quoted 5 lines

This alone will be a quite useful result already - and gives a basis for further work. No need to spend months to do the big grand design straight away, all of this can be done gradually and in the order of usefulness - and you'll always have something that actually works (and helps your other KVM projects) along the way.

It took me for a couple of hours to read the emails on the topic. Based on above idea, I worked out a prototype which is ugly, but does work with top/record when both guest side and host side use the same kernel image, while compiling most needed modules into kernel directly..

The commands are:
perf kvm top
perf kvm record
perf kvm report

They just collect guest kernel hot functions.

With my patch, I collected dbench data on Nehalem machine (2*4*2 logical cpu).

1) Vanilla host kernel (6G memory):
------------------------------------------------------------------------------------------------------------------------
PerfTop: 15491 irqs/sec kernel:93.6% [1000Hz cycles], (all, 16 CPUs)
------------------------------------------------------------------------------------------------------------------------

             samples  pcnt function                        DSO
             _______ _____ _______________________________ ________________________________________

            99376.00 40.5% ext3_test_allocatable           /lib/modules/2.6.33-kvmymz/build/vmlinux
            41239.00 16.8% bitmap_search_next_usable_block /lib/modules/2.6.33-kvmymz/build/vmlinux
             7019.00  2.9% __ticket_spin_lock              /lib/modules/2.6.33-kvmymz/build/vmlinux
             5350.00  2.2% copy_user_generic_string        /lib/modules/2.6.33-kvmymz/build/vmlinux
             5208.00  2.1% do_get_write_access             /lib/modules/2.6.33-kvmymz/build/vmlinux
             4484.00  1.8% journal_dirty_metadata          /lib/modules/2.6.33-kvmymz/build/vmlinux
             4078.00  1.7% ext3_free_blocks_sb             /lib/modules/2.6.33-kvmymz/build/vmlinux
             3856.00  1.6% ext3_new_blocks                 /lib/modules/2.6.33-kvmymz/build/vmlinux
             3485.00  1.4% journal_get_undo_access         /lib/modules/2.6.33-kvmymz/build/vmlinux
             2803.00  1.1% ext3_try_to_allocate            /lib/modules/2.6.33-kvmymz/build/vmlinux
             2241.00  0.9% __find_get_block                /lib/modules/2.6.33-kvmymz/build/vmlinux
             1957.00  0.8% find_revoke_record              /lib/modules/2.6.33-kvmymz/build/vmlinux

2) guest os: start one guest os with 4GB memory.
------------------------------------------------------------------------------------------------------------------------
PerfTop: 827 irqs/sec kernel: 0.0% [1000Hz cycles], (all, 16 CPUs)
------------------------------------------------------------------------------------------------------------------------

             samples  pcnt function                        DSO
             _______ _____ _______________________________ ________________________________________

            41701.00 28.1% __ticket_spin_lock              /lib/modules/2.6.33-kvmymz/build/vmlinux
            33843.00 22.8% ext3_test_allocatable           /lib/modules/2.6.33-kvmymz/build/vmlinux
            16862.00 11.4% bitmap_search_next_usable_block /lib/modules/2.6.33-kvmymz/build/vmlinux
             3278.00  2.2% native_flush_tlb_others         /lib/modules/2.6.33-kvmymz/build/vmlinux
             3200.00  2.2% copy_user_generic_string        /lib/modules/2.6.33-kvmymz/build/vmlinux
             3009.00  2.0% do_get_write_access             /lib/modules/2.6.33-kvmymz/build/vmlinux
             2834.00  1.9% journal_dirty_metadata          /lib/modules/2.6.33-kvmymz/build/vmlinux
             1965.00  1.3% journal_get_undo_access         /lib/modules/2.6.33-kvmymz/build/vmlinux
             1907.00  1.3% ext3_new_blocks                 /lib/modules/2.6.33-kvmymz/build/vmlinux
             1790.00  1.2% ext3_free_blocks_sb             /lib/modules/2.6.33-kvmymz/build/vmlinux
             1741.00  1.2% find_revoke_record              /lib/modules/2.6.33-kvmymz/build/vmlinux

With vanilla host kernel, perf top data is stable and spinlock doesn't take too much cpu time. With guest os, __ticket_spin_lock consumes 28% cpu time, and sometimes it fluctuates between 9%~28%.

Another interesting finding is aim7. If I start aim7 on tmpfs testing in guest os with 1GB memory, the login hangs and cpu is busy. With the new patch, I could check what happens in guest os, where spinlock is busy and kernel is shrinking memory mostly from slab.

+++ linux-2.6.33_perfkvm/arch/x86/kvm/vmx.c 2010-03-02 10:21:57.588586179 +0800

arch/x86/kvm/vmx.c +0 -0

@@ -3553,8 +3554,19 @@ static void vmx_complete_interrupts(stru
 
 	/* We need to handle NMIs before interrupts are enabled */
 	if ((exit_intr_info & INTR_INFO_INTR_TYPE_MASK) == INTR_TYPE_NMI_INTR &&

With vanilla host kernel, perf top data is stable and spinlock doesn't take too much cpu time. With guest os, __ticket_spin_lock consumes 28% cpu time, and sometimes it fluctuates between 9%~28%.

Another interesting finding is aim7. If I start aim7 on tmpfs testing in guest os with 1GB memory, the login hangs and cpu is busy. With the new patch, I could check what happens in guest os, where spinlock is busy and kernel is shrinking memory mostly from slab.

Quoted 20 lines · Zhang, Yanmin

Quoted 7 lines · Ingo Molnar

My suggestion, as always, would be to start very simple and very minimal:

Enable 'perf kvm top' to show guest overhead. Use the exact same kernel image both as a host and as guest (for testing), to not have to deal with the symbol space transport problem initially. Enable 'perf kvm record' to only record guest events by default. Etc.

Quoted 5 lines

This alone will be a quite useful result already - and gives a basis for further work. No need to spend months to do the big grand design straight away, all of this can be done gradually and in the order of usefulness - and you'll always have something that actually works (and helps your other KVM projects) along the way.

It took me for a couple of hours to read the emails on the topic. Based on above idea, I worked out a prototype which is ugly, but does work with top/record when both guest side and host side use the same kernel image, while compiling most needed modules into kernel directly..

Quoted 6 lines

The commands are:
perf kvm top
perf kvm record
perf kvm report

They just collect guest kernel hot functions.

Fantastic, and there's some really interesting KVM guest/host comparison profiles you've done with this prototype!

Quoted 45 lines

With my patch, I collected dbench data on Nehalem machine (2*4*2 logical cpu).

1) Vanilla host kernel (6G memory):
------------------------------------------------------------------------------------------------------------------------
PerfTop: 15491 irqs/sec kernel:93.6% [1000Hz cycles], (all, 16 CPUs)
------------------------------------------------------------------------------------------------------------------------

             samples  pcnt function                        DSO
             _______ _____ _______________________________ ________________________________________

            99376.00 40.5% ext3_test_allocatable           /lib/modules/2.6.33-kvmymz/build/vmlinux
            41239.00 16.8% bitmap_search_next_usable_block /lib/modules/2.6.33-kvmymz/build/vmlinux
             7019.00  2.9% __ticket_spin_lock              /lib/modules/2.6.33-kvmymz/build/vmlinux
             5350.00  2.2% copy_user_generic_string        /lib/modules/2.6.33-kvmymz/build/vmlinux
             5208.00  2.1% do_get_write_access             /lib/modules/2.6.33-kvmymz/build/vmlinux
             4484.00  1.8% journal_dirty_metadata          /lib/modules/2.6.33-kvmymz/build/vmlinux
             4078.00  1.7% ext3_free_blocks_sb             /lib/modules/2.6.33-kvmymz/build/vmlinux
             3856.00  1.6% ext3_new_blocks                 /lib/modules/2.6.33-kvmymz/build/vmlinux
             3485.00  1.4% journal_get_undo_access         /lib/modules/2.6.33-kvmymz/build/vmlinux
             2803.00  1.1% ext3_try_to_allocate            /lib/modules/2.6.33-kvmymz/build/vmlinux
             2241.00  0.9% __find_get_block                /lib/modules/2.6.33-kvmymz/build/vmlinux
             1957.00  0.8% find_revoke_record              /lib/modules/2.6.33-kvmymz/build/vmlinux

2) guest os: start one guest os with 4GB memory.
------------------------------------------------------------------------------------------------------------------------
PerfTop: 827 irqs/sec kernel: 0.0% [1000Hz cycles], (all, 16 CPUs)
------------------------------------------------------------------------------------------------------------------------

             samples  pcnt function                        DSO
             _______ _____ _______________________________ ________________________________________

            41701.00 28.1% __ticket_spin_lock              /lib/modules/2.6.33-kvmymz/build/vmlinux
            33843.00 22.8% ext3_test_allocatable           /lib/modules/2.6.33-kvmymz/build/vmlinux
            16862.00 11.4% bitmap_search_next_usable_block /lib/modules/2.6.33-kvmymz/build/vmlinux
             3278.00  2.2% native_flush_tlb_others         /lib/modules/2.6.33-kvmymz/build/vmlinux
             3200.00  2.2% copy_user_generic_string        /lib/modules/2.6.33-kvmymz/build/vmlinux
             3009.00  2.0% do_get_write_access             /lib/modules/2.6.33-kvmymz/build/vmlinux
             2834.00  1.9% journal_dirty_metadata          /lib/modules/2.6.33-kvmymz/build/vmlinux
             1965.00  1.3% journal_get_undo_access         /lib/modules/2.6.33-kvmymz/build/vmlinux
             1907.00  1.3% ext3_new_blocks                 /lib/modules/2.6.33-kvmymz/build/vmlinux
             1790.00  1.2% ext3_free_blocks_sb             /lib/modules/2.6.33-kvmymz/build/vmlinux
             1741.00  1.2% find_revoke_record              /lib/modules/2.6.33-kvmymz/build/vmlinux

Quoted 3 lines

With vanilla host kernel, perf top data is stable and spinlock doesn't take too much cpu time. With guest os, __ticket_spin_lock consumes 28% cpu time, and sometimes it fluctuates between 9%~28%.

Looks quite convenient to be able to profile guest and host from the same space, right?

Btw, another, convenient way to compare profiles is 'perf diff':

$ perf diff

# Baseline Delta Shared Object Symbol

# ........ ..........  .................  ......
#
     5.45%     +4.31%  [kernel.kallsyms]  [k] _raw_spin_lock
     3.52%     +3.74%  [kernel.kallsyms]  [k] copy_user_generic_string
     3.11%     +4.08%  [kernel.kallsyms]  [k] sock_alloc_send_pskb
     4.32%     +2.62%  [kernel.kallsyms]  [k] _raw_spin_lock_irqsave
     3.34%     +2.31%  [kernel.kallsyms]  [k] __cache_free
     1.49%     +3.49%  [kernel.kallsyms]  [k] _raw_read_lock
     7.44%     -3.06%  [kernel.kallsyms]  [k] avc_has_perm_noaudit
     0.14%     +2.49%  [kernel.kallsyms]  [k] skb_release_head_state
     0.22%     +2.29%  [kernel.kallsyms]  [k] vfs_read
     1.67%     +0.75%  [kernel.kallsyms]  [k] file_has_perm
     0.09%     +2.31%  [kernel.kallsyms]  [k] rw_verify_area

By default it compares the last two profiles done in the current directory, if you have two separate data files, say perf.data.host and perf.data.guest, you can do:

perf diff perf.data.host perf.data.guest

To get a host -> guest slowdown comparison.

Another suggestion: you could add --guest / --host convenience flag to 'perf kvm', to allow for an easy host/guest comparison workflow:

  perf kvm record --guest     # creates perf.data.guest
  perf kvm record --host      # creates perf.data.host

perf kvm diff # shortcut for: 'perf diff perf.data.host perf.data.guest'

Quoted 4 lines

Another interesting finding is aim7. If I start aim7 on tmpfs testing in guest os with 1GB memory, the login hangs and cpu is busy. With the new patch, I could check what happens in guest os, where spinlock is busy and kernel is shrinking memory mostly from slab.

This is exactly the kind of usage proper perf events integration would allow! Your 'perf kvm' looks very powerful, even in its early prototype.

Now, regarding the technical details of your patch:

Quoted 4 lines

+++ linux-2.6.33_perfkvm/arch/x86/kvm/vmx.c 2010-03-02 10:21:57.588586179 +0800

(quoted diff) +0 -0

@@ -3553,8 +3554,19 @@ static void vmx_complete_interrupts(stru
 

arch/x86/kvm/vmx.c static void vmx_complete_interrupts(stru · line 3565

Quoted 12 lines

arch/x86/kvm/vmx.c +9 -1

 	/* We need to handle NMIs before interrupts are enabled */
 	if ((exit_intr_info & INTR_INFO_INTR_TYPE_MASK) == INTR_TYPE_NMI_INTR &&
-	    (exit_intr_info & INTR_INFO_VALID_MASK))
+	    (exit_intr_info & INTR_INFO_VALID_MASK)) {
+		u64 rip = vmcs_readl(GUEST_RIP);
+		int user_mode = vmcs_read16(GUEST_CS_SELECTOR);
+
+#ifdef CONFIG_X86_32
+		user_mode = (user_mode & SEGMENT_RPL_MASK) == USER_RPL;
+#else
+		user_mode = !!(user_mode & 3);
+#endif

This test could use a helper i guess, to remove the #ifdef?

Quoted 5 lines

arch/x86/kvm/vmx.c +3 -0

+		perf_save_virt_ip(user_mode, rip);
 		asm("int $2");
+		perf_reset_virt_ip();
+	}

include/linux/perf_event.h enum perf_event_sample_format { · line 128

Quoted 7 lines

include/linux/perf_event.h +1 -0

@@ -125,8 +125,9 @@ enum perf_event_sample_format {
 	PERF_SAMPLE_PERIOD			= 1U << 8,
 	PERF_SAMPLE_STREAM_ID			= 1U << 9,
 	PERF_SAMPLE_RAW				= 1U << 10,
+        PERF_SAMPLE_KVM                         = 1U << 11,

yep, we can extend it like this, but maybe there's another method:

include/linux/perf_event.h static inline void perf_event_enable(str · line 913

Quoted 7 lines

include/linux/perf_event.h +6 -0

+//#if defined(CONFIG_PERF_EVENTS && CONFIG_PERF_HAS_VIRT_IP)
+#if defined(CONFIG_PERF_EVENTS)
+struct virt_ip_info {
+	int	user_mode;
+	u64	ip;
+};

basically what we want is a 'this is guest user mode' differentiator in the call frame stream data, right?

We already have such separators (it's just not used very much):

enum perf_callchain_context {
        PERF_CONTEXT_HV                 = (__u64)-32,
        PERF_CONTEXT_KERNEL             = (__u64)-128,
        PERF_CONTEXT_USER               = (__u64)-512,

        PERF_CONTEXT_GUEST              = (__u64)-2048,
        PERF_CONTEXT_GUEST_KERNEL       = (__u64)-2176,
        PERF_CONTEXT_GUEST_USER         = (__u64)-2560,

        PERF_CONTEXT_MAX                = (__u64)-4095,
};

Basically KVM's guest context could be expressed as a PERF_CONTEXT_GUEST separator pushed into the stream (and then recognized by 'perf kvm' - and generally by 'perf report' et al), followed by the virtual RIP as a regular IP.

That way we dont need PERF_SAMPLE_KVM. Note that the tooling doesnt know about such separators very well yet, so there might be some gotchas along the way. Please let us know if you run into any problems here.

Peter, what's your preference for this KVM profiling ABI detail?

include/linux/perf_event.h static inline void perf_event_enable(str · line 926

Quoted 12 lines

include/linux/perf_event.h +12 -0

+DECLARE_PER_CPU(struct virt_ip_info, perf_virt_ip);
+extern void perf_save_virt_ip(int user_mode, u64 ip);
+extern void perf_reset_virt_ip(void);
+extern int perf_get_virt_user_mode(void);
+static inline u64 perf_instruction_pointer(struct perf_event *event, struct pt_regs *regs)
+{
+	u64 ip;
+	if (event->attr.sample_type & PERF_SAMPLE_KVM)
+		ip = percpu_read(perf_virt_ip.ip);
+	else
+		ip = instruction_pointer(regs);
+	return ip;

And this complication we can perhaps avoid by extending the stack-trace engine (arch/x86/kernel/stacktrace.c) to 'know' about virtual guest RIPs automatically?

( Note that this would be a bit of an advantage for regular oops printing as
well: if a KVM thread crashes or generates a stack-dump it could automatically print the guest virtual RIP as well. )

Walking into the guest context is more complex, but not impossible either - and that bit definitely has to be done not in an NMI context but in the KVM thread context.

So maybe we should not extend dump_stack_trace() after all (it cannot really work from NMI or from oops context), but add the KVM variant and let it directly inject into the perf data stream? (like your patch does it pretty much)

include/linux/perf_event.h

Quoted 8 lines

+++ linux-2.6.33_perfkvm/tools/perf/perf.c 2010-03-02 09:57:03.164001069 +0800

include/linux/perf_event.h +6 -0

+	if (argc > 1 && !strcmp(argv[0], "kvm")) {
+		sample_kvm = 1;
+		argv++;
+		argc--;
+		cmd = argv[0];
+	}

this is fine as a quick hack. For the real thing i suspect we want to add 'perf kvm' as a real builtin-kvm.c command - see builtin-sched.c and builtin-lock.c about how to create such 'subsystem commands'. builtin-record.c can be extended with various host/guest recording detail switches (off by default), and builtin-kvm.c can use those. For example builtin-sched.c implements 'perf sched record' the following way:

static const char *record_args[] = {
        "record",
        "-a",
        "-R",
        "-M",
        "-f",
        "-m", "1024",
        "-c", "1",
        "-e", "sched:sched_switch:r",
        "-e", "sched:sched_stat_wait:r",
        "-e", "sched:sched_stat_sleep:r",
        "-e", "sched:sched_stat_iowait:r",
        "-e", "sched:sched_stat_runtime:r",
        "-e", "sched:sched_process_exit:r",
        "-e", "sched:sched_process_fork:r",
        "-e", "sched:sched_wakeup:r",
        "-e", "sched:sched_migrate_task:r",
};

So it simply passes these arguments to perf record. 'perf kvm record' could do something similar.

Anyway, your patch already shows great progress and it's the kind of direction for enhanced performance analysis of KVM that i think would be very fruitful to KVM developers to pursue.

Quoted 103 lines · Ingo Molnar

Quoted 20 lines · Zhang, Yanmin

Quoted 7 lines · Ingo Molnar

My suggestion, as always, would be to start very simple and very minimal:

Enable 'perf kvm top' to show guest overhead. Use the exact same kernel image both as a host and as guest (for testing), to not have to deal with the symbol space transport problem initially. Enable 'perf kvm record' to only record guest events by default. Etc.

Quoted 5 lines

This alone will be a quite useful result already - and gives a basis for further work. No need to spend months to do the big grand design straight away, all of this can be done gradually and in the order of usefulness - and you'll always have something that actually works (and helps your other KVM projects) along the way.

It took me for a couple of hours to read the emails on the topic. Based on above idea, I worked out a prototype which is ugly, but does work with top/record when both guest side and host side use the same kernel image, while compiling most needed modules into kernel directly..

Quoted 6 lines

The commands are:
perf kvm top
perf kvm record
perf kvm report

They just collect guest kernel hot functions.

Fantastic, and there's some really interesting KVM guest/host comparison profiles you've done with this prototype!

Quoted 45 lines

With my patch, I collected dbench data on Nehalem machine (2*4*2 logical cpu).

1) Vanilla host kernel (6G memory):
------------------------------------------------------------------------------------------------------------------------
PerfTop: 15491 irqs/sec kernel:93.6% [1000Hz cycles], (all, 16 CPUs)
------------------------------------------------------------------------------------------------------------------------

             samples  pcnt function                        DSO
             _______ _____ _______________________________ ________________________________________

            99376.00 40.5% ext3_test_allocatable           /lib/modules/2.6.33-kvmymz/build/vmlinux
            41239.00 16.8% bitmap_search_next_usable_block /lib/modules/2.6.33-kvmymz/build/vmlinux
             7019.00  2.9% __ticket_spin_lock              /lib/modules/2.6.33-kvmymz/build/vmlinux
             5350.00  2.2% copy_user_generic_string        /lib/modules/2.6.33-kvmymz/build/vmlinux
             5208.00  2.1% do_get_write_access             /lib/modules/2.6.33-kvmymz/build/vmlinux
             4484.00  1.8% journal_dirty_metadata          /lib/modules/2.6.33-kvmymz/build/vmlinux
             4078.00  1.7% ext3_free_blocks_sb             /lib/modules/2.6.33-kvmymz/build/vmlinux
             3856.00  1.6% ext3_new_blocks                 /lib/modules/2.6.33-kvmymz/build/vmlinux
             3485.00  1.4% journal_get_undo_access         /lib/modules/2.6.33-kvmymz/build/vmlinux
             2803.00  1.1% ext3_try_to_allocate            /lib/modules/2.6.33-kvmymz/build/vmlinux
             2241.00  0.9% __find_get_block                /lib/modules/2.6.33-kvmymz/build/vmlinux
             1957.00  0.8% find_revoke_record              /lib/modules/2.6.33-kvmymz/build/vmlinux

2) guest os: start one guest os with 4GB memory.
------------------------------------------------------------------------------------------------------------------------
PerfTop: 827 irqs/sec kernel: 0.0% [1000Hz cycles], (all, 16 CPUs)
------------------------------------------------------------------------------------------------------------------------

             samples  pcnt function                        DSO
             _______ _____ _______________________________ ________________________________________

            41701.00 28.1% __ticket_spin_lock              /lib/modules/2.6.33-kvmymz/build/vmlinux
            33843.00 22.8% ext3_test_allocatable           /lib/modules/2.6.33-kvmymz/build/vmlinux
            16862.00 11.4% bitmap_search_next_usable_block /lib/modules/2.6.33-kvmymz/build/vmlinux
             3278.00  2.2% native_flush_tlb_others         /lib/modules/2.6.33-kvmymz/build/vmlinux
             3200.00  2.2% copy_user_generic_string        /lib/modules/2.6.33-kvmymz/build/vmlinux
             3009.00  2.0% do_get_write_access             /lib/modules/2.6.33-kvmymz/build/vmlinux
             2834.00  1.9% journal_dirty_metadata          /lib/modules/2.6.33-kvmymz/build/vmlinux
             1965.00  1.3% journal_get_undo_access         /lib/modules/2.6.33-kvmymz/build/vmlinux
             1907.00  1.3% ext3_new_blocks                 /lib/modules/2.6.33-kvmymz/build/vmlinux
             1790.00  1.2% ext3_free_blocks_sb             /lib/modules/2.6.33-kvmymz/build/vmlinux
             1741.00  1.2% find_revoke_record              /lib/modules/2.6.33-kvmymz/build/vmlinux

Quoted 3 lines

With vanilla host kernel, perf top data is stable and spinlock doesn't take too much cpu time. With guest os, __ticket_spin_lock consumes 28% cpu time, and sometimes it fluctuates between 9%~28%.

Looks quite convenient to be able to profile guest and host from the same space, right?

Btw, another, convenient way to compare profiles is 'perf diff':

$ perf diff

# Baseline Delta Shared Object Symbol

# ........ ..........  .................  ......
#
     5.45%     +4.31%  [kernel.kallsyms]  [k] _raw_spin_lock
     3.52%     +3.74%  [kernel.kallsyms]  [k] copy_user_generic_string
     3.11%     +4.08%  [kernel.kallsyms]  [k] sock_alloc_send_pskb
     4.32%     +2.62%  [kernel.kallsyms]  [k] _raw_spin_lock_irqsave
     3.34%     +2.31%  [kernel.kallsyms]  [k] __cache_free
     1.49%     +3.49%  [kernel.kallsyms]  [k] _raw_read_lock
     7.44%     -3.06%  [kernel.kallsyms]  [k] avc_has_perm_noaudit
     0.14%     +2.49%  [kernel.kallsyms]  [k] skb_release_head_state
     0.22%     +2.29%  [kernel.kallsyms]  [k] vfs_read
     1.67%     +0.75%  [kernel.kallsyms]  [k] file_has_perm
     0.09%     +2.31%  [kernel.kallsyms]  [k] rw_verify_area

Quoted 7 lines

By default it compares the last two profiles done in the current directory, if you have two separate data files, say perf.data.host and perf.data.guest, you can do:

perf diff perf.data.host perf.data.guest

To get a host -> guest slowdown comparison.

Thanks for your good pointer. That would be more user friendly.

Quoted 7 lines

Another suggestion: you could add --guest / --host convenience flag to 'perf kvm', to allow for an easy host/guest comparison workflow:

  perf kvm record --guest     # creates perf.data.guest
  perf kvm record --host      # creates perf.data.host

So here we would have a new meaning that --guest means just collects guest os data and --host just collects host data.

Quoted 16 lines

perf kvm diff # shortcut for: 'perf diff perf.data.host perf.data.guest'

Quoted 4 lines

Another interesting finding is aim7. If I start aim7 on tmpfs testing in guest os with 1GB memory, the login hangs and cpu is busy. With the new patch, I could check what happens in guest os, where spinlock is busy and kernel is shrinking memory mostly from slab.

This is exactly the kind of usage proper perf events integration would allow! Your 'perf kvm' looks very powerful, even in its early prototype.

Now, regarding the technical details of your patch:

Quoted 4 lines

+++ linux-2.6.33_perfkvm/arch/x86/kvm/vmx.c 2010-03-02 10:21:57.588586179 +0800

(quoted diff) +0 -0

@@ -3553,8 +3554,19 @@ static void vmx_complete_interrupts(stru
 

Quoted 14 lines

Review static void vmx_complete_interrupts(stru · line 3565

Quoted 12 lines

(quoted diff) +9 -1

 	/* We need to handle NMIs before interrupts are enabled */
 	if ((exit_intr_info & INTR_INFO_INTR_TYPE_MASK) == INTR_TYPE_NMI_INTR &&
-	    (exit_intr_info & INTR_INFO_VALID_MASK))
+	    (exit_intr_info & INTR_INFO_VALID_MASK)) {
+		u64 rip = vmcs_readl(GUEST_RIP);
+		int user_mode = vmcs_read16(GUEST_CS_SELECTOR);
+
+#ifdef CONFIG_X86_32
+		user_mode = (user_mode & SEGMENT_RPL_MASK) == USER_RPL;
+#else
+		user_mode = !!(user_mode & 3);
+#endif

This test could use a helper i guess, to remove the #ifdef?

Right.

Quoted 6 lines

Quoted 5 lines

(quoted diff) +3 -0

+		perf_save_virt_ip(user_mode, rip);
 		asm("int $2");
+		perf_reset_virt_ip();
+	}

Quoted 9 lines

include/linux/perf_event.h enum perf_event_sample_format { · line 128

Quoted 7 lines

include/linux/perf_event.h +1 -0

@@ -125,8 +125,9 @@ enum perf_event_sample_format {
 	PERF_SAMPLE_PERIOD			= 1U << 8,
 	PERF_SAMPLE_STREAM_ID			= 1U << 9,
 	PERF_SAMPLE_RAW				= 1U << 10,
+        PERF_SAMPLE_KVM                         = 1U << 11,

yep, we can extend it like this, but maybe there's another method:

Here PERF_SAMPLE_KVM is just used by tool perf to notify kernel that we want to collect KVM guest os data instead of whole system. It isn't used when parsing data. Interface sys_perf_event_open has no flag to notify that we want to collect guest os event. Another solution is
add new member in perf_event_attr, such like
typedef enum {
PERF_SAMPLE_HOST,
PERF_SAMPLE_GUEST
} perf_os;

per_event_attr {
	...
	perf_os os;
	...
};

Quoted 10 lines

include/linux/perf_event.h enum perf_event_sample_format {

Quoted 7 lines

include/linux/perf_event.h +6 -0

+//#if defined(CONFIG_PERF_EVENTS && CONFIG_PERF_HAS_VIRT_IP)
+#if defined(CONFIG_PERF_EVENTS)
+struct virt_ip_info {
+	int	user_mode;
+	u64	ip;
+};

basically what we want is a 'this is guest user mode' differentiator in the call frame stream data, right?

Here user mode is to match PERF_RECORD_MISC_USER/PERF_RECORD_MISC_KERNEL. If checking current codes of tool perf and kernel, we could
find they uses both perf_callchain_context and PERF_RECORD_MISC_USER/PERF_RECORD_MISC_KERNEL. The 1st is used for callchain and the 2nd is used for non-callchain. As for the callchain, we could use perf_callchain_context=>PERF_CONTEXT_GUEST_KERNEL.

As for non callchain, the doable solution is just like what Peter suggested that we need add a new PERF_RECORD_MISC_CPUMODE_MASK, such like PERF_RECORD_MISC_GUEST_KERNEL and PERF_RECORD_MISC_GUEST_USER. The new flag will be used in perf's function thread__find_addr_location to find the right map for guest os.

Quoted 15 lines

We already have such separators (it's just not used very much):

enum perf_callchain_context {
        PERF_CONTEXT_HV                 = (__u64)-32,
        PERF_CONTEXT_KERNEL             = (__u64)-128,
        PERF_CONTEXT_USER               = (__u64)-512,

        PERF_CONTEXT_GUEST              = (__u64)-2048,
        PERF_CONTEXT_GUEST_KERNEL       = (__u64)-2176,
        PERF_CONTEXT_GUEST_USER         = (__u64)-2560,

        PERF_CONTEXT_MAX                = (__u64)-4095,
};

Quoted 4 lines

Basically KVM's guest context could be expressed as a PERF_CONTEXT_GUEST separator pushed into the stream (and then recognized by 'perf kvm' - and generally by 'perf report' et al), followed by the virtual RIP as a regular IP.

Right when we use it for callchain.

Quoted 72 lines

That way we dont need PERF_SAMPLE_KVM. Note that the tooling doesnt know about such separators very well yet, so there might be some gotchas along the way. Please let us know if you run into any problems here.

Peter, what's your preference for this KVM profiling ABI detail?

include/linux/perf_event.h

Quoted 12 lines

include/linux/perf_event.h +12 -0

+DECLARE_PER_CPU(struct virt_ip_info, perf_virt_ip);
+extern void perf_save_virt_ip(int user_mode, u64 ip);
+extern void perf_reset_virt_ip(void);
+extern int perf_get_virt_user_mode(void);
+static inline u64 perf_instruction_pointer(struct perf_event *event, struct pt_regs *regs)
+{
+	u64 ip;
+	if (event->attr.sample_type & PERF_SAMPLE_KVM)
+		ip = percpu_read(perf_virt_ip.ip);
+	else
+		ip = instruction_pointer(regs);
+	return ip;

And this complication we can perhaps avoid by extending the stack-trace engine (arch/x86/kernel/stacktrace.c) to 'know' about virtual guest RIPs automatically?

( Note that this would be a bit of an advantage for regular oops printing as
well: if a KVM thread crashes or generates a stack-dump it could automatically print the guest virtual RIP as well. )

Walking into the guest context is more complex, but not impossible either - and that bit definitely has to be done not in an NMI context but in the KVM thread context.

So maybe we should not extend dump_stack_trace() after all (it cannot really work from NMI or from oops context), but add the KVM variant and let it directly inject into the perf data stream? (like your patch does it pretty much)

include/linux/perf_event.h

Quoted 8 lines

+++ linux-2.6.33_perfkvm/tools/perf/perf.c 2010-03-02 09:57:03.164001069 +0800

include/linux/perf_event.h +6 -0

+	if (argc > 1 && !strcmp(argv[0], "kvm")) {
+		sample_kvm = 1;
+		argv++;
+		argc--;
+		cmd = argv[0];
+	}

this is fine as a quick hack. For the real thing i suspect we want to add 'perf kvm' as a real builtin-kvm.c command - see builtin-sched.c and builtin-lock.c about how to create such 'subsystem commands'. builtin-record.c can be extended with various host/guest recording detail switches (off by default), and builtin-kvm.c can use those. For example builtin-sched.c implements 'perf sched record' the following way:

static const char *record_args[] = {
        "record",
        "-a",
        "-R",
        "-M",
        "-f",
        "-m", "1024",
        "-c", "1",
        "-e", "sched:sched_switch:r",
        "-e", "sched:sched_stat_wait:r",
        "-e", "sched:sched_stat_sleep:r",
        "-e", "sched:sched_stat_iowait:r",
        "-e", "sched:sched_stat_runtime:r",
        "-e", "sched:sched_process_exit:r",
        "-e", "sched:sched_process_fork:r",
        "-e", "sched:sched_wakeup:r",
        "-e", "sched:sched_migrate_task:r",
};

Quoted 2 lines

So it simply passes these arguments to perf record. 'perf kvm record' could do something similar.

That's a good pointer. I will try it late.

Quoted 4 lines

Anyway, your patch already shows great progress and it's the kind of direction for enhanced performance analysis of KVM that i think would be very fruitful to KVM developers to pursue.

include/linux/perf_event.h +11 -0

+static inline u64 perf_instruction_pointer(struct pt_regs *regs)
+{
+       u64 ip;
+       ip = percpu_read(perf_virt_ip.ip);
+       if (!ip)
+               ip = instruction_pointer(regs);
+       else
+               perf_reset_virt_ip();
+       return ip;
+}
+

Quoted 17 lines · Yanmin

include/linux/perf_event.h +17 -0

+static inline u64 perf_instruction_pointer(struct pt_regs *regs)
+{
+       u64 ip;
+       ip = percpu_read(perf_virt_ip.ip);
+       if (!ip)
+               ip = instruction_pointer(regs);
+       else
+               perf_reset_virt_ip();
+       return ip;
+}
+
+static inline unsigned int perf_misc_flags(struct pt_regs *regs)
+{
+       if (percpu_read(perf_virt_ip.ip)) {
+               return percpu_read(perf_virt_ip.user_mode) ?
+                       PERF_RECORD_MISC_GUEST_USER :
+                       PERF_RECORD_MISC_GUEST_KERNEL;

include/linux/perf_event.h static inline void perf_event_mmap(struc · line 875

Quoted 4 lines

include/linux/perf_event.h +4 -0

+       } else
+               return user_mode(regs) ? PERF_RECORD_MISC_USER :
+                                PERF_RECORD_MISC_KERNEL;
+} 

This codes in the assumption that perf_misc_flags() must only be called before perf_instruction_pointer(), which is currently true, but you might want to put a comment near to remind us of this.

include/linux/perf_event.h extern void perf_tp_event(int event_id, · line 862

Quoted 5 lines · Yanmin

include/linux/perf_event.h +0 -5

-#ifndef perf_misc_flags
-#define perf_misc_flags(regs)  (user_mode(regs) ? PERF_RECORD_MISC_USER : \
-                                PERF_RECORD_MISC_KERNEL)
-#define perf_instruction_pointer(regs) instruction_pointer(regs)
-#endif 

Ah, that #ifndef is for powerpc, which I think you just broke.

Quoted 7 lines

2. For every emulated performance counter the guest activates kvm allocates a perf_event and configures it for the guest (we may allow kvm to specify the counter index, the guest would be able to use rdpmc unintercepted then). Event filtering is also done in this step.

rdpmc can never be used unintercepted, for perf might be multiplexing the actual hw.

Also, intel debugstore things requires a host linear address,

again, not
something a vcpu can easily provide (although that might be worked around with an msr trap, but that still limits you to 1 page data sizes, not a limitation all software will respect).

Quoted 5 lines

All that said, what we really want is for Intel+AMD to come up with proper hw PMU virtualization support that makes it easy to rotate the full PMU in and out for a guest. Then this whole discussion will become a non issue.

As it stands there simply are a number of PMU features that defy being virtualized, simply because the virt stuff doesn't do system topology. So even if they were to support a virtualized pmu, it would likely be a different beast than the native hardware is, and it will be several hardware models in the future, coming up with a paravirt interface and getting !linux hosts to adapt and !linux guests to use is probably as 'easy'.

Quoted 8 lines · Peter Zijlstra

Review

Quoted 5 lines · Yanmin

(quoted diff) +0 -5

-#ifndef perf_misc_flags
-#define perf_misc_flags(regs)  (user_mode(regs) ? PERF_RECORD_MISC_USER : \
-                                PERF_RECORD_MISC_KERNEL)
-#define perf_instruction_pointer(regs) instruction_pointer(regs)
-#endif 

Ah, that #ifndef is for powerpc, which I think you just broke.

Thanks for the reminder. I deleted powerpc codes when building cscope lib.

It seems perf_save_virt_ip/perf_reset_virt_ip interfaces are ugly. I plan to change them to a callback function struct and kvm registers its version to perf.

Such like:

struct perf_guest_info_callbacks {
	int (*is_in_guest)();
	u64 (*get_guest_ip)();
	int (*copy_guest_stack)();
	int (*reset_in_guest)();
	...
};

int perf_register_guest_info_callbacks(struct perf_guest_info_callbacks *); int perf_unregister_guest_info_callbacks(struct perf_guest_info_callbacks *);

It's more scalable and neater.