x86/kvm: Show guest system/user cputime in cpustat

Currently we can only get the cpu_stat of whole guest as one. This patch enhanced cpu_stat with more detail, has guest_system and guest_user cpu time statistics with a little overhead.

Signed-off-by Sheng Yang<[email protected]>

---

This draft patch based on KVM upstream to show the idea. I would split it into more kernel friendly version later.

The overhead is, the cost of get_cpl() after each exit from guest.

Quoted 10 lines · Sheng Yang

Currently we can only get the cpu_stat of whole guest as one. This patch enhanced cpu_stat with more detail, has guest_system and guest_user cpu time statistics with a little overhead.

Signed-off-by Sheng Yang<[email protected]>

---

This draft patch based on KVM upstream to show the idea. I would split it into more kernel friendly version later.

Quoted 1 line

The overhead is, the cost of get_cpl() after each exit from guest.

This can be very expensive in the nested virtualization case, so I wouldn't like this to be in normal paths. I think detailed profiling like that can be left to 'perf kvm', which only has overhead if enabled at runtime.

For example you can put the code to note the cpl in a tracepoint which is enabled dynamically.

Quoted 14 lines · On Thursday 11 March 2010 15:36:01 Avi Kivity wrote:

Quoted 12 lines · Sheng Yang

Currently we can only get the cpu_stat of whole guest as one. This patch enhanced cpu_stat with more detail, has guest_system and guest_user cpu time statistics with a little overhead.

Signed-off-by Sheng Yang<[email protected]>

---

This draft patch based on KVM upstream to show the idea. I would split it into more kernel friendly version later.

The overhead is, the cost of get_cpl() after each exit from guest.

Quoted 5 lines

This can be very expensive in the nested virtualization case, so I wouldn't like this to be in normal paths. I think detailed profiling like that can be left to 'perf kvm', which only has overhead if enabled at runtime.

Yes, that's my concern too(though nested vmcs/vmcb read already too expensive, they should be optimized...).

The other concern is, perf alike mechanism would
bring a lot more overhead compared to this.

Quoted 3 lines

For example you can put the code to note the cpl in a tracepoint which is enabled dynamically.

Yanmin have already implement "perf kvm" to support this. We are just arguing if a normal top-alike mechanism is necessary.

I am also considering to make it a feature that can be disabled. But seems it make things complicate and result in uncertain cpustat output.

Quoted 4 lines · Sheng Yang

Quoted 2 lines · On Thursday 11 March 2010 15:36:01 Avi Kivity wrote:

On 03/11/2010 09:20 AM, Sheng Yang wrote:

Quoted 5 lines

Quoted 5 lines

Quoted 5 lines

Currently we can only get the cpu_stat of whole guest as one. This patch enhanced cpu_stat with more detail, has guest_system and guest_user cpu time statistics with a little overhead.

Signed-off-by Sheng Yang<[email protected]>

Quoted 12 lines

Quoted 12 lines

Quoted 7 lines

---

This draft patch based on KVM upstream to show the idea. I would split it into more kernel friendly version later.

The overhead is, the cost of get_cpl() after each exit from guest.

This can be very expensive in the nested virtualization case, so I wouldn't like this to be in normal paths. I think detailed profiling like that can be left to 'perf kvm', which only has overhead if enabled at runtime.

Quoted 2 lines

Yes, that's my concern too(though nested vmcs/vmcb read already too expensive, they should be optimized...).

Any ideas on how to do that? Perhaps use paravirt_ops to covert the vmread into a memory read? We store the vmwrites in the vmcs anyway.

Quoted 4 lines

The other concern is, perf alike mechanism would
bring a lot more overhead compared to this.

Ordinarily users won't care if time is spent in guest kernel mode or guest user mode. They want to see which guest is imposing a load on a system. I consider a user profiling a guest from the host an advanced and rarer use case, so it's okay to require tools and additional overhead for this.

Quoted 6 lines

Quoted 4 lines

For example you can put the code to note the cpl in a tracepoint which is enabled dynamically.

Yanmin have already implement "perf kvm" to support this. We are just arguing if a normal top-alike mechanism is necessary.

Quoted 4 lines

I am also considering to make it a feature that can be disabled. But seems it make things complicate and result in uncertain cpustat output.

I'm not even sure that guest time was a good idea.

Quoted 19 lines · Sheng Yang

Quoted 17 lines · On Thursday 11 March 2010 15:36:01 Avi Kivity wrote:

Quoted 11 lines · Sheng Yang

Currently we can only get the cpu_stat of whole guest as one. This patch enhanced cpu_stat with more detail, has guest_system and guest_user cpu time statistics with a little overhead.

Signed-off-by Sheng Yang<[email protected]>

---

This draft patch based on KVM upstream to show the idea. I would split it into more kernel friendly version later.

The overhead is, the cost of get_cpl() after each exit from guest.

This can be very expensive in the nested virtualization case, so I wouldn't like this to be in normal paths. I think detailed profiling like that can be left to 'perf kvm', which only has overhead if enabled at runtime.

Quoted 2 lines

Yes, that's my concern too(though nested vmcs/vmcb read already too expensive, they should be optimized...).

Any ideas on how to do that? Perhaps use paravirt_ops to covert the vmread into a memory read? We store the vmwrites in the vmcs anyway.

Quoted 3 lines

The other concern is, perf alike mechanism would
bring a lot more overhead compared to this.

Ordinarily users won't care if time is spent in guest kernel mode or guest user mode. They want to see which guest is imposing a load on a system. I consider a user profiling a guest from the host an advanced and rarer use case, so it's okay to require tools and additional overhead for this.

Quoted 8 lines

Quoted 2 lines

For example you can put the code to note the cpl in a tracepoint which is enabled dynamically.

Yanmin have already implement "perf kvm" to support this. We are just arguing if a normal top-alike mechanism is necessary.

I am also considering to make it a feature that can be disabled. But seems it make things complicate and result in uncertain cpustat output.

I'm not even sure that guest time was a good idea.

Quoted 23 lines · On Thursday 11 March 2010 15:50:54 Avi Kivity wrote:

Quoted 21 lines · Sheng Yang

Quoted 11 lines · On Thursday 11 March 2010 15:36:01 Avi Kivity wrote:

Quoted 10 lines · Sheng Yang

Currently we can only get the cpu_stat of whole guest as one. This patch enhanced cpu_stat with more detail, has guest_system and guest_user cpu time statistics with a little overhead.

Signed-off-by Sheng Yang<[email protected]>

---

This draft patch based on KVM upstream to show the idea. I would split it into more kernel friendly version later.

Quoted 6 lines

Quoted 1 line

The overhead is, the cost of get_cpl() after each exit from guest.

This can be very expensive in the nested virtualization case, so I wouldn't like this to be in normal paths. I think detailed profiling like that can be left to 'perf kvm', which only has overhead if enabled at runtime.

Yes, that's my concern too(though nested vmcs/vmcb read already too expensive, they should be optimized...).

Quoted 2 lines

Any ideas on how to do that? Perhaps use paravirt_ops to covert the vmread into a memory read? We store the vmwrites in the vmcs anyway.

When Qing(CCed) was working on nested VMX in the past, he found PV vmread/vmwrite indeed works well(it would write to the virtual vmcs so vmwrite can also benefit). Though compared to old machine(one our internal patch shows improve more than 5%), NHM get less benefit due to the reduced vmexit cost.

Quoted 7 lines

When Qing(CCed) was working on nested VMX in the past, he found PV vmread/vmwrite indeed works well(it would write to the virtual vmcs so vmwrite can also benefit). Though compared to old machine(one our internal patch shows improve more than 5%), NHM get less benefit due to the reduced vmexit cost.

One of the hurdles to PVize vmread/vmwrite is the fact that the memory layout of physical vmcs remains unknown. Of course it can use the custom vmcs layout utilized by nested virtualization, but that looks a little weird, since different nested virtualization implementation may create different custom layout.

I once used another approach to partially accelerate the vmread/vmwrite in nested virtualization case, which also gives good performance gain (around 7% on pre-nehalem, based on this, PV vmread/vmwrite had another 7%). That is to make a shortcut to handle EXIT_REASON_VM{READ,WRITE}, without even turning on the IF.