Skip to contentSkip to conversation

KVM in-kernel PM Timer implementation

RFC In review Ulrich Obergfell 6 participants 15 messages Dec 14, 2010 – Dec 15, 2010 1 message not in the archive KVM

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 344060531.680691292328457867.JavaMail.root@zmail07.collab.prod.int.phx2.redhat.com

#1 lore raw

cover KVM in-kernel PM Timer implementation

Hi,

This is an RFC through which I would like to get feedback on how the idea of in-kernel PM Timer would be received.

The current implementation of PM Timer emulation is 'heavy-weight' because the code resides in qemu userspace. Guest operating systems that use PM Timer as a clock source (for example, older versions of Linux that do not have paravirtualized clock) would benefit from an in-kernel PM Timer emulation.

Parts 1 thru 4 of this RFC contain experimental source code which I recently used to investigate the performance benefit. In a Linux guest, I was running a program that calls gettimeofday() 'n' times in a loop (the PM Timer register is read during each call). With in-kernel PM Timer, I observed a significant reduction of program execution time.

The experimental code emulates the PM Timer register in KVM kernel. All other components of ACPI PM remain in qemu userspace. Also, the 'timer carry interrupt' feature is not implemented in-kernel. If a guest operating system needs to enable the 'timer carry interrupt', the code takes care that PM Timer emulation falls back to userspace. However, I think the design of the code has sufficient flexibility, so that anyone who would want to add the 'timer carry interrupt' feature in-kernel could try to do so later on.

Please review and please comment.

Regards,

Uli Obergfell

#2 lore raw
Quoted 18 lines · Ulrich Obergfell

Hi,

This is an RFC through which I would like to get feedback on how the idea of in-kernel PM Timer would be received.

The current implementation of PM Timer emulation is 'heavy-weight' because the code resides in qemu userspace. Guest operating systems that use PM Timer as a clock source (for example, older versions of Linux that do not have paravirtualized clock) would benefit from an in-kernel PM Timer emulation.

Parts 1 thru 4 of this RFC contain experimental source code which I recently used to investigate the performance benefit. In a Linux guest, I was running a program that calls gettimeofday() 'n' times in a loop (the PM Timer register is read during each call). With in-kernel PM Timer, I observed a significant reduction of program execution time.

Quoted 9 lines

The experimental code emulates the PM Timer register in KVM kernel. All other components of ACPI PM remain in qemu userspace. Also, the 'timer carry interrupt' feature is not implemented in-kernel. If a guest operating system needs to enable the 'timer carry interrupt', the code takes care that PM Timer emulation falls back to userspace. However, I think the design of the code has sufficient flexibility, so that anyone who would want to add the 'timer carry interrupt' feature in-kernel could try to do so later on.

What is the motivation for this? Are there any important guests that use the pmtimer?

If anything I'd expect hpet or the Microsoft synthetic timers to be a lot more important.

Signature · 1 line
error compiling committee.c: too many arguments to function
#3 lore raw
Quoted 29 lines · Avi Kivity
Quoted 18 lines · Ulrich Obergfell

Hi,

This is an RFC through which I would like to get feedback on how the idea of in-kernel PM Timer would be received.

The current implementation of PM Timer emulation is 'heavy-weight' because the code resides in qemu userspace. Guest operating systems that use PM Timer as a clock source (for example, older versions of Linux that do not have paravirtualized clock) would benefit from an in-kernel PM Timer emulation.

Parts 1 thru 4 of this RFC contain experimental source code which I recently used to investigate the performance benefit. In a Linux guest, I was running a program that calls gettimeofday() 'n' times in a loop (the PM Timer register is read during each call). With in-kernel PM Timer, I observed a significant reduction of program execution time.

Quoted 9 lines

The experimental code emulates the PM Timer register in KVM kernel. All other components of ACPI PM remain in qemu userspace. Also, the 'timer carry interrupt' feature is not implemented in-kernel. If a guest operating system needs to enable the 'timer carry interrupt', the code takes care that PM Timer emulation falls back to userspace. However, I think the design of the code has sufficient flexibility, so that anyone who would want to add the 'timer carry interrupt' feature in-kernel could try to do so later on.

Quoted 2 lines

What is the motivation for this? Are there any important guests that use the pmtimer?

Avi,

All older RHEL and Windows, for example, would benefit for this.

Quoted 2 lines

If anything I'd expect hpet or the Microsoft synthetic timers to be a lot more important.

True. But also a lot more work.
Implementing just the pm timer counter - not the whole of it - in kernel, gives us a lot of gain with not very much effort. Patch is pretty simple, as you can see, and most of it is even code to turn it on/off, etc.

#4 lore raw
Quoted 6 lines · Glauber Costa
Quoted 3 lines

What is the motivation for this? Are there any important guests that use the pmtimer?

Avi,

All older RHEL and Windows, for example, would benefit for this.

They only benefit from it because we don't provide HPET. If we did, the guests would use HPET in preference to pmtimer, since HPET is so much better than pmtimer (yet still sucks in an absolute sense).

Quoted 9 lines
Quoted 2 lines

If anything I'd expect hpet or the Microsoft synthetic timers to be a lot more important.

True. But also a lot more work.
Implementing just the pm timer counter - not the whole of it - in kernel, gives us a lot of gain with not very much effort. Patch is pretty simple, as you can see, and most of it is even code to turn it on/off, etc.

Partial emulation is not something I like since it causes a fuzzy kernel/user boundary. In this case, transitioning to userspace when interrupts are enabled doesn't look so hot. Are you sure all guests that benefit from this don't enable the pmtimer interrupt? What about the transition? Will we have a time discontinuity when that happens?

What I'd really like to see is this stuff implemented in bytecode, unfortunately that's a lot of work which will be very hard to upstream.

Signature · 1 line
error compiling committee.c: too many arguments to function
#5 lore raw
Quoted 29 lines · Avi Kivity
Quoted 6 lines · Glauber Costa
Quoted 3 lines

What is the motivation for this? Are there any important guests that use the pmtimer?

Avi,

All older RHEL and Windows, for example, would benefit for this.

They only benefit from it because we don't provide HPET. If we did, the guests would use HPET in preference to pmtimer, since HPET is so much better than pmtimer (yet still sucks in an absolute sense).

Quoted 9 lines
Quoted 2 lines

If anything I'd expect hpet or the Microsoft synthetic timers to be a lot more important.

True. But also a lot more work.
Implementing just the pm timer counter - not the whole of it - in kernel, gives us a lot of gain with not very much effort. Patch is pretty simple, as you can see, and most of it is even code to turn it on/off, etc.

Partial emulation is not something I like since it causes a fuzzy kernel/user boundary. In this case, transitioning to userspace when interrupts are enabled doesn't look so hot. Are you sure all guests that benefit from this don't enable the pmtimer interrupt? What about the transition? Will we have a time discontinuity when that happens?

Quoted 4 lines

What I'd really like to see is this stuff implemented in bytecode, unfortunately that's a lot of work which will be very hard to upstream.

<joke>
Just use ACPI bytecode. It is upstream already.
</joke>

--
Gleb.

#6 lore raw
In reply to Avi Kivity
Quoted 20 lines · Avi Kivity
Quoted 6 lines · Glauber Costa
Quoted 3 lines

What is the motivation for this? Are there any important guests that use the pmtimer?

Avi,

All older RHEL and Windows, for example, would benefit for this.

They only benefit from it because we don't provide HPET. If we did, the guests would use HPET in preference to pmtimer, since HPET is so much better than pmtimer (yet still sucks in an absolute sense).

Quoted 9 lines
Quoted 2 lines

If anything I'd expect hpet or the Microsoft synthetic timers to be a lot more important.

True. But also a lot more work.
Implementing just the pm timer counter - not the whole of it - in kernel, gives us a lot of gain with not very much effort. Patch is pretty simple, as you can see, and most of it is even code to turn it on/off, etc.

Quoted 3 lines

Partial emulation is not something I like since it causes a fuzzy kernel/user boundary. In this case, transitioning to userspace when interrupts are enabled doesn't look so hot. Are you sure all guests that benefit from this don't enable the pmtimer interrupt? What about the transition? Will we have a time discontinuity when that happens?

What I'd really like to see is this stuff implemented in bytecode, unfortunately that's a lot of work which will be very hard to upstream.

Fortunately, we have a very good bytecode interpreter that's accelerated in the kernel called KVM ;-)

Why not have the equivalent of a paravirtual SMM mode where we can reflect IO exits back to the guest in a well defined way? It could then implement PM timer in terms of HPET or something like that.

We already have a virtual address space that works for most guests thanks to the TPR optimization.

Regards,

Anthony Liguori

#7 lore raw
Quoted 12 lines · Anthony Liguori
Quoted 12 lines
Quoted 11 lines
Quoted 2 lines

If anything I'd expect hpet or the Microsoft synthetic timers to

be a

Quoted 1 line

lot more important.

True. But also a lot more work.
Implementing just the pm timer counter - not the whole of it - in kernel, gives us a lot of gain with not very much effort. Patch is pretty simple, as you can see, and most of it is even code to turn it on/off, etc.

Quoted 6 lines
Quoted 3 lines

Partial emulation is not something I like since it causes a fuzzy kernel/user boundary. In this case, transitioning to userspace when interrupts are enabled doesn't look so hot. Are you sure all guests that benefit from this don't enable the pmtimer interrupt? What about the transition? Will we have a time discontinuity when that happens?

What I'd really like to see is this stuff implemented in bytecode, unfortunately that's a lot of work which will be very hard to upstream.

Fortunately, we have a very good bytecode interpreter that's accelerated in the kernel called KVM ;-)

We have exactly the same bytecode interpreter under a different name, it's called userspace.

If you can afford to make the transition back to the guest for emulation, you might as well transition to userspace.

Quoted 2 lines

Why not have the equivalent of a paravirtual SMM mode where we can reflect IO exits back to the guest in a well defined way? It could then implement PM timer in terms of HPET or something like that.

More exits.

Quoted 2 lines

We already have a virtual address space that works for most guests thanks to the TPR optimization.

It only works for Windows XP and Windows XP with the /3GB extension.

Signature · 1 line
error compiling committee.c: too many arguments to function
#8 lore raw
Quoted 5 lines · Avi Kivity

Fortunately, we have a very good bytecode interpreter that's accelerated in the kernel called KVM ;-)

We have exactly the same bytecode interpreter under a different name, it's called userspace.

If you can afford to make the transition back to the guest for emulation, you might as well transition to userspace.

If you re-entered the guest and setup a stack that had the RIP of the source of the exit, then there's no additional need to exit the guest. The handler can just do an iret. Or am I missing something?

Quoted 4 lines
Quoted 2 lines

Why not have the equivalent of a paravirtual SMM mode where we can reflect IO exits back to the guest in a well defined way? It could then implement PM timer in terms of HPET or something like that.

More exits.

Yeah, I should have said, implement in terms of kvmclock so no additional exits.

Quoted 4 lines
Quoted 2 lines

We already have a virtual address space that works for most guests thanks to the TPR optimization.

It only works for Windows XP and Windows XP with the /3GB extension.

Is this a fundamental limitation or just a statement of today's heuristics? Does any guest not keep the BIOS in virtual memory in a static location?

Regards,

Anthony Liguori

#9 lore raw
Quoted 8 lines · Anthony Liguori
Quoted 5 lines · Avi Kivity

Fortunately, we have a very good bytecode interpreter that's accelerated in the kernel called KVM ;-)

We have exactly the same bytecode interpreter under a different name, it's called userspace.

If you can afford to make the transition back to the guest for emulation, you might as well transition to userspace.

If you re-entered the guest and setup a stack that had the RIP of the source of the exit, then there's no additional need to exit the guest. The handler can just do an iret. Or am I missing something?

I didn't even consider an iret-to-guest, to be honest. Let's consider the options:

- iret-to-guest (a la tpr patching) - need to have an executable page
in the guest virtual address space and some stack space (on 64-bit, can rely on iretq switching the stack). That is probably impossible to do in a generic way without guest cooperation. If we rely on guest cooperation, we might as well have the guest patch the IN instruction itself (no exits at all).

- architectural SMM - no need to find a virtual mapping, or even a physical page, since we're in our own physical address space. However, the RSM instruction will trap, and on Intel, at least the first few instructions need to be emulated since SMM starts in big real mode. Also needs a tlb flush.

- kvm-specific SMM (probably what you referred to as "paravirt SMM", but if the guest OS is not involved, it's not really paravirt) - can switch to our own cr3 so no problem with finding a virtual mapping; however still needs a tlb flush, and on pre-NPT/EPT machines, switching cr3 back will involve an exit.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

We already have a virtual address space that works for most guests thanks to the TPR optimization.

It only works for Windows XP and Windows XP with the /3GB extension.

Is this a fundamental limitation or just a statement of today's heuristics? Does any guest not keep the BIOS in virtual memory in a static location?

If you're looking for a fundamental limitation, then yes, a guest need not map the BIOS at all. Practically, I believe all common guest do map the BIOS, but IIRC modern guests use non-executable mappings.

Signature · 1 line
error compiling committee.c: too many arguments to function
#10 lore raw
In reply to Ulrich Obergfell
Quoted 11 lines · Ulrich Obergfell

Hi,

This is an RFC through which I would like to get feedback on how the idea of in-kernel PM Timer would be received.

The current implementation of PM Timer emulation is 'heavy-weight' because the code resides in qemu userspace. Guest operating systems that use PM Timer as a clock source (for example, older versions of Linux that do not have paravirtualized clock) would benefit from an in-kernel PM Timer emulation.

Quoted 7 lines

Parts 1 thru 4 of this RFC contain experimental source code which I recently used to investigate the performance benefit. In a Linux guest, I was running a program that calls gettimeofday() 'n' times in a loop (the PM Timer register is read during each call). With in-kernel PM Timer, I observed a significant reduction of program execution time.

I've played with this in the past. Can you post real numbers, preferably, with a real work load?

Regards,

Anthony Liguori

Quoted 20 lines

The experimental code emulates the PM Timer register in KVM kernel. All other components of ACPI PM remain in qemu userspace. Also, the 'timer carry interrupt' feature is not implemented in-kernel. If a guest operating system needs to enable the 'timer carry interrupt', the code takes care that PM Timer emulation falls back to userspace. However, I think the design of the code has sufficient flexibility, so that anyone who would want to add the 'timer carry interrupt' feature in-kernel could try to do so later on.

Please review and please comment.

Regards,

Uli Obergfell
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#11 lore raw
Quoted 9 lines · Anthony Liguori
Quoted 6 lines

I recently used to investigate the performance benefit. In a Linux guest, I was running a program that calls gettimeofday() 'n' times in a loop (the PM Timer register is read during each call). With in-kernel PM Timer, I observed a significant reduction of program execution time.

I've played with this in the past. Can you post real numbers, preferably, with a real work load?

2 years ago I posted relative comparisons of the time sources for older RHEL guests:
http://www.mail-archive.com/[email protected]/msg07231.html

What's the relative speed of the in-kernel pmtimer compared to the PIT?

David

#12 lore raw
Quoted 11 lines · David S. Ahern
Quoted 7 lines · Anthony Liguori
Quoted 7 lines

I recently used to investigate the performance benefit. In a Linux guest, I was running a program that calls gettimeofday() 'n' times in a loop (the PM Timer register is read during each call). With in-kernel PM Timer, I observed a significant reduction of program execution time.

Quoted 7 lines
Quoted 3 lines

I've played with this in the past. Can you post real numbers, preferably, with a real work load?

2 years ago I posted relative comparisons of the time sources for older RHEL guests:
http://www.mail-archive.com/[email protected]/msg07231.html

Any time you write a program in userspace that effectively equates to a single PIO operation that is easy to emulate, it's going to be remarkably faster to implement that PIO emulation in the kernel than in userspace because vmexit exit cost dominates the execution path.

But that doesn't tell you what the impact is in real world workloads. Before we start pushing all device emulation into the kernel, we need to quantify how often gettimeofday() is really called in real workloads.

Regards,

Anthony Liguori

Quoted 4 lines

What's the relative speed of the in-kernel pmtimer compared to the PIT?

David

#13 lore raw
Quoted 3 lines · Anthony Liguori

But that doesn't tell you what the impact is in real world workloads. Before we start pushing all device emulation into the kernel, we need to quantify how often gettimeofday() is really called in real workloads.

The workload that inspired that example program at its current max load calls gtod upwards of 1000 times per second. The overhead of gettimeofday was the biggest factor when comparing performance to bare metal and esx. That's why I wrote the test program --- boils a complex product/program to a single system call.

David

Quoted 10 lines

Regards,

Anthony Liguori

Quoted 4 lines

What's the relative speed of the in-kernel pmtimer compared to the PIT?

David

#14 lore raw
Quoted 7 lines · David S. Ahern
Quoted 4 lines · Anthony Liguori

But that doesn't tell you what the impact is in real world workloads. Before we start pushing all device emulation into the kernel, we need to quantify how often gettimeofday() is really called in real workloads.

Quoted 6 lines

The workload that inspired that example program at its current max load calls gtod upwards of 1000 times per second. The overhead of gettimeofday was the biggest factor when comparing performance to bare metal and esx. That's why I wrote the test program --- boils a complex product/program to a single system call.

So the absolute performance impact was on the order of what?

The difference in CPU time of a light weight vs. heavy weight exit should be something like 2-3us. That would mean 2-3ms of CPU time at a rate of 1000 per second.

That should be pretty much in the noise.

There are possibly second order effects that might make a large impact such as contention with the qemu_mutex. It's worth doing experimentation to see if a non-mutex acquiring fast path in userspace also resulted in a significant performance boost.

Regards,

Anthony Liguori

Quoted 14 lines

David

Quoted 11 lines

Regards,

Anthony Liguori

Quoted 5 lines

What's the relative speed of the in-kernel pmtimer compared to the PIT?

David

#15 lore raw
Quoted 8 lines · Anthony Liguori
Quoted 7 lines · David S. Ahern
Quoted 4 lines · Anthony Liguori

But that doesn't tell you what the impact is in real world workloads. Before we start pushing all device emulation into the kernel, we need to quantify how often gettimeofday() is really called in real workloads.

Quoted 8 lines
Quoted 6 lines

The workload that inspired that example program at its current max load calls gtod upwards of 1000 times per second. The overhead of gettimeofday was the biggest factor when comparing performance to bare metal and esx. That's why I wrote the test program --- boils a complex product/program to a single system call.

So the absolute performance impact was on the order of what?

At the time I did the investigations (18-24 months ago) KVM was on the order of 15-20% worse for a RHEL4 based workload and the overhead appeared to be due to the PIT or PM timer as the clock source. Switching the clock to the TSC brought the performance on par with bare metal, but that route has other issues.

Quoted 4 lines

The difference in CPU time of a light weight vs. heavy weight exit should be something like 2-3us. That would mean 2-3ms of CPU time at a rate of 1000 per second.

The PIT causes 3 VMEXITs for each gettimeofday (get_offset_pit in RHEL4):

	/* timer count may underflow right here */
	outb_p(0x00, PIT_MODE);	/* latch the count ASAP */
...
	count = inb_p(PIT_CH0);	/* read the latched count */
...
	count |= inb_p(PIT_CH0) << 8;

...

David

Quoted 27 lines

That should be pretty much in the noise.

There are possibly second order effects that might make a large impact such as contention with the qemu_mutex. It's worth doing experimentation to see if a non-mutex acquiring fast path in userspace also resulted in a significant performance boost.

Regards,

Anthony Liguori

Quoted 14 lines

David

Quoted 11 lines

Regards,

Anthony Liguori

Quoted 5 lines

What's the relative speed of the in-kernel pmtimer compared to the PIT?

David