In kernel PIT patch

Hi

Here is the last in-kernel PIT patch for KVM. The mainly change from last version is the supporting to save/restore. I also tested live migration.

The other modifies including some date structure changed to be better for supporting the save/restore. I moved the PIT timer to outside of channel structure, which explicitly means only one channel (channel 0) would trigger it.

After fix TSC problem on SMP PAE RHEL5/5.1 guest, now the patch works well without any modify of kernel parameter.

Quoted 10 lines · Sheng

Hi

Here is the last in-kernel PIT patch for KVM. The mainly change from last version is the supporting to save/restore. I also tested live migration.

The other modifies including some date structure changed to be better for supporting the save/restore. I moved the PIT timer to outside of channel structure, which explicitly means only one channel (channel 0) would trigger it.

Quoted 3 lines

After fix TSC problem on SMP PAE RHEL5/5.1 guest, now the patch works well without any modify of kernel parameter.

How are you measuring the improvements from an in-kernel PIT? From your mails, you're claiming it increases the timer accuracy. How are you measuring it and how much does it improve it?

Do you expect an overall performance improvement from this or is it simply about improving timer accuracy?

Regards,

Anthony Liguori

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Quoted 12 lines · Anthony Liguori

Quoted 10 lines · Sheng

Hi

Here is the last in-kernel PIT patch for KVM. The mainly change from last version is the supporting to save/restore. I also tested live migration.

The other modifies including some date structure changed to be better for supporting the save/restore. I moved the PIT timer to outside of channel structure, which explicitly means only one channel (channel 0) would trigger it.

Quoted 9 lines

Quoted 4 lines

After fix TSC problem on SMP PAE RHEL5/5.1 guest, now the patch works well without any modify of kernel parameter.

How are you measuring the improvements from an in-kernel PIT? From your mails, you're claiming it increases the timer accuracy. How are you measuring it and how much does it improve it?

It's also a functionality addition: userspace pit & pic combination needed to use -tdf option (time drift fix). The tdf took care of pending pit irqs and tried to make the guest ack the right number of irqs the pit was configured.

Once we switched to the default in-kernel pic, the userspace pit couldn't get the acks from the pit.
One can see the effect when running multiple guests (windows, standard HAL) playing video, the time slows down.

This patch set has a pending counter and takes care for it too.

Quoted 4 lines

Do you expect an overall performance improvement from this or is it simply about improving timer accuracy?

It will probably help older kernels with slow HZ run faster HZ guests. Without CONFIG_DYNTICK the guests behaved jumpy because of that.

Quoted 15 lines

Regards,

Anthony Liguori

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Quoted 16 lines · Dor Laor

Quoted 15 lines · Anthony Liguori

Quoted 13 lines · Sheng

Hi

Here is the last in-kernel PIT patch for KVM. The mainly change from last version is the supporting to save/restore. I also tested live migration.

The other modifies including some date structure changed to be better for supporting the save/restore. I moved the PIT timer to outside of channel structure, which explicitly means only one channel (channel 0) would trigger it.

After fix TSC problem on SMP PAE RHEL5/5.1 guest, now the patch works well without any modify of kernel parameter.

Quoted 8 lines

Quoted 3 lines

How are you measuring the improvements from an in-kernel PIT? From your mails, you're claiming it increases the timer accuracy. How are you measuring it and how much does it improve it?

It's also a functionality addition: userspace pit & pic combination needed to use -tdf option (time drift fix). The tdf took care of pending pit irqs and tried to make the guest ack the right number of irqs the pit was configured.

I thought there was some discussion about whether -tdf was every useful in practice?

Quoted 4 lines

Once we switched to the default in-kernel pic, the userspace pit couldn't get the acks from the pit.
One can see the effect when running multiple guests (windows, standard HAL) playing video, the time slows down.

Okay, that makes sense. So have you done any tests to confirm this? We suffered through a fair number of regressions when we moved to an in-kernel APIC. Before moving another big chunk of code in the kernel and going through possible regressions, I want to make sure we have a measurable argument that it's the right thing to do.

So how do we measure the benefits of an in-kernel PIT?

Regards,

Anthony Liguori

Quoted 22 lines

This patch set has a pending counter and takes care for it too.

Quoted 2 lines

Do you expect an overall performance improvement from this or is it simply about improving timer accuracy?

It will probably help older kernels with slow HZ run faster HZ guests. Without CONFIG_DYNTICK the guests behaved jumpy because of that.

Quoted 14 lines

Regards,

Anthony Liguori

------------------------------------------------------------------------
- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Quoted 2 lines · On Wednesday 05 March 2008 08:50:24 Anthony Liguori wrote:

So how do we measure the benefits of an in-kernel PIT?

On the time accuracy side, one typical example is in RHEL5 32E guest, time flows very slow compared to the host
(https://sourceforge.net/tracker/?func=detail&atid=893831&aid=1826080&group_id=180599).
You can simple using "sleep" to test it. And many people complained it before, e,g,
http://www.mail-archive.com/[email protected]/msg10928.html
And I have to say the timer problem in current KVM is very serious, and this patch can solve this.

I think you are most worrying about the regressions. That's why I spent a lot of time to solve TSC problem (PAE SMP RHEL5.1 can't boot up). For in kernel PIT accelerate the process, the same bug was exposed on PAE SMP RHEL5 with the patch. Though I don't think it's a real regression, I have got it done to prevent this patch bring any bad effect.

I would do more test to ensure this patch won't break something.

Quoted 11 lines · Sheng

Quoted 1 line · On Wednesday 05 March 2008 08:50:24 Anthony Liguori wrote:

So how do we measure the benefits of an in-kernel PIT?

On the time accuracy side, one typical example is in RHEL5 32E guest, time flows very slow compared to the host
(https://sourceforge.net/tracker/?func=detail&atid=893831&aid=1826080&gro
up_id=180599). You can simple using "sleep" to test it. And many people complained it before, e,g,
http://www.mail-archive.com/[email protected]/msg10928.html
And I have to say the timer problem in current KVM is very serious, and this patch can solve this.

Okay, then my question is, how much does this patch set improve the situation?

For instance, the bug report shows some circumstances where:

On IA32e RHEL4 guest with
Realtime 3min
Guest 3min15s

So what is the guest time with an in-kernel PIT? How is this affected by the various possible -clock options? What I'm looking for is an example of how much we're improving the situation and some assurance that this is the only way to solve the problem.

I'm not fundamentally opposed to an in-kernel PIT, I just am trying to understand the justification.

Regards,

Anthony Liguori

Quoted 7 lines

I think you are most worrying about the regressions. That's why I spent a lot of time to solve TSC problem (PAE SMP RHEL5.1 can't boot up). For in kernel PIT accelerate the process, the same bug was exposed on PAE SMP RHEL5 with the patch. Though I don't think it's a real regression, I have got it done to prevent this patch bring any bad effect.

I would do more test to ensure this patch won't break something.

Quoted 24 lines · Dor Laor

Quoted 21 lines · Anthony Liguori

Quoted 5 lines · Sheng

Hi

Here is the last in-kernel PIT patch for KVM. The mainly change from last version is the supporting to save/restore. I also tested live migration.

Quoted 9 lines

The other modifies including some date structure changed to be better for supporting the save/restore. I moved the PIT timer to outside of channel structure, which explicitly means only one channel (channel 0) would trigger it.

After fix TSC problem on SMP PAE RHEL5/5.1 guest, now the patch works well without any modify of kernel parameter.

How are you measuring the improvements from an in-kernel PIT? From your mails, you're claiming it increases the timer accuracy. How are you measuring it and how much does it improve it?

Quoted 5 lines

It's also a functionality addition: userspace pit & pic combination needed to use -tdf option (time drift fix). The tdf took care of pending pit irqs and tried to make the guest ack the right number of irqs the pit was configured.

I thought there was some discussion about whether -tdf was every useful in practice?

Quoted 6 lines

Once we switched to the default in-kernel pic, the userspace pit couldn't get the acks from the pit.
One can see the effect when running multiple guests (windows, standard HAL) playing video, the time slows down.

Okay, that makes sense. So have you done any tests to confirm this? We suffered through a fair number of regressions when we moved to an in-kernel APIC. Before moving another big chunk of code in the kernel and going through possible regressions, I want to make sure we have a measurable argument that it's the right thing to do.

So how do we measure the benefits of an in-kernel PIT?

Regards,

Anthony Liguori

Quoted 29 lines

This patch set has a pending counter and takes care for it too.

Quoted 4 lines

Do you expect an overall performance improvement from this or is it simply about improving timer accuracy?

It will probably help older kernels with slow HZ run faster HZ guests. Without CONFIG_DYNTICK the guests behaved jumpy because of that.

Quoted 15 lines

Regards,

Anthony Liguori

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
kvm-devel mailing list [email protected]
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Quoted 2 lines · Anthony Liguori

Dor Laor wrote:

Quoted 3 lines

I thought there was some discussion about whether -tdf was every useful in practice?

It works.
Just try to play a movie in windows standard HAL with and w/o -tdf --no-irq-chip and you'll see the difference.

Quoted 3 lines

So how do we measure the benefits of an in-kernel PIT?

Play the same movie using the kernel's pit.

Playing a movie is a bit subjective. I presume you're talking about the standard HAL as presumably the ACPI HAL is using the pm timer?

So the two cases I'm hearing where timer accuracy should improve is standard HAL on Windows and clock=pit on Linux? I'd still like to see what the actual difference in timer accuracy is.

I have no doubt that
moving the pit into the kernel is more efficient. Moving everything into the kernel is more efficient because light weight exits are cheaper than heavy weight exits.

The thing I'm trying to get at is a quantitative statement about why moving the pit into the kernel is the right thing. I'll try to give the patches a try myself in the next couple of days. I don't think it's obvious that it's the right thing to do without some sort of benchmark supporting it.

Quoted 3 lines · Anthony Liguori

Playing a movie is a bit subjective. I presume you're talking about the standard HAL as presumably the ACPI HAL is using the pm timer?

ACPI HAL uses the apic timer, IIRC; perhaps the pm timer as well.

Quoted 3 lines

So the two cases I'm hearing where timer accuracy should improve is standard HAL on Windows and clock=pit on Linux? I'd still like to see what the actual difference in timer accuracy is.

It depends on the load. As the load increases, the host process starts to miss timer signals. With both pic and pit in userspace, you can detect those missed interrupts and inject them later once you get your timeslice. With the pic in kernel, there is no way to do this.

The same thing happens with the apic timer, only there, it is easy to compensate because both parts are in the kernel.

Quoted 5 lines

I have no doubt that
moving the pit into the kernel is more efficient. Moving everything into the kernel is more efficient because light weight exits are cheaper than heavy weight exits.

Efficiency is only a secondary goal here. The userspace PIT does not consume large amounts of CPU.

Quoted 6 lines

The thing I'm trying to get at is a quantitative statement about why moving the pit into the kernel is the right thing. I'll try to give the patches a try myself in the next couple of days. I don't think it's obvious that it's the right thing to do without some sort of benchmark supporting it.

Playing a movie is better than any benchmark; it reflects actual user experience in a real and important use case. Benchmarks are substitutes for real use cases, not the goal of the optimization.

Quoted 6 lines · Avi Kivity

Playing a movie is better than any benchmark; it reflects actual user experience in a real and important use case. Benchmarks are substitutes for real use cases, not the goal of the optimization.

I forgot to mention that the benchmark is measuring time drift in the guest. Playing a movie in winxp changes the guest clock frequency from 100HZ to 1000HZ, thus causing a 250HZ host to coalesce pit irqs.

So good pic & pit combination can handle guest multimedia without drifts while insufficient implementation just can't.

Quoted 13 lines · Dor Laor

Quoted 6 lines · Avi Kivity

Playing a movie is better than any benchmark; it reflects actual user experience in a real and important use case. Benchmarks are substitutes for real use cases, not the goal of the optimization.

I forgot to mention that the benchmark is measuring time drift in the guest. Playing a movie in winxp changes the guest clock frequency from 100HZ to 1000HZ, thus causing a 250HZ host to coalesce pit irqs.

Quoted 3 lines

So good pic & pit combination can handle guest multimedia without drifts while insufficient implementation just can't.

I'll try out these patches in the next day or so. Is the expectation that Standard HAL + in-kernel APIC/PIT will have smoother playback than ACPI HAL w/o in-kernel APIC/PIT? I presume that the ACPI HAL is unaffected by in-kernel PIT?

Regards,

Anthony Liguori

Quoted 7 lines

The thing I'm trying to get at is a quantitative statement about why moving the pit into the kernel is the right thing. I'll try to give the patches a try myself in the next couple of days. I don't think it's obvious that it's the right thing to do without some sort of benchmark supporting it.

Playing a movie is better than any benchmark; it reflects actual user experience in a real and important use case. Benchmarks are substitutes for real use cases, not the goal of the optimization.

Quoted 11 lines · Avi Kivity

Quoted 7 lines

The thing I'm trying to get at is a quantitative statement about why moving the pit into the kernel is the right thing. I'll try to give the patches a try myself in the next couple of days. I don't think it's obvious that it's the right thing to do without some sort of benchmark supporting it.

Playing a movie is better than any benchmark; it reflects actual user experience in a real and important use case. Benchmarks are substitutes for real use cases, not the goal of the optimization.

I tried out WinXP with the standard HAL and the in-kernel APIC patches. I did not see any appreciable improvement in multimedia playback (video or audio). I still get the same amount of jitters. I've tried with and without -tdf too.

So far, the smoothest play back I've gotten is using the ACPI HAL. Can you point to a particular example where you see an improvement? Perhaps a divx or movie trailer that is better with it than without it?

Regards,

Anthony Liguori