The HPET issue on Linux
Hi Beth
I still found the emulated HPET would result in some boot failure. For example, on my 2.6.30, with HPET enabled, the kernel would fail check_timer(), especially in timer_irq_works().
The testing of timer_irq_works() is let 10 ticks pass(using mdelay()), and want to confirm the clock source with at least 5 ticks advanced in jiffies. I've checked that, on my machine, it would mostly get only 4 ticks when HPET enabled, then fail the test. On the other hand, if I using PIT, it would get more than 10 ticks(maybe understandable if some complementary ticks there). Of course, extend the ticks count/mdelay() time can work.
I think it's a major issue of HPET. And it maybe just due to a too long userspace path for interrupt injection... If it's true, I think it's not easy to deal with it.
Signature · 2 lines
regards Yang, Sheng
Quoted 13 lines · Sheng Yang
Hi Beth
I still found the emulated HPET would result in some boot failure. For example, on my 2.6.30, with HPET enabled, the kernel would fail
check_timer(), especially intimer_irq_works().The testing of
timer_irq_works()is let 10 ticks pass(usingmdelay()), and want to confirm the clock source with at least 5 ticks advanced in jiffies. I've checked that, on my machine, it would mostly get only 4 ticks when HPET enabled, then fail the test. On the other hand, if I using PIT, it would get more than 10 ticks(maybe understandable if some complementary ticks there). Of course, extend the ticks count/mdelay() time can work.
Quoted 4 lines
I think it's a major issue of HPET. And it maybe just due to a too long userspace path for interrupt injection... If it's true, I think it's not easy to deal with it.
PIT tick are reinjected automatically, HPET should probably do the same although it may just create another set of problems.
--
Gleb.
Quoted 14 lines · Gleb Natapov
Quoted 13 lines · Sheng Yang
Hi Beth
I still found the emulated HPET would result in some boot failure. For example, on my 2.6.30, with HPET enabled, the kernel would fail
check_timer(), especially intimer_irq_works().The testing of
timer_irq_works()is let 10 ticks pass(usingmdelay()), and want to confirm the clock source with at least 5 ticks advanced in jiffies. I've checked that, on my machine, it would mostly get only 4 ticks when HPET enabled, then fail the test. On the other hand, if I using PIT, it would get more than 10 ticks(maybe understandable if some complementary ticks there). Of course, extend the ticks count/mdelay() time can work.
Quoted 6 lines
Quoted 4 lines
I think it's a major issue of HPET. And it maybe just due to a too long userspace path for interrupt injection... If it's true, I think it's not easy to deal with it.
PIT tick are reinjected automatically, HPET should probably do the same although it may just create another set of problems.
Older Linux do automatic adjustment for lost ticks so automatic reinjection causes time to run too fast. This is why we added the -no-kvm-pit-reinject flag...
It took lots of time to pit/rtc to stabilize, in order of seriously consider the hpet emulation, lots of testing should be done.
Quoted 7 lines
--
Gleb.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 19 lines · Dor Laor
Quoted 18 lines · Gleb Natapov
Quoted 17 lines · Sheng Yang
Hi Beth
I still found the emulated HPET would result in some boot failure. For example, on my 2.6.30, with HPET enabled, the kernel would fail
check_timer(), especially intimer_irq_works().The testing of
timer_irq_works()is let 10 ticks pass(usingmdelay()), and want to confirm the clock source with at least 5 ticks advanced in jiffies. I've checked that, on my machine, it would mostly get only 4 ticks when HPET enabled, then fail the test. On the other hand, if I using PIT, it would get more than 10 ticks(maybe understandable if some complementary ticks there). Of course, extend the ticks count/mdelay() time can work.I think it's a major issue of HPET. And it maybe just due to a too long userspace path for interrupt injection... If it's true, I think it's not easy to deal with it.
Quoted 7 lines
Quoted 2 lines
PIT tick are reinjected automatically, HPET should probably do the same although it may just create another set of problems.
Older Linux do automatic adjustment for lost ticks so automatic reinjection causes time to run too fast. This is why we added the -no-kvm-pit-reinject flag...
It took lots of time to pit/rtc to stabilize, in order of seriously consider the hpet emulation, lots of testing should be done.
I will try to look into this. Since HPET is edge-triggered, looks like this problem is of a different nature than PIT. Is this a solid failure or intermittent?
Quoted 13 lines
Quoted 8 lines
Signature · 6 lines
-- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlSignature · 3 lines
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
Regards, Beth Kon
Quoted 20 lines · Beth Kon
Quoted 19 lines · Dor Laor
Quoted 18 lines · Gleb Natapov
Quoted 17 lines · Sheng Yang
Hi Beth
I still found the emulated HPET would result in some boot failure. For example, on my 2.6.30, with HPET enabled, the kernel would fail
check_timer(), especially intimer_irq_works().The testing of
timer_irq_works()is let 10 ticks pass(usingmdelay()), and want to confirm the clock source with at least 5 ticks advanced in jiffies. I've checked that, on my machine, it would mostly get only 4 ticks when HPET enabled, then fail the test. On the other hand, if I using PIT, it would get more than 10 ticks(maybe understandable if some complementary ticks there). Of course, extend the ticks count/mdelay() time can work.I think it's a major issue of HPET. And it maybe just due to a too long userspace path for interrupt injection... If it's true, I think it's not easy to deal with it.
Quoted 8 lines
Quoted 7 lines
Quoted 2 lines
PIT tick are reinjected automatically, HPET should probably do the same although it may just create another set of problems.
Older Linux do automatic adjustment for lost ticks so automatic reinjection causes time to run too fast. This is why we added the -no-kvm-pit-reinject flag...
It took lots of time to pit/rtc to stabilize, in order of seriously consider the hpet emulation, lots of testing should be done.
I will try to look into this. Since HPET is edge-triggered, looks like this problem is of a different nature than PIT. Is this a solid failure or intermittent?
Anthony just explained that on x86, even edge-triggered interrupts are queued in the apic and an eoi will occur, so this is not different than the PIT.
Quoted 16 lines
Quoted 14 lines
Quoted 9 lines
Signature · 6 lines
-- Gleb. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.htmlSignature · 3 lines
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 3 lines
Regards, Beth Kon
Quoted 3 lines · Beth Kon
Quoted 1 line · Beth Kon
I will try to look into this. Since HPET is edge-triggered, looks like this problem is of a different nature than PIT. Is this a solid failure or intermittent?
Anthony just explained that on x86, even edge-triggered interrupts are queued in the apic and an eoi will occur, so this is not different than the PIT.
Not quite queued in the sense that multiple events will be delivered in order, but I think the point is that you can still detect whether delivery succeeded by counting APIC EOIs.
The trouble is that historically we've struggled with doing this in userspace. Maybe it's time to revisit.
Regards,
Anthony Liguori
Quoted 9 lines · Anthony Liguori
Quoted 7 lines · Beth Kon
Quoted 3 lines · Beth Kon
I will try to look into this. Since HPET is edge-triggered, looks like this problem is of a different nature than PIT. Is this a solid failure or intermittent?
Anthony just explained that on x86, even edge-triggered interrupts are queued in the apic and an eoi will occur, so this is not different than the PIT.
Quoted 7 lines
Not quite queued in the sense that multiple events will be delivered in order, but I think the point is that you can still detect whether delivery succeeded by counting APIC EOIs.
The trouble is that historically we've struggled with doing this in userspace. Maybe it's time to revisit.
We reinject PIT interrupts from kernel and RTC interrupts from userspace.
--
Gleb.
Quoted 18 lines · Gleb Natapov
Quoted 16 lines · Anthony Liguori
Quoted 10 lines · Beth Kon
Quoted 4 lines · Beth Kon
I will try to look into this. Since HPET is edge-triggered, looks like this problem is of a different nature than PIT. Is this a solid failure or intermittent?
Anthony just explained that on x86, even edge-triggered interrupts are queued in the apic and an eoi will occur, so this is not different than the PIT.
Not quite queued in the sense that multiple events will be delivered in order, but I think the point is that you can still detect whether delivery succeeded by counting APIC EOIs.
Quoted 7 lines
Quoted 4 lines
The trouble is that historically we've struggled with doing this in userspace. Maybe it's time to revisit.
We reinject PIT interrupts from kernel and RTC interrupts from userspace.
Because we can determine that we've missed an RTC interrupt in userspace. We cannot determine this with the PIT without adding a hook into the userspace apic that lets us know whether an injection failed or not.
Regards,
Anthony Liguori
Quoted 17 lines · Anthony Liguori
Quoted 16 lines · Gleb Natapov
Quoted 15 lines · Anthony Liguori
Quoted 7 lines · Beth Kon
Quoted 3 lines · Beth Kon
I will try to look into this. Since HPET is edge-triggered, looks like this problem is of a different nature than PIT. Is this a solid failure or intermittent?
Anthony just explained that on x86, even edge-triggered interrupts are queued in the apic and an eoi will occur, so this is not different than the PIT.
Not quite queued in the sense that multiple events will be delivered in order, but I think the point is that you can still detect whether delivery succeeded by counting APIC EOIs.
The trouble is that historically we've struggled with doing this in userspace. Maybe it's time to revisit.
Quoted 8 lines
Quoted 2 lines
We reinject PIT interrupts from kernel and RTC interrupts from userspace.
Because we can determine that we've missed an RTC interrupt in userspace. We cannot determine this with the PIT without adding a hook into the userspace apic that lets us know whether an injection failed or not.
We have exactly that hook in apic already and that's how RTC determines that interrupt was coalesced.
--
Gleb.
Quoted 3 lines · Gleb Natapov
We have exactly that hook in apic already and that's how RTC determines that interrupt was coalesced.
AFAICT, apic_irq_delivered is only reset explicitly by the RTC when the line is lowered. It's not currently lowered based on EOI.
How can this mechanism be used with the HPET when operating in edge triggered mode?
Regards,
Anthony Liguori
Quoted 3 lines
--
Gleb.
Quoted 34 lines · On Thursday 07 January 2010 02:36:26 Beth Kon wrote:
Quoted 33 lines · Dor Laor
Quoted 20 lines · Gleb Natapov
Quoted 19 lines · Sheng Yang
Hi Beth
I still found the emulated HPET would result in some boot failure. For example, on my 2.6.30, with HPET enabled, the kernel would fail
check_timer(),
especially intimer_irq_works().The testing of
timer_irq_works()is let 10 ticks pass(usingmdelay()), and
want to confirm the clock source with at least 5 ticks advanced in jiffies.
I've checked that, on my machine, it would mostly get only 4 ticks when HPET
enabled, then fail the test. On the other hand, if I using PIT, it would get
more than 10 ticks(maybe understandable if some complementary ticks there). Of
course, extend the ticks count/mdelay() time can work.Quoted 7 lines
Quoted 4 lines
I think it's a major issue of HPET. And it maybe just due to a too long userspace path for interrupt injection... If it's true, I think it's not easy
to deal with it.PIT tick are reinjected automatically, HPET should probably do the same although it may just create another set of problems.
Older Linux do automatic adjustment for lost ticks so automatic reinjection causes time to run too fast. This is why we added the -no-kvm-pit-reinject flag...
Quoted 7 lines
Quoted 2 lines
It took lots of time to pit/rtc to stabilize, in order of seriously consider the hpet emulation, lots of testing should be done.
I will try to look into this. Since HPET is edge-triggered, looks like this problem is of a different nature than PIT. Is this a solid failure or intermittent?
At least for v2.6.30 in my box, it always fails... Of course, I believe the chance of successful injecting enough interrupt depends on the many factors. So I think out target can be: not far behind what PIT can do...
Signature · 2 lines
regards Yang, Sheng
Quoted 7 lines · Anthony Liguori
Quoted 2 lines · Gleb Natapov
We have exactly that hook in apic already and that's how RTC determines that interrupt was coalesced.
AFAICT,
apic_irq_deliveredis only reset explicitly by the RTC when the line is lowered. It's not currently lowered based on EOI.
Correct. We can expose ACK notifiers to userspace (and if we want to move assigned devices into userspace we have to), but I'd rather avoid it.
Quoted 3 lines
How can this mechanism be used with the HPET when operating in edge triggered mode?
If interrupt is coalesced increment counter and double HPET timer frequency. When counter is zeroed return HPET timer to normal frequency.
--
Gleb.
No longer working on HPET
In reply to Gleb NatapovI have decided to take a job outside of IBM and so will not be involved with HPET any longer. Working on KVM has been great fun... top-notch people and a top-notch technology. Wishing KVM and you all the best!
Signature · 3 lines
Regards, Beth Kon