Skip to contentSkip to conversation

KVM: x86: Avoid busy loops over uninjectable pending APIC timers

PATCH In review Jan Kiszka 4 participants 26 messages Mar 16, 2013 – Apr 28, 2013

Subsystems (3)
x86 KVM KVM x86
lore
Share & tools
lore

Jan Kiszka

author
#0 lore raw

PATCH 1/1 KVM: x86: Avoid busy loops over uninjectable pending APIC timers

From: Jan Kiszka <[email protected]>

If the guest didn't take the last APIC timer interrupt yet and generates another one on top, e.g. via periodic mode, we do not block the VCPU even if the guest state is halted. The reason is that apic_has_pending_timer continues to return a non-zero value.

Fix this busy loop by taking the IRR content for the LVT vector in apic_has_pending_timer into account.

Signed-off-by Jan Kiszka <[email protected]>

---

Not a critical issue, we are looping fully interruptible, but it's ugly to do so IMHO.

 arch/x86/kvm/lapic.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
arch/x86/kvm/lapic.c +3 -1
index a8e9369..658abf5 100644
@@ -1473,7 +1473,9 @@ int apic_has_pending_timer(struct kvm_vcpu *vcpu)
 	struct kvm_lapic *apic = vcpu->arch.apic;
 
 	if (kvm_vcpu_has_lapic(vcpu) && apic_enabled(apic) &&
-			apic_lvt_enabled(apic, APIC_LVTT))
+	    apic_lvt_enabled(apic, APIC_LVTT) &&
+	    !apic_test_vector(apic_lvt_vector(apic, APIC_LVTT),
+					      apic->regs + APIC_IRR))
 		return atomic_read(&apic->lapic_timer.pending);
 
 	return 0;
Signature · 1 line
1.7.3.4
#1 lore raw
Quoted 10 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

If the guest didn't take the last APIC timer interrupt yet and generates another one on top, e.g. via periodic mode, we do not block the VCPU even if the guest state is halted. The reason is that apic_has_pending_timer continues to return a non-zero value.

Fix this busy loop by taking the IRR content for the LVT vector in apic_has_pending_timer into account.

Just drop coalescing tacking for lapic interrupt. After posted interrupt will be merged __apic_accept_irq() will not longer return coalescing information, so the code will be dead anyway.

Quoted 26 lines
Signed-off-by Jan Kiszka <[email protected]>

---

Not a critical issue, we are looping fully interruptible, but it's ugly to do so IMHO.

 arch/x86/kvm/lapic.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
arch/x86/kvm/lapic.c +3 -1
index a8e9369..658abf5 100644
@@ -1473,7 +1473,9 @@ int apic_has_pending_timer(struct kvm_vcpu *vcpu)
 	struct kvm_lapic *apic = vcpu->arch.apic;
 
 	if (kvm_vcpu_has_lapic(vcpu) && apic_enabled(apic) &&
-			apic_lvt_enabled(apic, APIC_LVTT))
+	    apic_lvt_enabled(apic, APIC_LVTT) &&
+	    !apic_test_vector(apic_lvt_vector(apic, APIC_LVTT),
+					      apic->regs + APIC_IRR))
 		return atomic_read(&apic->lapic_timer.pending);
 
 	return 0;
Signature · 1 line
1.7.3.4

--
Gleb.

author
#2 lore raw
Quoted 8 lines · Gleb Natapov
Quoted 7 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

If the guest didn't take the last APIC timer interrupt yet and generates another one on top, e.g. via periodic mode, we do not block the VCPU even if the guest state is halted. The reason is that apic_has_pending_timer continues to return a non-zero value.

Quoted 6 lines
Quoted 3 lines

Fix this busy loop by taking the IRR content for the LVT vector in apic_has_pending_timer into account.

Just drop coalescing tacking for lapic interrupt. After posted interrupt will be merged __apic_accept_irq() will not longer return coalescing information, so the code will be dead anyway.

That requires the RTC decoalescing series to go first to avoid a regression, no? Then let's postpone this topic for now.

Jan

#3 lore raw
Quoted 12 lines · Jan Kiszka
Quoted 11 lines · Gleb Natapov
Quoted 10 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

If the guest didn't take the last APIC timer interrupt yet and generates another one on top, e.g. via periodic mode, we do not block the VCPU even if the guest state is halted. The reason is that apic_has_pending_timer continues to return a non-zero value.

Fix this busy loop by taking the IRR content for the LVT vector in apic_has_pending_timer into account.

Quoted 7 lines
Quoted 3 lines

Just drop coalescing tacking for lapic interrupt. After posted interrupt will be merged __apic_accept_irq() will not longer return coalescing information, so the code will be dead anyway.

That requires the RTC decoalescing series to go first to avoid a regression, no? Then let's postpone this topic for now.

Yes, but decoalescing will work only for RTC :(

--
Gleb.

#4 lore raw
Quoted 13 lines · Gleb Natapov
Quoted 12 lines · Jan Kiszka
Quoted 11 lines · Gleb Natapov
Quoted 10 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

If the guest didn't take the last APIC timer interrupt yet and generates another one on top, e.g. via periodic mode, we do not block the VCPU even if the guest state is halted. The reason is that apic_has_pending_timer continues to return a non-zero value.

Fix this busy loop by taking the IRR content for the LVT vector in apic_has_pending_timer into account.

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

Just drop coalescing tacking for lapic interrupt. After posted interrupt will be merged __apic_accept_irq() will not longer return coalescing information, so the code will be dead anyway.

That requires the RTC decoalescing series to go first to avoid a regression, no? Then let's postpone this topic for now.

Yes, but decoalescing will work only for RTC :(

Are you proposing to drop LAPIC interrupt reinjection?

#5 lore raw
Quoted 18 lines · Marcelo Tosatti
Quoted 17 lines · Gleb Natapov
Quoted 16 lines · Jan Kiszka
Quoted 14 lines · Gleb Natapov
Quoted 10 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

If the guest didn't take the last APIC timer interrupt yet and generates another one on top, e.g. via periodic mode, we do not block the VCPU even if the guest state is halted. The reason is that apic_has_pending_timer continues to return a non-zero value.

Fix this busy loop by taking the IRR content for the LVT vector in apic_has_pending_timer into account.

Just drop coalescing tacking for lapic interrupt. After posted interrupt will be merged __apic_accept_irq() will not longer return coalescing information, so the code will be dead anyway.

Quoted 6 lines
Quoted 4 lines
Quoted 3 lines

That requires the RTC decoalescing series to go first to avoid a regression, no? Then let's postpone this topic for now.

Yes, but decoalescing will work only for RTC :(

Are you proposing to drop LAPIC interrupt reinjection?

Since timer handling and injection is VCPU-local for LAPIC, __apic_accept_irq can (and must) return coalesced information (cannot drop LAPIC interrupt reinjection).

#6 lore raw
Quoted 22 lines · Marcelo Tosatti
Quoted 21 lines · Marcelo Tosatti
Quoted 20 lines · Gleb Natapov
Quoted 19 lines · Jan Kiszka
Quoted 8 lines · Gleb Natapov
Quoted 7 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

If the guest didn't take the last APIC timer interrupt yet and generates another one on top, e.g. via periodic mode, we do not block the VCPU even if the guest state is halted. The reason is that apic_has_pending_timer continues to return a non-zero value.

Quoted 6 lines
Quoted 3 lines

Fix this busy loop by taking the IRR content for the LVT vector in apic_has_pending_timer into account.

Just drop coalescing tacking for lapic interrupt. After posted interrupt will be merged __apic_accept_irq() will not longer return coalescing information, so the code will be dead anyway.

That requires the RTC decoalescing series to go first to avoid a regression, no? Then let's postpone this topic for now.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Yes, but decoalescing will work only for RTC :(

Are you proposing to drop LAPIC interrupt reinjection?

Since timer handling and injection is VCPU-local for LAPIC, __apic_accept_irq can (and must) return coalesced information (cannot drop LAPIC interrupt reinjection).

Why can't we drop LAPIC interrupt reinjection? Proposed posted interrupt patches do not properly check for interrupt coalescing even for VCPU-local injection.

--
Gleb.

#7 lore raw
Quoted 31 lines · Gleb Natapov
Quoted 22 lines · Marcelo Tosatti
Quoted 21 lines · Marcelo Tosatti
Quoted 20 lines · Gleb Natapov
Quoted 19 lines · Jan Kiszka
Quoted 8 lines · Gleb Natapov
Quoted 7 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

If the guest didn't take the last APIC timer interrupt yet and generates another one on top, e.g. via periodic mode, we do not block the VCPU even if the guest state is halted. The reason is that apic_has_pending_timer continues to return a non-zero value.

Quoted 6 lines
Quoted 3 lines

Fix this busy loop by taking the IRR content for the LVT vector in apic_has_pending_timer into account.

Just drop coalescing tacking for lapic interrupt. After posted interrupt will be merged __apic_accept_irq() will not longer return coalescing information, so the code will be dead anyway.

That requires the RTC decoalescing series to go first to avoid a regression, no? Then let's postpone this topic for now.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Yes, but decoalescing will work only for RTC :(

Are you proposing to drop LAPIC interrupt reinjection?

Since timer handling and injection is VCPU-local for LAPIC, __apic_accept_irq can (and must) return coalesced information (cannot drop LAPIC interrupt reinjection).

Quoted 6 lines

Why can't we drop LAPIC interrupt reinjection? Proposed posted interrupt patches do not properly check for interrupt coalescing even for VCPU-local injection.

--
Gleb.

Because older Linux guests depend on reinjection for proper timekeeping.

#8 lore raw
Quoted 32 lines · Marcelo Tosatti
Quoted 31 lines · Gleb Natapov
Quoted 30 lines · Marcelo Tosatti
Quoted 18 lines · Marcelo Tosatti
Quoted 17 lines · Gleb Natapov
Quoted 16 lines · Jan Kiszka
Quoted 14 lines · Gleb Natapov
Quoted 10 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

If the guest didn't take the last APIC timer interrupt yet and generates another one on top, e.g. via periodic mode, we do not block the VCPU even if the guest state is halted. The reason is that apic_has_pending_timer continues to return a non-zero value.

Fix this busy loop by taking the IRR content for the LVT vector in apic_has_pending_timer into account.

Just drop coalescing tacking for lapic interrupt. After posted interrupt will be merged __apic_accept_irq() will not longer return coalescing information, so the code will be dead anyway.

Quoted 6 lines
Quoted 4 lines
Quoted 3 lines

That requires the RTC decoalescing series to go first to avoid a regression, no? Then let's postpone this topic for now.

Yes, but decoalescing will work only for RTC :(

Are you proposing to drop LAPIC interrupt reinjection?

Since timer handling and injection is VCPU-local for LAPIC, __apic_accept_irq can (and must) return coalesced information (cannot drop LAPIC interrupt reinjection).

Quoted 8 lines
Quoted 6 lines

Why can't we drop LAPIC interrupt reinjection? Proposed posted interrupt patches do not properly check for interrupt coalescing even for VCPU-local injection.

--
Gleb.

Because older Linux guests depend on reinjection for proper timekeeping.

Which versions? Those without kvmclock? Can we make them use PIT instead? Posted interrupts going to break them.

--
Gleb.

#9 lore raw
Quoted 37 lines · Gleb Natapov
Quoted 36 lines · Marcelo Tosatti
Quoted 35 lines · Gleb Natapov
Quoted 22 lines · Marcelo Tosatti
Quoted 21 lines · Marcelo Tosatti
Quoted 20 lines · Gleb Natapov
Quoted 19 lines · Jan Kiszka
Quoted 8 lines · Gleb Natapov
Quoted 7 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

If the guest didn't take the last APIC timer interrupt yet and generates another one on top, e.g. via periodic mode, we do not block the VCPU even if the guest state is halted. The reason is that apic_has_pending_timer continues to return a non-zero value.

Quoted 6 lines
Quoted 3 lines

Fix this busy loop by taking the IRR content for the LVT vector in apic_has_pending_timer into account.

Just drop coalescing tacking for lapic interrupt. After posted interrupt will be merged __apic_accept_irq() will not longer return coalescing information, so the code will be dead anyway.

That requires the RTC decoalescing series to go first to avoid a regression, no? Then let's postpone this topic for now.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Yes, but decoalescing will work only for RTC :(

Are you proposing to drop LAPIC interrupt reinjection?

Since timer handling and injection is VCPU-local for LAPIC, __apic_accept_irq can (and must) return coalesced information (cannot drop LAPIC interrupt reinjection).

Why can't we drop LAPIC interrupt reinjection? Proposed posted interrupt patches do not properly check for interrupt coalescing even for VCPU-local injection.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

--
Gleb.

Because older Linux guests depend on reinjection for proper timekeeping.

Which versions? Those without kvmclock? Can we make them use PIT instead? Posted interrupts going to break them.

There is no reason to break them if its OK to receive reinjection info from LAPIC... its a matter of returning the information from apic_accept_irq, no big deal.

#10 lore raw
Quoted 41 lines · Marcelo Tosatti
Quoted 40 lines · Gleb Natapov
Quoted 39 lines · Marcelo Tosatti
Quoted 37 lines · Gleb Natapov
Quoted 22 lines · Marcelo Tosatti
Quoted 21 lines · Marcelo Tosatti
Quoted 20 lines · Gleb Natapov
Quoted 19 lines · Jan Kiszka
Quoted 8 lines · Gleb Natapov
Quoted 7 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

If the guest didn't take the last APIC timer interrupt yet and generates another one on top, e.g. via periodic mode, we do not block the VCPU even if the guest state is halted. The reason is that apic_has_pending_timer continues to return a non-zero value.

Quoted 6 lines
Quoted 3 lines

Fix this busy loop by taking the IRR content for the LVT vector in apic_has_pending_timer into account.

Just drop coalescing tacking for lapic interrupt. After posted interrupt will be merged __apic_accept_irq() will not longer return coalescing information, so the code will be dead anyway.

That requires the RTC decoalescing series to go first to avoid a regression, no? Then let's postpone this topic for now.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Yes, but decoalescing will work only for RTC :(

Are you proposing to drop LAPIC interrupt reinjection?

Since timer handling and injection is VCPU-local for LAPIC, __apic_accept_irq can (and must) return coalesced information (cannot drop LAPIC interrupt reinjection).

Why can't we drop LAPIC interrupt reinjection? Proposed posted interrupt patches do not properly check for interrupt coalescing even for VCPU-local injection.

--
Gleb.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Because older Linux guests depend on reinjection for proper timekeeping.

Which versions? Those without kvmclock? Can we make them use PIT instead? Posted interrupts going to break them.

There is no reason to break them if its OK to receive reinjection info from LAPIC... its a matter of returning the information from apic_accept_irq, no big deal.

But current PI patches do break them, thats my point. So we either need to revise them again, or drop LAPIC timer reinjection. Making apic_accept_irq semantics "it returns coalescing info, but only sometimes" is dubious though.

--
Gleb.

#11 lore raw

Gleb Natapov wrote on 2013-03-21:

Quoted 47 lines · Marcelo Tosatti
Quoted 46 lines · Gleb Natapov
Quoted 44 lines · Marcelo Tosatti
Quoted 33 lines · Gleb Natapov
Quoted 32 lines · Marcelo Tosatti
Quoted 31 lines · Marcelo Tosatti
Quoted 19 lines · Gleb Natapov
Quoted 18 lines · Jan Kiszka
Quoted 17 lines · Gleb Natapov
Quoted 15 lines · Jan Kiszka
Quoted 11 lines

From: Jan Kiszka <[email protected]>

If the guest didn't take the last APIC timer interrupt yet and generates another one on top, e.g. via periodic mode, we do not block the VCPU even if the guest state is halted. The reason is that apic_has_pending_timer continues to return a non-zero value.

Fix this busy loop by taking the IRR content for the LVT vector in apic_has_pending_timer into account.

Just drop coalescing tacking for lapic interrupt. After posted interrupt will be merged __apic_accept_irq() will not longer return coalescing information, so the code will be dead anyway.

Quoted 6 lines
Quoted 4 lines
Quoted 3 lines

That requires the RTC decoalescing series to go first to avoid a regression, no? Then let's postpone this topic for now.

Yes, but decoalescing will work only for RTC :(

Are you proposing to drop LAPIC interrupt reinjection?

Since timer handling and injection is VCPU-local for LAPIC, __apic_accept_irq can (and must) return coalesced information (cannot drop LAPIC interrupt reinjection).

Quoted 8 lines
Quoted 6 lines

Why can't we drop LAPIC interrupt reinjection? Proposed posted interrupt patches do not properly check for interrupt coalescing even for VCPU-local injection.

--
Gleb.

Because older Linux guests depend on reinjection for proper timekeeping.

Which versions? Those without kvmclock? Can we make them use PIT instead? Posted interrupts going to break them.

Quoted 8 lines
Quoted 4 lines

There is no reason to break them if its OK to receive reinjection info from LAPIC... its a matter of returning the information from apic_accept_irq, no big deal.

But current PI patches do break them, thats my point. So we either need to revise them again, or drop LAPIC timer reinjection. Making apic_accept_irq semantics "it returns coalescing info, but only sometimes" is dubious though.

We may rollback to the initial idea: test both irr and pir to get coalescing info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info. Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.

Best regards,
Yang

#12 lore raw
Quoted 52 lines · Yang Z

Gleb Natapov wrote on 2013-03-21:

Quoted 51 lines · Marcelo Tosatti
Quoted 50 lines · Gleb Natapov
Quoted 38 lines · Marcelo Tosatti
Quoted 37 lines · Gleb Natapov
Quoted 36 lines · Marcelo Tosatti
Quoted 23 lines · Marcelo Tosatti
Quoted 22 lines · Gleb Natapov
Quoted 21 lines · Jan Kiszka
Quoted 20 lines · Gleb Natapov
Quoted 9 lines · Jan Kiszka
Quoted 8 lines

From: Jan Kiszka <[email protected]>

If the guest didn't take the last APIC timer interrupt yet and generates another one on top, e.g. via periodic mode, we do not block the VCPU even if the guest state is halted. The reason is that apic_has_pending_timer continues to return a non-zero value.

Quoted 6 lines
Quoted 3 lines

Fix this busy loop by taking the IRR content for the LVT vector in apic_has_pending_timer into account.

Just drop coalescing tacking for lapic interrupt. After posted interrupt will be merged __apic_accept_irq() will not longer return coalescing information, so the code will be dead anyway.

That requires the RTC decoalescing series to go first to avoid a regression, no? Then let's postpone this topic for now.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Yes, but decoalescing will work only for RTC :(

Are you proposing to drop LAPIC interrupt reinjection?

Since timer handling and injection is VCPU-local for LAPIC, __apic_accept_irq can (and must) return coalesced information (cannot drop LAPIC interrupt reinjection).

Why can't we drop LAPIC interrupt reinjection? Proposed posted interrupt patches do not properly check for interrupt coalescing even for VCPU-local injection.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

--
Gleb.

Because older Linux guests depend on reinjection for proper timekeeping.

Which versions? Those without kvmclock? Can we make them use PIT instead? Posted interrupts going to break them.

There is no reason to break them if its OK to receive reinjection info from LAPIC... its a matter of returning the information from apic_accept_irq, no big deal.

Quoted 7 lines
Quoted 4 lines

But current PI patches do break them, thats my point. So we either need to revise them again, or drop LAPIC timer reinjection. Making apic_accept_irq semantics "it returns coalescing info, but only sometimes" is dubious though.

We may rollback to the initial idea: test both irr and pir to get coalescing info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info. Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.

We cannot touch irr while vcpu is in non-root operation, so we will have to pass flag to apic_accept_irq() to let it know that it is called synchronously. While all this is possible I want to know which guests exactly will we break if we will not track interrupt coalescing for lapic timer. If only 2.0 smp kernels will break we can probably drop it.

--
Gleb.

#13 lore raw
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

But current PI patches do break them, thats my point. So we either need to revise them again, or drop LAPIC timer reinjection. Making apic_accept_irq semantics "it returns coalescing info, but only sometimes" is dubious though.

Quoted 8 lines
Quoted 3 lines

We may rollback to the initial idea: test both irr and pir to get coalescing info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info. Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.

We cannot touch irr while vcpu is in non-root operation, so we will have to pass flag to apic_accept_irq() to let it know that it is called synchronously. While all this is possible I want to know which guests exactly will we break if we will not track interrupt coalescing for lapic timer. If only 2.0 smp kernels will break we can probably drop it.

RHEL4 / RHEL5 guests.

#14 lore raw
Quoted 7 lines · Marcelo Tosatti
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

But current PI patches do break them, thats my point. So we either need to revise them again, or drop LAPIC timer reinjection. Making apic_accept_irq semantics "it returns coalescing info, but only sometimes" is dubious though.

We may rollback to the initial idea: test both irr and pir to get coalescing info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info. Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.

Quoted 7 lines
Quoted 5 lines

We cannot touch irr while vcpu is in non-root operation, so we will have to pass flag to apic_accept_irq() to let it know that it is called synchronously. While all this is possible I want to know which guests exactly will we break if we will not track interrupt coalescing for lapic timer. If only 2.0 smp kernels will break we can probably drop it.

RHEL4 / RHEL5 guests.

RHEL5 has kvmclock no? We should not break RHEL4 though.

--
Gleb.

#15 lore raw
Quoted 8 lines · Gleb Natapov
Quoted 7 lines · Marcelo Tosatti
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

But current PI patches do break them, thats my point. So we either need to revise them again, or drop LAPIC timer reinjection. Making apic_accept_irq semantics "it returns coalescing info, but only sometimes" is dubious though.

We may rollback to the initial idea: test both irr and pir to get coalescing info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info. Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines

We cannot touch irr while vcpu is in non-root operation, so we will have to pass flag to apic_accept_irq() to let it know that it is called synchronously. While all this is possible I want to know which guests exactly will we break if we will not track interrupt coalescing for lapic timer. If only 2.0 smp kernels will break we can probably drop it.

RHEL4 / RHEL5 guests.

RHEL5 has kvmclock no? We should not break RHEL4 though.

kvmclock provides no timer interrupt... either LAPIC or PIT must be used with kvmclock.

#16 lore raw

Marcelo Tosatti wrote on 2013-03-22:

Quoted 19 lines · Gleb Natapov
Quoted 6 lines · Marcelo Tosatti
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

But current PI patches do break them, thats my point. So we either need to revise them again, or drop LAPIC timer reinjection. Making apic_accept_irq semantics "it returns coalescing info, but only sometimes" is dubious though.

We may rollback to the initial idea: test both irr and pir to get coalescing

info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines

Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.

We cannot touch irr while vcpu is in non-root operation, so we will have to pass flag to apic_accept_irq() to let it know that it is called synchronously. While all this is possible I want to know which guests exactly will we break if we will not track interrupt coalescing for lapic timer. If only 2.0 smp kernels will break we can probably drop it.

Quoted 5 lines
Quoted 2 lines
Quoted 1 line

RHEL4 / RHEL5 guests.

RHEL5 has kvmclock no? We should not break RHEL4 though.

kvmclock provides no timer interrupt... either LAPIC or PIT must be used with kvmclock.

Ok, Here is the conclusion:
-- According Marcelo's comments, RHEL4/RHEL5 rely on precise LAPIC timer injection. So LAPIC timer injection logic is necessary.
--LAPIC timer injection always occurred in vcpu context, so it's safe to touch irr and pir for LAPIC timer injection.
--We cannot touch virtual apic page while vcpu is in non-root operation, so the best solution is pass a flag to apic_accept_irq and check whether it's safe to touch vIRR according this flag.

Right?

Best regards,
Yang

#17 lore raw
In reply to Marcelo Tosatti
Quoted 15 lines · Marcelo Tosatti
Quoted 14 lines · Gleb Natapov
Quoted 13 lines · Marcelo Tosatti
Quoted 12 lines
Quoted 7 lines
Quoted 4 lines

But current PI patches do break them, thats my point. So we either need to revise them again, or drop LAPIC timer reinjection. Making apic_accept_irq semantics "it returns coalescing info, but only sometimes" is dubious though.

We may rollback to the initial idea: test both irr and pir to get coalescing info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info. Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.

We cannot touch irr while vcpu is in non-root operation, so we will have to pass flag to apic_accept_irq() to let it know that it is called synchronously. While all this is possible I want to know which guests exactly will we break if we will not track interrupt coalescing for lapic timer. If only 2.0 smp kernels will break we can probably drop it.

Quoted 5 lines
Quoted 2 lines
Quoted 1 line

RHEL4 / RHEL5 guests.

RHEL5 has kvmclock no? We should not break RHEL4 though.

kvmclock provides no timer interrupt... either LAPIC or PIT must be used with kvmclock.

I am confused now. If LAPIC is not used for wallclock time keeping, but only for scheduling the reinjection is actually harmful. Reinjecting the interrupt will cause needles task rescheduling. So the question is if there is a Linux kernel that uses LAPIC for wallclock time keeping and relies on accurate number of injected interrupts to not time drift. Knowing that Linux tend to disable interrupt it is likely that it tries to detect and compensate for missing interrupt.

--
Gleb.

#18 lore raw
Quoted 19 lines · Gleb Natapov
Quoted 18 lines · Marcelo Tosatti
Quoted 16 lines · Gleb Natapov
Quoted 7 lines · Marcelo Tosatti
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

But current PI patches do break them, thats my point. So we either need to revise them again, or drop LAPIC timer reinjection. Making apic_accept_irq semantics "it returns coalescing info, but only sometimes" is dubious though.

We may rollback to the initial idea: test both irr and pir to get coalescing info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info. Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.

Quoted 7 lines
Quoted 5 lines

We cannot touch irr while vcpu is in non-root operation, so we will have to pass flag to apic_accept_irq() to let it know that it is called synchronously. While all this is possible I want to know which guests exactly will we break if we will not track interrupt coalescing for lapic timer. If only 2.0 smp kernels will break we can probably drop it.

RHEL4 / RHEL5 guests.

RHEL5 has kvmclock no? We should not break RHEL4 though.

Quoted 7 lines
Quoted 2 lines

kvmclock provides no timer interrupt... either LAPIC or PIT must be used with kvmclock.

I am confused now. If LAPIC is not used for wallclock time keeping, but only for scheduling the reinjection is actually harmful. Reinjecting the interrupt will cause needles task rescheduling. So the question is if there is a Linux kernel that uses LAPIC for wallclock time keeping and relies on accurate number of injected interrupts to not time drift.

See 4acd47cfea9c18134e0cbf915780892ef0ff433a on RHEL5, RHEL5 kernels before that commit did not reinject. Which means that all non-RHEL Linux guests based on that upstream code also suffer from the same problem.

Also any other algorithm which uses LAPIC timers and compare that with other clocks (such as NMI watchdog) are potentially vulnerable.

Can drop it, and then wait until someone complains (if so).

Quoted 2 lines

Knowing that Linux tend to disable interrupt it is likely that it tries to detect and compensate for missing interrupt.

As said above, any algorithm which compares LAPIC timer interrupt with another clock is vulnerable.

#19 lore raw
Quoted 28 lines · Marcelo Tosatti
Quoted 19 lines · Gleb Natapov
Quoted 18 lines · Marcelo Tosatti
Quoted 16 lines · Gleb Natapov
Quoted 7 lines · Marcelo Tosatti
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines

But current PI patches do break them, thats my point. So we either need to revise them again, or drop LAPIC timer reinjection. Making apic_accept_irq semantics "it returns coalescing info, but only sometimes" is dubious though.

We may rollback to the initial idea: test both irr and pir to get coalescing info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info. Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.

Quoted 7 lines
Quoted 5 lines

We cannot touch irr while vcpu is in non-root operation, so we will have to pass flag to apic_accept_irq() to let it know that it is called synchronously. While all this is possible I want to know which guests exactly will we break if we will not track interrupt coalescing for lapic timer. If only 2.0 smp kernels will break we can probably drop it.

RHEL4 / RHEL5 guests.

RHEL5 has kvmclock no? We should not break RHEL4 though.

Quoted 7 lines
Quoted 2 lines

kvmclock provides no timer interrupt... either LAPIC or PIT must be used with kvmclock.

I am confused now. If LAPIC is not used for wallclock time keeping, but only for scheduling the reinjection is actually harmful. Reinjecting the interrupt will cause needles task rescheduling. So the question is if there is a Linux kernel that uses LAPIC for wallclock time keeping and relies on accurate number of injected interrupts to not time drift.

Quoted 4 lines

See 4acd47cfea9c18134e0cbf915780892ef0ff433a on RHEL5, RHEL5 kernels before that commit did not reinject. Which means that all non-RHEL Linux guests based on that upstream code also suffer from the same problem.

The commit actually fixes guest, not host. The existence of the commit also means that LAPIC timer reinjection does not solve the problem and all guests without this commit will suffer from the bug regardless of what we will decide to do here. Without LAPIC timer reinfection the effect of the bug will be much more visible and long lasting though.

Quoted 2 lines

Also any other algorithm which uses LAPIC timers and compare that with other clocks (such as NMI watchdog) are potentially vulnerable.

They are with or without timer reinjection as commit you pointed to shows.

Quoted 3 lines

Can drop it, and then wait until someone complains (if so).

Yes, tough decision to make. All the complains will be guest bugs which can be hit without reinjection too, but with less probability. Why we so keen on keeping RTC reinject is that the guests that depends on it cannot be fixed.

Quoted 5 lines
Quoted 2 lines

Knowing that Linux tend to disable interrupt it is likely that it tries to detect and compensate for missing interrupt.

As said above, any algorithm which compares LAPIC timer interrupt with another clock is vulnerable.

--
Gleb.

#20 lore raw

Gleb Natapov wrote on 2013-03-22:

Quoted 51 lines · Marcelo Tosatti
Quoted 9 lines · Gleb Natapov
Quoted 8 lines · Marcelo Tosatti
Quoted 7 lines · Gleb Natapov
Quoted 6 lines · Marcelo Tosatti
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

But current PI patches do break them, thats my point. So we either need to revise them again, or drop LAPIC timer reinjection. Making apic_accept_irq semantics "it returns coalescing info, but only sometimes" is dubious though.

We may rollback to the initial idea: test both irr and pir to get

coalescing info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info.

Quoted 22 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.

Quoted 8 lines
Quoted 6 lines

We cannot touch irr while vcpu is in non-root operation, so we will have to pass flag to apic_accept_irq() to let it know that it is called synchronously. While all this is possible I want to know which guests exactly will we break if we will not track interrupt coalescing for lapic timer. If only 2.0 smp kernels will break we can probably drop it.

RHEL4 / RHEL5 guests.

RHEL5 has kvmclock no? We should not break RHEL4 though.

Quoted 7 lines
Quoted 2 lines

kvmclock provides no timer interrupt... either LAPIC or PIT must be used with kvmclock.

I am confused now. If LAPIC is not used for wallclock time keeping, but only for scheduling the reinjection is actually harmful. Reinjecting the interrupt will cause needles task rescheduling. So the question is if there is a Linux kernel that uses LAPIC for wallclock time keeping and relies on accurate number of injected interrupts to not time drift.

Quoted 5 lines

See 4acd47cfea9c18134e0cbf915780892ef0ff433a on RHEL5, RHEL5 kernels before that commit did not reinject. Which means that all non-RHEL Linux guests based on that upstream code also suffer from the same problem.

The commit actually fixes guest, not host. The existence of the commit also means that LAPIC timer reinjection does not solve the problem and all guests without this commit will suffer from the bug regardless of what we will decide to do here. Without LAPIC timer reinfection the effect of the bug will be much more visible and long lasting though.

Quoted 2 lines

Also any other algorithm which uses LAPIC timers and compare that with other clocks (such as NMI watchdog) are potentially vulnerable.

They are with or without timer reinjection as commit you pointed to shows.

Quoted 1 line
Quoted 12 lines
Quoted 2 lines

Can drop it, and then wait until someone complains (if so).

Yes, tough decision to make. All the complains will be guest bugs which can be hit without reinjection too, but with less probability. Why we so keen on keeping RTC reinject is that the guests that depends on it cannot be fixed.

Quoted 5 lines
Quoted 2 lines

Knowing that Linux tend to disable interrupt it is likely that it tries to detect and compensate for missing interrupt.

As said above, any algorithm which compares LAPIC timer interrupt with another clock is vulnerable.

Any conclusion?

Best regards,
Yang

#21 lore raw
Quoted 59 lines · Yang Z

Gleb Natapov wrote on 2013-03-22:

Quoted 58 lines · Marcelo Tosatti
Quoted 9 lines · Gleb Natapov
Quoted 8 lines · Marcelo Tosatti
Quoted 7 lines · Gleb Natapov
Quoted 6 lines · Marcelo Tosatti
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

But current PI patches do break them, thats my point. So we either need to revise them again, or drop LAPIC timer reinjection. Making apic_accept_irq semantics "it returns coalescing info, but only sometimes" is dubious though.

We may rollback to the initial idea: test both irr and pir to get

coalescing info. In this case, inject LAPIC timer always in vcpu context. So apic_accept_irq() will return right coalescing info.

Quoted 22 lines
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

Also, we need to add comments to tell caller, apic_accept_irq() can ensure the return value is correct only when caller is in target vcpu context.

Quoted 8 lines
Quoted 6 lines

We cannot touch irr while vcpu is in non-root operation, so we will have to pass flag to apic_accept_irq() to let it know that it is called synchronously. While all this is possible I want to know which guests exactly will we break if we will not track interrupt coalescing for lapic timer. If only 2.0 smp kernels will break we can probably drop it.

RHEL4 / RHEL5 guests.

RHEL5 has kvmclock no? We should not break RHEL4 though.

Quoted 7 lines
Quoted 2 lines

kvmclock provides no timer interrupt... either LAPIC or PIT must be used with kvmclock.

I am confused now. If LAPIC is not used for wallclock time keeping, but only for scheduling the reinjection is actually harmful. Reinjecting the interrupt will cause needles task rescheduling. So the question is if there is a Linux kernel that uses LAPIC for wallclock time keeping and relies on accurate number of injected interrupts to not time drift.

Quoted 5 lines

See 4acd47cfea9c18134e0cbf915780892ef0ff433a on RHEL5, RHEL5 kernels before that commit did not reinject. Which means that all non-RHEL Linux guests based on that upstream code also suffer from the same problem.

The commit actually fixes guest, not host. The existence of the commit also means that LAPIC timer reinjection does not solve the problem and all guests without this commit will suffer from the bug regardless of what we will decide to do here. Without LAPIC timer reinfection the effect of the bug will be much more visible and long lasting though.

Quoted 2 lines

Also any other algorithm which uses LAPIC timers and compare that with other clocks (such as NMI watchdog) are potentially vulnerable.

They are with or without timer reinjection as commit you pointed to shows.

Quoted 3 lines

Can drop it, and then wait until someone complains (if so).

Yes, tough decision to make. All the complains will be guest bugs which can be hit without reinjection too, but with less probability. Why we so keen on keeping RTC reinject is that the guests that depends on it cannot be fixed.

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

Knowing that Linux tend to disable interrupt it is likely that it tries to detect and compensate for missing interrupt.

As said above, any algorithm which compares LAPIC timer interrupt with another clock is vulnerable.

Any conclusion?

Lets not check for coalescing in PI patches for now.

--
Gleb.

author
#22 lore raw
In reply to Gleb Natapov
Quoted 8 lines · Gleb Natapov
Quoted 7 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

If the guest didn't take the last APIC timer interrupt yet and generates another one on top, e.g. via periodic mode, we do not block the VCPU even if the guest state is halted. The reason is that apic_has_pending_timer continues to return a non-zero value.

Quoted 6 lines
Quoted 3 lines

Fix this busy loop by taking the IRR content for the LVT vector in apic_has_pending_timer into account.

Just drop coalescing tacking for lapic interrupt. After posted interrupt will be merged __apic_accept_irq() will not longer return coalescing information, so the code will be dead anyway.

If I understood the follow-up discussion correctly, we aren't dropping de-coalescing support yet. So how to proceed with this fix here?

Jan

Quoted 30 lines
Quoted 27 lines
Signed-off-by Jan Kiszka <[email protected]>

---

Not a critical issue, we are looping fully interruptible, but it's ugly to do so IMHO.

 arch/x86/kvm/lapic.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
arch/x86/kvm/lapic.c +3 -1
index a8e9369..658abf5 100644
@@ -1473,7 +1473,9 @@ int apic_has_pending_timer(struct kvm_vcpu *vcpu)
 	struct kvm_lapic *apic = vcpu->arch.apic;
 
 	if (kvm_vcpu_has_lapic(vcpu) && apic_enabled(apic) &&
-			apic_lvt_enabled(apic, APIC_LVTT))
+	    apic_lvt_enabled(apic, APIC_LVTT) &&
+	    !apic_test_vector(apic_lvt_vector(apic, APIC_LVTT),
+					      apic->regs + APIC_IRR))
 		return atomic_read(&apic->lapic_timer.pending);
 
 	return 0;
Signature · 1 line
1.7.3.4

--
Gleb.

#23 lore raw
Quoted 12 lines · Jan Kiszka
Quoted 11 lines · Gleb Natapov
Quoted 10 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

If the guest didn't take the last APIC timer interrupt yet and generates another one on top, e.g. via periodic mode, we do not block the VCPU even if the guest state is halted. The reason is that apic_has_pending_timer continues to return a non-zero value.

Fix this busy loop by taking the IRR content for the LVT vector in apic_has_pending_timer into account.

Quoted 7 lines
Quoted 3 lines

Just drop coalescing tacking for lapic interrupt. After posted interrupt will be merged __apic_accept_irq() will not longer return coalescing information, so the code will be dead anyway.

If I understood the follow-up discussion correctly, we aren't dropping de-coalescing support yet. So how to proceed with this fix here?

We do. It does not work if you run on CPU with apicv support already.

Quoted 34 lines

Jan

Quoted 30 lines
Quoted 27 lines
Signed-off-by Jan Kiszka <[email protected]>

---

Not a critical issue, we are looping fully interruptible, but it's ugly to do so IMHO.

 arch/x86/kvm/lapic.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
arch/x86/kvm/lapic.c +3 -1
index a8e9369..658abf5 100644
@@ -1473,7 +1473,9 @@ int apic_has_pending_timer(struct kvm_vcpu *vcpu)
 	struct kvm_lapic *apic = vcpu->arch.apic;
 
 	if (kvm_vcpu_has_lapic(vcpu) && apic_enabled(apic) &&
-			apic_lvt_enabled(apic, APIC_LVTT))
+	    apic_lvt_enabled(apic, APIC_LVTT) &&
+	    !apic_test_vector(apic_lvt_vector(apic, APIC_LVTT),
+					      apic->regs + APIC_IRR))
 		return atomic_read(&apic->lapic_timer.pending);
 
 	return 0;
Signature · 1 line
1.7.3.4

--
Gleb.

--
Gleb.

author
#24 lore raw
Quoted 13 lines · Gleb Natapov
Quoted 12 lines · Jan Kiszka
Quoted 11 lines · Gleb Natapov
Quoted 10 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

If the guest didn't take the last APIC timer interrupt yet and generates another one on top, e.g. via periodic mode, we do not block the VCPU even if the guest state is halted. The reason is that apic_has_pending_timer continues to return a non-zero value.

Fix this busy loop by taking the IRR content for the LVT vector in apic_has_pending_timer into account.

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

Just drop coalescing tacking for lapic interrupt. After posted interrupt will be merged __apic_accept_irq() will not longer return coalescing information, so the code will be dead anyway.

If I understood the follow-up discussion correctly, we aren't dropping de-coalescing support yet. So how to proceed with this fix here?

We do. It does not work if you run on CPU with apicv support already.

But isn't the code still there and working when apicv is absent?

Jan

#25 lore raw
Quoted 18 lines · Jan Kiszka
Quoted 17 lines · Gleb Natapov
Quoted 16 lines · Jan Kiszka
Quoted 14 lines · Gleb Natapov
Quoted 10 lines · Jan Kiszka

From: Jan Kiszka <[email protected]>

If the guest didn't take the last APIC timer interrupt yet and generates another one on top, e.g. via periodic mode, we do not block the VCPU even if the guest state is halted. The reason is that apic_has_pending_timer continues to return a non-zero value.

Fix this busy loop by taking the IRR content for the LVT vector in apic_has_pending_timer into account.

Just drop coalescing tacking for lapic interrupt. After posted interrupt will be merged __apic_accept_irq() will not longer return coalescing information, so the code will be dead anyway.

Quoted 7 lines
Quoted 4 lines
Quoted 3 lines

If I understood the follow-up discussion correctly, we aren't dropping de-coalescing support yet. So how to proceed with this fix here?

We do. It does not work if you run on CPU with apicv support already.

But isn't the code still there and working when apicv is absent?

Remove it as a fix for busy loop. It is not a good idea to behave differently on different types of hardware.

--
Gleb.