X86/vmx: Dump PIR and vIRR before ASSERT()
PATCH 1/1 X86/vmx: Dump PIR and vIRR before ASSERT()
Commit c7bdecae42 ("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).
The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.
--- xen/arch/x86/hvm/vmx/intr.c | 43 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+)
xen/arch/x86/hvm/vmx/intr.c +43 -0
@@ -23,6 +23,7 @@#include <xen/errno.h> #include <xen/trace.h> #include <xen/event.h> +#include <asm/apicdef.h> #include <asm/current.h> #include <asm/cpufeature.h> #include <asm/processor.h>
@@ -318,6 +319,48 @@ void vmx_intr_assist(void)*/ if ( pt_vector !=-1 ) { +/* +* We assert that intack.vector is the highest priority vector for +* only an interrupt from vlapic can reach this point and the +* highest vector is chosen in hvm_vcpu_has_pending_irq(). +* But, in fact, the assertion failed sometimes. It is suspected +* that PIR is not synced to vIRR which makes pt_vector is left in +* PIR. In order to verify this suspicion, dump some information +* when the assertion fails. +*/ +if (unlikely (intack.vector < pt_vector) ) + { +struct vlapic *vlapic; +struct pi_desc *pi_desc; +uint32_t *word; +int i; + +printk ("Assertion failed on %pv. Some info are below.\n" , + current); +printk ("intack info: vector 0x%2x, source %x\n" , intack.vector, + intack.source); +printk ("pt_vector info: vector 0x%2x\n" , pt_vector); + + vlapic =vcpu_vlapic (v); +if ( vlapic && vlapic->regs->data ) + { + word = (void *)&vlapic->regs->data[APIC_IRR ]; +printk ("vIRR:" ); +for ( i =NR_VECTORS /32 -1 ; i >=0 ; i-- ) +printk (" %08x" , word[i*4 ]); +printk ("\n" ); + } + + pi_desc = &v->arch.hvm_vmx.pi_desc; +if ( pi_desc && pi_desc->pir ) + { + word = (void *)&pi_desc->pir; +printk (" PIR:" ); +for ( i =NR_VECTORS /32 -1 ; i >=0 ; i-- ) +printk (" %08x" , word[i]); +printk ("\n" ); + } + }ASSERT (intack.vector >= pt_vector);vmx_set_eoi_exit_bitmap (v, intack.vector); }
Signature · 7 lines
1.8.3.1 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
Quoted 45 lines · Chao Gao
Commit
c7bdecae42("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.ht
ml).The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.
Signed-off-by Chao Gao <[email protected]>---
xen/arch/x86/hvm/vmx/intr.c| 43
+++++++++++++++++++++++++++++++++++++++++++1 file changed, 43 insertions(+)xen/arch/x86/hvm/vmx/intr.c +3
-0@@ -23,6 +23,7 @@#include <xen/errno.h> #include <xen/trace.h> #include <xen/event.h> +#include <asm/apicdef.h> #include <asm/current.h> #include <asm/cpufeature.h> #include <asm/processor.h> @@ -318,6 +319,48 @@ void vmx_intr_assist(void)*/ if ( pt_vector !=-1 ) { +/* +* We assert that intack.vector is the highest priority vector for
+ * only an interrupt from vlapic can reach this point and
thexen/arch/x86/hvm/vmx/intr.c +2
-0+* highest vector is chosen in hvm_vcpu_has_pending_irq(). +* But, in fact, the assertion failed sometimes. It is suspected
+ * that PIR is not synced to vIRR which makespt_vectoris
left in
+ * PIR. In order to verify this suspicion, dump some
information
xen/arch/x86/hvm/vmx/intr.c void vmx_intr_assist(void) · line 330
Quoted 2 lines
xen/arch/x86/hvm/vmx/intr.c +2
-0+* when the assertion fails. +*/
Some comment from QEMU/KVM code, in <linux-kernel>/arch/x86/kvm/lapic.c,
int kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
{
/* This may race with setting of irr in __apic_accept_irq() and
* value returned may be wrong, but kvm_vcpu_kick() in __apic_accept_irq
* will cause vmexit immediately and the value will be recalculated
* on the next vmentry.
*/
...
}
I am afraid, there may be a similar race with setting of vIRR..
Quan
Quoted 39 lines
xen/arch/x86/hvm/vmx/intr.c +8
-0+if (unlikely (intack.vector < pt_vector) ) + { +struct vlapic *vlapic; +struct pi_desc *pi_desc; +uint32_t *word; +int i; + + printk("Assertion failed on %pv. Some info arebelow.\n",
xen/arch/x86/hvm/vmx/intr.c +2
-0+ current); + printk("intack info: vector 0x%2x, source %x\n",intack.vector,
xen/arch/x86/hvm/vmx/intr.c +23
-1+ intack.source); +printk ("pt_vector info: vector 0x%2x\n" , pt_vector); + + vlapic =vcpu_vlapic (v); +if ( vlapic && vlapic->regs->data ) + { + word = (void *)&vlapic->regs->data[APIC_IRR ]; +printk ("vIRR:" ); +for ( i =NR_VECTORS /32 -1 ; i >=0 ; i-- ) +printk (" %08x" , word[i*4 ]); +printk ("\n" ); + } + + pi_desc = &v->arch.hvm_vmx.pi_desc; +if ( pi_desc && pi_desc->pir ) + { + word = (void *)&pi_desc->pir; +printk (" PIR:" ); +for ( i =NR_VECTORS /32 -1 ; i >=0 ; i-- ) +printk (" %08x" , word[i]); +printk ("\n" ); + } + }ASSERT (intack.vector >= pt_vector);vmx_set_eoi_exit_bitmap (v, intack.vector); } --
Quoted 1 line
1.8.3.1
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:48, <[email protected]> wrote:
Some comment from QEMU/KVM code, in <linux-kernel>/arch/x86/kvm/lapic.c,
int
kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
{
/* This may race with setting of irr in__apic_accept_irq()and
* value returned may be wrong, butkvm_vcpu_kick()in__apic_accept_irq
* will cause vmexit immediately and the value will be recalculated
* on the next vmentry.
*/
...
}
Quoted 1 line
I am afraid, there may be a similar race with setting of vIRR..
I think this is unrelated: If another interrupt came in, the highest set bit in vIRR can only increase. Plus pt_update_irq(), before returning, calls vlapic_set_irq(), which ought to result in pt_vector's vIRR bit to be set (either directly or via setting its PIR bit). I.e. the highest priority interrupt should still have a vector >= pt_vector.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 13 lines · Jan Beulich
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:48, <[email protected]> wrote:
Some comment from QEMU/KVM code, in <linux-kernel>/arch/x86/kvm/lapic.c,
int
kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
{
/* This may race with setting of irr in__apic_accept_irq()and
* value returned may be wrong, butkvm_vcpu_kick()in__apic_accept_irq
* will cause vmexit immediately and the value will be recalculated
* on the next vmentry.
*/
...
}
Quoted 7 lines
Quoted 1 line
I am afraid, there may be a similar race with setting of vIRR..
I think this is unrelated: If another interrupt came in, the highest set bit in vIRR can only increase. Plus
pt_update_irq(), before returning, callsvlapic_set_irq(), which ought to result in pt_vector's vIRR bit to be set (either directly or via setting its PIR bit). I.e. the highest priority interrupt should still have a vector >=pt_vector.
I have noticed that pt_vector was 0x38 and the hightest vector was 0x30 when the assertion failed. In the process of debugging pt_update_irq() when I was working on another feature, I noticed that 0x30 is always the vector of IRQ2. I suspect that the source of the periodic timer interrupt is not lapic. If that, pt_update_irq() reads the vioapic entry twice, one in hvm_isa_irq_assert() and the other in pt_irq_vector().
If guest changed the content in viopaic entry between the two read (ie. change the vector from 0x30 to 0x38), the assertion would fail. Do you think it is a reasonable explanation?
Thanks,
chao
Quoted 3 lines
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 23 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:32, <[email protected]> wrote:
Quoted 13 lines · Jan Beulich
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:48, <[email protected]> wrote:
Some comment from QEMU/KVM code, in <linux-kernel>/arch/x86/kvm/lapic.c,
int
kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
{
/* This may race with setting of irr in__apic_accept_irq()and
* value returned may be wrong, butkvm_vcpu_kick()in__apic_accept_irq
* will cause vmexit immediately and the value will be recalculated
* on the next vmentry.
*/
...
}Quoted 7 lines
Quoted 1 line
I am afraid, there may be a similar race with setting of vIRR..
I think this is unrelated: If another interrupt came in, the highest set bit in vIRR can only increase. Plus
pt_update_irq(), before returning, callsvlapic_set_irq(), which ought to result in pt_vector's vIRR bit to be set (either directly or via setting its PIR bit). I.e. the highest priority interrupt should still have a vector >=pt_vector.
Quoted 9 lines
I have noticed that
pt_vectorwas0x38and the hightest vector was0x30when the assertion failed. In the process of debuggingpt_update_irq()when I was working on another feature, I noticed that0x30is always the vector of IRQ2. I suspect that the source of the periodic timer interrupt is not lapic. If that,pt_update_irq()reads the vioapic entry twice, one inhvm_isa_irq_assert()and the other inpt_irq_vector().
If guest changed the content in viopaic entry between the two read (ie. change the vector from0x30to0x38), the assertion would fail. Do you think it is a reasonable explanation?
IRQ2? If this was going via the PIC, that would be impossible, as that's the cascade pin. IRQ0, otoh, normally arrives at pin2 of the IO-APIC (and we follow this model - see tools/libacpi/build.c). How did you determine that 0x30 is the vector of IRQ2?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 33 lines · Jan Beulich
Quoted 23 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:32, <[email protected]> wrote:
Quoted 13 lines · Jan Beulich
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:48, <[email protected]> wrote:
Some comment from QEMU/KVM code, in <linux-kernel>/arch/x86/kvm/lapic.c,
int
kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
{
/* This may race with setting of irr in__apic_accept_irq()and
* value returned may be wrong, butkvm_vcpu_kick()in__apic_accept_irq
* will cause vmexit immediately and the value will be recalculated
* on the next vmentry.
*/
...
}Quoted 7 lines
Quoted 1 line
I am afraid, there may be a similar race with setting of vIRR..
I think this is unrelated: If another interrupt came in, the highest set bit in vIRR can only increase. Plus
pt_update_irq(), before returning, callsvlapic_set_irq(), which ought to result in pt_vector's vIRR bit to be set (either directly or via setting its PIR bit). I.e. the highest priority interrupt should still have a vector >=pt_vector.Quoted 9 lines
I have noticed that
pt_vectorwas0x38and the hightest vector was0x30when the assertion failed. In the process of debuggingpt_update_irq()when I was working on another feature, I noticed that0x30is always the vector of IRQ2. I suspect that the source of the periodic timer interrupt is not lapic. If that,pt_update_irq()reads the vioapic entry twice, one inhvm_isa_irq_assert()and the other inpt_irq_vector().
If guest changed the content in viopaic entry between the two read (ie. change the vector from0x30to0x38), the assertion would fail. Do you think it is a reasonable explanation?
Quoted 3 lines
IRQ2? If this was going via the PIC, that would be impossible, as that's the cascade pin. IRQ0, otoh, normally arrives at pin2 of the IO-APIC (and we follow this model - see
tools/libacpi/build.c). How
Sorry, I meaned IRQ0. I was confused with IRQ0 and pin2 of the IO-APIC.
Quoted 1 line
did you determine that
0x30is the vector of IRQ2?
By monitoring the write operation to IO-APIC redirection entry. From experimental data, linux kernel always writes vector 0x30 to redirection entry of pin2 of IO-APIC.
Thank,
chao
Quoted 3 lines
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 40 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 08:28, <[email protected]> wrote:
Quoted 23 lines · Jan Beulich
Quoted 23 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:32, <[email protected]> wrote:
Quoted 20 lines · Jan Beulich
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:48, <[email protected]> wrote:
Some comment from QEMU/KVM code, in <linux-kernel>/arch/x86/kvm/lapic.c,
int
kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
{
/* This may race with setting of irr in__apic_accept_irq()and
* value returned may be wrong, butkvm_vcpu_kick()in__apic_accept_irq
* will cause vmexit immediately and the value will be recalculated
* on the next vmentry.
*/
...
}Quoted 1 line
I am afraid, there may be a similar race with setting of vIRR..
I think this is unrelated: If another interrupt came in, the highest set bit in vIRR can only increase. Plus
pt_update_irq(), before returning, callsvlapic_set_irq(), which ought to result in pt_vector's vIRR bit to be set (either directly or via setting its PIR bit). I.e. the highest priority interrupt should still have a vector >=pt_vector.Quoted 4 lines
Quoted 4 lines
I have noticed that
pt_vectorwas0x38and the hightest vector was0x30when the assertion failed. In the process of debuggingpt_update_irq()when I was working on another feature, I noticed that0x30is always the vector of IRQ2. I suspect that the source of the periodic timerinterrupt
Quoted 9 lines
Quoted 5 lines
is not lapic. If that,
pt_update_irq()reads the vioapic entry twice, one inhvm_isa_irq_assert()and the other inpt_irq_vector(). If guest changed the content in viopaic entry between the two read (ie. change the vector from0x30to0x38), the assertion would fail. Do you think it is a reasonable explanation?IRQ2? If this was going via the PIC, that would be impossible, as that's the cascade pin. IRQ0, otoh, normally arrives at pin2 of the IO-APIC (and we follow this model - see
tools/libacpi/build.c). How
Quoted 7 lines
Sorry, I meaned IRQ0. I was confused with IRQ0 and pin2 of the IO-APIC.
Quoted 1 line
did you determine that
0x30is the vector of IRQ2?By monitoring the write operation to IO-APIC redirection entry. From experimental data, linux kernel always writes vector
0x30to redirection entry of pin2 of IO-APIC.
But if it goes through the IO-APIC, it'll also go through the LAPIC (and in the ExtInt case you wouldn't be able to read a valid vector from the RTE).
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 44 lines · Jan Beulich
Quoted 44 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 08:28, <[email protected]> wrote:
Quoted 23 lines · Jan Beulich
Quoted 23 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:32, <[email protected]> wrote:
Quoted 20 lines · Jan Beulich
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:48, <[email protected]> wrote:
Some comment from QEMU/KVM code, in <linux-kernel>/arch/x86/kvm/lapic.c,
int
kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
{
/* This may race with setting of irr in__apic_accept_irq()and
* value returned may be wrong, butkvm_vcpu_kick()in__apic_accept_irq
* will cause vmexit immediately and the value will be recalculated
* on the next vmentry.
*/
...
}Quoted 1 line
I am afraid, there may be a similar race with setting of vIRR..
I think this is unrelated: If another interrupt came in, the highest set bit in vIRR can only increase. Plus
pt_update_irq(), before returning, callsvlapic_set_irq(), which ought to result in pt_vector's vIRR bit to be set (either directly or via setting its PIR bit). I.e. the highest priority interrupt should still have a vector >=pt_vector.Quoted 4 lines
Quoted 4 lines
I have noticed that
pt_vectorwas0x38and the hightest vector was0x30when the assertion failed. In the process of debuggingpt_update_irq()when I was working on another feature, I noticed that0x30is always the vector of IRQ2. I suspect that the source of the periodic timerinterrupt
Quoted 9 lines
Quoted 5 lines
is not lapic. If that,
pt_update_irq()reads the vioapic entry twice, one inhvm_isa_irq_assert()and the other inpt_irq_vector(). If guest changed the content in viopaic entry between the two read (ie. change the vector from0x30to0x38), the assertion would fail. Do you think it is a reasonable explanation?IRQ2? If this was going via the PIC, that would be impossible, as that's the cascade pin. IRQ0, otoh, normally arrives at pin2 of the IO-APIC (and we follow this model - see
tools/libacpi/build.c). HowSorry, I meaned IRQ0. I was confused with IRQ0 and pin2 of the IO-APIC.
Quoted 1 line
did you determine that
0x30is the vector of IRQ2?
Quoted 7 lines
Quoted 3 lines
By monitoring the write operation to IO-APIC redirection entry. From experimental data, linux kernel always writes vector
0x30to redirection entry of pin2 of IO-APIC.But if it goes through the IO-APIC, it'll also go through the LAPIC (and in the ExtInt case you wouldn't be able to read a valid vector from the RTE).
Yes. If a pt interrupt goes through IO-APIC, pt_update_irq() will read the RTE twice. One is in hvm_isa_irq_assert() to determine which bit in vIRR should be set. The other is in pt_irq_vector() to get the bit we have set and return it. Is it possible that the return vector is different from the bit set in vIRR of LAPIC, since guest has changed RTE of IO-APIC pin2.
thanks,
chao
Quoted 3 lines
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 57 lines
From: Gao, Chao
Sent: Wednesday, February 08, 2017 8:12 AMQuoted 6 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 08:28, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:32, <[email protected]> wrote:
Quoted 2 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:48, <[email protected]> wrote:
Some comment from QEMU/KVM code, in
<linux-kernel>/arch/x86/kvm/lapic.c,
Quoted 38 lines
Quoted 38 lines
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Quoted 11 lines
int
kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
{
/* This may race with setting of irr in__apic_accept_irq()and
* value returned may be wrong, butkvm_vcpu_kick()in__apic_accept_irq
* will cause vmexit immediately and the value will be recalculated
* on the next vmentry.
*/
...
}Quoted 1 line
I am afraid, there may be a similar race with setting of vIRR..
I think this is unrelated: If another interrupt came in, the highest set bit in vIRR can only increase. Plus
pt_update_irq(), before returning, callsvlapic_set_irq(), which ought to result in pt_vector's vIRR bit to be set (either directly or via setting its PIR bit). I.e. the highest priority interrupt should still have a vector >=pt_vector.Quoted 4 lines
Quoted 4 lines
I have noticed that
pt_vectorwas0x38and the hightest vector was0x30when the assertion failed. In the process of debuggingpt_update_irq()when I was working on another feature, I noticed that0x30is always the vector of IRQ2. I suspect that the source of the periodic timerinterrupt
Quoted 9 lines
Quoted 5 lines
is not lapic. If that,
pt_update_irq()reads the vioapic entry twice, one inhvm_isa_irq_assert()and the other inpt_irq_vector(). If guest changed the content in viopaic entry between the two read (ie. change the vector from0x30to0x38), the assertion would fail. Do you think it is a reasonable explanation?IRQ2? If this was going via the PIC, that would be impossible, as that's the cascade pin. IRQ0, otoh, normally arrives at pin2 of the IO-APIC (and we follow this model - see
tools/libacpi/build.c). HowSorry, I meaned IRQ0. I was confused with IRQ0 and pin2 of the IO-APIC.
Quoted 1 line
did you determine that
0x30is the vector of IRQ2?Quoted 7 lines
Quoted 3 lines
By monitoring the write operation to IO-APIC redirection entry. From experimental data, linux kernel always writes vector
0x30to redirection entry of pin2 of IO-APIC.But if it goes through the IO-APIC, it'll also go through the LAPIC (and in the ExtInt case you wouldn't be able to read a valid vector from the RTE).
Quoted 7 lines
Yes. If a pt interrupt goes through IO-APIC,
pt_update_irq()will read the RTE twice. One is inhvm_isa_irq_assert()to determine which bit in vIRR should be set. The other is inpt_irq_vector()to get the bit we have set and return it. Is it possible that the return vector is different from the bit set in vIRR of LAPIC, since guest has changed RTE of IO-APIC pin2.
In theory it could happen. There is no lock to prevent vIOAPIC RTE being modified between above two operations. But I'm not sure whether it's the reason for this bug. My feeling is that vector for IRQ0 should be fixed one. Did you capture such vector change in your side?
Thanks
Kevin
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 65 lines · Kevin
Quoted 57 lines
From: Gao, Chao
Sent: Wednesday, February 08, 2017 8:12 AMQuoted 6 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 08:28, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:32, <[email protected]> wrote:
Quoted 2 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:48, <[email protected]> wrote:
Some comment from QEMU/KVM code, in
<linux-kernel>/arch/x86/kvm/lapic.c,
Quoted 38 lines
Quoted 38 lines
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Quoted 11 lines
int
kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
{
/* This may race with setting of irr in__apic_accept_irq()and
* value returned may be wrong, butkvm_vcpu_kick()in__apic_accept_irq
* will cause vmexit immediately and the value will be recalculated
* on the next vmentry.
*/
...
}Quoted 1 line
I am afraid, there may be a similar race with setting of vIRR..
I think this is unrelated: If another interrupt came in, the highest set bit in vIRR can only increase. Plus
pt_update_irq(), before returning, callsvlapic_set_irq(), which ought to result in pt_vector's vIRR bit to be set (either directly or via setting its PIR bit). I.e. the highest priority interrupt should still have a vector >=pt_vector.Quoted 4 lines
Quoted 4 lines
I have noticed that
pt_vectorwas0x38and the hightest vector was0x30when the assertion failed. In the process of debuggingpt_update_irq()when I was working on another feature, I noticed that0x30is always the vector of IRQ2. I suspect that the source of the periodic timerinterrupt
Quoted 9 lines
Quoted 5 lines
is not lapic. If that,
pt_update_irq()reads the vioapic entry twice, one inhvm_isa_irq_assert()and the other inpt_irq_vector(). If guest changed the content in viopaic entry between the two read (ie. change the vector from0x30to0x38), the assertion would fail. Do you think it is a reasonable explanation?IRQ2? If this was going via the PIC, that would be impossible, as that's the cascade pin. IRQ0, otoh, normally arrives at pin2 of the IO-APIC (and we follow this model - see
tools/libacpi/build.c). HowSorry, I meaned IRQ0. I was confused with IRQ0 and pin2 of the IO-APIC.
Quoted 1 line
did you determine that
0x30is the vector of IRQ2?Quoted 7 lines
Quoted 3 lines
By monitoring the write operation to IO-APIC redirection entry. From experimental data, linux kernel always writes vector
0x30to redirection entry of pin2 of IO-APIC.But if it goes through the IO-APIC, it'll also go through the LAPIC (and in the ExtInt case you wouldn't be able to read a valid vector from the RTE).
Quoted 7 lines
Yes. If a pt interrupt goes through IO-APIC,
pt_update_irq()will read the RTE twice. One is inhvm_isa_irq_assert()to determine which bit in vIRR should be set. The other is inpt_irq_vector()to get the bit we have set and return it. Is it possible that the return vector is different from the bit set in vIRR of LAPIC, since guest has changed RTE of IO-APIC pin2.
Quoted 4 lines
In theory it could happen. There is no lock to prevent vIOAPIC RTE being modified between above two operations. But I'm not sure whether it's the reason for this bug. My feeling is that vector for IRQ0 should be fixed one. Did you capture such vector change in your side?
I agree. I don't see guest changes the vector for IRQ0 in hundreds of tests (simply create and shutdown guest).
Thanks
chao
Quoted 3 lines
Thanks
Kevin
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:
Commit
c7bdecae42("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).
Quoted 5 lines
The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.
Signed-off-by Chao Gao <[email protected]>
If this wasn't just a temporary thing, I'd have some comments, but it's good enough (perhaps with a little massaging when committing) for this purpose.
Ian, Wei, is there a way to automatically scan osstest logs for occurrences of the extra output, or does someone need to look through the logs of every spurious failure?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 17 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:
Commit
c7bdecae42("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).Quoted 5 lines
The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.
Signed-off-by Chao Gao <[email protected]>If this wasn't just a temporary thing, I'd have some comments, but it's good enough (perhaps with a little massaging when committing) for this purpose.
Quoted 6 lines
Reviewed-by Jan Beulich <[email protected]>Ian, Wei, is there a way to automatically scan osstest logs for occurrences of the extra output, or does someone need to look through the logs of every spurious failure?
I don't think there is automatically scanning in place.
Quoted 2 lines
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 21 lines
From: Wei Liu [mailto:[email protected]]
Sent: Tuesday, February 07, 2017 6:05 PMQuoted 17 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:
Commit
c7bdecae42("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).Quoted 5 lines
The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.
Signed-off-by Chao Gao <[email protected]>If this wasn't just a temporary thing, I'd have some comments, but it's good enough (perhaps with a little massaging when committing) for this purpose.
Quoted 8 lines
Quoted 6 lines
Reviewed-by Jan Beulich <[email protected]>Ian, Wei, is there a way to automatically scan osstest logs for occurrences of the extra output, or does someone need to look through the logs of every spurious failure?
I don't think there is automatically scanning in place.
Curious how this issue was initially caught? Would same practice make sure next fail catching our eye?
Thanks
Kevin
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 08.02.17 at 08:49, <[email protected]> wrote:
Curious how this issue was initially caught? Would same practice make sure next fail catching our eye?
I guess Andrew just happened to look at the logs of a spurious failure of some test.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:
Commit
c7bdecae42("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).
Quoted 5 lines
The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.
Signed-off-by Chao Gao <[email protected]>
Jun, Kevin - can you ack this? Or was Chao expected to make any changes? I'd like to see this go in rather sooner than later, so we can get it back out well before 4.9 is going to settle.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Tian, Kevin
Acked-byQuoted 14 lines
From: Jan Beulich [mailto:[email protected]]
Sent: Friday, February 10, 2017 5:36 PMQuoted 11 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:
Commit
c7bdecae42("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.
Quoted 6 lines
Quoted 1 line
Signed-off-by Chao Gao <[email protected]>Jun, Kevin - can you ack this? Or was Chao expected to make any changes? I'd like to see this go in rather sooner than later, so we can get it back out well before 4.9 is going to settle.
My bad... thought I've acked but seems distracted by later discussions.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:
Commit
c7bdecae42("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.
Julien, could you add a note on the 4.9 tracking list for us to not forget to revert this (or at least add NDEBUG conditionals) the latest in the RC phase?
Thanks, Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
Hi Jan,
Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:
Commit
c7bdecae42("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).
Quoted 7 lines
Quoted 3 lines
The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.
Julien, could you add a note on the 4.9 tracking list for us to not forget to revert this (or at least add NDEBUG conditionals) the latest in the RC phase?
I have added a note in the 4.9 tracking list.
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
(Resending the e-mail due to SMTP issue)
Quoted 27 lines · Julien Grall
Hi Jan,
Quoted 16 lines · Jan Beulich
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:
Commit
c7bdecae42("x86/apicv: fix RTC periodic timer and apicv issue") has
added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can
be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).Quoted 5 lines
The assertion failure is hard to reproduce. In order to root cause issue, this
patch is to add logs to dump PIR and vIRR when failure takes place. It should
be reverted once the root cause is found.Quoted 3 lines
Julien, could you add a note on the 4.9 tracking list for us to not forget to revert this (or at least add NDEBUG conditionals) the latest in the RC phase?
I have added a note in the 4.9 tracking list.
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
Hi,
Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:
Commit
c7bdecae42("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).
Quoted 7 lines
Quoted 3 lines
The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.
Julien, could you add a note on the 4.9 tracking list for us to not forget to revert this (or at least add NDEBUG conditionals) the latest in the RC phase?
Bumping this discussion as we are in the RC phase. Was the root cause found and we can revert the patch? Or shall we surround the code with NDEBUG?
Cheers,
Signature · 6 lines
Julien Grall _______________________________________________ Xen-devel mailing list [email protected] https://lists.xen.org/xen-devel
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 04.05.17 at 11:47, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:
Commit
c7bdecae42("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.
Quoted 6 lines
Quoted 3 lines
Julien, could you add a note on the 4.9 tracking list for us to not forget to revert this (or at least add NDEBUG conditionals) the latest in the RC phase?
Bumping this discussion as we are in the RC phase. Was the root cause found and we can revert the patch?
Sadly not, so far.
Quoted 1 line
Or shall we surround the code with NDEBUG?
Yes, I'll submit a patch.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel