Skip to contentSkip to conversation

X86/vmx: Dump PIR and vIRR before ASSERT()

PATCH In review Chao Gao 6 participants 21 messages Feb 7, 2017 – May 4, 2017

lore
Share & tools
lore

Chao Gao

author
#0 lore raw

PATCH 1/1 X86/vmx: Dump PIR and vIRR before ASSERT()

Commit c7bdecae42 ("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).

The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.

---
 xen/arch/x86/hvm/vmx/intr.c | 43 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
xen/arch/x86/hvm/vmx/intr.c +43 -0
index 24e4505..5e5b37a 100644
@@ -23,6 +23,7 @@
 #include <xen/errno.h>
 #include <xen/trace.h>
 #include <xen/event.h>
+#include <asm/apicdef.h>
 #include <asm/current.h>
 #include <asm/cpufeature.h>
 #include <asm/processor.h>
@@ -318,6 +319,48 @@ void vmx_intr_assist(void)
         */
         if ( pt_vector != -1 )
         {
+            /*
+             * We assert that intack.vector is the highest priority vector for
+             * only an interrupt from vlapic can reach this point and the
+             * highest vector is chosen in hvm_vcpu_has_pending_irq().
+             * But, in fact, the assertion failed sometimes. It is suspected
+             * that PIR is not synced to vIRR which makes pt_vector is left in
+             * PIR. In order to verify this suspicion, dump some information
+             * when the assertion fails.
+             */
+            if ( unlikely(intack.vector < pt_vector) )
+            {
+                struct vlapic *vlapic;
+                struct pi_desc *pi_desc;
+                uint32_t *word;
+                int i;
+
+                printk("Assertion failed on %pv. Some info are below.\n",
+                       current);
+                printk("intack info: vector 0x%2x, source %x\n", intack.vector,
+                       intack.source);
+                printk("pt_vector info: vector 0x%2x\n", pt_vector);
+
+                vlapic = vcpu_vlapic(v);
+                if ( vlapic && vlapic->regs->data )
+                {
+                    word = (void *)&vlapic->regs->data[APIC_IRR];
+                    printk("vIRR:");
+                    for ( i = NR_VECTORS / 32 - 1; i >= 0 ; i-- )
+                        printk(" %08x", word[i*4]);
+                    printk("\n");
+                }
+
+                pi_desc = &v->arch.hvm_vmx.pi_desc;
+                if ( pi_desc && pi_desc->pir )
+                {
+                    word = (void *)&pi_desc->pir;
+                    printk(" PIR:");
+                    for ( i = NR_VECTORS / 32 - 1; i >= 0 ; i-- )
+                        printk(" %08x", word[i]);
+                    printk("\n");
+                }
+            }
             ASSERT(intack.vector >= pt_vector);
             vmx_set_eoi_exit_bitmap(v, intack.vector);
         }
Signature · 7 lines
1.8.3.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
#1 lore raw
Quoted 45 lines · Chao Gao

Commit c7bdecae42 ("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.ht
ml).

The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.

---
xen/arch/x86/hvm/vmx/intr.c | 43
+++++++++++++++++++++++++++++++++++++++++++

1 file changed, 43 insertions(+)
xen/arch/x86/hvm/vmx/intr.c +3 -0
index 24e4505..5e5b37a 100644
@@ -23,6 +23,7 @@
 #include <xen/errno.h>
 #include <xen/trace.h>
 #include <xen/event.h>
+#include <asm/apicdef.h>
 #include <asm/current.h>
 #include <asm/cpufeature.h>
 #include <asm/processor.h>
@@ -318,6 +319,48 @@ void vmx_intr_assist(void)
        */
        if ( pt_vector != -1 )
        {
+            /*
+             * We assert that intack.vector is the highest priority vector

for
+ * only an interrupt from vlapic can reach this point and
the

xen/arch/x86/hvm/vmx/intr.c +2 -0
+             * highest vector is chosen in hvm_vcpu_has_pending_irq().
+             * But, in fact, the assertion failed sometimes. It is

suspected
+ * that PIR is not synced to vIRR which makes pt_vector is
left in
+ * PIR. In order to verify this suspicion, dump some
information

xen/arch/x86/hvm/vmx/intr.c void vmx_intr_assist(void) · line 330

Quoted 2 lines
xen/arch/x86/hvm/vmx/intr.c +2 -0
+             * when the assertion fails.
+             */

Some comment from QEMU/KVM code, in <linux-kernel>/arch/x86/kvm/lapic.c,

int kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
{
/* This may race with setting of irr in __apic_accept_irq() and
* value returned may be wrong, but kvm_vcpu_kick() in __apic_accept_irq
* will cause vmexit immediately and the value will be recalculated
* on the next vmentry.
*/
...
}

I am afraid, there may be a similar race with setting of vIRR..

Quan

Quoted 39 lines
xen/arch/x86/hvm/vmx/intr.c +8 -0
+            if ( unlikely(intack.vector < pt_vector) )
+            {
+                struct vlapic *vlapic;
+                struct pi_desc *pi_desc;
+                uint32_t *word;
+                int i;
+
+                printk("Assertion failed on %pv. Some info are

below.\n",

xen/arch/x86/hvm/vmx/intr.c +2 -0
+                       current);
+                printk("intack info: vector 0x%2x, source %x\n",

intack.vector,

xen/arch/x86/hvm/vmx/intr.c +23 -1
+                       intack.source);
+                printk("pt_vector info: vector 0x%2x\n", pt_vector);
+
+                vlapic = vcpu_vlapic(v);
+                if ( vlapic && vlapic->regs->data )
+                {
+                    word = (void *)&vlapic->regs->data[APIC_IRR];
+                    printk("vIRR:");
+                    for ( i = NR_VECTORS / 32 - 1; i >= 0 ; i-- )
+                        printk(" %08x", word[i*4]);
+                    printk("\n");
+                }
+
+                pi_desc = &v->arch.hvm_vmx.pi_desc;
+                if ( pi_desc && pi_desc->pir )
+                {
+                    word = (void *)&pi_desc->pir;
+                    printk(" PIR:");
+                    for ( i = NR_VECTORS / 32 - 1; i >= 0 ; i-- )
+                        printk(" %08x", word[i]);
+                    printk("\n");
+                }
+            }
            ASSERT(intack.vector >= pt_vector);
            vmx_set_eoi_exit_bitmap(v, intack.vector);
        }
--
Quoted 1 line

1.8.3.1

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

#2 lore raw
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:48, <[email protected]> wrote:

Some comment from QEMU/KVM code, in <linux-kernel>/arch/x86/kvm/lapic.c,

int kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
{
/* This may race with setting of irr in __apic_accept_irq() and
* value returned may be wrong, but kvm_vcpu_kick() in __apic_accept_irq
* will cause vmexit immediately and the value will be recalculated
* on the next vmentry.
*/
...
}

Quoted 1 line

I am afraid, there may be a similar race with setting of vIRR..

I think this is unrelated: If another interrupt came in, the highest set bit in vIRR can only increase. Plus pt_update_irq(), before returning, calls vlapic_set_irq(), which ought to result in pt_vector's vIRR bit to be set (either directly or via setting its PIR bit). I.e. the highest priority interrupt should still have a vector >= pt_vector.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

author
#3 lore raw
Quoted 13 lines · Jan Beulich
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:48, <[email protected]> wrote:

Some comment from QEMU/KVM code, in <linux-kernel>/arch/x86/kvm/lapic.c,

int kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
{
/* This may race with setting of irr in __apic_accept_irq() and
* value returned may be wrong, but kvm_vcpu_kick() in __apic_accept_irq
* will cause vmexit immediately and the value will be recalculated
* on the next vmentry.
*/
...
}

Quoted 7 lines
Quoted 1 line

I am afraid, there may be a similar race with setting of vIRR..

I think this is unrelated: If another interrupt came in, the highest set bit in vIRR can only increase. Plus pt_update_irq(), before returning, calls vlapic_set_irq(), which ought to result in pt_vector's vIRR bit to be set (either directly or via setting its PIR bit). I.e. the highest priority interrupt should still have a vector >= pt_vector.

I have noticed that pt_vector was 0x38 and the hightest vector was 0x30 when the assertion failed. In the process of debugging pt_update_irq() when I was working on another feature, I noticed that 0x30 is always the vector of IRQ2. I suspect that the source of the periodic timer interrupt is not lapic. If that, pt_update_irq() reads the vioapic entry twice, one in hvm_isa_irq_assert() and the other in pt_irq_vector().
If guest changed the content in viopaic entry between the two read (ie. change the vector from 0x30 to 0x38), the assertion would fail. Do you think it is a reasonable explanation?

Thanks,
chao

Quoted 3 lines

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

#4 lore raw
Quoted 23 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:32, <[email protected]> wrote:
Quoted 13 lines · Jan Beulich
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:48, <[email protected]> wrote:

Some comment from QEMU/KVM code, in <linux-kernel>/arch/x86/kvm/lapic.c,

int kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
{
/* This may race with setting of irr in __apic_accept_irq() and
* value returned may be wrong, but kvm_vcpu_kick() in __apic_accept_irq
* will cause vmexit immediately and the value will be recalculated
* on the next vmentry.
*/
...
}

Quoted 7 lines
Quoted 1 line

I am afraid, there may be a similar race with setting of vIRR..

I think this is unrelated: If another interrupt came in, the highest set bit in vIRR can only increase. Plus pt_update_irq(), before returning, calls vlapic_set_irq(), which ought to result in pt_vector's vIRR bit to be set (either directly or via setting its PIR bit). I.e. the highest priority interrupt should still have a vector >= pt_vector.

Quoted 9 lines

I have noticed that pt_vector was 0x38 and the hightest vector was 0x30 when the assertion failed. In the process of debugging pt_update_irq() when I was working on another feature, I noticed that 0x30 is always the vector of IRQ2. I suspect that the source of the periodic timer interrupt is not lapic. If that, pt_update_irq() reads the vioapic entry twice, one in hvm_isa_irq_assert() and the other in pt_irq_vector().
If guest changed the content in viopaic entry between the two read (ie. change the vector from 0x30 to 0x38), the assertion would fail. Do you think it is a reasonable explanation?

IRQ2? If this was going via the PIC, that would be impossible, as that's the cascade pin. IRQ0, otoh, normally arrives at pin2 of the IO-APIC (and we follow this model - see tools/libacpi/build.c). How did you determine that 0x30 is the vector of IRQ2?

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

author
#5 lore raw
Quoted 33 lines · Jan Beulich
Quoted 23 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:32, <[email protected]> wrote:
Quoted 13 lines · Jan Beulich
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:48, <[email protected]> wrote:

Some comment from QEMU/KVM code, in <linux-kernel>/arch/x86/kvm/lapic.c,

int kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
{
/* This may race with setting of irr in __apic_accept_irq() and
* value returned may be wrong, but kvm_vcpu_kick() in __apic_accept_irq
* will cause vmexit immediately and the value will be recalculated
* on the next vmentry.
*/
...
}

Quoted 7 lines
Quoted 1 line

I am afraid, there may be a similar race with setting of vIRR..

I think this is unrelated: If another interrupt came in, the highest set bit in vIRR can only increase. Plus pt_update_irq(), before returning, calls vlapic_set_irq(), which ought to result in pt_vector's vIRR bit to be set (either directly or via setting its PIR bit). I.e. the highest priority interrupt should still have a vector >= pt_vector.

Quoted 9 lines

I have noticed that pt_vector was 0x38 and the hightest vector was 0x30 when the assertion failed. In the process of debugging pt_update_irq() when I was working on another feature, I noticed that 0x30 is always the vector of IRQ2. I suspect that the source of the periodic timer interrupt is not lapic. If that, pt_update_irq() reads the vioapic entry twice, one in hvm_isa_irq_assert() and the other in pt_irq_vector().
If guest changed the content in viopaic entry between the two read (ie. change the vector from 0x30 to 0x38), the assertion would fail. Do you think it is a reasonable explanation?

Quoted 3 lines

IRQ2? If this was going via the PIC, that would be impossible, as that's the cascade pin. IRQ0, otoh, normally arrives at pin2 of the IO-APIC (and we follow this model - see tools/libacpi/build.c). How

Sorry, I meaned IRQ0. I was confused with IRQ0 and pin2 of the IO-APIC.

Quoted 1 line

did you determine that 0x30 is the vector of IRQ2?

By monitoring the write operation to IO-APIC redirection entry. From experimental data, linux kernel always writes vector 0x30 to redirection entry of pin2 of IO-APIC.

Thank,
chao

Quoted 3 lines

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

#6 lore raw
Quoted 40 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 08:28, <[email protected]> wrote:
Quoted 23 lines · Jan Beulich
Quoted 23 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:32, <[email protected]> wrote:
Quoted 20 lines · Jan Beulich
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:48, <[email protected]> wrote:

Some comment from QEMU/KVM code, in <linux-kernel>/arch/x86/kvm/lapic.c,

int kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
{
/* This may race with setting of irr in __apic_accept_irq() and
* value returned may be wrong, but kvm_vcpu_kick() in __apic_accept_irq
* will cause vmexit immediately and the value will be recalculated
* on the next vmentry.
*/
...
}

Quoted 1 line

I am afraid, there may be a similar race with setting of vIRR..

I think this is unrelated: If another interrupt came in, the highest set bit in vIRR can only increase. Plus pt_update_irq(), before returning, calls vlapic_set_irq(), which ought to result in pt_vector's vIRR bit to be set (either directly or via setting its PIR bit). I.e. the highest priority interrupt should still have a vector >= pt_vector.

Quoted 4 lines
Quoted 4 lines

I have noticed that pt_vector was 0x38 and the hightest vector was 0x30 when the assertion failed. In the process of debugging pt_update_irq() when I was working on another feature, I noticed that 0x30 is always the vector of IRQ2. I suspect that the source of the periodic timer

interrupt

Quoted 9 lines
Quoted 5 lines

is not lapic. If that, pt_update_irq() reads the vioapic entry twice, one in hvm_isa_irq_assert() and the other in pt_irq_vector(). If guest changed the content in viopaic entry between the two read (ie. change the vector from 0x30 to 0x38), the assertion would fail. Do you think it is a reasonable explanation?

IRQ2? If this was going via the PIC, that would be impossible, as that's the cascade pin. IRQ0, otoh, normally arrives at pin2 of the IO-APIC (and we follow this model - see tools/libacpi/build.c). How

Quoted 7 lines

Sorry, I meaned IRQ0. I was confused with IRQ0 and pin2 of the IO-APIC.

Quoted 1 line

did you determine that 0x30 is the vector of IRQ2?

By monitoring the write operation to IO-APIC redirection entry. From experimental data, linux kernel always writes vector 0x30 to redirection entry of pin2 of IO-APIC.

But if it goes through the IO-APIC, it'll also go through the LAPIC (and in the ExtInt case you wouldn't be able to read a valid vector from the RTE).

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

author
#7 lore raw
Quoted 44 lines · Jan Beulich
Quoted 44 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 08:28, <[email protected]> wrote:
Quoted 23 lines · Jan Beulich
Quoted 23 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:32, <[email protected]> wrote:
Quoted 20 lines · Jan Beulich
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:48, <[email protected]> wrote:

Some comment from QEMU/KVM code, in <linux-kernel>/arch/x86/kvm/lapic.c,

int kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
{
/* This may race with setting of irr in __apic_accept_irq() and
* value returned may be wrong, but kvm_vcpu_kick() in __apic_accept_irq
* will cause vmexit immediately and the value will be recalculated
* on the next vmentry.
*/
...
}

Quoted 1 line

I am afraid, there may be a similar race with setting of vIRR..

I think this is unrelated: If another interrupt came in, the highest set bit in vIRR can only increase. Plus pt_update_irq(), before returning, calls vlapic_set_irq(), which ought to result in pt_vector's vIRR bit to be set (either directly or via setting its PIR bit). I.e. the highest priority interrupt should still have a vector >= pt_vector.

Quoted 4 lines
Quoted 4 lines

I have noticed that pt_vector was 0x38 and the hightest vector was 0x30 when the assertion failed. In the process of debugging pt_update_irq() when I was working on another feature, I noticed that 0x30 is always the vector of IRQ2. I suspect that the source of the periodic timer

interrupt

Quoted 9 lines
Quoted 5 lines

is not lapic. If that, pt_update_irq() reads the vioapic entry twice, one in hvm_isa_irq_assert() and the other in pt_irq_vector(). If guest changed the content in viopaic entry between the two read (ie. change the vector from 0x30 to 0x38), the assertion would fail. Do you think it is a reasonable explanation?

IRQ2? If this was going via the PIC, that would be impossible, as that's the cascade pin. IRQ0, otoh, normally arrives at pin2 of the IO-APIC (and we follow this model - see tools/libacpi/build.c). How

Sorry, I meaned IRQ0. I was confused with IRQ0 and pin2 of the IO-APIC.

Quoted 1 line

did you determine that 0x30 is the vector of IRQ2?

Quoted 7 lines
Quoted 3 lines

By monitoring the write operation to IO-APIC redirection entry. From experimental data, linux kernel always writes vector 0x30 to redirection entry of pin2 of IO-APIC.

But if it goes through the IO-APIC, it'll also go through the LAPIC (and in the ExtInt case you wouldn't be able to read a valid vector from the RTE).

Yes. If a pt interrupt goes through IO-APIC, pt_update_irq() will read the RTE twice. One is in hvm_isa_irq_assert() to determine which bit in vIRR should be set. The other is in pt_irq_vector() to get the bit we have set and return it. Is it possible that the return vector is different from the bit set in vIRR of LAPIC, since guest has changed RTE of IO-APIC pin2.

thanks,
chao

Quoted 3 lines

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

#8 lore raw
Quoted 57 lines

From: Gao, Chao
Sent: Wednesday, February 08, 2017 8:12 AM

Quoted 6 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 08:28, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:32, <[email protected]> wrote:
Quoted 2 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:48, <[email protected]> wrote:

Some comment from QEMU/KVM code, in

<linux-kernel>/arch/x86/kvm/lapic.c,

Quoted 38 lines
Quoted 38 lines
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Quoted 11 lines

int kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
{
/* This may race with setting of irr in __apic_accept_irq() and
* value returned may be wrong, but kvm_vcpu_kick() in __apic_accept_irq
* will cause vmexit immediately and the value will be recalculated
* on the next vmentry.
*/
...
}

Quoted 1 line

I am afraid, there may be a similar race with setting of vIRR..

I think this is unrelated: If another interrupt came in, the highest set bit in vIRR can only increase. Plus pt_update_irq(), before returning, calls vlapic_set_irq(), which ought to result in pt_vector's vIRR bit to be set (either directly or via setting its PIR bit). I.e. the highest priority interrupt should still have a vector >= pt_vector.

Quoted 4 lines
Quoted 4 lines

I have noticed that pt_vector was 0x38 and the hightest vector was 0x30 when the assertion failed. In the process of debugging pt_update_irq() when I was working on another feature, I noticed that 0x30 is always the vector of IRQ2. I suspect that the source of the periodic timer

interrupt

Quoted 9 lines
Quoted 5 lines

is not lapic. If that, pt_update_irq() reads the vioapic entry twice, one in hvm_isa_irq_assert() and the other in pt_irq_vector(). If guest changed the content in viopaic entry between the two read (ie. change the vector from 0x30 to 0x38), the assertion would fail. Do you think it is a reasonable explanation?

IRQ2? If this was going via the PIC, that would be impossible, as that's the cascade pin. IRQ0, otoh, normally arrives at pin2 of the IO-APIC (and we follow this model - see tools/libacpi/build.c). How

Sorry, I meaned IRQ0. I was confused with IRQ0 and pin2 of the IO-APIC.

Quoted 1 line

did you determine that 0x30 is the vector of IRQ2?

Quoted 7 lines
Quoted 3 lines

By monitoring the write operation to IO-APIC redirection entry. From experimental data, linux kernel always writes vector 0x30 to redirection entry of pin2 of IO-APIC.

But if it goes through the IO-APIC, it'll also go through the LAPIC (and in the ExtInt case you wouldn't be able to read a valid vector from the RTE).

Quoted 7 lines

Yes. If a pt interrupt goes through IO-APIC, pt_update_irq() will read the RTE twice. One is in hvm_isa_irq_assert() to determine which bit in vIRR should be set. The other is in pt_irq_vector() to get the bit we have set and return it. Is it possible that the return vector is different from the bit set in vIRR of LAPIC, since guest has changed RTE of IO-APIC pin2.

In theory it could happen. There is no lock to prevent vIOAPIC RTE being modified between above two operations. But I'm not sure whether it's the reason for this bug. My feeling is that vector for IRQ0 should be fixed one. Did you capture such vector change in your side?

Thanks
Kevin

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

author
#9 lore raw
Quoted 65 lines · Kevin
Quoted 57 lines

From: Gao, Chao
Sent: Wednesday, February 08, 2017 8:12 AM

Quoted 6 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 08:28, <[email protected]> wrote:
Quoted 4 lines · Jan Beulich
Quoted 4 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:32, <[email protected]> wrote:
Quoted 2 lines · Jan Beulich
Quoted 2 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 07:48, <[email protected]> wrote:

Some comment from QEMU/KVM code, in

<linux-kernel>/arch/x86/kvm/lapic.c,

Quoted 38 lines
Quoted 38 lines
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines
Quoted 11 lines

int kvm_lapic_find_highest_irr(struct kvm_vcpu *vcpu)
{
/* This may race with setting of irr in __apic_accept_irq() and
* value returned may be wrong, but kvm_vcpu_kick() in __apic_accept_irq
* will cause vmexit immediately and the value will be recalculated
* on the next vmentry.
*/
...
}

Quoted 1 line

I am afraid, there may be a similar race with setting of vIRR..

I think this is unrelated: If another interrupt came in, the highest set bit in vIRR can only increase. Plus pt_update_irq(), before returning, calls vlapic_set_irq(), which ought to result in pt_vector's vIRR bit to be set (either directly or via setting its PIR bit). I.e. the highest priority interrupt should still have a vector >= pt_vector.

Quoted 4 lines
Quoted 4 lines

I have noticed that pt_vector was 0x38 and the hightest vector was 0x30 when the assertion failed. In the process of debugging pt_update_irq() when I was working on another feature, I noticed that 0x30 is always the vector of IRQ2. I suspect that the source of the periodic timer

interrupt

Quoted 9 lines
Quoted 5 lines

is not lapic. If that, pt_update_irq() reads the vioapic entry twice, one in hvm_isa_irq_assert() and the other in pt_irq_vector(). If guest changed the content in viopaic entry between the two read (ie. change the vector from 0x30 to 0x38), the assertion would fail. Do you think it is a reasonable explanation?

IRQ2? If this was going via the PIC, that would be impossible, as that's the cascade pin. IRQ0, otoh, normally arrives at pin2 of the IO-APIC (and we follow this model - see tools/libacpi/build.c). How

Sorry, I meaned IRQ0. I was confused with IRQ0 and pin2 of the IO-APIC.

Quoted 1 line

did you determine that 0x30 is the vector of IRQ2?

Quoted 7 lines
Quoted 3 lines

By monitoring the write operation to IO-APIC redirection entry. From experimental data, linux kernel always writes vector 0x30 to redirection entry of pin2 of IO-APIC.

But if it goes through the IO-APIC, it'll also go through the LAPIC (and in the ExtInt case you wouldn't be able to read a valid vector from the RTE).

Quoted 7 lines

Yes. If a pt interrupt goes through IO-APIC, pt_update_irq() will read the RTE twice. One is in hvm_isa_irq_assert() to determine which bit in vIRR should be set. The other is in pt_irq_vector() to get the bit we have set and return it. Is it possible that the return vector is different from the bit set in vIRR of LAPIC, since guest has changed RTE of IO-APIC pin2.

Quoted 4 lines

In theory it could happen. There is no lock to prevent vIOAPIC RTE being modified between above two operations. But I'm not sure whether it's the reason for this bug. My feeling is that vector for IRQ0 should be fixed one. Did you capture such vector change in your side?

I agree. I don't see guest changes the vector for IRQ0 in hundreds of tests (simply create and shutdown guest).

Thanks
chao

Quoted 3 lines

Thanks
Kevin

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Reviewed-by on v1 1/1
#10 lore raw
In reply to Chao Gao
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:

Commit c7bdecae42 ("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).

Quoted 5 lines

The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.

If this wasn't just a temporary thing, I'd have some comments, but it's good enough (perhaps with a little massaging when committing) for this purpose.

Ian, Wei, is there a way to automatically scan osstest logs for occurrences of the extra output, or does someone need to look through the logs of every spurious failure?

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

#11 lore raw
Quoted 17 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:

Commit c7bdecae42 ("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).

Quoted 5 lines

The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.

If this wasn't just a temporary thing, I'd have some comments, but it's good enough (perhaps with a little massaging when committing) for this purpose.

Quoted 6 lines

Ian, Wei, is there a way to automatically scan osstest logs for occurrences of the extra output, or does someone need to look through the logs of every spurious failure?

I don't think there is automatically scanning in place.

Quoted 2 lines

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

#12 lore raw
Quoted 21 lines

From: Wei Liu [mailto:[email protected]]
Sent: Tuesday, February 07, 2017 6:05 PM

Quoted 17 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:

Commit c7bdecae42 ("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).

Quoted 5 lines

The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.

If this wasn't just a temporary thing, I'd have some comments, but it's good enough (perhaps with a little massaging when committing) for this purpose.

Quoted 8 lines
Quoted 6 lines

Ian, Wei, is there a way to automatically scan osstest logs for occurrences of the extra output, or does someone need to look through the logs of every spurious failure?

I don't think there is automatically scanning in place.

Curious how this issue was initially caught? Would same practice make sure next fail catching our eye?

Thanks
Kevin

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

#13 lore raw
Quoted 3 lines
Quoted 1 line
Quoted 1 line · On 08.02.17 at 08:49, <[email protected]> wrote:

Curious how this issue was initially caught? Would same practice make sure next fail catching our eye?

I guess Andrew just happened to look at the logs of a spurious failure of some test.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

#14 lore raw
In reply to Chao Gao
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:

Commit c7bdecae42 ("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).

Quoted 5 lines

The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.

Jun, Kevin - can you ack this? Or was Chao expected to make any changes? I'd like to see this go in rather sooner than later, so we can get it back out well before 4.9 is going to settle.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

Acked-by
#15 lore raw
Quoted 14 lines

From: Jan Beulich [mailto:[email protected]]
Sent: Friday, February 10, 2017 5:36 PM

Quoted 11 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:

Commit c7bdecae42 ("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).

The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.

Quoted 6 lines
Quoted 1 line

Jun, Kevin - can you ack this? Or was Chao expected to make any changes? I'd like to see this go in rather sooner than later, so we can get it back out well before 4.9 is going to settle.

My bad... thought I've acked but seems distracted by later discussions.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

#16 lore raw
In reply to Chao Gao
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:

Commit c7bdecae42 ("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).

The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.

Julien, could you add a note on the 4.9 tracking list for us to not forget to revert this (or at least add NDEBUG conditionals) the latest in the RC phase?

Thanks, Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel

#17 lore raw

Hi Jan,

Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:

Commit c7bdecae42 ("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).

Quoted 7 lines
Quoted 3 lines

The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.

Julien, could you add a note on the 4.9 tracking list for us to not forget to revert this (or at least add NDEBUG conditionals) the latest in the RC phase?

I have added a note in the 4.9 tracking list.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
#18 lore raw

(Resending the e-mail due to SMTP issue)

Quoted 27 lines · Julien Grall

Hi Jan,

Quoted 16 lines · Jan Beulich
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:

Commit c7bdecae42 ("x86/apicv: fix RTC periodic timer and apicv issue") has
added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can
be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).

Quoted 5 lines

The assertion failure is hard to reproduce. In order to root cause issue, this
patch is to add logs to dump PIR and vIRR when failure takes place. It should
be reverted once the root cause is found.

Quoted 3 lines

Julien, could you add a note on the 4.9 tracking list for us to not forget to revert this (or at least add NDEBUG conditionals) the latest in the RC phase?

I have added a note in the 4.9 tracking list.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
#19 lore raw
In reply to Jan Beulich

Hi,

Quoted 7 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:

Commit c7bdecae42 ("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).

Quoted 7 lines
Quoted 3 lines

The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.

Julien, could you add a note on the 4.9 tracking list for us to not forget to revert this (or at least add NDEBUG conditionals) the latest in the RC phase?

Bumping this discussion as we are in the RC phase. Was the root cause found and we can revert the patch? Or shall we surround the code with NDEBUG?

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel
#20 lore raw
Quoted 13 lines
Quoted 1 line
Quoted 1 line · On 04.05.17 at 11:47, <[email protected]> wrote:
Quoted 11 lines · Jan Beulich
Quoted 10 lines
Quoted 1 line
Quoted 1 line · On 07.02.17 at 00:32, <[email protected]> wrote:

Commit c7bdecae42 ("x86/apicv: fix RTC periodic timer and apicv issue") has added a assertion that intack.vector is the highest priority vector. But according to the osstest, the assertion failed sometimes. More discussion can be found in the thread
(https://lists.xenproject.org/archives/html/xen-devel/2017-01/msg01019.html).

The assertion failure is hard to reproduce. In order to root cause issue, this patch is to add logs to dump PIR and vIRR when failure takes place. It should be reverted once the root cause is found.

Quoted 6 lines
Quoted 3 lines

Julien, could you add a note on the 4.9 tracking list for us to not forget to revert this (or at least add NDEBUG conditionals) the latest in the RC phase?

Bumping this discussion as we are in the RC phase. Was the root cause found and we can revert the patch?

Sadly not, so far.

Quoted 1 line

Or shall we surround the code with NDEBUG?

Yes, I'll submit a patch.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xen.org/xen-devel