Skip to contentSkip to conversation

xen vtd : set msi guest_masked 0 by default

PATCH In review Jianzhong,Chang 8 participants 18 messages Jan 26, 2016 – Oct 10, 2018

lore
Share & tools
lore

Jianzhong,Chang

author
#0 lore raw

PATCH 1/1 xen vtd : set msi guest_masked 0 by default

There are some problems when msi guest_masked is set to 1 by default. When guest os is windows 2008 r2 server,
the device(eg X540-AT2 vf) is not initialized correctly.
Host will always receive message like this :"VF Reset msg received from vf". Guest has network connectivity issues,
and can not correctly receive/send the packet.
So, guest_masked is set to 0 by default.

Signed-off-by Jianzhong,Chang <[email protected]>
---
 xen/arch/x86/msi.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
xen/arch/x86/msi.c +2 -2
index 5a481f6..b4f60a3 100644
@@ -512,7 +512,7 @@ void guest_mask_msi_irq(struct irq_desc *desc, bool_t mask)
 
 static unsigned int startup_msi_irq(struct irq_desc *desc)
 {
-    if ( unlikely(!msi_set_mask_bit(desc, 0, !!(desc->status & IRQ_GUEST))) )
+    if ( unlikely(!msi_set_mask_bit(desc, 0, 0) ))
         WARN();
     return 0;
 }

@@ -972,7 +972,7 @@ static int msix_capability_init(struct pci_dev *dev,
         entry->msi_attrib.entry_nr = msi->entry_nr;
         entry->msi_attrib.maskbit = 1;
         entry->msi_attrib.host_masked = 1;
-        entry->msi_attrib.guest_masked = 1;
+        entry->msi_attrib.guest_masked = 0;
         entry->msi_attrib.pos = pos;
         entry->irq = msi->irq;
         entry->dev = dev;
Signature · 1 line
1.7.1
#1 lore raw
Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 26.01.16 at 02:34, <[email protected]> wrote:

There are some problems when msi guest_masked is set to 1 by default. When guest os is windows 2008 r2 server,
the device(eg X540-AT2 vf) is not initialized correctly.
Host will always receive message like this :"VF Reset msg received from vf". Guest has network connectivity issues,
and can not correctly receive/send the packet.
So, guest_masked is set to 0 by default.

You describe a problem and half of your change, but there's no connection between the two: What is actually wrong with current behavior (matching the hardware's - MSI-X mask bits are set when coming out of reset).

xen/arch/x86/msi.c void guest_mask_msi_irq(struct irq_desc *desc, bool_t mask) · line 518

Quoted 11 lines
xen/arch/x86/msi.c +1 -1
@@ -512,7 +512,7 @@ void guest_mask_msi_irq(struct irq_desc *desc, bool_t mask)
 
 static unsigned int startup_msi_irq(struct irq_desc *desc)
 {
-    if ( unlikely(!msi_set_mask_bit(desc, 0, !!(desc->status & IRQ_GUEST))) )
+    if ( unlikely(!msi_set_mask_bit(desc, 0, 0) ))
         WARN();
     return 0;
 }

Whether this part can go under "set ... by default" is highly questionable. Plus, while this affects MSI and MSI-X, ...

xen/arch/x86/msi.c static int msix_capability_init(struct pci_dev *dev, · line 978

Quoted 9 lines
xen/arch/x86/msi.c +1 -1
@@ -972,7 +972,7 @@ static int msix_capability_init(struct pci_dev *dev,
         entry->msi_attrib.entry_nr = msi->entry_nr;
         entry->msi_attrib.maskbit = 1;
         entry->msi_attrib.host_masked = 1;
-        entry->msi_attrib.guest_masked = 1;
+        entry->msi_attrib.guest_masked = 0;
         entry->msi_attrib.pos = pos;
         entry->irq = msi->irq;
         entry->dev = dev;

... this change affect MSI-X only, and doing some guessing from what you write above I suspect you only really tested one of the two cases.

So while the change _may_ be necessary, you'll need to do a better job at explaining why you what you do.

Jan

#2 lore raw
Quoted 15 lines

From: Jan Beulich
Sent: Tuesday, January 26, 2016 8:57 PM
To: Chang Jianzhong
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [Xen-devel] [PATCH] xen vtd : set msi guest_masked 0 by default

Quoted 8 lines
Quoted 1 line
Quoted 1 line · On 26.01.16 at 02:34, <[email protected]> wrote:

There are some problems when msi guest_masked is set to 1 by default. When guest os is windows 2008 r2 server,
the device(eg X540-AT2 vf) is not initialized correctly.
Host will always receive message like this :"VF Reset msg received from vf". Guest has network connectivity issues,
and can not correctly receive/send the packet.
So, guest_masked is set to 0 by default.

Quoted 5 lines

You describe a problem and half of your change, but there's no connection between the two: What is actually wrong with current behavior (matching the hardware's - MSI-X mask bits are set when coming out of reset).

Agree. Jianzhong, could you elaborate how mask bit is related to the error message above? Also instead of changing the behavior completely, you need consider there must be a reason for guest_masked set to 1 originally. So when you fix one issue please pay attention to not break other scenarios.

Thanks
Kevin

author
#3 lore raw
In reply to Jan Beulich

2016-01-26 20:56 GMT+08:00 Jan Beulich <[email protected]>:

Quoted 29 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line · On 26.01.16 at 02:34, <[email protected]> wrote:

There are some problems when msi guest_masked is set to 1 by default. When guest os is windows 2008 r2 server,
the device(eg X540-AT2 vf) is not initialized correctly. Host will always receive message like this :"VF Reset msg received from

vf".

Quoted 3 lines

Guest has network connectivity issues,
and can not correctly receive/send the packet.
So, guest_masked is set to 0 by default.

You describe a problem and half of your change, but there's no connection between the two: What is actually wrong with current behavior (matching the hardware's - MSI-X mask bits are set when coming out of reset).

xen/arch/x86/msi.c void guest_mask_msi_irq(struct irq_desc *desc,

Quoted 3 lines
xen/arch/x86/msi.c +0 -0
@@ -512,7 +512,7 @@ void guest_mask_msi_irq(struct irq_desc *desc,

bool_t mask)

xen/arch/x86/msi.c void guest_mask_msi_irq(struct irq_desc *desc, bool_t mask)

Quoted 4 lines
xen/arch/x86/msi.c +0 -1
 static unsigned int startup_msi_irq(struct irq_desc *desc)
 {
-    if ( unlikely(!msi_set_mask_bit(desc, 0, !!(desc->status &

IRQ_GUEST))) )

xen/arch/x86/msi.c void guest_mask_msi_irq(struct irq_desc *desc, bool_t mask) · line 518

Quoted 4 lines
xen/arch/x86/msi.c +1 -0
+    if ( unlikely(!msi_set_mask_bit(desc, 0, 0) ))
         WARN();
     return 0;
 }
Quoted 4 lines

Whether this part can go under "set ... by default" is highly questionable. Plus, while this affects MSI and MSI-X, ...

If irq is owned by guest,in function msi_set_mask_bit():

...
bool_t flag = host || guest; //The flag should be true.
...
writel(flag, entry->mask_base + PCI_MSIX_ENTRY_VECTOR_CTRL_OFFSET);
...
PCI device can not generate interrrupt.
windows guest can not change vector_ctrl_mask, guest os get abnormal status of nic.

Quoted 14 lines

xen/arch/x86/msi.c static int msix_capability_init(struct pci_dev *dev, · line 978

Quoted 9 lines
xen/arch/x86/msi.c +1 -1
@@ -972,7 +972,7 @@ static int msix_capability_init(struct pci_dev *dev,
         entry->msi_attrib.entry_nr = msi->entry_nr;
         entry->msi_attrib.maskbit = 1;
         entry->msi_attrib.host_masked = 1;
-        entry->msi_attrib.guest_masked = 1;
+        entry->msi_attrib.guest_masked = 0;
         entry->msi_attrib.pos = pos;
         entry->irq = msi->irq;
         entry->dev = dev;

... this change affect MSI-X only, and doing some guessing from what you write above I suspect you only really tested one of the two cases.

Quoted 3 lines

So while the change _may_ be necessary, you'll need to do a better job at explaining why you what you do.

Msi guest_masked is set to 0 in the original code, only msi-x guest_masked is modifed in msix_capability_init() function by patch.

Quoted 4 lines

Jan

This issue appears after commited the variable guest_mask. Initialization operations of pci device may be changed in windows guest,or Xen need to change the initial state of vtd pci device.

Signature · 1 line
Jianzhong,Chang

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#4 lore raw
Quoted 36 lines
Quoted 1 line
Quoted 1 line · On 07.03.16 at 09:12, <[email protected]> wrote:

2016-01-26 20:56 GMT+08:00 Jan Beulich <[email protected]>:

Quoted 29 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line · On 26.01.16 at 02:34, <[email protected]> wrote:

There are some problems when msi guest_masked is set to 1 by default. When guest os is windows 2008 r2 server,
the device(eg X540-AT2 vf) is not initialized correctly. Host will always receive message like this :"VF Reset msg received from

vf".

Quoted 3 lines

Guest has network connectivity issues,
and can not correctly receive/send the packet.
So, guest_masked is set to 0 by default.

You describe a problem and half of your change, but there's no connection between the two: What is actually wrong with current behavior (matching the hardware's - MSI-X mask bits are set when coming out of reset).

xen/arch/x86/msi.c void guest_mask_msi_irq(struct irq_desc *desc,

Quoted 3 lines
xen/arch/x86/msi.c +0 -0
@@ -512,7 +512,7 @@ void guest_mask_msi_irq(struct irq_desc *desc,

bool_t mask)

xen/arch/x86/msi.c void guest_mask_msi_irq(struct irq_desc *desc, bool_t mask)

Quoted 4 lines
xen/arch/x86/msi.c +0 -1
 static unsigned int startup_msi_irq(struct irq_desc *desc)
 {
-    if ( unlikely(!msi_set_mask_bit(desc, 0, !!(desc->status &

IRQ_GUEST))) )

xen/arch/x86/msi.c void guest_mask_msi_irq(struct irq_desc *desc, bool_t mask) · line 518

Quoted 4 lines
xen/arch/x86/msi.c +1 -0
+    if ( unlikely(!msi_set_mask_bit(desc, 0, 0) ))
         WARN();
     return 0;
 }
Quoted 4 lines

Whether this part can go under "set ... by default" is highly questionable. Plus, while this affects MSI and MSI-X, ...

If irq is owned by guest,in function msi_set_mask_bit():

Quoted 8 lines

...
bool_t flag = host || guest; //The flag should be true.
...
writel(flag, entry->mask_base + PCI_MSIX_ENTRY_VECTOR_CTRL_OFFSET);
...
PCI device can not generate interrrupt.
windows guest can not change vector_ctrl_mask, guest os get abnormal status of nic.

Here and below - I'm sorry, I do not understand what you're trying to tell us, or how this is meant to extend beyond the original (too vague) description of your change. Among unclear things is why "windows guest can not change vector_ctrl_mask" - you again just make statements without dealing with any of the why-s.

Please can you try to explain things by matching operations done by the guest, qemu, and the hypervisor to the effect they have on the state of the mask bit, and then point out which of those steps needs changing or doesn't work as intended?

Jan

Quoted 21 lines
Quoted 14 lines

xen/arch/x86/msi.c static int msix_capability_init(struct pci_dev *dev, · line 978

Quoted 9 lines
xen/arch/x86/msi.c +1 -1
@@ -972,7 +972,7 @@ static int msix_capability_init(struct pci_dev *dev,
         entry->msi_attrib.entry_nr = msi->entry_nr;
         entry->msi_attrib.maskbit = 1;
         entry->msi_attrib.host_masked = 1;
-        entry->msi_attrib.guest_masked = 1;
+        entry->msi_attrib.guest_masked = 0;
         entry->msi_attrib.pos = pos;
         entry->irq = msi->irq;
         entry->dev = dev;

... this change affect MSI-X only, and doing some guessing from what you write above I suspect you only really tested one of the two cases.

Quoted 3 lines

So while the change _may_ be necessary, you'll need to do a better job at explaining why you what you do.

Msi guest_masked is set to 0 in the original code, only msi-x guest_masked is modifed in msix_capability_init() function by patch.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

Jan

This issue appears after commited the variable guest_mask. Initialization operations of pci device may be changed in windows guest,or Xen need to change the initial state of vtd pci device.

Signature · 1 line
Jianzhong,Chang

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#5 lore raw
In reply to Jianzhong,Chang
Quoted 6 lines · Jianzhong,Chang

There are some problems when msi guest_masked is set to 1 by default. When guest os is windows 2008 r2 server,
the device(eg X540-AT2 vf) is not initialized correctly.
Host will always receive message like this :"VF Reset msg received from vf". Guest has network connectivity issues,
and can not correctly receive/send the packet.

In other words "the guest doesn't get any interrupts from the NIC".

Quoted 1 line

So, guest_masked is set to 0 by default.

This seems consistent with the PCI spec, which says that "After reset, the state of all implemented Mask and Pending bits is 0 (no vectors are masked and no messages are pending)."

That's what we *used* to have in Xen, before these commits changed it to assume that IRQs were guest-masked by default:

https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=ad28e42bd1d28d746988ed71654e8aa670629753
https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=84d6add5593d865736831d150da7c38588f669f6

Quoted 11 lines
xen/arch/x86/msi.c +0 -0
@@ -512,7 +512,7 @@ void guest_mask_msi_irq(struct irq_desc *desc, bool_t mask)


 static unsigned int startup_msi_irq(struct irq_desc *desc)
 {
-    if ( unlikely(!msi_set_mask_bit(desc, 0, !!(desc->status & IRQ_GUEST))) ) +    if ( unlikely(!msi_set_mask_bit(desc, 0, 0) ))
         WARN();
     return 0;
 }

In testing, this part actually seems to make the difference in practice. Interrupts now work, and Windows guests have connectivity again.

xen/arch/x86/msi.c static int msix_capability_init(struct pci_dev *dev, · line 978

Quoted 9 lines
xen/arch/x86/msi.c +1 -1
@@ -972,7 +972,7 @@ static int msix_capability_init(struct pci_dev *dev,
          entry->msi_attrib.entry_nr = msi->entry_nr;
          entry->msi_attrib.maskbit = 1;
          entry->msi_attrib.host_masked = 1;
-        entry->msi_attrib.guest_masked = 1;
+        entry->msi_attrib.guest_masked = 0;
          entry->msi_attrib.pos = pos;
          entry->irq = msi->irq;
          entry->dev = dev;

That also seems correct though, since it reflects the actual state we intend to emulate.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw
Quoted 38 lines · David Woodhouse
Quoted 6 lines · Jianzhong,Chang

There are some problems when msi guest_masked is set to 1 by default. When guest os is windows 2008 r2 server,
the device(eg X540-AT2 vf) is not initialized correctly.
Host will always receive message like this :"VF Reset msg received from vf". Guest has network connectivity issues,
and can not correctly receive/send the packet.

In other words "the guest doesn't get any interrupts from the NIC".

Quoted 1 line

So, guest_masked is set to 0 by default.

This seems consistent with the PCI spec, which says that "After reset, the state of all implemented Mask and Pending bits is 0 (no vectors are masked and no messages are pending)."

That's what we *used* to have in Xen, before these commits changed it to assume that IRQs were guest-masked by default:

https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=ad28e42bd1d28d746988ed71654e8aa670629753
https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=84d6add5593d865736831d150da7c38588f669f6

Quoted 11 lines
xen/arch/x86/msi.c +0 -0
@@ -512,7 +512,7 @@ void guest_mask_msi_irq(struct irq_desc *desc, bool_t mask)


 static unsigned int startup_msi_irq(struct irq_desc *desc)
 {
-    if ( unlikely(!msi_set_mask_bit(desc, 0, !!(desc->status & IRQ_GUEST))) ) +    if ( unlikely(!msi_set_mask_bit(desc, 0, 0) ))
         WARN();
     return 0;
 }

In testing, this part actually seems to make the difference in practice. Interrupts now work, and Windows guests have connectivity again.

Quoted 12 lines

xen/arch/x86/msi.c static int msix_capability_init(struct pci_dev *dev, · line 978

Quoted 9 lines
xen/arch/x86/msi.c +1 -1
@@ -972,7 +972,7 @@ static int msix_capability_init(struct pci_dev *dev,
          entry->msi_attrib.entry_nr = msi->entry_nr;
          entry->msi_attrib.maskbit = 1;
          entry->msi_attrib.host_masked = 1;
-        entry->msi_attrib.guest_masked = 1;
+        entry->msi_attrib.guest_masked = 0;
          entry->msi_attrib.pos = pos;
          entry->irq = msi->irq;
          entry->dev = dev;

That also seems correct though, since it reflects the actual state we intend to emulate.

Hm, I think I might have fixed this issue, see:

https://git.qemu.org/?p=qemu.git;a=commit;h=a8036336609d2e184fc3543a4c439c0ba7d7f3a2

And the Xen side:

http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=69d99d1b223fc5082400374698ddd7486e5ea953

The original report of the issue is at:

https://lists.xenproject.org/archives/html/xen-devel/2017-07/msg00915.html

Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#8 lore raw

PATCH 1/1 xen/pt: allow QEMU to request MSI unmasking at bind time

When a MSI interrupt is bound to a guest using xc_domain_update_msi_irq (XEN_DOMCTL_bind_pt_irq) the interrupt is left masked by default.

This causes problems with guests that first configure interrupts and clean the per-entry MSIX table mask bit and afterwards enable MSIX globally. In such scenario the Xen internal msixtbl handlers would not detect the unmasking of MSIX entries because vectors are not yet registered since MSIX is not enabled, and vectors would be left masked.

Introduce a new flag in the gflags field to signal Xen whether a MSI interrupt should be unmasked after being bound.

This also requires to track the mask register for MSI interrupts, so QEMU can also notify to Xen whether the MSI interrupt should be bound masked or unmasked

Signed-off-by Roger Pau Monné <[email protected]>
Reviewed-by Jan Beulich <[email protected]>
Reported-by Andreas Kinzler <[email protected]>
Reviewed-by Stefano Stabellini <[email protected]>
Signed-off-by Stefano Stabellini <[email protected]>

[Backported from commit a8036336609d2e184fc3543a4c439c0ba7d7f3a2
https://git.qemu.org/?p=qemu.git;a=commit;h=a8036336609d2e184fc3543a4c439c0ba7d7f3a2
https://xenbits.xen.org/gitweb/?p=qemu-xen.git;a=commit;h=a8036336609d2e184fc3543a4c439c0ba7d7f3a2]

Signed-off-by Andra Paraschiv <[email protected]>
---
 hw/pass-through.c | 23 +++++++++++++++++++++--
 hw/pass-through.h |  1 +
 hw/pt-msi.c       | 25 +++++++++++++++++++++----
 hw/pt-msi.h       |  9 +++++++++
 4 files changed, 52 insertions(+), 6 deletions(-)
hw/pass-through.c +21 -2
index 0b76585..c6cb60e 100644
@@ -188,6 +188,9 @@ static int pt_word_reg_write(struct pt_dev *ptdev,
 static int pt_long_reg_write(struct pt_dev *ptdev,
     struct pt_reg_tbl *cfg_entry,
     uint32_t *value, uint32_t dev_value, uint32_t valid_mask);
+static int pt_mask_reg_write(struct pt_dev *ptdev,
+    struct pt_reg_tbl *cfg_entry,
+    uint32_t *value, uint32_t dev_value, uint32_t valid_mask);
 static int pt_cmd_reg_write(struct pt_dev *ptdev,
     struct pt_reg_tbl *cfg_entry,
     uint16_t *value, uint16_t dev_value, uint16_t valid_mask);

@@ -755,7 +758,7 @@ static struct pt_reg_info_tbl pt_emu_reg_msi_tbl[] = {
         .emu_mask   = 0xFFFFFFFF,
         .init       = pt_mask_reg_init,
         .u.dw.read  = pt_long_reg_read,
-        .u.dw.write = pt_long_reg_write,
+        .u.dw.write = pt_mask_reg_write,
     },
     /* Mask reg (if PCI_MSI_FLAGS_MASK_BIT set, for 64-bit devices) */
     {

@@ -766,7 +769,7 @@ static struct pt_reg_info_tbl pt_emu_reg_msi_tbl[] = {
         .emu_mask   = 0xFFFFFFFF,
         .init       = pt_mask_reg_init,
         .u.dw.read  = pt_long_reg_read,
-        .u.dw.write = pt_long_reg_write,
+        .u.dw.write = pt_mask_reg_write,
     },
     /* Pending reg (if PCI_MSI_FLAGS_MASK_BIT set, for 32-bit devices) */
     {

@@ -3572,6 +3575,22 @@ static int pt_long_reg_write(struct pt_dev *ptdev,
     return 0;
 }
 
+/* write guest mask bits */
+static int pt_mask_reg_write(struct pt_dev *ptdev,
+        struct pt_reg_tbl *cfg_entry,
+        uint32_t *value, uint32_t dev_value, uint32_t valid_mask)
+{
+    int rc;
+
+    rc = pt_long_reg_write(ptdev, cfg_entry, value, dev_value, valid_mask);
+    if (rc)
+        return rc;
+
+    ptdev->msi->mask = *value;
+
+    return 0;
+}
+
 /* write Command register */
 static int pt_cmd_reg_write(struct pt_dev *ptdev,
         struct pt_reg_tbl *cfg_entry,
hw/pass-through.h +1 -0
index bb6ddce..0b58224 100644
@@ -191,6 +191,7 @@ struct pt_region {
 struct pt_msi_info {
     uint32_t flags;
     uint32_t ctrl_offset; /* saved control offset */
+    uint32_t mask;     /* guest mask bits */
     int pirq;          /* guest pirq corresponding */
     uint32_t addr_lo;  /* guest message address */
     uint32_t addr_hi;  /* guest message upper address */
hw/pt-msi.c +21 -4
index c6baea9..8b06450 100644
@@ -138,6 +138,9 @@ int pt_msi_update(struct pt_dev *d)
     addr = (uint64_t)d->msi->addr_hi << 32 | d->msi->addr_lo;
     gflags = __get_msi_gflags(d->msi->data, addr);
 
+    /* Current MSI emulation in QEMU only supports 1 vector */
+    gflags |= (d->msi->mask & 1) ? 0 : (1u << GLFAGS_SHIFT_UNMASKED);
+
     PT_LOG("Update msi with pirq %x gvec %x gflags %x\n",
            d->msi->pirq, gvec, gflags);
 

@@ -275,7 +278,8 @@ void pt_disable_msi_translate(struct pt_dev *dev)
     }
 }
 
-static int pt_msix_update_one(struct pt_dev *dev, int entry_nr)
+static int pt_msix_update_one(struct pt_dev *dev, int entry_nr,
+                              uint32_t vec_ctrl)
 {
     struct msix_entry_info *entry = &dev->msix->msix_entry[entry_nr];
     int pirq = entry->pirq;

@@ -316,6 +320,9 @@ static int pt_msix_update_one(struct pt_dev *dev, int entry_nr)
         entry->pirq = pirq;
     }
 
+    gflags |= (vec_ctrl & PCI_MSIX_ENTRY_CTRL_MASKBIT) ? 0 :
+        (1u << GLFAGS_SHIFT_UNMASKED);
+
     PT_LOG("Update msix entry %x with pirq %x gvec %x\n",
             entry_nr, pirq, gvec);
 

@@ -343,7 +350,7 @@ int pt_msix_update(struct pt_dev *dev)
 
     for ( i = 0; i < msix->total_entries; i++ )
     {
-        pt_msix_update_one(dev, i);
+        pt_msix_update_one(dev, i, msix->msix_entry[i].io_mem[3]);
     }
 
     return 0;

@@ -479,8 +486,18 @@ static void pci_msix_writel(void *opaque, target_phys_addr_t addr, uint32_t val)
 
     if ( offset == 3 )
     {
-        if ( msix->enabled && !(val & 0x1) )
-            pt_msix_update_one(dev, entry_nr);
+        if ( msix->enabled && !(val & 0x1) ) {
+            const volatile uint32_t *vec_ctrl;
+
+            /*
+             * If Xen intercepts the mask bit access, io_mem[3] may not be
+             * up-to-date. Read from hardware directly.
+             */
+            vec_ctrl = dev->msix->phys_iomem_base +
+                PCI_MSIX_ENTRY_SIZE * entry_nr + PCI_MSIX_ENTRY_VECTOR_CTRL;
+
+            pt_msix_update_one(dev, entry_nr, *vec_ctrl);
+        }
     }
 }
hw/pt-msi.h +9 -0
index 94e0d35..a80d76e 100644
@@ -33,6 +33,14 @@
 #define PCI_MSIX_PBA        8
 #define  PCI_MSIX_BIR       0x7
 
+/* MSI-X Table entry format */
+#define PCI_MSIX_ENTRY_SIZE             16
+#define  PCI_MSIX_ENTRY_LOWER_ADDR      0
+#define  PCI_MSIX_ENTRY_UPPER_ADDR      4
+#define  PCI_MSIX_ENTRY_DATA            8
+#define  PCI_MSIX_ENTRY_VECTOR_CTRL     12
+#define   PCI_MSIX_ENTRY_CTRL_MASKBIT   1
+
 #define MSI_FLAG_UNINIT 0x1000
 #define PT_MSI_MAPPED   0x2000
 

@@ -77,6 +85,7 @@
 #define GFLAGS_SHIFT_DM             9
 #define GLFAGS_SHIFT_DELIV_MODE     12
 #define GLFAGS_SHIFT_TRG_MODE       15
+#define GLFAGS_SHIFT_UNMASKED       16
 
 void
 msi_set_enable(struct pt_dev *dev, int en);
Signature · 10 lines
2.7.3.AMZN




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#9 lore raw

Re: [PATCH qemu-xen-traditional] xen/pt: allow QEMU to request MSI unmasking at bind time

Quoted 24 lines · Andra Paraschiv

When a MSI interrupt is bound to a guest using xc_domain_update_msi_irq (XEN_DOMCTL_bind_pt_irq) the interrupt is left masked by default.

This causes problems with guests that first configure interrupts and clean the per-entry MSIX table mask bit and afterwards enable MSIX globally. In such scenario the Xen internal msixtbl handlers would not detect the unmasking of MSIX entries because vectors are not yet registered since MSIX is not enabled, and vectors would be left masked.

Introduce a new flag in the gflags field to signal Xen whether a MSI interrupt should be unmasked after being bound.

This also requires to track the mask register for MSI interrupts, so QEMU can also notify to Xen whether the MSI interrupt should be bound masked or unmasked

Signed-off-by Roger Pau Monné <[email protected]>
Reviewed-by Jan Beulich <[email protected]>
Reported-by Andreas Kinzler <[email protected]>
Reviewed-by Stefano Stabellini <[email protected]>
Signed-off-by Stefano Stabellini <[email protected]>
Quoted 5 lines

[Backported from commit a8036336609d2e184fc3543a4c439c0ba7d7f3a2
https://git.qemu.org/?p=qemu.git;a=commit;h=a8036336609d2e184fc3543a4c439c0ba7d7f3a2
https://xenbits.xen.org/gitweb/?p=qemu-xen.git;a=commit;h=a8036336609d2e184fc3543a4c439c0ba7d7f3a2]

Signed-off-by Andra Paraschiv <[email protected]>

Thanks for doing the backport, LGTM.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#10 lore raw

Re: [PATCH qemu-xen-traditional] xen/pt: allow QEMU to request MSI unmasking at bind time

+ Cc: Ian Jackson <[email protected]> for review.

Thanks, Roger, for review and feedback.

Andra
________________________________________
From: Roger Pau Monné <[email protected]>
Sent: Monday, June 18, 2018 2:43 PM
To: Paraschiv, Andra-Irina
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Woodhouse, David
Subject: Re: [PATCH qemu-xen-traditional] xen/pt: allow QEMU to request MSI unmasking at bind time

Quoted 24 lines · Andra Paraschiv

When a MSI interrupt is bound to a guest using xc_domain_update_msi_irq (XEN_DOMCTL_bind_pt_irq) the interrupt is left masked by default.

This causes problems with guests that first configure interrupts and clean the per-entry MSIX table mask bit and afterwards enable MSIX globally. In such scenario the Xen internal msixtbl handlers would not detect the unmasking of MSIX entries because vectors are not yet registered since MSIX is not enabled, and vectors would be left masked.

Introduce a new flag in the gflags field to signal Xen whether a MSI interrupt should be unmasked after being bound.

This also requires to track the mask register for MSI interrupts, so QEMU can also notify to Xen whether the MSI interrupt should be bound masked or unmasked

Signed-off-by Roger Pau Monné <[email protected]>
Reviewed-by Jan Beulich <[email protected]>
Reported-by Andreas Kinzler <[email protected]>
Reviewed-by Stefano Stabellini <[email protected]>
Signed-off-by Stefano Stabellini <[email protected]>
Quoted 5 lines

[Backported from commit a8036336609d2e184fc3543a4c439c0ba7d7f3a2
https://git.qemu.org/?p=qemu.git;a=commit;h=a8036336609d2e184fc3543a4c439c0ba7d7f3a2
https://xenbits.xen.org/gitweb/?p=qemu-xen.git;a=commit;h=a8036336609d2e184fc3543a4c439c0ba7d7f3a2]

Signed-off-by Andra Paraschiv <[email protected]>

Thanks for doing the backport, LGTM.

Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#11 lore raw

Re: [PATCH qemu-xen-traditional] xen/pt: allow QEMU to request MSI unmasking at bind time

Ian, any objections?

Quoted 61 lines · Andra-Irina

+ Cc: Ian Jackson <[email protected]> for review.

Thanks, Roger, for review and feedback.

Andra
________________________________________
From: Roger Pau Monné <[email protected]>
Sent: Monday, June 18, 2018 2:43 PM
To: Paraschiv, Andra-Irina
Cc: [email protected]; [email protected]; jbeulich@ suse.com; [email protected]; Woodhouse, David
Subject: Re: [PATCH qemu-xen-traditional] xen/pt: allow QEMU to request MSI unmasking at bind time

Quoted 30 lines · Andra Paraschiv

When a MSI interrupt is bound to a guest using xc_domain_update_msi_irq (XEN_DOMCTL_bind_pt_irq) the interrupt is left masked by default.

This causes problems with guests that first configure interrupts and
clean the per-entry MSIX table mask bit and afterwards enable MSIX globally. In such scenario the Xen internal msixtbl handlers would not
detect the unmasking of MSIX entries because vectors are not yet registered since MSIX is not enabled, and vectors would be left masked.

Introduce a new flag in the gflags field to signal Xen whether a MSI
interrupt should be unmasked after being bound.

This also requires to track the mask register for MSI interrupts, so
QEMU can also notify to Xen whether the MSI interrupt should be bound
masked or unmasked

Signed-off-by Roger Pau Monné <[email protected]>
Reviewed-by Jan Beulich <[email protected]>
Reported-by Andreas Kinzler <[email protected]>
Reviewed-by Stefano Stabellini <[email protected]>
Signed-off-by Stefano Stabellini <[email protected]>
Quoted 7 lines

[Backported from commit a8036336609d2e184fc3543a4c439c0ba7d7f3a2
 https://git.qemu.org/?p=qemu.git;a=commit;h=a8036336609d2e184fc354
3a4c439c0ba7d7f3a2
 https://xenbits.xen.org/gitweb/?p=qemu-xen.git;a=commit;h=a8036336
609d2e184fc3543a4c439c0ba7d7f3a2]

Signed-off-by Andra Paraschiv <[email protected]>

Thanks for doing the backport, LGTM.

Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

Quoted 4 lines

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Amazon Development Centre (London) Ltd. Registered in England and Wales with registration number 04543232 with its registered office at 1 Principal Place, Worship Street, London EC2A 2FA, United Kingdom.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#12 lore raw

Re: [PATCH qemu-xen-traditional] xen/pt: allow QEMU to request MSI unmasking at bind time

Quoted 6 lines

From: Woodhouse, David
Sent: Tuesday, July 31, 2018 1:22 PM
To: [email protected]; Paraschiv, Andra-Irina
Cc: [email protected]; [email protected]; [email protected]; [email protected]; xen->[email protected]
Subject: Re: [Xen-devel] [PATCH qemu-xen-traditional] xen/pt: allow QEMU to request MSI unmasking at bind >time

Quoted 57 lines

Ian, any objections?

Quoted 54 lines · Andra-Irina

+ Cc: Ian Jackson <[email protected]> for review.

Thanks, Roger, for review and feedback.

Andra
________________________________________
From: Roger Pau Monné <[email protected]>
Sent: Monday, June 18, 2018 2:43 PM
To: Paraschiv, Andra-Irina
Cc: [email protected]; [email protected]; jbeulich@ suse.com; [email protected]; Woodhouse, David
Subject: Re: [PATCH qemu-xen-traditional] xen/pt: allow QEMU to request MSI unmasking at bind time

Quoted 2 lines · Andra Paraschiv

When a MSI interrupt is bound to a guest using

xc_domain_update_msi_irq (XEN_DOMCTL_bind_pt_irq) the interrupt is

Quoted 27 lines

left masked by default.

This causes problems with guests that first configure interrupts and
clean the per-entry MSIX table mask bit and afterwards enable MSIX globally. In such scenario the Xen internal msixtbl handlers would not
detect the unmasking of MSIX entries because vectors are not yet registered since MSIX is not enabled, and vectors would be left masked.

Introduce a new flag in the gflags field to signal Xen whether a MSI
interrupt should be unmasked after being bound.

This also requires to track the mask register for MSI interrupts, so
QEMU can also notify to Xen whether the MSI interrupt should be bound
masked or unmasked

Signed-off-by Roger Pau Monné <[email protected]>
Reviewed-by Jan Beulich <[email protected]>
Reported-by Andreas Kinzler <[email protected]>
Reviewed-by Stefano Stabellini <[email protected]>
Signed-off-by Stefano Stabellini <[email protected]>
Quoted 7 lines

[Backported from commit a8036336609d2e184fc3543a4c439c0ba7d7f3a2
https://git.qemu.org/?p=qemu.git;a=commit;h=a8036336609d2e184fc354
3a4c439c0ba7d7f3a2
https://xenbits.xen.org/gitweb/?p=qemu-xen.git;a=commit;h=a8036336
609d2e184fc3543a4c439c0ba7d7f3a2]

Signed-off-by Andra Paraschiv <[email protected]>

Thanks for doing the backport, LGTM.

Ian, can you please take a look on this patch that is considered for backporting from QEMU upstream repo to QEMU traditional?

Thanks.

Quoted 11 lines
Quoted 11 lines

Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#13 lore raw

Re: [PATCH qemu-xen-traditional] xen/pt: allow QEMU to request MSI unmasking at bind time

Quoted 1 line
Quoted 1 line

Ian, any objections?

Sorry for dropping this. It's been a while!

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

When a MSI interrupt is bound to a guest using xc_domain_update_msi_irq (XEN_DOMCTL_bind_pt_irq) the interrupt is left masked by default.

This causes problems with guests that first configure interrupts and clean the per-entry MSIX table mask bit and afterwards enable MSIX globally. In such scenario the Xen internal msixtbl handlers would not detect the unmasking of MSIX entries because vectors are not yet registered since MSIX is not enabled, and vectors would be left masked.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Introduce a new flag in the gflags field to signal Xen whether a MSI interrupt should be unmasked after being bound.

This also requires to track the mask register for MSI interrupts, so QEMU can also notify to Xen whether the MSI interrupt should be bound masked or unmasked

I searched my email archives but I haven't been able to find a copy of the actual patch. Was it ever sent to me ?

Also, qemu-trad is quite deeply frozen. I'd like an explanation of the practical impact of this bug, and the use cases where the fix is needed, to justify the change. Sorry to be awkward.

Regards,
Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#14 lore raw

Re: [PATCH qemu-xen-traditional] xen/pt: allow QEMU to request MSI unmasking at bind time

Quoted 3 lines
Quoted 1 line
Quoted 1 line

Ian, any objections?

Sorry for dropping this. It's been a while!

No problem, Ian, we have this heads-up now.

Quoted 19 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines

When a MSI interrupt is bound to a guest using xc_domain_update_msi_irq (XEN_DOMCTL_bind_pt_irq) the interrupt is left masked by default.

This causes problems with guests that first configure interrupts and clean the per-entry MSIX table mask bit and afterwards enable MSIX globally. In such scenario the Xen internal msixtbl handlers would not detect the unmasking of MSIX entries because vectors are not yet registered since MSIX is not enabled, and vectors would be left masked.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Introduce a new flag in the gflags field to signal Xen whether a MSI interrupt should be unmasked after being bound.

This also requires to track the mask register for MSI interrupts, so QEMU can also notify to Xen whether the MSI interrupt should be bound masked or unmasked

Quoted 6 lines

I searched my email archives but I haven't been able to find a copy of the actual patch. Was it ever sent to me ?

Also, qemu-trad is quite deeply frozen. I'd like an explanation of the practical impact of this bug, and the use cases where the fix is needed, to justify the change. Sorry to be awkward.

That's the link to the patch for qemu-xen-traditional:
https://lists.xenproject.org/archives/html/xen-devel/2018-06/msg01248.html

(https://lists.xenproject.org/archives/html/xen-devel/2018-06/msg01373.html - added you in CC)

The issue was notified here - https://lists.xenproject.org/archives/html/xen-devel/2018-05/msg01238.html and fixes were added in xen and qemu. QEMU needs to interact with Xen to have guest interrupts masked/unmasked.

This patch is a backport from qemu.

One of the use cases where this fix is needed is: guest OS is Windows and the host has the latest stable version of xen and qemu-xen-traditional. Using this environment, setting up a PCI device (using MSI/MSI-X interrupts) will not successfully finish.

Thanks,
Andra

Quoted 3 lines

Regards,
Ian.

Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#15 lore raw

Re: [PATCH qemu-xen-traditional] xen/pt: allow QEMU to request MSI unmasking at bind time

Paraschiv, Andra-Irina writes ("Re: [Xen-devel] [PATCH qemu-xen-traditional] xen/pt: allow QEMU to request MSI unmasking at bind time"):

Quoted 1 line

One of the use cases where this fix is needed is: guest OS is Windows and the host has the latest stable version of xen and qemu-xen-traditional. Using this environment, setting up a PCI device (using MSI/MSI-X interrupts) will not successfully finish.

OK, I'm sold on applying the patch to qemu-trad. Great.

Quoted 2 lines

That's the link to the patch for qemu-xen-traditional:
https://lists.xenproject.org/archives/html/xen-devel/2018-06/msg01248.html

Right, thanks.

Quoted 1 line

(https://lists.xenproject.org/archives/html/xen-devel/2018-06/msg01373.html - added you in CC)

Unfortunately I don't want to fish the patch out of the mailing list archives (because the conversion from email to html and back etc. might mangle it) and the email with the CC to me didn't actually have the patch in it.

So can you please resend the backported patch to me ? Or point me at a git tree that contains the backported patch ?

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#16 lore raw

Re: [PATCH qemu-xen-traditional] xen/pt: allow QEMU to request MSI unmasking at bind time

Ian Jackson writes ("Re: [Xen-devel] [PATCH qemu-xen-traditional] xen/pt: allow QEMU to request MSI unmasking at bind time"):

Quoted 5 lines

Paraschiv, Andra-Irina writes ("Re: [Xen-devel] [PATCH qemu-xen-traditional] xen/pt: allow QEMU to request MSI unmasking at bind time"):

Quoted 1 line

One of the use cases where this fix is needed is: guest OS is Windows and the host has the latest stable version of xen and qemu-xen-traditional. Using this environment, setting up a PCI device (using MSI/MSI-X interrupts) will not successfully finish.

OK, I'm sold on applying the patch to qemu-trad. Great.

Good :) It's useful to have it applied.

Quoted 8 lines
Quoted 3 lines

That's the link to the patch for qemu-xen-traditional:
https://lists.xenproject.org/archives/html/xen-devel/2018-06/msg01248.html

Right, thanks.

Quoted 1 line

(https://lists.xenproject.org/archives/html/xen-devel/2018-06/msg01373.html - added you in CC)

Quoted 7 lines

Unfortunately I don't want to fish the patch out of the mailing list archives (because the conversion from email to html and back etc. might mangle it) and the email with the CC to me didn't actually have the patch in it.

So can you please resend the backported patch to me ? Or point me at a git tree that contains the backported patch ?

Resent the patch, here it is on the list:
https://lists.xenproject.org/archives/html/xen-devel/2018-10/msg00937.html

Thanks for helping with this.

Andra

Quoted 3 lines

Thanks,
Ian.

Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#17 lore raw
In reply to David Woodhouse
Quoted 9 lines · David Woodhouse
Quoted 4 lines · Roger Pau Monné

Hm, I think I might have fixed this issue, see:

https://git.qemu.org/?p=qemu.git;a=commit;h=a8036336609d2e184fc3543a4c439c0ba7d7f3a2

Hm, thanks. We'll look at porting that change to qemu-traditional which still doesn't do it.

Backported patch to qemu-xen-traditional repo:
https://lists.xenproject.org/archives/html/xen-devel/2018-06/msg01248.html

Thank you.

Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel