Skip to contentSkip to conversation

Refactor msi/msix restore code Part2

PATCH Superseded Zhenzhong Duan 3 participants 18 messages Jul 24, 2013 – Aug 5, 2013 PCI

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 2/3 Refactor msi/msix restore code Part2

xen_initdom_restore_msi_irqs trigger a hypercall to restore addr/data/mask in dom0. It's better to do the same for default_restore_msi_irqs in baremetal.

Move restore of mask in default_restore_msi_irqs, this could avoid mask restored twice in dom0, once in hypercall, the other in kernel.

Without that, qlcnic driver calling pci_reset_function will lost interrupt in dom0.

---
 drivers/pci/msi.c |   17 ++++++++++++++---
 1 files changed, 14 insertions(+), 3 deletions(-)
drivers/pci/msi.c +14 -3
index 87223ae..922fb49 100644
@@ -216,6 +216,8 @@ void unmask_msi_irq(struct irq_data *data)
 #ifdef HAVE_DEFAULT_MSI_RESTORE_IRQS
 void default_restore_msi_irqs(struct pci_dev *dev, int irq)
 {
+	int pos;
+	u16 control;
 	struct msi_desc *entry;
 
 	entry = NULL;
@@ -228,8 +230,19 @@ void default_restore_msi_irqs(struct pci_dev *dev, int irq)
 		entry = irq_get_msi_desc(irq);
 	}
 
-	if (entry)
+	if (entry) {
 		write_msi_msg(irq, &entry->msg);
+		if (dev->msix_enabled) {
+			msix_mask_irq(entry, entry->masked);
+			readl(entry->mask_base);
+		} else {
+			pos = entry->msi_attrib.pos;
+			pci_read_config_word(dev, pos + PCI_MSI_FLAGS,
+					     &control);
+			msi_mask_irq(entry, msi_capable_mask(control),
+				     entry->masked);
+		}
+	}
 }
 #endif
 
@@ -406,7 +419,6 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
 	arch_restore_msi_irqs(dev, dev->irq);
 
 	pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
-	msi_mask_irq(entry, msi_capable_mask(control), entry->masked);
 	control &= ~PCI_MSI_FLAGS_QSIZE;
 	control |= (entry->msi_attrib.multiple << 4) | PCI_MSI_FLAGS_ENABLE;
 	pci_write_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, control);
@@ -430,7 +442,6 @@ static void __pci_restore_msix_state(struct pci_dev *dev)
 
 	list_for_each_entry(entry, &dev->msi_list, list) {
 		arch_restore_msi_irqs(dev, entry->irq);
-		msix_mask_irq(entry, entry->masked);
 	}
 
 	control &= ~PCI_MSIX_FLAGS_MASKALL;
Signature · 1 line
1.7.3
#1 lore raw
Quoted 5 lines · Zhenzhong Duan

xen_initdom_restore_msi_irqs trigger a hypercall to restore addr/data/mask in dom0. It's better to do the same for default_restore_msi_irqs in baremetal.

Move restore of mask in default_restore_msi_irqs, this could avoid mask restored twice in dom0, once in hypercall, the other in kernel.

Why not remove the hypercall then? Or alter the function to detect whether the restore of the mask has occurred?

Quoted 45 lines

Without that, qlcnic driver calling pci_reset_function will lost interrupt in dom0.

---
 drivers/pci/msi.c |   17 ++++++++++++++---
 1 files changed, 14 insertions(+), 3 deletions(-)
drivers/pci/msi.c +14 -1
index 87223ae..922fb49 100644
@@ -216,6 +216,8 @@ void unmask_msi_irq(struct irq_data *data)
 #ifdef HAVE_DEFAULT_MSI_RESTORE_IRQS
 void default_restore_msi_irqs(struct pci_dev *dev, int irq)
 {
+	int pos;
+	u16 control;
 	struct msi_desc *entry;
 
 	entry = NULL;
@@ -228,8 +230,19 @@ void default_restore_msi_irqs(struct pci_dev *dev, int irq)
 		entry = irq_get_msi_desc(irq);
 	}
 
-	if (entry)
+	if (entry) {
 		write_msi_msg(irq, &entry->msg);
+		if (dev->msix_enabled) {
+			msix_mask_irq(entry, entry->masked);
+			readl(entry->mask_base);
+		} else {
+			pos = entry->msi_attrib.pos;
+			pci_read_config_word(dev, pos + PCI_MSI_FLAGS,
+					     &control);
+			msi_mask_irq(entry, msi_capable_mask(control),
+				     entry->masked);
+		}
+	}
 }
 #endif
 

drivers/pci/msi.c static void __pci_restore_msi_state(struct pci_dev *dev) · line 409

Quoted 4 lines
drivers/pci/msi.c +0 -1
@@ -406,7 +419,6 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
 	arch_restore_msi_irqs(dev, dev->irq);
 
 	pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
-	msi_mask_irq(entry, msi_capable_mask(control), entry->masked);

Before this patch we had:

	write_msi_msg(..)
	pci_read_config_work(PCI_MSI_FLAGS, &control)
	pci_write_config_dword(~msi_capable_mask(control) | entry->masked)
	control &= ~_PCI_MSI_FLAGS_QSIZE;
	control |= ...
	pci_write_config_dword(PCI_MSI_FLAGS, control)

while with this you have now:

	write_msi_msg(..)
	pci_read_config_work(PCI_MSI_FLAGS, &_control)
	pci_write_config_dword(~msi_capable_mask(_control) | entry->masked)
--> 	pci_read_config_work(PCI_MSI_FLAGS, &control)
	control &= ~_PCI_MSI_FLAGS_QSIZE;
	control |= ...
	pci_write_config_dword(PCI_MSI_FLAGS, control)

see the problem? The 'control' value in __pci_restore_msi_state reads the value _after_ it has been masked (which is now done in default_restore_msi_irqs).

Wouldn't that cause problems?

Quoted 14 lines
drivers/pci/msi.c +0 -1
 	control &= ~PCI_MSI_FLAGS_QSIZE;
 	control |= (entry->msi_attrib.multiple << 4) | PCI_MSI_FLAGS_ENABLE;
 	pci_write_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, control);
@@ -430,7 +442,6 @@ static void __pci_restore_msix_state(struct pci_dev *dev)
 
 	list_for_each_entry(entry, &dev->msi_list, list) {
 		arch_restore_msi_irqs(dev, entry->irq);
-		msix_mask_irq(entry, entry->masked);
 	}
 
 	control &= ~PCI_MSIX_FLAGS_MASKALL;
Signature · 1 line
1.7.3
#2 lore raw
In reply to Zhenzhong Duan
Quoted 5 lines · Zhenzhong Duan

xen_initdom_restore_msi_irqs trigger a hypercall to restore addr/data/mask in dom0. It's better to do the same for default_restore_msi_irqs in baremetal.

Move restore of mask in default_restore_msi_irqs, this could avoid mask restored twice in dom0, once in hypercall, the other in kernel.

Why not remove the hypercall then? Or alter the function to detect whether the restore of the mask has occurred?

Quoted 45 lines

Without that, qlcnic driver calling pci_reset_function will lost interrupt in dom0.

---
 drivers/pci/msi.c |   17 ++++++++++++++---
 1 files changed, 14 insertions(+), 3 deletions(-)
drivers/pci/msi.c +14 -1
index 87223ae..922fb49 100644
@@ -216,6 +216,8 @@ void unmask_msi_irq(struct irq_data *data)
 #ifdef HAVE_DEFAULT_MSI_RESTORE_IRQS
 void default_restore_msi_irqs(struct pci_dev *dev, int irq)
 {
+	int pos;
+	u16 control;
 	struct msi_desc *entry;
 
 	entry = NULL;
@@ -228,8 +230,19 @@ void default_restore_msi_irqs(struct pci_dev *dev, int irq)
 		entry = irq_get_msi_desc(irq);
 	}
 
-	if (entry)
+	if (entry) {
 		write_msi_msg(irq, &entry->msg);
+		if (dev->msix_enabled) {
+			msix_mask_irq(entry, entry->masked);
+			readl(entry->mask_base);
+		} else {
+			pos = entry->msi_attrib.pos;
+			pci_read_config_word(dev, pos + PCI_MSI_FLAGS,
+					     &control);
+			msi_mask_irq(entry, msi_capable_mask(control),
+				     entry->masked);
+		}
+	}
 }
 #endif
 

drivers/pci/msi.c static void __pci_restore_msi_state(struct pci_dev *dev) · line 409

Quoted 4 lines
drivers/pci/msi.c +0 -1
@@ -406,7 +419,6 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
 	arch_restore_msi_irqs(dev, dev->irq);
 
 	pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
-	msi_mask_irq(entry, msi_capable_mask(control), entry->masked);

Before this patch we had:

	write_msi_msg(..)
	pci_read_config_work(PCI_MSI_FLAGS, &control)
	pci_write_config_dword(~msi_capable_mask(control) | entry->masked)
	control &= ~_PCI_MSI_FLAGS_QSIZE;
	control |= ...
	pci_write_config_dword(PCI_MSI_FLAGS, control)

while with this you have now:

	write_msi_msg(..)
	pci_read_config_work(PCI_MSI_FLAGS, &_control)
	pci_write_config_dword(~msi_capable_mask(_control) | entry->masked)
--> 	pci_read_config_work(PCI_MSI_FLAGS, &control)
	control &= ~_PCI_MSI_FLAGS_QSIZE;
	control |= ...
	pci_write_config_dword(PCI_MSI_FLAGS, control)

see the problem? The 'control' value in __pci_restore_msi_state reads the value _after_ it has been masked (which is now done in default_restore_msi_irqs).

Wouldn't that cause problems?

Quoted 14 lines
drivers/pci/msi.c +0 -1
 	control &= ~PCI_MSI_FLAGS_QSIZE;
 	control |= (entry->msi_attrib.multiple << 4) | PCI_MSI_FLAGS_ENABLE;
 	pci_write_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, control);
@@ -430,7 +442,6 @@ static void __pci_restore_msix_state(struct pci_dev *dev)
 
 	list_for_each_entry(entry, &dev->msi_list, list) {
 		arch_restore_msi_irqs(dev, entry->irq);
-		msix_mask_irq(entry, entry->masked);
 	}
 
 	control &= ~PCI_MSIX_FLAGS_MASKALL;
Signature · 1 line
1.7.3
#3 lore raw
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Zhenzhong Duan

xen_initdom_restore_msi_irqs trigger a hypercall to restore addr/data/mask in dom0. It's better to do the same for default_restore_msi_irqs in baremetal.

Move restore of mask in default_restore_msi_irqs, this could avoid mask restored twice in dom0, once in hypercall, the other in kernel.

Why not remove the hypercall then?

If removed, msi entry couldn't be restored, such as pci_reset_function who will reset pci registers.

Quoted 2 lines

Or alter the function to detect
whether the restore of the mask has occurred?

Then we need to add the check for dom0 only.

Quoted 68 lines
Quoted 45 lines

Without that, qlcnic driver calling pci_reset_function will lost interrupt in dom0.

---
  drivers/pci/msi.c |   17 ++++++++++++++---
  1 files changed, 14 insertions(+), 3 deletions(-)
drivers/pci/msi.c +14 -1
index 87223ae..922fb49 100644
@@ -216,6 +216,8 @@ void unmask_msi_irq(struct irq_data *data)
  #ifdef HAVE_DEFAULT_MSI_RESTORE_IRQS
  void default_restore_msi_irqs(struct pci_dev *dev, int irq)
  {
+	int pos;
+	u16 control;
  	struct msi_desc *entry;
 
  	entry = NULL;
@@ -228,8 +230,19 @@ void default_restore_msi_irqs(struct pci_dev *dev, int irq)
  		entry = irq_get_msi_desc(irq);
  	}
 
-	if (entry)
+	if (entry) {
  		write_msi_msg(irq, &entry->msg);
+		if (dev->msix_enabled) {
+			msix_mask_irq(entry, entry->masked);
+			readl(entry->mask_base);
+		} else {
+			pos = entry->msi_attrib.pos;
+			pci_read_config_word(dev, pos + PCI_MSI_FLAGS,
+					     &control);
+			msi_mask_irq(entry, msi_capable_mask(control),
+				     entry->masked);
+		}
+	}
  }
  #endif
 

drivers/pci/msi.c static void __pci_restore_msi_state(struct pci_dev *dev) · line 409

Quoted 4 lines
drivers/pci/msi.c +0 -1
@@ -406,7 +419,6 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
  	arch_restore_msi_irqs(dev, dev->irq);
 
  	pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
-	msi_mask_irq(entry, msi_capable_mask(control), entry->masked);

Before this patch we had:

	write_msi_msg(..)
	pci_read_config_work(PCI_MSI_FLAGS, &control)
	pci_write_config_dword(~msi_capable_mask(control) | entry->masked)
	control &= ~_PCI_MSI_FLAGS_QSIZE;
	control |= ...
	pci_write_config_dword(PCI_MSI_FLAGS, control)

while with this you have now:

	write_msi_msg(..)
	pci_read_config_work(PCI_MSI_FLAGS, &_control)
	pci_write_config_dword(~msi_capable_mask(_control) | entry->masked)
--> 	pci_read_config_work(PCI_MSI_FLAGS, &control)
	control &= ~_PCI_MSI_FLAGS_QSIZE;
	control |= ...
	pci_write_config_dword(PCI_MSI_FLAGS, control)
Quoted 5 lines

see the problem? The 'control' value in __pci_restore_msi_state reads the value _after_ it has been masked (which is now done in default_restore_msi_irqs).

Wouldn't that cause problems?

pci_write_config_dword(~msi_capable_mask(_control) | entry->masked) restore per vector msi mask bits based on the support of PCI_MSI_FLAGS_MASKBIT. This is different from the global mask bit in PCI_MSI_FLAGS.

#4 lore raw
In reply to Konrad Rzeszutek Wilk
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Zhenzhong Duan

xen_initdom_restore_msi_irqs trigger a hypercall to restore addr/data/mask in dom0. It's better to do the same for default_restore_msi_irqs in baremetal.

Move restore of mask in default_restore_msi_irqs, this could avoid mask restored twice in dom0, once in hypercall, the other in kernel.

Why not remove the hypercall then?

If removed, msi entry couldn't be restored, such as pci_reset_function who will reset pci registers.

Quoted 2 lines

Or alter the function to detect
whether the restore of the mask has occurred?

Then we need to add the check for dom0 only.

Quoted 68 lines
Quoted 45 lines

Without that, qlcnic driver calling pci_reset_function will lost interrupt in dom0.

---
  drivers/pci/msi.c |   17 ++++++++++++++---
  1 files changed, 14 insertions(+), 3 deletions(-)
drivers/pci/msi.c +14 -1
index 87223ae..922fb49 100644
@@ -216,6 +216,8 @@ void unmask_msi_irq(struct irq_data *data)
  #ifdef HAVE_DEFAULT_MSI_RESTORE_IRQS
  void default_restore_msi_irqs(struct pci_dev *dev, int irq)
  {
+	int pos;
+	u16 control;
  	struct msi_desc *entry;
 
  	entry = NULL;
@@ -228,8 +230,19 @@ void default_restore_msi_irqs(struct pci_dev *dev, int irq)
  		entry = irq_get_msi_desc(irq);
  	}
 
-	if (entry)
+	if (entry) {
  		write_msi_msg(irq, &entry->msg);
+		if (dev->msix_enabled) {
+			msix_mask_irq(entry, entry->masked);
+			readl(entry->mask_base);
+		} else {
+			pos = entry->msi_attrib.pos;
+			pci_read_config_word(dev, pos + PCI_MSI_FLAGS,
+					     &control);
+			msi_mask_irq(entry, msi_capable_mask(control),
+				     entry->masked);
+		}
+	}
  }
  #endif
 

drivers/pci/msi.c static void __pci_restore_msi_state(struct pci_dev *dev) · line 409

Quoted 4 lines
drivers/pci/msi.c +0 -1
@@ -406,7 +419,6 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
  	arch_restore_msi_irqs(dev, dev->irq);
 
  	pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
-	msi_mask_irq(entry, msi_capable_mask(control), entry->masked);

Before this patch we had:

	write_msi_msg(..)
	pci_read_config_work(PCI_MSI_FLAGS, &control)
	pci_write_config_dword(~msi_capable_mask(control) | entry->masked)
	control &= ~_PCI_MSI_FLAGS_QSIZE;
	control |= ...
	pci_write_config_dword(PCI_MSI_FLAGS, control)

while with this you have now:

	write_msi_msg(..)
	pci_read_config_work(PCI_MSI_FLAGS, &_control)
	pci_write_config_dword(~msi_capable_mask(_control) | entry->masked)
--> 	pci_read_config_work(PCI_MSI_FLAGS, &control)
	control &= ~_PCI_MSI_FLAGS_QSIZE;
	control |= ...
	pci_write_config_dword(PCI_MSI_FLAGS, control)
Quoted 5 lines

see the problem? The 'control' value in __pci_restore_msi_state reads the value _after_ it has been masked (which is now done in default_restore_msi_irqs).

Wouldn't that cause problems?

pci_write_config_dword(~msi_capable_mask(_control) | entry->masked) restore per vector msi mask bits based on the support of PCI_MSI_FLAGS_MASKBIT. This is different from the global mask bit in PCI_MSI_FLAGS.

#5 lore raw
Quoted 11 lines · Zhenzhong Duan
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Zhenzhong Duan

xen_initdom_restore_msi_irqs trigger a hypercall to restore addr/data/mask in dom0. It's better to do the same for default_restore_msi_irqs in baremetal.

Move restore of mask in default_restore_msi_irqs, this could avoid mask restored twice in dom0, once in hypercall, the other in kernel.

Why not remove the hypercall then?

If removed, msi entry couldn't be restored, such as pci_reset_function who will reset pci registers.

I did not read your email first time correctly. You are saying that we restore it twice in the host kernel (aka dom0), once in the hypervisor (b/c the guest tries to do MSI-X write and it ends up in the hypervisor), and then we also do it in the guest kernel?

That is a lot of duplicate calls.

Quoted 3 lines
Quoted 2 lines

Or alter the function to detect
whether the restore of the mask has occurred?

Then we need to add the check for dom0 only.

I am not sure I completly follow this. Is the reason for the lost of interrupt b/c one of those four MSI-X writes ends up masking and the subsequent writes end up with invalid data?

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Without that, qlcnic driver calling pci_reset_function will lost interrupt in dom0.

But if you pass said PCI device to a guest there is no need for the interrupts to go to the host (dom0). They should go to the hypervisor which will deliever them to the guest.

Is that what you meant by 'in dom0' ?

Quoted 61 lines
Quoted 61 lines
Quoted 39 lines
---
 drivers/pci/msi.c |   17 ++++++++++++++---
 1 files changed, 14 insertions(+), 3 deletions(-)
drivers/pci/msi.c +14 -1
index 87223ae..922fb49 100644
@@ -216,6 +216,8 @@ void unmask_msi_irq(struct irq_data *data)
 #ifdef HAVE_DEFAULT_MSI_RESTORE_IRQS
 void default_restore_msi_irqs(struct pci_dev *dev, int irq)
 {
+	int pos;
+	u16 control;
 	struct msi_desc *entry;
 	entry = NULL;
@@ -228,8 +230,19 @@ void default_restore_msi_irqs(struct pci_dev *dev, int irq)
 		entry = irq_get_msi_desc(irq);
 	}
-	if (entry)
+	if (entry) {
 		write_msi_msg(irq, &entry->msg);
+		if (dev->msix_enabled) {
+			msix_mask_irq(entry, entry->masked);
+			readl(entry->mask_base);
+		} else {
+			pos = entry->msi_attrib.pos;
+			pci_read_config_word(dev, pos + PCI_MSI_FLAGS,
+					     &control);
+			msi_mask_irq(entry, msi_capable_mask(control),
+				     entry->masked);
+		}
+	}
 }
 #endif

drivers/pci/msi.c static void __pci_restore_msi_state(struct pci_dev *dev) · line 408

Quoted 3 lines
drivers/pci/msi.c +0 -1
@@ -406,7 +419,6 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
 	arch_restore_msi_irqs(dev, dev->irq);
 	pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
-	msi_mask_irq(entry, msi_capable_mask(control), entry->masked);

Before this patch we had:

	write_msi_msg(..)
	pci_read_config_work(PCI_MSI_FLAGS, &control)
	pci_write_config_dword(~msi_capable_mask(control) | entry->masked)
	control &= ~_PCI_MSI_FLAGS_QSIZE;
	control |= ...
	pci_write_config_dword(PCI_MSI_FLAGS, control)

while with this you have now:

	write_msi_msg(..)
	pci_read_config_work(PCI_MSI_FLAGS, &_control)
	pci_write_config_dword(~msi_capable_mask(_control) | entry->masked)
--> 	pci_read_config_work(PCI_MSI_FLAGS, &control)
	control &= ~_PCI_MSI_FLAGS_QSIZE;
	control |= ...
	pci_write_config_dword(PCI_MSI_FLAGS, control)
Quoted 8 lines
Quoted 5 lines

see the problem? The 'control' value in __pci_restore_msi_state reads the value _after_ it has been masked (which is now done in default_restore_msi_irqs).

Wouldn't that cause problems?

pci_write_config_dword(~msi_capable_mask(_control) | entry->masked) restore per vector msi mask bits based on the support of PCI_MSI_FLAGS_MASKBIT. This is different from the global mask bit in PCI_MSI_FLAGS.

So I think you are saying that it won't cause problems?

Quoted 1 line
#6 lore raw
In reply to Zhenzhong Duan
Quoted 11 lines · Zhenzhong Duan
Quoted 7 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Zhenzhong Duan

xen_initdom_restore_msi_irqs trigger a hypercall to restore addr/data/mask in dom0. It's better to do the same for default_restore_msi_irqs in baremetal.

Move restore of mask in default_restore_msi_irqs, this could avoid mask restored twice in dom0, once in hypercall, the other in kernel.

Why not remove the hypercall then?

If removed, msi entry couldn't be restored, such as pci_reset_function who will reset pci registers.

I did not read your email first time correctly. You are saying that we restore it twice in the host kernel (aka dom0), once in the hypervisor (b/c the guest tries to do MSI-X write and it ends up in the hypervisor), and then we also do it in the guest kernel?

That is a lot of duplicate calls.

Quoted 3 lines
Quoted 2 lines

Or alter the function to detect
whether the restore of the mask has occurred?

Then we need to add the check for dom0 only.

I am not sure I completly follow this. Is the reason for the lost of interrupt b/c one of those four MSI-X writes ends up masking and the subsequent writes end up with invalid data?

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Without that, qlcnic driver calling pci_reset_function will lost interrupt in dom0.

But if you pass said PCI device to a guest there is no need for the interrupts to go to the host (dom0). They should go to the hypervisor which will deliever them to the guest.

Is that what you meant by 'in dom0' ?

Quoted 61 lines
Quoted 61 lines
Quoted 39 lines
---
 drivers/pci/msi.c |   17 ++++++++++++++---
 1 files changed, 14 insertions(+), 3 deletions(-)
drivers/pci/msi.c +14 -1
index 87223ae..922fb49 100644
@@ -216,6 +216,8 @@ void unmask_msi_irq(struct irq_data *data)
 #ifdef HAVE_DEFAULT_MSI_RESTORE_IRQS
 void default_restore_msi_irqs(struct pci_dev *dev, int irq)
 {
+	int pos;
+	u16 control;
 	struct msi_desc *entry;
 	entry = NULL;
@@ -228,8 +230,19 @@ void default_restore_msi_irqs(struct pci_dev *dev, int irq)
 		entry = irq_get_msi_desc(irq);
 	}
-	if (entry)
+	if (entry) {
 		write_msi_msg(irq, &entry->msg);
+		if (dev->msix_enabled) {
+			msix_mask_irq(entry, entry->masked);
+			readl(entry->mask_base);
+		} else {
+			pos = entry->msi_attrib.pos;
+			pci_read_config_word(dev, pos + PCI_MSI_FLAGS,
+					     &control);
+			msi_mask_irq(entry, msi_capable_mask(control),
+				     entry->masked);
+		}
+	}
 }
 #endif

drivers/pci/msi.c static void __pci_restore_msi_state(struct pci_dev *dev) · line 408

Quoted 3 lines
drivers/pci/msi.c +0 -1
@@ -406,7 +419,6 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
 	arch_restore_msi_irqs(dev, dev->irq);
 	pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
-	msi_mask_irq(entry, msi_capable_mask(control), entry->masked);

Before this patch we had:

	write_msi_msg(..)
	pci_read_config_work(PCI_MSI_FLAGS, &control)
	pci_write_config_dword(~msi_capable_mask(control) | entry->masked)
	control &= ~_PCI_MSI_FLAGS_QSIZE;
	control |= ...
	pci_write_config_dword(PCI_MSI_FLAGS, control)

while with this you have now:

	write_msi_msg(..)
	pci_read_config_work(PCI_MSI_FLAGS, &_control)
	pci_write_config_dword(~msi_capable_mask(_control) | entry->masked)
--> 	pci_read_config_work(PCI_MSI_FLAGS, &control)
	control &= ~_PCI_MSI_FLAGS_QSIZE;
	control |= ...
	pci_write_config_dword(PCI_MSI_FLAGS, control)
Quoted 8 lines
Quoted 5 lines

see the problem? The 'control' value in __pci_restore_msi_state reads the value _after_ it has been masked (which is now done in default_restore_msi_irqs).

Wouldn't that cause problems?

pci_write_config_dword(~msi_capable_mask(_control) | entry->masked) restore per vector msi mask bits based on the support of PCI_MSI_FLAGS_MASKBIT. This is different from the global mask bit in PCI_MSI_FLAGS.

So I think you are saying that it won't cause problems?

Quoted 1 line
#7 lore raw
Quoted 8 lines · Konrad Rzeszutek Wilk
Quoted 7 lines · Zhenzhong Duan
Quoted 6 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Zhenzhong Duan

xen_initdom_restore_msi_irqs trigger a hypercall to restore addr/data/mask in dom0. It's better to do the same for default_restore_msi_irqs in baremetal.

Move restore of mask in default_restore_msi_irqs, this could avoid mask restored twice in dom0, once in hypercall, the other in kernel.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Why not remove the hypercall then?

If removed, msi entry couldn't be restored, such as pci_reset_function who will reset pci registers.

I did not read your email first time correctly. You are saying that we restore it twice in the host kernel (aka dom0), once in the hypervisor (b/c the guest tries to do MSI-X write and it ends up in the hypervisor), and then we also do it in the guest kernel?

Non business of guest kernel, this patch is fixing driver load issue in dom0. Driver qlcnic called pci_reset_function during init. The call path:
pci_reset_function->pci_restore_state->__pci_restore_msix_state->arch_restore_msi_irqs-> xen_initdom_restore_msi_irqs->PHYSDEVOP_restore_msi hypercall

First mask restore is in xen_initdom_restore_msi_irqs->PHYSDEVOP_restore_msi hypercall Second restore is __pci_restore_msix_state->msix_mask_irq(entry, entry->masked)

Mask bits are under full control of xen, and the entry->masked in dom0 kernel is invalid. We restore an invalid value to mask register could mask the msix interrupt.

Quoted 8 lines

That is a lot of duplicate calls.

Quoted 3 lines
Quoted 2 lines

Or alter the function to detect
whether the restore of the mask has occurred?

Then we need to add the check for dom0 only.

I am not sure I completly follow this. Is the reason for the lost of interrupt b/c one of those four MSI-X writes ends up masking and the subsequent writes end up with invalid data?

The first restore in hypercall is needed, and second restore should be removed for dom0 But baremetal need that restore, so I move it in default_restore_msi_irqs which is the func for baremetal.

Quoted 72 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Without that, qlcnic driver calling pci_reset_function will lost interrupt in dom0.

But if you pass said PCI device to a guest there is no need for the interrupts to go to the host (dom0). They should go to the hypervisor which will deliever them to the guest.

Is that what you meant by 'in dom0' ?

Quoted 63 lines
Quoted 63 lines
Quoted 38 lines
---
  drivers/pci/msi.c |   17 ++++++++++++++---
  1 files changed, 14 insertions(+), 3 deletions(-)
drivers/pci/msi.c +14 -1
index 87223ae..922fb49 100644
@@ -216,6 +216,8 @@ void unmask_msi_irq(struct irq_data *data)
  #ifdef HAVE_DEFAULT_MSI_RESTORE_IRQS
  void default_restore_msi_irqs(struct pci_dev *dev, int irq)
  {
+	int pos;
+	u16 control;
  	struct msi_desc *entry;
  	entry = NULL;
@@ -228,8 +230,19 @@ void default_restore_msi_irqs(struct pci_dev *dev, int irq)
  		entry = irq_get_msi_desc(irq);
  	}
-	if (entry)
+	if (entry) {
  		write_msi_msg(irq, &entry->msg);
+		if (dev->msix_enabled) {
+			msix_mask_irq(entry, entry->masked);
+			readl(entry->mask_base);
+		} else {
+			pos = entry->msi_attrib.pos;
+			pci_read_config_word(dev, pos + PCI_MSI_FLAGS,
+					     &control);
+			msi_mask_irq(entry, msi_capable_mask(control),
+				     entry->masked);
+		}
+	}
  }
  #endif

drivers/pci/msi.c static void __pci_restore_msi_state(struct pci_dev *dev) · line 408

Quoted 3 lines
drivers/pci/msi.c +0 -1
@@ -406,7 +419,6 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
  	arch_restore_msi_irqs(dev, dev->irq);
  	pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
-	msi_mask_irq(entry, msi_capable_mask(control), entry->masked);

Before this patch we had:

	write_msi_msg(..)
	pci_read_config_work(PCI_MSI_FLAGS, &control)
	pci_write_config_dword(~msi_capable_mask(control) | entry->masked)
	control &= ~_PCI_MSI_FLAGS_QSIZE;
	control |= ...
	pci_write_config_dword(PCI_MSI_FLAGS, control)

while with this you have now:

	write_msi_msg(..)
	pci_read_config_work(PCI_MSI_FLAGS, &_control)
	pci_write_config_dword(~msi_capable_mask(_control) | entry->masked)
--> 	pci_read_config_work(PCI_MSI_FLAGS, &control)
	control &= ~_PCI_MSI_FLAGS_QSIZE;
	control |= ...
	pci_write_config_dword(PCI_MSI_FLAGS, control)

see the problem? The 'control' value in __pci_restore_msi_state reads the value _after_ it has been masked (which is now done in default_restore_msi_irqs).

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

Wouldn't that cause problems?

pci_write_config_dword(~msi_capable_mask(_control) | entry->masked) restore per vector msi mask bits based on the support of PCI_MSI_FLAGS_MASKBIT. This is different from the global mask bit in PCI_MSI_FLAGS.

So I think you are saying that it won't cause problems?

Yes

#8 lore raw
In reply to Konrad Rzeszutek Wilk
Quoted 8 lines · Konrad Rzeszutek Wilk
Quoted 7 lines · Zhenzhong Duan
Quoted 6 lines · Konrad Rzeszutek Wilk
Quoted 5 lines · Zhenzhong Duan

xen_initdom_restore_msi_irqs trigger a hypercall to restore addr/data/mask in dom0. It's better to do the same for default_restore_msi_irqs in baremetal.

Move restore of mask in default_restore_msi_irqs, this could avoid mask restored twice in dom0, once in hypercall, the other in kernel.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Why not remove the hypercall then?

If removed, msi entry couldn't be restored, such as pci_reset_function who will reset pci registers.

I did not read your email first time correctly. You are saying that we restore it twice in the host kernel (aka dom0), once in the hypervisor (b/c the guest tries to do MSI-X write and it ends up in the hypervisor), and then we also do it in the guest kernel?

Non business of guest kernel, this patch is fixing driver load issue in dom0. Driver qlcnic called pci_reset_function during init. The call path:
pci_reset_function->pci_restore_state->__pci_restore_msix_state->arch_restore_msi_irqs-> xen_initdom_restore_msi_irqs->PHYSDEVOP_restore_msi hypercall

First mask restore is in xen_initdom_restore_msi_irqs->PHYSDEVOP_restore_msi hypercall Second restore is __pci_restore_msix_state->msix_mask_irq(entry, entry->masked)

Mask bits are under full control of xen, and the entry->masked in dom0 kernel is invalid. We restore an invalid value to mask register could mask the msix interrupt.

Quoted 8 lines

That is a lot of duplicate calls.

Quoted 3 lines
Quoted 2 lines

Or alter the function to detect
whether the restore of the mask has occurred?

Then we need to add the check for dom0 only.

I am not sure I completly follow this. Is the reason for the lost of interrupt b/c one of those four MSI-X writes ends up masking and the subsequent writes end up with invalid data?

The first restore in hypercall is needed, and second restore should be removed for dom0 But baremetal need that restore, so I move it in default_restore_msi_irqs which is the func for baremetal.

Quoted 72 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Without that, qlcnic driver calling pci_reset_function will lost interrupt in dom0.

But if you pass said PCI device to a guest there is no need for the interrupts to go to the host (dom0). They should go to the hypervisor which will deliever them to the guest.

Is that what you meant by 'in dom0' ?

Quoted 63 lines
Quoted 63 lines
Quoted 38 lines
---
  drivers/pci/msi.c |   17 ++++++++++++++---
  1 files changed, 14 insertions(+), 3 deletions(-)
drivers/pci/msi.c +14 -1
index 87223ae..922fb49 100644
@@ -216,6 +216,8 @@ void unmask_msi_irq(struct irq_data *data)
  #ifdef HAVE_DEFAULT_MSI_RESTORE_IRQS
  void default_restore_msi_irqs(struct pci_dev *dev, int irq)
  {
+	int pos;
+	u16 control;
  	struct msi_desc *entry;
  	entry = NULL;
@@ -228,8 +230,19 @@ void default_restore_msi_irqs(struct pci_dev *dev, int irq)
  		entry = irq_get_msi_desc(irq);
  	}
-	if (entry)
+	if (entry) {
  		write_msi_msg(irq, &entry->msg);
+		if (dev->msix_enabled) {
+			msix_mask_irq(entry, entry->masked);
+			readl(entry->mask_base);
+		} else {
+			pos = entry->msi_attrib.pos;
+			pci_read_config_word(dev, pos + PCI_MSI_FLAGS,
+					     &control);
+			msi_mask_irq(entry, msi_capable_mask(control),
+				     entry->masked);
+		}
+	}
  }
  #endif

drivers/pci/msi.c static void __pci_restore_msi_state(struct pci_dev *dev) · line 408

Quoted 3 lines
drivers/pci/msi.c +0 -1
@@ -406,7 +419,6 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
  	arch_restore_msi_irqs(dev, dev->irq);
  	pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
-	msi_mask_irq(entry, msi_capable_mask(control), entry->masked);

Before this patch we had:

	write_msi_msg(..)
	pci_read_config_work(PCI_MSI_FLAGS, &control)
	pci_write_config_dword(~msi_capable_mask(control) | entry->masked)
	control &= ~_PCI_MSI_FLAGS_QSIZE;
	control |= ...
	pci_write_config_dword(PCI_MSI_FLAGS, control)

while with this you have now:

	write_msi_msg(..)
	pci_read_config_work(PCI_MSI_FLAGS, &_control)
	pci_write_config_dword(~msi_capable_mask(_control) | entry->masked)
--> 	pci_read_config_work(PCI_MSI_FLAGS, &control)
	control &= ~_PCI_MSI_FLAGS_QSIZE;
	control |= ...
	pci_write_config_dword(PCI_MSI_FLAGS, control)

see the problem? The 'control' value in __pci_restore_msi_state reads the value _after_ it has been masked (which is now done in default_restore_msi_irqs).

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

Wouldn't that cause problems?

pci_write_config_dword(~msi_capable_mask(_control) | entry->masked) restore per vector msi mask bits based on the support of PCI_MSI_FLAGS_MASKBIT. This is different from the global mask bit in PCI_MSI_FLAGS.

So I think you are saying that it won't cause problems?

Yes

maintainer
#9 lore raw
In reply to Zhenzhong Duan
Quoted 58 lines · Zhenzhong Duan

xen_initdom_restore_msi_irqs trigger a hypercall to restore addr/data/mask in dom0. It's better to do the same for default_restore_msi_irqs in baremetal.

Move restore of mask in default_restore_msi_irqs, this could avoid mask restored twice in dom0, once in hypercall, the other in kernel.

Without that, qlcnic driver calling pci_reset_function will lost interrupt in dom0.

---
 drivers/pci/msi.c |   17 ++++++++++++++---
 1 files changed, 14 insertions(+), 3 deletions(-)
drivers/pci/msi.c +14 -2
index 87223ae..922fb49 100644
@@ -216,6 +216,8 @@ void unmask_msi_irq(struct irq_data *data)
 #ifdef HAVE_DEFAULT_MSI_RESTORE_IRQS
 void default_restore_msi_irqs(struct pci_dev *dev, int irq)
 {
+       int pos;
+       u16 control;
        struct msi_desc *entry;
 
        entry = NULL;
@@ -228,8 +230,19 @@ void default_restore_msi_irqs(struct pci_dev *dev, int irq)
                entry = irq_get_msi_desc(irq);
        }
 
-       if (entry)
+       if (entry) {
                write_msi_msg(irq, &entry->msg);
+               if (dev->msix_enabled) {
+                       msix_mask_irq(entry, entry->masked);
+                       readl(entry->mask_base);
+               } else {
+                       pos = entry->msi_attrib.pos;
+                       pci_read_config_word(dev, pos + PCI_MSI_FLAGS,
+                                            &control);
+                       msi_mask_irq(entry, msi_capable_mask(control),
+                                    entry->masked);
+               }
+       }
 }
 #endif
 
@@ -406,7 +419,6 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
        arch_restore_msi_irqs(dev, dev->irq);
 
        pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
-       msi_mask_irq(entry, msi_capable_mask(control), entry->masked);
        control &= ~PCI_MSI_FLAGS_QSIZE;
        control |= (entry->msi_attrib.multiple << 4) | PCI_MSI_FLAGS_ENABLE;
        pci_write_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, control);

drivers/pci/msi.c static void __pci_restore_msix_state(struct pci_dev *dev) · line 447

Quoted 7 lines
drivers/pci/msi.c +0 -1
@@ -430,7 +442,6 @@ static void __pci_restore_msix_state(struct pci_dev *dev)
 
        list_for_each_entry(entry, &dev->msi_list, list) {
                arch_restore_msi_irqs(dev, entry->irq);
-               msix_mask_irq(entry, entry->masked);
        }
 
        control &= ~PCI_MSIX_FLAGS_MASKALL;

Konrad, are you OK with this patch now?

If so, can you (Zhenzhong) update the changelog with some of the info from the conversation with Konrad? It seems like a few more details are needed to make it clear why we need this change.

Bjorn

maintainer
#10 lore raw
In reply to Zhenzhong Duan
Quoted 58 lines · Zhenzhong Duan

xen_initdom_restore_msi_irqs trigger a hypercall to restore addr/data/mask in dom0. It's better to do the same for default_restore_msi_irqs in baremetal.

Move restore of mask in default_restore_msi_irqs, this could avoid mask restored twice in dom0, once in hypercall, the other in kernel.

Without that, qlcnic driver calling pci_reset_function will lost interrupt in dom0.

---
 drivers/pci/msi.c |   17 ++++++++++++++---
 1 files changed, 14 insertions(+), 3 deletions(-)
drivers/pci/msi.c +14 -2
index 87223ae..922fb49 100644
@@ -216,6 +216,8 @@ void unmask_msi_irq(struct irq_data *data)
 #ifdef HAVE_DEFAULT_MSI_RESTORE_IRQS
 void default_restore_msi_irqs(struct pci_dev *dev, int irq)
 {
+       int pos;
+       u16 control;
        struct msi_desc *entry;
 
        entry = NULL;
@@ -228,8 +230,19 @@ void default_restore_msi_irqs(struct pci_dev *dev, int irq)
                entry = irq_get_msi_desc(irq);
        }
 
-       if (entry)
+       if (entry) {
                write_msi_msg(irq, &entry->msg);
+               if (dev->msix_enabled) {
+                       msix_mask_irq(entry, entry->masked);
+                       readl(entry->mask_base);
+               } else {
+                       pos = entry->msi_attrib.pos;
+                       pci_read_config_word(dev, pos + PCI_MSI_FLAGS,
+                                            &control);
+                       msi_mask_irq(entry, msi_capable_mask(control),
+                                    entry->masked);
+               }
+       }
 }
 #endif
 
@@ -406,7 +419,6 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
        arch_restore_msi_irqs(dev, dev->irq);
 
        pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
-       msi_mask_irq(entry, msi_capable_mask(control), entry->masked);
        control &= ~PCI_MSI_FLAGS_QSIZE;
        control |= (entry->msi_attrib.multiple << 4) | PCI_MSI_FLAGS_ENABLE;
        pci_write_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, control);

drivers/pci/msi.c static void __pci_restore_msix_state(struct pci_dev *dev) · line 447

Quoted 7 lines
drivers/pci/msi.c +0 -1
@@ -430,7 +442,6 @@ static void __pci_restore_msix_state(struct pci_dev *dev)
 
        list_for_each_entry(entry, &dev->msi_list, list) {
                arch_restore_msi_irqs(dev, entry->irq);
-               msix_mask_irq(entry, entry->masked);
        }
 
        control &= ~PCI_MSIX_FLAGS_MASKALL;

Konrad, are you OK with this patch now?

If so, can you (Zhenzhong) update the changelog with some of the info from the conversation with Konrad? It seems like a few more details are needed to make it clear why we need this change.

Bjorn

#11 lore raw
Quoted 61 lines · Bjorn Helgaas
Quoted 59 lines · Zhenzhong Duan

xen_initdom_restore_msi_irqs trigger a hypercall to restore addr/data/mask in dom0. It's better to do the same for default_restore_msi_irqs in baremetal.

Move restore of mask in default_restore_msi_irqs, this could avoid mask restored twice in dom0, once in hypercall, the other in kernel.

Without that, qlcnic driver calling pci_reset_function will lost interrupt in dom0.

---
 drivers/pci/msi.c |   17 ++++++++++++++---
 1 files changed, 14 insertions(+), 3 deletions(-)
drivers/pci/msi.c +14 -2
index 87223ae..922fb49 100644
@@ -216,6 +216,8 @@ void unmask_msi_irq(struct irq_data *data)
 #ifdef HAVE_DEFAULT_MSI_RESTORE_IRQS
 void default_restore_msi_irqs(struct pci_dev *dev, int irq)
 {
+       int pos;
+       u16 control;
        struct msi_desc *entry;
 
        entry = NULL;
@@ -228,8 +230,19 @@ void default_restore_msi_irqs(struct pci_dev *dev, int irq)
                entry = irq_get_msi_desc(irq);
        }
 
-       if (entry)
+       if (entry) {
                write_msi_msg(irq, &entry->msg);
+               if (dev->msix_enabled) {
+                       msix_mask_irq(entry, entry->masked);
+                       readl(entry->mask_base);
+               } else {
+                       pos = entry->msi_attrib.pos;
+                       pci_read_config_word(dev, pos + PCI_MSI_FLAGS,
+                                            &control);
+                       msi_mask_irq(entry, msi_capable_mask(control),
+                                    entry->masked);
+               }
+       }
 }
 #endif
 
@@ -406,7 +419,6 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
        arch_restore_msi_irqs(dev, dev->irq);
 
        pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
-       msi_mask_irq(entry, msi_capable_mask(control), entry->masked);
        control &= ~PCI_MSI_FLAGS_QSIZE;
        control |= (entry->msi_attrib.multiple << 4) | PCI_MSI_FLAGS_ENABLE;
        pci_write_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, control);
@@ -430,7 +442,6 @@ static void __pci_restore_msix_state(struct pci_dev *dev)
 
Quoted 8 lines

drivers/pci/msi.c static void __pci_restore_msix_state(struct pci_dev *dev) · line 447

Quoted 6 lines
drivers/pci/msi.c +0 -1
        list_for_each_entry(entry, &dev->msi_list, list) {
                arch_restore_msi_irqs(dev, entry->irq);
-               msix_mask_irq(entry, entry->masked);
        }
 
        control &= ~PCI_MSIX_FLAGS_MASKALL;

Konrad, are you OK with this patch now?

Yes, I wanted to make sure that the existing behavior under baremetal was not altered.

And Duan had confirmed it was not - thought to be on a safe side it would be good to confirm this via testing just in case.

Quoted 6 lines

If so, can you (Zhenzhong) update the changelog with some of the info from the conversation with Konrad? It seems like a few more details are needed to make it clear why we need this change.

Bjorn

#12 lore raw
In reply to Bjorn Helgaas
Quoted 61 lines · Bjorn Helgaas
Quoted 59 lines · Zhenzhong Duan

xen_initdom_restore_msi_irqs trigger a hypercall to restore addr/data/mask in dom0. It's better to do the same for default_restore_msi_irqs in baremetal.

Move restore of mask in default_restore_msi_irqs, this could avoid mask restored twice in dom0, once in hypercall, the other in kernel.

Without that, qlcnic driver calling pci_reset_function will lost interrupt in dom0.

---
 drivers/pci/msi.c |   17 ++++++++++++++---
 1 files changed, 14 insertions(+), 3 deletions(-)
drivers/pci/msi.c +14 -2
index 87223ae..922fb49 100644
@@ -216,6 +216,8 @@ void unmask_msi_irq(struct irq_data *data)
 #ifdef HAVE_DEFAULT_MSI_RESTORE_IRQS
 void default_restore_msi_irqs(struct pci_dev *dev, int irq)
 {
+       int pos;
+       u16 control;
        struct msi_desc *entry;
 
        entry = NULL;
@@ -228,8 +230,19 @@ void default_restore_msi_irqs(struct pci_dev *dev, int irq)
                entry = irq_get_msi_desc(irq);
        }
 
-       if (entry)
+       if (entry) {
                write_msi_msg(irq, &entry->msg);
+               if (dev->msix_enabled) {
+                       msix_mask_irq(entry, entry->masked);
+                       readl(entry->mask_base);
+               } else {
+                       pos = entry->msi_attrib.pos;
+                       pci_read_config_word(dev, pos + PCI_MSI_FLAGS,
+                                            &control);
+                       msi_mask_irq(entry, msi_capable_mask(control),
+                                    entry->masked);
+               }
+       }
 }
 #endif
 
@@ -406,7 +419,6 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
        arch_restore_msi_irqs(dev, dev->irq);
 
        pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
-       msi_mask_irq(entry, msi_capable_mask(control), entry->masked);
        control &= ~PCI_MSI_FLAGS_QSIZE;
        control |= (entry->msi_attrib.multiple << 4) | PCI_MSI_FLAGS_ENABLE;
        pci_write_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, control);
@@ -430,7 +442,6 @@ static void __pci_restore_msix_state(struct pci_dev *dev)
 
Quoted 8 lines

drivers/pci/msi.c static void __pci_restore_msix_state(struct pci_dev *dev) · line 447

Quoted 6 lines
drivers/pci/msi.c +0 -1
        list_for_each_entry(entry, &dev->msi_list, list) {
                arch_restore_msi_irqs(dev, entry->irq);
-               msix_mask_irq(entry, entry->masked);
        }
 
        control &= ~PCI_MSIX_FLAGS_MASKALL;

Konrad, are you OK with this patch now?

Yes, I wanted to make sure that the existing behavior under baremetal was not altered.

And Duan had confirmed it was not - thought to be on a safe side it would be good to confirm this via testing just in case.

Quoted 6 lines

If so, can you (Zhenzhong) update the changelog with some of the info from the conversation with Konrad? It seems like a few more details are needed to make it clear why we need this change.

Bjorn

#13 lore raw
Quoted 67 lines · Konrad Rzeszutek Wilk
Quoted 66 lines · Bjorn Helgaas
Quoted 64 lines · Zhenzhong Duan

xen_initdom_restore_msi_irqs trigger a hypercall to restore addr/data/mask in dom0. It's better to do the same for default_restore_msi_irqs in baremetal.

Move restore of mask in default_restore_msi_irqs, this could avoid mask restored twice in dom0, once in hypercall, the other in kernel.

Without that, qlcnic driver calling pci_reset_function will lost interrupt in dom0.

---
  drivers/pci/msi.c |   17 ++++++++++++++---
  1 files changed, 14 insertions(+), 3 deletions(-)
drivers/pci/msi.c +14 -3
index 87223ae..922fb49 100644
@@ -216,6 +216,8 @@ void unmask_msi_irq(struct irq_data *data)
  #ifdef HAVE_DEFAULT_MSI_RESTORE_IRQS
  void default_restore_msi_irqs(struct pci_dev *dev, int irq)
  {
+       int pos;
+       u16 control;
         struct msi_desc *entry;
 
         entry = NULL;
@@ -228,8 +230,19 @@ void default_restore_msi_irqs(struct pci_dev *dev, int irq)
                 entry = irq_get_msi_desc(irq);
         }
 
-       if (entry)
+       if (entry) {
                 write_msi_msg(irq, &entry->msg);
+               if (dev->msix_enabled) {
+                       msix_mask_irq(entry, entry->masked);
+                       readl(entry->mask_base);
+               } else {
+                       pos = entry->msi_attrib.pos;
+                       pci_read_config_word(dev, pos + PCI_MSI_FLAGS,
+                                            &control);
+                       msi_mask_irq(entry, msi_capable_mask(control),
+                                    entry->masked);
+               }
+       }
  }
  #endif
 
@@ -406,7 +419,6 @@ static void __pci_restore_msi_state(struct pci_dev *dev)
         arch_restore_msi_irqs(dev, dev->irq);
 
         pci_read_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, &control);
-       msi_mask_irq(entry, msi_capable_mask(control), entry->masked);
         control &= ~PCI_MSI_FLAGS_QSIZE;
         control |= (entry->msi_attrib.multiple << 4) | PCI_MSI_FLAGS_ENABLE;
         pci_write_config_word(dev, dev->msi_cap + PCI_MSI_FLAGS, control);
@@ -430,7 +442,6 @@ static void __pci_restore_msix_state(struct pci_dev *dev)
 
         list_for_each_entry(entry, &dev->msi_list, list) {
                 arch_restore_msi_irqs(dev, entry->irq);
-               msix_mask_irq(entry, entry->masked);
         }
 
Quoted 7 lines
Quoted 2 lines

drivers/pci/msi.c static void __pci_restore_msix_state(struct pci_dev *dev) · line 447

Quoted 1 line
drivers/pci/msi.c +0 -0
         control &= ~PCI_MSIX_FLAGS_MASKALL;

Konrad, are you OK with this patch now?

Yes, I wanted to make sure that the existing behavior under baremetal was not altered.

And Duan had confirmed it was not - thought to be on a safe side it would be good to confirm this via testing just in case.

I had ever let our customer test on baremetal and it passed.

zduan

#14 lore raw
Quoted 7 lines
Quoted 6 lines
Quoted 1 line

Konrad, are you OK with this patch now?

Yes, I wanted to make sure that the existing behavior under baremetal was not altered.

And Duan had confirmed it was not - thought to be on a safe side it would be good to confirm this via testing just in case.

I had ever let our customer test on baremetal and it passed.

Are they OK with being credited for this in the patch? Meaning are they OK with Reported-by-and-Tested-by: flag?

Either way, I think the only remaining issue is to expand the git commit a bit to include the discussion about the flow of the pci_read/pci_write and point out that the logic remains the same.

Quoted 2 lines

zduan

#15 lore raw
In reply to Zhenzhong Duan
Quoted 7 lines
Quoted 6 lines
Quoted 1 line

Konrad, are you OK with this patch now?

Yes, I wanted to make sure that the existing behavior under baremetal was not altered.

And Duan had confirmed it was not - thought to be on a safe side it would be good to confirm this via testing just in case.

I had ever let our customer test on baremetal and it passed.

Are they OK with being credited for this in the patch? Meaning are they OK with Reported-by-and-Tested-by: flag?

Either way, I think the only remaining issue is to expand the git commit a bit to include the discussion about the flow of the pci_read/pci_write and point out that the logic remains the same.

Quoted 2 lines

zduan

#16 lore raw
Quoted 9 lines · Konrad Rzeszutek Wilk
Quoted 7 lines
Quoted 6 lines
Quoted 1 line

Konrad, are you OK with this patch now?

Yes, I wanted to make sure that the existing behavior under baremetal was not altered.

And Duan had confirmed it was not - thought to be on a safe side it would be good to confirm this via testing just in case.

I had ever let our customer test on baremetal and it passed.

Are they OK with being credited for this in the patch? Meaning are they OK with Reported-by-and-Tested-by: flag?

sucheta.chakraborty from Qlogic also reproduced the same and I got his confirm patch worked. Already add him in tested-by.

zduan

Quoted 7 lines

Either way, I think the only remaining issue is to expand the git commit a bit to include the discussion about the flow of the pci_read/pci_write and point out that the logic remains the same.

Quoted 1 line

zduan

#17 lore raw
In reply to Konrad Rzeszutek Wilk
Quoted 9 lines · Konrad Rzeszutek Wilk
Quoted 7 lines
Quoted 6 lines
Quoted 1 line

Konrad, are you OK with this patch now?

Yes, I wanted to make sure that the existing behavior under baremetal was not altered.

And Duan had confirmed it was not - thought to be on a safe side it would be good to confirm this via testing just in case.

I had ever let our customer test on baremetal and it passed.

Are they OK with being credited for this in the patch? Meaning are they OK with Reported-by-and-Tested-by: flag?

sucheta.chakraborty from Qlogic also reproduced the same and I got his confirm patch worked. Already add him in tested-by.

zduan

Quoted 7 lines

Either way, I think the only remaining issue is to expand the git commit a bit to include the discussion about the flow of the pci_read/pci_write and point out that the logic remains the same.

Quoted 1 line

zduan