Skip to contentSkip to conversation

PCI: vmd: Add AHCI to fast interrupt list

PATCH In review Jon Derrick 4 participants 6 messages Sep 4, 2020 – Oct 20, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 PCI: vmd: Add AHCI to fast interrupt list

Some platforms have an AHCI controller behind VMD. These platforms are working correctly except for a case when the AHCI MSI is programmed with VMD IRQ vector 0 (0xfee00000). When programmed with any other interrupt (0xfeeNN000), the MSI is routed correctly and is handled by VMD. Placing the AHCI MSI(s) in the fast-interrupt allow list solves the issue.

This also requires that VMD allocate more than one MSI/X vector and changes the minimum MSI/X vectors allocated to two.

---
 drivers/pci/controller/vmd.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
drivers/pci/controller/vmd.c +3 -5
index f69ef8c89f72..d9c72613082a 100644
@@ -202,15 +202,13 @@ static struct vmd_irq_list *vmd_next_irq(struct vmd_dev *vmd, struct msi_desc *d
 	int i, best = 1;
 	unsigned long flags;
 
-	if (vmd->msix_count == 1)
-		return &vmd->irqs[0];
-
 	/*
-	 * White list for fast-interrupt handlers. All others will share the
+	 * Allow list for fast-interrupt handlers. All others will share the
 	 * "slow" interrupt vector.
 	 */
 	switch (msi_desc_to_pci_dev(desc)->class) {
 	case PCI_CLASS_STORAGE_EXPRESS:
+	case PCI_CLASS_STORAGE_SATA_AHCI:
 		break;
 	default:
 		return &vmd->irqs[0];
@@ -657,7 +655,7 @@ static int vmd_probe(struct pci_dev *dev, const struct pci_device_id *id)
 	if (vmd->msix_count < 0)
 		return -ENODEV;
 
-	vmd->msix_count = pci_alloc_irq_vectors(dev, 1, vmd->msix_count,
+	vmd->msix_count = pci_alloc_irq_vectors(dev, 2, vmd->msix_count,
 					PCI_IRQ_MSIX);
 	if (vmd->msix_count < 0)
 		return vmd->msix_count;
Signature · 1 line
2.17.1
#1 lore raw
Quoted 10 lines · Jon Derrick

Some platforms have an AHCI controller behind VMD. These platforms are working correctly except for a case when the AHCI MSI is programmed with VMD IRQ vector 0 (0xfee00000). When programmed with any other interrupt (0xfeeNN000), the MSI is routed correctly and is handled by VMD. Placing the AHCI MSI(s) in the fast-interrupt allow list solves the issue.

This also requires that VMD allocate more than one MSI/X vector and changes the minimum MSI/X vectors allocated to two.

Verified two platforms with such configuration. Thank you.

You-Sheng Yang

Quoted 36 lines
---
 drivers/pci/controller/vmd.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
drivers/pci/controller/vmd.c +3 -5
index f69ef8c89f72..d9c72613082a 100644
@@ -202,15 +202,13 @@ static struct vmd_irq_list *vmd_next_irq(struct vmd_dev *vmd, struct msi_desc *d
 	int i, best = 1;
 	unsigned long flags;
 
-	if (vmd->msix_count == 1)
-		return &vmd->irqs[0];
-
 	/*
-	 * White list for fast-interrupt handlers. All others will share the
+	 * Allow list for fast-interrupt handlers. All others will share the
 	 * "slow" interrupt vector.
 	 */
 	switch (msi_desc_to_pci_dev(desc)->class) {
 	case PCI_CLASS_STORAGE_EXPRESS:
+	case PCI_CLASS_STORAGE_SATA_AHCI:
 		break;
 	default:
 		return &vmd->irqs[0];
@@ -657,7 +655,7 @@ static int vmd_probe(struct pci_dev *dev, const struct pci_device_id *id)
 	if (vmd->msix_count < 0)
 		return -ENODEV;
 
-	vmd->msix_count = pci_alloc_irq_vectors(dev, 1, vmd->msix_count,
+	vmd->msix_count = pci_alloc_irq_vectors(dev, 2, vmd->msix_count,
 					PCI_IRQ_MSIX);
 	if (vmd->msix_count < 0)
 		return vmd->msix_count;
#2 lore raw
In reply to Jon Derrick
Quoted 5 lines · Jon Derrick

Some platforms have an AHCI controller behind VMD. These platforms are working correctly except for a case when the AHCI MSI is programmed with VMD IRQ vector 0 (0xfee00000). When programmed with any other interrupt (0xfeeNN000), the MSI is routed correctly and is handled by VMD. Placing the AHCI MSI(s) in the fast-interrupt allow list solves the issue.

The only reason we have the fast vs. slow is because of the non-posted transactions from slow driver's interrupt handlers tanking performance of the nvme devices sharing the same vector. The microsemi switchtec was one of the first such devices identified that led to the current split in the VMD domain. AHCI's driver also has non-posted transactions in their interrupt handling, so you probably don't want those devices sharing vectors with your fast nvme devices.

#3 lore raw
Quoted 7 lines · Keith Busch
Quoted 5 lines · Jon Derrick

Some platforms have an AHCI controller behind VMD. These platforms are working correctly except for a case when the AHCI MSI is programmed with VMD IRQ vector 0 (0xfee00000). When programmed with any other interrupt (0xfeeNN000), the MSI is routed correctly and is handled by VMD. Placing the AHCI MSI(s) in the fast-interrupt allow list solves the issue.

Quoted 7 lines

The only reason we have the fast vs. slow is because of the non-posted transactions from slow driver's interrupt handlers tanking performance of the nvme devices sharing the same vector. The microsemi switchtec was one of the first such devices identified that led to the current split in the VMD domain. AHCI's driver also has non-posted transactions in their interrupt handling, so you probably don't want those devices sharing vectors with your fast nvme devices.

Good points and I think this would be simply resolved by offsetting the starting vector and avoiding vector 0 for all vectors.

Thanks

Tested-by on v1 1/1
#4 lore raw
In reply to Jon Derrick
Quoted 7 lines · Jon Derrick
Quoted 6 lines

Some platforms have an AHCI controller behind VMD. These platforms are working correctly except for a case when the AHCI MSI is programmed with VMD IRQ vector 0 (0xfee00000). When programmed with any other interrupt (0xfeeNN000), the MSI is routed correctly and is handled by VMD. Placing the AHCI MSI(s) in the fast-interrupt allow list solves the issue.

Quoted 8 lines
Quoted 4 lines

This also requires that VMD allocate more than one MSI/X vector and changes the minimum MSI/X vectors allocated to two.

Verified two platforms with such configuration. Thank you.

You-Sheng Yang

We have some laptops equipped with Tiger Lake chips and such configuration that hit the same issue, too. They all could be fixed by this patch. Thank you

#5 lore raw
Quoted 12 lines · Jian-Hong Pan
Quoted 12 lines · Jon Derrick
Quoted 10 lines

Some platforms have an AHCI controller behind VMD. These platforms are working correctly except for a case when the AHCI MSI is programmed with VMD IRQ vector 0 (0xfee00000). When programmed with any other interrupt (0xfeeNN000), the MSI is routed correctly and is handled by VMD. Placing the AHCI MSI(s) in the fast-interrupt allow list solves the issue.

This also requires that VMD allocate more than one MSI/X vector and changes the minimum MSI/X vectors allocated to two.

Quoted 8 lines
Quoted 3 lines

Verified two platforms with such configuration. Thank you.

You-Sheng Yang

We have some laptops equipped with Tiger Lake chips and such configuration that hit the same issue, too. They all could be fixed by this patch. Thank you

Please use the v3 instead:

https://patchwork.kernel.org/project/linux-pci/patch/[email protected]/

I'm not sure if there's anything remaining that's gating inclusion, though.