Skip to contentSkip to conversation

Remove PCIE root bridge LSI on powernv

PATCH Superseded Frederic Barrat 3 participants 9 messages Mar 21, 2022 – Apr 6, 2022

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Remove PCIE root bridge LSI on powernv

The powernv8/powernv9/powernv10 machines allocate a LSI for their root port bridge, which is not the case on real hardware. The default root port implementation in qemu requests a LSI. Since the powernv implementation derives from it, that's where the LSI is coming from. This series fixes it, so that the model matches the hardware.

However, the code in hw/pci to handle AER and hotplug events assume a LSI is defined. It tends to assert/deassert a LSI if MSI or MSIX is not enabled. Since we have hardware where that is not true, this patch also fixes a few code paths to check if a LSI is configured before trying to trigger it.

Frederic Barrat (2):
pcie: Don't try triggering a LSI when not defined ppc/pnv: Remove LSI on the PCIE host bridge

 hw/pci-host/pnv_phb3.c | 1 +
 hw/pci-host/pnv_phb4.c | 1 +
 hw/pci/pcie.c          | 8 ++++++--
 hw/pci/pcie_aer.c      | 4 +++-
 4 files changed, 11 insertions(+), 3 deletions(-)
Signature · 1 line
2.35.1
#1 lore raw

PATCH 1/2 pcie: Don't try triggering a LSI when not defined

This patch skips [de]asserting a LSI interrupt if the device doesn't have any LSI defined. Doing so would trigger an assert in pci_irq_handler().

The PCIE root port implementation in qemu requests a LSI (INTA), but a subclass may want to change that behavior since it's a valid configuration. For example on the POWER8/POWER9/POWER10 systems, the root bridge doesn't request any LSI.

---
 hw/pci/pcie.c     | 8 ++++++--
 hw/pci/pcie_aer.c | 4 +++-
 2 files changed, 9 insertions(+), 3 deletions(-)
hw/pci/pcie.c +6 -2
index 67a5d67372..71c5194b80 100644
@@ -354,7 +354,9 @@ static void hotplug_event_notify(PCIDevice *dev)
     } else if (msi_enabled(dev)) {
         msi_notify(dev, pcie_cap_flags_get_vector(dev));
     } else {
-        pci_set_irq(dev, dev->exp.hpev_notified);
+        if (pci_intx(dev) != -1) {
+            pci_set_irq(dev, dev->exp.hpev_notified);
+        }
     }
 }
 
@@ -362,7 +364,9 @@ static void hotplug_event_clear(PCIDevice *dev)
 {
     hotplug_event_update_event_status(dev);
     if (!msix_enabled(dev) && !msi_enabled(dev) && !dev->exp.hpev_notified) {
-        pci_irq_deassert(dev);
+        if (pci_intx(dev) != -1) {
+            pci_irq_deassert(dev);
+        }
     }
 }
 
hw/pci/pcie_aer.c +3 -1
index e1a8a88c8c..d936bfca20 100644
@@ -291,7 +291,9 @@ static void pcie_aer_root_notify(PCIDevice *dev)
     } else if (msi_enabled(dev)) {
         msi_notify(dev, pcie_aer_root_get_vector(dev));
     } else {
-        pci_irq_assert(dev);
+        if (pci_intx(dev) != -1) {
+            pci_irq_assert(dev);
+        }
     }
 }
 
Signature · 1 line
2.35.1
#2 lore raw

Re: [PATCH 1/2] pcie: Don't try triggering a LSI when not defined

Quoted 11 lines · Frederic Barrat

This patch skips [de]asserting a LSI interrupt if the device doesn't have any LSI defined. Doing so would trigger an assert in pci_irq_handler().

The PCIE root port implementation in qemu requests a LSI (INTA), but a subclass may want to change that behavior since it's a valid configuration. For example on the POWER8/POWER9/POWER10 systems, the root bridge doesn't request any LSI.

---

I assume that it's easier to handle just the codepaths that powernv PHBs uses rather than handling all instances where pci_irq_handler() would be asserting without LSIs.

Patch LGTM. Small nits below:

Quoted 9 lines
  hw/pci/pcie.c     | 8 ++++++--
  hw/pci/pcie_aer.c | 4 +++-
  2 files changed, 9 insertions(+), 3 deletions(-)
hw/pci/pcie.c +0 -0
index 67a5d67372..71c5194b80 100644

hw/pci/pcie.c static void hotplug_event_notify(PCIDevice *dev) · line 359

Quoted 7 lines
hw/pci/pcie.c +3 -1
@@ -354,7 +354,9 @@ static void hotplug_event_notify(PCIDevice *dev)
      } else if (msi_enabled(dev)) {
          msi_notify(dev, pcie_cap_flags_get_vector(dev));
      } else {
-        pci_set_irq(dev, dev->exp.hpev_notified);
+        if (pci_intx(dev) != -1) {
+            pci_set_irq(dev, dev->exp.hpev_notified);
+        }

Since you're not doing anything unless the condition is met, you can use 'else if' like it's done in the other conditionals:

    if (msix_enabled(dev)) {
        msix_notify(dev, pcie_cap_flags_get_vector(dev));
    } else if (msi_enabled(dev)) {
        msi_notify(dev, pcie_cap_flags_get_vector(dev));
    } else if (pci_intx(dev) != -1) {
        pci_set_irq(dev, dev->exp.hpev_notified);
    }

hw/pci/pcie.c static void hotplug_event_clear(PCIDevice *dev) · line 370

Quoted 12 lines
hw/pci/pcie.c +3 -1
      }
  }
 
@@ -362,7 +364,9 @@ static void hotplug_event_clear(PCIDevice *dev)
  {
      hotplug_event_update_event_status(dev);
      if (!msix_enabled(dev) && !msi_enabled(dev) && !dev->exp.hpev_notified) {
-        pci_irq_deassert(dev);
+        if (pci_intx(dev) != -1) {
+            pci_irq_deassert(dev);
+        }
      }

Similar comment here:

    if (!msix_enabled(dev) && !msi_enabled(dev) && !dev->exp.hpev_notified &&
        pci_intx(dev) != -1) {
        pci_irq_deassert(dev);
    }
Quoted 7 lines
hw/pci/pcie.c +0 -0
  }
 
hw/pci/pcie_aer.c +0 -0
index e1a8a88c8c..d936bfca20 100644

hw/pci/pcie_aer.c static void pcie_aer_root_notify(PCIDevice *dev) · line 296

Quoted 7 lines
hw/pci/pcie_aer.c +3 -1
@@ -291,7 +291,9 @@ static void pcie_aer_root_notify(PCIDevice *dev)
      } else if (msi_enabled(dev)) {
          msi_notify(dev, pcie_aer_root_get_vector(dev));
      } else {
-        pci_irq_assert(dev);
+        if (pci_intx(dev) != -1) {
+            pci_irq_assert(dev);
+        }

And here:

    if (msix_enabled(dev)) {
        msix_notify(dev, pcie_aer_root_get_vector(dev));
    } else if (msi_enabled(dev)) {
        msi_notify(dev, pcie_aer_root_get_vector(dev));
    } else if (pci_intx(dev) != -1) {
        pci_irq_assert(dev);
    }

Thanks,

Daniel

Quoted 3 lines
hw/pci/pcie_aer.c +0 -0
      }
  }
 
#3 lore raw

Re: [PATCH 1/2] pcie: Don't try triggering a LSI when not defined

Quoted 12 lines · Daniel Henrique Barboza
Quoted 9 lines · Frederic Barrat

This patch skips [de]asserting a LSI interrupt if the device doesn't have any LSI defined. Doing so would trigger an assert in pci_irq_handler().

The PCIE root port implementation in qemu requests a LSI (INTA), but a subclass may want to change that behavior since it's a valid configuration. For example on the POWER8/POWER9/POWER10 systems, the root bridge doesn't request any LSI.

Quoted 6 lines
Quoted 2 lines

---

I assume that it's easier to handle just the codepaths that powernv PHBs uses rather than handling all instances where pci_irq_handler() would be asserting without LSIs.

The real reason is that the LSI is added when we realize the TYPE_PCIE_ROOT_PORT object. See rp_realize(). So I'm only trying to fix the code paths that can be called from a subclass of TYPE_PCIE_ROOT_PORT which would choose to override the "interrupt pin" setting in the config space. I believe they are all covered by this patch. The assert() in pci_irq_handler() is there for a reason and I don't want to mess with that.

Fred

Quoted 92 lines

Patch LGTM. Small nits below:

Quoted 9 lines

  hw/pci/pcie.c     | 8 ++++++--
  hw/pci/pcie_aer.c | 4 +++-

  2 files changed, 9 insertions(+), 3 deletions(-)
hw/pci/pcie.c +0 -0
index 67a5d67372..71c5194b80 100644

hw/pci/pcie.c static void hotplug_event_notify(PCIDevice *dev) · line 359

Quoted 7 lines
hw/pci/pcie.c +3 -1
@@ -354,7 +354,9 @@ static void hotplug_event_notify(PCIDevice *dev)
       } else if (msi_enabled(dev)) {
           msi_notify(dev, pcie_cap_flags_get_vector(dev));
       } else {
-        pci_set_irq(dev, dev->exp.hpev_notified);
+        if (pci_intx(dev) != -1) {
+            pci_set_irq(dev, dev->exp.hpev_notified);
+        }

Since you're not doing anything unless the condition is met, you can use 'else if' like it's done in the other conditionals:

    if (msix_enabled(dev)) {
        msix_notify(dev, pcie_cap_flags_get_vector(dev)); } else if (msi_enabled(dev)) { msi_notify(dev, pcie_cap_flags_get_vector(dev)); } else if (pci_intx(dev) != -1) { pci_set_irq(dev, dev->exp.hpev_notified); }

hw/pci/pcie.c static void hotplug_event_clear(PCIDevice *dev) · line 370

Quoted 11 lines
hw/pci/pcie.c +3 -1
       }
   }
@@ -362,7 +364,9 @@ static void hotplug_event_clear(PCIDevice *dev)
   {
       hotplug_event_update_event_status(dev);
       if (!msix_enabled(dev) && !msi_enabled(dev) && !dev->exp.hpev_notified) {
-        pci_irq_deassert(dev);
+        if (pci_intx(dev) != -1) {
+            pci_irq_deassert(dev);
+        }
       }

Similar comment here:

    if (!msix_enabled(dev) && !msi_enabled(dev) &&
!dev->exp.hpev_notified &&
        pci_intx(dev) != -1) { pci_irq_deassert(dev);
    }

hw/pci/pcie_aer.c static void pcie_aer_root_notify(PCIDevice *dev) · line 296

Quoted 13 lines
hw/pci/pcie.c +0 -0
   }
hw/pci/pcie_aer.c +3 -1
index e1a8a88c8c..d936bfca20 100644
@@ -291,7 +291,9 @@ static void pcie_aer_root_notify(PCIDevice *dev)
       } else if (msi_enabled(dev)) {
           msi_notify(dev, pcie_aer_root_get_vector(dev));
       } else {
-        pci_irq_assert(dev);
+        if (pci_intx(dev) != -1) {
+            pci_irq_assert(dev);
+        }

And here:

    if (msix_enabled(dev)) {
        msix_notify(dev, pcie_aer_root_get_vector(dev)); } else if (msi_enabled(dev)) { msi_notify(dev, pcie_aer_root_get_vector(dev)); } else if (pci_intx(dev) != -1) { pci_irq_assert(dev);
    }

Thanks,

Daniel

Quoted 2 lines
hw/pci/pcie_aer.c +0 -0
       }
   }
#4 lore raw

Re: [PATCH 1/2] pcie: Don't try triggering a LSI when not defined

Quoted 18 lines · Frederic Barrat
Quoted 15 lines · Daniel Henrique Barboza
Quoted 11 lines · Frederic Barrat

This patch skips [de]asserting a LSI interrupt if the device doesn't have any LSI defined. Doing so would trigger an assert in pci_irq_handler().

The PCIE root port implementation in qemu requests a LSI (INTA), but a subclass may want to change that behavior since it's a valid configuration. For example on the POWER8/POWER9/POWER10 systems, the root bridge doesn't request any LSI.

---

Quoted 7 lines
Quoted 3 lines

I assume that it's easier to handle just the codepaths that powernv PHBs uses rather than handling all instances where pci_irq_handler() would be asserting without LSIs.

The real reason is that the LSI is added when we realize the TYPE_PCIE_ROOT_PORT object. See rp_realize(). So I'm only trying to fix the code paths that can be called from a subclass of TYPE_PCIE_ROOT_PORT which would choose to override the "interrupt pin" setting in the config space. I believe they are all covered by this patch. The assert() in pci_irq_handler() is there for a reason and I don't want to mess with that.

Yes, handling this situation inside pci_irq_handler() would require changing the "assert(0 <= irq_num && irq_num < PCI_NUM_PINS)" assert or doing some sanity before it to avoid the trigger.

Since this is a common code used everywhere we're better of doing minimalist changes as you're doing. We can reevaluate this design if more machines/devices start to get in the same situation we have now with powernv PHBs.

Daniel

Quoted 95 lines

  Fred

Quoted 91 lines

Patch LGTM. Small nits below:

Quoted 9 lines

  hw/pci/pcie.c     | 8 ++++++--
  hw/pci/pcie_aer.c | 4 +++-

  2 files changed, 9 insertions(+), 3 deletions(-)
hw/pci/pcie.c +0 -0
index 67a5d67372..71c5194b80 100644

hw/pci/pcie.c static void hotplug_event_notify(PCIDevice *dev) · line 359

Quoted 7 lines
hw/pci/pcie.c +3 -1
@@ -354,7 +354,9 @@ static void hotplug_event_notify(PCIDevice *dev)
       } else if (msi_enabled(dev)) {
           msi_notify(dev, pcie_cap_flags_get_vector(dev));
       } else {
-        pci_set_irq(dev, dev->exp.hpev_notified);
+        if (pci_intx(dev) != -1) {
+            pci_set_irq(dev, dev->exp.hpev_notified);
+        }

Since you're not doing anything unless the condition is met, you can use 'else if' like it's done in the other conditionals:

     if (msix_enabled(dev)) {
         msix_notify(dev, pcie_cap_flags_get_vector(dev)); } else if (msi_enabled(dev)) { msi_notify(dev, pcie_cap_flags_get_vector(dev)); } else if (pci_intx(dev) != -1) { pci_set_irq(dev, dev->exp.hpev_notified); }

hw/pci/pcie.c static void hotplug_event_clear(PCIDevice *dev) · line 370

Quoted 11 lines
hw/pci/pcie.c +3 -1
       }
   }
@@ -362,7 +364,9 @@ static void hotplug_event_clear(PCIDevice *dev)
   {
       hotplug_event_update_event_status(dev);
       if (!msix_enabled(dev) && !msi_enabled(dev) && !dev->exp.hpev_notified) {
-        pci_irq_deassert(dev);
+        if (pci_intx(dev) != -1) {
+            pci_irq_deassert(dev);
+        }
       }

Similar comment here:

     if (!msix_enabled(dev) && !msi_enabled(dev) && !dev->exp.hpev_notified && pci_intx(dev) != -1) {
         pci_irq_deassert(dev);
     }

hw/pci/pcie_aer.c static void pcie_aer_root_notify(PCIDevice *dev) · line 296

Quoted 13 lines
hw/pci/pcie.c +0 -0
   }
hw/pci/pcie_aer.c +3 -1
index e1a8a88c8c..d936bfca20 100644
@@ -291,7 +291,9 @@ static void pcie_aer_root_notify(PCIDevice *dev)
       } else if (msi_enabled(dev)) {
           msi_notify(dev, pcie_aer_root_get_vector(dev));
       } else {
-        pci_irq_assert(dev);
+        if (pci_intx(dev) != -1) {
+            pci_irq_assert(dev);
+        }

And here:

     if (msix_enabled(dev)) {
         msix_notify(dev, pcie_aer_root_get_vector(dev)); } else if (msi_enabled(dev)) { msi_notify(dev, pcie_aer_root_get_vector(dev)); } else if (pci_intx(dev) != -1) { pci_irq_assert(dev);
     }

Thanks,

Daniel

Quoted 2 lines
hw/pci/pcie_aer.c +0 -0
       }
   }
#5 lore raw

PATCH 2/2 ppc/pnv: Remove LSI on the PCIE host bridge

In reply to Frederic Barrat

The phb3/phb4/phb5 root ports inherit from the default PCIE root port implementation, which requests a LSI interrupt (#INTA). On real hardware (POWER8/POWER9/POWER10), there is no such LSI. This patch corrects it so that it matches the hardware.

As a consequence, the device tree previously generated was bogus, as the root bridge LSI was not properly mapped. On some implementation (powernv9), it was leading to inconsistent interrupt controller (xive) data. With this patch, it is now clean.

---
 hw/pci-host/pnv_phb3.c | 1 +
 hw/pci-host/pnv_phb4.c | 1 +
 2 files changed, 2 insertions(+)
hw/pci-host/pnv_phb3.c +1 -0
index ac801ac835..0d18c96117 100644
@@ -1162,6 +1162,7 @@ static void pnv_phb3_root_port_realize(DeviceState *dev, Error **errp)
         error_propagate(errp, local_err);
         return;
     }
+    pci_config_set_interrupt_pin(pci->config, 0);
 }
 
 static void pnv_phb3_root_port_class_init(ObjectClass *klass, void *data)
hw/pci-host/pnv_phb4.c +1 -0
index b301762093..b66b75d4d7 100644
@@ -1772,6 +1772,7 @@ static void pnv_phb4_root_port_reset(DeviceState *dev)
     pci_set_word(conf + PCI_PREF_MEMORY_LIMIT, 0xfff1);
     pci_set_long(conf + PCI_PREF_BASE_UPPER32, 0x1); /* Hack */
     pci_set_long(conf + PCI_PREF_LIMIT_UPPER32, 0xffffffff);
+    pci_config_set_interrupt_pin(conf, 0);
 }
 
 static void pnv_phb4_root_port_realize(DeviceState *dev, Error **errp)
Signature · 1 line
2.35.1
Reviewed-by on v1 2/2
#6 lore raw

Re: [PATCH 2/2] ppc/pnv: Remove LSI on the PCIE host bridge

Quoted 5 lines · Frederic Barrat

The phb3/phb4/phb5 root ports inherit from the default PCIE root port implementation, which requests a LSI interrupt (#INTA). On real hardware (POWER8/POWER9/POWER10), there is no such LSI. This patch corrects it so that it matches the hardware.

Quoted 7 lines

As a consequence, the device tree previously generated was bogus, as the root bridge LSI was not properly mapped. On some implementation (powernv9), it was leading to inconsistent interrupt controller (xive) data. With this patch, it is now clean.

---

Quoted 28 lines
  hw/pci-host/pnv_phb3.c | 1 +
  hw/pci-host/pnv_phb4.c | 1 +
  2 files changed, 2 insertions(+)
hw/pci-host/pnv_phb3.c +1 -0
index ac801ac835..0d18c96117 100644
@@ -1162,6 +1162,7 @@ static void pnv_phb3_root_port_realize(DeviceState *dev, Error **errp)
          error_propagate(errp, local_err);
          return;
      }
+    pci_config_set_interrupt_pin(pci->config, 0);
  }
 
  static void pnv_phb3_root_port_class_init(ObjectClass *klass, void *data)
hw/pci-host/pnv_phb4.c +1 -0
index b301762093..b66b75d4d7 100644
@@ -1772,6 +1772,7 @@ static void pnv_phb4_root_port_reset(DeviceState *dev)
      pci_set_word(conf + PCI_PREF_MEMORY_LIMIT, 0xfff1);
      pci_set_long(conf + PCI_PREF_BASE_UPPER32, 0x1); /* Hack */
      pci_set_long(conf + PCI_PREF_LIMIT_UPPER32, 0xffffffff);
+    pci_config_set_interrupt_pin(conf, 0);
  }
 
  static void pnv_phb4_root_port_realize(DeviceState *dev, Error **errp)
#7 lore raw
In reply to Frederic Barrat
Quoted 11 lines · Frederic Barrat

The powernv8/powernv9/powernv10 machines allocate a LSI for their root port bridge, which is not the case on real hardware. The default root port implementation in qemu requests a LSI. Since the powernv implementation derives from it, that's where the LSI is coming from. This series fixes it, so that the model matches the hardware.

However, the code in hw/pci to handle AER and hotplug events assume a LSI is defined. It tends to assert/deassert a LSI if MSI or MSIX is not enabled. Since we have hardware where that is not true, this patch also fixes a few code paths to check if a LSI is configured before trying to trigger it.

Hi Frederic, thanks for the patch!
I assume you will address Daniel's comments and post a new version, right?

Quoted 13 lines

Frederic Barrat (2):
pcie: Don't try triggering a LSI when not defined ppc/pnv: Remove LSI on the PCIE host bridge

 hw/pci-host/pnv_phb3.c | 1 +
 hw/pci-host/pnv_phb4.c | 1 +
 hw/pci/pcie.c          | 8 ++++++--
 hw/pci/pcie_aer.c      | 4 +++-
 4 files changed, 11 insertions(+), 3 deletions(-)
Signature · 1 line
2.35.1
#8 lore raw
Quoted 14 lines · Michael S. Tsirkin
Quoted 11 lines · Frederic Barrat

The powernv8/powernv9/powernv10 machines allocate a LSI for their root port bridge, which is not the case on real hardware. The default root port implementation in qemu requests a LSI. Since the powernv implementation derives from it, that's where the LSI is coming from. This series fixes it, so that the model matches the hardware.

However, the code in hw/pci to handle AER and hotplug events assume a LSI is defined. It tends to assert/deassert a LSI if MSI or MSIX is not enabled. Since we have hardware where that is not true, this patch also fixes a few code paths to check if a LSI is configured before trying to trigger it.

Quoted 3 lines

Hi Frederic, thanks for the patch!
I assume you will address Daniel's comments and post a new version, right?

Yes, I will. I'm obviously only targeting 7.1

Fred

Quoted 15 lines
Quoted 14 lines

Frederic Barrat (2):
pcie: Don't try triggering a LSI when not defined ppc/pnv: Remove LSI on the PCIE host bridge

  hw/pci-host/pnv_phb3.c | 1 +
  hw/pci-host/pnv_phb4.c | 1 +
  hw/pci/pcie.c          | 8 ++++++--
  hw/pci/pcie_aer.c      | 4 +++-
  4 files changed, 11 insertions(+), 3 deletions(-)
Signature · 1 line
2.35.1