Skip to contentSkip to conversation

Whitelist chipsets supporting MSI and check Hyper-transport capabilities

RFC In review Brice Goglin 9 participants 15 messages Jun 17, 2006 – Jun 19, 2006 1 message not in the archive

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

RFC 1/1 Whitelist chipsets supporting MSI and check Hyper-transport capabilities

Several chipsets are known to not support MSI. Some support MSI but disable it by default. Thus, several drivers implement their own way to detect whether MSI works.

We introduce whitelisting of chipsets that are known to support MSI and keep the existing backlisting to disable MSI for other chipsets. When it is unknown whether the root chipset support MSI or not, we disable MSI by default except if pci=forcemsi was passed.

Whitelisting is done by setting a new PCI_BUS_FLAGS_MSI in the chipset subordinate bus. pci_enable_msi() thus starts by checking whether the root chipset of the device has the MSI or NOMSI flag set.

Bus flags inheritance is dropped since it has been reported to be broken.

We currently whitelist all Intel chipsets, nVidia CK804 and ServerWorks HT2000. Some others are probably missing such as nVidia chipsets for Intel processors.

We introduces a generic code to check the MSI capability in the HyperTransport configuration space, and use it in the generic HT MSI quirk (only used for HT2000 so far) and in the CK804 specific quirk (needs to check 2 HT MSI mapping).

Finally, we rename PCI_CAP_ID_HT_IRQCONF to PCI_CAP_ID_HT since its value is the capability and not the irqconf subtype.

---

I do not split this patch in multiple pieces since it is still small and this is just a RFC. I will split it for the actual submission.

The patch adds PCI_DEVICE_ID_NVIDIA_CK804_PCIE to pciids.h and may thus conflict with pci-nvidia-quirk-to-make-aer-pci-e-extended-capability-visible.patch which adds it too (it has been pushed in Greg K-H's patchset today).

 Documentation/kernel-parameters.txt |    6 +
 arch/powerpc/sysdev/mpic.c          |    2
 drivers/pci/msi.c                   |   58 ++++++++++-----
 drivers/pci/pci.c                   |    2
 drivers/pci/pci.h                   |    1
 drivers/pci/probe.c                 |    2
 drivers/pci/quirks.c                |   95 +++++++++++++++++++++++++-
 include/linux/pci.h                 |    3
 include/linux/pci_ids.h             |    2
 include/linux/pci_regs.h            |    2
 10 files changed, 147 insertions(+), 26 deletions(-)
drivers/pci/msi.c +38 -20
Index: linux-mm/drivers/pci/msi.c
===================================================================
@@ -28,6 +28,7 @@
 static kmem_cache_t* msi_cachep;
 
 static int pci_msi_enable = 1;
+static int pci_msi_force = 0;
 static int last_alloc_vector;
 static int nr_released_vectors;
 static int nr_reserved_vectors = NR_HP_RESERVED_VECTORS;
@@ -902,6 +903,34 @@
 }
 
 /**
+ * pci_msi_enabled - check whether MSI may be enabled on device
+ * @dev: pointer to the pci_dev data structure of MSI device function
+ *
+ * Check parent busses for MSI or NO_MSI flags, or disable except
+ * if forced.
+ **/
+int pci_msi_enabled(struct pci_dev * dev)
+{
+	struct pci_dev *pdev;
+
+	if (!pci_msi_enable || !dev || dev->no_msi)
+ 		return -1;
+
+	/* find root complex for our device */
+	pdev = dev;
+	while (pdev->bus && pdev->bus->self)
+		pdev = pdev->bus->self;
+
+	/* check its bus flags */
+	if (pdev->subordinate->bus_flags & PCI_BUS_FLAGS_MSI)
+		return 0;
+	if (pdev->subordinate->bus_flags & PCI_BUS_FLAGS_NO_MSI)
+		return -1;
+
+	return pci_msi_force ? 0 : -1;
+}
+
+/**
  * pci_enable_msi - configure device's MSI capability structure
  * @dev: pointer to the pci_dev data structure of MSI device function
  *
@@ -913,19 +942,11 @@
  **/
 int pci_enable_msi(struct pci_dev* dev)
 {
-	struct pci_bus *bus;
-	int pos, temp, status = -EINVAL;
+	int pos, temp, status;
 	u16 control;
 
-	if (!pci_msi_enable || !dev)
- 		return status;
-
-	if (dev->no_msi)
-		return status;
-
-	for (bus = dev->bus; bus; bus = bus->parent)
-		if (bus->bus_flags & PCI_BUS_FLAGS_NO_MSI)
-			return -EINVAL;
+	if (pci_msi_enabled(dev) < 0)
+		return -EINVAL;
 
 	temp = dev->irq;
 
@@ -1135,22 +1156,14 @@
  **/
 int pci_enable_msix(struct pci_dev* dev, struct msix_entry *entries, int nvec)
 {
-	struct pci_bus *bus;
 	int status, pos, nr_entries, free_vectors;
 	int i, j, temp;
 	u16 control;
 	unsigned long flags;
 
-	if (!pci_msi_enable || !dev || !entries)
- 		return -EINVAL;
-
-	if (dev->no_msi)
+	if (!entries || pci_msi_enabled(dev) < 0)
 		return -EINVAL;
 
-	for (bus = dev->bus; bus; bus = bus->parent)
-		if (bus->bus_flags & PCI_BUS_FLAGS_NO_MSI)
-			return -EINVAL;
-
 	status = msi_init();
 	if (status < 0)
 		return status;
@@ -1350,6 +1363,11 @@
 	pci_msi_enable = 0;
 }
 
+void pci_force_msi(void)
+{
+	pci_msi_force = 1;
+}
+
 EXPORT_SYMBOL(pci_enable_msi);
 EXPORT_SYMBOL(pci_disable_msi);
 EXPORT_SYMBOL(pci_enable_msix);
drivers/pci/pci.c +2 -0
Index: linux-mm/drivers/pci/pci.c
===================================================================
@@ -978,6 +978,8 @@
 		if (*str && (str = pcibios_setup(str)) && *str) {
 			if (!strcmp(str, "nomsi")) {
 				pci_no_msi();
+			} else if (!strcmp(str, "forcemsi")) {
+				pci_force_msi();
 			} else {
 				printk(KERN_ERR "PCI: Unknown option `%s'\n",
 						str);
drivers/pci/pci.h +1 -0
Index: linux-mm/drivers/pci/pci.h
===================================================================
@@ -51,6 +51,7 @@
 #ifdef CONFIG_PCI_MSI
 void disable_msi_mode(struct pci_dev *dev, int pos, int type);
 void pci_no_msi(void);
+void pci_force_msi(void);
 #else
 static inline void disable_msi_mode(struct pci_dev *dev, int pos, int type) { }
 static inline void pci_no_msi(void) { }
drivers/pci/quirks.c +93 -2

113 lines not on this page · raw message

include/linux/pci.h +2 -1
Index: linux-mm/include/linux/pci.h
===================================================================
@@ -97,6 +97,7 @@
 typedef unsigned short __bitwise pci_bus_flags_t;
 enum pci_bus_flags {
 	PCI_BUS_FLAGS_NO_MSI = (__force pci_bus_flags_t) 1,
+	PCI_BUS_FLAGS_MSI = (__force pci_bus_flags_t) 2,
 };
 
 struct pci_cap_saved_state {
@@ -242,7 +243,7 @@
 	char		name[48];
 
 	unsigned short  bridge_ctl;	/* manage NO_ISA/FBB/et al behaviors */
-	pci_bus_flags_t bus_flags;	/* Inherited by child busses */
+	pci_bus_flags_t bus_flags;
 	struct device		*bridge;
 	struct class_device	class_dev;
 	struct bin_attribute	*legacy_io; /* legacy I/O for this bus */
include/linux/pci_ids.h +2 -0
Index: linux-mm/include/linux/pci_ids.h
===================================================================
@@ -1027,6 +1027,7 @@
 #define PCI_DEVICE_ID_NVIDIA_NVENET_8		0x0056
 #define PCI_DEVICE_ID_NVIDIA_NVENET_9		0x0057
 #define PCI_DEVICE_ID_NVIDIA_CK804_AUDIO	0x0059
+#define PCI_DEVICE_ID_NVIDIA_CK804_PCIE		0x005d
 #define PCI_DEVICE_ID_NVIDIA_NFORCE2_SMBUS	0x0064
 #define PCI_DEVICE_ID_NVIDIA_NFORCE2_IDE	0x0065
 #define PCI_DEVICE_ID_NVIDIA_NVENET_2		0x0066
@@ -1405,6 +1406,7 @@
 #define PCI_DEVICE_ID_SERVERWORKS_LE	  0x0009
 #define PCI_DEVICE_ID_SERVERWORKS_GCNB_LE 0x0017
 #define PCI_DEVICE_ID_SERVERWORKS_EPB	  0x0103
+#define PCI_DEVICE_ID_SERVERWORKS_HT2000_PCIE	0x0132
 #define PCI_DEVICE_ID_SERVERWORKS_OSB4	  0x0200
 #define PCI_DEVICE_ID_SERVERWORKS_CSB5	  0x0201
 #define PCI_DEVICE_ID_SERVERWORKS_CSB6    0x0203
Documentation/kernel-parameters.txt +6 -0
Index: linux-mm/Documentation/kernel-parameters.txt
===================================================================
@@ -1190,6 +1190,12 @@
 		nomsi		[MSI] If the PCI_MSI kernel config parameter is
 				enabled, this kernel boot option can be used to
 				disable the use of MSI interrupts system-wide.
+		forcemsi	[MSI] If the PCI_MSI kernel config parameter is
+				enabled, force MSI to be enabled when its support
+				in parent busses is unknown. By default, MSI
+				are only enabled when parent busses are known
+				to support it. This option does not force MSI
+				when parent busses are known to not support it.
 		nosort		[IA-32] Don't sort PCI devices according to
 				order given by the PCI BIOS. This sorting is
 				done to get a device order compatible with
drivers/pci/probe.c +1 -1
Index: linux-mm/drivers/pci/probe.c
===================================================================
@@ -351,7 +351,7 @@
 	child->parent = parent;
 	child->ops = parent->ops;
 	child->sysdata = parent->sysdata;
-	child->bus_flags = parent->bus_flags;
+	child->bus_flags = 0;
 	child->bridge = get_device(&bridge->dev);
 
 	child->class_dev.class = &pcibus_class;
arch/powerpc/sysdev/mpic.c +1 -1
Index: linux-mm/arch/powerpc/sysdev/mpic.c
===================================================================
@@ -250,7 +250,7 @@
 	for (pos = readb(devbase + PCI_CAPABILITY_LIST); pos != 0;
 	     pos = readb(devbase + pos + PCI_CAP_LIST_NEXT)) {
 		u8 id = readb(devbase + pos + PCI_CAP_LIST_ID);
-		if (id == PCI_CAP_ID_HT_IRQCONF) {
+		if (id == PCI_CAP_ID_HT) {
 			id = readb(devbase + pos + 3);
 			if (id == 0x80)
 				break;
include/linux/pci_regs.h +1 -1
Index: linux-mm/include/linux/pci_regs.h
===================================================================
@@ -196,7 +196,7 @@
 #define  PCI_CAP_ID_MSI		0x05	/* Message Signalled Interrupts */
 #define  PCI_CAP_ID_CHSWP	0x06	/* CompactPCI HotSwap */
 #define  PCI_CAP_ID_PCIX	0x07	/* PCI-X */
-#define  PCI_CAP_ID_HT_IRQCONF	0x08	/* HyperTransport IRQ Configuration */
+#define  PCI_CAP_ID_HT		0x08	/* HyperTransport capability */
 #define  PCI_CAP_ID_VNDR	0x09	/* Vendor specific capability */
 #define  PCI_CAP_ID_SHPC 	0x0C	/* PCI Standard Hot-Plug Controller */
 #define  PCI_CAP_ID_EXP 	0x10	/* PCI Express */
#1 lore raw
Quoted 3 lines · Brice Goglin

Several chipsets are known to not support MSI. Some support MSI but disable it by default. Thus, several drivers implement their own way to detect whether MSI works.

Yes, and that needs to go away. To be fair, we're in the early stages of introducing generic MSI support, so it's understandable that people have made expedient rather than architectural solutions to problems.

Quoted 4 lines

We introduce whitelisting of chipsets that are known to support MSI and keep the existing backlisting to disable MSI for other chipsets. When it is unknown whether the root chipset support MSI or not, we disable MSI by default except if pci=forcemsi was passed.

I think that's a bad idea. Blacklisting is the better idea in the long-term.

Quoted 1 line

Bus flags inheritance is dropped since it has been reported to be broken.

I must have missed that report. Please elucidate.

#2 lore raw
Quoted 7 lines · Matthew Wilcox
Quoted 4 lines

We introduce whitelisting of chipsets that are known to support MSI and keep the existing backlisting to disable MSI for other chipsets. When it is unknown whether the root chipset support MSI or not, we disable MSI by default except if pci=forcemsi was passed.

I think that's a bad idea. Blacklisting is the better idea in the long-term.

In this case, we absolutely need an exhaustive list of chipsets that do not support MSI. And it looks like there are a lot of them in the non-PCI-E world. Our concern is that trying to enable MSI when it does not work breaks drivers. While disabling MSI generally keeps drivers working (except for instance for the Infinipath hardware, but our whitelisting of the nVidia chipset might help them :)).
For the long term, I hope we'll end up having an exhaustive list of chipsets that do not support MSI. But in the meantime, our "whitelisting
+ disable when we don't know" might help getting MSI as much as possible without breaking to many things.

Quoted 5 lines
Quoted 2 lines

Bus flags inheritance is dropped since it has been reported to be broken.

I must have missed that report. Please elucidate.

It is based on the following thread:

    http://www.ussg.iu.edu/hypermail/linux/kernel/0605.2/0929.html

The amd8131 MSI quirk cannot be EARLY or HEADER because it would be called before dev->subordinate has been set (then, PCI_BUS_FLAGS_NO_MSI could not be set in the bus flags). But, flags are inherited while the PCI hierarchy is scanned, which means the quirk has to be EARLY or HEADER. To solve this issue, we would need a quirk that is called while the PCI hierarchy is scanned (ie, before FINAL or ENABLE) and after the pci child bus of the bridge is created (ie, after EARLY and HEADER). But I am not sure whether adding a new quirk type for this is a good idea.

Since the MSI is always actually processed on the root chipset, we know we have to check there. We thought that inheriting bus flags was not really required. That's why my patch finds the root chipset and checks bus flags there.

Brice

#5 lore raw
In reply to Brice Goglin
Quoted 4 lines · Brice Goglin

Several chipsets are known to not support MSI. Some support MSI but disable it by default. Thus, several drivers implement their own way to detect whether MSI works.

Quoted 8 lines

We introduce whitelisting of chipsets that are known to support MSI and keep the existing backlisting to disable MSI for other chipsets. When it is unknown whether the root chipset support MSI or not, we disable MSI by default except if pci=forcemsi was passed.

Whitelisting is done by setting a new PCI_BUS_FLAGS_MSI in the chipset subordinate bus. pci_enable_msi() thus starts by checking whether the root chipset of the device has the MSI or NOMSI flag set.

Whitelisting looks all well and good today, and maybe for the rest of the year. But what about 3 years from now when everyone has shaken all of the MSI bugs out of their chipsets finally? Do you really want to add a new quirk for _every_ new chipset that comes out? I don't think that it is managable over the long run.

I do like your checks to see if MSI is able to be enabled or not, and maybe we can just invert them to mark those chips that do not support MSI today?

Quoted 2 lines

I do not split this patch in multiple pieces since it is still small and this is just a RFC. I will split it for the actual submission.

Please do, you are doing a lot of different things in here.

thanks,

greg k-h

#6 lore raw
Quoted 6 lines · Greg KH

Whitelisting looks all well and good today, and maybe for the rest of the year. But what about 3 years from now when everyone has shaken all of the MSI bugs out of their chipsets finally? Do you really want to add a new quirk for _every_ new chipset that comes out? I don't think that it is managable over the long run.

We could still reverse the default. Right now, unless pci=forcemsi is passed, I disable MSI if we don't know whether the chipset supports it. Once blacklisting has been improved/completed, we can enable MSI by default (and keep "pci=nomsi" in case a non-blacklisted chipset appears).

Or we could enable MSI by default on PCI-E chipsets and disable by default on non-PCI-E (ie we whitelist non-PCI-E only) ? PCI-E chipsets seem to support MSI pretty well.

I am ok with any strategy as long as I don't end up passing "pci=nomsi" on most new machines I will install in the next 10 years (as I did with "noapic" in the past) :)

Brice

#7 lore raw
Quoted 4 lines · Brice Goglin

Or we could enable MSI by default on PCI-E chipsets and disable by default on non-PCI-E (ie we whitelist non-PCI-E only) ? PCI-E chipsets seem to support MSI pretty well.

It looks like at least Serverworks HT1000 has trouble with MSI too, but it's PCI-Express. But I guess those can be black listed

Also I think Intel has supported it well for a long time so might want to white list all from VENDOR == Intel.

Blacklisting all old non PCI-E bridges non Intel seems reasonable

It seems AMD 8132 can be made to work, but it needs a special quirk too and then it can be white listed.

The rules will be relatively complicated I guess, but should be doable.

-Andi

#13 lore raw
In reply to Greg KH
Quoted 20 lines · Greg KH
Quoted 4 lines · Brice Goglin

Several chipsets are known to not support MSI. Some support MSI but disable it by default. Thus, several drivers implement their own way to detect whether MSI works.

Quoted 8 lines

We introduce whitelisting of chipsets that are known to support MSI and keep the existing backlisting to disable MSI for other chipsets. When it is unknown whether the root chipset support MSI or not, we disable MSI by default except if pci=forcemsi was passed.

Whitelisting is done by setting a new PCI_BUS_FLAGS_MSI in the chipset subordinate bus. pci_enable_msi() thus starts by checking whether the root chipset of the device has the MSI or NOMSI flag set.

Whitelisting looks all well and good today, and maybe for the rest of the year. But what about 3 years from now when everyone has shaken all of the MSI bugs out of their chipsets finally? Do you really want to add a new quirk for _every_ new chipset that comes out? I don't think that it is managable over the long run.

Quoted 3 lines

I do like your checks to see if MSI is able to be enabled or not, and maybe we can just invert them to mark those chips that do not support MSI today?

My gut feeling is:

blacklist -> any Intel machines which fail (most work) blacklist -> any PCI Express which fails (most should work) whitelist -> any other situation which works

Regards,

Jeff

#8 lore raw
In reply to Andi Kleen
Quoted 10 lines · Andi Kleen
Quoted 4 lines · Brice Goglin

Or we could enable MSI by default on PCI-E chipsets and disable by default on non-PCI-E (ie we whitelist non-PCI-E only) ? PCI-E chipsets seem to support MSI pretty well.

It looks like at least Serverworks HT1000 has trouble with MSI too, but it's PCI-Express. But I guess those can be black listed

IIRC, HT1000 is the southbridge part of the HT2000 chipset. We have been told that MSI works on this chipset. And from what we've seen/tested, it is true. The problem is that MSI is often disabled by the BIOS. My hypertransport MSI capability quirk should check it right.

Brice

#14 lore raw
In reply to Brice Goglin
Quoted 8 lines · Brice Goglin

Several chipsets are known to not support MSI. Some support MSI but disable it by default. Thus, several drivers implement their own way to detect whether MSI works.

We introduce whitelisting of chipsets that are known to support MSI and keep the existing backlisting to disable MSI for other chipsets. When it is unknown whether the root chipset support MSI or not, we disable MSI by default except if pci=forcemsi was passed.

I tried your patch on 2.6.17-rc6 (fixed up a view rejects) in an attempt to get MSI working with my forcedeth NIC, which advertises the capability. The kernel now gives me the following:

PCI: Found MSI HT mapping on 0000:00:0b.0
PCI: Found MSI HT mapping on 0000:00:00.0
PCI: MSI quirk detected. PCI_BUS_FLAGS_MSI set for 0000:00:0b.0 subordinate bus.
PCI: Found MSI HT mapping on 0000:00:0c.0
PCI: Found MSI HT mapping on 0000:00:00.0
PCI: MSI quirk detected. PCI_BUS_FLAGS_MSI set for 0000:00:0c.0 subordinate bus.
PCI: Found MSI HT mapping on 0000:00:0d.0
PCI: Found MSI HT mapping on 0000:00:00.0
PCI: MSI quirk detected. PCI_BUS_FLAGS_MSI set for 0000:00:0d.0 subordinate bus.
PCI: Found MSI HT mapping on 0000:00:0e.0
PCI: Found MSI HT mapping on 0000:00:00.0
PCI: MSI quirk detected. PCI_BUS_FLAGS_MSI set for 0000:00:0e.0 subordinate bus.
PCI: Setting latency timer of device 0000:00:0b.0 to 64 pcie_portdrv_probe->Dev[005d:10de] has invalid IRQ. Check vendor BIOS assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:0b.0:pcie00]
Allocate Port Service[0000:00:0b.0:pcie03]
PCI: Setting latency timer of device 0000:00:0c.0 to 64 pcie_portdrv_probe->Dev[005d:10de] has invalid IRQ. Check vendor BIOS assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:0c.0:pcie00]
Allocate Port Service[0000:00:0c.0:pcie03]
PCI: Setting latency timer of device 0000:00:0d.0 to 64 pcie_portdrv_probe->Dev[005d:10de] has invalid IRQ. Check vendor BIOS assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:0d.0:pcie00]
Allocate Port Service[0000:00:0d.0:pcie03]
PCI: Setting latency timer of device 0000:00:0e.0 to 64 pcie_portdrv_probe->Dev[005d:10de] has invalid IRQ. Check vendor BIOS assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:0e.0:pcie00]
Allocate Port Service[0000:00:0e.0:pcie03]

Is the "vendor BIOS" causing the problem? Here's my /proc/interrupts after booting.

CPU0 CPU1

  0:     574494          0    IO-APIC-edge  timer
  8:          0          0    IO-APIC-edge  rtc
  9:          0          0   IO-APIC-level  acpi
 15:       4891          0    IO-APIC-edge  ide1
 50:          0          0   IO-APIC-level  NVidia CK804
 58:          4          0   IO-APIC-level  ehci_hcd:usb1
 66:       6203          0   IO-APIC-level  ohci1394, wlan
 74:          3          0   IO-APIC-level  ohci1394
217:      21315          0   IO-APIC-level  libata, ohci_hcd:usb2
225:       1031          0   IO-APIC-level  libata
233:      28124          0   IO-APIC-level  EMU10K1, skge
NMI:        171         90
LOC:     574330     574263
ERR:          0
MIS:          0

It's an x86-64 kernel.

Signature · 5 lines
Cheers,
Alistair.

Third year Computer Science undergraduate.
1F2 55 South Clerk Street, Edinburgh, UK.
#15 lore raw
Quoted 6 lines · Alistair John Strachan

On Saturday 17 June 2006 04:01, Brice Goglin wrote:

I tried your patch on 2.6.17-rc6 (fixed up a view rejects) in an attempt to get MSI working with my forcedeth NIC, which advertises the capability. The kernel now gives me the following:

Well, the aim of my patch is to stop enabling MSI by default, and try to find out when we can and when we cannot use MSI. 2.6.17 will only disable MSI on a few chipsets. My patch will disable on all chipsets but Intel and those having Hypertransport MSI capabilities (Serverworks HT2000 and nVidia CK804). Thus, you cannot expect MSI to be enabled with my patch if it was not before :/

Quoted 8 lines

PCI: Found MSI HT mapping on 0000:00:0b.0
PCI: Found MSI HT mapping on 0000:00:00.0
PCI: MSI quirk detected. PCI_BUS_FLAGS_MSI set for 0000:00:0b.0 subordinate bus.
PCI: Found MSI HT mapping on 0000:00:0c.0
PCI: Found MSI HT mapping on 0000:00:00.0
PCI: MSI quirk detected. PCI_BUS_FLAGS_MSI set for 0000:00:0c.0 subordinate bus.

Quoted 9 lines

PCI: Found MSI HT mapping on 0000:00:0d.0
PCI: Found MSI HT mapping on 0000:00:00.0
PCI: MSI quirk detected. PCI_BUS_FLAGS_MSI set for 0000:00:0d.0 subordinate bus.
PCI: Found MSI HT mapping on 0000:00:0e.0
PCI: Found MSI HT mapping on 0000:00:00.0
PCI: MSI quirk detected. PCI_BUS_FLAGS_MSI set for 0000:00:0e.0 subordinate bus.

If your board is behind one of these bridges, MSI could work (looks like you have 4 nVidia CK804 with MSI enabled in the Hypertransport mapping).

Quoted 5 lines

pcie_portdrv_probe->Dev[005d:10de] has invalid IRQ. Check vendor BIOS assign_interrupt_mode Found MSI capability
Allocate Port Service[0000:00:0b.0:pcie00]
Allocate Port Service[0000:00:0b.0:pcie03]

It might be the reason why MSI does not work. But I am not familiar with this code, so I can't help, sorry.

Brice

#10 lore raw
In reply to a message not in the archive

[you destroyed the cc list. Don't do that]

Quoted 7 lines · Greg Lindahl
Quoted 3 lines · Brice Goglin

IIRC, HT1000 is the southbridge part of the HT2000 chipset. We have been told that MSI works on this chipset. And from what we've seen/tested, it is true.

We can also say this is true -- our InfiniPath HCA requires MSI, so we really notice this.

Isn't your HCA directly connected to HTX? If yes it will likely not run into PCI bridge bugs.

We got a Serverworks based Supermicro system where the driver for the integrated tg3 NIC complains about MSI not working. So either that particular system has a specific BIOS issue or it is broken for on motherboard devices.

I was extrapolating from that.

It doesn't complain on another Supermicro system, but that seems to be because that particular BCM570x silicon revision is blacklisted for MSI in the tg3 driver.

Quoted 1 line

From that experience I certainly cannot say that MSI works

very well on Serverworks so far. It might be safer to blacklist until someone can explain what's going on on these systems. The problem is that not all drivers do the MSI probing tg3 do, so if you got a system where MSI doesn't work. but it tells the kernel it does, and a driver turns on MSI things just break.

For me so far it looks like MSI is another HPET or mmconfig - something that looks nice in the specification, but hardware/BIOS developers don't seem to pay particular attention to and is ridden with platform bugs.

-Andi

#3 lore raw
In reply to Matthew Wilcox
Quoted 7 lines · Matthew Wilcox
Quoted 4 lines · Brice Goglin

We introduce whitelisting of chipsets that are known to support MSI and keep the existing backlisting to disable MSI for other chipsets. When it is unknown whether the root chipset support MSI or not, we disable MSI by default except if pci=forcemsi was passed.

I think that's a bad idea. Blacklisting is the better idea in the long-term.

I think the option adopted elsewhere is: whitelist for old chipsets, and blacklist for new chipsets. You just have to decide for a good date to separate "old" and "new" to minimize the lists size.

Xav

#11 lore raw
In reply to Andi Kleen
Quoted 11 lines · Andi Kleen

We got a Serverworks based Supermicro system where the driver for the integrated tg3 NIC complains about MSI not working. So either that particular system has a specific BIOS issue or it is broken for on motherboard devices.

I was extrapolating from that.

It doesn't complain on another Supermicro system, but that seems to be because that particular BCM570x silicon revision is blacklisted for MSI in the tg3 driver.

Quoted 8 lines

From that experience I certainly cannot say that MSI works very well on Serverworks so far. It might be safer to blacklist until someone can explain what's going on on these systems. The problem is that not all drivers do the MSI probing tg3 do, so if you got a system where MSI doesn't work. but it tells the kernel it does, and a driver turns on MSI things just break.

Initially, we thought MSI did not work. But, we actually discovered that it was caused by the MSI cap being disabled in the Hypertransport config space. Enabling the MSI cap there seems to make it work.

My patches check the HT MSI cap in a quirk. So you will only get MSI for your driver if the BIOS/chipset has been correctly initialized.

The next step would be to explicitely set the MSI cap in the Hypertransport config space in my quirk. I have patches to do so, but I'd rather wait until my current patches to are stabilized/merged.

Brice

#4 lore raw
In reply to Xavier Bestel
Quoted 7 lines · Xavier Bestel
Quoted 6 lines · Matthew Wilcox
Quoted 4 lines · Brice Goglin

We introduce whitelisting of chipsets that are known to support MSI and keep the existing backlisting to disable MSI for other chipsets. When it is unknown whether the root chipset support MSI or not, we disable MSI by default except if pci=forcemsi was passed.

Quoted 5 lines
Quoted 1 line

I think that's a bad idea. Blacklisting is the better idea in the long-term.

I think the option adopted elsewhere is: whitelist for old chipsets, and blacklist for new chipsets. You just have to decide for a good date to separate "old" and "new" to minimize the lists size.

I agree with willy.

White lists work "well" only if one's goal is to reduce the number of bug reports about _all_ HW. Most folks with _working_ MSI (but not already on the whitelist) won't know they could report this as a bug. Ie these chipsets likely won't ever get added to the whitelist.

thanks,
grant

#12 lore raw
In reply to Andi Kleen
Quoted 2 lines · Andi Kleen

Isn't your HCA directly connected to HTX? If yes it will likely not run into PCI bridge bugs.

I was referring to our new PCI Express HCA.

-- greg