Sun V880 + Infiniband?

So, I'm trying to get a Mellanox IB card to work on my Sun V880, and I'm getting this error output to dmesg:

[   99.529658] ib_mthca: Mellanox InfiniBand HCA driver v1.0 (April 4, 
2008)
[   99.609988] ib_mthca: Initializing 0003:01:00.0
[   99.664180] PCI: Enabling device: (0003:01:00.0), cmd 2
[   99.664193] ib_mthca 0003:01:00.0: Missing DCS, aborting.

I'm using a kernel compiled off of Linus's git tree as of a few days ago (to fix other SPARC issues). From what I've seen from google searches, this means that the first BAR isn't mapped or visible to the driver, so maybe this is a SPARC related thing?

lspci -v output for the device:

0003:00:01.0 PCI bridge: Mellanox Technologies MT23108 PCI Bridge (rev a1) (prog-if 00 [Normal decode])
Flags: bus master, 66MHz, slow devsel, latency 64 [virtual] Memory at fffff80500000000 (32-bit, non-prefetchable)
[size=1]

        [virtual] Memory at fffff80500000000 (32-bit, non-prefetchable) 
[size=1]
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        Memory behind bridge: 00100000-0fffffff
        [virtual] Expansion ROM at fffff80500000000 [disabled] [size=1]
        Capabilities: [70] PCI-X bridge device

0003:01:00.0 InfiniBand: Mellanox Technologies MT23108 InfiniHost (rev a1)
Subsystem: Mellanox Technologies MT23108 InfiniHost
Flags: 66MHz, slow devsel, IRQ 30
Memory at fffff80500000000 (64-bit, non-prefetchable) [size=1] Memory at fffff80500000000 (64-bit, prefetchable) [size=1] Memory at fffff80500000000 (64-bit, prefetchable) [size=1] [virtual] Expansion ROM at fffff80500000000 [disabled] [size=1]
Capabilities: [40] MSI-X: Enable- Mask- TabSize=32
Capabilities: [50] Vital Product Data <?>
Capabilities: [60] Message Signalled Interrupts: Mask- 64bit+
Queue=0/5 Enable-
Capabilities: [70] PCI-X non-bridge device
Kernel modules: ib_mthca

Pat

Purdue University Research Computing ---  http://www.rcac.purdue.edu/
The Computer Refuge                  ---  http://computer-refuge.org

I think the problem is related to:

So it says you have 3 BARs at the same address, all with size 1 (?!) which means the PCI setup or probing is messed up. The mthca driver checks that the first BAR has size 1 megabyte as it should, and it's bailing out because the kernel is telling it that it's the wrong size.

- R.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

From: Roland Dreier <[email protected]>
Date: Wed, 02 Dec 2009 09:55:19 -0800

These BARs are allocated and setup by the boot firmware long before Linus boots up. Not being able to handle 64-bit BARs properly wouldn't surprise me.

But there could also be a Linux bug in decoding the openfirmware property values as well, so let's investigate that.

Patrick, can you post a new "prtconf -pv" dump under Linux with this card in the machine? Also, please bootup with:

of_debug=1 ofpci_debug=1

added to the kernel command line and post the resulting "dmesg".

If there's something Linux isn't doing right, those dumps will help me spot it.

Thanks!

I have placed an updated prtconf here:

http://ned.cc.purdue.edu/prtconf-v880-8cpu

The last two entries on it (PCI bridge and pci15b3,5a44) are from the card.

Based upon my best guess as to what the fields indicate, it looks like the OF tree may have the proper BAR sizes in it at least (I believe that they're supposed to be 1MB, 8MB and 128MB respectively), so I think it's a Linux thing not an OpenBoot thing.

The console output after booting with the debugging options is here:

http://ned.cc.purdue.edu/v880-of_debug-dmesg

I see that the output claims that the 3 BARs start and end at address "0" which doesn't look right.

Thanks again!

Pat

Purdue University Research Computing ---  http://www.rcac.purdue.edu/
The Computer Refuge                  ---  http://computer-refuge.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Please don't post in HTML, the list server rejects all such postings. Use plain ASCII text only.

Thank you.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

From: Patrick Finnegan <[email protected]>
Date: Thu, 3 Dec 2009 01:44:18 -0500

Please give this patch a try:

sparc64: Fix overly strict range type matching for PCI devices.

When we are trying to see if a range property entry applies to a given address, we are overly strict about the type.

We should only allow I/O ranges for I/O addresses, and only allow CONFIG space ranges for CONFIG space address.

However for MEM ranges, they come in 32-bit and 64-bit flavors. And a lack of an exact match is OK if the range is 32-bit and the address is 64-bit. We can assign a 64-bit address properly into a 32-bit parent range just fine.

So allow it.

---
 arch/sparc/kernel/of_device_64.c |   14 ++++++++++++--
 1 files changed, 12 insertions(+), 2 deletions(-)

index 881947e..0a6f2d1 100644
@@ -104,9 +104,19 @@ static int of_bus_pci_map(u32 *addr, const u32 *range,
 	int i;
 
 	/* Check address type match */
-	if ((addr[0] ^ range[0]) & 0x03000000)
-		return -EINVAL;
+	if (!((addr[0] ^ range[0]) & 0x03000000))
+		goto type_match;
+
+	/* Special exception, we can map a 64-bit address into
+	 * a 32-bit range.
+	 */
+	if ((addr[0] & 0x03000000) == 0x03000000 &&
+	    (range[0] & 0x03000000) == 0x02000000)
+		goto type_match;
+
+	return -EINVAL;
 
+type_match:
 	if (of_out_of_range(addr + 1, range + 1, range + na + pna,
 			    na - 1, ns))
 		return -EINVAL;

1.6.5

This appears to work for me.

Thanks!

Pat

Purdue University Research Computing ---  http://www.rcac.purdue.edu/
The Computer Refuge                  ---  http://computer-refuge.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

From: Patrick Finnegan <[email protected]>
Date: Wed, 9 Dec 2009 14:29:56 -0500

Thanks for your report and testing the fix.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html