Skip to contentSkip to conversation

Intel RAID Controller SRCU42X in SGI Altix 350

DISCUSSION Robert Nagy 2 participants 11 messages Jun 29, 2006 – Jun 30, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

Distribution: Debian testing/unstable
Hardware Environment: SGI Altix 350, 2xItanium 2, EFI (read dmesg)
http://bsd.hu/~robert/altix.dmesg
http://bsd.hu/~robert/altix.kconf

Problem Description: I've installed an Intel(r) RAID Controller SRCU42X (PCI-X) controller to this machine.
http://www.intel.com/design/servers/raid/srcu42x/index.htm
I've never used such a controller so if someone has any idea about this please tell me. The dmesg will show everyhing, but:

megaraid cmm: 2.20.2.6 (Release Date: Mon Mar 7 00:01:03 EST 2005)
megaraid: 2.20.4.8 (Release Date: Mon Apr 11 12:27:22 EST 2006)
megaraid: probe new device 0x1000:0x0407:0x8086:0x0532: bus 2:slot 0:func 0
megaraid: out of memory, megaraid_alloc_cmd_packets 965
megaraid: maibox adapter did not initialize

0001:01:01.0 Co-processor: Silicon Graphics, Inc. IOC4 I/O controller (rev 4f) 0001:01:03.0 SCSI storage controller: QLogic Corp. ISP12160 Dual Channel Ultra3 SCSI Processor (rev 06)
0001:01:04.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15)
0002:01:02.0 PCI bridge: IBM PCI-X to PCI-X Bridge (rev 03) 0002:02:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID (rev 02)

Thanks

#1 lore raw
Quoted 13 lines · Robert Nagy

Hi,

Distribution: Debian testing/unstable
Hardware Environment: SGI Altix 350, 2xItanium 2, EFI (read dmesg)
http://bsd.hu/~robert/altix.dmesg
http://bsd.hu/~robert/altix.kconf

Problem Description: I've installed an Intel(r) RAID Controller SRCU42X (PCI-X) controller to this machine.
http://www.intel.com/design/servers/raid/srcu42x/index.htm
I've never used such a controller so if someone has any idea about this please tell me. The dmesg will show everyhing, but:

Quoted 5 lines

megaraid cmm: 2.20.2.6 (Release Date: Mon Mar 7 00:01:03 EST 2005)
megaraid: 2.20.4.8 (Release Date: Mon Apr 11 12:27:22 EST 2006)
megaraid: probe new device 0x1000:0x0407:0x8086:0x0532: bus 2:slot 0:func 0 megaraid: out of memory, megaraid_alloc_cmd_packets 965
megaraid: maibox adapter did not initialize

IIRC some Altix boxes don't support 32 bit DMA for PCI-X devices. Based on the initialization code I looked at (just a quick scan), it looks like the command packet initialization is done before the switch to a 64 bit DMA mask, which might cause the failure you see here. You can try this patch out (totally untested). It's not fully correct, it should probably try 64 bit first then fall back to 32 bit if that fails then give up, and the other 64 bit DMA mask call should probably be removed.

Anyway, good luck. If this one doesn't work you'll have to talk with one of the SGI guys and get some more debug info about the allocation failure for the command packets.

Jesse

drivers/scsi/megaraid/megaraid_mbox.c +1 -1
index b7caf60..032a3d7 100644
@@ -457,7 +457,7 @@ megaraid_probe_one(struct pci_dev *pdev,
 
 	// Setup the default DMA mask. This would be changed later on
 	// depending on hardware capabilities
-	if (pci_set_dma_mask(adapter->pdev, DMA_32BIT_MASK) != 0) {
+	if (pci_set_dma_mask(adapter->pdev, DMA_64BIT_MASK) != 0) {
 
 		con_log(CL_ANN, (KERN_WARNING
 			"megaraid: pci_set_dma_mask failed:%d\n", __LINE__));
#2 lore raw

I've tried the diff but there is no difference.
I've also tried to use the EFI driver from Intel, but that did not work either.

2006/6/29, Jesse Barnes <[email protected]>:

Quoted 36 lines · Robert Nagy
Quoted 13 lines

Hi,

Distribution: Debian testing/unstable
Hardware Environment: SGI Altix 350, 2xItanium 2, EFI (read dmesg)
http://bsd.hu/~robert/altix.dmesg
http://bsd.hu/~robert/altix.kconf

Problem Description: I've installed an Intel(r) RAID Controller SRCU42X (PCI-X) controller to this machine.
http://www.intel.com/design/servers/raid/srcu42x/index.htm
I've never used such a controller so if someone has any idea about this please tell me. The dmesg will show everyhing, but:

Quoted 5 lines

megaraid cmm: 2.20.2.6 (Release Date: Mon Mar 7 00:01:03 EST 2005)
megaraid: 2.20.4.8 (Release Date: Mon Apr 11 12:27:22 EST 2006)
megaraid: probe new device 0x1000:0x0407:0x8086:0x0532: bus 2:slot 0:func 0 megaraid: out of memory, megaraid_alloc_cmd_packets 965
megaraid: maibox adapter did not initialize

IIRC some Altix boxes don't support 32 bit DMA for PCI-X devices. Based on the initialization code I looked at (just a quick scan), it looks like the command packet initialization is done before the switch to a 64 bit DMA mask, which might cause the failure you see here. You can try this patch out (totally untested). It's not fully correct, it should probably try 64 bit first then fall back to 32 bit if that fails then give up, and the other 64 bit DMA mask call should probably be removed.

Anyway, good luck. If this one doesn't work you'll have to talk with one of the SGI guys and get some more debug info about the allocation failure for the command packets.

Jesse

#3 lore raw
Quoted 3 lines · Robert Nagy

I've tried the diff but there is no difference.
I've also tried to use the EFI driver from Intel, but that did not work either.

Yeah, using the EFI driver won't help at all, as it's only available in EFI context (it might let you boot of the raid but that's about it).

If you applied the diff and recompiled the megaraid driver and still got the same error, I'm not sure what the problem is....

Jesse

#4 lore raw

I know that it only works in EFI context. But I haven't seen the disk attached to it. I am going to try forcing it to PCI mode.

2006/6/29, Jesse Barnes <[email protected]>:

Quoted 13 lines · Robert Nagy
Quoted 3 lines

I've tried the diff but there is no difference.
I've also tried to use the EFI driver from Intel, but that did not work either.

Yeah, using the EFI driver won't help at all, as it's only available in EFI context (it might let you boot of the raid but that's about it).

If you applied the diff and recompiled the megaraid driver and still got the same error, I'm not sure what the problem is....

Jesse

#5 lore raw
Quoted 2 lines · Robert Nagy

I know that it only works in EFI context. But I haven't seen the disk attached to it. I am going to try forcing it to PCI mode.

Ah ok, just as a sanity check? It's been awhile since I looked at the EFI driver API, it could very well have problems on a machine like Altix due to memory mapping constraints and such.

Jesse

#6 lore raw
In reply to Robert Nagy
Quoted 3 lines · Robert Nagy

I've tried the diff but there is no difference.
I've also tried to use the EFI driver from Intel, but that did not work either.

I've just been informed that megaraid has command ring addressing limitations, so you may not be able to use this card in PCI-X mode at all, at least on your Altix. You can force it into PCI mode by putting an old PCI device in the same bus though, I think, that might get things working (without my patch).

Jesse

#7 lore raw

I've tried that with two different cards. Now the error is different. Even the firmware boots on the controller but then the machine resets. Same thing happens if I load the EFI driver but that drops me to the debugger. More info can be found at http://pastebin.ca/75652

megaraid cmm: 2.20.2.6 (Release Date: Mon Mar 7 00:01:03 EST 2005)
megaraid: 2.20.4.8 (Release Date: Mon Apr 11 12:27:22 EST 2006)
megaraid: probe new device 0x1000:0x0407:0x8086:0x0532: bus 2:slot 0:func 0
ACPI: PCI Interrupt 0002:02:00.0[A]: no GSI
megaraid mailbox: wait for FW to boot [ok]
Entered OS MCA handler. PSP=20000000fff21120 cpu=0 monarch=1 All OS MCA slaves have reached rendezvous

2006/6/29, Jesse Barnes <[email protected]>:

Quoted 13 lines · Robert Nagy
Quoted 3 lines

I've tried the diff but there is no difference.
I've also tried to use the EFI driver from Intel, but that did not work either.

I've just been informed that megaraid has command ring addressing limitations, so you may not be able to use this card in PCI-X mode at all, at least on your Altix. You can force it into PCI mode by putting an old PCI device in the same bus though, I think, that might get things working (without my patch).

Jesse

#8 lore raw
Quoted 5 lines · Robert Nagy

I've tried that with two different cards. Now the error is different. Even the firmware boots on the controller but then the machine resets. Same thing happens if I load the EFI driver but that drops me to the debugger. More info can be found at http://pastebin.ca/75652

Quoted 7 lines

megaraid cmm: 2.20.2.6 (Release Date: Mon Mar 7 00:01:03 EST 2005)
megaraid: 2.20.4.8 (Release Date: Mon Apr 11 12:27:22 EST 2006)
megaraid: probe new device 0x1000:0x0407:0x8086:0x0532: bus 2:slot 0:func 0 ACPI: PCI Interrupt 0002:02:00.0[A]: no GSI megaraid mailbox: wait for FW to boot [ok]
Entered OS MCA handler. PSP=20000000fff21120 cpu=0 monarch=1 All OS MCA slaves have reached rendezvous

This is what happens when you have PCI card in the bus next to your RAID card and run without my patch? Hm... this might be a regular driver bug. Interesting that this driver might do an msleep right after the [ok] is printed. Do you have kdb builtin to your kernel? If so, maybe you could get a backtrace. Otherwise you could put in some printk statements to see if we can figure out where the MCA is occuring...

Jesse

#9 lore raw

no i do not have kdb. and i cannot even boot the box now. is there any way to disable the megaraid driver with an argument?

2006/6/30, Jesse Barnes <[email protected]>:

Quoted 23 lines · Robert Nagy
Quoted 5 lines

I've tried that with two different cards. Now the error is different. Even the firmware boots on the controller but then the machine resets. Same thing happens if I load the EFI driver but that drops me to the debugger. More info can be found at http://pastebin.ca/75652

Quoted 7 lines

megaraid cmm: 2.20.2.6 (Release Date: Mon Mar 7 00:01:03 EST 2005)
megaraid: 2.20.4.8 (Release Date: Mon Apr 11 12:27:22 EST 2006)
megaraid: probe new device 0x1000:0x0407:0x8086:0x0532: bus 2:slot 0:func 0 ACPI: PCI Interrupt 0002:02:00.0[A]: no GSI megaraid mailbox: wait for FW to boot [ok]
Entered OS MCA handler. PSP=20000000fff21120 cpu=0 monarch=1 All OS MCA slaves have reached rendezvous

This is what happens when you have PCI card in the bus next to your RAID card and run without my patch? Hm... this might be a regular driver bug. Interesting that this driver might do an msleep right after the [ok] is printed. Do you have kdb builtin to your kernel? If so, maybe you could get a backtrace. Otherwise you could put in some printk statements to see if we can figure out where the MCA is occuring...

Jesse

#10 lore raw
Quoted 2 lines · Robert Nagy

no i do not have kdb. and i cannot even boot the box now. is there any way to disable the megaraid driver with an argument?

Not that I know of. But you can use your system controller to disable the PCI slot containing the RAID card... that should let you boot. I don't remember what the command is, but you can type 'help' at your L1 or L2 prompt.

Jesse