Skip to contentSkip to conversation

qlogicpti driver hangs the machine

DISCUSSION Meelis Roos 7 participants 15 messages Sep 23, 2008 – Sep 29, 2008 SCSI

lore
Share & tools
lore
#0 lore raw

I tried to use Qlogic wide SE SCSI SBus controllers. First tied one in Sparcstation 5 (sparc32) with etch and sarge installers, then tried another in E3500. Both tries on SS5 hung when loadin qlogicpti module. E3500 conf doesn't autoload SBus drivers and the card was properly inserted only after installation, so I tried to modprobe qlogicpti after booting up. This also results in a hang but outputs that much on console before hanging. Maybe the error handling spinlock lockup gives a clue to someone...

[ 3570.576359] qlogicpti0: IRQ 12 SCSI ID 7 (Firmware v1.31.32)(FCode 1.21 95/05/18)
[ 3571.002975] qlogicpti0: [Fast Wide, using single ended interface]
[ 3571.078001] scsi2 : PTI Qlogic,ISP SBUS SCSI irq 12 regs at 000001c520010000
[ 3571.667582] scsi 2:0:2:0: Direct-Access     FUJITSU  M2954ESP SUN4.2G 2545 PQ: 0 ANSI: 2
[ 3571.768584] sd 2:0:2:0: [sdg] 8385121 512-byte hardware sectors (4293 MB)
[ 3571.852563] sd 2:0:2:0: [sdg] Write Protect is off
[ 3571.907932] sd 2:0:2:0: [sdg] Mode Sense: 9f 00 00 08
[ 3571.973016] sd 2:0:2:0: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 3572.082467] sd 2:0:2:0: [sdg] 8385121 512-byte hardware sectors (4293 MB)
[ 3572.166373] sd 2:0:2:0: [sdg] Write Protect is off
[ 3572.221718] sd 2:0:2:0: [sdg] Mode Sense: 9f 00 00 08
[ 3572.286814] sd 2:0:2:0: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 3572.394250]  sdg: unknown partition table
[ 3572.455839] sd 2:0:2:0: [sdg] Attached SCSI disk
[ 3572.511006] sd 2:0:2:0: Attached scsi generic sg7 type 0
[ 3578.075788] qlogicpti0: Aborting cmd for tgt[2] lun[0]
[ 3578.137633] qlogicpti0: Aborting cmd for tgt[2] lun[1]
[ 3696.467069] BUG: spinlock lockup on CPU#7, scsi_eh_2/1737, fffff8004d0f8060
[ 3696.548319] Call Trace:
[ 3696.577498]  [0000000000590e5c] _raw_spin_lock+0xfc/0x160
[ 3696.642105]  [000000000069dc5c] _spin_lock_irqsave+0x1c/0x40
[ 3696.709816]  [00000000005d5090] scsi_send_eh_cmnd+0x70/0x280
[ 3696.777530]  [00000000005d52c4] scsi_eh_tur+0x24/0xa0
[ 3696.837958]  [00000000005d5f8c] scsi_error_handler+0x2cc/0x3c0
[ 3696.907783]  [000000000047676c] kthread+0x4c/0x80
[ 3696.964032]  [000000000042707c] kernel_thread+0x3c/0x60
[ 3697.026543]  [0000000000476694] kthreadd+0x174/0x200
[ 3697.086064]   CPU[  6]: TSTATE[0000009980009602] TPC[000000000069dc40] TNPC[000000000069dc44] TASK[NULL:-1]
[ 3697.202709]              TPC[_spin_lock_irqsave+0x0/0x40] O7[complete+0x14/0x60] I7[scsi_eh_done+0x24/0x40] RPC[qpti_intr+0]
[ 3697.359973] * CPU[  7]: TSTATE[0000000080009625] TPC[0000000000427074] TNPC[0000000000427078] TASK[scsi_eh_2:1737]
[ 3697.483975]              TPC[kernel_thread+0x34/0x60] O7[kernel_thread+0x8/0x60] I7[kthreadd+0x174/0x200] RPC[0x0]
Signature · 1 line
Meelis Roos ([email protected])      http://www.cs.ut.ee/~mroos/
#1 lore raw
Quoted 8 lines · Meelis Roos

I tried to use Qlogic wide SE SCSI SBus controllers. First tied one in Sparcstation 5 (sparc32) with etch and sarge installers, then tried another in E3500. Both tries on SS5 hung when loadin qlogicpti module. E3500 conf doesn't autoload SBus drivers and the card was properly inserted only after installation, so I tried to modprobe qlogicpti after booting up. This also results in a hang but outputs that much on console before hanging. Maybe the error handling spinlock lockup gives a clue to someone...

What kernel version is this trace from?

Quoted 38 lines
[ 3570.576359] qlogicpti0: IRQ 12 SCSI ID 7 (Firmware v1.31.32)(FCode 1.21 95/05/18)
[ 3571.002975] qlogicpti0: [Fast Wide, using single ended interface]
[ 3571.078001] scsi2 : PTI Qlogic,ISP SBUS SCSI irq 12 regs at 000001c520010000
[ 3571.667582] scsi 2:0:2:0: Direct-Access     FUJITSU  M2954ESP SUN4.2G 2545 PQ: 0 ANSI: 2
[ 3571.768584] sd 2:0:2:0: [sdg] 8385121 512-byte hardware sectors (4293 MB)
[ 3571.852563] sd 2:0:2:0: [sdg] Write Protect is off
[ 3571.907932] sd 2:0:2:0: [sdg] Mode Sense: 9f 00 00 08
[ 3571.973016] sd 2:0:2:0: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 3572.082467] sd 2:0:2:0: [sdg] 8385121 512-byte hardware sectors (4293 MB)
[ 3572.166373] sd 2:0:2:0: [sdg] Write Protect is off
[ 3572.221718] sd 2:0:2:0: [sdg] Mode Sense: 9f 00 00 08
[ 3572.286814] sd 2:0:2:0: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 3572.394250]  sdg: unknown partition table
[ 3572.455839] sd 2:0:2:0: [sdg] Attached SCSI disk
[ 3572.511006] sd 2:0:2:0: Attached scsi generic sg7 type 0
[ 3578.075788] qlogicpti0: Aborting cmd for tgt[2] lun[0]
[ 3578.137633] qlogicpti0: Aborting cmd for tgt[2] lun[1]
[ 3696.467069] BUG: spinlock lockup on CPU#7, scsi_eh_2/1737, fffff8004d0f8060
[ 3696.548319] Call Trace:
[ 3696.577498]  [0000000000590e5c] _raw_spin_lock+0xfc/0x160
[ 3696.642105]  [000000000069dc5c] _spin_lock_irqsave+0x1c/0x40
[ 3696.709816]  [00000000005d5090] scsi_send_eh_cmnd+0x70/0x280
[ 3696.777530]  [00000000005d52c4] scsi_eh_tur+0x24/0xa0
[ 3696.837958]  [00000000005d5f8c] scsi_error_handler+0x2cc/0x3c0
[ 3696.907783]  [000000000047676c] kthread+0x4c/0x80
[ 3696.964032]  [000000000042707c] kernel_thread+0x3c/0x60
[ 3697.026543]  [0000000000476694] kthreadd+0x174/0x200
[ 3697.086064]   CPU[  6]: TSTATE[0000009980009602] TPC[000000000069dc40] TNPC[000000000069dc44] TASK[NULL:-1]
[ 3697.202709]              TPC[_spin_lock_irqsave+0x0/0x40] O7[complete+0x14/0x60] I7[scsi_eh_done+0x24/0x40] RPC[qpti_intr+0]
[ 3697.359973] * CPU[  7]: TSTATE[0000000080009625] TPC[0000000000427074] TNPC[0000000000427078] TASK[scsi_eh_2:1737]
[ 3697.483975]              TPC[kernel_thread+0x34/0x60] O7[kernel_thread+0x8/0x60] I7[kthreadd+0x174/0x200] RPC[0x0]
Signature · 5 lines
Meelis Roos ([email protected])      http://www.cs.ut.ee/~mroos/
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 4 lines
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
#2 lore raw
Quoted 1 line

What kernel version is this trace from?

Oops, forgot - todays 2.6.27-rc7+git.

Signature · 1 line
Meelis Roos ([email protected])
Acked-by
#3 lore raw
In reply to Matthew Wilcox

From: Matthew Wilcox <[email protected]>
Date: Tue, 23 Sep 2008 09:59:18 -0600

Quoted 11 lines · Meelis Roos
Quoted 8 lines

I tried to use Qlogic wide SE SCSI SBus controllers. First tied one in Sparcstation 5 (sparc32) with etch and sarge installers, then tried another in E3500. Both tries on SS5 hung when loadin qlogicpti module. E3500 conf doesn't autoload SBus drivers and the card was properly inserted only after installation, so I tried to modprobe qlogicpti after booting up. This also results in a hang but outputs that much on console before hanging. Maybe the error handling spinlock lockup gives a clue to someone...

What kernel version is this trace from?

FWIW, I personally haven't tested this driver in a long time, did anyone give it at least a simple test when you made those ->slave_configure changes to this driver?

I'm talking about this one:

commit 9ec76fbf7d6da3e98070a7059699d0ca019b0c9b

Author: Matthew Wilcox <[email protected]>
Date:   Mon Oct 1 08:07:52 2007 -0600

[SCSI] qlogicpti: Add a slave_configure method

By configuring targets in slave_configure, we can eliminate a shadow queuecommand, a shadow scsi_done, a write to the host template, abuse of SCp->Message and SCp->Status, a use of kmap_atomic() and sniffing the results of INQUIRY.

#4 lore raw
Quoted 13 lines · David Miller

From: Matthew Wilcox <[email protected]>
Date: Tue, 23 Sep 2008 09:59:18 -0600

Quoted 10 lines · Meelis Roos
Quoted 8 lines

I tried to use Qlogic wide SE SCSI SBus controllers. First tied one in Sparcstation 5 (sparc32) with etch and sarge installers, then tried another in E3500. Both tries on SS5 hung when loadin qlogicpti module. E3500 conf doesn't autoload SBus drivers and the card was properly inserted only after installation, so I tried to modprobe qlogicpti after booting up. This also results in a hang but outputs that much on console before hanging. Maybe the error handling spinlock lockup gives a clue to someone...

Quoted 5 lines
Quoted 1 line

What kernel version is this trace from?

FWIW, I personally haven't tested this driver in a long time, did anyone give it at least a simple test when you made those ->slave_configure changes to this driver?

Not that I know of, but I do have an incriminating email from you where you promise to pick up the pieces if anything explodes ;-)

There've also been commits from Jens, Boaz and Tomo that could plausibly be related (I think we can discount the one from Adrian). Meelis, can you run through the various patches to qlogicpti, backing them out one at a time? I worry that some infrastructure may have been taken away by followups to these patches, so it may not be feasible.

Signature · 4 lines
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
#5 lore raw

From: Matthew Wilcox <[email protected]>
Date: Tue, 23 Sep 2008 22:42:11 -0600

Quoted 14 lines · David Miller
Quoted 13 lines

From: Matthew Wilcox <[email protected]>
Date: Tue, 23 Sep 2008 09:59:18 -0600

Quoted 10 lines · Meelis Roos
Quoted 8 lines

I tried to use Qlogic wide SE SCSI SBus controllers. First tied one in Sparcstation 5 (sparc32) with etch and sarge installers, then tried another in E3500. Both tries on SS5 hung when loadin qlogicpti module. E3500 conf doesn't autoload SBus drivers and the card was properly inserted only after installation, so I tried to modprobe qlogicpti after booting up. This also results in a hang but outputs that much on console before hanging. Maybe the error handling spinlock lockup gives a clue to someone...

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

What kernel version is this trace from?

FWIW, I personally haven't tested this driver in a long time, did anyone give it at least a simple test when you made those ->slave_configure changes to this driver?

Not that I know of, but I do have an incriminating email from you where you promise to pick up the pieces if anything explodes ;-)

And here I am :-)

Quoted 5 lines

There've also been commits from Jens, Boaz and Tomo that could plausibly be related (I think we can discount the one from Adrian). Meelis, can you run through the various patches to qlogicpti, backing them out one at a time? I worry that some infrastructure may have been taken away by followups to these patches, so it may not be feasible.

I know, I looked at those too. Backing many of them out is infeasible, especially the scatterlist ones.

I'm about to stick a qlogicpti card into my ultra2 and try to reproduce this.

#6 lore raw

From: David Miller <[email protected]>
Date: Tue, 23 Sep 2008 21:53:02 -0700 (PDT)

Quoted 2 lines

I'm about to stick a qlogicpti card into my ultra2 and try to reproduce this.

Unfortunately, the driver works just fine for me, in fact it's checking the root filesystem right now :-)

Meelis, if Matthew isn't able to give you some debugging patches or other stuff to try, I'll try to get this fixed when I get back from the netfilter workshop in Paris in 2 weeks.

#7 lore raw
Quoted 9 lines · David Miller

From: David Miller <[email protected]>
Date: Tue, 23 Sep 2008 21:53:02 -0700 (PDT)

Quoted 2 lines

I'm about to stick a qlogicpti card into my ultra2 and try to reproduce this.

Unfortunately, the driver works just fine for me, in fact it's checking the root filesystem right now :-)

Quoted 7 lines

Meelis, if Matthew isn't able to give you some debugging patches or other stuff to try, I'll try to get this fixed when I get back from the netfilter workshop in Paris in 2 weeks.
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Subject: [PATCH] fix fall out of sg-chaining patch

I've reviewed all patches since Matthew's, and I find one small problem.

In the load_cmd() there is a compound loop where the first 4 sg's are set then the rest are set into a memory structure in group of 7 sg's.

Well the second 7-group and on is a bug because sg pointer does not advance. This is a fall out from Jens's patch.

I'm not sure if this is it but it should be fixed
---
git diff --stat -p

 drivers/scsi/qlogicpti.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
drivers/scsi/qlogicpti.c +1 -0
index 4a1cf63..9053508 100644
@@ -914,6 +914,7 @@ static inline int load_cmd(struct scsi_cmnd *Cmnd, struct Command_Entry *cmd,
 				ds[i].d_count = sg_dma_len(s);
 			}
 			sg_count -= n;
+			sg = s;
 		}
 	} else {
 		cmd->dataseg[0].d_base = 0;
#8 lore raw

From: Boaz Harrosh <[email protected]>
Date: Wed, 24 Sep 2008 12:00:22 +0300

Quoted 5 lines

Subject: [PATCH] fix fall out of sg-chaining patch

I've reviewed all patches since Matthew's, and I find one small problem.

Quoted 7 lines

In the load_cmd() there is a compound loop where the first 4 sg's are set then the rest are set into a memory structure in group of 7 sg's.

Well the second 7-group and on is a bug because sg pointer does not advance. This is a fall out from Jens's patch.

I'm not sure if this is it but it should be fixed

Definitely a bug, and I looked at the patch that addded this code, specifically audited this stuff, and didn't notice the problem. :-)

Meelis, see if this patch helps out your case, thanks!

Quoted 21 lines

---
git diff --stat -p

 drivers/scsi/qlogicpti.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
drivers/scsi/qlogicpti.c +1 -1
index 4a1cf63..9053508 100644
@@ -914,6 +914,7 @@ static inline int load_cmd(struct scsi_cmnd *Cmnd, struct Command_Entry *cmd,
 				ds[i].d_count = sg_dma_len(s);
 			}
 			sg_count -= n;
+			sg = s;
 		}
 	} else {
 		cmd->dataseg[0].d_base = 0;
--

To unsubscribe from this list: send the line "unsubscribe sparclinux" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#9 lore raw
Quoted 1 line

Meelis, see if this patch helps out your case, thanks!

Will try soon.

Signature · 1 line
Meelis Roos ([email protected])      http://www.cs.ut.ee/~mroos/
#10 lore raw
In reply to Boaz Harrosh
Quoted 5 lines

Subject: [PATCH] fix fall out of sg-chaining patch

I've reviewed all patches since Matthew's, and I find one small problem.

Quoted 7 lines

In the load_cmd() there is a compound loop where the first 4 sg's are set then the rest are set into a memory structure in group of 7 sg's.

Well the second 7-group and on is a bug because sg pointer does not advance. This is a fall out from Jens's patch.

I'm not sure if this is it but it should be fixed

This definitely helps,

# modprobe qlogicpti
[  271.742828] qlogicpti0: IRQ 12 SCSI ID 7 (Firmware v1.31.32)(FCode 1.21 95/05/18)
[  272.168804] qlogicpti0: [Fast Wide, using single ended interface]
[  272.243809] scsi3 : PTI Qlogic,ISP SBUS SCSI irq 12 regs at 000001c520010000
[  272.835116] scsi 3:0:2:0: Direct-Access     FUJITSU  M2954ESP SUN4.2G 2545 PQ: 0 ANSI: 2
[  272.942031] sd 3:0:2:0: [sdg] 8385121 512-byte hardware sectors (4293 MB)
[  273.026123] sd 3:0:2:0: [sdg] Write Protect is off
[  273.087748] sd 3:0:2:0: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[  273.198290] sd 3:0:2:0: [sdg] 8385121 512-byte hardware sectors (4293 MB)
[  273.282354] sd 3:0:2:0: [sdg] Write Protect is off
[  273.342641] sd 3:0:2:0: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[  273.450154]  sdg: unknown partition table
[  273.511662] sd 3:0:2:0: [sdg] Attached SCSI disk
[  273.566006] sd 3:0:2:0: Attached scsi generic sg7 type 0
[  279.133183] qlogicpti0: Aborting cmd for tgt[2] lun[0]
[  279.193222] qlogicpti0: Aborting cmd for tgt[2] lun[1]
[  279.259841] qlogicpti0: Resetting SCSI bus!

and here it finishes and sdg seems to be working fine. There are two strange command aborts followed by a bus reset, but it recovers and seems to work (reading whole disc with dd seems to work at least).

Thanks!

Signature · 1 line
Meelis Roos ([email protected])      http://www.cs.ut.ee/~mroos/
#11 lore raw

From: Meelis Roos <[email protected]>
Date: Wed, 24 Sep 2008 12:51:09 +0300 (EEST)

Quoted 8 lines
Quoted 8 lines

Subject: [PATCH] fix fall out of sg-chaining patch

I've reviewed all patches since Matthew's, and I find one small problem.

In the load_cmd() there is a compound loop where the first 4 sg's are set then the rest are set into a memory structure in group of 7 sg's.

Quoted 6 lines
Quoted 4 lines

Well the second 7-group and on is a bug because sg pointer does not advance. This is a fall out from Jens's patch.

I'm not sure if this is it but it should be fixed

This definitely helps,

Can we get this bug fix merged if it isn't already?

Thanks.

#12 lore raw
Quoted 11 lines · David Miller

From: Meelis Roos <[email protected]>
Date: Wed, 24 Sep 2008 12:51:09 +0300 (EEST)

Quoted 8 lines
Quoted 8 lines

Subject: [PATCH] fix fall out of sg-chaining patch

I've reviewed all patches since Matthew's, and I find one small problem.

In the load_cmd() there is a compound loop where the first 4 sg's are set then the rest are set into a memory structure in group of 7 sg's.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

Well the second 7-group and on is a bug because sg pointer does not advance. This is a fall out from Jens's patch.

I'm not sure if this is it but it should be fixed

This definitely helps,

Can we get this bug fix merged if it isn't already?

I'll take care of it.

Signature · 1 line
Jens Axboe
maintainer
#13 lore raw
Quoted 15 lines · Jens Axboe
Quoted 14 lines · David Miller

From: Meelis Roos <[email protected]>
Date: Wed, 24 Sep 2008 12:51:09 +0300 (EEST)

Quoted 11 lines
Quoted 11 lines

Subject: [PATCH] fix fall out of sg-chaining patch

I've reviewed all patches since Matthew's, and I find one small problem.

In the load_cmd() there is a compound loop where the first 4 sg's are set then the rest are set into a memory structure in group of 7 sg's.

Well the second 7-group and on is a bug because sg pointer does not advance. This is a fall out from Jens's patch.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

I'm not sure if this is it but it should be fixed

This definitely helps,

Can we get this bug fix merged if it isn't already?

I'll take care of it.

Actually, I already have it in rc fixes which are due out this morning.

James

#14 lore raw
In reply to David Miller
Quoted 2 lines

Unfortunately, the driver works just fine for me, in fact it's checking the root filesystem right now :-)

Hmm, it might to do with the disk that I tried with qlogicpti's on both systems. It's an externa SUN4.2G disk in unipack enclosure. I don't know about its health, only that it shows up in probe-scsi-all PROM command.

Signature · 1 line
Meelis Roos ([email protected])      http://www.cs.ut.ee/~mroos/