Skip to contentSkip to conversation

sata_mv, io stucks

DISCUSSION Artem Bokhan 4 participants 18 messages Oct 17, 2008 – Nov 17, 2008 libata

lore
Share & tools
lore
#0 lore raw

I try to simulate random reads with "sysbench --test=fileio --num-threads=16 --max-requests=9999999 --max-time=60 --init-rng=on --file-num=16 --file-fsync-freq=0 --file-test-mode=rndrd --file-total-size=30G run"

Two marvell controllers, 16 disks, software raid10, IO stucks on different disks, kernel 2.6.26.5. With default ubuntu's 8.04 2.6.24 kernel the problem can not be repeated

[  289.851609] ata11.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
[  289.851695] ata11.00: cmd 61/08:00:60:1e:bf/00:00:01:00:00/40 tag 0 ncq 4096 out
[  289.851697]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  289.851774] ata11.00: status: { DRDY }
[  289.851834] ata11: hard resetting link
[  290.649259] ata11: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[  290.749239] ata11.00: max_sectors limited to 256 for NCQ
[  290.809189] ata11.00: max_sectors limited to 256 for NCQ
[  290.809194] ata11.00: configured for UDMA/133
[  290.809200] ata11: EH complete
[  290.809242] sd 10:0:0:0: [sdk] 1953525168 512-byte hardware sectors (1000205 MB)
[  290.809258] sd 10:0:0:0: [sdk] Write Protect is off
[  290.809263] sd 10:0:0:0: [sdk] Mode Sense: 00 3a 00 00
[  290.809286] sd 10:0:0:0: [sdk] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

lspci:

05:01.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09)
Subsystem: Marvell Technology Group Ltd. Unknown device 11ab
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr+ Stepping- SERR+ FastB2B+
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 32, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 24
Region 0: Memory at d8200000 (64-bit, non-prefetchable) [size=1M]
Region 2: I/O ports at 3000 [size=256]
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+
Queue=0/0 Enable-

               Address: 0000000000000000  Data: 0000
       Capabilities: [60] PCI-X non-bridge device
               Command: DPERE- ERO- RBC=512 OST=4
               Status: Dev=05:01.0 64bit+ 133MHz+ SCD- USC- DC=simple 

DMMRBC=512 DMOST=4 DMCRS=8 RSCEM- 266MHz- 533MHz-

05:02.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09)
Subsystem: Marvell Technology Group Ltd. Unknown device 11ab
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr+ Stepping- SERR+ FastB2B+
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-

       Latency: 32, Cache Line Size: 32 bytes
       Interrupt: pin A routed to IRQ 28
       Region 0: Memory at d8300000 (64-bit, non-prefetchable) [size=1M]
       Region 2: I/O ports at 3400 [size=256]
       Region 3: [virtual] Memory at d8800000 (32-bit, 

non-prefetchable) [size=4M]
[virtual] Expansion ROM at d9000000 [disabled] [size=4M]
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+
Queue=0/0 Enable-

               Address: 0000000000000000  Data: 0000
       Capabilities: [60] PCI-X non-bridge device
               Command: DPERE- ERO- RBC=512 OST=4
               Status: Dev=05:02.0 64bit+ 133MHz+ SCD- USC- DC=simple 

DMMRBC=512 DMOST=4 DMCRS=8 RSCEM- 266MHz- 533MHz-

#1 lore raw

Any thought?

Artem Bokhan пишет:

Quoted 78 lines

I try to simulate random reads with "sysbench --test=fileio --num-threads=16 --max-requests=9999999 --max-time=60 --init-rng=on --file-num=16 --file-fsync-freq=0 --file-test-mode=rndrd --file-total-size=30G run"

Two marvell controllers, 16 disks, software raid10, IO stucks on different disks, kernel 2.6.26.5. With default ubuntu's 8.04 2.6.24 kernel the problem can not be repeated

[  289.851609] ata11.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
[  289.851695] ata11.00: cmd 61/08:00:60:1e:bf/00:00:01:00:00/40 tag 0 ncq 4096 out
[  289.851697]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  289.851774] ata11.00: status: { DRDY }
[  289.851834] ata11: hard resetting link
[  290.649259] ata11: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[  290.749239] ata11.00: max_sectors limited to 256 for NCQ
[  290.809189] ata11.00: max_sectors limited to 256 for NCQ
[  290.809194] ata11.00: configured for UDMA/133
[  290.809200] ata11: EH complete
[  290.809242] sd 10:0:0:0: [sdk] 1953525168 512-byte hardware sectors (1000205 MB)
[  290.809258] sd 10:0:0:0: [sdk] Write Protect is off
[  290.809263] sd 10:0:0:0: [sdk] Mode Sense: 00 3a 00 00
[  290.809286] sd 10:0:0:0: [sdk] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

lspci:

05:01.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09)
Subsystem: Marvell Technology Group Ltd. Unknown device 11ab
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr+ Stepping- SERR+ FastB2B+
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium

Quoted 1 line

TAbort- <TAbort- <MAbort- >SERR- <PERR-

Latency: 32, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 24
Region 0: Memory at d8200000 (64-bit, non-prefetchable) [size=1M] Region 2: I/O ports at 3000 [size=256]
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+
Queue=0/0 Enable-

               Address: 0000000000000000  Data: 0000
       Capabilities: [60] PCI-X non-bridge device
               Command: DPERE- ERO- RBC=512 OST=4
               Status: Dev=05:01.0 64bit+ 133MHz+ SCD- USC- DC=simple 

DMMRBC=512 DMOST=4 DMCRS=8 RSCEM- 266MHz- 533MHz-

05:02.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09)
Subsystem: Marvell Technology Group Ltd. Unknown device 11ab
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr+ Stepping- SERR+ FastB2B+
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium

Quoted 1 line

TAbort- <TAbort- <MAbort- >SERR- <PERR-

       Latency: 32, Cache Line Size: 32 bytes
       Interrupt: pin A routed to IRQ 28
       Region 0: Memory at d8300000 (64-bit, non-prefetchable) [size=1M]
       Region 2: I/O ports at 3400 [size=256]
       Region 3: [virtual] Memory at d8800000 (32-bit, 

non-prefetchable) [size=4M]
[virtual] Expansion ROM at d9000000 [disabled] [size=4M]
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+
Queue=0/0 Enable-

               Address: 0000000000000000  Data: 0000
       Capabilities: [60] PCI-X non-bridge device
               Command: DPERE- ERO- RBC=512 OST=4
               Status: Dev=05:02.0 64bit+ 133MHz+ SCD- USC- DC=simple 

DMMRBC=512 DMOST=4 DMCRS=8 RSCEM- 266MHz- 533MHz-

Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-ide" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#16 lore raw
In reply to Artem Bokhan
Quoted 7 lines · Artem Bokhan

I try to simulate random reads with "sysbench --test=fileio --num-threads=16 --max-requests=9999999 --max-time=60 --init-rng=on --file-num=16 --file-fsync-freq=0 --file-test-mode=rndrd --file-total-size=30G run"

Two marvell controllers, 16 disks, software raid10, IO stucks on different disks, kernel 2.6.26.5. With default ubuntu's 8.04 2.6.24 kernel the problem can not be repeated

I have the same problem with recent kernels with updated sata_mv driver. First IO stops for a while and afer EH runs, everything works again for a while. Happens on 3 different computers using WD5000ABYS, WD5000YS and WD7500AYYS hard disks, RAID5 and 6 configurations using Linux MD.

Stalls seem to happen only on controller ports 0-3, ports 4-7 work without problems.

Contoller is Supermicro AOC-SAT2-MV8, connected to 133MHz PCI-X slot on one computer, 66MHz 64bit PCI slot on the second machine and to normal 32bit PCI slot on third computer.
http://www.supermicro.com/products/accessories/addon/AoC-SAT2-MV8.cfm

At the moment I don't have disks connected to failing ports, but if needed, I can test patches.

Oct 10 18:56:17 mizar kernel: ata10.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Oct 10 18:56:17 mizar kernel: ata10.00: cmd 35/00:08:3f:52:54/00:00:57:00:00/e0 tag 0 dma 4096 out
Oct 10 18:56:17 mizar kernel:          res 40/00:ff:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
Oct 10 18:56:17 mizar kernel: ata10.00: status: { DRDY }
Oct 10 18:56:17 mizar kernel: ata10: hard resetting link
Oct 10 18:56:17 mizar kernel: ata10: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Oct 10 18:56:17 mizar kernel: ata10.00: max_sectors limited to 256 for NCQ
Oct 10 18:56:17 mizar kernel: ata10.00: max_sectors limited to 256 for NCQ
Oct 10 18:56:17 mizar kernel: ata10.00: configured for UDMA/33
Oct 10 18:56:17 mizar kernel: ata10: EH complete
Oct 10 18:56:17 mizar kernel: sd 9:0:0:0: [sdg] 1465149168 512-byte hardware sectors (750156 MB)
Oct 10 18:56:17 mizar kernel: sd 9:0:0:0: [sdg] Write Protect is off
Oct 10 18:56:17 mizar kernel: sd 9:0:0:0: [sdg] Mode Sense: 00 3a 00 00
Oct 10 18:56:17 mizar kernel: sd 9:0:0:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Oct 10 19:34:58 mizar kernel: ata10.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Oct 10 19:34:58 mizar kernel: ata10.00: cmd 35/00:08:3f:52:54/00:00:57:00:00/e0 tag 0 dma 4096 out
Oct 10 19:34:58 mizar kernel:          res 40/00:ff:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
Oct 10 19:34:58 mizar kernel: ata10.00: status: { DRDY }
Oct 10 19:34:58 mizar kernel: ata10: hard resetting link
Oct 10 19:34:58 mizar kernel: ata10: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Oct 10 19:34:58 mizar kernel: ata10.00: max_sectors limited to 256 for NCQ
Oct 10 19:34:58 mizar kernel: ata10.00: max_sectors limited to 256 for NCQ
Oct 10 19:34:58 mizar kernel: ata10.00: configured for UDMA/33
Oct 10 19:34:58 mizar kernel: ata10: EH complete
Oct 10 19:34:58 mizar kernel: sd 9:0:0:0: [sdg] 1465149168 512-byte hardware sectors (750156 MB)
Oct 10 19:34:58 mizar kernel: sd 9:0:0:0: [sdg] Write Protect is off
Oct 10 19:34:58 mizar kernel: sd 9:0:0:0: [sdg] Mode Sense: 00 3a 00 00
Oct 10 19:34:58 mizar kernel: sd 9:0:0:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Oct 10 19:37:05 mizar kernel: ata10.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Oct 10 19:37:05 mizar kernel: ata10.00: cmd 35/00:08:3f:52:54/00:00:57:00:00/e0 tag 0 dma 4096 out
Oct 10 19:37:05 mizar kernel:          res 40/00:ff:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
Oct 10 19:37:05 mizar kernel: ata10.00: status: { DRDY }
Oct 10 19:37:05 mizar kernel: ata10: hard resetting link
Oct 10 19:37:06 mizar kernel: ata10: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Oct 10 19:37:06 mizar kernel: ata10.00: max_sectors limited to 256 for NCQ
Oct 10 19:37:06 mizar kernel: ata10.00: max_sectors limited to 256 for NCQ
Oct 10 19:37:06 mizar kernel: ata10.00: configured for UDMA/33
Oct 10 19:37:06 mizar kernel: ata10: EH complete
Oct 10 19:37:06 mizar kernel: sd 9:0:0:0: [sdg] 1465149168 512-byte hardware sectors (750156 MB)
Oct 10 19:37:06 mizar kernel: sd 9:0:0:0: [sdg] Write Protect is off
Oct 10 19:37:06 mizar kernel: sd 9:0:0:0: [sdg] Mode Sense: 00 3a 00 00
Oct 10 19:37:06 mizar kernel: sd 9:0:0:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: exception Emask 0x0 SAct 0xf SErr 0x0 action 0x6 frozen
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: cmd 60/40:00:7f:a1:e2/00:00:28:00:00/40 tag 0 ncq 32768 in
Sep 26 15:47:14 mvsrv02 kernel:          res 40/00:00:09:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: status: { DRDY }
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: cmd 60/40:08:3f:a1:e2/00:00:28:00:00/40 tag 1 ncq 32768 in
Sep 26 15:47:14 mvsrv02 kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: status: { DRDY }
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: cmd 60/40:10:3f:a2:e2/00:00:28:00:00/40 tag 2 ncq 32768 in
Sep 26 15:47:14 mvsrv02 kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: status: { DRDY }
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: cmd 60/c0:18:7f:a2:e2/00:00:28:00:00/40 tag 3 ncq 98304 in
Sep 26 15:47:14 mvsrv02 kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: status: { DRDY }
Sep 26 15:47:14 mvsrv02 kernel: ata5: hard resetting link
Sep 26 15:47:14 mvsrv02 kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: max_sectors limited to 256 for NCQ
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: max_sectors limited to 256 for NCQ
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: configured for UDMA/133
Sep 26 15:47:14 mvsrv02 kernel: ata5: EH complete
Sep 26 15:47:14 mvsrv02 kernel: sd 4:0:0:0: [sdb] 976773168 512-byte hardware sectors (500108 MB)
Sep 26 15:47:14 mvsrv02 kernel: sd 4:0:0:0: [sdb] Write Protect is off
Sep 26 15:47:14 mvsrv02 kernel: sd 4:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Sep 26 15:47:14 mvsrv02 kernel: sd 4:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

1st comuter: 133MHz PCI-X slot
03:01.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09)
Subsystem: Marvell Technology Group Ltd. Unknown device 11ab
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr- Stepping- SERR- FastB2B+ DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 32, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 48
Region 0: Memory at d8800000 (64-bit, non-prefetchable) [size=1M]
Region 2: I/O ports at 3000 [size=256]
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+
Queue=0/0 Enable-

               Address: 0000000000000000  Data: 0000
       Capabilities: [60] PCI-X non-bridge device
               Command: DPERE- ERO- RBC=512 OST=4
               Status: Dev=03:01.0 64bit+ 133MHz+ SCD- USC- DC=simple 

DMMRBC=512 DMOST=4 DMCRS=8 RSCEM- 266MHz- 533MHz-
Kernel driver in use: sata_mv

2nd: 66MHz 64bit PCI
02:01.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09)
Subsystem: Marvell Technology Group Ltd. Unknown device 11ab
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 32, Cache Line Size: 128 bytes
Interrupt: pin A routed to IRQ 24
Region 0: Memory at f2800000 (64-bit, non-prefetchable) [size=1M]
Region 2: I/O ports at c000 [size=256]
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+
Queue=0/0 Enable-

               Address: 0000000000000000  Data: 0000
       Capabilities: [60] PCI-X non-bridge device
               Command: DPERE- ERO- RBC=512 OST=4
               Status: Dev=02:01.0 64bit+ 133MHz+ SCD- USC- DC=simple 

DMMRBC=512 DMOST=4 DMCRS=8 RSCEM- 266MHz- 533MHz-

3rd computer: 32bit 33MHz PCI
00:0a.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09)
Subsystem: Marvell Technology Group Ltd. Unknown device 11ab
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 32, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 16
Region 0: Memory at cfe00000 (64-bit, non-prefetchable) [size=1M]
Region 2: I/O ports at dc00 [size=256]
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: 64bit+
Queue=0/0 Enable-

               Address: 0000000000000000  Data: 0000
       Capabilities: [60] PCI-X non-bridge device
               Command: DPERE- ERO- RBC=512 OST=4
               Status: Dev=ff:1f.0 64bit+ 133MHz+ SCD- USC- DC=simple 

DMMRBC=512 DMOST=4 DMCRS=8 RSCEM- 266MHz- 533MHz-

Signature · 1 line
Harri.
#2 lore raw
In reply to Artem Bokhan
Quoted 26 lines · Artem Bokhan

Any thought?

Artem Bokhan пишет:

Quoted 23 lines

I try to simulate random reads with "sysbench --test=fileio --num-threads=16 --max-requests=9999999 --max-time=60 --init-rng=on --file-num=16 --file-fsync-freq=0 --file-test-mode=rndrd --file-total-size=30G run"

Two marvell controllers, 16 disks, software raid10, IO stucks on different disks, kernel 2.6.26.5. With default ubuntu's 8.04 2.6.24 kernel the problem can not be repeated

[  289.851609] ata11.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
[  289.851695] ata11.00: cmd 61/08:00:60:1e:bf/00:00:01:00:00/40 tag 0 ncq 4096 out
[  289.851697]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  289.851774] ata11.00: status: { DRDY }
[  289.851834] ata11: hard resetting link
[  290.649259] ata11: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[  290.749239] ata11.00: max_sectors limited to 256 for NCQ
[  290.809189] ata11.00: max_sectors limited to 256 for NCQ
[  290.809194] ata11.00: configured for UDMA/133
[  290.809200] ata11: EH complete
[  290.809242] sd 10:0:0:0: [sdk] 1953525168 512-byte hardware sectors (1000205 MB)
[  290.809258] sd 10:0:0:0: [sdk] Write Protect is off
[  290.809263] sd 10:0:0:0: [sdk] Mode Sense: 00 3a 00 00
[  290.809286] sd 10:0:0:0: [sdk] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

...

I've just returned here from a month holiday in Italy, and I'll have a look at this and other sata_mv issues next week or so.

Cheers

#17 lore raw
In reply to Harri Olin

The controller is AOC-SAT2-MV8 too.

Harri Olin пишет:

Quoted 157 lines
Quoted 7 lines · Artem Bokhan

I try to simulate random reads with "sysbench --test=fileio --num-threads=16 --max-requests=9999999 --max-time=60 --init-rng=on --file-num=16 --file-fsync-freq=0 --file-test-mode=rndrd --file-total-size=30G run"

Two marvell controllers, 16 disks, software raid10, IO stucks on different disks, kernel 2.6.26.5. With default ubuntu's 8.04 2.6.24 kernel the problem can not be repeated

I have the same problem with recent kernels with updated sata_mv driver. First IO stops for a while and afer EH runs, everything works again for a while. Happens on 3 different computers using WD5000ABYS, WD5000YS and WD7500AYYS hard disks, RAID5 and 6 configurations using Linux MD.

Stalls seem to happen only on controller ports 0-3, ports 4-7 work without problems.

Contoller is Supermicro AOC-SAT2-MV8, connected to 133MHz PCI-X slot on one computer, 66MHz 64bit PCI slot on the second machine and to normal 32bit PCI slot on third computer.
http://www.supermicro.com/products/accessories/addon/AoC-SAT2-MV8.cfm

At the moment I don't have disks connected to failing ports, but if needed, I can test patches.

Oct 10 18:56:17 mizar kernel: ata10.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Oct 10 18:56:17 mizar kernel: ata10.00: cmd 35/00:08:3f:52:54/00:00:57:00:00/e0 tag 0 dma 4096 out
Oct 10 18:56:17 mizar kernel:          res 40/00:ff:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
Oct 10 18:56:17 mizar kernel: ata10.00: status: { DRDY }
Oct 10 18:56:17 mizar kernel: ata10: hard resetting link
Oct 10 18:56:17 mizar kernel: ata10: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Oct 10 18:56:17 mizar kernel: ata10.00: max_sectors limited to 256 for NCQ
Oct 10 18:56:17 mizar kernel: ata10.00: max_sectors limited to 256 for NCQ
Oct 10 18:56:17 mizar kernel: ata10.00: configured for UDMA/33
Oct 10 18:56:17 mizar kernel: ata10: EH complete
Oct 10 18:56:17 mizar kernel: sd 9:0:0:0: [sdg] 1465149168 512-byte hardware sectors (750156 MB)
Oct 10 18:56:17 mizar kernel: sd 9:0:0:0: [sdg] Write Protect is off
Oct 10 18:56:17 mizar kernel: sd 9:0:0:0: [sdg] Mode Sense: 00 3a 00 00
Oct 10 18:56:17 mizar kernel: sd 9:0:0:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Oct 10 19:34:58 mizar kernel: ata10.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Oct 10 19:34:58 mizar kernel: ata10.00: cmd 35/00:08:3f:52:54/00:00:57:00:00/e0 tag 0 dma 4096 out
Oct 10 19:34:58 mizar kernel:          res 40/00:ff:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
Oct 10 19:34:58 mizar kernel: ata10.00: status: { DRDY }
Oct 10 19:34:58 mizar kernel: ata10: hard resetting link
Oct 10 19:34:58 mizar kernel: ata10: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Oct 10 19:34:58 mizar kernel: ata10.00: max_sectors limited to 256 for NCQ
Oct 10 19:34:58 mizar kernel: ata10.00: max_sectors limited to 256 for NCQ
Oct 10 19:34:58 mizar kernel: ata10.00: configured for UDMA/33
Oct 10 19:34:58 mizar kernel: ata10: EH complete
Oct 10 19:34:58 mizar kernel: sd 9:0:0:0: [sdg] 1465149168 512-byte hardware sectors (750156 MB)
Oct 10 19:34:58 mizar kernel: sd 9:0:0:0: [sdg] Write Protect is off
Oct 10 19:34:58 mizar kernel: sd 9:0:0:0: [sdg] Mode Sense: 00 3a 00 00
Oct 10 19:34:58 mizar kernel: sd 9:0:0:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Oct 10 19:37:05 mizar kernel: ata10.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Oct 10 19:37:05 mizar kernel: ata10.00: cmd 35/00:08:3f:52:54/00:00:57:00:00/e0 tag 0 dma 4096 out
Oct 10 19:37:05 mizar kernel:          res 40/00:ff:00:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
Oct 10 19:37:05 mizar kernel: ata10.00: status: { DRDY }
Oct 10 19:37:05 mizar kernel: ata10: hard resetting link
Oct 10 19:37:06 mizar kernel: ata10: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Oct 10 19:37:06 mizar kernel: ata10.00: max_sectors limited to 256 for NCQ
Oct 10 19:37:06 mizar kernel: ata10.00: max_sectors limited to 256 for NCQ
Oct 10 19:37:06 mizar kernel: ata10.00: configured for UDMA/33
Oct 10 19:37:06 mizar kernel: ata10: EH complete
Oct 10 19:37:06 mizar kernel: sd 9:0:0:0: [sdg] 1465149168 512-byte hardware sectors (750156 MB)
Oct 10 19:37:06 mizar kernel: sd 9:0:0:0: [sdg] Write Protect is off
Oct 10 19:37:06 mizar kernel: sd 9:0:0:0: [sdg] Mode Sense: 00 3a 00 00
Oct 10 19:37:06 mizar kernel: sd 9:0:0:0: [sdg] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: exception Emask 0x0 SAct 0xf SErr 0x0 action 0x6 frozen
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: cmd 60/40:00:7f:a1:e2/00:00:28:00:00/40 tag 0 ncq 32768 in
Sep 26 15:47:14 mvsrv02 kernel:          res 40/00:00:09:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: status: { DRDY }
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: cmd 60/40:08:3f:a1:e2/00:00:28:00:00/40 tag 1 ncq 32768 in
Sep 26 15:47:14 mvsrv02 kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: status: { DRDY }
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: cmd 60/40:10:3f:a2:e2/00:00:28:00:00/40 tag 2 ncq 32768 in
Sep 26 15:47:14 mvsrv02 kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: status: { DRDY }
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: cmd 60/c0:18:7f:a2:e2/00:00:28:00:00/40 tag 3 ncq 98304 in
Sep 26 15:47:14 mvsrv02 kernel:          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: status: { DRDY }
Sep 26 15:47:14 mvsrv02 kernel: ata5: hard resetting link
Sep 26 15:47:14 mvsrv02 kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: max_sectors limited to 256 for NCQ
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: max_sectors limited to 256 for NCQ
Sep 26 15:47:14 mvsrv02 kernel: ata5.00: configured for UDMA/133
Sep 26 15:47:14 mvsrv02 kernel: ata5: EH complete
Sep 26 15:47:14 mvsrv02 kernel: sd 4:0:0:0: [sdb] 976773168 512-byte hardware sectors (500108 MB)
Sep 26 15:47:14 mvsrv02 kernel: sd 4:0:0:0: [sdb] Write Protect is off
Sep 26 15:47:14 mvsrv02 kernel: sd 4:0:0:0: [sdb] Mode Sense: 00 3a 00 00
Sep 26 15:47:14 mvsrv02 kernel: sd 4:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

1st comuter: 133MHz PCI-X slot
03:01.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09)
Subsystem: Marvell Technology Group Ltd. Unknown device 11ab
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr- Stepping- SERR- FastB2B+ DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium

Quoted 1 line

TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Latency: 32, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 48
Region 0: Memory at d8800000 (64-bit, non-prefetchable) [size=1M] Region 2: I/O ports at 3000 [size=256]
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+
Queue=0/0 Enable-

               Address: 0000000000000000  Data: 0000
       Capabilities: [60] PCI-X non-bridge device
               Command: DPERE- ERO- RBC=512 OST=4
               Status: Dev=03:01.0 64bit+ 133MHz+ SCD- USC- DC=simple 

DMMRBC=512 DMOST=4 DMCRS=8 RSCEM- 266MHz- 533MHz-
Kernel driver in use: sata_mv

2nd: 66MHz 64bit PCI
02:01.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09)
Subsystem: Marvell Technology Group Ltd. Unknown device 11ab
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium

Quoted 1 line

TAbort- <TAbort- <MAbort- >SERR- <PERR-

Latency: 32, Cache Line Size: 128 bytes
Interrupt: pin A routed to IRQ 24
Region 0: Memory at f2800000 (64-bit, non-prefetchable) [size=1M] Region 2: I/O ports at c000 [size=256]
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+
Queue=0/0 Enable-

               Address: 0000000000000000  Data: 0000
       Capabilities: [60] PCI-X non-bridge device
               Command: DPERE- ERO- RBC=512 OST=4
               Status: Dev=02:01.0 64bit+ 133MHz+ SCD- USC- DC=simple 

DMMRBC=512 DMOST=4 DMCRS=8 RSCEM- 266MHz- 533MHz-

3rd computer: 32bit 33MHz PCI
00:0a.0 SCSI storage controller: Marvell Technology Group Ltd. MV88SX6081 8-port SATA II PCI-X Controller (rev 09)
Subsystem: Marvell Technology Group Ltd. Unknown device 11ab
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr- Stepping- SERR+ FastB2B-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium

Quoted 1 line

TAbort- <TAbort- <MAbort- >SERR- <PERR-

Latency: 32, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 16
Region 0: Memory at cfe00000 (64-bit, non-prefetchable) [size=1M] Region 2: I/O ports at dc00 [size=256]
Capabilities: [40] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] Message Signalled Interrupts: 64bit+
Queue=0/0 Enable-

               Address: 0000000000000000  Data: 0000
       Capabilities: [60] PCI-X non-bridge device
               Command: DPERE- ERO- RBC=512 OST=4
               Status: Dev=ff:1f.0 64bit+ 133MHz+ SCD- USC- DC=simple 

DMMRBC=512 DMOST=4 DMCRS=8 RSCEM- 266MHz- 533MHz-

#3 lore raw
In reply to Mark Lord
Quoted 18 lines · Mark Lord
Quoted 4 lines
Quoted 4 lines

Two marvell controllers, 16 disks, software raid10, IO stucks on different disks, kernel 2.6.26.5. With default ubuntu's 8.04 2.6.24 kernel the problem can not be repeated

Quoted 14 lines
Quoted 6 lines
[  289.851609] ata11.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
[  289.851695] ata11.00: cmd 61/08:00:60:1e:bf/00:00:01:00:00/40 tag 0 ncq 4096 out
[  289.851697]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  289.851774] ata11.00: status: { DRDY }
[  289.851834] ata11: hard resetting link
[  290.649259] ata11: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Quoted 8 lines
[  290.749239] ata11.00: max_sectors limited to 256 for NCQ
[  290.809189] ata11.00: max_sectors limited to 256 for NCQ
[  290.809194] ata11.00: configured for UDMA/133
[  290.809200] ata11: EH complete
[  290.809242] sd 10:0:0:0: [sdk] 1953525168 512-byte hardware sectors (1000205 MB)
[  290.809258] sd 10:0:0:0: [sdk] Write Protect is off
[  290.809263] sd 10:0:0:0: [sdk] Mode Sense: 00 3a 00 00
[  290.809286] sd 10:0:0:0: [sdk] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Quoted 5 lines

...

I've just returned here from a month holiday in Italy, and I'll have a look at this and other sata_mv issues next week or so.

I ran git-bisect on it and it returned a3718c1f230240361ed92d3e53342df0ff7efa8c as first bad commit. Also verified by hand that patching it on working tree breaks it.

Signature · 1 line
Harri.
#4 lore raw
Quoted 19 lines · Harri Olin
Quoted 18 lines · Mark Lord
Quoted 18 lines
Quoted 18 lines

Two marvell controllers, 16 disks, software raid10, IO stucks on different disks, kernel 2.6.26.5. With default ubuntu's 8.04 2.6.24 kernel the problem can not be repeated

[  289.851609] ata11.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
[  289.851695] ata11.00: cmd 61/08:00:60:1e:bf/00:00:01:00:00/40 tag 0 ncq 4096 out
[  289.851697]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  289.851774] ata11.00: status: { DRDY }
[  289.851834] ata11: hard resetting link
[  290.649259] ata11: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[  290.749239] ata11.00: max_sectors limited to 256 for NCQ
[  290.809189] ata11.00: max_sectors limited to 256 for NCQ
[  290.809194] ata11.00: configured for UDMA/133
[  290.809200] ata11: EH complete
[  290.809242] sd 10:0:0:0: [sdk] 1953525168 512-byte hardware sectors (1000205 MB)
[  290.809258] sd 10:0:0:0: [sdk] Write Protect is off
[  290.809263] sd 10:0:0:0: [sdk] Mode Sense: 00 3a 00 00
[  290.809286] sd 10:0:0:0: [sdk] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Quoted 7 lines
Quoted 5 lines

...

I've just returned here from a month holiday in Italy, and I'll have a look at this and other sata_mv issues next week or so.

I ran git-bisect on it and it returned a3718c1f230240361ed92d3e53342df0ff7efa8c as first bad commit. Also verified by hand that patching it on working tree breaks it.

..

Wow.. thanks for all of the hard work there.

So it has something to do with (qc->tf.flags & ATA_TFLAG_POLLING). But what I don't see (yet), is exactly which path through the driver is reporting this error.

Any path through mv_unexpected_intr() should have the string "unexpected device interrupt" as part of the error report.

Similarly, any path through mv_err_intr() should have the string "edma_err_cause=xxxxxxxx" in the error report.

I see neither. So I wonder what path is being taken
through the driver that results in the error?
(That's the trouble with having been away from the code for five months..).

Since cmd 61 is an NCQ write, it pretty much has to be in EDMA mode, which means it will likely be calling mv_process_crpb_entries() and then dropping through to check ERR_IRQ and DEV_IRQ.

The old code checked ERR_IRQ above, and never saw it, so the new code is probably not seeing ERR_IRQ either.

So it must be seeing DEV_IRQ after process_crpb_entries(), something that the old code never checked for. And which is not supposed to happen here.

Weird.

Looking at later kernels (after the commit in question), I see that the code was further fixed to remove some possible races and stuff, but that's still just 2.6.26.5, which you guys see failures on.

So here's some instrumentation to help us figure it out. Please apply and report back once it triggers again. Thanks.

drivers/ata/sata_mv.c +0 -0
@@ -1999,12 +1999,15 @@

* Error will be seen/handled by mv_err_intr().
* So do nothing at all here.
*/

+				ata_port_printk(ap, KERN_WARNING, "mv_process_crpb_response1: err_cause=0x%x\n", err_cause);
				return;
			}
		}
		ata_status = edma_status >> CRPB_FLAG_STATUS_SHIFT;
		if (!ac_err_mask(ata_status))
			ata_qc_complete(qc);
+		else
+			ata_port_printk(ap, KERN_WARNING, "mv_process_crpb_response2: edma_status=0x%x\n", edma_status);
		/* else: leave it for mv_err_intr() */
	} else {
		ata_port_printk(ap, KERN_ERR, "%s: no qc for tag=%d\n",
@@ -2070,20 +2073,25 @@
	 */
	if (edma_was_enabled && (port_cause & DONE_IRQ)) {
		mv_process_crpb_entries(ap, pp);
-		if (pp->pp_flags & MV_PP_FLAG_DELAYED_EH)
+		if (pp->pp_flags & MV_PP_FLAG_DELAYED_EH) {
+			ata_port_printk(ap, KERN_WARNING, "mv_port_intr1: port_cause=0x%x(ERR_IRQ), ppflags=0x%x\n", port_cause, pp->pp_flags);
			mv_handle_fbs_ncq_dev_err(ap);
+		}
	}
	/*
	 * Handle chip-reported errors, or continue on to handle PIO.
	 */
	if (unlikely(port_cause & ERR_IRQ)) {
+		ata_port_printk(ap, KERN_WARNING, "mv_port_intr2: port_cause=0x%x(ERR_IRQ), edma=%d, ppflags=0x%x\n", port_cause, edma_was_enabled, pp->pp_flags);
		mv_err_intr(ap);
	} else if (!edma_was_enabled) {
		struct ata_queued_cmd *qc = mv_get_active_qc(ap);
		if (qc)
			ata_sff_host_intr(ap, qc);
-		else
+		else {
+			ata_port_printk(ap, KERN_WARNING, "mv_port_intr3: port_cause=0x%x(ERR_IRQ), ppflags=0x%x\n", port_cause, pp->pp_flags);
			mv_unexpected_intr(ap, edma_was_enabled);
+		}
	}
}
#5 lore raw
Quoted 26 lines · Mark Lord
Quoted 25 lines · Harri Olin
Quoted 23 lines · Mark Lord
Quoted 4 lines
Quoted 4 lines

Two marvell controllers, 16 disks, software raid10, IO stucks on different disks, kernel 2.6.26.5. With default ubuntu's 8.04 2.6.24 kernel the problem can not be repeated

Quoted 14 lines
Quoted 6 lines
[  289.851609] ata11.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
[  289.851695] ata11.00: cmd 61/08:00:60:1e:bf/00:00:01:00:00/40 tag 0 ncq 4096 out
[  289.851697]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  289.851774] ata11.00: status: { DRDY }
[  289.851834] ata11: hard resetting link
[  290.649259] ata11: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Quoted 8 lines
[  290.749239] ata11.00: max_sectors limited to 256 for NCQ
[  290.809189] ata11.00: max_sectors limited to 256 for NCQ
[  290.809194] ata11.00: configured for UDMA/133
[  290.809200] ata11: EH complete
[  290.809242] sd 10:0:0:0: [sdk] 1953525168 512-byte hardware sectors (1000205 MB)
[  290.809258] sd 10:0:0:0: [sdk] Write Protect is off
[  290.809263] sd 10:0:0:0: [sdk] Mode Sense: 00 3a 00 00
[  290.809286] sd 10:0:0:0: [sdk] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

...

I've just returned here from a month holiday in Italy, and I'll have a look at this and other sata_mv issues next week or so.

Quoted 8 lines
Quoted 1 line

I ran git-bisect on it and it returned a3718c1f230240361ed92d3e53342df0ff7efa8c as first bad commit. Also verified by hand that patching it on working tree breaks it.

Looking at later kernels (after the commit in question), I see that the code was further fixed to remove some possible races and stuff, but that's still just 2.6.26.5, which you guys see failures on.

So here's some instrumentation to help us figure it out. Please apply and report back once it triggers again. Thanks.

I have to take back that bisect, as just couple of minutes ago it happened again, with last 'good' kernel from bisect. Just the frequency of stalls has dropped quite much. I also noticed that on current kernels are much better too. pre-..0ff7efa8c: only once after 6 hours of testing
post-..0ff7efa8c: one hd stalled while filesystem was mounting. Before boot was complete, 3 stalls. Also at shutdown kernel hung at Synchronizing SCSI cache for a while.
2.6.27: once in 5 minutes or so on heavy load

When some hd/port stalls, other ports sill work fine.

I applied your patch on 2.6.27.1, no results:

ata14.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen ata14.00: cmd 61/08:00:3f:52:54/00:00:57:00:00/40 tag 0 ncq 4096 out

        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)

ata14.00: status: { DRDY }
ata14: hard resetting link
ata14: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata14.00: max_sectors limited to 256 for NCQ
ata14.00: max_sectors limited to 256 for NCQ
ata14.00: configured for UDMA/133
ata14: EH complete
sd 13:0:0:0: [sdh] 1465149168 512-byte hardware sectors (750156 MB)
sd 13:0:0:0: [sdh] Write Protect is off
sd 13:0:0:0: [sdh] Mode Sense: 00 3a 00 00
sd 13:0:0:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

Do I have to enable something somewhere else too?

I also compiled and patched linux-2.6-stable tree from git but it just paniced after stall instead of recovering. I'm currently trying to reproduce that on second computer where I can capture the panic.

Signature · 1 line
Harri.
#6 lore raw
Quoted 60 lines · Harri Olin
Quoted 26 lines · Mark Lord
Quoted 25 lines · Harri Olin
Quoted 23 lines · Mark Lord
Quoted 4 lines
Quoted 4 lines

Two marvell controllers, 16 disks, software raid10, IO stucks on different disks, kernel 2.6.26.5. With default ubuntu's 8.04 2.6.24 kernel the problem can not be repeated

Quoted 14 lines
Quoted 6 lines
[  289.851609] ata11.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
[  289.851695] ata11.00: cmd 61/08:00:60:1e:bf/00:00:01:00:00/40 tag 0 ncq 4096 out
[  289.851697]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  289.851774] ata11.00: status: { DRDY }
[  289.851834] ata11: hard resetting link
[  290.649259] ata11: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Quoted 8 lines
[  290.749239] ata11.00: max_sectors limited to 256 for NCQ
[  290.809189] ata11.00: max_sectors limited to 256 for NCQ
[  290.809194] ata11.00: configured for UDMA/133
[  290.809200] ata11: EH complete
[  290.809242] sd 10:0:0:0: [sdk] 1953525168 512-byte hardware sectors (1000205 MB)
[  290.809258] sd 10:0:0:0: [sdk] Write Protect is off
[  290.809263] sd 10:0:0:0: [sdk] Mode Sense: 00 3a 00 00
[  290.809286] sd 10:0:0:0: [sdk] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

...

I've just returned here from a month holiday in Italy, and I'll have a look at this and other sata_mv issues next week or so.

Quoted 8 lines
Quoted 1 line

I ran git-bisect on it and it returned a3718c1f230240361ed92d3e53342df0ff7efa8c as first bad commit. Also verified by hand that patching it on working tree breaks it.

Looking at later kernels (after the commit in question), I see that the code was further fixed to remove some possible races and stuff, but that's still just 2.6.26.5, which you guys see failures on.

So here's some instrumentation to help us figure it out. Please apply and report back once it triggers again. Thanks.

I have to take back that bisect, as just couple of minutes ago it happened again, with last 'good' kernel from bisect. Just the frequency of stalls has dropped quite much. I also noticed that on current kernels are much better too. pre-..0ff7efa8c: only once after 6 hours of testing
post-..0ff7efa8c: one hd stalled while filesystem was mounting. Before boot was complete, 3 stalls. Also at shutdown kernel hung at Synchronizing SCSI cache for a while.
2.6.27: once in 5 minutes or so on heavy load

When some hd/port stalls, other ports sill work fine.

I applied your patch on 2.6.27.1, no results:

ata14.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen ata14.00: cmd 61/08:00:3f:52:54/00:00:57:00:00/40 tag 0 ncq 4096 out

       res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)

ata14.00: status: { DRDY }
ata14: hard resetting link
ata14: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata14.00: max_sectors limited to 256 for NCQ
ata14.00: max_sectors limited to 256 for NCQ
ata14.00: configured for UDMA/133
ata14: EH complete
sd 13:0:0:0: [sdh] 1465149168 512-byte hardware sectors (750156 MB)
sd 13:0:0:0: [sdh] Write Protect is off
sd 13:0:0:0: [sdh] Mode Sense: 00 3a 00 00
sd 13:0:0:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

Quoted 3 lines

Do I have to enable something somewhere else too?

I also compiled and patched linux-2.6-stable tree from git but it just paniced after stall instead of recovering. I'm currently trying to reproduce that on second computer where I can capture the panic.

What type of disks are you using?

Justin.

#7 lore raw
Quoted 64 lines · Justin Piszcz
Quoted 60 lines · Harri Olin
Quoted 26 lines · Mark Lord
Quoted 25 lines · Harri Olin
Quoted 23 lines · Mark Lord
Quoted 4 lines
Quoted 4 lines

Two marvell controllers, 16 disks, software raid10, IO stucks on different disks, kernel 2.6.26.5. With default ubuntu's 8.04 2.6.24 kernel the problem can not be repeated

Quoted 14 lines
Quoted 6 lines
[  289.851609] ata11.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
[  289.851695] ata11.00: cmd 61/08:00:60:1e:bf/00:00:01:00:00/40 tag 0 ncq 4096 out
[  289.851697]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  289.851774] ata11.00: status: { DRDY }
[  289.851834] ata11: hard resetting link
[  290.649259] ata11: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Quoted 8 lines
[  290.749239] ata11.00: max_sectors limited to 256 for NCQ
[  290.809189] ata11.00: max_sectors limited to 256 for NCQ
[  290.809194] ata11.00: configured for UDMA/133
[  290.809200] ata11: EH complete
[  290.809242] sd 10:0:0:0: [sdk] 1953525168 512-byte hardware sectors (1000205 MB)
[  290.809258] sd 10:0:0:0: [sdk] Write Protect is off
[  290.809263] sd 10:0:0:0: [sdk] Mode Sense: 00 3a 00 00
[  290.809286] sd 10:0:0:0: [sdk] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

...

I've just returned here from a month holiday in Italy, and I'll have a look at this and other sata_mv issues next week or so.

Quoted 8 lines
Quoted 1 line

I ran git-bisect on it and it returned a3718c1f230240361ed92d3e53342df0ff7efa8c as first bad commit. Also verified by hand that patching it on working tree breaks it.

Looking at later kernels (after the commit in question), I see that the code was further fixed to remove some possible races and stuff, but that's still just 2.6.26.5, which you guys see failures on.

So here's some instrumentation to help us figure it out. Please apply and report back once it triggers again. Thanks.

I have to take back that bisect, as just couple of minutes ago it happened again, with last 'good' kernel from bisect. Just the frequency of stalls has dropped quite much. I also noticed that on current kernels are much better too. pre-..0ff7efa8c: only once after 6 hours of testing
post-..0ff7efa8c: one hd stalled while filesystem was mounting. Before boot was complete, 3 stalls. Also at shutdown kernel hung at Synchronizing SCSI cache for a while.
2.6.27: once in 5 minutes or so on heavy load

When some hd/port stalls, other ports sill work fine.

I applied your patch on 2.6.27.1, no results:

ata14.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen ata14.00: cmd 61/08:00:3f:52:54/00:00:57:00:00/40 tag 0 ncq 4096 out

       res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)

ata14.00: status: { DRDY }
ata14: hard resetting link
ata14: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
ata14.00: max_sectors limited to 256 for NCQ
ata14.00: max_sectors limited to 256 for NCQ
ata14.00: configured for UDMA/133
ata14: EH complete
sd 13:0:0:0: [sdh] 1465149168 512-byte hardware sectors (750156 MB)
sd 13:0:0:0: [sdh] Write Protect is off
sd 13:0:0:0: [sdh] Mode Sense: 00 3a 00 00
sd 13:0:0:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

Quoted 7 lines
Quoted 3 lines

Do I have to enable something somewhere else too?

I also compiled and patched linux-2.6-stable tree from git but it just paniced after stall instead of recovering. I'm currently trying to reproduce that on second computer where I can capture the panic.

What type of disks are you using?

Justin.

I have seen this happening on on 3 different computers using WD5000ABYS, WD5000YS and WD7500AYYS hard disks. All have same Supermicro controller. Stalls happen only on controller ports 0-3, never on ports 4-7. Moving cables around doesn't help.

Signature · 1 line
Harri.
#8 lore raw
Quoted 7 lines · Harri Olin
Quoted 6 lines · Justin Piszcz
Quoted 2 lines · Harri Olin
Quoted 1 line · Mark Lord

Harri Olin wrote:

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines · Mark Lord
Quoted 2 lines
Quoted 2 lines

Two marvell controllers, 16 disks, software raid10, IO stucks on different disks, kernel 2.6.26.5. With default ubuntu's 8.04 2.6.24 kernel the problem can not be repeated

What type of disks are you using?

Justin.

I have seen this happening on on 3 different computers using WD5000ABYS, WD5000YS and WD7500AYYS hard disks. All have same Supermicro controller. Stalls happen only on controller ports 0-3, never on ports 4-7. Moving cables around doesn't help.

I have been compiling my own list of reports on this problem:

Bug 462425 - Kernel 2.6.26.3-29.fc9.x86_64 drive goes offline
https://bugzilla.redhat.com/show_bug.cgi?id=462425

hardy / ibex - raid5 - ata#: hard resetting link [edit]
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/263160/

exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen
http://groups.google.com/group/linux.kernel/browse_thread/thread/8f3c7ea69a900e51?q="exception+Emask+0x0+SAct"

Harri,

That is interesting that it never happens on port 4-7. I have two machines, each with Sil 3132 cards, sata_mv and Intel ICH8 and the same problem happens across all (12) Velociraptors, I have RMA'd 8 disks so far. It also happens on my Raptor 150s in a different box, but MUCH less. I am currently testing a patch from Alan Cox and will report if/when the problem recurs:

Subject: [PATCH] libata: Drain data on errors

From: Alan Cox <[email protected]>

If the device is signalling that there is data to drain after an error we should read the bytes out and throw them away. Without this some devices and controllers get wedged and don't recover.

Based on earlier work by Mark Lord

---

 drivers/ata/libata-sff.c  |   44 +++++++++++++++++++++++++++++++++++++++++++-
 drivers/ata/pata_pcmcia.c |   34 +++++++++++++++++++++++++++++++++-
 include/linux/libata.h    |    3 +++
 3 files changed, 79 insertions(+), 2 deletions(-)

Justin.

#9 lore raw
In reply to Harri Olin

Harri Olin wrote:
..

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines

Two marvell controllers, 16 disks, software raid10, IO stucks on different disks, kernel 2.6.26.5. With default ubuntu's 8.04 2.6.24 kernel the problem can not be repeated

[  289.851609] ata11.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen
[  289.851695] ata11.00: cmd 61/08:00:60:1e:bf/00:00:01:00:00/40 tag 0 ncq 4096 out
[  289.851697]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[  289.851774] ata11.00: status: { DRDY }
[  289.851834] ata11: hard resetting link
[  290.649259] ata11: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[  290.749239] ata11.00: max_sectors limited to 256 for NCQ
[  290.809189] ata11.00: max_sectors limited to 256 for NCQ
[  290.809194] ata11.00: configured for UDMA/133
[  290.809200] ata11: EH complete
[  290.809242] sd 10:0:0:0: [sdk] 1953525168 512-byte hardware sectors (1000205 MB)
[  290.809258] sd 10:0:0:0: [sdk] Write Protect is off
[  290.809263] sd 10:0:0:0: [sdk] Mode Sense: 00 3a 00 00
[  290.809286] sd 10:0:0:0: [sdk] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

..

Quoted 5 lines

I have to take back that bisect, as just couple of minutes ago it happened again, with last 'good' kernel from bisect. Just the frequency of stalls has dropped quite much. I also noticed that on current kernels are much better too. pre-..0ff7efa8c: only once after 6 hours of testing
post-..0ff7efa8c: one hd stalled while filesystem was mounting. Before boot was complete, 3 stalls. Also at shutdown kernel hung at Synchronizing SCSI cache for a while.
2.6.27: once in 5 minutes or so on heavy load

Quoted 7 lines

When some hd/port stalls, other ports sill work fine.

I applied your patch on 2.6.27.1, no results:

ata14.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen ata14.00: cmd 61/08:00:3f:52:54/00:00:57:00:00/40 tag 0 ncq 4096 out

        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)

..

Yeah, I see what I was missing earlier: "(timeout)".
So it's "none of" the driver paths.

This could very well be due to one/several of the as-yet un-addressed chipset errata for the 6081. Someday we'll have software workarounds for those, but I'm (still) waiting on Marvell for stuff.

I will look and see if this makes sense based on the errata info that I have already though (under NDA).

Harri / Artem: what type/speed of slots are your 6081 controllers in?
PCI, or PCI-X? Bus speed?

Thanks

#10 lore raw
Quoted 13 lines · Mark Lord

Yeah, I see what I was missing earlier: "(timeout)".
So it's "none of" the driver paths.

This could very well be due to one/several of the as-yet un-addressed chipset errata for the 6081. Someday we'll have software workarounds for those, but I'm (still) waiting on Marvell for stuff.

I will look and see if this makes sense based on the errata info that I have already though (under NDA).

Harri / Artem: what type/speed of slots are your 6081 controllers in?
PCI, or PCI-X? Bus speed?

..

Quoted 2 lines · Mmm.. Harri at least previously

Contoller is Supermicro AOC-SAT2-MV8, connected to 133MHz PCI-X slot on one computer, 66MHz 64bit PCI slot on the second machine and to normal 32bit PCI slot on third computer.

..

Quoted 1 line · And Artem

The controller is AOC-SAT2-MV8 too.

..

So I guess I need to know if Artem is using PCI-X, and the bus width + speed.

Cheers

#11 lore raw
Quoted 16 lines · Mark Lord
Quoted 13 lines · Mark Lord

Yeah, I see what I was missing earlier: "(timeout)".
So it's "none of" the driver paths.

This could very well be due to one/several of the as-yet un-addressed chipset errata for the 6081. Someday we'll have software workarounds for those, but I'm (still) waiting on Marvell for stuff.

I will look and see if this makes sense based on the errata info that I have already though (under NDA).

Harri / Artem: what type/speed of slots are your 6081 controllers in?
PCI, or PCI-X? Bus speed?

..

Quoted 8 lines
Quoted 2 lines · Mmm.. Harri at least previously

Contoller is Supermicro AOC-SAT2-MV8, connected to 133MHz PCI-X slot on one computer, 66MHz 64bit PCI slot on the second machine and to normal 32bit PCI slot on third computer.

..

Quoted 1 line · And Artem

The controller is AOC-SAT2-MV8 too.

..

As I mentioned earlier, it happens on all of my drives, 4 of 12 disks are connected to a Marvell controller, in my case:

01:00.0 SCSI storage controller: Marvell Technology Group Ltd. 88SX7042 PCI-e 4-port SATA-II (rev 02)

It is in the x16 slot.

Justin.

#14 lore raw
In reply to Mark Lord
Quoted 8 lines · Mark Lord

Harri Olin wrote:
..

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Two marvell controllers, 16 disks, software raid10, IO stucks on different disks, kernel 2.6.26.5. With default ubuntu's 8.04 2.6.24 kernel the problem can not be repeated

This could very well be due to one/several of the as-yet un-addressed chipset errata for the 6081. Someday we'll have software workarounds for those, but I'm (still) waiting on Marvell for stuff.

I was going through logs for the last night and noticed something:
Currently I have only 7 disks connected to 6081 controller, port 5 is free, and now stalls have been happening only on ports 0, 2 and 3. So it seems something on high port is affecting corresponding lower port too.

Quoted 3 lines

Harri / Artem: what type/speed of slots are your 6081 controllers in?
PCI, or PCI-X? Bus speed?

As mentioned earlier, 33MHz, 66MHz and 133MHz. Motherboards are: some Asus board with VIA chipset, Supermicro P4SCI and Supermicro X7SBE. I think P4SCI has 66MHz 64bit PCI, not PCI-X.

Signature · 1 line
Harri.
#15 lore raw
In reply to Mark Lord
Quoted 10 lines · Mark Lord
Quoted 3 lines

ata14.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x6 frozen ata14.00: cmd 61/08:00:3f:52:54/00:00:57:00:00/40 tag 0 ncq 4096 out

        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)

Yeah, I see what I was missing earlier: "(timeout)".
So it's "none of" the driver paths.

This could very well be due to one/several of the as-yet un-addressed chipset errata for the 6081. Someday we'll have software workarounds for those, but I'm (still) waiting on Marvell for stuff.

After a bit of testing, it seems that writing is required to trigger the bug, dstat output follows:

--dsk/sde-----dsk/sdf-----dsk/sdg-----dsk/sdh-----dsk/sdi-----dsk/sdj-----dsk/sdk-- read writ: read writ: read writ: read writ: read writ: read writ: read writ

37M    0 :  35M    0 :  35M    0 :  37M    0 :  34M    0 :  35M    0 :  32M    0
35M    0 :  34M    0 :  34M    0 :  35M    0 :  37M    0 :  37M    0 :  36M    0
34M    0 :  35M    0 :  35M    0 :  40M    0 :  36M    0 :  33M    0 :  35M    0
30M 8192B:  28M 8192B:  30M 8192B:  30M    0 :  28M 8192B:  30M 8192B:  28M 8192B
35M    0 :  37M    0 :  33M    0 :   0     0 :  36M    0 :  34M    0 :  35M    0
36M    0 :  35M    0 :  35M    0 :   0     0 :  35M    0 :  34M    0 :  34M    0
34M    0 :  37M    0 :  38M    0 :   0     0 :  36M    0 :  36M    0 :  35M    0

I was running fio, reading from all drives connected to 6081. After nothing happened for a while, I decided to mount the xfs filesystem read-write and it hung immediately before mount was even complete.

I also managed to catch the panic I mentioned, running kernel 2.6.28-rc5:

[  503.918122] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
[  503.918399] IP: [<ffffffff804d3938>] scsi_times_out+0x8/0x70
[  503.918561] PGD 229068067 PUD 22a1f0067 PMD 0
[  503.918814] Oops: 0000 [#1] SMP
[  503.919009] last sysfs file: /sys/block/sdk/stat
[  503.919123] CPU 2
[  503.919273] Modules linked in: kvm_intel kvm coretemp w83627hf w83793 hwmon_vid hwmon nf_conntrack_ftp 3c59x i2c_i801 i2c_core e100 iTCO_wdt
[  503.920074] Pid: 0, comm: swapper Not tainted 2.6.28-rc5 #4
[  503.920190] RIP: 0010:[<ffffffff804d3938>]  [<ffffffff804d3938>] scsi_times_out+0x8/0x70
[  503.920417] RSP: 0018:ffff88022f0f3e60  EFLAGS: 00010046
[  503.920540] RAX: ffff88022d4f5470 RBX: 0000000000000000 RCX: ffff88022d4f5ac8
[  503.920659] RDX: ffff88022d4f57e8 RSI: 0000000000000eae RDI: ffff8801f8188848
[  503.920777] RBP: ffff88022d4f5988 R08: 0000000000000000 R09: 0000000000000000
[  503.920897] R10: ffffffff804d6142 R11: ffffffff805dc480 R12: ffff88022f0e4000
[  503.921015] R13: ffff88022d4f57e8 R14: 0000000000000000 R15: ffff88022d4f5470
[  503.921134] FS:  0000000000000000(0000) GS:ffff88022f08bac0(0000) knlGS:0000000000000000
[  503.921317] CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
[  503.921434] CR2: 0000000000000000 CR3: 000000022a0cf000 CR4: 00000000000026e0
[  503.921553] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  503.921674] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  503.921793] Process swapper (pid: 0, threadinfo ffff88022f0ee000, task ffff88022f0e2c30)
[  503.921985] Stack:
[  503.922094]  ffff8801f8188848 ffffffff80416eee ffff8801f8188848 ffffffff80416fea
[  503.922116]  0000000000000282 ffff88022d4f5470 0000000000000100 ffff88022f0e4000
[  503.922116]  ffff88022f0f3ee0 ffffffff80416f30 ffff88022f0e5018 ffffffff8024393b
[  503.922116] Call Trace:
[  503.922116]  <IRQ> <0> [<ffffffff80416eee>] ? blk_rq_timed_out+0xe/0x50
[  503.922116]  [<ffffffff80416fea>] ? blk_rq_timed_out_timer+0xba/0x120
[  503.922116]  [<ffffffff80416f30>] ? blk_rq_timed_out_timer+0x0/0x120
[  503.922116]  [<ffffffff8024393b>] ? run_timer_softirq+0x1bb/0x230
[  503.922116]  [<ffffffff8023f00b>] ? __do_softirq+0x8b/0x150
[  503.922116]  [<ffffffff8020e7db>] ? profile_pc+0x3b/0x80
[  503.922116]  [<ffffffff8020c8fc>] ? call_softirq+0x1c/0x40
[  503.922116]  [<ffffffff8020db55>] ? do_softirq+0x35/0x70
[  503.922116]  [<ffffffff802205b5>] ? smp_apic_timer_interrupt+0x85/0xd0
[  503.922116]  [<ffffffff8020c34b>] ? apic_timer_interrupt+0x6b/0x70
[  503.922116]  <EOI> <0> [<ffffffff805dc480>] ? udp_poll+0x0/0x150
[  503.922116]  [<ffffffff80212d8c>] ? mwait_idle+0x3c/0x40
[  503.922116]  [<ffffffff80209d5a>] ? cpu_idle+0x3a/0x70
[  503.922116] Code: 18 4c 8b 74 24 20 48 83 c4 28 c3 be 06 00 00 00 48 89 df e8 9b c8 ff ff 85 c0 75 c3 eb 87 0f 1f 44 00 00 53 48 8b 9f e0 00 00 00 <48> 8b 03 48
[  503.922116] RIP  [<ffffffff804d3938>] scsi_times_out+0x8/0x70
[  503.922116]  RSP <ffff88022f0f3e60>
[  503.922116] CR2: 0000000000000000
[  503.922116] Kernel panic - not syncing: Fatal exception in interrupt
[  503.922116] ------------[ cut here ]------------
[  503.922116] WARNING: at kernel/smp.c:333 smp_call_function_mask+0x236/0x240()
[  503.922116] Modules linked in: kvm_intel kvm coretemp w83627hf w83793 hwmon_vid hwmon nf_conntrack_ftp 3c59x i2c_i801 i2c_core e100 iTCO_wdt
[  503.922116] Pid: 0, comm: swapper Tainted: G      D    2.6.28-rc5 #4
[  503.922116] Call Trace:
[  503.922116]  <IRQ>  [<ffffffff80239ea4>] warn_on_slowpath+0x64/0xa0
[  503.922116]  [<ffffffff80252396>] up+0x16/0x50
[  503.922116]  [<ffffffff8023a657>] release_console_sem+0x197/0x1e0
[  503.922116]  [<ffffffff8025c126>] smp_call_function_mask+0x236/0x240
[  503.922116]  [<ffffffff8023b0fe>] printk+0x4e/0x60
[  503.922116]  [<ffffffff80252396>] up+0x16/0x50
[  503.922116]  [<ffffffff8021f290>] native_smp_send_stop+0x20/0x30
[  503.922116]  [<ffffffff80239f7e>] panic+0x8e/0x150
[  503.922116]  [<ffffffff8020e582>] show_registers+0x192/0x250
[  503.922116]  [<ffffffff8047d745>] do_unblank_screen+0x15/0x140
[  503.922116]  [<ffffffff80636370>] oops_end+0xa0/0xb0
[  503.922116]  [<ffffffff80637f43>] do_page_fault+0x6a3/0x830
[  503.922116]  [<ffffffff80635799>] error_exit+0x0/0x51
[  503.922116]  [<ffffffff805dc480>] udp_poll+0x0/0x150
[  503.922116]  [<ffffffff804d6142>] scsi_request_fn+0xe2/0x400
[  503.922116]  [<ffffffff804d3938>] scsi_times_out+0x8/0x70
[  503.922116]  [<ffffffff80416eee>] blk_rq_timed_out+0xe/0x50
[  503.922116]  [<ffffffff80416fea>] blk_rq_timed_out_timer+0xba/0x120
[  503.922116]  [<ffffffff80416f30>] blk_rq_timed_out_timer+0x0/0x120
[  503.922116]  [<ffffffff8024393b>] run_timer_softirq+0x1bb/0x230
[  503.922116]  [<ffffffff8023f00b>] __do_softirq+0x8b/0x150
[  503.922116]  [<ffffffff8020e7db>] profile_pc+0x3b/0x80
[  503.922116]  [<ffffffff8020c8fc>] call_softirq+0x1c/0x40
[  503.922116]  [<ffffffff8020db55>] do_softirq+0x35/0x70
[  503.922116]  [<ffffffff802205b5>] smp_apic_timer_interrupt+0x85/0xd0
[  503.922116]  [<ffffffff8020c34b>] apic_timer_interrupt+0x6b/0x70
[  503.922116]  <EOI>  [<ffffffff805dc480>] udp_poll+0x0/0x150
[  503.922116]  [<ffffffff80212d8c>] mwait_idle+0x3c/0x40
[  503.922116]  [<ffffffff80209d5a>] cpu_idle+0x3a/0x70
[  503.922116] ---[ end trace 3eef0898db52fd7a ]---
Signature · 1 line
Harri.
#12 lore raw
In reply to Justin Piszcz
Quoted 4 lines · Justin Piszcz

..
As I mentioned earlier, it happens on all of my drives, 4 of 12 disks are connected to a Marvell controller, in my case:

01:00.0 SCSI storage controller: Marvell Technology Group Ltd. 88SX7042 PCI-e 4-port SATA-II (rev 02)

..

Whatever problem you are seeing is likely different from what the others see. The 7042 is a much newer rev of the chipset, with a lot less in the way of errata and stuff to go wrong.

So, yes, you're apparently seeing some problems, but there's nothing to suggest that those are the *same* problems as the 6081 users are seeing.

Cheers

#13 lore raw
In reply to Mark Lord

Mark Lord пишет:

Quoted 7 lines
Quoted 1 line · And Artem

The controller is AOC-SAT2-MV8 too.

..

So I guess I need to know if Artem is using PCI-X, and the bus width + speed.

The only configuration I tried was 2 controllers on 133 mhz pci-x

Quoted 5 lines

Cheers

Signature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-ide" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html