Backplane mystery
I have an external enclosure with an SATA backplane with 20 drive slots. I am using 4 port multipliers to drive all 20 slots.
One set of 5 slots connected to a single port multiplier always fails with
[ 16.449673] ata2: FORCE: PHY spd limit set to 1.5Gbps [ 16.449740] ata2: SATA max UDMA/100 host m128@0xfdeff000 port 0xfdefa000 irq 16 [ 20.616038] ata2: SATA link down (SStatus 0 SControl 10) [ 80.596644] ata2: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0xe frozen [ 80.596956] ata2: irq_stat 0x00a00080, device exchanged [ 80.597039] ata2: hard resetting link [ 90.604024] ata2: softreset failed (timeout) [ 90.604108] ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 10) [ 90.604183] ata2: link online but device misclassified, retrying [ 90.604264] ata2: hard resetting link [ 100.604026] ata2: softreset failed (timeout) [ 100.604109] ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 10) [ 100.604186] ata2: link online but device misclassified, retrying [ 100.604259] ata2: hard resetting link [ 135.612026] ata2: softreset failed (timeout) [ 135.612109] ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 10) [ 135.612191] ata2: link online but device misclassified, retrying [ 135.612273] ata2: hard resetting link [ 140.616024] ata2: softreset failed (timeout) [ 140.616107] ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 10) [ 140.616183] ata2: link online but device misclassified, device detection might fail [ 140.616296] ata2: EH complete
I've swapped cables, port multipliers, and so on. It's not the backplace, as I can swap cables from one of the port multipliers that works and it will detect drives. It's not the port multiplier, as I've swapped those. The only thing I can think of is that it's failing initialization. If I swap cables from a port multiplier that's initialized while the system is running, it works.
Before I start taking random actions, can someone interpret the above for me? What's it telling me?
For comparison, here's the initialization sequence from the adjacent port multiplier:
[ 16.449504] ata1: FORCE: PHY spd limit set to 1.5Gbps [ 16.449571] ata1: SATA max UDMA/100 host m128@0xfdeff000 port 0xfdef8000 irq 16 [ 18.536033] ata1: SATA link down (SStatus 0 SControl 10) [ 19.374191] ata10: SATA max UDMA/133 cmd 0xbc00 ctl 0xb800 bmdma 0xb408 irq 21 [ 80.596647] ata1: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0xe frozen [ 80.596649] ata1: irq_stat 0x00a00080, device exchanged [ 80.596653] ata1: hard resetting link [ 82.796045] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 10) [ 82.796408] ata1.15: Port Multiplier 1.1, 0x1095:0x3726 r23, 6 ports, feat 0x1/0x9 [ 82.796796] ata1.00: hard resetting link [ 83.116406] ata1.00: SATA link down (SStatus 0 SControl 10) [ 83.116482] ata1.01: hard resetting link [ 83.436402] ata1.01: SATA link down (SStatus 0 SControl 310) [ 83.436479] ata1.02: hard resetting link [ 83.756394] ata1.02: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 83.756467] ata1.03: hard resetting link [ 84.076432] ata1.03: SATA link down (SStatus 0 SControl 310) [ 84.076510] ata1.04: hard resetting link [ 84.396387] ata1.04: SATA link down (SStatus 0 SControl 310) [ 84.396465] ata1.05: hard resetting link [ 84.716330] ata1.05: SATA link up 1.5 Gbps (SStatus 113 SControl 310) [ 84.828010] ata1.02: failed to IDENTIFY (I/O error, err_mask=0x11) [ 84.828082] ata1.15: hard resetting link [ 84.828145] ata1: controller in dubious state, performing PORT_RST [ 87.060032] ata1.15: SATA link up 1.5 Gbps (SStatus 113 SControl 10) [ 87.060351] ata1.00: hard resetting link [ 87.380330] ata1.00: SATA link down (SStatus 0 SControl 10) [ 87.380404] ata1.01: hard resetting link [ 87.700329] ata1.01: SATA link down (SStatus 0 SControl 310) [ 88.756028] ata1.02: hard resetting link [ 89.076366] ata1.02: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 89.716017] ata1.05: hard resetting link [ 90.036392] ata1.05: SATA link up 1.5 Gbps (SStatus 113 SControl 310) [ 90.148021] ata1.02: failed to IDENTIFY (I/O error, err_mask=0x11) [ 90.148104] ata1.15: hard resetting link [ 90.148177] ata1: controller in dubious state, performing PORT_RST [ 92.380043] ata1.15: SATA link up 1.5 Gbps (SStatus 113 SControl 10) [ 92.380415] ata1.00: hard resetting link [ 92.700329] ata1.00: SATA link down (SStatus 0 SControl 10) [ 92.700398] ata1.01: hard resetting link [ 93.020334] ata1.01: SATA link down (SStatus 0 SControl 310) [ 94.076081] ata1.02: hard resetting link [ 94.396398] ata1.02: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 94.396479] ata1.03: hard resetting link [ 94.716392] ata1.03: SATA link down (SStatus 0 SControl 310) [ 94.716465] ata1.04: hard resetting link [ 95.036397] ata1.04: SATA link down (SStatus 0 SControl 310) [ 95.036474] ata1.05: hard resetting link [ 95.356426] ata1.05: SATA link up 1.5 Gbps (SStatus 113 SControl 310) [ 95.369116] ata1.02: ATA-8: Hitachi HDT721010SLA360, ST6OA31B, max UDMA/133 [ 95.369193] ata1.02: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32) [ 95.370256] ata1.02: configured for UDMA/100 [ 95.370769] ata1: EH complete
If I swap cables from ata1 to ata2 while the system is running, the backplane will detect drives and work fine.
I'm already forcing a slower speed:
yan@selene:/etc/modprobe.d$ cat libata options libata force=1.15:1.5Gbps,2.15:1.5Gbps,3.15:1.5Gbps,4.15:1.5Gbps
I'm using a Sil 3132 controller:
02:00.0 Mass storage controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)
The system sees 3 port multipliers; not 4 as it should:
[ 82.488354] ata4.15: Port Multiplier 1.1, 0x1095:0x3726 r23, 6 ports, feat 0x1/0x9 [ 82.488507] ata3.15: Port Multiplier 1.1, 0x1095:0x3726 r23, 6 ports, feat 0x1/0x9 [ 82.796408] ata1.15: Port Multiplier 1.1, 0x1095:0x3726 r23, 6 ports, feat 0x1/0x9
I suspect it's cabling length as the 5 "dead" slots are the farthest down the backplane but swapping cables randomly just doesn't seem like a good approach.
What's my next step?
Thanks,
--Yan
Signature · 9 lines
o__
,>/'_ o__
(_)\(_) ,>/'_ o__
Yan Seiner (_)\(_) ,>/'_ o__
Personal Trainer (_)\(_) ,>/'_ o__
Professional Engineer (_)\(_) ,>/'_
Who says engineers have to be pencil necked geeks? (_)\(_)
I like paying taxes. With them I buy civilization.Quoted 100 lines · Yan Seiner
I have an external enclosure with an SATA backplane with 20 drive slots. I am using 4 port multipliers to drive all 20 slots.
One set of 5 slots connected to a single port multiplier always fails with
[ 16.449673] ata2: FORCE: PHY spd limit set to 1.5Gbps [ 16.449740] ata2: SATA max UDMA/100 host m128@0xfdeff000 port 0xfdefa000 irq 16 [ 20.616038] ata2: SATA link down (SStatus 0 SControl 10) [ 80.596644] ata2: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0xe frozen [ 80.596956] ata2: irq_stat 0x00a00080, device exchanged [ 80.597039] ata2: hard resetting link [ 90.604024] ata2: softreset failed (timeout) [ 90.604108] ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 10) [ 90.604183] ata2: link online but device misclassified, retrying [ 90.604264] ata2: hard resetting link [ 100.604026] ata2: softreset failed (timeout) [ 100.604109] ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 10) [ 100.604186] ata2: link online but device misclassified, retrying [ 100.604259] ata2: hard resetting link [ 135.612026] ata2: softreset failed (timeout) [ 135.612109] ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 10) [ 135.612191] ata2: link online but device misclassified, retrying [ 135.612273] ata2: hard resetting link [ 140.616024] ata2: softreset failed (timeout) [ 140.616107] ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 10) [ 140.616183] ata2: link online but device misclassified, device detection might fail [ 140.616296] ata2: EH completeI've swapped cables, port multipliers, and so on. It's not the backplace, as I can swap cables from one of the port multipliers that works and it will detect drives. It's not the port multiplier, as I've swapped those. The only thing I can think of is that it's failing initialization. If I swap cables from a port multiplier that's initialized while the system is running, it works.
Before I start taking random actions, can someone interpret the above for me? What's it telling me?
For comparison, here's the initialization sequence from the adjacent port multiplier:
[ 16.449504] ata1: FORCE: PHY spd limit set to 1.5Gbps [ 16.449571] ata1: SATA max UDMA/100 host m128@0xfdeff000 port 0xfdef8000 irq 16 [ 18.536033] ata1: SATA link down (SStatus 0 SControl 10) [ 19.374191] ata10: SATA max UDMA/133 cmd 0xbc00 ctl 0xb800 bmdma 0xb408 irq 21 [ 80.596647] ata1: exception Emask 0x10 SAct 0x0 SErr 0x0 action 0xe frozen [ 80.596649] ata1: irq_stat 0x00a00080, device exchanged [ 80.596653] ata1: hard resetting link [ 82.796045] ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 10) [ 82.796408] ata1.15: Port Multiplier 1.1, 0x1095:0x3726 r23, 6 ports, feat 0x1/0x9 [ 82.796796] ata1.00: hard resetting link [ 83.116406] ata1.00: SATA link down (SStatus 0 SControl 10) [ 83.116482] ata1.01: hard resetting link [ 83.436402] ata1.01: SATA link down (SStatus 0 SControl 310) [ 83.436479] ata1.02: hard resetting link [ 83.756394] ata1.02: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 83.756467] ata1.03: hard resetting link [ 84.076432] ata1.03: SATA link down (SStatus 0 SControl 310) [ 84.076510] ata1.04: hard resetting link [ 84.396387] ata1.04: SATA link down (SStatus 0 SControl 310) [ 84.396465] ata1.05: hard resetting link [ 84.716330] ata1.05: SATA link up 1.5 Gbps (SStatus 113 SControl 310) [ 84.828010] ata1.02: failed to IDENTIFY (I/O error, err_mask=0x11) [ 84.828082] ata1.15: hard resetting link [ 84.828145] ata1: controller in dubious state, performing PORT_RST [ 87.060032] ata1.15: SATA link up 1.5 Gbps (SStatus 113 SControl 10) [ 87.060351] ata1.00: hard resetting link [ 87.380330] ata1.00: SATA link down (SStatus 0 SControl 10) [ 87.380404] ata1.01: hard resetting link [ 87.700329] ata1.01: SATA link down (SStatus 0 SControl 310) [ 88.756028] ata1.02: hard resetting link [ 89.076366] ata1.02: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 89.716017] ata1.05: hard resetting link [ 90.036392] ata1.05: SATA link up 1.5 Gbps (SStatus 113 SControl 310) [ 90.148021] ata1.02: failed to IDENTIFY (I/O error, err_mask=0x11) [ 90.148104] ata1.15: hard resetting link [ 90.148177] ata1: controller in dubious state, performing PORT_RST [ 92.380043] ata1.15: SATA link up 1.5 Gbps (SStatus 113 SControl 10) [ 92.380415] ata1.00: hard resetting link [ 92.700329] ata1.00: SATA link down (SStatus 0 SControl 10) [ 92.700398] ata1.01: hard resetting link [ 93.020334] ata1.01: SATA link down (SStatus 0 SControl 310) [ 94.076081] ata1.02: hard resetting link [ 94.396398] ata1.02: SATA link up 3.0 Gbps (SStatus 123 SControl 300) [ 94.396479] ata1.03: hard resetting link [ 94.716392] ata1.03: SATA link down (SStatus 0 SControl 310) [ 94.716465] ata1.04: hard resetting link [ 95.036397] ata1.04: SATA link down (SStatus 0 SControl 310) [ 95.036474] ata1.05: hard resetting link [ 95.356426] ata1.05: SATA link up 1.5 Gbps (SStatus 113 SControl 310) [ 95.369116] ata1.02: ATA-8: Hitachi HDT721010SLA360, ST6OA31B, max UDMA/133 [ 95.369193] ata1.02: 1953525168 sectors, multi 0: LBA48 NCQ (depth 31/32) [ 95.370256] ata1.02: configured for UDMA/100 [ 95.370769] ata1: EH completeIf I swap cables from ata1 to ata2 while the system is running, the backplane will detect drives and work fine.
I'm already forcing a slower speed:
yan@selene:/etc/modprobe.d$ cat libata options libata force=1.15:1.5Gbps,2.15:1.5Gbps,3.15:1.5Gbps,4.15:1.5GbpsI'm using a Sil 3132 controller:
02:00.0 Mass storage controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)
The system sees 3 port multipliers; not 4 as it should:
Quoted 7 lines
[ 82.488354] ata4.15: Port Multiplier 1.1, 0x1095:0x3726 r23, 6 ports, feat 0x1/0x9 [ 82.488507] ata3.15: Port Multiplier 1.1, 0x1095:0x3726 r23, 6 ports, feat 0x1/0x9 [ 82.796408] ata1.15: Port Multiplier 1.1, 0x1095:0x3726 r23, 6 ports, feat 0x1/0x9I suspect it's cabling length as the 5 "dead" slots are the farthest down the backplane but swapping cables randomly just doesn't seem like a good approach.
What's my next step?
..
Perhaps try a different chipset controller card, in hope that it will have stronger drive capability?
Quoted 5 lines · Mark Lord
Quoted 4 lines · Yan Seiner
I have an external enclosure with an SATA backplane with 20 drive slots. I am using 4 port multipliers to drive all 20 slots.
One set of 5 slots connected to a single port multiplier always fails with
.....
<snippage>
Quoted 10 lines
Quoted 10 lines
02:00.0 Mass storage controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)
The system sees 3 port multipliers; not 4 as it should:
[ 82.488354] ata4.15: Port Multiplier 1.1, 0x1095:0x3726 r23, 6 ports, feat 0x1/0x9 [ 82.488507] ata3.15: Port Multiplier 1.1, 0x1095:0x3726 r23, 6 ports, feat 0x1/0x9 [ 82.796408] ata1.15: Port Multiplier 1.1, 0x1095:0x3726 r23, 6 ports, feat 0x1/0x9
Quoted 7 lines
Quoted 3 lines
I suspect it's cabling length as the 5 "dead" slots are the farthest down the backplane but swapping cables randomly just doesn't seem like a good approach.
What's my next step?
..
Perhaps try a different chipset controller card, in hope that it will have stronger drive capability?
OK, did that. No joy.
[ 82.533526] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310) [ 82.533606] ata3: link online but device misclassified, retrying [ 82.533686] ata3: reset failed (errno=-11), retrying in 9 secs
Is there some way to be more forceful with the reset? I suspect one of the port multiplier boards is bad at this point. Is there some way to force a hard reset from user space? What can I from userspace short of hitting the thing with a hammer?
--Yan
Signature · 9 lines
o__
,>/'_ o__
(_)\(_) ,>/'_ o__
Yan Seiner (_)\(_) ,>/'_ o__
Personal Trainer (_)\(_) ,>/'_ o__
Professional Engineer (_)\(_) ,>/'_
Who says engineers have to be pencil necked geeks? (_)\(_)
I like paying taxes. With them I buy civilization.Quoted 8 lines · Yan Seiner
[ 82.533526] ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310) [ 82.533606] ata3: link online but device misclassified, retrying [ 82.533686] ata3: reset failed (errno=-11), retrying in 9 secsIs there some way to be more forceful with the reset? I suspect one of the port multiplier boards is bad at this point. Is there some way to force a hard reset from user space? What can I from userspace short of hitting the thing with a hammer?
ATA already uses hardreset. You can trigger it by echoing "- - -" to /sys/class/scsi_host/hostN/scan but I doubt it would do any good. :-(
Signature · 1 line
tejun