Skip to contentSkip to conversation

And again help on deciphering an error

DISCUSSION Peter Rabbitson 4 participants 11 messages Oct 28, 2008 – Nov 15, 2008

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello,

Before I ask my question - if there is some documentation that answers what I am asking - I would be thrilled to read it. So far google has not helped me at all.

This being said I need help with understanding what is going on here (full log):

Oct 27 11:33:41 Arzamas kernel: ata6.00: exception Emask 0x10 SAct 0x0
SErr 0x80000 action 0xe frozen
Oct 27 11:33:41 Arzamas kernel: ata6.00: irq_stat 0x01100010, PHY RDY
changed
Oct 27 11:33:41 Arzamas kernel: ata6: SError: { 10B8B }
Oct 27 11:33:41 Arzamas kernel: ata6.00: cmd
ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Oct 27 11:33:41 Arzamas kernel: res 06/37:00:00:00:00/00:00:00:00:06/00
Emask 0x12 (ATA bus error)
Oct 27 11:33:41 Arzamas kernel: ata6.00: error: { IDNF ABRT }
Oct 27 11:33:41 Arzamas kernel: ata6: hard resetting link
Oct 27 11:33:46 Arzamas kernel: ata6: SATA link up 3.0 Gbps (SStatus 123
SControl 0)
Oct 27 11:33:46 Arzamas kernel: ata6.00: configured for UDMA/100
Oct 27 11:33:46 Arzamas kernel: ata6: EH complete
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] 488397168 512-byte
hardware sectors (250059 MB)
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] Write Protect is off
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] Mode Sense: 00 3a 00 00
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] Write cache: enabled,
read cache: enabled, doesn't support DPO or FUA
Oct 27 11:33:46 Arzamas kernel: end_request: I/O error, dev sde, sector
488166955
Oct 27 11:33:46 Arzamas kernel: md: super_written gets error=-5, uptodate=0
Oct 27 11:33:46 Arzamas kernel: raid10: Disk failure on sde2, disabling
device.
Oct 27 11:33:46 Arzamas kernel: raid10: Operation continuing on 3 devices.
Oct 27 11:33:46 Arzamas kernel: RAID10 conf printout:
Oct 27 11:33:46 Arzamas kernel: --- wd:3 rd:4
Oct 27 11:33:46 Arzamas kernel: disk 0, wo:0, o:1, dev:sdc2
Oct 27 11:33:46 Arzamas kernel: disk 1, wo:0, o:1, dev:sdb2
Oct 27 11:33:46 Arzamas kernel: disk 2, wo:1, o:0, dev:sde2
Oct 27 11:33:46 Arzamas kernel: disk 3, wo:0, o:1, dev:sdd2
Oct 27 11:33:46 Arzamas kernel: RAID10 conf printout:
Oct 27 11:33:46 Arzamas kernel: --- wd:3 rd:4
Oct 27 11:33:46 Arzamas kernel: disk 0, wo:0, o:1, dev:sdc2
Oct 27 11:33:46 Arzamas kernel: disk 1, wo:0, o:1, dev:sdb2
Oct 27 11:33:46 Arzamas kernel: disk 3, wo:0, o:1, dev:sdd2

Afterwards I removed sde from all arrays and mounts, and grilled it repeatedly with badblocks and smartctl -t long. Nothing came out for over a day. Below see the smartclt -a output. This happened one day after I upgraded to 2.6.27.4 from 2.6.23.

Can someone please tell me what seems to be dying? Thank you!

root@Arzamas:~# smartctl -a /dev/sde
smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===

Model Family:     Western Digital Caviar Second Generation Serial ATA family
Device Model:     WDC WD2500AAJS-00B4A0
Serial Number:    WD-WCAT11669666
Firmware Version: 01.03A01
User Capacity:    250,059,350,016 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Tue Oct 28 19:52:16 2008 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error. Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine
completed
without error or no self-test has ever been run.
Total time to complete Offline
data collection: (6180) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support. Suspend Offline collection upon new command.
Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 75) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303f) SCT Status supported.
SCT Feature Control supported. SCT Data Table supported.

SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always
- 0
3 Spin_Up_Time 0x0027 175 153 021 Pre-fail Always
- 2250
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always
- 17
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always
- 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always
- 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always
- 694
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always
- 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always
- 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always
- 17
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always
- 13
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always
- 17
194 Temperature_Celsius 0x0022 106 100 000 Old_age Always
- 37
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always
- 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always
- 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always
- 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
Offline - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 694

     -
# 2  Extended offline    Completed without error       00%       684
     -
# 3  Extended offline    Completed without error       00%       663
     -
# 4  Short offline       Aborted by host               60%       662
     -
# 5  Short offline       Completed without error       00%       662
     -

SMART Selective self-test log data structure revision number 1

 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing

Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@Arzamas:~#

#1 lore raw
Quoted 49 lines · Peter Rabbitson

Hello,

Before I ask my question - if there is some documentation that answers what I am asking - I would be thrilled to read it. So far google has not helped me at all.

This being said I need help with understanding what is going on here (full log):

Oct 27 11:33:41 Arzamas kernel: ata6.00: exception Emask 0x10 SAct 0x0
SErr 0x80000 action 0xe frozen
Oct 27 11:33:41 Arzamas kernel: ata6.00: irq_stat 0x01100010, PHY RDY
changed
Oct 27 11:33:41 Arzamas kernel: ata6: SError: { 10B8B }
Oct 27 11:33:41 Arzamas kernel: ata6.00: cmd
ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Oct 27 11:33:41 Arzamas kernel: res 06/37:00:00:00:00/00:00:00:00:06/00
Emask 0x12 (ATA bus error)
Oct 27 11:33:41 Arzamas kernel: ata6.00: error: { IDNF ABRT }
Oct 27 11:33:41 Arzamas kernel: ata6: hard resetting link
Oct 27 11:33:46 Arzamas kernel: ata6: SATA link up 3.0 Gbps (SStatus 123
SControl 0)
Oct 27 11:33:46 Arzamas kernel: ata6.00: configured for UDMA/100
Oct 27 11:33:46 Arzamas kernel: ata6: EH complete
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] 488397168 512-byte
hardware sectors (250059 MB)
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] Write Protect is off
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] Mode Sense: 00 3a 00 00
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] Write cache: enabled,
read cache: enabled, doesn't support DPO or FUA
Oct 27 11:33:46 Arzamas kernel: end_request: I/O error, dev sde, sector
488166955
Oct 27 11:33:46 Arzamas kernel: md: super_written gets error=-5, uptodate=0
Oct 27 11:33:46 Arzamas kernel: raid10: Disk failure on sde2, disabling
device.
Oct 27 11:33:46 Arzamas kernel: raid10: Operation continuing on 3 devices.
Oct 27 11:33:46 Arzamas kernel: RAID10 conf printout:
Oct 27 11:33:46 Arzamas kernel: --- wd:3 rd:4
Oct 27 11:33:46 Arzamas kernel: disk 0, wo:0, o:1, dev:sdc2
Oct 27 11:33:46 Arzamas kernel: disk 1, wo:0, o:1, dev:sdb2
Oct 27 11:33:46 Arzamas kernel: disk 2, wo:1, o:0, dev:sde2
Oct 27 11:33:46 Arzamas kernel: disk 3, wo:0, o:1, dev:sdd2
Oct 27 11:33:46 Arzamas kernel: RAID10 conf printout:
Oct 27 11:33:46 Arzamas kernel: --- wd:3 rd:4
Oct 27 11:33:46 Arzamas kernel: disk 0, wo:0, o:1, dev:sdc2
Oct 27 11:33:46 Arzamas kernel: disk 1, wo:0, o:1, dev:sdb2
Oct 27 11:33:46 Arzamas kernel: disk 3, wo:0, o:1, dev:sdd2
Quoted 6 lines

Afterwards I removed sde from all arrays and mounts, and grilled it repeatedly with badblocks and smartctl -t long. Nothing came out for over a day. Below see the smartclt -a output. This happened one day after I upgraded to 2.6.27.4 from 2.6.23.

Can someone please tell me what seems to be dying? Thank you!

Oct 27 11:33:46 Arzamas kernel: end_request: I/O error, dev sde, sector

488166955

Disk is going bad, I had the same problem, a week later, it was another I/O error, a week after that, another I/O error, replace/RMA disk.

Justin.

#2 lore raw

Re: And again help on deciphering an error (continued)

In reply to Peter Rabbitson
Quoted 28 lines · Peter Rabbitson

Hello,

I need help with understanding what is going on here (full log):

Oct 27 11:33:41 Arzamas kernel: ata6.00: exception Emask 0x10 SAct 0x0
SErr 0x80000 action 0xe frozen
Oct 27 11:33:41 Arzamas kernel: ata6.00: irq_stat 0x01100010, PHY RDY
changed
Oct 27 11:33:41 Arzamas kernel: ata6: SError: { 10B8B }
Oct 27 11:33:41 Arzamas kernel: ata6.00: cmd
ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Oct 27 11:33:41 Arzamas kernel: res 06/37:00:00:00:00/00:00:00:00:06/00
Emask 0x12 (ATA bus error)
Oct 27 11:33:41 Arzamas kernel: ata6.00: error: { IDNF ABRT }
Oct 27 11:33:41 Arzamas kernel: ata6: hard resetting link
Oct 27 11:33:46 Arzamas kernel: ata6: SATA link up 3.0 Gbps (SStatus 123
SControl 0)
Oct 27 11:33:46 Arzamas kernel: ata6.00: configured for UDMA/100
Oct 27 11:33:46 Arzamas kernel: ata6: EH complete
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] 488397168 512-byte
hardware sectors (250059 MB)
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] Write Protect is off
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] Mode Sense: 00 3a 00 00
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] Write cache: enabled,
read cache: enabled, doesn't support DPO or FUA
Oct 27 11:33:46 Arzamas kernel: end_request: I/O error, dev sde, sector
Quoted 1 line

488166955

The saga continues. After replacing the drive (and I didn't make a mistake as the serial numbers don't match) I got _exaclty_ the same error a week later. I have replaced the cable for the drive in question,
will see what happens. The controller is:

02:03.0 Mass storage controller: Silicon Image, Inc. SiI 3124 PCI-X Serial ATA Controller (rev 02)
Subsystem: Silicon Image, Inc. SiI 3124 PCI-X Serial ATA Controller
Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV+ VGASnoop- ParErr+
Stepping+ SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-

	Latency: 64, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 25
	Region 0: Memory at fc5fd800 (64-bit, non-prefetchable) [size=128]
	Region 2: Memory at fc5f0000 (64-bit, non-prefetchable) [size=32K]
	Region 4: I/O ports at b000 [size=16]
	Expansion ROM at fc480000 [disabled] [size=512K]
	Capabilities: [64] Power Management version 2
		Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [40] PCI-X non-bridge device
		Command: DPERE- ERO+ RBC=512 OST=12
		Status: Dev=ff:1f.0 64bit+ 133MHz+ SCD- USC- DC=simple DMMRBC=2048
DMOST=12 DMCRS=128 RSCEM- 266MHz- 533MHz-
	Capabilities: [54] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0
Enable-
		Address: 0000000000000000  Data: 0000
	Kernel driver in use: sata_sil24

Anyone has any bright ideas?

Thanks

New log follows:

Nov  8 15:41:18 Arzamas kernel: ata6.00: exception Emask 0x10 SAct 0x0
SErr 0x80000 action 0xe frozen
Nov  8 15:41:18 Arzamas kernel: ata6.00: irq_stat 0x01100010, PHY RDY
changed
Nov  8 15:41:18 Arzamas kernel: ata6: SError: { 10B8B }
Nov  8 15:41:18 Arzamas kernel: ata6.00: cmd
ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Nov  8 15:41:18 Arzamas kernel: res 06/37:00:00:00:00/00:00:00:00:06/00
Emask 0x12 (ATA bus error)
Nov  8 15:41:18 Arzamas kernel: ata6.00: error: { IDNF ABRT }
Nov  8 15:41:18 Arzamas kernel: ata6: hard resetting link
Nov  8 15:41:22 Arzamas kernel: ata6: SATA link up 3.0 Gbps (SStatus 123
SControl 0)
Nov  8 15:41:22 Arzamas kernel: ata6.00: configured for UDMA/100
Nov  8 15:41:22 Arzamas kernel: ata6: EH complete
Nov  8 15:41:22 Arzamas kernel: sd 6:0:0:0: [sde] 488397168 512-byte
hardware sectors (250059 MB)
Nov  8 15:41:22 Arzamas kernel: sd 6:0:0:0: [sde] Write Protect is off
Nov  8 15:41:22 Arzamas kernel: sd 6:0:0:0: [sde] Mode Sense: 00 3a 00 00
Nov  8 15:41:22 Arzamas kernel: sd 6:0:0:0: [sde] Write cache: enabled,
read cache: enabled, doesn't support DPO or FUA
Nov  8 15:41:22 Arzamas kernel: end_request: I/O error, dev sde, sector
488166955
Nov  8 15:41:22 Arzamas kernel: md: super_written gets error=-5, uptodate=0
Nov  8 15:41:22 Arzamas kernel: raid10: Disk failure on sde2, disabling
device.
Nov  8 15:41:22 Arzamas kernel: raid10: Operation continuing on 3 devices.

smartctl -a of new drive:

smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===

Model Family:     Western Digital Caviar Second Generation Serial ATA family
Device Model:     WDC WD2500AAJS-08B4A0
Serial Number:    WD-WMAT14036837
Firmware Version: 01.03A01
User Capacity:    250,059,350,016 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Fri Nov 14 13:22:23 2008 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine
completed
without error or no self-test has ever been run.
Total time to complete Offline
data collection: (5580) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support. Suspend Offline collection upon new command.
Offline surface scan supported. Self-test supported.
Conveyance Self-test supported. Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 68) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303f) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always
- 0
3 Spin_Up_Time 0x0027 174 174 021 Pre-fail Always
- 2266
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always
- 9
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always
- 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always
- 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always
- 329
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always
- 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always
- 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always
- 7
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always
- 6
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always
- 9
194 Temperature_Celsius 0x0022 106 103 000 Old_age Always
- 37
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always
- 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always
- 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age
Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always
- 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age
Offline - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 20

     -
# 2  Extended offline    Completed without error       00%        14
     -
# 3  Short offline       Completed without error       00%         1
     -

SMART Selective self-test log data structure revision number 1

 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing

Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

#3 lore raw

Re: And again help on deciphering an error (continued)

Quoted 62 lines · Peter Rabbitson
Quoted 28 lines · Peter Rabbitson

Hello,

I need help with understanding what is going on here (full log):

Oct 27 11:33:41 Arzamas kernel: ata6.00: exception Emask 0x10 SAct 0x0
SErr 0x80000 action 0xe frozen
Oct 27 11:33:41 Arzamas kernel: ata6.00: irq_stat 0x01100010, PHY RDY
changed
Oct 27 11:33:41 Arzamas kernel: ata6: SError: { 10B8B }
Oct 27 11:33:41 Arzamas kernel: ata6.00: cmd
ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Oct 27 11:33:41 Arzamas kernel: res 06/37:00:00:00:00/00:00:00:00:06/00
Emask 0x12 (ATA bus error)
Oct 27 11:33:41 Arzamas kernel: ata6.00: error: { IDNF ABRT }
Oct 27 11:33:41 Arzamas kernel: ata6: hard resetting link
Oct 27 11:33:46 Arzamas kernel: ata6: SATA link up 3.0 Gbps (SStatus 123
SControl 0)
Oct 27 11:33:46 Arzamas kernel: ata6.00: configured for UDMA/100
Oct 27 11:33:46 Arzamas kernel: ata6: EH complete
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] 488397168 512-byte
hardware sectors (250059 MB)
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] Write Protect is off
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] Mode Sense: 00 3a 00 00
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] Write cache: enabled,
read cache: enabled, doesn't support DPO or FUA
Oct 27 11:33:46 Arzamas kernel: end_request: I/O error, dev sde, sector
Quoted 1 line

488166955

The saga continues. After replacing the drive (and I didn't make a mistake as the serial numbers don't match) I got _exaclty_ the same error a week later. I have replaced the cable for the drive in question, will see what happens. The controller is:

02:03.0 Mass storage controller: Silicon Image, Inc. SiI 3124 PCI-X Serial ATA Controller (rev 02)
Subsystem: Silicon Image, Inc. SiI 3124 PCI-X Serial ATA Controller
Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV+ VGASnoop- ParErr+
Stepping+ SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-

	Latency: 64, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 25
	Region 0: Memory at fc5fd800 (64-bit, non-prefetchable) [size=128]
	Region 2: Memory at fc5f0000 (64-bit, non-prefetchable) [size=32K]
	Region 4: I/O ports at b000 [size=16]
	Expansion ROM at fc480000 [disabled] [size=512K]
	Capabilities: [64] Power Management version 2
		Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [40] PCI-X non-bridge device
		Command: DPERE- ERO+ RBC=512 OST=12
		Status: Dev=ff:1f.0 64bit+ 133MHz+ SCD- USC- DC=simple DMMRBC=2048
DMOST=12 DMCRS=128 RSCEM- 266MHz- 533MHz-
	Capabilities: [54] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0
Enable-
		Address: 0000000000000000  Data: 0000
	Kernel driver in use: sata_sil24
Quoted 1 line

Anyone has any bright ideas?

Hm, someone else also has similar problems, maybe we ought to buy new hw or something, I have the same/similar issue as you.

BTW, after that error, if you re-run a short/long test, do they succeed or hang (never complete)?

Justin.

#4 lore raw

Re: And again help on deciphering an error (continued)

Quoted 4 lines · Justin Piszcz

On Fri, 14 Nov 2008, Peter Rabbitson wrote:

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines · Peter Rabbitson

Hello,

I need help with understanding what is going on here (full log):

Anyone has any bright ideas?

Here is the bugzilla URL he created:
https://bugzilla.redhat.com/show_bug.cgi?id=462425

What type of motherboard do you have?

I have that error and it occurs on sata_mv, Intel ICH8 and sil24, I don't think its related to the card(s) themselves.

Justin.

#6 lore raw

Re: And again help on deciphering an error (continued)

In reply to Richard Scobie
Quoted 9 lines · Richard Scobie
Quoted 8 lines · Justin Piszcz
Quoted 4 lines · Peter Rabbitson
Quoted 2 lines · Peter Rabbitson

Hello,

Quoted 9 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I need help with understanding what is going on here (full log):

These posts might be worth sending to linux-ide.

Regards,

Richard

I have a whole bunch, but here is the last one:

[64894.610696] ata9.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[64894.610703] ata9.00: irq_stat 0x00020002, device error via D2H FIS
[64894.610708] ata9.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
[64894.610709]          res 51/04:00:34:cf:f3/00:00:00:f3:40/a3 Emask 0x1 (devic
e error)
[64894.610714] ata9.00: status: { DRDY ERR }
[64894.610718] ata9.00: error: { ABRT }
[64894.626437] ata9.00: configured for UDMA/100
[64894.626449] ata9: EH complete
[64894.626654] sd 8:0:0:0: [sdh] 586072368 512-byte hardware sectors (300069 MB)
[64894.637744] end_request: I/O error, dev sdh, sector 586067135
[64894.637759] md: super_written gets error=-5, uptodate=0
[64894.637762] raid5: Disk failure on sdh1, disabling device.
[64894.637763] raid5: Operation continuing on 9 devices.
[64894.638129] sd 8:0:0:0: [sdh] Write Protect is off
[64894.638131] sd 8:0:0:0: [sdh] Mode Sense: 00 3a 00 00
[64894.638148] sd 8:0:0:0: [sdh] Write cache: enabled, read cache: enabled, does

n't support DPO or FUA

Justin.

#5 lore raw

Re: And again help on deciphering an error (continued)

In reply to Justin Piszcz
Quoted 10 lines · Justin Piszcz
Quoted 6 lines · Peter Rabbitson
Quoted 4 lines · Peter Rabbitson

Hello,

I need help with understanding what is going on here (full log):

These posts might be worth sending to linux-ide.

Regards,

Richard

#7 lore raw

Re: And again help on deciphering an error (continued)

In reply to Justin Piszcz
Quoted 16 lines · Justin Piszcz
Quoted 12 lines · Richard Scobie
Quoted 10 lines · Justin Piszcz
Quoted 6 lines · Peter Rabbitson
Quoted 4 lines · Peter Rabbitson

Hello,

I need help with understanding what is going on here (full log):

Quoted 8 lines
Quoted 6 lines

These posts might be worth sending to linux-ide.

Regards,

Richard

I have a whole bunch, but here is the last one:

Here are some more:

Jul 14 05:54:55 p34 kernel: [478107.751877] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Jul 14 05:54:55 p34 kernel: [478107.751883] ata4.00: irq_stat 0x40000001
Jul 14 05:54:55 p34 kernel: [478107.751889] ata4.00: cmd 25/00:f8:08:b0:ee/00:00:22:00:00/e0 tag 0 dma 126976 in
Jul 14 05:54:55 p34 kernel: [478107.751890]          res 51/40:f8:47:b0:ee/00:00:22:00:00/e0 Emask 0x9 (media error)
Jul 14 05:54:55 p34 kernel: [478107.751895] ata4.00: status: { DRDY ERR }
Jul 14 05:54:55 p34 kernel: [478107.751898] ata4.00: error: { UNC }
Jul 14 05:54:55 p34 kernel: [478107.773836] ata4.00: configured for UDMA/133
Jul 14 05:54:55 p34 kernel: [478107.773845] ata4: EH complete
Jul 14 05:54:55 p34 kernel: [478107.844912] sd 3:0:0:0: [sdd] 586072368 512-byte hardware sectors (300069 MB)
Jul 14 05:54:55 p34 kernel: [478107.846321] sd 3:0:0:0: [sdd] Write Protect is off
Jul 14 05:54:55 p34 kernel: [478107.846325] sd 3:0:0:0: [sdd] Mode Sense: 00 3a 00 00
Jul 14 05:54:55 p34 kernel: [478107.849297] sd 3:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Oct 12 09:25:22 p34 kernel: [1434196.959129] ata6.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Oct 12 09:25:30 p34 kernel: [1434196.959137] ata6.00: irq_stat 0x40000001
Oct 12 09:25:30 p34 kernel: [1434196.959142] ata6.00: cmd 25/00:00:00:af:ee/00:02:22:00:00/e0 tag 0 dma 262144 in
Oct 12 09:25:30 p34 kernel: [1434196.959144]          res 51/40:00:4f:b0:ee/00:02:22:00:00/e0 Emask 0x9 (media error)
Oct 12 09:25:30 p34 kernel: [1434196.959149] ata6.00: status: { DRDY ERR }
Oct 12 09:25:30 p34 kernel: [1434196.959152] ata6.00: error: { UNC }
Oct 12 09:25:30 p34 kernel: [1434196.999110] ata6.00: configured for UDMA/133
Oct 12 09:25:30 p34 kernel: [1434196.999144] ata6: EH complete
Oct 12 09:25:30 p34 kernel: [1434203.096157] ata6.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Oct 12 09:25:30 p34 kernel: [1434203.096165] ata6.00: irq_stat 0x40000001
Oct 12 09:25:30 p34 kernel: [1434203.096171] ata6.00: cmd 25/00:00:00:af:ee/00:02:22:00:00/e0 tag 0 dma 262144 in
Oct 12 09:25:30 p34 kernel: [1434203.096172]          res 51/40:00:4f:b0:ee/00:02:22:00:00/e0 Emask 0x9 (media error)
Oct 12 09:25:30 p34 kernel: [1434203.096177] ata6.00: status: { DRDY ERR }
Oct 12 09:25:30 p34 kernel: [1434203.096181] ata6.00: error: { UNC }
Oct 12 09:25:30 p34 kernel: [1434203.104811] ata6.00: configured for UDMA/133
Oct 12 09:25:30 p34 kernel: [1434203.104819] ata6: EH complete
Nov  9 09:25:19 p34 kernel: [ 3686.460568] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Nov  9 09:25:19 p34 kernel: [ 3686.460574] ata4.00: irq_stat 0x40000001
Nov  9 09:25:19 p34 kernel: [ 3686.460579] ata4.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Nov  9 09:25:19 p34 kernel: [ 3686.460580]          res 51/10:00:34:cf:f3/00:00:00:f3:40/a3 Emask 0x81 (invalid argument)
Nov  9 09:25:19 p34 kernel: [ 3686.460585] ata4.00: status: { DRDY ERR }
Nov  9 09:25:19 p34 kernel: [ 3686.460589] ata4.00: error: { IDNF }
Nov  9 09:25:24 p34 kernel: [ 3691.461173] ata4.00: qc timeout (cmd 0xec)
Nov  9 09:25:24 p34 kernel: [ 3691.461183] ata4.00: failed to IDENTIFY (I/O error, err_mask=0x5)
Nov  9 09:25:24 p34 kernel: [ 3691.461187] ata4.00: revalidation failed (errno=-5)
Nov  9 09:25:24 p34 kernel: [ 3691.461192] ata4: hard resetting link
Nov  9 09:25:24 p34 kernel: [ 3691.766178] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Nov  9 09:25:24 p34 kernel: [ 3691.803834] ata4.00: configured for UDMA/133
Nov  9 09:25:24 p34 kernel: [ 3691.803846] end_request: I/O error, dev sdd, sector 586067135
Nov  9 09:25:24 p34 kernel: [ 3691.803849] md: super_written gets error=-5, uptodate=0
Nov  9 09:25:24 p34 kernel: [ 3691.803853] raid5: Disk failure on sdd1, disabling device.
Nov  9 09:25:24 p34 kernel: [ 3691.803854] raid5: Operation continuing on 8 devices.
Nov 12 01:22:28 p34 kernel: [64894.610696] ata9.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Nov 12 01:22:28 p34 kernel: [64894.610703] ata9.00: irq_stat 0x00020002, device error via D2H FIS
Nov 12 01:22:28 p34 kernel: [64894.610708] ata9.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Nov 12 01:22:28 p34 kernel: [64894.610709]          res 51/04:00:34:cf:f3/00:00:00:f3:40/a3 Emask 0x1 (device error)
Nov 12 01:22:28 p34 kernel: [64894.610714] ata9.00: status: { DRDY ERR }
Nov 12 01:22:28 p34 kernel: [64894.610718] ata9.00: error: { ABRT }
Nov 12 01:22:28 p34 kernel: [64894.626437] ata9.00: configured for UDMA/100
Nov 12 01:22:28 p34 kernel: [64894.626449] ata9: EH complete
Nov 12 01:22:28 p34 kernel: [64894.626654] sd 8:0:0:0: [sdh] 586072368 512-byte hardware sectors (300069 MB)
Nov 12 01:22:28 p34 kernel: [64894.637744] end_request: I/O error, dev sdh, sector 586067135
Nov 12 01:22:28 p34 kernel: [64894.637759] md: super_written gets error=-5, uptodate=0
Nov 12 01:22:28 p34 kernel: [64894.637762] raid5: Disk failure on sdh1, disabling device.
Nov 12 01:22:28 p34 kernel: [64894.637763] raid5: Operation continuing on 9 devices.
Nov 12 01:22:28 p34 kernel: [64894.638129] sd 8:0:0:0: [sdh] Write Protect is off
Nov 12 01:22:28 p34 kernel: [64894.638131] sd 8:0:0:0: [sdh] Mode Sense: 00 3a 00 00
Nov 12 01:22:28 p34 kernel: [64894.638148] sd 8:0:0:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
#10 lore raw

Re: And again help on deciphering an error (continued)

In reply to Justin Piszcz
Quoted 70 lines · Justin Piszcz
Quoted 66 lines · Peter Rabbitson
Quoted 29 lines · Peter Rabbitson

Hello,

I need help with understanding what is going on here (full log):

Oct 27 11:33:41 Arzamas kernel: ata6.00: exception Emask 0x10 SAct 0x0
SErr 0x80000 action 0xe frozen
Oct 27 11:33:41 Arzamas kernel: ata6.00: irq_stat 0x01100010, PHY RDY
changed
Oct 27 11:33:41 Arzamas kernel: ata6: SError: { 10B8B }
Oct 27 11:33:41 Arzamas kernel: ata6.00: cmd
ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Oct 27 11:33:41 Arzamas kernel: res 06/37:00:00:00:00/00:00:00:00:06/00
Emask 0x12 (ATA bus error)
Oct 27 11:33:41 Arzamas kernel: ata6.00: error: { IDNF ABRT }
Oct 27 11:33:41 Arzamas kernel: ata6: hard resetting link
Oct 27 11:33:46 Arzamas kernel: ata6: SATA link up 3.0 Gbps (SStatus 123
SControl 0)
Oct 27 11:33:46 Arzamas kernel: ata6.00: configured for UDMA/100
Oct 27 11:33:46 Arzamas kernel: ata6: EH complete
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] 488397168 512-byte
hardware sectors (250059 MB)
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] Write Protect is off
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] Mode Sense: 00 3a
00 00
Oct 27 11:33:46 Arzamas kernel: sd 6:0:0:0: [sde] Write cache: enabled,
read cache: enabled, doesn't support DPO or FUA
Oct 27 11:33:46 Arzamas kernel: end_request: I/O error, dev sde, sector
Quoted 1 line

488166955

The saga continues. After replacing the drive (and I didn't make a mistake as the serial numbers don't match) I got _exaclty_ the same error a week later. I have replaced the cable for the drive in question, will see what happens. The controller is:

02:03.0 Mass storage controller: Silicon Image, Inc. SiI 3124 PCI-X Serial ATA Controller (rev 02)
Subsystem: Silicon Image, Inc. SiI 3124 PCI-X Serial ATA Controller
Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV+ VGASnoop- ParErr+
Stepping+ SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 25
Region 0: Memory at fc5fd800 (64-bit, non-prefetchable) [size=128] Region 2: Memory at fc5f0000 (64-bit, non-prefetchable) [size=32K] Region 4: I/O ports at b000 [size=16]
Expansion ROM at fc480000 [disabled] [size=512K]
Capabilities: [64] Power Management version 2
Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [40] PCI-X non-bridge device
Command: DPERE- ERO+ RBC=512 OST=12
Status: Dev=ff:1f.0 64bit+ 133MHz+ SCD- USC- DC=simple
DMMRBC=2048
DMOST=12 DMCRS=128 RSCEM- 266MHz- 533MHz-
Capabilities: [54] Message Signalled Interrupts: Mask- 64bit+
Queue=0/0
Enable-
Address: 0000000000000000 Data: 0000
Kernel driver in use: sata_sil24

Quoted 8 lines
Quoted 1 line

Anyone has any bright ideas?

Hm, someone else also has similar problems, maybe we ought to buy new hw or something, I have the same/similar issue as you.

BTW, after that error, if you re-run a short/long test, do they succeed or hang (never complete)?

They succeed with flying colors. Moreover I had badblocks -svw running at the same time (so the drive was really loaded). And I did this repeatedly with the first one - no issues.

Peter

#8 lore raw

Re: And again help on deciphering an error (continued)

In reply to Justin Piszcz
Quoted 8 lines · Justin Piszcz

Here are some more:

Jul 14 05:54:55 p34 kernel: [478107.751877] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Jul 14 05:54:55 p34 kernel: [478107.751883] ata4.00: irq_stat 0x40000001
Jul 14 05:54:55 p34 kernel: [478107.751889] ata4.00: cmd 25/00:f8:08:b0:ee/00:00:22:00:00/e0 tag 0 dma 126976 in
Jul 14 05:54:55 p34 kernel: [478107.751890]          res 51/40:f8:47:b0:ee/00:00:22:00:00/e0 Emask 0x9 (media error)
Jul 14 05:54:55 p34 kernel: [478107.751895] ata4.00: status: { DRDY ERR }
Jul 14 05:54:55 p34 kernel: [478107.751898] ata4.00: error: { UNC }

Drive reporting an uncorrectable error on a read command.

Quoted 6 lines
Nov  9 09:25:19 p34 kernel: [ 3686.460568] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Nov  9 09:25:19 p34 kernel: [ 3686.460574] ata4.00: irq_stat 0x40000001
Nov  9 09:25:19 p34 kernel: [ 3686.460579] ata4.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Nov  9 09:25:19 p34 kernel: [ 3686.460580]          res 51/10:00:34:cf:f3/00:00:00:f3:40/a3 Emask 0x81 (invalid argument)
Nov  9 09:25:19 p34 kernel: [ 3686.460585] ata4.00: status: { DRDY ERR }
Nov  9 09:25:19 p34 kernel: [ 3686.460589] ata4.00: error: { IDNF }

Flush cache returned ID not found (drive couldn't find a sector)

Quoted 11 lines
Nov  9 09:25:24 p34 kernel: [ 3691.461173] ata4.00: qc timeout (cmd 0xec)
Nov  9 09:25:24 p34 kernel: [ 3691.461183] ata4.00: failed to IDENTIFY (I/O error, err_mask=0x5)
Nov  9 09:25:24 p34 kernel: [ 3691.461187] ata4.00: revalidation failed (errno=-5)
Nov  9 09:25:24 p34 kernel: [ 3691.461192] ata4: hard resetting link
Nov  9 09:25:24 p34 kernel: [ 3691.766178] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Nov  9 09:25:24 p34 kernel: [ 3691.803834] ata4.00: configured for UDMA/133
Nov  9 09:25:24 p34 kernel: [ 3691.803846] end_request: I/O error, dev sdd, sector 586067135
Nov  9 09:25:24 p34 kernel: [ 3691.803849] md: super_written gets error=-5, uptodate=0
Nov  9 09:25:24 p34 kernel: [ 3691.803853] raid5: Disk failure on sdd1, disabling device.
Nov  9 09:25:24 p34 kernel: [ 3691.803854] raid5: Operation continuing on 8 devices.
Quoted 6 lines
Nov 12 01:22:28 p34 kernel: [64894.610696] ata9.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Nov 12 01:22:28 p34 kernel: [64894.610703] ata9.00: irq_stat 0x00020002, device error via D2H FIS
Nov 12 01:22:28 p34 kernel: [64894.610708] ata9.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Nov 12 01:22:28 p34 kernel: [64894.610709]          res 51/04:00:34:cf:f3/00:00:00:f3:40/a3 Emask 0x1 (device error)
Nov 12 01:22:28 p34 kernel: [64894.610714] ata9.00: status: { DRDY ERR }
Nov 12 01:22:28 p34 kernel: [64894.610718] ata9.00: error: { ABRT }

Flush cache returned command aborted, presumably because the write to media failed.

Looks like some sick drives, or some other more global problem (power, temperature, etc.) to me..

#9 lore raw

Re: And again help on deciphering an error (continued)

Quoted 57 lines · Robert Hancock
Quoted 11 lines · Justin Piszcz

Here are some more:

Jul 14 05:54:55 p34 kernel: [478107.751877] ata4.00: exception Emask
0x0 SAct 0x0 SErr 0x0 action 0x0
Jul 14 05:54:55 p34 kernel: [478107.751883] ata4.00: irq_stat 0x40000001
Jul 14 05:54:55 p34 kernel: [478107.751889] ata4.00: cmd
25/00:f8:08:b0:ee/00:00:22:00:00/e0 tag 0 dma 126976 in
Jul 14 05:54:55 p34 kernel: [478107.751890]          res
51/40:f8:47:b0:ee/00:00:22:00:00/e0 Emask 0x9 (media error)
Jul 14 05:54:55 p34 kernel: [478107.751895] ata4.00: status: { DRDY ERR }
Jul 14 05:54:55 p34 kernel: [478107.751898] ata4.00: error: { UNC }

Drive reporting an uncorrectable error on a read command.

Quoted 9 lines
Nov  9 09:25:19 p34 kernel: [ 3686.460568] ata4.00: exception Emask
0x0 SAct 0x0 SErr 0x0 action 0x0
Nov  9 09:25:19 p34 kernel: [ 3686.460574] ata4.00: irq_stat 0x40000001
Nov  9 09:25:19 p34 kernel: [ 3686.460579] ata4.00: cmd
ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Nov  9 09:25:19 p34 kernel: [ 3686.460580]          res
51/10:00:34:cf:f3/00:00:00:f3:40/a3 Emask 0x81 (invalid argument)
Nov  9 09:25:19 p34 kernel: [ 3686.460585] ata4.00: status: { DRDY ERR }
Nov  9 09:25:19 p34 kernel: [ 3686.460589] ata4.00: error: { IDNF }

Flush cache returned ID not found (drive couldn't find a sector)

Quoted 19 lines
Nov  9 09:25:24 p34 kernel: [ 3691.461173] ata4.00: qc timeout (cmd 0xec)
Nov  9 09:25:24 p34 kernel: [ 3691.461183] ata4.00: failed to IDENTIFY
(I/O error, err_mask=0x5)
Nov  9 09:25:24 p34 kernel: [ 3691.461187] ata4.00: revalidation
failed (errno=-5)
Nov  9 09:25:24 p34 kernel: [ 3691.461192] ata4: hard resetting link
Nov  9 09:25:24 p34 kernel: [ 3691.766178] ata4: SATA link up 3.0 Gbps
(SStatus 123 SControl 300)
Nov  9 09:25:24 p34 kernel: [ 3691.803834] ata4.00: configured for
UDMA/133
Nov  9 09:25:24 p34 kernel: [ 3691.803846] end_request: I/O error, dev
sdd, sector 586067135
Nov  9 09:25:24 p34 kernel: [ 3691.803849] md: super_written gets
error=-5, uptodate=0
Nov  9 09:25:24 p34 kernel: [ 3691.803853] raid5: Disk failure on
sdd1, disabling device.
Nov  9 09:25:24 p34 kernel: [ 3691.803854] raid5: Operation continuing

on 8 devices.

Quoted 10 lines
Nov 12 01:22:28 p34 kernel: [64894.610696] ata9.00: exception Emask
0x0 SAct 0x0 SErr 0x0 action 0x0
Nov 12 01:22:28 p34 kernel: [64894.610703] ata9.00: irq_stat
0x00020002, device error via D2H FIS
Nov 12 01:22:28 p34 kernel: [64894.610708] ata9.00: cmd
ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
Nov 12 01:22:28 p34 kernel: [64894.610709]          res
51/04:00:34:cf:f3/00:00:00:f3:40/a3 Emask 0x1 (device error)
Nov 12 01:22:28 p34 kernel: [64894.610714] ata9.00: status: { DRDY ERR }
Nov 12 01:22:28 p34 kernel: [64894.610718] ata9.00: error: { ABRT }
Quoted 6 lines

Flush cache returned command aborted, presumably because the write to media failed.

Looks like some sick drives, or some other more global problem (power, temperature, etc.) to me..

But an _identical_ sector number? Looks too unlikely... I might try to crank open hexedit and write repeatedly to this place. Sectors are still 512 bytes right?

Peter