Skip to contentSkip to conversation

array sync looping

DISCUSSION [email protected] 4 participants 7 messages Dec 8, 2006 – Dec 20, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi -

I recently upgraded to the 2.6.17-2-686 SMP kernel image (debian).

Now, when I hot-add a disk (mdadm /dev/md1 --add /dev/hdc3) the array resyncs to 100%, a bunch of errors appear and then the array resync starts from 0% again.

mdadm --zero-superblock on the "new" disk doesn't help.

Any suggestions on what might be happening or how to debug/fix would be welcome.

Thanks -

Leni

$ dmesg  # output truncated
...
md: md1: sync done.
hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=390716106, high=23, low=4840138, sector=390716042
ide: failed opcode was: unknown
end_request: I/O error, dev hda, sector 390716042
...
raid1: hda: unrecoverable I/O read error for block 388499072
RAID1 conf printout:
 --- wd:1 rd:3
 disk 0, wo:1, o:1, dev:hdc3
 disk 2, wo:0, o:1, dev:hda3
RAID1 conf printout:
 --- wd:1 rd:3
 disk 2, wo:0, o:1, dev:hda3
RAID1 conf printout:
 --- wd:1 rd:3
 disk 0, wo:1, o:1, dev:hdc3
 disk 2, wo:0, o:1, dev:hda3
md: syncing RAID array md
...
#1 lore raw

Frequent SATA errors / port timeouts in 2.6.18.3?

Hi all,
this may not be the best list for this question, but I figure that the number of disks connected to users here should be pretty big...

I upgraded from 2.6.17-rc4 to 2.6.18.3 about a week ago, and I've since had 3 drives kicked out of my 10-drive RAID5 array. Previously, I had no kicks over almost a year. The kernel message is:

ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 ata7.00: (BMDMA stat 0x20)
ata7.00: tag 0 cmd 0xc8 Emask 0x1 stat 0x41 err 0x4 (device error)
ata7: EH complete
SCSI device sdc: 488397168 512-byte hdwr sectors (250059 MB)
sdc: Write Protect is off
sdc: Mode Sense: 00 3a 00 00
SCSI device sdc: drive cache: write back
ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen ata7.00: (BMDMA stat 0x20)
ata7.00: tag 0 cmd 0xca Emask 0x4 stat 0x40 err 0x0 (timeout)
ata7: port is slow to respond, please be patient
ata7: port failed to respond (30 secs)
ata7: soft resetting port
ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ata7.00: failed to IDENTIFY (I/O error, err_mask=0x2) ata7.00: revalidation failed (errno=-5)
ata7: failed to recover some devices, retrying in 5 secs
ata7: hard resetting port
ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ata7.00: failed to IDENTIFY (I/O error, err_mask=0x2) ata7.00: revalidation failed (errno=-5)
ata7: failed to recover some devices, retrying in 5 secs
ata7: hard resetting port
ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ata7.00: failed to IDENTIFY (I/O error, err_mask=0x2) ata7.00: revalidation failed (errno=-5)
ata7.00: disabled
ata7: EH complete

First I thought it was a cabling or card issue, because the same drive got kicked twice. That drive was connected to a 2-port SIG sata_sil24 card. However, I just had another drive kicked that's connected to sata_nv, which leads me to suspect that the upgraded kernel might have something to do with it. A quick googling seems to indicate that others are seeing this with 2.6.18, too, so I was wondering if anyone here knows more.

I did at the same time also install an Areca ARC1260 controller and connected a bunch of drives to it, so another idea I had was cable interference or something (there are now 18 drives in the machine).

Any ideas or thought would be appreciated,

/Patrik

#2 lore raw

Re: Frequent SATA errors / port timeouts in 2.6.18.3?

Quoted 8 lines · Patrik Jonsson

Hi all,
this may not be the best list for this question, but I figure that the number of disks connected to users here should be pretty big...

I upgraded from 2.6.17-rc4 to 2.6.18.3 about a week ago, and I've since had 3 drives kicked out of my 10-drive RAID5 array. Previously, I had no kicks over almost a year. The kernel message is:

Quoted 6 lines

ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 ata7.00: (BMDMA stat 0x20)
ata7.00: tag 0 cmd 0xc8 Emask 0x1 stat 0x41 err 0x4 (device error)
ata7: EH complete

Any ideas or thought would be appreciated,

SMART?

Read the manpage and then try running:
smartctl -data -S on /dev/...
and
smartctl -data -s on /dev/...

Then look at your smartd timing and see if it's related; possibly just do a manual smartd poll.

I've had smart/libata problems (well, no, glitches) for about 2 years now but as the irq handler occasionally says "no one cared" ;)

It may well not be your problem but...

David

#3 lore raw

Frequent SATA errors / port timeouts in 2.6.18.3?

In reply to [email protected]

Hi all,

Hopefully someone here will know what's up with my machine. It's an nforce4 ultra box that's running a 10-drive RAID5 array. I upgraded from 2.6.17-rc4 to 2.6.18.3 about a week ago, and I've since had 3 drives kicked out. Previously, I had no kicks over almost a year. The kernel message is:

ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 ata7.00: (BMDMA stat 0x20)
ata7.00: tag 0 cmd 0xc8 Emask 0x1 stat 0x41 err 0x4 (device error)
ata7: EH complete
SCSI device sdc: 488397168 512-byte hdwr sectors (250059 MB)
sdc: Write Protect is off
sdc: Mode Sense: 00 3a 00 00
SCSI device sdc: drive cache: write back
ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x2 frozen ata7.00: (BMDMA stat 0x20)
ata7.00: tag 0 cmd 0xca Emask 0x4 stat 0x40 err 0x0 (timeout)
ata7: port is slow to respond, please be patient
ata7: port failed to respond (30 secs)
ata7: soft resetting port
ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ata7.00: failed to IDENTIFY (I/O error, err_mask=0x2) ata7.00: revalidation failed (errno=-5)
ata7: failed to recover some devices, retrying in 5 secs
ata7: hard resetting port
ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ata7.00: failed to IDENTIFY (I/O error, err_mask=0x2) ata7.00: revalidation failed (errno=-5)
ata7: failed to recover some devices, retrying in 5 secs
ata7: hard resetting port
ata7: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ata7.00: failed to IDENTIFY (I/O error, err_mask=0x2) ata7.00: revalidation failed (errno=-5)
ata7.00: disabled
ata7: EH complete

First I thought it was a cabling or card issue, because the same drive got kicked twice. That drive was connected to a 2-port SIG sata_sil24 card. However, I just had another drive kicked that's connected to the onboard sata_nv, which leads me to suspect that the upgraded kernel might have something to do with it. A quick googling seems to indicate that others are seeing this with 2.6.18, too, so I was wondering if anyone knows more. The drives contain science data for analysis, so it would be a pain (though not a disaster) to lose it. Would it be advisable to revert to the previous 2.6.17 that I was running before or is this a problem that's fixed in a later kernel than the one I'm running now?

I did at the same time also install an Areca ARC1260 controller and connected a bunch of drives to it, so another idea I had was cable interference or something (there are now 18 drives in the machine).

Any ideas or thought would be appreciated,

/Patrik

#4 lore raw

Re: Frequent SATA errors / port timeouts in 2.6.18.3?

Quoted 7 lines · Patrik Jonsson

First I thought it was a cabling or card issue, because the same drive got kicked twice. That drive was connected to a 2-port SIG sata_sil24 card. However, I just had another drive kicked that's connected to the onboard sata_nv, which leads me to suspect that the upgraded kernel might have something to do with it. A quick googling seems to indicate that others are seeing this with 2.6.18, too, so I was wondering if anyone knows more.

I don't think there has been any change which can affect both sata_sil24 and sata_nv to cause timeouts. Please post the result of 'smartctl -d ata -a /dev/sdX' of failed devices.

Quoted 11 lines

The drives contain science data for analysis, so it
would be a pain (though not a disaster) to lose it. Would it be advisable to revert to the previous 2.6.17 that I was running before or is this a problem that's fixed in a later kernel than the one I'm running now?

I did at the same time also install an Areca ARC1260 controller and connected a bunch of drives to it, so another idea I had was cable interference or something (there are now 18 drives in the machine).

Any ideas or thought would be appreciated,

Power quality degradation can cause transmission failures which can result in timeouts. How are your power lines hooked?

Signature · 1 line
tejun
#5 lore raw

Re: Frequent SATA errors / port timeouts in 2.6.18.3?

Quoted 9 lines · Tejun Heo
Quoted 7 lines · Patrik Jonsson

First I thought it was a cabling or card issue, because the same drive got kicked twice. That drive was connected to a 2-port SIG sata_sil24 card. However, I just had another drive kicked that's connected to the onboard sata_nv, which leads me to suspect that the upgraded kernel might have something to do with it. A quick googling seems to indicate that others are seeing this with 2.6.18, too, so I was wondering if anyone knows more.

Quoted 3 lines

I don't think there has been any change which can affect both sata_sil24 and sata_nv to cause timeouts. Please post the result of 'smartctl -d ata -a /dev/sdX' of failed devices.

At the time it happened, none of the devices responded to smart commands ("not an ATA device"). After resetting, they do. The outputs are at the bottom.

Quoted 7 lines
Quoted 7 lines

The drives contain science data for analysis, so it
would be a pain (though not a disaster) to lose it. Would it be advisable to revert to the previous 2.6.17 that I was running before or is this a problem that's fixed in a later kernel than the one I'm running now?

Quoted 8 lines
Quoted 5 lines

I did at the same time also install an Areca ARC1260 controller and connected a bunch of drives to it, so another idea I had was cable interference or something (there are now 18 drives in the machine).

Any ideas or thought would be appreciated,

Power quality degradation can cause transmission failures which can result in timeouts. How are your power lines hooked?

The drives are connected to the Seasonic M12-600 PSU, connected to an APC BackUPS and then it's in the wall. It's in a dedicated computer room, but I'm not sure if the power in conditioned in any way.

cheers,

/Patrik

**** Drive 1 SMART output ****

=== START OF INFORMATION SECTION ===

Device Model:     WDC WD2500SD-01KCC0
Serial Number:    WD-WCAL74587210
Firmware Version: 08.02D08
User Capacity:    250,059,350,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   6
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Tue Dec 19 16:25:01 2006 PST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x05) Offline data collection activity
was aborted by an interrupting
command from host.
Auto Offline Data Collection:
Disabled.
Self-test execution status: ( 0) The previous self-test routine
completed
without error or no self-test
has ever
been run.
Total time to complete Offline
data collection: (6942) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection
on/off support.
Suspend Offline collection upon new command.
Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode. Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
No General Purpose Logging support.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 88) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.

SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 200 159 051 Pre-fail Always
- 0
3 Spin_Up_Time 0x0007 123 121 021 Pre-fail Always
- 6375
4 Start_Stop_Count 0x0032 100 100 040 Old_age Always
- 172
5 Reallocated_Sector_Ct 0x0033 183 183 140 Pre-fail Always
- 271
7 Seek_Error_Rate 0x000b 200 200 051 Pre-fail Always
- 0
9 Power_On_Hours 0x0032 083 083 000 Old_age Always
- 12742
10 Spin_Retry_Count 0x0013 100 100 051 Pre-fail Always
- 0
11 Calibration_Retry_Count 0x0013 100 100 051 Pre-fail Always
- 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always
- 171
194 Temperature_Celsius 0x0022 125 103 000 Old_age Always
- 25
196 Reallocated_Event_Count 0x0032 161 161 000 Old_age Always
- 39
197 Current_Pending_Sector 0x0012 200 200 000 Old_age Always
- 1
198 Offline_Uncorrectable 0x0012 200 200 000 Old_age Always
- 0
199 UDMA_CRC_Error_Count 0x000a 200 253 000 Old_age Always
- 0
200 Multi_Zone_Error_Rate 0x0009 200 200 051 Pre-fail
Offline - 0

SMART Error Log Version: 1
ATA Error Count: 899 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 899 occurred at disk power-on lifetime: 12737 hours (530 days + 17 hours)
When the command that caused the error occurred, the device was doing
SMART Offline or Self-test.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 2a c3 90 e5 Error: UNC 8 sectors at LBA = 0x0590c32a = 93373226

Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 28 c3 90 05 00   5d+19:30:10.900  READ DMA
  ec 00 08 2a c3 90 00 00   5d+19:30:10.900  IDENTIFY DEVICE
  c8 00 08 28 c3 90 05 00   5d+19:30:10.900  READ DMA
  ec 00 08 2a c3 90 00 00   5d+19:30:10.900  IDENTIFY DEVICE
  c8 00 08 28 c3 90 05 00   5d+19:30:10.900  READ DMA

Error 898 occurred at disk power-on lifetime: 12737 hours (530 days + 17 hours)
When the command that caused the error occurred, the device was doing
SMART Offline or Self-test.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 2a c3 90 e5 Error: UNC 8 sectors at LBA = 0x0590c32a = 93373226

Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 28 c3 90 05 00   5d+19:30:08.850  READ DMA
  ec 00 08 2a c3 90 00 00   5d+19:30:08.850  IDENTIFY DEVICE
  c8 00 08 28 c3 90 05 00   5d+19:30:08.850  READ DMA
  ec 00 08 2a c3 90 00 00   5d+19:30:08.850  IDENTIFY DEVICE
  c8 00 08 28 c3 90 05 00   5d+19:30:08.850  READ DMA

Error 897 occurred at disk power-on lifetime: 12737 hours (530 days + 17 hours)
When the command that caused the error occurred, the device was doing
SMART Offline or Self-test.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 2a c3 90 e5 Error: UNC 8 sectors at LBA = 0x0590c32a = 93373226

Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 28 c3 90 05 00   5d+19:30:06.800  READ DMA
  ec 00 08 2a c3 90 00 00   5d+19:30:06.800  IDENTIFY DEVICE
  c8 00 08 28 c3 90 05 00   5d+19:30:06.800  READ DMA
  ec 00 08 2a c3 90 00 00   5d+19:30:06.800  IDENTIFY DEVICE
  c8 00 08 28 c3 90 05 00   5d+19:30:06.800  READ DMA

Error 896 occurred at disk power-on lifetime: 12737 hours (530 days + 17 hours)
When the command that caused the error occurred, the device was doing
SMART Offline or Self-test.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 2a c3 90 e5 Error: UNC 8 sectors at LBA = 0x0590c32a = 93373226

Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 28 c3 90 05 00   5d+19:30:04.650  READ DMA
  ec 00 08 2a c3 90 00 00   5d+19:30:04.650  IDENTIFY DEVICE
  c8 00 08 28 c3 90 05 00   5d+19:30:04.650  READ DMA
  ec 00 08 2a c3 90 00 00   5d+19:30:04.650  IDENTIFY DEVICE
  c8 00 08 28 c3 90 05 00   5d+19:30:04.650  READ DMA

Error 895 occurred at disk power-on lifetime: 12737 hours (530 days + 17 hours)
When the command that caused the error occurred, the device was doing
SMART Offline or Self-test.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 08 2a c3 90 e5 Error: UNC 8 sectors at LBA = 0x0590c32a = 93373226

Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 28 c3 90 05 00   5d+19:30:02.550  READ DMA
  ec 00 08 2a c3 90 00 00   5d+19:30:02.550  IDENTIFY DEVICE
  c8 00 08 28 c3 90 05 00   5d+19:30:02.550  READ DMA
  ec 00 08 2a c3 90 00 00   5d+19:30:02.550  IDENTIFY DEVICE
  c8 00 08 28 c3 90 05 00   5d+19:30:02.550  READ DMA

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 12742

     -
# 2  Extended offline    Completed: read failure       70%     12738
     110374951
# 3  Short offline       Completed: read failure       10%     12729
     93373219
# 4  Short offline       Completed: read failure       10%     12705
     93373219
# 5  Extended offline    Completed: read failure       70%     12681
     93373219
# 6  Short offline       Completed: read failure       10%     12657
     93373219
# 7  Short offline       Completed: read failure       10%     12633
     93373219
# 8  Short offline       Completed: read failure       10%     12609
     93373219
# 9  Extended offline    Completed: read failure       70%     12598
     93373219
#10  Short offline       Completed without error       00%     12587
     -
#11  Short offline       Completed without error       00%     12563
     -
#12  Short offline       Completed without error       00%     12543
     -
#13  Short offline       Completed without error       00%     12519
     -
#14  Extended offline    Completed without error       00%     12396
     -
#15  Short offline       Completed without error       00%     12342
     -
#16  Short offline       Completed without error       00%     12318
     -
#17  Short offline       Completed without error       00%     12294
     -
#18  Extended offline    Completed without error       00%     12272
     -
#19  Short offline       Completed without error       00%     12247
     -
#20  Short offline       Completed without error       00%     12223
     -
#21  Short offline       Completed without error       00%     12202
     -

**** Drive 2 SMART output ****

=== START OF INFORMATION SECTION ===

Device Model:     WDC WD2500SD-01KCC0
Serial Number:    WD-WCAL74586965
Firmware Version: 08.02D08
User Capacity:    250,059,350,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   6
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Tue Dec 19 16:27:31 2006 PST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x02) Offline data collection activity
was completed without error. Auto Offline Data Collection:
Disabled.
Self-test execution status: ( 0) The previous self-test routine
completed
without error or no self-test
has ever
been run.
Total time to complete Offline
data collection: (7275) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection
on/off support.
Suspend Offline collection upon new command.
Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode. Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
No General Purpose Logging support.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 91) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.

SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 200 158 051 Pre-fail Always
- 0
3 Spin_Up_Time 0x0007 146 126 021 Pre-fail Always
- 5225
4 Start_Stop_Count 0x0032 100 100 040 Old_age Always
- 94
5 Reallocated_Sector_Ct 0x0033 183 183 140 Pre-fail Always
- 258
7 Seek_Error_Rate 0x000b 200 200 051 Pre-fail Always
- 0
9 Power_On_Hours 0x0032 094 094 000 Old_age Always
- 4878
10 Spin_Retry_Count 0x0013 100 253 051 Pre-fail Always
- 0
11 Calibration_Retry_Count 0x0013 100 253 051 Pre-fail Always
- 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always
- 93
194 Temperature_Celsius 0x0022 123 101 000 Old_age Always
- 27
196 Reallocated_Event_Count 0x0032 175 175 000 Old_age Always
- 25
197 Current_Pending_Sector 0x0012 200 200 000 Old_age Always
- 0
198 Offline_Uncorrectable 0x0012 200 200 000 Old_age Always
- 0
199 UDMA_CRC_Error_Count 0x000a 200 253 000 Old_age Always
- 0
200 Multi_Zone_Error_Rate 0x0009 200 200 051 Pre-fail
Offline - 0

SMART Error Log Version: 1
ATA Error Count: 246 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 246 occurred at disk power-on lifetime: 2573 hours (107 days + 5 hours)
When the command that caused the error occurred, the device was active
or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 28 15 bd 69 e1 Error: UNC 40 sectors at LBA = 0x0169bd15 = 23706901

Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 28 0f bd 69 01 00  30d+01:23:50.954  READ DMA
  c8 00 28 0f bd 69 01 00  30d+01:23:50.954  READ DMA
  c8 00 28 0f bd 69 01 00  30d+01:23:50.954  READ DMA
  c8 00 28 0f bd 69 01 00  30d+01:23:50.954  READ DMA
  c8 00 28 0f bd 69 01 00  30d+01:23:50.954  READ DMA

Error 245 occurred at disk power-on lifetime: 2573 hours (107 days + 5 hours)
When the command that caused the error occurred, the device was active
or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 28 15 bd 69 e1 Error: UNC 40 sectors at LBA = 0x0169bd15 = 23706901

Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 28 0f bd 69 01 00  30d+01:23:48.904  READ DMA
  c8 00 28 0f bd 69 01 00  30d+01:23:48.904  READ DMA
  c8 00 28 0f bd 69 01 00  30d+01:23:48.904  READ DMA
  c8 00 28 0f bd 69 01 00  30d+01:23:48.904  READ DMA
  c8 00 28 0f bd 69 01 00  30d+01:23:48.904  READ DMA

Error 244 occurred at disk power-on lifetime: 2573 hours (107 days + 5 hours)
When the command that caused the error occurred, the device was active
or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 28 15 bd 69 e1 Error: UNC 40 sectors at LBA = 0x0169bd15 = 23706901

Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 28 0f bd 69 01 00  30d+01:23:46.854  READ DMA
  c8 00 28 0f bd 69 01 00  30d+01:23:46.854  READ DMA
  c8 00 28 0f bd 69 01 00  30d+01:23:46.854  READ DMA
  c8 00 28 0f bd 69 01 00  30d+01:23:46.854  READ DMA
  c8 00 30 07 bd 69 01 00  30d+01:23:46.854  READ DMA

Error 243 occurred at disk power-on lifetime: 2573 hours (107 days + 5 hours)
When the command that caused the error occurred, the device was active
or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 28 15 bd 69 e1 Error: UNC 40 sectors at LBA = 0x0169bd15 = 23706901

Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 28 0f bd 69 01 00  30d+01:23:44.754  READ DMA
  c8 00 28 0f bd 69 01 00  30d+01:23:44.754  READ DMA
  c8 00 28 0f bd 69 01 00  30d+01:23:44.754  READ DMA
  c8 00 30 07 bd 69 01 00  30d+01:23:44.754  READ DMA
  c8 00 30 07 bd 69 01 00  30d+01:23:44.754  READ DMA

Error 242 occurred at disk power-on lifetime: 2573 hours (107 days + 5 hours)
When the command that caused the error occurred, the device was active
or idle.

After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 28 15 bd 69 e1 Error: UNC 40 sectors at LBA = 0x0169bd15 = 23706901

Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 28 0f bd 69 01 00  30d+01:23:42.654  READ DMA
  c8 00 28 0f bd 69 01 00  30d+01:23:42.654  READ DMA
  c8 00 30 07 bd 69 01 00  30d+01:23:42.654  READ DMA
  c8 00 30 07 bd 69 01 00  30d+01:23:42.654  READ DMA
  c8 00 30 07 bd 69 01 00  30d+01:23:42.654  READ DMA

SMART Self-test log structure revision number 1
Num Test_Description Status Remaining
LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 4865

     -
# 2  Short offline       Completed without error       00%      4841
     -
# 3  Extended offline    Completed without error       00%      4818
     -
# 4  Short offline       Completed without error       00%      4793
     -
# 5  Short offline       Completed without error       00%      4769
     -
# 6  Short offline       Completed without error       00%      4745
     -
# 7  Short offline       Completed without error       00%      4722
     -
# 8  Short offline       Completed without error       00%      4698
     -
# 9  Short offline       Completed: read failure       10%      4687
     254963387
#10  Short offline       Completed without error       00%      4678
     -
#11  Short offline       Completed without error       00%      4654
     -
#12  Extended offline    Completed without error       00%      4532
     -
#13  Short offline       Completed without error       00%      4478
     -
#14  Short offline       Completed without error       00%      4454
     -
#15  Short offline       Completed without error       00%      4430
     -
#16  Extended offline    Completed without error       00%      4408
     -
#17  Short offline       Completed without error       00%      4382
     -
#18  Short offline       Completed without error       00%      4358
     -
#19  Short offline       Completed without error       00%      4338
     -
#20  Short offline       Completed without error       00%      4314
     -
#21  Short offline       Completed without error       00%      4290
     -
#6 lore raw

Re: Frequent SATA errors / port timeouts in 2.6.18.3?

Quoted 3 lines · Patrik Jonsson

At the time it happened, none of the devices responded to smart commands ("not an ATA device"). After resetting, they do. The outputs are at the bottom.

Hmmm... Nothing indicative of power problem. The first drive seems to have suffered quite some number of read failures due to bad sectors tho.

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

The drives contain science data for analysis, so it
would be a pain (though not a disaster) to lose it. Would it be advisable to revert to the previous 2.6.17 that I was running before or is this a problem that's fixed in a later kernel than the one I'm running now?

I did at the same time also install an Areca ARC1260 controller and connected a bunch of drives to it, so another idea I had was cable interference or something (there are now 18 drives in the machine).

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

Any ideas or thought would be appreciated,

Power quality degradation can cause transmission failures which can result in timeouts. How are your power lines hooked?

The drives are connected to the Seasonic M12-600 PSU, connected to an APC BackUPS and then it's in the wall. It's in a dedicated computer room, but I'm not sure if the power in conditioned in any way.

My primary suspect is still power. Please do one of the followings.

* Buy an extra power and hang 2/3 of drives there. Just cheap 350w one should do. Using PS without mainboard attached is easy, google it. In my experience, single-lane 300 ~ 350w PS is better than 450w multilane one for this purpose. More juice on molex/SATA 12v.

* If that's not an option, disconnect the extra drives and see how the system behaves.

Signature · 1 line
tejun