Recent drive errors
Hi,
I have this one drive that dropped out of one of my arrays once. It shows UNC errors in SMART (log appended), and Reported_Uncorrect is 5. There are no smart test failures or any other SMART values that look spectacularly wrong, other than maybe Load_Cycle_Count which is 10625 (these seagates used to constantly park and unpark before i updated the firmware).
I'm wondering whether or not this drive is still safe to use. I feel like I can't trust it, especially after all the other Seagates I had that failed in the past few years. I'm running a tool called whdd on it right now and it shows very consistent latency spikes above 150ms. Really, I'm wondering if this drive is RMAable as is, or if i have to wait for it to degrade further as i have another drive with like 10k reallocated sectors to send in. I have already replaced both with WD Red's so I can do whatever tests are needed to figure it out.
Thanks for any help,
SMART log:
# smartctl -a /dev/sdf smartctl 6.4 2014-10-07 r4002 [x86_64-linux-4.0.0-1-amd64] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST3000DM001-9YN166 Serial Number: W1F2G312 LU WWN Device Id: 5 000c50 060014689 Firmware Version: CC4H User Capacity: 3,000,592,982,016 bytes [3.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Tue May 19 04:42:33 2015 MDT SMART support is: Available - device has SMART capability. SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Status not supported: Incomplete response, ATA output registers missing SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error. Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever been run.
Total time to complete Offline
data collection: ( 592) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support. Suspend Offline collection upon new command.
Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 345) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes. SCT capabilities: (0x3085) SCT Status supported.
SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 115 099 006 Pre-fail Always - 100468288 3 Spin_Up_Time 0x0003 092 091 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 526 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 069 060 030 Pre-fail Always - 9795200 9 Power_On_Hours 0x0032 094 094 000 Old_age Always - 5590 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 36 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 095 095 000 Old_age Always - 5 188 Command_Timeout 0x0032 100 099 000 Old_age Always - 0 0 1 189 High_Fly_Writes 0x003a 099 099 000 Old_age Always - 1 190 Airflow_Temperature_Cel 0x0022 064 054 045 Old_age Always - 36 (Min/Max 35/36) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 524 193 Load_Cycle_Count 0x0032 095 095 000 Old_age Always - 10625 194 Temperature_Celsius 0x0022 036 046 000 Old_age Always - 36 (0 18 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 5572h+04m+02.074s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 44489827126891 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 22877192047256
SMART Error Log Version: 1
ATA Error Count: 5
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 5 occurred at disk power-on lifetime: 5309 hours (221 days + 5 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 c0 ff ff ff 4f 00 48d+03:55:47.791 READ FPDMA QUEUED 61 00 80 ff ff ff 4f 00 48d+03:55:47.791 WRITE FPDMA QUEUED e5 00 00 00 00 00 00 00 48d+03:55:47.784 CHECK POWER MODE 60 00 08 08 00 00 40 00 48d+03:55:47.755 READ FPDMA QUEUED 60 00 08 00 00 00 40 00 48d+03:55:47.751 READ FPDMA QUEUED
Error 4 occurred at disk power-on lifetime: 5309 hours (221 days + 5 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 00 ff ff ff 4f 00 48d+03:55:44.672 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 48d+03:55:44.672 READ FPDMA QUEUED 60 00 c0 ff ff ff 4f 00 48d+03:55:44.672 READ FPDMA QUEUED 60 00 08 ff ff ff 4f 00 48d+03:55:44.672 READ FPDMA QUEUED 60 00 08 00 00 00 40 00 48d+03:55:44.672 READ FPDMA QUEUED
Error 3 occurred at disk power-on lifetime: 5309 hours (221 days + 5 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 00 ff ff ff 4f 00 48d+03:55:41.802 READ FPDMA QUEUED 60 00 08 ff ff ff 4f 00 48d+03:55:41.802 READ FPDMA QUEUED 60 00 c0 ff ff ff 4f 00 48d+03:55:41.802 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 48d+03:55:41.802 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 48d+03:55:41.801 READ FPDMA QUEUED
Error 2 occurred at disk power-on lifetime: 5309 hours (221 days + 5 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 08 ff ff ff 4f 00 48d+03:55:38.809 READ FPDMA QUEUED 60 00 08 ff ff ff 4f 00 48d+03:55:38.793 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 48d+03:55:38.793 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 48d+03:55:38.792 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 48d+03:55:38.792 READ FPDMA QUEUED
Error 1 occurred at disk power-on lifetime: 5309 hours (221 days + 5 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 60 00 00 ff ff ff 4f 00 48d+03:55:35.636 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 48d+03:55:35.636 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 48d+03:55:35.636 READ FPDMA QUEUED 60 00 40 ff ff ff 4f 00 48d+03:55:35.636 READ FPDMA QUEUED 60 00 00 ff ff ff 4f 00 48d+03:55:35.636 READ FPDMA QUEUED
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 5574 - # 2 Extended offline Completed without error 00% 5571 - # 3 Extended offline Completed without error 00% 5561 - # 4 Short offline Completed without error 00% 5556 - # 5 Short offline Completed without error 00% 5553 - # 6 Short offline Completed without error 00% 5529 - # 7 Short offline Completed without error 00% 5505 - # 8 Short offline Completed without error 00% 5481 - # 9 Short offline Completed without error 00% 5457 - #10 Short offline Completed without error 00% 5433 - #11 Short offline Completed without error 00% 5409 - #12 Short offline Completed without error 00% 5385 - #13 Short offline Completed without error 00% 5361 - #14 Short offline Completed without error 00% 5337 - #15 Short offline Completed without error 00% 5313 - #16 Short offline Completed without error 00% 5289 - #17 Short offline Completed without error 00% 5265 - #18 Short offline Completed without error 00% 5241 - #19 Short offline Completed without error 00% 5217 - #20 Short offline Completed without error 00% 5193 - #21 Short offline Completed without error 00% 5169 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testingSelective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Signature · 2 lines
Thomas Fjellstrom [email protected]
Hi Thomas,
Quoted 8 lines · Thomas Fjellstrom
Hi,
I have this one drive that dropped out of one of my arrays once. It shows UNC errors in SMART (log appended), and
Reported_Uncorrectis 5. There are no smart test failures or any other SMART values that look spectacularly wrong, other than maybeLoad_Cycle_Countwhich is 10625 (these seagates used to constantly park and unpark before i updated the firmware).
Quoted 8 lines
I'm wondering whether or not this drive is still safe to use. I feel like I can't trust it, especially after all the other Seagates I had that failed in the past few years. I'm running a tool called whdd on it right now and it shows very consistent latency spikes above 150ms. Really, I'm wondering if this drive is RMAable as is, or if i have to wait for it to degrade further as i have another drive with like 10k reallocated sectors to send in. I have already replaced both with WD Red's so I can do whatever tests are needed to figure it out.
Based on the smart report, this drive is perfectly healthy. A small number of uncorrectable read errors is normal in the life of any drive.
It has no relocations, and no pending sectors. The latency spikes are
likely due to slow degradation of some sectors that the drive is having to internally retry to read successfully. Again, normal.
I own some "DM001" drives -- they are unsuited to raid duty as they don't support ERC. So, out of the box, they are time bombs for any array you put them in. That's almost certainly why they were ejected from your array.
If you absolutely must use them, you *must* set the *driver* timeout to 120 seconds or more.
HTH,
Phil
http://marc.info/?l=linux-raid&m=133761065622164&w=2
http://marc.info/?l=linux-raid&m=135811522817345&w=1
http://marc.info/?l=linux-raid&m=133761065622164&w=2
http://marc.info/?l=linux-raid&m=133665797115876&w=2
Quoted 22 lines · Phil Turmel
Hi Thomas,
Quoted 9 lines · Thomas Fjellstrom
Hi,
I have this one drive that dropped out of one of my arrays once. It shows UNC errors in SMART (log appended), and
Reported_Uncorrectis 5. There are no smart test failures or any other SMART values that look spectacularly wrong, other than maybeLoad_Cycle_Countwhich is 10625 (these seagates used to constantly park and unpark before i updated the firmware).Quoted 9 lines
I'm wondering whether or not this drive is still safe to use. I feel like I
can't trust it, especially after all the other Seagates I had that failed in the past few years. I'm running a tool called whdd on it right now and it shows very consistent latency spikes above 150ms. Really, I'm wondering if this drive is RMAable as is, or if i have to wait for it to degrade further as i have another drive with like 10k reallocated sectors to send in. I have already replaced both with WD Red's so I can do whatever tests are needed to figure it out.
Quoted 2 lines
Based on the smart report, this drive is perfectly healthy. A small number of uncorrectable read errors is normal in the life of any drive.
Is it perfectly normal for the same sector to be reported uncorrectable 5 times in a row like it did?
How many UREs are considered "ok"? Tens, hundreds, thousands, tens of thousands?
These drives have been barely used. Most of their life, they were either off, or not actually being used. (it took a while to collect enough 3TB drives, and then find time to build the array, and set it up as a regular backup of my 11TB nas).
Quoted 3 lines
It has no relocations, and no pending sectors. The latency spikes are
likely due to slow degradation of some sectors that the drive is having to internally retry to read successfully. Again, normal.
The latency spikes are /very/ regular and theres quite a lot of them.
See: http://i.imgur.com/QjTl6o3.png
Quoted 7 lines
I own some "DM001" drives -- they are unsuited to raid duty as they don't support ERC. So, out of the box, they are time bombs for any array you put them in. That's almost certainly why they were ejected from your array.
If you absolutely must use them, you *must* set the *driver* timeout to 120 seconds or more.
I've been planning on looking into the ERC stuff. I now actually have some drives that do support ERC, so it'll be interesting to make sure everything is set up properly.
Quoted 1 line
HTH,
Thank you :)
Quoted 12 lines
Phil
http://marc.info/?l=linux-raid&m=133761065622164&w=2
http://marc.info/?l=linux-raid&m=135811522817345&w=1
http://marc.info/?l=linux-raid&m=133761065622164&w=2
http://marc.info/?l=linux-raid&m=133665797115876&w=2--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 7 lines · Thomas Fjellstrom
Quoted 2 lines · Phil Turmel
Based on the smart report, this drive is perfectly healthy. A small number of uncorrectable read errors is normal in the life of any drive.
Is it perfectly normal for the same sector to be reported uncorrectable 5 times in a row like it did?
Yes, if you keep trying to read it. Unreadable sectors stay unreadable, generally, until they are re-written. That's the first opportunity the drive has to decide if a relocation is necessary.
Quoted 2 lines
How many UREs are considered "ok"? Tens, hundreds, thousands, tens of thousands?
Depends. In a properly functioning array that gets scrubbed occasionally, or sufficiently heavy use to read the entire contents occasionally, the UREs get rewritten by MD right away. Any UREs then only show up once.
In a desktop environment, or non-raid, or improperly configured raid, the UREs will build up, and get reported on every read attempt.
Most consumer-grade drives claim a URE average below 1 per 1E14 bits read. So by the end of their warranty period, getting one every 12TB read wouldn't be unusual. This sort of thing follows a Poisson distribution:
http://marc.info/?l=linux-raid&m=135863964624202&w=2
Quoted 4 lines
These drives have been barely used. Most of their life, they were either off, or not actually being used. (it took a while to collect enough 3TB drives, and then find time to build the array, and set it up as a regular backup of my 11TB nas).
While being off may lengthen their life somewhat, the magnetic domains on these things are so small that some degradation will happen just sitting there. Diffusion in the p- and n-doped regions of the semiconductors is also happening while sitting unused, degrading the electronics.
Quoted 6 lines
Quoted 3 lines
It has no relocations, and no pending sectors. The latency spikes are
likely due to slow degradation of some sectors that the drive is having to internally retry to read successfully. Again, normal.The latency spikes are /very/ regular and theres quite a lot of them.
See: http://i.imgur.com/QjTl6o3.png
Interesting. I suspect that if you wipe that disk with noise, read it all back, and wipe it again, you'll have a handful of relocations.
Your latency test will show different numbers then, as the head will have to seek to the spare sector and back whenever you read through one of those spots.
Or the rewrites will fix them all, and you'll have no further problems.
Hard to tell. Bottom line is that drives can't fix any problems they
have unless they are *written* in previously identified problem areas.
Quoted 11 lines
Quoted 7 lines
I own some "DM001" drives -- they are unsuited to raid duty as they don't support ERC. So, out of the box, they are time bombs for any array you put them in. That's almost certainly why they were ejected from your array.
If you absolutely must use them, you *must* set the *driver* timeout to 120 seconds or more.
I've been planning on looking into the ERC stuff. I now actually have some drives that do support ERC, so it'll be interesting to make sure everything is set up properly.
You have it backwards. If you have WD Reds, they are correct out of the box. It's when you *don't* have ERC support, or you only have desktop ERC, that you need to take special action.
If you have consumer grade drives in a raid array, and you don't have boot scripts or udev rules to deal with timeout mismatch, your *ss is hanging in the wind. The links in my last msg should help you out.
Also, I noticed that you used "smartctl -a" to post a complete report of your drive's status. It's not complete. You should get in the habit of using "smartctl -x" instead, so you see the ERC status, too.
Phil
Quoted 12 lines · Phil Turmel
Quoted 6 lines · Thomas Fjellstrom
Quoted 2 lines · Phil Turmel
Based on the smart report, this drive is perfectly healthy. A small number of uncorrectable read errors is normal in the life of any drive.
Is it perfectly normal for the same sector to be reported uncorrectable 5 times in a row like it did?
Yes, if you keep trying to read it. Unreadable sectors stay unreadable, generally, until they are re-written. That's the first opportunity the drive has to decide if a relocation is necessary.
Quoted 7 lines
Quoted 2 lines
How many UREs are considered "ok"? Tens, hundreds, thousands, tens of thousands?
Depends. In a properly functioning array that gets scrubbed occasionally, or sufficiently heavy use to read the entire contents occasionally, the UREs get rewritten by MD right away. Any UREs then only show up once.
I have made sure that it's doing regular scrubs, and regular SMART scans. This time...
Quoted 23 lines
In a desktop environment, or non-raid, or improperly configured raid, the UREs will build up, and get reported on every read attempt.
Most consumer-grade drives claim a URE average below 1 per 1E14 bits read. So by the end of their warranty period, getting one every 12TB read wouldn't be unusual. This sort of thing follows a Poisson distribution:
http://marc.info/?l=linux-raid&m=135863964624202&w=2
Quoted 4 lines
These drives have been barely used. Most of their life, they were either off, or not actually being used. (it took a while to collect enough 3TB drives, and then find time to build the array, and set it up as a regular backup of my 11TB nas).
While being off may lengthen their life somewhat, the magnetic domains on these things are so small that some degradation will happen just sitting there. Diffusion in the p- and n-doped regions of the semiconductors is also happening while sitting unused, degrading the electronics.
Quoted 2 lines
Quoted 2 lines
It has no relocations, and no pending sectors. The latency spikes are
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
likely due to slow degradation of some sectors that the drive is having to internally retry to read successfully. Again, normal.
The latency spikes are /very/ regular and theres quite a lot of them.
See: http://i.imgur.com/QjTl6o3.pngInteresting. I suspect that if you wipe that disk with noise, read it all back, and wipe it again, you'll have a handful of relocations.
It looks like each one of the blocks in that display is 128KiB. Which i think means those red blocks aren't very far apart. Maybe 80MiB apart? Would it reallocate all of those? That'd be a lot of reallocated sectors.
Quoted 16 lines
Your latency test will show different numbers then, as the head will have to seek to the spare sector and back whenever you read through one of those spots.
Or the rewrites will fix them all, and you'll have no further problems.
Hard to tell. Bottom line is that drives can't fix any problems they
have unless they are *written* in previously identified problem areas.Quoted 8 lines
Quoted 7 lines
I own some "DM001" drives -- they are unsuited to raid duty as they don't support ERC. So, out of the box, they are time bombs for any array you put them in. That's almost certainly why they were ejected from your array.
If you absolutely must use them, you *must* set the *driver* timeout to 120 seconds or more.
Quoted 7 lines
Quoted 3 lines
I've been planning on looking into the ERC stuff. I now actually have some drives that do support ERC, so it'll be interesting to make sure everything is set up properly.
You have it backwards. If you have WD Reds, they are correct out of the box. It's when you *don't* have ERC support, or you only have desktop ERC, that you need to take special action.
I was under the impression you still had to enable ERC on boot. And I /thought/ I read that you still want to adjust the timeouts, though not the same as for consumer drives.
Quoted 3 lines
If you have consumer grade drives in a raid array, and you don't have boot scripts or udev rules to deal with timeout mismatch, your *ss is hanging in the wind. The links in my last msg should help you out.
There was some talk of ERC/TLER and md. I'll still have to find or write a script to properly set up timeouts and enable TLER on drives capable of it (that don't come with it enabled by default).
Quoted 3 lines
Also, I noticed that you used "smartctl -a" to post a complete report of your drive's status. It's not complete. You should get in the habit of using "smartctl -x" instead, so you see the ERC status, too.
Good to know. Thanks.
Quoted 5 lines
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 9 lines · Thomas Fjellstrom
Quoted 4 lines · Phil Turmel
Depends. In a properly functioning array that gets scrubbed occasionally, or sufficiently heavy use to read the entire contents occasionally, the UREs get rewritten by MD right away. Any UREs then only show up once.
I have made sure that it's doing regular scrubs, and regular SMART scans. This time...
Yes, and this drive was kicked out. Because it wouldn't be listening when MD tried to write over the error it found.
I posted this link earlier, but it is particularly relevant:
http://marc.info/?l=linux-raid&m=133665797115876&w=2
Quoted 6 lines
Quoted 2 lines
Interesting. I suspect that if you wipe that disk with noise, read it all back, and wipe it again, you'll have a handful of relocations.
It looks like each one of the blocks in that display is 128KiB. Which i think means those red blocks aren't very far apart. Maybe 80MiB apart? Would it reallocate all of those? That'd be a lot of reallocated sectors.
Drives will only reallocate where a previous read failed (making it pending), then write and follow-up verification fails. In general, writes are unverified at the time of write (or your write performance would be dramatically slower than read).
Quoted 7 lines
Quoted 3 lines
You have it backwards. If you have WD Reds, they are correct out of the box. It's when you *don't* have ERC support, or you only have desktop ERC, that you need to take special action.
I was under the impression you still had to enable ERC on boot. And I /thought/ I read that you still want to adjust the timeouts, though not the same as for consumer drives.
Desktop / consumer drives that support ERC typically ship with it disabled, so they behave just like drives that don't support it at all.
So a boot script would enable ERC on drives where it can (and not
already OK), and set long driver timeouts on the rest.
Any drive that claims "raid" compatibility will have ERC enabled by default. Typically 7.0 seconds. WD Reds do. Enterprise drives do, and have better URE specs, too.
Quoted 7 lines
Quoted 3 lines
If you have consumer grade drives in a raid array, and you don't have boot scripts or udev rules to deal with timeout mismatch, your *ss is hanging in the wind. The links in my last msg should help you out.
There was some talk of ERC/TLER and md. I'll still have to find or write a script to properly set up timeouts and enable TLER on drives capable of it (that don't come with it enabled by default).
Before I got everything onto proper drives, I just put what I needed into rc.local.
Chris Murphy posted some udev rules that will likely work for you. I haven't tried them myself, though.
https://www.marc.info/?l=linux-raid&m=142487508806844&w=3
Phil
Quoted 7 lines · you
Quoted 6 lines · Thomas Fjellstrom
Quoted 4 lines · Phil Turmel
Depends. In a properly functioning array that gets scrubbed occasionally, or sufficiently heavy use to read the entire contents occasionally, the UREs get rewritten by MD right away. Any UREs then only show up once.
Quoted 5 lines
Quoted 2 lines
I have made sure that it's doing regular scrubs, and regular SMART scans. This time...
Yes, and this drive was kicked out. Because it wouldn't be listening when MD tried to write over the error it found.
I didn't actually re-install this drive after the last time it was kicked out, which was when i didn't have regular scrubs (actually, it may have been, as it was probably the only thing that would cause activity on that array for many many months) or smart tests set up. I noticed the high start stop count, and the 5 errors, and decided to keep it out of the new array. I seem to recall one or more drives having suspiciously high start stop counts that then went on to fail, but it seems that isn't true, one of them is still in use (64k start stop events apparently, or it maxed out the counter).
Basically, I had an unused array of 3TB seagates, it sat doing virtually nothing but spinning its platters for quite a long time due to lack of time on my part, and some time between last summer and winter, it kicked out two drives all on its own. It was probably the monthly scrub. After I got back from a three month long trip, I rebuilt that array (and my main NAS, which also kicked out two drives... But thats a story for another time) with four of the old Seagates, and one new WD red. It was this drive that I removed at that time because it looked suspicious. Sadly, about a month or two later, a second drive got kicked out and was unambiguously faulty (thousands, if not 10k+ reallocated sectors), so I replaced it with two new WD Reds, and reshaped to a raid6. After that, I just decided to re-check the first drive to drop out, just to be safe, and here we are...
I'm running a badblocks -w on the drive as we speak, it'll probably be done in a day or two. We'll see if it changes anything. It's not exactly writing noise, but it aught to do the trick.
Quoted 6 lines
I posted this link earlier, but it is particularly relevant:
http://marc.info/?l=linux-raid&m=133665797115876&w=2Quoted 3 lines
Quoted 2 lines
Interesting. I suspect that if you wipe that disk with noise, read it all back, and wipe it again, you'll have a handful of relocations.
Quoted 8 lines
Quoted 3 lines
It looks like each one of the blocks in that display is 128KiB. Which i think means those red blocks aren't very far apart. Maybe 80MiB apart? Would it reallocate all of those? That'd be a lot of reallocated sectors.
Drives will only reallocate where a previous read failed (making it pending), then write and follow-up verification fails. In general, writes are unverified at the time of write (or your write performance would be dramatically slower than read).
Right. I was just thinking about how you mentioned that I'd get a handful of reallocations based on the latency shown in the image I posted. It's a lot of sectors that seem to be affected by the latency spikes, so I assumed (probably wrongly) that many of them may be reallocated afterwards.
If this drive ends up not reallocating a single sector, or only a few, I may just keep it around as a hot spare, though i feel that's not the best idea, if it is degrading, then when it actually goes to use that disk it has a higher chance of failing.
Quoted 9 lines
Quoted 8 lines
Quoted 3 lines
You have it backwards. If you have WD Reds, they are correct out of the box. It's when you *don't* have ERC support, or you only have desktop ERC, that you need to take special action.
I was under the impression you still had to enable ERC on boot. And I /thought/ I read that you still want to adjust the timeouts, though not the
same as for consumer drives.
Quoted 8 lines
Desktop / consumer drives that support ERC typically ship with it disabled, so they behave just like drives that don't support it at all.
So a boot script would enable ERC on drives where it can (and not
already OK), and set long driver timeouts on the rest.Any drive that claims "raid" compatibility will have ERC enabled by default. Typically 7.0 seconds. WD Reds do. Enterprise drives do, and have better URE specs, too.
Good to know.
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
If you have consumer grade drives in a raid array, and you don't have boot scripts or udev rules to deal with timeout mismatch, your *ss is hanging in the wind. The links in my last msg should help you out.
There was some talk of ERC/TLER and md. I'll still have to find or write a script to properly set up timeouts and enable TLER on drives capable of it (that don't come with it enabled by default).
Before I got everything onto proper drives, I just put what I needed into rc.local.
It's going to be a long time before I can swap out the rest of the seagates. I just can't justify the cost atm, especially as its the backup for my main nas which used to be all 2TB seagates, but has since been retrofitted with two WD reds as two had thousands of reallocated sectors, funny thing is one of the seagates had already been replaced prior to that, so 3 out of 5 of the original setup failed. And before that, at least two 1TB seagates have failed on me (out of 7ish), I think maybe one 640G one went, and a couple 320s went. I won't blame the two 80s that I had that failed on seagate though, that was a power supply fault. Took out two drives out of five (total), memory, and made the motherboard a bit flaky.
Just a little bit jaded when it comes to seagates these days, but i still can't just up and swap them all out, even if its a good idea. If I had the money, I wouldn't mind just replacing them all with enterprise/nearline or NAS drives, and slap the seagates in a big zfs pool or something for some scratch space or just sell them...
Quoted 4 lines
Chris Murphy posted some udev rules that will likely work for you. I haven't tried them myself, though.
Thanks :)
Quoted 5 lines
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 15 lines · Thomas Fjellstrom
Quoted 13 lines · you
Quoted 9 lines · Thomas Fjellstrom
Quoted 4 lines · Phil Turmel
Depends. In a properly functioning array that gets scrubbed occasionally, or sufficiently heavy use to read the entire contents occasionally, the UREs get rewritten by MD right away. Any UREs then only show up once.
I have made sure that it's doing regular scrubs, and regular SMART scans.
This time...Yes, and this drive was kicked out. Because it wouldn't be listening when MD tried to write over the error it found.
[snip]
Quoted 22 lines
Quoted 12 lines
I posted this link earlier, but it is particularly relevant:
http://marc.info/?l=linux-raid&m=133665797115876&w=2Quoted 7 lines
Quoted 2 lines
Interesting. I suspect that if you wipe that disk with noise, read it all back, and wipe it again, you'll have a handful of relocations.
It looks like each one of the blocks in that display is 128KiB. Which i think means those red blocks aren't very far apart. Maybe 80MiB apart? Would it reallocate all of those? That'd be a lot of reallocated sectors.
Quoted 4 lines
Drives will only reallocate where a previous read failed (making it pending), then write and follow-up verification fails. In general, writes are unverified at the time of write (or your write performance would be dramatically slower than read).
Right. I was just thinking about how you mentioned that I'd get a handful of reallocations based on the latency shown in the image I posted. It's a lot of sectors that seem to be affected by the latency spikes, so I assumed (probably wrongly) that many of them may be reallocated afterwards.
Quoted 4 lines
If this drive ends up not reallocating a single sector, or only a few, I may just keep it around as a hot spare, though i feel that's not the best idea, if it is degrading, then when it actually goes to use that disk it has a higher chance of failing.
Well here's something:
[78447.747221] sd 0:0:15:0: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [78447.749092] sd 0:0:15:0: [sdf] Sense Key : Medium Error [current] [78447.751034] sd 0:0:15:0: [sdf] Add. Sense: Unrecovered read error [78447.752925] sd 0:0:15:0: [sdf] CDB: Read(16) 88 00 00 00 00 00 ef 7a 0f b0 00 00 00 08 00 00 [78447.754746] blk_update_request: critical medium error, dev sdf, sector 4017754032 [78447.756700] Buffer I/O error on dev sdf, logical block 502219254, async page read
<many many more of the above>
5 Reallocated_Sector_Ct PO--CK 087 087 036 - 17232 187 Reported_Uncorrect -O--CK 001 001 000 - 8236 197 Current_Pending_Sector -O--C- 024 024 000 - 12584 198 Offline_Uncorrectable ----C- 024 024 000 - 12584
Badblocks is showing a bunch of errors now, and the above is what's in dmesg and smartctl.
So I guess it was dead after all.
Quoted 36 lines
Quoted 13 lines
Quoted 9 lines
Quoted 4 lines
You have it backwards. If you have WD Reds, they are correct out of the
box. It's when you *don't* have ERC support, or you only have desktop ERC, that you need to take special action.I was under the impression you still had to enable ERC on boot. And I /thought/ I read that you still want to adjust the timeouts, though not the
same as for consumer drives.Desktop / consumer drives that support ERC typically ship with it disabled, so they behave just like drives that don't support it at all.
Quoted 7 lines
So a boot script would enable ERC on drives where it can (and not
already OK), and set long driver timeouts on the rest.
Any drive that claims "raid" compatibility will have ERC enabled by default. Typically 7.0 seconds. WD Reds do. Enterprise drives do, and have better URE specs, too.
Good to know.
Quoted 12 lines
Quoted 9 lines
Quoted 3 lines
If you have consumer grade drives in a raid array, and you don't have boot scripts or udev rules to deal with timeout mismatch, your *ss is hanging in the wind. The links in my last msg should help you out.
There was some talk of ERC/TLER and md. I'll still have to find or write a
script to properly set up timeouts and enable TLER on drives capable of it
(that don't come with it enabled by default).Before I got everything onto proper drives, I just put what I needed into rc.local.
[snip]
Quoted 13 lines
Quoted 5 lines
Chris Murphy posted some udev rules that will likely work for you. I haven't tried them myself, though.
Thanks :)
Quoted 5 lines
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 1 line · Thomas Fjellstrom
How many UREs are considered "ok"? Tens, hundreds, thousands, tens of thousands?
I will replace any drive that have developed UNC sectors a few times, so I'd say "less than 10".
+1 on the "set kernel timeout to more than 120 seconds". I have this in /etc/rc.local:
for x in /sys/block/sd[a-z] ; do
echo 180 > $x/device/timeout
done
echo 4096 > /sys/block/md0/md/stripe_cache_size
Signature · 1 line
Mikael Abrahamsson email: [email protected]
Quoted 6 lines · Mikael Abrahamsson
Quoted 2 lines · Thomas Fjellstrom
How many UREs are considered "ok"? Tens, hundreds, thousands, tens of thousands?
I will replace any drive that have developed UNC sectors a few times, so I'd say "less than 10".
In this case, it looked like 5 UNC errors for a single sector, and some weird latency patterns, till I ran badblocks -w on it, then it gave me > 10k relocated sectors and many thousands more uncorrectable sectors. Before the badblocks test, it "looked" ok, now It's most definitely dead.
Quoted 8 lines
+1 on the "set kernel timeout to more than 120 seconds". I have this in
/etc/rc.local:for x in /sys/block/sd[a-z] ; do
echo 180 > $x/device/timeout
doneecho 4096 >
/sys/block/md0/md/stripe_cache_size
I presume it's ok to do that even if the drives do ERC/TLER? Just woke up, but my brain seems to be telling me it shouldn't break anything since the ERC drives should always return after 7s no matter what...
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 9 lines · Mikael Abrahamsson
Quoted 2 lines · Thomas Fjellstrom
How many UREs are considered "ok"? Tens, hundreds, thousands, tens of thousands?
I will replace any drive that have developed UNC sectors a few times, so I'd say "less than 10".
Quoted 8 lines
+1 on the "set kernel timeout to more than 120 seconds". I have this in
/etc/rc.local:for x in /sys/block/sd[a-z] ; do
echo 180 > $x/device/timeout
doneecho 4096 >
/sys/block/md0/md/stripe_cache_size
#!/bin/sh
if [ -e /proc/sys/dev/raid/speed_limit_min ]; then
echo $((1024*4)) > /proc/sys/dev/raid/speed_limit_min 2> /dev/null
echo $((1024*24)) > /proc/sys/dev/raid/speed_limit_max 2> /dev/nullfi
if [ -e /dev/md0 -o -d /dev/md ]; then
for md in /dev/md*; do
echo $((1024*4)) > /sys/block/${md#/dev/}/md/stripe_cache_size blockdev --setra $((1024*4)) /dev/${md#/dev/}
done
for disk in /sys/block/sd*; do
echo 45 > $disk/device/timeout
smartctl -l scterc,70,70 /dev/${disk#/sys/block/}
done
fi
for disk in /sys/block/sd*; do
echo cfq > $disk/queue/scheduler
echo 768 > $disk/queue/read_ahead_kb
echo 256 > $disk/queue/nr_requests echo 8 2>/dev/null > $disk/device/queue_depth
done
Quoted 7 lines · Thomas Fjellstrom
Quoted 3 lines
for x in /sys/block/sd[a-z] ; do
echo 180 > $x/device/timeout
doneI presume it's ok to do that even if the drives do ERC/TLER? Just woke up, but my brain seems to be telling me it shouldn't break anything since the ERC drives should always return after 7s no matter what...
Correct, the only downside is that if the drive really dies, it's going to take longer to detect this.
I'd rather have longer timeouts to make sure drives never kicked out because of lack of ERC or something else, than to have drives kicked for some reason (controller reset, lack of ERC, or something else).
I really really want to avoid drives being kicked for any other reason than them really being dead. I'd rather have reads stalled for a few minutes than this happening. For other use-cases, the requirements may be different.
Signature · 1 line
Mikael Abrahamsson email: [email protected]
Quoted 14 lines · Mikael Abrahamsson
Quoted 9 lines · Thomas Fjellstrom
Quoted 5 lines
for x in /sys/block/sd[a-z] ; do
echo 180 > $x/device/timeout
done
I presume it's ok to do that even if the drives do ERC/TLER? Just woke up, but my brain seems to be telling me it shouldn't break anything since the ERC drives should always return after 7s no matter what...
Correct, the only downside is that if the drive really dies, it's going to take longer to detect this.
Quoted 8 lines
I'd rather have longer timeouts to make sure drives never kicked out because of lack of ERC or something else, than to have drives kicked for some reason (controller reset, lack of ERC, or something else).
I really really want to avoid drives being kicked for any other reason than them really being dead. I'd rather have reads stalled for a few minutes than this happening. For other use-cases, the requirements may be different.
Yeah, I agree. Especially given this is just my home nas and nas backup setup, It's better to just stall rather than die horribly.
Signature · 2 lines
Thomas Fjellstrom [email protected]