Raid check didn't fix Current_Pending_Sector, but badblocks -nsv did
Howdy, I have a raid 5 where one drive reported this:
197 Current_Pending_Sector 0x0032 200 199 000 Old_age Always - 29
So I did this:
myth:~# echo check > /sys/block/md5/md/sync_action [173947.749761] md: data-check of RAID array md5 (...) [370316.769230] md: md5: data-check done.
My understanding was that it was supposed to read every block of every drive, and if some blocks were unreadable, use parity to rewrite them on some fresh backup blocks.
If a block returned garbage instead, md5 cannot fix this not knowing which block is wrong, but I'm assuming the check would have failed with an error.
However after the check is over, I still have 29 Current_Pending_Sector on that drive.
Since raid check succeeded, I'm going to assume that the sectors were readable and did not return garbage, or I'd have gotten a parity mismatch error.
Should then assume that either
1) the smart counter/logic is wrong?
2) the pending sectors started returning correct data again, so linux md has no idea those blocks are "weak" and I have no easy way to forcibly remap them.
3) the bad blocks did get remapped somehow, but the smart counter did not get reset due to a firmware bug
4) other
After 2 days of testing with badblocks, it seems that it's #2, and I'm not sure if there is anything raid check could have done (probably not)
Since raid check didn't do the job I was hoping for, I ran this instead:
myth:~# badblocks -nsv /dev/sdg Checking for bad blocks in non-destructive read-write mode From block 0 to 3907018583 Checking for bad blocks (non-destructive read-write test) Testing with random pattern: 57.87% done, 22:24:15 elapsed. (0/0/0 errors))
And this worked:
197 Current_Pending_Sector 0x0032 200 199 000 Old_age Always - 0
strangely, I also have:
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
I guess this means that my drive's auto reallocation logic is faulty and that it will not re-allocate blocks that are weak, even after it was able to read them.
Does that sound correct?
More drive details from before I ran badblocks (not an SMR drive):
Device Model: WDC WD40EFRX-68WT0N0 Serial Number: WD-WCC4E0642444 LU WWN Device Id: 5 0014ee 2b437e9a6 Firmware Version: 80.00A80 User Capacity: 4,000,787,030,016 bytes [4.00 TB]
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 1617 3 Spin_Up_Time 0x0027 175 173 021 Pre-fail Always - 8250 4 Start_Stop_Count 0x0032 094 094 000 Old_age Always - 6773 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 074 074 000 Old_age Always - 19092 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 158 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 91 193 Load_Cycle_Count 0x0032 182 182 000 Old_age Always - 54642 194 Temperature_Celsius 0x0022 121 103 000 Old_age Always - 31 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 199 000 Old_age Always - 29 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 199 000 Old_age Always - 2 200 Multi_Zone_Error_Rate 0x0008 200 189 000 Old_age Offline - 0
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 19081 - # 2 Short offline Completed without error 00% 19057 - # 3 Short offline Completed without error 00% 19035 - # 4 Short offline Completed without error 00% 18984 - # 5 Extended offline Completed without error 00% 18974 -
Thanks,
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 9 lines · Marc MERLIN
Howdy, I have a raid 5 where one drive reported this:
197Current_Pending_Sector0x0032200 199 000Old_ageAlways - 29So I did this:
myth:~# echo check > /sys/block/md5/md/sync_action [173947.749761] md: data-check of RAID array md5 (...) [370316.769230] md: md5: data-check done.
Quoted 5 lines
My understanding was that it was supposed to read every block of every drive, and if some blocks were unreadable, use parity to rewrite them on some fresh backup blocks.
If a block returned garbage instead, md5 cannot fix this not knowing which block is wrong, but I'm assuming the check would have failed with an error.
https://www.kernel.org/doc/Documentation/md.txt shows for sync_action
check - A full check of redundancy was requested and is
happening. This reads all blocks and checks them. A repair may also happen for some raid levels.
repair - A full check and repair is happening. This is
similar to 'resync', but was requested by the user, and the write-intent bitmap is NOT used to optimise the process.
I think you wanted 'repair' not 'check'.
--Sarah
Quoted 11 lines · Sarah Newman
https://www.kernel.org/doc/Documentation/md.txt shows for
sync_actioncheck - A full check of redundancy was requested and is
happening. This reads all blocks and checks them. A repair may also happen for some raid levels.
repair - A full check and repair is happening. This is
similar to 'resync', but was requested by the user, and the write-intent bitmap is NOT used to optimise the process.
Quoted 1 line
I think you wanted 'repair' not 'check'.
From what I understand, the only difference between the 2 is that repair does not use the write-intent bitmap, but both will repair an error if found.
https://www.thomas-krenn.com/en/wiki/Mdadm_checkarray#Check_vs._Repair
Or are you saying that check after getting a read error from one drive, would not rewrite the bad block on that drive? I thought it did...
Either way, it seems that neither would have worked because while those blocks were marked as "need to be reallocated" by the drive, I think the kernel was actually able to read them without problem, so the md layer never saw anything and therefore never did anything either.
Whereas badblocks forced an unconditional rewrite of all blocks, and forced the drive to re-allocate those "weak" blocks, even though it was able to read them.
Does that sound about right?
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ Quoted 3 lines · Marc MERLIN
From what I understand, the only difference between the 2 is that repair does not use the write-intent bitmap, but both will repair an error if found.
Repair doesn't even read the parity (and Q syndrome) blocks if it doesn't need them. It unconditionally recomputes and writes parity (and
Q) blocks.
Both processes will compute and rewrite a block that reports a read error.
Quoted 4 lines
https://www.thomas-krenn.com/en/wiki/Mdadm_checkarray#Check_vs._Repair
Or are you saying that check after getting a read error from one drive, would not rewrite the bad block on that drive? I thought it did...
They both do.
Quoted 4 lines
Either way, it seems that neither would have worked because while those blocks were marked as "need to be reallocated" by the drive, I think the kernel was actually able to read them without problem, so the md layer never saw anything and therefore never did anything either.
Both processes only apply to the actual data area of each member device, which is usually substantially less than the whole disk. Both processes will read or write (or both) every data area sector.
Sectors that haven't ever been deliberately written and aren't read by normal mdadm processes are often the first to pop up in disk self-diagnostics. If you look at the sector offsets in the SMART data, I bet you find they're all outside the area mdadm is using.
Magnetism does slowly decay on drive platters, faster in the spots where manufacturing didn't maintain the intended oxide thickness. Completely normal and is the basis for URE specifications.
Quoted 3 lines
Whereas badblocks forced an unconditional rewrite of all blocks, and forced the drive to re-allocate those "weak" blocks, even though it was able to read them.
Probably not. It got 'em because it wasn't limited to the data area.
Phil
Quoted 3 lines · Phil Turmel
Repair doesn't even read the parity (and Q syndrome) blocks if it doesn't need them. It unconditionally recomputes and writes parity (and
Q) blocks.
I see. So if it happens to rewrite a parity block that was unreadable but that it didn't have to read, I won't even know it was unreadable to start with (but only one change out of n of that happening in raid5)
Quoted 7 lines
Both processes will compute and rewrite a block that reports a read error.
Quoted 4 lines
https://www.thomas-krenn.com/en/wiki/Mdadm_checkarray#Check_vs._Repair
Or are you saying that check after getting a read error from one drive, would not rewrite the bad block on that drive? I thought it did...
They both do.
Thanks for confirming.
Quoted 3 lines
Both processes only apply to the actual data area of each member device, which is usually substantially less than the whole disk. Both processes will read or write (or both) every data area sector.
Mmmh, so it keeps track of how much of my block device is used by filesystem blocks and skips the rest? I didn't know that.
Quoted 4 lines
Sectors that haven't ever been deliberately written and aren't read by normal mdadm processes are often the first to pop up in disk self-diagnostics. If you look at the sector offsets in the SMART data, I bet you find they're all outside the area mdadm is using.
Got it.
Quoted 5 lines
Quoted 3 lines
Whereas badblocks forced an unconditional rewrite of all blocks, and forced the drive to re-allocate those "weak" blocks, even though it was able to read them.
Probably not. It got 'em because it wasn't limited to the data area.
Right, I understand now, good to know.
So I'll use badblocks next time I have this issue.
Thanks for clearing this up for me.
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901Quoted 8 lines · Marc MERLIN
Quoted 3 lines · Phil Turmel
Both processes only apply to the actual data area of each member device, which is usually substantially less than the whole disk. Both processes will read or write (or both) every data area sector.
Mmmh, so it keeps track of how much of my block device is used by filesystem blocks and skips the rest? I didn't know that.
No, it doesn't keep track of anything the filesystem does. Look at the mdadm -E report for one of you member devices -- it shows the location of the superblock, the start location and size of the data area, and possibly information about the optional bitmap or bad block table. Areas outside the data area are not touched by check or repair scrubs.
Quoted 2 lines
Right, I understand now, good to know.
So I'll use badblocks next time I have this issue.
Or just ignore them. You aren't using them, so they can't hurt you. However, do look at the sector numbers in the SMART reports to make sure they aren't in the data area. (If you aren't using the whole disk for mdadm, be sure to adjust for the partition start sector.) If they *are* in the data area, watch dmesg while scrubbing to see what's really happening.
Phil
Quoted 1 line · Phil Turmel
Or just ignore them. You aren't using them, so they can't hurt you. However, do look at the sector numbers in the SMART reports to make sure they aren't in the data area. (If you aren't using the whole disk for mdadm, be sure to adjust for the partition start sector.) If they *are* in the data area, watch dmesg while scrubbing to see what's really happening.
I had a similar problem a few years back.
I seem to remember that there was a problem that the pending blocks were in the superblock area, and these weren't written out when I did "check" or "repair". I might be misremembering though, but I seem to remember I had some kind of problem that when I tried to add the dive back in again, it got a read error when trying to read the superblock.
They might have also been in the offset area. What I did was that I did a --replace to another drive, then I dd:ed /dev/zero to the first gigabyte of the drive, which removed the pending blocks. I then added the drive back in.
Could someone confirm whether the entire (actually valuable portion) of the superblock area is written out at any kind of frequency, or when a check/repair is issued? If not, I think that would be a valuable thing to do?
Signature · 1 line
Mikael Abrahamsson email: [email protected]
Quoted 6 lines · Phil Turmel
Quoted 2 lines · Marc MERLIN
Right, I understand now, good to know.
So I'll use badblocks next time I have this issue.Or just ignore them. You aren't using them, so they can't hurt you.
That's actually not necessarily true.
If you have a dud sector early on the disk (so before the start of the RAID data) you will terminate every SMART long test in the first couple of meg of the disk. So while a dud down there won't necessarily impact your usage from a RAID perspective, it'll knacker your ability to regularly check the disks in their entirety. SMART tests abort on the first bad read.
It's ugly, but in the single instance I had that happen, I removed the drive from the array, wrote zero to the entire disk and then added it back. That forced a reallocation in the affected area.
Usually if it is in the RAID zone, a check scrub will clear it up. Having said that I've had a very peculiar one here in the last couple of days.
A WD 2TB Green drive with TLER set to 7 seconds. The first read would error out in 7 seconds (as it should), but a second read succeeded. After returning the error, the drive must have kept trying to recover in the background and eventually succeeded and cached the result. So subsequent reads were ok. After reading and writing enough to other parts of the drive to flush the drives cache, the process would repeat.
In this case, it took about 3 check scrubs to actually hit the read error and force a re-write.
Quoted 8 lines · Brad Campbell
Quoted 6 lines · Phil Turmel
Quoted 2 lines · Marc MERLIN
Right, I understand now, good to know.
So I'll use badblocks next time I have this issue.Or just ignore them. You aren't using them, so they can't hurt you.
Quoted 8 lines
That's actually not necessarily true.
If you have a dud sector early on the disk (so before the start of the RAID data) you will terminate every SMART long test in the first couple of meg of the disk. So while a dud down there won't necessarily impact your usage from a RAID perspective, it'll knacker your ability to regularly check the disks in their entirety. SMART tests abort on the first bad read.
Don't bother doing long self-tests on drives participating in an array -- check scrubs do everything a long self-test does on the area of interest, plus actually fixing UREs that are found. And check scrubs don't abort on a read failure.
My advice stands: ignore the UREs in unused areas of the disk.
Quoted 3 lines
It's ugly, but in the single instance I had that happen, I removed the drive from the array, wrote zero to the entire disk and then added it back. That forced a reallocation in the affected area.
Completely pointless exercise that opened a window of higher-risk of failure of your array. Unless you used --replace with another spare to maintain redundancy on your array while that disk was out.
Quoted 10 lines
Usually if it is in the RAID zone, a check scrub will clear it up. Having said that I've had a very peculiar one here in the last couple of days.
A WD 2TB Green drive with TLER set to 7 seconds. The first read would error out in 7 seconds (as it should), but a second read succeeded. After returning the error, the drive must have kept trying to recover in the background and eventually succeeded and cached the result. So subsequent reads were ok. After reading and writing enough to other parts of the drive to flush the drives cache, the process would repeat.
Pure speculation. Unless you can show better evidence that those drives will cache a read in that case, I would say it was just a mild enough weak spot that it randomly succeeded more than not. And if you follow my advice, it doesn't matter: if the array is the only process reading from the disk, the first appearance of the URE would be the last, as the array would re-write it immediately. Whether during a scrub or due to normal access.
Regular long self-tests are highly recommended for stand-alone disks and for array hot spares.
Phil
Quoted 2 lines · On Mon, 6 Jun 2016 10:41:13 -0700 Marc MERLIN <[email protected]> wrote:
However after the check is over, I still have 29
Current_Pending_Sectoron that drive.
I had that once, even asked on this list (can't find that now), and it turned out the pending sector(s) were in the gap between the start of the partition and the start of the mdadm actual data (which turned out to be surprisingly large) -- so naturally neither a check nor repair have ever tried to read or write there.
Quoted 3 lines
More drive details from before I ran badblocks (not an SMR drive):
Device Model: WDC WD40EFRX-68WT0N0 Serial Number: WD-WCC4E0642444
At least if it's out of warranty, consider checking the contact pads underside the drive PCB and cleaning them. AFAIK the WD drives in particular are known to develop rust there, which can lead to all sorts of problems including transient "pending" sectors like that.
https://www.youtube.com/watch?v=qmT32VC22PU
https://www.youtube.com/watch?v=tDTt_yjYYQ8
Signature · 2 lines
With respect, Roman
Quoted 2 lines · Marc MERLIN
Howdy, I have a raid 5 where one drive reported this:
197Current_Pending_Sector0x0032200 199 000Old_ageAlways - 29
RAID survival depends on healthy drives. This one does not look healthy. If the data on your RAID is important, I'd replace this drive. Getting this count back down to zero doesn't make it any more trustworthy.
Quoted 5 lines · Phil Turmel
No, it doesn't keep track of anything the filesystem does. Look at the mdadm -E report for one of you member devices -- it shows the location of the superblock, the start location and size of the data area, and possibly information about the optional bitmap or bad block table. Areas outside the data area are not touched by check or repair scrubs.
Ok, that makes more sense. In my case, the md array is the entire drive, minus partition and boot sector header, which is why I expected scanning that would be good enough.
But as been pointed out already, it's very possible that the bad sectors happened either on the parity drive that didn't get scanned, or the unused space before the md array data actually starts on the drive.
Quoted 3 lines
Or just ignore them. You aren't using them, so they can't hurt you. However, do look at the sector numbers in the SMART reports to make sure they aren't in the data area. (If you aren't using the whole disk for
The smart scan log for that drive didn't seem to show where said bad sectores were (smartctl -a, log at the end). Either way, now they've been remapped, so it's fine.
Quoted 7 lines · Andreas Klauer
Quoted 2 lines · Marc MERLIN
Howdy, I have a raid 5 where one drive reported this:
197Current_Pending_Sector0x0032200 199 000Old_ageAlways - 29RAID survival depends on healthy drives. This one does not look healthy. If the data on your RAID is important, I'd replace this drive. Getting this count back down to zero doesn't make it any more trustworthy.
Yeah, I hear what you're saying here. I usually do this actually. If this drive grows another single bad sector, I will do this, but at this time the hassle fo replacing the drive is not worth it yet. The array is also used to hold offsite backups of backups of personal data (not work stuff)
Thanks for the replies.
Marc
Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/ | PGP 1024R/763BE901Quoted 7 lines · Andreas Klauer
Quoted 2 lines · Marc MERLIN
Howdy, I have a raid 5 where one drive reported this:
197Current_Pending_Sector0x0032200 199 000Old_ageAlways - 29RAID survival depends on healthy drives. This one does not look healthy. If the data on your RAID is important, I'd replace this drive. Getting this count back down to zero doesn't make it any more trustworthy.
That's excessively paranoid. You missed:
Quoted 1 line
5
Reallocated_Sector_Ct0x0033200 200 140 Pre-fail Always - 0
In other words, the totally normal accumulation of weak spots on his disk aren't being cleaned up by overwriting as expected from raid scrubbing. The vast majority of such flaws are successfully overwritten and are not a problem. When they are overwritten and the drive firmware decides to reallocate, then you can start worrying. I replace drives when *reallocations* hit double digits, as my experience says they go to hell quickly between 10 and 100.
FWIW, all of my troubles with drives were consumer-grade, mostly w/ scterc support, @ > 35k hours. I'll report to the list when my first big batch of WD reds starts to die.