Skip to contentSkip to conversation

Raid check didn't fix Current_Pending_Sector, but badblocks -nsv did

DISCUSSION Marc MERLIN 7 participants 13 messages Jun 6, 2016 – Jun 8, 2016 MD RAID

lore
Share & tools
lore
#0 lore raw

Howdy, I have a raid 5 where one drive reported this:
197 Current_Pending_Sector 0x0032 200 199 000 Old_age Always - 29

So I did this:

myth:~# echo check > /sys/block/md5/md/sync_action
[173947.749761] md: data-check of RAID array md5
(...)
[370316.769230] md: md5: data-check done.

My understanding was that it was supposed to read every block of every drive, and if some blocks were unreadable, use parity to rewrite them on some fresh backup blocks.
If a block returned garbage instead, md5 cannot fix this not knowing which block is wrong, but I'm assuming the check would have failed with an error.

However after the check is over, I still have 29 Current_Pending_Sector on that drive.

Since raid check succeeded, I'm going to assume that the sectors were readable and did not return garbage, or I'd have gotten a parity mismatch error.
Should then assume that either
1) the smart counter/logic is wrong?
2) the pending sectors started returning correct data again, so linux md has no idea those blocks are "weak" and I have no easy way to forcibly remap them.
3) the bad blocks did get remapped somehow, but the smart counter did not get reset due to a firmware bug
4) other

After 2 days of testing with badblocks, it seems that it's #2, and I'm not sure if there is anything raid check could have done (probably not)

Since raid check didn't do the job I was hoping for, I ran this instead:

myth:~# badblocks -nsv /dev/sdg  
Checking for bad blocks in non-destructive read-write mode  
From block 0 to 3907018583  
Checking for bad blocks (non-destructive read-write test)
Testing with random pattern:  57.87% done, 22:24:15 elapsed. (0/0/0 errors))

And this worked:
197 Current_Pending_Sector 0x0032 200 199 000 Old_age Always - 0
strangely, I also have:

  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0

I guess this means that my drive's auto reallocation logic is faulty and that it will not re-allocate blocks that are weak, even after it was able to read them.
Does that sound correct?

More drive details from before I ran badblocks (not an SMR drive):

Device Model:     WDC WD40EFRX-68WT0N0
Serial Number:    WD-WCC4E0642444
LU WWN Device Id: 5 0014ee 2b437e9a6
Firmware Version: 80.00A80
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       1617
  3 Spin_Up_Time            0x0027   175   173   021    Pre-fail  Always       -       8250
  4 Start_Stop_Count        0x0032   094   094   000    Old_age   Always       -       6773
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   074   074   000    Old_age   Always       -       19092
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       158
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       91
193 Load_Cycle_Count        0x0032   182   182   000    Old_age   Always       -       54642
194 Temperature_Celsius     0x0022   121   103   000    Old_age   Always       -       31
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   199   000    Old_age   Always       -       29
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   199   000    Old_age   Always       -       2
200 Multi_Zone_Error_Rate   0x0008   200   189   000    Old_age   Offline      -       0

SMART Self-test log structure revision number 1

Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     19081         -
# 2  Short offline       Completed without error       00%     19057         -
# 3  Short offline       Completed without error       00%     19035         -
# 4  Short offline       Completed without error       00%     18984         -
# 5  Extended offline    Completed without error       00%     18974         -

Thanks,
Marc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  
#1 lore raw
Quoted 9 lines · Marc MERLIN

Howdy, I have a raid 5 where one drive reported this:
197 Current_Pending_Sector 0x0032 200 199 000 Old_age Always - 29

So I did this:

myth:~# echo check > /sys/block/md5/md/sync_action
[173947.749761] md: data-check of RAID array md5
(...)
[370316.769230] md: md5: data-check done.
Quoted 5 lines

My understanding was that it was supposed to read every block of every drive, and if some blocks were unreadable, use parity to rewrite them on some fresh backup blocks.
If a block returned garbage instead, md5 cannot fix this not knowing which block is wrong, but I'm assuming the check would have failed with an error.

https://www.kernel.org/doc/Documentation/md.txt shows for sync_action

check - A full check of redundancy was requested and is
happening. This reads all blocks and checks them. A repair may also happen for some raid levels.
repair - A full check and repair is happening. This is
similar to 'resync', but was requested by the user, and the write-intent bitmap is NOT used to optimise the process.

I think you wanted 'repair' not 'check'.

--Sarah

#2 lore raw
Quoted 11 lines · Sarah Newman

https://www.kernel.org/doc/Documentation/md.txt shows for sync_action

check - A full check of redundancy was requested and is
happening. This reads all blocks and checks them. A repair may also happen for some raid levels.
repair - A full check and repair is happening. This is
similar to 'resync', but was requested by the user, and the write-intent bitmap is NOT used to optimise the process.

Quoted 1 line

I think you wanted 'repair' not 'check'.

From what I understand, the only difference between the 2 is that repair does not use the write-intent bitmap, but both will repair an error if found.
https://www.thomas-krenn.com/en/wiki/Mdadm_checkarray#Check_vs._Repair

Or are you saying that check after getting a read error from one drive, would not rewrite the bad block on that drive? I thought it did...

Either way, it seems that neither would have worked because while those blocks were marked as "need to be reallocated" by the drive, I think the kernel was actually able to read them without problem, so the md layer never saw anything and therefore never did anything either.

Whereas badblocks forced an unconditional rewrite of all blocks, and forced the drive to re-allocate those "weak" blocks, even though it was able to read them.

Does that sound about right?

Marc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  
#3 lore raw
Quoted 3 lines · Marc MERLIN

From what I understand, the only difference between the 2 is that repair does not use the write-intent bitmap, but both will repair an error if found.

Repair doesn't even read the parity (and Q syndrome) blocks if it doesn't need them. It unconditionally recomputes and writes parity (and
Q) blocks.

Both processes will compute and rewrite a block that reports a read error.

Quoted 4 lines

https://www.thomas-krenn.com/en/wiki/Mdadm_checkarray#Check_vs._Repair

Or are you saying that check after getting a read error from one drive, would not rewrite the bad block on that drive? I thought it did...

They both do.

Quoted 4 lines

Either way, it seems that neither would have worked because while those blocks were marked as "need to be reallocated" by the drive, I think the kernel was actually able to read them without problem, so the md layer never saw anything and therefore never did anything either.

Both processes only apply to the actual data area of each member device, which is usually substantially less than the whole disk. Both processes will read or write (or both) every data area sector.

Sectors that haven't ever been deliberately written and aren't read by normal mdadm processes are often the first to pop up in disk self-diagnostics. If you look at the sector offsets in the SMART data, I bet you find they're all outside the area mdadm is using.

Magnetism does slowly decay on drive platters, faster in the spots where manufacturing didn't maintain the intended oxide thickness. Completely normal and is the basis for URE specifications.

Quoted 3 lines

Whereas badblocks forced an unconditional rewrite of all blocks, and forced the drive to re-allocate those "weak" blocks, even though it was able to read them.

Probably not. It got 'em because it wasn't limited to the data area.

Phil

#4 lore raw
Quoted 3 lines · Phil Turmel

Repair doesn't even read the parity (and Q syndrome) blocks if it doesn't need them. It unconditionally recomputes and writes parity (and
Q) blocks.

I see. So if it happens to rewrite a parity block that was unreadable but that it didn't have to read, I won't even know it was unreadable to start with (but only one change out of n of that happening in raid5)

Quoted 7 lines

Both processes will compute and rewrite a block that reports a read error.

Quoted 4 lines

https://www.thomas-krenn.com/en/wiki/Mdadm_checkarray#Check_vs._Repair

Or are you saying that check after getting a read error from one drive, would not rewrite the bad block on that drive? I thought it did...

They both do.

Thanks for confirming.

Quoted 3 lines

Both processes only apply to the actual data area of each member device, which is usually substantially less than the whole disk. Both processes will read or write (or both) every data area sector.

Mmmh, so it keeps track of how much of my block device is used by filesystem blocks and skips the rest? I didn't know that.

Quoted 4 lines

Sectors that haven't ever been deliberately written and aren't read by normal mdadm processes are often the first to pop up in disk self-diagnostics. If you look at the sector offsets in the SMART data, I bet you find they're all outside the area mdadm is using.

Got it.

Quoted 5 lines
Quoted 3 lines

Whereas badblocks forced an unconditional rewrite of all blocks, and forced the drive to re-allocate those "weak" blocks, even though it was able to read them.

Probably not. It got 'em because it wasn't limited to the data area.

Right, I understand now, good to know.
So I'll use badblocks next time I have this issue.

Thanks for clearing this up for me.
Marc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
#9 lore raw
In reply to Marc MERLIN
Quoted 2 lines · On Mon, 6 Jun 2016 10:41:13 -0700 Marc MERLIN <[email protected]> wrote:

However after the check is over, I still have 29 Current_Pending_Sector on that drive.

I had that once, even asked on this list (can't find that now), and it turned out the pending sector(s) were in the gap between the start of the partition and the start of the mdadm actual data (which turned out to be surprisingly large) -- so naturally neither a check nor repair have ever tried to read or write there.

Quoted 3 lines

More drive details from before I ran badblocks (not an SMR drive):

Device Model:     WDC WD40EFRX-68WT0N0
Serial Number:    WD-WCC4E0642444

At least if it's out of warranty, consider checking the contact pads underside the drive PCB and cleaning them. AFAIK the WD drives in particular are known to develop rust there, which can lead to all sorts of problems including transient "pending" sectors like that.
https://www.youtube.com/watch?v=qmT32VC22PU
https://www.youtube.com/watch?v=tDTt_yjYYQ8

Signature · 2 lines
With respect,
Roman
#5 lore raw
In reply to Marc MERLIN
Quoted 8 lines · Marc MERLIN
Quoted 3 lines · Phil Turmel

Both processes only apply to the actual data area of each member device, which is usually substantially less than the whole disk. Both processes will read or write (or both) every data area sector.

Mmmh, so it keeps track of how much of my block device is used by filesystem blocks and skips the rest? I didn't know that.

No, it doesn't keep track of anything the filesystem does. Look at the mdadm -E report for one of you member devices -- it shows the location of the superblock, the start location and size of the data area, and possibly information about the optional bitmap or bad block table. Areas outside the data area are not touched by check or repair scrubs.

Quoted 2 lines

Right, I understand now, good to know.
So I'll use badblocks next time I have this issue.

Or just ignore them. You aren't using them, so they can't hurt you. However, do look at the sector numbers in the SMART reports to make sure they aren't in the data area. (If you aren't using the whole disk for mdadm, be sure to adjust for the partition start sector.) If they *are* in the data area, watch dmesg while scrubbing to see what's really happening.

Phil

#6 lore raw
Quoted 1 line · Phil Turmel

Or just ignore them. You aren't using them, so they can't hurt you. However, do look at the sector numbers in the SMART reports to make sure they aren't in the data area. (If you aren't using the whole disk for mdadm, be sure to adjust for the partition start sector.) If they *are* in the data area, watch dmesg while scrubbing to see what's really happening.

I had a similar problem a few years back.

I seem to remember that there was a problem that the pending blocks were in the superblock area, and these weren't written out when I did "check" or "repair". I might be misremembering though, but I seem to remember I had some kind of problem that when I tried to add the dive back in again, it got a read error when trying to read the superblock.

They might have also been in the offset area. What I did was that I did a --replace to another drive, then I dd:ed /dev/zero to the first gigabyte of the drive, which removed the pending blocks. I then added the drive back in.

Could someone confirm whether the entire (actually valuable portion) of the superblock area is written out at any kind of frequency, or when a check/repair is issued? If not, I think that would be a valuable thing to do?

Signature · 1 line
Mikael Abrahamsson    email: [email protected]
#10 lore raw
In reply to Marc MERLIN
Quoted 2 lines · Marc MERLIN

Howdy, I have a raid 5 where one drive reported this:
197 Current_Pending_Sector 0x0032 200 199 000 Old_age Always - 29

RAID survival depends on healthy drives. This one does not look healthy. If the data on your RAID is important, I'd replace this drive. Getting this count back down to zero doesn't make it any more trustworthy.

#11 lore raw
Quoted 5 lines · Phil Turmel

No, it doesn't keep track of anything the filesystem does. Look at the mdadm -E report for one of you member devices -- it shows the location of the superblock, the start location and size of the data area, and possibly information about the optional bitmap or bad block table. Areas outside the data area are not touched by check or repair scrubs.

Ok, that makes more sense. In my case, the md array is the entire drive, minus partition and boot sector header, which is why I expected scanning that would be good enough.
But as been pointed out already, it's very possible that the bad sectors happened either on the parity drive that didn't get scanned, or the unused space before the md array data actually starts on the drive.

Quoted 3 lines

Or just ignore them. You aren't using them, so they can't hurt you. However, do look at the sector numbers in the SMART reports to make sure they aren't in the data area. (If you aren't using the whole disk for

The smart scan log for that drive didn't seem to show where said bad sectores were (smartctl -a, log at the end). Either way, now they've been remapped, so it's fine.

Quoted 7 lines · Andreas Klauer
Quoted 2 lines · Marc MERLIN

Howdy, I have a raid 5 where one drive reported this:
197 Current_Pending_Sector 0x0032 200 199 000 Old_age Always - 29

RAID survival depends on healthy drives. This one does not look healthy. If the data on your RAID is important, I'd replace this drive. Getting this count back down to zero doesn't make it any more trustworthy.

Yeah, I hear what you're saying here. I usually do this actually. If this drive grows another single bad sector, I will do this, but at this time the hassle fo replacing the drive is not worth it yet. The array is also used to hold offsite backups of backups of personal data (not work stuff)

Thanks for the replies.
Marc

Signature · 4 lines
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901
#12 lore raw
In reply to Andreas Klauer
Quoted 7 lines · Andreas Klauer
Quoted 2 lines · Marc MERLIN

Howdy, I have a raid 5 where one drive reported this:
197 Current_Pending_Sector 0x0032 200 199 000 Old_age Always - 29

RAID survival depends on healthy drives. This one does not look healthy. If the data on your RAID is important, I'd replace this drive. Getting this count back down to zero doesn't make it any more trustworthy.

That's excessively paranoid. You missed:

Quoted 1 line

5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0

In other words, the totally normal accumulation of weak spots on his disk aren't being cleaned up by overwriting as expected from raid scrubbing. The vast majority of such flaws are successfully overwritten and are not a problem. When they are overwritten and the drive firmware decides to reallocate, then you can start worrying. I replace drives when *reallocations* hit double digits, as my experience says they go to hell quickly between 10 and 100.

FWIW, all of my troubles with drives were consumer-grade, mostly w/ scterc support, @ > 35k hours. I'll report to the list when my first big batch of WD reds starts to die.

#7 lore raw
In reply to Phil Turmel
Quoted 6 lines · Phil Turmel
Quoted 2 lines · Marc MERLIN

Right, I understand now, good to know.
So I'll use badblocks next time I have this issue.

Or just ignore them. You aren't using them, so they can't hurt you.

That's actually not necessarily true.

If you have a dud sector early on the disk (so before the start of the RAID data) you will terminate every SMART long test in the first couple of meg of the disk. So while a dud down there won't necessarily impact your usage from a RAID perspective, it'll knacker your ability to regularly check the disks in their entirety. SMART tests abort on the first bad read.

It's ugly, but in the single instance I had that happen, I removed the drive from the array, wrote zero to the entire disk and then added it back. That forced a reallocation in the affected area.

Usually if it is in the RAID zone, a check scrub will clear it up. Having said that I've had a very peculiar one here in the last couple of days.

A WD 2TB Green drive with TLER set to 7 seconds. The first read would error out in 7 seconds (as it should), but a second read succeeded. After returning the error, the drive must have kept trying to recover in the background and eventually succeeded and cached the result. So subsequent reads were ok. After reading and writing enough to other parts of the drive to flush the drives cache, the process would repeat.

In this case, it took about 3 check scrubs to actually hit the read error and force a re-write.

#8 lore raw
Quoted 8 lines · Brad Campbell
Quoted 6 lines · Phil Turmel
Quoted 2 lines · Marc MERLIN

Right, I understand now, good to know.
So I'll use badblocks next time I have this issue.

Or just ignore them. You aren't using them, so they can't hurt you.

Quoted 8 lines

That's actually not necessarily true.

If you have a dud sector early on the disk (so before the start of the RAID data) you will terminate every SMART long test in the first couple of meg of the disk. So while a dud down there won't necessarily impact your usage from a RAID perspective, it'll knacker your ability to regularly check the disks in their entirety. SMART tests abort on the first bad read.

Don't bother doing long self-tests on drives participating in an array -- check scrubs do everything a long self-test does on the area of interest, plus actually fixing UREs that are found. And check scrubs don't abort on a read failure.

My advice stands: ignore the UREs in unused areas of the disk.

Quoted 3 lines

It's ugly, but in the single instance I had that happen, I removed the drive from the array, wrote zero to the entire disk and then added it back. That forced a reallocation in the affected area.

Completely pointless exercise that opened a window of higher-risk of failure of your array. Unless you used --replace with another spare to maintain redundancy on your array while that disk was out.

Quoted 10 lines

Usually if it is in the RAID zone, a check scrub will clear it up. Having said that I've had a very peculiar one here in the last couple of days.

A WD 2TB Green drive with TLER set to 7 seconds. The first read would error out in 7 seconds (as it should), but a second read succeeded. After returning the error, the drive must have kept trying to recover in the background and eventually succeeded and cached the result. So subsequent reads were ok. After reading and writing enough to other parts of the drive to flush the drives cache, the process would repeat.

Pure speculation. Unless you can show better evidence that those drives will cache a read in that case, I would say it was just a mild enough weak spot that it randomly succeeded more than not. And if you follow my advice, it doesn't matter: if the array is the only process reading from the disk, the first appearance of the URE would be the last, as the array would re-write it immediately. Whether during a scrub or due to normal access.

Regular long self-tests are highly recommended for stand-alone disks and for array hot spares.

Phil