Uncorrectable errors: how do I fix it?
One of the drives in my RAID-5 array is showing uncorrectable errors:
Nov 28 17:52:36 beast smartd[8184]: Device: /dev/sdc, 1 Currently unreadable (pending) sectors Nov 28 17:52:36 beast smartd[8184]: Device: /dev/sdc, 1 Offline uncorrectable sectors
And it fails a self-test:
SMART Self-test log structure revision number 0
Warning: ATA Specification requires self-test log structure revision number = 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed: read failure 20% 931
1953520763
Now that's not good but it's probably not bad enough to get the drive replaced. (Opinions?) Anyway, rewriting the sector ought to "cure" it, so how do I do that?
Here's the details of my array:
[root@beast md]# mdadm --detail /dev/md1/dev/md1:
Version : 00.90.03
Creation Time : Mon Jul 28 15:49:09 2008
Raid Level : raid5
Array Size : 1953310720 (1862.82 GiB 2000.19 GB)
Used Dev Size : 976655360 (931.41 GiB 1000.10 GB)
Raid Devices : 3
Total Devices : 3
Preferred Minor : 1
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Fri Nov 28 17:56:22 2008
State : active
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
UUID : d8c57a89:166ee722:23adec48:1574b5fc
Events : 0.6112
Number Major Minor RaidDevice State
0 8 2 0 active sync /dev/sda2
1 8 18 1 active sync /dev/sdb2
2 8 34 2 active sync /dev/sdc2
I tried:
[root@beast md]# mdadm /dev/md1 --fail /dev/sdc2
mdadm: set /dev/sdc2 faulty in /dev/md1 [root@beast md]# mdadm /dev/md1 --remove /dev/sdc2
mdadm: hot removed /dev/sdc2
[root@beast md]# mdadm /dev/md1 --add /dev/sdc2
mdadm: re-added /dev/sdc2
but that finished instantly. I guess it would since the array has a write-intent bitmap and it's noticed that sdc2 is being re-added. I could tell the system to do a complete resync with:
# echo repair > /sys/block/md1/md/sync_action
but really I want to tell the system to rebuild entirely from sda2 and sdb2, onto sdc2. At least I think I do. I've a feeling the answer is to zero the superblock, but I'm not confident about doing that because I'm not sure if re-adding the thing without a superblock will either work or do the Right Thing[tm].
Cheers,
John.
Quoted 10 lines · John Robinson
One of the drives in my RAID-5 array is showing uncorrectable errors:
I tried:
[root@beast md]# mdadm/dev/md1--fail/dev/sdc2
mdadm: set/dev/sdc2faulty in/dev/md1[root@beast md]# mdadm/dev/md1--remove/dev/sdc2
mdadm: hot removed/dev/sdc2
[root@beast md]# mdadm/dev/md1--add/dev/sdc2
mdadm: re-added/dev/sdc2
Quoted 4 lines
but that finished instantly. I guess it would since the array has a write-intent bitmap and it's noticed that sdc2 is being re-added. I could tell the system to do a complete resync with:
# echo repair > /sys/block/md1/md/sync_actionbut really I want to tell the system to rebuild entirely from sda2 and sdb2, onto sdc2. At least I think I do. I've a feeling the answer is to zero the superblock, but I'm not confident about doing that because I'm not sure if re-adding the thing without a superblock will either work or do the Right Thing[tm].
Before you do this you should backup your data elsewhere just incase you cannot rebuild the array later.
You may be able to fix it using the hdparm --write-sector command if you know what you are doing, otherwise, the quickest way is to fail it from the array as you did before, remove it from the raid set.
zero the disk:
dd if=/dev/zero of=/dev/disk
run badblocks on it a few times:
run short+long test when its all done:
re-check smart statistics and they should complete successfully.
then sfdisk -d /dev/sda | sfdisk /dev/sdc
then pop it back into the array with mdadm /dev/md1 -a /dev/sdc2
Justin.
Quoted 18 lines · John Robinson
One of the drives in my RAID-5 array is showing uncorrectable errors:
Nov 28 17:52:36 beast smartd[8184]: Device: /dev/sdc, 1 Currently unreadable (pending) sectors Nov 28 17:52:36 beast smartd[8184]: Device: /dev/sdc, 1 Offlineuncorrectable sectors
And it fails a self-test:
SMART Self-test log structure revision number 0
Warning: ATA Specification requires self-test log structure revision number = 1
NumTest_DescriptionStatus Remaining
LifeTime(hours)LBA_of_first_error
# 1 Short offline Completed: read failure 20% 931
1953520763Now that's not good but it's probably not bad enough to get the drive replaced. (Opinions?) Anyway, rewriting the sector ought to "cure" it, so how do I do that?
..
Quoted 11 lines
I tried:
[root@beast md]# mdadm/dev/md1--fail/dev/sdc2
mdadm: set/dev/sdc2faulty in/dev/md1[root@beast md]# mdadm/dev/md1--remove/dev/sdc2
mdadm: hot removed/dev/sdc2
[root@beast md]# mdadm/dev/md1--add/dev/sdc2
mdadm: re-added/dev/sdc2but that finished instantly. I guess it would since the array has a write-intent bitmap and it's noticed that sdc2 is being re-added. I could tell the system to do a complete resync with:
Quoted 7 lines
# echo repair > /sys/block/md1/md/sync_actionbut really I want to tell the system to rebuild entirely from sda2 and sdb2, onto sdc2. At least I think I do. I've a feeling the answer is to zero the superblock, but I'm not confident about doing that because I'm not sure if re-adding the thing without a superblock will either work or do the Right Thing[tm].
I would recommend the "echo repair" approach. It won't write every block on sdc, but you don't really need that.
And if you hit a bad block on some other drive, it will cope much better than removing a drive and adding it back in.
However if you really want to write all of sdc and you are willing to risk the possibility of a bad block on sda or sdb, then zeroing the superblock on sdc before adding it back in will do what you expect.
The suggestion made by Justin of always having backups is, of course, a good one.
NeilBrown