Question: how to identify failing disk in a RAID1

Hi there.

Recently I have been frequently seeing a damaged filesystem on a RAID1 on boot. a lengthy fsck does get it working, but I am seeing files disappearing as a result.

I am pretty sure that one of the drives has developed some issues and needs to be replaced.

How does one identify which of the 2 disks is the one that is failing?

The system has 2 identical disks, and / is on md0

fstab:

/dev/md0                /                       ext3    defaults        1 1
LABEL=/boot1            /boot                   ext2    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=/boot11           /boot1                  ext2    defaults        1 2
LABEL=SWAP-sdb3         swap                    swap    defaults        0 0
LABEL=SWAP-sda2         swap                    swap    defaults        0 0

fdisk -l shows me:
Disk /dev/sda: 400.0 GB, 400088457216 bytes
255 heads, 63 sectors/track, 48641 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14         535     4192965   82  Linux swap / Solaris
/dev/sda3             536       48641   386411445   fd  Linux raid autodetect

Disk /dev/sdb: 400.0 GB, 400088457216 bytes
255 heads, 63 sectors/track, 48641 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

/dev/sdb1   *           1          13      104391   83  Linux
/dev/sdb2              14       48118   386403412+  fd  Linux raid autodetect
/dev/sdb3           48119       48640     4192965   82  Linux swap / Solaris

Disk /dev/md0: 395.6 GB, 395677007872 bytes
2 heads, 4 sectors/track, 96600832 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Anyone have a suggestion, please?
Responses off list are probably most appropriate.

Thanks for any help.

Signature · 9 lines

Regards, Maurice
[email protected]



--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 41 lines · Maurice Hilarius

Hi there.

Recently I have been frequently seeing a damaged filesystem on a RAID1 on boot. a lengthy fsck does get it working, but I am seeing files disappearing as a result.

I am pretty sure that one of the drives has developed some issues and needs to be replaced.

How does one identify which of the 2 disks is the one that is failing?

The system has 2 identical disks, and / is on md0

fstab:

/dev/md0                /                       ext3    defaults        1 1
LABEL=/boot1            /boot                   ext2    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=/boot11           /boot1                  ext2    defaults        1 2
LABEL=SWAP-sdb3         swap                    swap    defaults        0 0
LABEL=SWAP-sda2         swap                    swap    defaults        0 0

fdisk -l shows me:
Disk /dev/sda: 400.0 GB, 400088457216 bytes
255 heads, 63 sectors/track, 48641 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

/dev/sda1   *           1          13      104391   83  Linux
/dev/sda2              14         535     4192965   82  Linux swap / Solaris
/dev/sda3             536       48641   386411445   fd  Linux raid autodetect

Disk /dev/sdb: 400.0 GB, 400088457216 bytes
255 heads, 63 sectors/track, 48641 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

/dev/sdb1   *           1          13      104391   83  Linux
/dev/sdb2              14       48118   386403412+  fd  Linux raid autodetect
/dev/sdb3           48119       48640     4192965   82  Linux swap / Solaris

Quoted 13 lines

Disk /dev/md0: 395.6 GB, 395677007872 bytes
2 heads, 4 sectors/track, 96600832 cylinders
Units = cylinders of 8 * 512 = 4096 bytes

Anyone have a suggestion, please?
Responses off list are probably most appropriate.

Thanks for any help.

Signature · 2 lines

Regards, Maurice
[email protected]

smartctl -a /dev/sda
smartctl -a /dev/sdb

also, how come swap was not on the raid1?

Quoted 4 lines · Bill Davidsen

..

Quoted 2 lines

Quoted 2 lines

I am pretty sure that one of the drives has developed some issues and needs to be replaced. ..

Quoted 1 line

Very unexpected that the data would be bad without any hardware errors.

I DID say:
"I am pretty sure that one of the drives has developed some issues and needs to be replaced. "

Quoted 1 line

Did you look at your logs to see if one of your drives, or perhasps both, are getting hardware errors?

Oh, I KNOW one does..
The question is WHICH one?

Quoted 2 lines

I would run a 'check' and and see what mdadm finds on the array, you may have other problems.

Pardon my stupidity, care to share some syntax for that?

Quoted 1 line

Actually, I think I would run memtest86 for at least a few hours, starting from a really cold system (not just a cold boot, off for a few hours).

Did that already.

Quoted 2 lines

Your comment "on boot" may come from memory or other component which needs to physically get up to temperature before working reliably. Particularly if you don't get additional errors after you have been up for a while.

It happens cold or hot.

Signature · 1 line

Regards, Maurice

Quoted 4 lines · Bill Davidsen

..
cd /sys/block/md0/md
echo check >sync_action; cat mismatch_cnt

Hi Bill,

I am doing this as root.
I am seeing:

[root@localhost md]# echo check >sync_action; cat mismatch_cnt -bash: echo: write error: Device or resource busy
0

any suggestions?

Morning Bill.

BTW< I want to say "Thanks for your help with this" first. Just in case I forgot.

So, I ran "check" once. It complained, and failed.

A few hours later, I ran it again, and it immediately returned "0"

I am still puzzled:
Why it failed the first time
Why it returned a result in a couple of seconds the second time. What it tells me?
I gather this means the md0 device is healthy.

So, meanwhile back at the ranch, I still think sda is failing..

Any thoughts on that?

Quoted 10 lines · Bill Davidsen

Quoted 7 lines · Maurice Hilarius

Morning Bill.

BTW< I want to say "Thanks for your help with this" first. Just in case I forgot.

So, I ran "check" once. It complained, and failed.

Does the failure provide any useful information?

No.
Here is what I got the first time:

root@localhost md]# echo check >sync_action; cat mismatch_cnt -bash: echo: write error: Device or resource busy
0

Later, on my second try, a few hours later, it worked, reporting no error. ..
[maurice@localhost ~]$ su -
Password:
[root@localhost ~]# cd /sys/block/md0/md
[root@localhost md]# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md0 : active raid1 sda3[0] sdb2[1]

      386403328 blocks [2/2] [UU]

unused devices: <none>
[root@localhost md]# echo check >sync_action; cat mismatch_cnt 0

Quoted 3 lines

I think it's time to be keeping a good backup, and hopefully someone else has a good thought on running this down more.

Thanks, updated that backup at the first sign of trouble

Quoted 2 lines

Quoted 1 line

Any thoughts on that?

Quoted 2 lines

The only thought I have at the moment is marginal power supply, and that's just because it can generate all manner of odd behaviors, rather than any other hints. Sorry.

Yeah. I am going to replace *both* disks, and then run the manufacturers utility (Seatest) on them.

Quoted 1 line

If you aren't getting errors from SMART or logs, and I don't remember you sending me that info, I'm not sure how you determine which drive is the problem.

Exactly.

Thanks a LOT for trying, Bill..

Hi all.

With much appreciated help from Bell Davidsen and Justin Piszcz I recently dealt with a problem with a RAID1 set, caused by a failing hard disk.

At the end, there is one question remaining, which I think is quite important:
When one has a RAID5 or RAID6, and a disk starts "acting up" mdadm rapidly kicks out the offending device. Some might say "too easily" but that is another thread.

On a RAID1 set, until the failing disk completely "packs it in" it remains part of the RAID.

Why??

Some more background:
Since the issue was reported and explored I have recreated this on a test machine.
Installed RAID1 with one known good and one know error prone drive.
Easy to do as the error drive has a thermal issue.
Keep it cold, no problems, but after 30 minutes use in a +25C room it start to generate data errors.
I reproduced exactly the problem I saw before:
Data errors occur, the other drive in the RAID1 set gets "infected" with the bad data, and the file system will get corrupted. On BOTH drives.

This is highly reproducible.

In summary:
1) RAID1 lacks significant protection from the effects of a data error condition on a failing drive
2) I recommend anyone using madadm refrain from using RAID1 until this issue is addressed and resolved.

Thanks again.

Quoted 34 lines · Justin Piszcz

Quoted 21 lines · Maurice Hilarius

Hi all.

With much appreciated help from Bell Davidsen and Justin Piszcz I recently dealt with a problem with a RAID1 set, caused by a failing hard disk.

At the end, there is one question remaining, which I think is quite important:
When one has a RAID5 or RAID6, and a disk starts "acting up" mdadm rapidly kicks out the offending device. Some might say "too easily" but that is another thread.

On a RAID1 set, until the failing disk completely "packs it in" it remains part of the RAID.

Why??

Some more background:
Since the issue was reported and explored I have recreated this on a test machine.
Installed RAID1 with one known good and one know error prone drive.
Easy to do as the error drive has a thermal issue.
Keep it cold, no problems, but after 30 minutes use in a +25C room it start to generate data errors.
I reproduced exactly the problem I saw before:
Data errors occur, the other drive in the RAID1 set gets "infected" with the bad data, and the file system will get corrupted. On BOTH drives.

Quoted 7 lines

This is highly reproducible.

In summary:
1) RAID1 lacks significant protection from the effects of a data error condition on a failing drive
2) I recommend anyone using madadm refrain from using RAID1 until this issue is addressed and resolved.

Thanks again.

I can confirm this, until you actually REBOOT the host with RAID1 only then will it kick it out. Whereas with RAID5, I experienced the same thing, it kicks it out right away, would need to wait for the linux-raid/developers to answer this one.

Quoted 2 lines

Justin.

Actually reboot does not help me.
mdadm seems to NEVER "kick out" the bad disk.
Even when it is horribly erroring.

I think this is a CRITICAL problem, as, if one is using RAID1 thinking it will enhance their data reliability, they stand a very good chance of getting a nasty surprise.

To: Maurice Hilarius <[email protected]>
From: Justin Piszcz <[email protected]>
Subject: Re: Question: how to identify failing disk in a RAID1

Quoted 41 lines · Maurice Hilarius

Quoted 27 lines · Justin Piszcz

Quoted 23 lines · Maurice Hilarius

Hi all.

With much appreciated help from Bell Davidsen and Justin Piszcz I recently dealt with a problem with a RAID1 set, caused by a failing hard disk.

At the end, there is one question remaining, which I think is quite important:
When one has a RAID5 or RAID6, and a disk starts "acting up" mdadm rapidly kicks out the offending device. Some might say "too easily" but that is another thread.

On a RAID1 set, until the failing disk completely "packs it in" it remains part of the RAID.

Why??

Some more background:
Since the issue was reported and explored I have recreated this on a test machine.
Installed RAID1 with one known good and one know error prone drive.
Easy to do as the error drive has a thermal issue.
Keep it cold, no problems, but after 30 minutes use in a +25C room it start to generate data errors.
I reproduced exactly the problem I saw before:
Data errors occur, the other drive in the RAID1 set gets "infected" with the bad data, and the file system will get corrupted. On BOTH drives.

This is highly reproducible.

Quoted 9 lines

Quoted 5 lines

In summary:
1) RAID1 lacks significant protection from the effects of a data error condition on a failing drive
2) I recommend anyone using madadm refrain from using RAID1 until this issue is addressed and resolved.

Thanks again.

I can confirm this, until you actually REBOOT the host with RAID1 only then will it kick it out. Whereas with RAID5, I experienced the same thing, it kicks it out right away, would need to wait for the linux-raid/developers to answer this one.

Justin.

Actually reboot does not help me.
mdadm seems to NEVER "kick out" the bad disk.
Even when it is horribly erroring.

Quoted 2 lines

I think this is a CRITICAL problem, as, if one is using RAID1 thinking it will enhance their data reliability, they stand a very good chance of getting a nasty surprise.

Yikes, what kernel+mobo+chipset+drives are in use (the developers will want to know) also are you using drives on different channels? Or e.g., two drives on one ide cable? (To summarize for the developers)

Justin.

It would be nice to be able to use smartmontools to monitor the health of the drives in a RAID1 array. Get the best of both worlds then.

To: Linux RAID Mailing List <[email protected]>
From: Richard Scobie <[email protected]>
Subject: Re: Question: how to identify failing disk in a RAID1

Quoted 1 line · Keith Roberts

It would be nice to be able to use smartmontools to monitor the health of the drives in a RAID1 array. Get the best of both worlds then.

I have been doing this for years.

What problems are you seeing using smartd on a RAID1?

Regards,

Richard

All email addresses are challenge-response protected with TMDA [http://tmda.net]

Reading the documentation for smartmontools I got the impression that it cannot work with RAID controllers, apart from 3ware and some Highpoint. Maybe I'm getting mixed up with hardware raid?

So is it safe to use all features of smartmontools, including running tests, on a Linux software RAID1 array?

Is there any way that the smartmontools code can be included in the md driver code, to allow mdadm access to the SMART data on a RAID1 set of disks please?