device with newer data added as spare - data now gone?

Hi all

I've lost quite a lot of data on my /home raid partition and I'm wondering what exactly I did to make it happen. I'd like to know so something similar won't happen in the future.

I'm pretty much a raid newbie. I setup raid1 on my home server and I'm guessing that something like this happened. Please tell me if it's possible.

* Some time ago I did something to have one device fail which resulted md3 in having only 1 device.
* Time went by without me noticing (because I suck)
* An update broke my raid setup and gave me a kernel panic (because I suck). Didn't put the mdadm and raid hooks in mkinitcpio.conf
* Booted a live-cd, mounted the drives and chrooted back into the system and fixed the mkinitcpio.conf
* Rebooted and noticed that md3 was running with only 1 device
* Added sdb4 to md3 and it then read 1 device with 1 spare
* cat /proc/mdstat started to say "recovery"
* All data from approx. 1 year is gone

I guessing that the old (not updated) device was set as "master" and the data on the drive (containing newer data) was overwritten by data on the old device - is this plausible?

If not what exactly did I do to delete all of the data?

It's not the end of the world but it does suck a lot - especially some email and pictures are gone that I will miss.

Any clarifications will be much appreciated.

View this message in context: http://www.nabble.com/device-with-newer-data-added-as-spare---data-now-gone--tp24218843p24218843.html
Sent from the linux-raid mailing list archive at Nabble.com.

Well, first of all, of course learning from your mistakes will
prevent them from happening in the future. If you ask me, a second very important utility to enable is mdadm's ability to notify you via e-mail whenever a significant event transpires. You will then be notified quickly of any significant changes to any RAID array, such as losing a hard drive.

Finally, more important than anything else: BACK UP YOUR IMPORTANT
DATA. If it is data that can be recovered through some external process, but takes a bit of doing, back it up once, and keep it handy. A different drive or array on the same machine or a different machine in the same room is fine for this level of backup. If it is data that cannot be recovered and would cause some heartache if lost, then include it in the local handy backup, but also include it in an off-premise backup. If it is critical data - like financial information, then back it up 16 ways from Sunday. I keep all critical data backed up on two different servers with independent RAID arrays, DVD-ROM backups offsite, and independent multi-generation backups on every workstation which accesses the data. If it is a commercial application and the revenue supports it, or if it is important enough to you and you can personally afford it, I suggest you might look into an online storage solution.

Remember, RAID arrays are fault tolerant, not fault-free, and while
hard drives are frail, the most likely source of data failure by far is user error.

I presume md3 is the /home array and this was a 2 drive RAID1 array,
yes?

See above. We human beings all tend to suck from time to time.
Computers can help by reminding or notifying us of things - if we bother to set them up to do so.

This all sounds like lessons learned.

Was sdb4 originally the second partition in the array? What is the
first partition? What was the apparent cause of the failure?

Well, I suppose, yeah.

What command did you use to add the partition back to the array?

If the old device was brought up as md3 and had dropped out months ago, the data would now be the data that existed when that disk dropped off. And when a device drops out, there is no mark on that device marking it as bad since the typical reasons for the device dropping off are that it is not longer talking. And sometimes mirrors are intentionally broken for various reasons to preserve a copy for one reason or another such as to be able to back out of a serious OS upgrade that did not go well quickly.

If you added the current device as a spare it would have copied the data from the old device over the current device.

That is one thing that would make 3+ disk raid5 a bit more resistant to this, with a dropped off disk you could not start the array with only the dropped device, and with all 3, 2 of the devices will know the 3rd was dropped at some time in the past, and with any 2 on of those devices would believe the other one was marked bad.

I've got myself into the babit of comparing the output from "cat /proc/mdstat" and "mdadm -Esbv" to see if there's any old md metadata
floating around on disks I'm about to use before using them. Just as
a precaution. If I find any then I --zero-superblock the disk first
before re-using it, just to prevent myself getting caught out by events like this.

Rgds,

John