Skip to contentSkip to conversation

raid5 disaster

DISCUSSION Bruno Seoane 2 participants 2 messages May 23, 2006 MD RAID

lore
Share & tools
lore
#0 lore raw

Hi,

I had a working raid5 setup with 5 SATA disks, 3 attached to a Promise TX4 and 2 more attached to the mainboard controller.

It has been working flawlessly for a long time, but I had to add a sat card to the machine so I also upgraded to 2.6.16.16

I don't know if there was some problem with that kernel, but CPU usage was almost 100% when writing to the raid, so I decided to go back to me old and trusty 2.6.15.1, but when shutting down the system it wouldn't finish so I had to power it down.

On the next reboot I saw this:

md: Autodetecting RAID arrays.
md: invalid superblock checksum on sdc1
md: sdc1 has invalid sb, not importing!
md: invalid superblock checksum on sde1
md: sde1 has invalid sb, not importing!
md: autorun ...
md: considering sdd1 ...
md:  adding sdd1 ...
md:  adding sdb1 ...
md:  adding sda1 ...
md: created md0
md: bind<sda1>
md: bind<sdb1>
md: bind<sdd1>
md: running: <sdd1><sdb1><sda1>
raid5: device sdd1 operational as raid disk 1
raid5: device sdb1 operational as raid disk 0
raid5: device sda1 operational as raid disk 4
raid5: not enough operational devices for md0 (2/5 failed)

RAID5 conf printout:
--- rd:5 wd:3 fd:2
disk 0, o:1, dev:sdb1
disk 1, o:1, dev:sdd1
disk 4, o:1, dev:sda1
raid5: failed to run raid set md0

So sdc1 and sde1 have an invalid superblock (I assume this was because there was some I/O activity when I switched it down).

Now, as you suppose, I'd like to access my data.

This is what I get from the faulty (and one of the working disks) with the 'examine' parameter:

/dev/sda1:

          Magic : a92b4efc
        Version : 00.90.03
           UUID : 8e47d871:51e2f219:52b05fbf:44206fa0

Creation Time : Sat Jan 21 00:20:33 2006
Raid Level : raid5
Raid Devices : 5
Total Devices : 5
Preferred Minor : 0

Update Time : Tue May 23 19:06:48 2006
State : clean
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Checksum : ba51512f - correct
Events : 0.3551144

Layout : left-symmetric
Chunk Size : 128K

Number Major Minor RaidDevice State
this 4 8 1 4 active sync /dev/sda1

   0     0       8       17        0      active sync   /dev/sdb1
   1     1       8       49        1      active sync   /dev/sdd1
   2     2       8       65        2      active sync   /dev/sde1
   3     3       8       33        3      active sync   /dev/sdc1
   4     4       8        1        4      active sync   /dev/sda1

/dev/sdc1:

          Magic : a92b4efc
        Version : 00.90.03
           UUID : 8e47d871:00000000:00000000:260f0100

Creation Time : Sat Jan 21 00:20:33 2006
Raid Level : raid5
Raid Devices : 16777216
Total Devices : 0
Preferred Minor : 5058

Update Time : Fri Dec 13 20:45:52 1901
State : active
Active Devices : -2147483648
Working Devices : -2147483648
Failed Devices : -2147483648
Spare Devices : -2147483648

       Checksum : 80000000 - expected 2255ae19
         Events : -2147483648.-2147483648

Floating point exception

/dev/sde1:

          Magic : a92b4efc
        Version : 00.90.03
           UUID : 8e47d871:00000000:00000000:260f0100

Creation Time : Sat Jan 21 00:20:33 2006
Raid Level : raid5
Raid Devices : 16777216
Total Devices : 0
Preferred Minor : 5058

Update Time : Fri Dec 13 20:45:52 1901
State : active
Active Devices : -2147483648
Working Devices : -2147483648
Failed Devices : -2147483648
Spare Devices : -2147483648

       Checksum : 80000000 - expected 2255ae37
         Events : -2147483648.-2147483648

FS on the raid is XFS.

I've been crawling through the list and noticed I could create the array again and data would still be there and I should be able to mount the fs. Am I correct?

Is this the only solution?

I've assembled this command for mdadm:

mdadm -C -l5 -n5
-c=128 /dev/md0 /dev/sdb1 /dev/sdd1 /dev/sde1 /dev/sdc1 /dev/sda1

I took the devices order from the mdadm output on a working device. Is this the way it's supposed to be the command assembled?

Is there anything alse I should consider or any other valid solution to gain access to my data?

Thanks,

Signature · 1 line
Bruno Seoane
#1 lore raw
Quoted 8 lines · Bruno Seoane

mdadm -C -l5 -n5
-c=128 /dev/md0 /dev/sdb1 /dev/sdd1 /dev/sde1 /dev/sdc1 /dev/sda1

I took the devices order from the mdadm output on a working device. Is this the way it's supposed to be the command assembled?

Is there anything alse I should consider or any other valid solution to gain access to my data?

If you create the array, it will immediately start resyncing unless you list one of the devices in your command line as "missing". Just pick one (ideally one of the ones that isn't getting picked up anyway) and put 'missing' in its place.

Using "missing" is the only way to have it be read-only in the data regions. That'll let you make a mistake and still be able to recover data after you find the right command line.

-Mike