Linux RAID autodetect partitions go missing from /dev, but fdisk can see them

Hi all,

I have a very strange problem that I've been trying to debug for days now. I'm hoping someone on this list may have ran into this before, or have any tips on how I can continue debugging this, because I have to admit I'm a little lost...

I had a RAID5 with four drives and one spare, /dev/sd[bcde]1 + /dev/sdf1, and everything was working fine, until one day one of the drives in the array (sdb) no longer had a partition (sdb1). Letting the spare take over I ignored this for a few days, but then it happened again, this time with sdc. Kernel is 2.6.22.17, and I've compiled all raid support in. The symptoms are:

- The kernel seems to detect the partitions (lines 396 and 407 in the
dmesg [1])

- But once the boot process finishes and the RAID is started, there is
no longer any sdc1 or sdb1, so the RAID fails to start (lines 550-576 in dmesg [1])

- Running fdisk -l shows that the drives in question (sdb and sdc) do
have similar partitions as the other working drives, namely one Linux RAID autodetect partition each (see command output [2])

- But, the partitions are missing from /proc/partitions (see [3])

- Manually adding device nodes using mknod works, but doing file -sL
on the device gives "writable, no read permission", even though permissions are the same as the other sd* nodes in /dev

- Running 'partprobe -s' successfully finds the two missing partitions
and adds device nodes, and the nodes can be 'file -sL'ed, but when trying to assemble the array again with these new nodes in the system, I'm told that sdc1 is not found, and after the --assemble is done, the device nodes are once again missing (!) see [4]

- I've tried using the 'dmraid' command to look for fakeraid
partitions or meta data on the drives, which I was told could mess up the auto-detection of Linux software ride partitions, but could not find any issues.

As you can tell I've exhausted all my current options, so any help on what I could try next would be very much appreciated. I am especially curious as to why I lose the partitions when mdadm tries to assemble the array?

Thanks!

Tor Arne Vestbø

[1] http://pastebin.com/m15b9c275   dmesg
[2] http://pastebin.com/f50fb323a   fdisk -l
[3] http://pastebin.com/f4547c2ca   cat /proc/partitions
[4] http://pastebin.com/m4475c9ae   partprobe + mdadm --assemble

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Yes, it does sound rather weird.
Can you:

mdadm -Esv

and
mdadm --stop /dev/md0
strace -o /tmp/str -s 200 mdadm --assemble --scan --verbose /dev/md0

and send me the output and /tmp/str.
Also the contents of /etc/mdadm.conf might help.

Thanks,
NeilBrown

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Neil!

First of all, thank you so much for helping me out with this, as I'm still very lost :)

In addition to the things listed in the first e-mail, I've also tried installing the latest kernel from kernel.org, but that did not solve anything. Also, in case it's relevant, I'm running openSUSE 10.3.

http://pastebin.com/d7b14d14e

For some reason it seems to think that /dev/sdc and /dev/sdb are part of the array, while it really is /dev/sdc1 and /dev/sdb1. I'm guessing since they are missing somehow from the device nodes in /dev mdadm assumes the disk itself is the member?

http://pastebin.com/f2c1db2e4

The original array had sd[bcde]1 + sdf1 as spare. Then sdb1 went missing and the spare kicked in, and then sdc1 went missing, leaving me with a degraded array.

http://pastebin.com/f573346ef

Is there anything else I can run, cat, and/or paste that would shed light over what's going on?

Thank _you_ :)

Tor Arne

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

FYI, I decided to delete the Linux auto-detect partition on /dev/sdb so I could re-create it and add /dev/sdb1 to the array again, so I at least have four drives and not a degraded array (don't want to lose all my data).

This worked fine; after fdisk'ing /dev/sdb, deleting the partition and recreating it with the same parameters is showed up as /dev/sdb1, and I was able to add it to the array using mdadm /dev/md0 -a /dev/sdb1. The array is now rebuilding.

We still have /dev/sdc1 as a test-case though, so the original problem is still there, just not for two drives but for one. I would really like to figure out why it's happening, so that suddenly it does not happen to two drives at the same time, rendering my array dead :/

Tor Arne
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi again,

Sorry for the delay (Christmas vacation, ++). I still have the problem though.

My attempt to re-create /dev/sdb1 by fdisk'ing it again and adding it to the array failed. That is, mdadm started re-constructing the array, and that succeeded, but when I rebooted /dev/sdb1 was missing again :/

I think you are on to something here! I did partition all my disks with just one partition, starting on the first block and taking up all the remaining space, and set it to type Linux RAID auto detect.

Here's the output of the fdisk -pu and mdmadm -E commands before running blockdev:

    http://pastebin.com/f4cad67ea

    http://pastebin.com/f34be2b86

It seems that mdadm thinks sd[bc] are part of the array directly somehow, but for sd[def] the partition itself is in the array, which I guess supports your suspicion?

Here's the output after running blockdev --rereadpt /dev/sdc :

http://pastebin.com/f725ee0c3

Suddenly mdadm -E /dev/sdc* shows info for both sdc and sdc1, and guess what? /dev/sdc1 is back as a device node :)

So is this a case of the problem you described about superblock-confusion? If so, is it fixable?

Thank you so much for your help in this! I really do feel I'm one big step closer to solving this!

Tor Arne

Btw, I cloned mdadm from your site, so I have an updated version now. I haven't installed it yet, in case it can damage the existing RAID, but if there are any tests I can run with the updated version I'm ready for them :)

Tor Arne
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

[snip]

It appears it is :)

For googlability, here's what I did:

Changed the DEVICE line in my mdadm.conf from:

DEVICE partitions

to

DEVICE /dev/sd[bcdef]1

Now the array is assembled correctly at boot.

I'm still confused why the device nodes went missing as a result of the failed assembly earlier, but that's a minor issue I can live with :)

Tor Arne
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

A safer way of acheiving this is to list the array in mdadm.conf by UUID, which will guarantee all the correct devices are used - sd devices can move around under some conditions.

eg.

ARRAY /dev/md1 level=raid1 num-devices=2 UUID=f6bd7495:52288189:40f44282:1c220686 ARRAY /dev/md0 level=raid1 num-devices=2 UUID=f7d720fe:a5d0724c:c9bd10fa:a496ed51

Note each ARRAY entry and UUID are entered as a single line.

Regards,

Richard
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Should I try this after doing partprobe first? In that case /dev/sdb1 and /dev/sdc1 exist, and we might see from the trace what causes them to disappear?

Tor Arne