Strange RAID behaviour when faced with user error

A user actually ran into this in the field (RHEL 5.3) and I'm able to reproduce with:
Linux vm1 2.6.30-rc8 #1 SMP Mon Jun 8 11:32:59 EDT 2009 x86_64 GNU/Linux mdadm - v2.6.7 - 6th June 2008

I'll investigate (i.e. read/debug code) when I have time but any insights would be apprecated.

1. Assemble a RAID array incorrectly (forgetting the array name):

# mdadm --assemble /dev/sdb1 /dev/sdb2 /dev/sdb3 /dev/sdb4 --run
mdadm: /dev/sdb1 has been started with 3 drives (out of 4).
(The user did not require --run; I'm not sure why.)

2. An array is actually started. That's not so weird...

# cat /proc/mdstat
Personalities : [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md127 : active raid10 sdb2[1] sdb4[3] sdb3[2]
      513792 blocks 64K chunks 2 near-copies [4/3] [_UUU]

unused devices: <none>

3. But:

# mdadm --examine /dev/sdb1
mdadm: No md superblock detected on /dev/sdb1.
# mdadm --stop /dev/sdb1
mdadm: stopped /dev/sdb1
# mdadm --examine /dev/sdb1
mdadm: No md superblock detected on /dev/sdb1.
# mdadm --assemble /dev/md0 /dev/sdb1 /dev/sdb2 /dev/sdb3 /dev/sdb4
mdadm: no recogniseable superblock on /dev/sdb1
mdadm: /dev/sdb1 has no superblock - assembly aborted

The problem goes away after a reboot.

Cheers,
Jody

The device node /dev/sdb1 was turned into a md one - probably 9,127 - when you started the array, and stopping the array hasn't reverted it. mknod /dev/sdb1 8 17 would probably have fixed it. Assuming you're using udev, the /dev/sdb1 device is recreated the next time you reboot, so the problem goes away.

I think mdadm 3.0 would have deleted the device node when the array was stopped, but mdadm 2.6.x doesn't.

Cheers,

John.

Or possibly /dev/sdb1 was turned into a symlink to /dev/md127.

Sorry, mknod /dev/sdb1 b 8 17, after removing the symlink if that's what it is.

Cheers,

John.

That's exactly right. Thanks for the insight!

brw-rw---- 1 root disk 9, 127 2009-06-10 17:54 /dev/sdb1

Actually, mdadm 3.0 is even smarter than that:

# ls -l /dev/sdb1
brw-rw---- 1 root disk 8, 17 2009-06-12 13:40 /dev/sdb1
# ./mdadm --version
mdadm - v3.0 - 2nd June 2009
# ./mdadm --assemble --verbose /dev/sdb1 /dev/sdb2 /dev/sdb3 /dev/sdb4
mdadm: device /dev/sdb1 exists but is not an md array.

Cheers,
Jody

How do I resize my volume group with lvm2?

vgextend needs new devices but I only have one device md0 for all my physical volumes.

I know how to resize my logical volumes under the volume group, but somehow the volume group st1500 did not extend after raid5 reshape with mdadm?

Is my only option backing up 8TB of data and creating a new volume group?

I dont want to add sd[ai] here with vgextend as lvm2 is on top of mdadm so I guess I should have only one device for my VG?

/Michael Ole Olsen

mfs:~# fdisk -l /dev/md0

Disk /dev/md0: 12002.4 GB, 12002414559232 bytes
2 heads, 4 sectors/track, -1364690304 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000

Disk /dev/md0 doesn't contain a valid partition table

mfs:~# vgdisplay
  --- Volume group ---
  VG Name               st1500
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               1
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               8,19 TB
  PE Size               4,00 MB
  Total PE              2146198
  Alloc PE / Size       2104832 / 8,03 TB
  Free  PE / Size       41366 / 161,59 GB
  VG UUID               1Rl7ly-OguV-fEbS-TU8F-7tdM-9YH3-wKc9F6

lvm> help vgextend
vgextend: Add physical volumes to a volume group

vgextend

        [-A|--autobackup y|n]
        [-d|--debug]
        [-h|--help]
        [-t|--test]
        [-v|--verbose]

[--version]
VolumeGroupName PhysicalDevicePath [PhysicalDevicePath...]

Command failed with status code 0.
lvm> vgextend st1500 /dev/md0
Physical volume '/dev/md0' is already in volume group 'st1500' Unable to add physical volume '/dev/md0' to volume group 'st1500'.

Guess I was too quick with that one..

pvresize /dev/md0 was all that was needed, no need for restart :-)

pvresize is the command you're looking for, I think.

Tapani Tarvainen