md/mdadm fails to properly run on 2.6.15 after upgrading from 2.6.11
Hi,
(I just subscribed, sorry if this is a dupe. I did try to match the subject from the archives, but couldn't find any...)
I ran into trouble after upgrading a Debian Sarge system from 2.6.11 to 2.6.15. To be more precise, it turned out that md/mdadm seems to not function properly during the boot process of 2.6.15.
My /etc/mdadm/mdadm.conf contains this:
Quoted 1 line
Quoted 1 line
Quoted 1 line
---[mdadm.conf]---
DEVICE /dev/hdi1 /dev/hdg1 /dev/hdc1
ARRAY /dev/md1 level=raid5 num-devices=3 UUID=09c58ab6:f706e37b:504cf890:1a597046 devices=/dev/hdi1,/dev/hdg1,/dev/hdc1
DEVICE /dev/hdg2 /dev/hdc2
ARRAY /dev/md2 level=raid1 num-devices=2 UUID=86210844:6abbf533:dc82f982:fe417066 devices=/dev/hdg2,/dev/hdc2
DEVICE /dev/hda2 /dev/hdb2
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=da619c37:6c072dc8:52e45423:f4a58b7c devices=/dev/hda2,/dev/hdb2
DEVICE /dev/hda1 /dev/hdb1
ARRAY /dev/md4 level=raid1 num-devices=2 UUID=bfc30f9b:d2c21677:c4ae5f90:b2bddb75 devices=/dev/hda1,/dev/hdb1
DEVICE /dev/hdc3 /dev/hdg3
ARRAY /dev/md3 level=raid1 num-devices=2 UUID=fced78ce:54f00a78:8662e7eb:2ad01d0b devices=/dev/hdc3,/dev/hdg3
Quoted 1 line
Quoted 1 line
Quoted 1 line
---[/mdadm.conf]---
On 2.6.11, it booted (and still boots) correctly. The interesting parts from the boot-sequence are:
Quoted 1 line
Quoted 1 line
Quoted 1 line
---[2.6.11 dmesg]---
md: md driver 0.90.1 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: raid1 personality registered as nr 3
[...]
md: md0 stopped.
md: bind<hdb2>
md: bind<hda2>
[...]
md: md1 stopped.
md: bind<hdg1>
md: bind<hdc1>
md: bind<hdi1>
raid5: automatically using best checksumming function: pIII_sse
pIII_sse : 3872.000 MB/sec
raid5: using function: pIII_sse (3872.000 MB/sec)
md: raid5 personality registered as nr 4
raid5: device hdi1 operational as raid disk 0
raid5: device hdc1 operational as raid disk 2
raid5: device hdg1 operational as raid disk 1
raid5: allocated 3161kB for md1
raid5: raid level 5 set md1 active with 3 out of 3 devices, algorithm 2 RAID5 conf printout:
--- rd:3 wd:3 fd:0
disk 0, o:1, dev:hdi1
disk 1, o:1, dev:hdg1
disk 2, o:1, dev:hdc1
md: md2 stopped.
md: bind<hdc2>
md: bind<hdg2>
raid1: raid set md2 active with 2 out of 2 mirrors
md: md4 stopped.
md: bind<hdb1>
md: bind<hda1>
raid1: raid set md4 active with 2 out of 2 mirrors
md: md3 stopped.
md: bind<hdg3>
md: bind<hdc3>
raid1: raid set md3 active with 2 out of 2 mirrors
Quoted 1 line
Quoted 1 line
Quoted 1 line
---[/2.6.11 dmesg]---
This all looks great and is as expected by the mdadm.conf file. The bootlog daemon continues to report ordinary things such as:
Quoted 1 line
Quoted 1 line
Quoted 1 line
---[2.6.11 bootlog]---
Sat Apr 8 16:47:53 2006: bootlogd. Sat Apr 8 16:47:53 2006: Setting parameters of disc: (none). Sat Apr 8 16:47:53 2006: Activating swap. Sat Apr 8 16:47:53 2006: Checking root file system... Sat Apr 8 16:47:53 2006: fsck 1.37 (21-Mar-2005) Sat Apr 8 16:47:53 2006: /: clean, 122183/524288 files, 508881/1048576 blocks [...] Sat Apr 8 14:47:55 2006: Creating device-mapper devices...done. Sat Apr 8 14:47:55 2006: Creating device-mapper devices...done. Sat Apr 8 14:47:56 2006: Starting raid devices: mdadm-raid5: Sat Apr 8 14:47:56 2006: mdadm: /dev/md1 has been started with 3 drives. Sat Apr 8 14:47:56 2006: mdadm: /dev/md2 has been started with 2 drives. Sat Apr 8 14:47:56 2006: mdadm: /dev/md4 has been started with 2 drives. Sat Apr 8 14:47:56 2006: mdadm: /dev/md3 has been started with 2 drives. Sat Apr 8 14:47:56 2006: done. Sat Apr 8 14:47:56 2006: Setting up LVM Volume Groups... Sat Apr 8 14:47:57 2006: Reading all physical volumes. This may take a while... Sat Apr 8 14:47:58 2006: Found volume group "vg" using metadata type lvm2 Sat Apr 8 14:47:58 2006: 2 logical volume(s) in volume group "vg" now active Sat Apr 8 14:47:58 2006: Checking all file systems... Sat Apr 8 14:47:58 2006: fsck 1.37 (21-Mar-2005) Sat Apr 8 14:47:58 2006: /dev/md4: clean, 54/48192 files, 43630/192640 blocks Sat Apr 8 14:47:58 2006: /dev/mapper/vg-home: clean, 7560/219520 files, 120502/438272 blocks Sat Apr 8 14:47:58 2006: /dev/md1: clean, 38614/9781248 files, 15097260/19539008 blocks Sat Apr 8 14:47:58 2006: /dev/md2: clean, 18/7325696 files, 8634921/14651264 blocks Sat Apr 8 14:47:58 2006: /dev/md3: clean, 2079183/7094272 files, 10865102/14185376 blocks Sat Apr 8 14:47:58 2006: /dev/hde1: clean, 74/28640 files, 26855696/29296527 blocks Sat Apr 8 14:47:58 2006: /dev/hde2: clean, 573/9781248 files, 13186560/19543072 blocks Sat Apr 8 14:47:58 2006: Setting kernel variables ... Sat Apr 8 14:47:58 2006: ... done. Sat Apr 8 14:47:59 2006: Mounting local filesystems... Sat Apr 8 14:47:59 2006: /dev/md4 on /boot type ext3 (rw) Sat Apr 8 14:47:59 2006: /dev/mapper/vg-home on /home type ext3 (rw) Sat Apr 8 14:47:59 2006: /dev/md1 on /mnt/raid5 type ext3 (rw) Sat Apr 8 14:47:59 2006: /dev/md2 on /mnt/others2 type ext3 (rw) Sat Apr 8 14:47:59 2006: /dev/md3 on /mnt/others type ext3 (rw) Sat Apr 8 14:47:59 2006: proc on /mnt/others/sid-chrooted/proc type proc (rw) Sat Apr 8 14:47:59 2006: /dev/hde1 on /mnt/vmsdata type ext3 (rw) Sat Apr 8 14:47:59 2006: /dev/hde2 on /mnt/vms type ext3 (rw) Sat Apr 8 14:47:59 2006: Cleaning /tmp /var/run /var/lock.
Quoted 1 line
Quoted 1 line
Quoted 1 line
---[/2.6.11 bootlog]---
Again, this all looks great.
But...
now...
booting 2.6.15 leads to a disaster.
Quoted 1 line
Quoted 1 line
Quoted 1 line
---[2.6.15 dmesg]---
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
md: raid1 personality registered as nr 3
raid5: automatically using best checksumming function: pIII_sse
pIII_sse : 3916.000 MB/sec
raid5: using function: pIII_sse (3916.000 MB/sec)
md: raid5 personality registered as nr 4
md: md0 stopped.
md: bind<hdb1>
md: bind<hda1>
raid1: raid set md0 active with 2 out of 2 mirrors
md: md1 stopped.
md: bind<hdb2>
md: bind<hda2>
raid1: raid set md1 active with 2 out of 2 mirrors
md: md2 stopped.
md: bind<hdg1>
md: bind<hdc1>
md: bind<hdi1>
raid5: device hdi1 operational as raid disk 0
raid5: device hdc1 operational as raid disk 2
raid5: device hdg1 operational as raid disk 1
raid5: allocated 3162kB for md2
raid5: raid level 5 set md2 active with 3 out of 3 devices, algorithm 2 RAID5 conf printout:
--- rd:3 wd:3 fd:0
disk 0, o:1, dev:hdi1
disk 1, o:1, dev:hdg1
disk 2, o:1, dev:hdc1
md: md3 stopped.
md: bind<hdc2>
md: bind<hdg2>
raid1: raid set md3 active with 2 out of 2 mirrors
md: md4 stopped.
md: bind<hdg3>
md: bind<hdc3>
raid1: raid set md4 active with 2 out of 2 mirrors
device-mapper: 4.4.0-ioctl (2005-01-12) initialised: [email protected]
Quoted 1 line
Quoted 1 line
Quoted 1 line
---[/2.6.15 dmesg]---
As you might already have noticed, md0 does NOT get /dev/hda2 and /dev/hdb2 attached, but /dev/hda1 and /dev/hdb1! Same goes for md1, md2, md3 and md4. They all get wrong partitions.
Things get even worse further on:
Quoted 1 line
Quoted 1 line
Quoted 1 line
---[2.6.15 bootlog]---
Sat Apr 8 16:36:23 2006: bootlogd. Sat Apr 8 16:36:23 2006: Setting parameters of disc: (none). Sat Apr 8 16:36:23 2006: Activating swap. Sat Apr 8 16:36:23 2006: Checking root file system... Sat Apr 8 16:36:23 2006: fsck 1.37 (21-Mar-2005) Sat Apr 8 16:36:23 2006: /: clean, 122181/524288 files, 508826/1048576 blocks [...] Sat Apr 8 14:36:28 2006: Creating device-mapper devices...done. Sat Apr 8 14:36:28 2006: Creating device-mapper devices...done. Sat Apr 8 14:36:28 2006: Starting raid devices: mdadm-raid5: done. Sat Apr 8 14:36:28 2006: Setting up LVM Volume Groups... Sat Apr 8 14:36:29 2006: Reading all physical volumes. This may take a while... Sat Apr 8 14:36:29 2006: Found volume group "vg" using metadata type lvm2 Sat Apr 8 14:36:29 2006: 2 logical volume(s) in volume group "vg" now active Sat Apr 8 14:36:30 2006: Checking all file systems... Sat Apr 8 14:36:30 2006: fsck 1.37 (21-Mar-2005) Sat Apr 8 14:36:30 2006: /dev/md4: clean, 2079183/7094272 files, 10865102/14185376 blocks Sat Apr 8 14:36:30 2006: /dev/mapper/vg-home: clean, 7560/219520 files, 120502/438272 blocks Sat Apr 8 14:36:30 2006: /: Note: if there is several inode or block bitmap blocks Sat Apr 8 14:36:30 2006: which require relocation, or one part of the inode table Sat Apr 8 14:36:30 2006: which must be moved, you may wish to try running e2fsck Sat Apr 8 14:36:30 2006: with the '-b 32768' option first. The problem may lie only Sat Apr 8 14:36:30 2006: with the primary block group descriptor, and the backup Sat Apr 8 14:36:30 2006: block group descriptor may be OK. Sat Apr 8 14:36:30 2006: Sat Apr 8 14:36:30 2006: /: Block bitmap for group 0 is not in group. (block 1852402720) Sat Apr 8 14:36:30 2006: Sat Apr 8 14:36:30 2006: Sat Apr 8 14:36:30 2006: /: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. Sat Apr 8 14:36:30 2006: Sat Apr 8 14:36:30 2006: /dev/md2: clean, 38614/9781248 files, 15097260/19539008 blocks Sat Apr 8 14:36:30 2006: /dev/md3: clean, 18/7325696 files, 8634921/14651264 blocks Sat Apr 8 14:36:30 2006: /dev/hde1: clean, 74/28640 files, 26855696/29296527 blocks Sat Apr 8 14:36:30 2006: /dev/hde2: clean, 573/9781248 files, 13186560/19543072 blocks Sat Apr 8 14:36:30 2006: Sat Apr 8 14:36:30 2006: fsck failed. Please repair manually. Sat Apr 8 14:36:30 2006: Sat Apr 8 14:36:30 2006: CONTROL-D will exit from this shell and continue system startup. Sat Apr 8 14:36:30 2006: Sat Apr 8 14:36:30 2006: Give root password for maintenance Sat Apr 8 14:36:30 2006: (or type Control-D to continue):
Quoted 1 line
Quoted 1 line
Quoted 1 line
---[/2.6.15 bootlog]---
Okay, just pressing Control-D continues the boot process and AFAIK the root filesystemen actually isn't corrupt. Running e2fsck returns no errors and booting 2.6.11 works just fine, but I have no clue why it picked the wrong partitions to build md[01234].
What could have happened here?
Thanks!
Quoted 15 lines · Marc L. de Bruin
Hi,
(I just subscribed, sorry if this is a dupe. I did try to match the subject from the archives, but couldn't find any...)
I ran into trouble after upgrading a Debian Sarge system from 2.6.11 to 2.6.15. To be more precise, it turned out that md/mdadm seems to not function properly during the boot process of 2.6.15.
My
/etc/mdadm/mdadm.confcontains this:Quoted 1 line
Quoted 1 line
Quoted 1 line
---[mdadm.conf]---
DEVICE
/dev/hdi1/dev/hdg1/dev/hdc1
ARRAY/dev/md1level=raid5 num-devices=3 UUID=09c58ab6:f706e37b:504cf890:1a597046 devices=/dev/hdi1,/dev/hdg1,/dev/hdc1DEVICE
/dev/hdg2/dev/hdc2
ARRAY/dev/md2level=raid1 num-devices=2 UUID=86210844:6abbf533:dc82f982:fe417066 devices=/dev/hdg2,/dev/hdc2
Quoted 9 lines
DEVICE
/dev/hda2/dev/hdb2
ARRAY/dev/md0level=raid1 num-devices=2 UUID=da619c37:6c072dc8:52e45423:f4a58b7c devices=/dev/hda2,/dev/hdb2DEVICE
/dev/hda1/dev/hdb1
ARRAY/dev/md4level=raid1 num-devices=2 UUID=bfc30f9b:d2c21677:c4ae5f90:b2bddb75 devices=/dev/hda1,/dev/hdb1DEVICE
/dev/hdc3/dev/hdg3
ARRAY/dev/md3level=raid1 num-devices=2 UUID=fced78ce:54f00a78:8662e7eb:2ad01d0b devices=/dev/hdc3,/dev/hdg3Quoted 1 line
Quoted 1 line
Quoted 1 line
---[/mdadm.conf]---
replace all
DEVICE .....
lines
with a single
DEVICE partitions
remove all the device=... part from the array lines.
L.
On Sun, 9 Apr 2006, Marc L. de Bruin wrote:
...
Quoted 6 lines
Okay, just pressing Control-D continues the boot process and AFAIK the root filesystemen actually isn't corrupt. Running e2fsck returns no errors and booting 2.6.11 works just fine, but I have no clue why it picked the wrong partitions to build md[01234].
What could have happened here?
i didn't know sarge had 2.6.11 or 2.6.15 packages... but i'm going to assume you've installed one of initramfs-tools or yaird in order to use the unstable 2.6.11 or 2.6.15 packages... so my comments might not apply.
initramfs-tools generates an "mdrun /dev" which starts all the raids it can find... but does not include the mdadm.conf in the initrd so i'm not sure it will necessarily start them in the right minor devices. try doing an "mdadm --examine /dev/xxx" on some of your partitions to see if the "preferred minor" is what you expect it to be...
if the preferred minors are wrong there's some mdadm incantation to update them... see the man page.
or switch to yaird (you'll have to install yaird and purge initramfs-tools) and dpkg-reconfigure your kernel packages to cause the initrds to be rebuilt. yaird starts only the raid required for the root filesystem, and specifies the correct minor for it. then later after the initrd /etc/init.d/mdadm-raid will start the rest of your raids using your mdadm.conf.
-dean
Quoted 10 lines · dean gaudet
On Sun, 9 Apr 2006, Marc L. de Bruin wrote:
...
Quoted 5 lines
Okay, just pressing Control-D continues the boot process and AFAIK the root filesystemen actually isn't corrupt. Running e2fsck returns no errors and booting 2.6.11 works just fine, but I have no clue why it picked the wrong partitions to build md[01234].
Quoted 7 lines
Quoted 3 lines
What could have happened here?
i didn't know sarge had 2.6.11 or 2.6.15 packages... but i'm going to assume you've installed one of initramfs-tools or yaird in order to use the unstable 2.6.11 or 2.6.15 packages... so my comments might not apply.
You are right about this. Sarge does not contain 2.6.11 or 2.6.15. 'linux-image-2.6.15-1-686-smp' depends on "initramfs-tools | yaird | linux-initramfs-tool". I picked initramfs-tools, but in the future I might go for yaird because initramfs-tools depends on udev and I may not want to lose hotplug.
Quoted 3 lines
initramfs-tools generates an "mdrun /dev" which starts all the raids it can find... but does not include the mdadm.conf in the initrd so i'm not sure it will necessarily start them in the right minor devices. try doing an "mdadm --examine
/dev/xxx" on some of your partitions to see if the "preferred minor" is what you expect it to be...
blah@frodo:~# sudo mdadm --examine /dev/md[01234] mdadm: No super block found on /dev/md0 (Expected magic a92b4efc, got 5e4cff37) mdadm: No super block found on /dev/md1 (Expected magic a92b4efc, got 940922e3) mdadm: No super block found on /dev/md2 (Expected magic a92b4efc, got 00000000) mdadm: No super block found on /dev/md3 (Expected magic a92b4efc, got 006c038d) mdadm: No super block found on /dev/md4 (Expected magic a92b4efc, got 00000000) blah@frodo:~#
Is this a problem (for 2.6.15)? The system seems to work without a problem, at least with 2.6.11...
Quoted 4 lines
if the preferred minors are wrong there's some mdadm incantation to update them... see the man page.
or switch to yaird (you'll have to install yaird and purge initramfs-tools) and dpkg-reconfigure your kernel packages to cause the initrds to be rebuilt. yaird starts only the raid required for the root filesystem, and specifies the correct minor for it. then later after the initrd
/etc/init.d/mdadm-raidwill start the rest of your raids using your mdadm.conf.
Yeah, this is what is happening today with 2.6.11 as well.
Thanks for your time,
Marc.
Quoted 9 lines · Marc L. de Bruin
Quoted 6 lines · dean gaudet
initramfs-tools generates an "mdrun /dev" which starts all the raids it can find... but does not include the mdadm.conf in the initrd so i'm not sure it will necessarily start them in the right minor devices. try doing an "mdadm --examine
/dev/xxx" on some of your partitions to see if the "preferred minor" is what you expect it to be...blah@frodo:~# sudo mdadm --examine /dev/md[01234]
try running it on /dev/sda1 or whatever the component devices are for your array... not on the array devices.
-dean
Quoted 10 lines · dean gaudet
Quoted 8 lines · Marc L. de Bruin
Quoted 6 lines · dean gaudet
initramfs-tools generates an "mdrun /dev" which starts all the raids it can find... but does not include the mdadm.conf in the initrd so i'm not sure it will necessarily start them in the right minor devices. try doing an "mdadm --examine
/dev/xxx" on some of your partitions to see if the "preferred minor" is what you expect it to be...
Quoted 5 lines
Quoted 1 line
blah@frodo:~# sudo mdadm --examine /dev/md[01234]try running it on
/dev/sda1or whatever the component devices are for your array... not on the array devices.-dean
Briljant, that turned out to work.
However, all "preferred minor"s are correct, meaning that the output is in sync with what I expected it to be from /etc/mdadm/mdadm.conf.
Any other ideas? Just adding /etc/mdadm/mdadm.conf to the initrd does not seem to work, since mdrun seems to ignore it?!
Marc.
Quoted 5 lines · Marc L. de Bruin
However, all "preferred minor"s are correct, meaning that the output is in sync with what I expected it to be from
/etc/mdadm/mdadm.conf.Any other ideas? Just adding
/etc/mdadm/mdadm.confto the initrd does not seem to work, since mdrun seems to ignore it?!
yeah it looks like "mdrun /dev" just seems to assign things in the order they're discovered without consulting the preferred minor.
it seems to me "mdrun /dev" is about the worst thing possible to use in an initrd.
i opened a bug yesterday
<http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=361674> ... it seems really they should stop using mdrun entirely... when i get a chance i'll try updating the bug (or go ahead and add your own experiences to it).
oh hey take a look at this bug for debian mdadm package
<http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=354705> ... he intends to deprecate mdrun.
-dean
Quoted 9 lines · dean gaudet
Quoted 5 lines · Marc L. de Bruin
However, all "preferred minor"s are correct, meaning that the output is in sync with what I expected it to be from
/etc/mdadm/mdadm.conf.Any other ideas? Just adding
/etc/mdadm/mdadm.confto the initrd does not seem to work, since mdrun seems to ignore it?!it seems to me "mdrun /dev" is about the worst thing possible to use in an initrd.
:-)
I guess I'll have to change to yaird asap then. I can't think of any other solid solution...
Marc.
Quoted 10 lines · Marc L. de Bruin
Quoted 8 lines · dean gaudet
Quoted 6 lines · Marc L. de Bruin
However, all "preferred minor"s are correct, meaning that the output is in sync with what I expected it to be from
/etc/mdadm/mdadm.conf.Any other ideas? Just adding
/etc/mdadm/mdadm.confto the initrd does not seem
to work, since mdrun seems to ignore it?!
Quoted 7 lines
Quoted 2 lines
it seems to me "mdrun /dev" is about the worst thing possible to use in an initrd.
:-)
I guess I'll have to change to yaird asap then. I can't think of any other solid solution...
yeah i've been yaird... it's not perfect -- take a look at
<http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=351183> for a patch i use to improve the ability of a yaird initrd booting when you've moved devices or a device has failed.
-dean