switching root fs '/' to boot from RAID1 with grub

Hello,

I have and olde HDD and two new HDDs:

- hda1 - my current root filesystem '/'
- sda1 - part of raid1 /dev/md0 [U_U]
- hdc1 - part of raid1 /dev/md0 [U_U]

I want all hda1, sda1, hdc1 to be a raid1. I remounted hda1 readonly then I did 'dd if=/dev/hda1 of=/dev/md0'. I carefully checked that the partition sizes match exactly. So now md0 contains the same thing as hda1.

But hda1 is still outside of the array. I want to add it to the array. But before I do this I think that I should boot from /dev/md0 ? Otherwise I might hose this system. I tried grub-install /dev/sda1 (assuming that grub would see no problem with reading raid1 partition, and boot from it, until mdadm detects an array). I tried grub-install /dev/sda as well as on /dev/hdc and /dev/hdc1. I turned off 'active' flag for partition hda1 and turned it on for hdc1 and sda1. But still grub is booting from hda1.

I did all this with version 1.1

mdadm --create --verbose /dev/md0 --chunk=64 --level=raid1 \
      --metadata=1.1  --bitmap=internal --raid-devices=3 /dev/sda1 \
      missing /dev/hdc1

I'm NOT using LVM here.

Can someone tell me how should I switch grub to boot from /dev/md0 ?

After the boot I will add hda1 to the array, and all three partitions should become a raid1.

Janek Kozicki                                                         |

Janek Kozicki said: (by the date of Tue, 30 Oct 2007 21:07:21 +0100)

in fact, to check the size I was using 'fdisk -l' because it gives size in bytes (not in blocks), like this:

backup:~# fdisk -l /dev/md0

Disk /dev/md0: 1003 MB, 1003356160 bytes

And the same for /dev/hda1

But that's a detail, just so you know that I dd'ed my root partition correctly and I can mount /dev/md0 without problems.

Janek Kozicki                                                         |

Well, without going into a lot of detail, you would have to boot from /dev/hda1 and specify a root=/dev/md0 option to the kernel to actually boot to the new / filesystem before grub-install will do what you are expecting. The fact that hda1 is mounted as / and that hda1 contains /boot with all your kernels and initrd images means that when you run grub-install it looks up the current location of the /boot files, sees they are on /dev/hda1, and regardless of where you put the boot sector (sda1, hdc1), those sectors point to the files grub found in /boot which are on hda1.

Which won't work, and you'll see that as soon as you have md0 mounted as / and try to run grub-install again.

Grub doesn't work with version 1.1 superblocks at the moment. It could be made to work quick and dirty in a short period of time, making it work properly would take longer.

So, here's what I would do in your case. Scrap the current /dev/md0 setup. Make a new /dev/md0 using a version 1.0 superblock with all the other options the same as before. BTW, your partition sizes don't need to match exactly. If the new device is larger than your /dev/hda1, then no big deal, just do the dd like you did before and when you are done you can resize the fs to fit the device. If the new device is slightly smaller than /dev/hda1, then just run resizefs to shrink your /dev/hda1 to the same size as the fs on the /dev/md0 *before* you do the dd from hda1 to md0. Once you have the data copied to /dev/md0, you'll need to reboot the system and this time specify /dev/md0 as your root device (you may need to remake your initrd before you reboot, I don't know if your initrd starts /dev/md0, but it needs to). Once you are running with md0 as your root partition, you need to run grub to install the boot sectors on the md0 devices. You can't use grub-install though, it gets it wrong. Run the grub command, then enter these commands:

device /dev/sda (hd0)
root (hd0,0)
install --stage2=/boot/grub/stage2 /boot/grub/stage1 (hd0) /boot/grub/e2fs_stage1_5 p /boot/grub/stage2 /boot/grub/menu.lst device /dev/hdc (hd0)
root (hd0,0)
install --stage2=/boot/grub/stage2 /boot/grub/stage1 (hd0) /boot/grub/e2fs_stage1_5 p /boot/grub/stage2 /boot/grub/menu.lst

That will install grub on the master boot record of hdc and sda, and in both cases grub will look to whatever drive it is running on for the files to boot instead of going to a specific drive.

Next you need to modify the /etc/grub.conf and change all the root= lines to be root=/dev/md0, and you need to modify /etc/fstab the same way. Then you probably need to remake all the initrd images so that they contain the update.

Once you've done that, shut the system down, remove /dev/hda from the machine entirely, move /dev/hdc to /dev/hda, then reboot. The system should boot up to your raid array just fine. If it doesn't work, you can always put your old hda back in and boot up from it. If it does work, shut the machine down one more time, put the old hda in as hdc, boot back up (which should boot from hda to the md0 root, it should not touch hdc), add hdc to the raid array, let it resync, and then the final step is to run the grub install on hdc to make it match the other two disks. After that, you have a fully functional and booting raid1 array.

Doug Ledford <[email protected]>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband

Thanks!

It all worked really great, I had to correct only one typo:

into this:

device (hd0) /dev/sda

I have debian etch here, and I didn't have to rebuild initrd kernel. Also I didn't have to disconnect hda drive, I just selected a different boot device during boot (without even going into bios, or opening the case).

In fact.. the only "trouble" I had here is that I needed to connect a PS2 keyboard to this box ;-) (it's a just a backup machine)

And now I have a full RAID1 array. Now just two questions:

1. when I shutdown -r now I see a worrying message at the end:

Stopping array md2 .... done (stopped)
Stopping array md1 .... done (stopped)
Stopping array md0 .... failed (busy)
Will now reboot
md: Stopping all md devices
md: md0 still in use

<reboots>

Is that ok ?

2. Will grub update all drives automatically, for instance when I will upgrade kernel by 'aptitude upgrade'? Or do I need to repeat your grub instructions each time a new kernel is installed?

thanks again!

Janek Kozicki                                                         |

Yes. That's typical for the root device because root is never unmounted prior to shutdown. The messages are probably more worrying than they need to be. The system should have successfully switched the array to read only mode at the first attempt to stop the array. Neil, any chance on getting the messages for the / device to be less worrisome?

Now that grub's installed, you won't have to do anything manual again. The only time you might have to repeat that grub install procedure is if you loose a drive and need to add a new one back in, then the new one will need it.

Doug Ledford <[email protected]>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband

Doug Ledford said: (by the date of Wed, 31 Oct 2007 13:38:08 -0400)

great! many thanks again.

Another thing..

I'm using xosview to monitor my system activity (others prefer gkremml, or sth else ;-). To see RAID I can run xosview like this:

xosview -xrm "xosview*RAID:true" -xrm "xosview*RAIDdevicecount:2"

but I have three devices (md0, md1, md2), so I should use RAIDdevicecount:3 but it gives following error:

terminate called after throwing an instance of 'std::bad_alloc'
what(): St9bad_alloc
Aborted

anybody else here is using xosview?

Janek Kozicki                                                         |

funny you should mention that :)
I submitted a bug report in Mar 2005

It just got some attention a couple of weeks back:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=+300924

David

No, it won't... it'll look for the first drive in the system (BIOS drive 80h). This means that if the BIOS can see the bad drive, but it doesn't work, you're still screwed.

-hpa

Yes, and except for some fantastic BIOS I've never heard of, the drive that the BIOS reads the boot sector from is always the 0x80 drive. This is either because the drive truly is the first drive, or because the BIOS is remapping a later drive to 0x80 for boot purposes. In either case, what I said is still true: the boot sector will look to read the data files from the drive the boot sector itself was read from.

Correct, and that's what you want. The alternative is that if the BIOS can see the first disk but it's broken and can't be used, and if you have the boot sector on the second disk set to read from BIOS disk 0x81 because you ASSuMEd the first disk would be broken but still present in the BIOS tables, then your machine won't boot unless that first dead but preset disk is present. If you remove the disk entirely, thereby bumping disk 0x81 to 0x80, then you are screwed. If you have any drive failure that prevents the first disk from being recognized (blown fuse, blown electronics, etc), you are screwed until you get a new disk to replace it.

Follow these simple rules when setting up boot sectors and you'll be OK:

1) If you are using RAID1, then a boot sector should *never* try and read data from anything other than the disk the boot sector is on. To do otherwise defeats the whole purpose of RAID1 which is that you only need 1 disk to survive in order for the array to survive.
2) If the BIOS runs any given MBR in the RAID array, then that MBR will be on the disk the BIOS has mapped to 0x80.
3) While there are failure scenarios that would leave a disk unusable but still visible to the OS, there are no magic BIOS switches to fake a totally dead device. So, since you can remove a defunct but present disk in order to allow disk B to become disk A, but you can't magic a new disk A out of thin air should it fail to the point of not being recognized, set all your raid boot sectors to think they are the first disk in the system and you will always be able to start your machine.

So, what I said is true, the MBR will search on the disk it is being run from for the files it needs: 0x80.

Doug Ledford <[email protected]>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband

What you want is for it to use the drive number that BIOS passes into it (register DL), not a hard-coded number. That was my (only) point -- you're obviously right that hard-coding a number to 0x81 would be worse than useless.

-hpa

Oh, and I forgot to mention that in grub2, the DL register is ignored for RAID1 devices. Well, maybe not ignored, but once grub2 has determined that the intended boot partition is a raid partition, the raid code takes over and the raid code doesn't care about the DL register. Instead, it scans for all the other members of the raid array and utilizes whichever drives it needs to in order to complete the boot process. And since it does reads a sector (or a small group of sectors) at a time, it doesn't need any member of a raid1 array to be perfect, it will attempt a round robin read on all the sectors and only fail if all drives return an error for a given read.

Doug Ledford <[email protected]>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband

Doug Ledford said: (by the date of Thu, 01 Nov 2007 14:30:58 -0400)

my motherboard allows to pick a boot device if I press F11 during boot. Do you mean, that no matter which HDD I will choose it will have 0x80 number?

Janek Kozicki                                                         |

All the motherboard BIOS drive mapping things I've seen will do exactly that. In order to boot from say drive sda when hda is present, they map BIOS device 0x80 to sda instead of hda.

Doug Ledford <[email protected]>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband

Depends how "bad" the drive is. Just to align the thread on this - If the boot sector is bad - the bios on newer boxes will skip to the next one. But if it is "good", and you boot into garbage - - could be Windows.. does it crash?

b

Right, if the drive is dead almost every BIOS will fail over, if the read gets a CRC or similar most recent BIOS will fail over, but if an error-free read returns bad data, how can the BIOS know.

bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979

Unfortunately the Linux boot format doesn't contain any sort of integrity check. Otherwise the bootloader could catch this kind of error and throw a failure, letting the next disk boot (or another kernel.)

-hpa

I don't understand your point, unless there's a Linux bootloader in the BIOS it will boot whatever 512 bytes are in sector 0. So if that's crap it doesn't matter what it would do if it was valid, some other bytes came off the drive instead. Maybe Windows, since there seems to be an option in Windows to check the boot sector on boot and rewrite it if it isn't the WinXP one. One of my offspring has that problem, dual boot system, every time he boots Windows he has to boot from rescue and reinstall grub.

I think he could install grub in the partition, make that the active partition, and the boot would work, but he tried and only type FAT or VFAT seem to boot, active or not.

bill davidsen <[email protected]>
 CTO TMR Associates, Inc
 Doing interesting things with small computers since 1979

The Grub-promoted practice of stuffing the Linux bootloader in the MBR is a bad idea, but that's not the issue here.

The issue here is that the bootloader itself is capable of making the decision to reject a corrupt image and boot the next device. The Linux kernel, unfortunately, doesn't have a sane way to do that.

-hpa