Automatically adding spares on boot

Hi,

Over the years I've had various problems with the degree to which the adding of spares on boot is automatic. There have been times when this has just worked, and other times (such as now) where it only works very occasionally (or not at all). Google is full of stories like this.

I have no clue why it's not working automatically, I just know that it isn't.

So I wrote a little script that automates this, and hooked it up to mdmon as an event handler.

It can probably do with an elegance upgrade, but it works for me.

HTH, Jan

#!/bin/bash

# mdadm --monitor event handler, by Jan Ceuleers
# Licence: GPL v2
# Handles SparesMissing event. Uses blkid output to find devices that have
# the same UUID as the RAID device, but which are not yet members of the array.
# Have this script executed by mdadm by passing its name to the mdadm --monitor
# daemon using the --program parameter.
# On Debian-like systems this can be accomplished like so:
#   echo "DAEMON_OPTIONS=\"\$DAEMON_OPTIONS --program /pathto/script\"" >> /etc/default/mdadm

BLKID="/sbin/blkid"
MDADM="/sbin/mdadm"

(
case $1 in

    SparesMissing)
      COMPUUID=`$MDADM  -D $2 | grep UUID | cut -f 2 -d 'D' | cut -f 3 -d ' '`
      COMPUUID2=${COMPUUID//:/}
      COMPUUID3=${COMPUUID2:0:8}-${COMPUUID2:8:4}-${COMPUUID2:12:4}-${COMPUUID2:16:4}-${COMPUUID2:20}
      echo The UUID of component devices is $COMPUUID \($COMPUUID3\)
      blkid | grep $COMPUUID3 | grep linux_raid_member | (
        while read l; do
          COMPDEV=`echo $l | cut -f 1 -d ':'`
          $MDADM -D $2 | grep $COMPDEV > /dev/null
          if [ $? -eq 0 ] ; then
            echo Device $COMPDEV is already a member of the array
          else
            echo Found device $COMPDEV with the same UUID but not yet a component device.
            $MDADM $2 --add $COMPDEV 2>&1
          fi
        done
      )
      ;;
    *)
      echo "Unknown event type; can't handle."
      ;;

esac

) | mail -s "Handling $1 mdadm event on $2" root

Here is sample output of the script, which it mails to root:

The UUID of component devices is 442e9934:97191d8e:6d0cf7a9:41621837 (442e9934-9719-1d8e-6d0c-f7a941621837)
Device /dev/sdb2 is already a member of the array
Device /dev/sdd2 is already a member of the array
Found device /dev/sde3 with the same UUID but not yet a component device.
mdadm: re-added /dev/sde3

It's a great shame that there does not appear to be a standard format for representing UUIDs: mdadm uses an 8:8:8:8 format, whereas blkid wants an 8-4-4-4-12 format. Hence the conversion shenanigans in the script. Makes the code brittle.

Cheers, Jan

hm… I don't se the point, really. I have a RAID-6 with a spare, and last time a drive died, the spare kicked in automatically. Why would you add something at boot?

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
[email protected]
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

For the spare to be automatically swung into action when a drive fails, the spare must first have been added to the array. In other words: it can't just be an unused drive somewhere: mdadm must know that it has permission to use this drive (or partition or whatever) as a hot spare.

Normally you would expect mdadm's auto-assembly code, or the udev scripts that ship with the OS, to add all relevant devices to each
array: not only the active devices but also the spares. The problem I'm facing is that (for reasons unknown) my arrays start up with only the active members added, and the spare devices remaining unused.

Specifically, my /proc/mdstat looks like this upon boot:

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[1] sdd1[0]

      521984 blocks [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md1 : active raid1 sdb2[1] sdd2[0]

      487861824 blocks [2/2] [UU]
      bitmap: 1/4 pages [4KB], 65536KB chunk

unused devices: <none>

Note that mdadm doesn't think that there are any unused devices, but there are. Because when mdadm --monitor notices that each of the above arrays should have a spare it throws a SparesMissing event, which triggers the script that I sent to the list. Once both of these events have been handled /proc/mdstat looks like this:

Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sde2[2](S) sdb1[1] sdd1[0]

      521984 blocks [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md1 : active raid1 sde3[2](S) sdb2[1] sdd2[0]

      487861824 blocks [2/2] [UU]
      bitmap: 1/4 pages [4KB], 65536KB chunk

unused devices: <none>

Hope this clarifies. You don't need this script if your OS always finds your spares and adds them automatically to your arrays. Mine doesn't.

Cheers, Jan
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Really, if md 'forgets' a spare on reboot, that's a bug and should be reported. This is how my raid looks

Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] md0 : active raid6 sdg[5] sdh[6](S) sdc[7] sdb[0] sde[3] sdf[4] sdd[8]

      7814051840 blocks super 1.2 level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU]

unused devices: <none>

What distro/version is this?

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
[email protected]
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

http://www.google.com/#hl=en&q=mdadm+does+not+add+spares+on+boot

First hit.

root@zotac:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 11.10
Release:	11.10
Codename:	oneiric
root@zotac:~# uname -a
Linux zotac 3.0.0-30-generic #47-Ubuntu SMP Wed Jan 2 23:16:29 UTC 2013
x86_64 x86_64 x86_64 GNU/Linux
root@zotac:~# mdadm -V
mdadm - v3.1.4 - 31st August 2010

----- Opprinnelig melding -----

That's rather old…

That's rather old too, and soon out of support. Better upgrade to 12.04 LTS and try again.

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
[email protected]
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk. --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I do know that.

I reported this bug when the release on which I reported it was still current. I didn't have this problem on 2 subsequent Ubuntu releases and now I'm having it again on 11.10. So I did something about it myself and I posted the solution so others can help themselves too.