Care and feeding of RAID?

Hi all,

I have a RAID6 array and I wondering about care and feeding instructions :-)

Here is what I currently do:
- daily incremental and weekly full backups to a separate machine
- run smartd tests (short once a day, long once a week)
- check the raid for bad blocks every week

What else can I do make sure the array keeps humming? Thanks in advance!

The mdadm man-page has information about running mdadm from cron to check for 'unusual' activity.
You may want to consider that. I run it as daemon, personally.

Greetings,
Benjamin

#!/bin/sh #!/bin/bash #!/bin/tcsh #!/bin/csh #!/bin/kiss #!/bin/ksh
#!/bin/pdksh #!/usr/bin/perl #!/usr/bin/python #!/bin/zsh #!/bin/ash

Feel at home? Got some of them? Want to show some magic?

	http://shellscripts.org

Stop fiddling with it :)

I run similar stuff, but don't forget running mdadm in daemon mode to send you an email should a drive fail. I also check each device individually, rather than the array although I don't know the value of doing this over the SMART tests on modern drives though...

Gordon

Would people be willing to list their setup? Including such things as mdadm.conf file, crontab -l, plus scripts that they use to check the smart data and the array, mdadm daemon parameters and anything else that is relevant to checking and maintaining an array?

I'm running the mdmonitor script at startup and a sample mdadm.conf (one of 3 machines) looks like:

MAILADDR [email protected]
ARRAY /dev/md0 level=raid5 num-devices=3 UUID=39d07542:f3c97e69:fbb63d9d:64a052d3 devices=/dev/sdb1,/dev/sdc1,/dev/sdd1

These are SATA drives and except for the one machine that has a 3Ware 8506 card in it I haven't been able to get SMART programs to do anything with these drives. How do others deal with this?

Thanks,

Steve

Personally, I use this script from cron:
http://shellscripts.org/project/hdtest
0 3 * * * root /root/sbin/hdtest.sh -l /var/log/smart_ata-ST3250624A_4ND33CLT.log /dev/disk/by-id/ata-ST3250624A_4ND33CLT short 1 3 * * * root /root/sbin/hdtest.sh -l /var/log/smart_ata-ST3250624A_4ND33EJE.log /dev/disk/by-id/ata-ST3250624A_4ND33EJE short 2 3 * * * root /root/sbin/hdtest.sh -l /var/log/smart_ata-ST3250624A_4ND33ELA.log /dev/disk/by-id/ata-ST3250624A_4ND33ELA short

I have made the experience that long tests slow down the raid to a point where the system becomes unusable.

My mdadm.conf is like this:
---
DEVICE partitions
ARRAY /dev/md/0 level=raid1 num-devices=3 UUID=3559ffcf:14eb9889:3826d6c2:c13731d7 ARRAY /dev/md/1 level=raid5 num-devices=3 UUID=649fc7cc:d4b52c31:240fce2c:c64686e7 ARRAY /dev/md/2 level=raid5 num-devices=3 UUID=9a3bf634:58f39e44:27ba8087:d5189766
spares=1
ARRAY /dev/md/4 level=raid5 num-devices=3 UUID=d4799be3:5b157884:e38718c2:c05ab840
spares=1
ARRAY /dev/md/5 level=raid5 num-devices=3 UUID=ca4a6110:4533d8d5:0e2ed4e1:2f5805b2
spares=1

MAIL root@localhost
---

Also, I run

	mdadm --monitor /dev/md/* --daemonise

from an init script.

Greetings,
Benjamin

 _  _     _   _            _   
| \| |___| |_| |_  __ _ __| |__
| .` / -_)  _| ' \/ _` / _| / /
|_|\_\___|\__|_||_\__,_\__|_\_\
| |  (_)_ _ _  ___ __
| |__| | ' \ || \ \ /
|____|_|_||_\_,_/_\_\
Play Nethack anywhere with an x86 computer:
	http://www.crash-override.net/nethacklinux.html

If you list the devices explicitly, you're opening the possibility for errors when the devices are re-ordered following insertion (or removal) of any other SATA or SCSI (or USB storage) device

I think you want is a "DEVICE partitions" line accompanied by ARRAY lines that have the UUID attribute you've already got in there.

-Mike

Make sure the drives are adequately cooled. I use this nifty utility to look at my drive temps:
http://martybugs.net/linux/hddtemp.cgi

mtbf seems to have an exponential dependence on temperature, so it pays off to keep temp down. Exactly what temp you consider safe is individual, but my drives only occasionally go above 40C.

cheers,

/Patrik

I don't have any mdadm.conf files ... What am I missing? (I've always been under the impression that after needing the /etc/raidtab file with the old raidtools, you didn't need a config file as such under mdadm... However, I'm willing to be enlightened!)

For checking the smart stuff, I use the standard Debian packages and a smartd.conf file typically looks like:

#DEVICESCAN

/dev/hda -d ata -o on -S on -a -m [email protected] -s (S/../.././04|L/../../1/20) -M daily -M test
/dev/hdc -d ata -o on -S on -a -m [email protected] -s (S/../.././04|L/../../1/20) -M daily
/dev/hde -d ata -o on -S on -a -m [email protected] -s (S/../.././04|L/../../1/20) -M daily
/dev/hdi -d ata -o on -S on -a -m [email protected] -s (S/../.././04|L/../../1/20) -M daily

The running mdadm in monitor mode looks like:

/sbin/mdadm -F -i /var/run/mdadm.pid -m root -f -s

and my weekly badblocks script looks like:

#!/bin/csh

echo "uname -n: Badblocks test starting at [date]"

foreach disk ( a c )
foreach partition ( 1 2 3 5 6 )
echo -n "hd$disk${partition}: "
badblocks -c 128 /dev/hd$disk$partition
end
echo ""
end

echo "uname -n: Badblocks test ending at [date]"

I do loads of stuff with disk temperatures (when I can), etc. but thats just for making pretty graphs I can point at my customers... (eg
http://lion.drogon.net/mrtg/diskTemp.html and tell me when that data centre upgraded their AC ;-)

Gordon

I had a pair (2 x Hitachi IDE 80GB) that ran in a sealed case at the top of a lift-shaft for 2 years. They averaged 55C... I never got to see the box after it was decomissioned...

Gordon

Excellent idea, Steve.

In my crontab, I have this:
# Check RAID arrays for bad blocks once a week
30 2 * * Tue echo check >> /sys/block/md0/md/sync_action ; echo "Checking md0 bad blocks" 30 2 * * Wed echo check >> /sys/block/md1/md/sync_action ; echo "Checking md1 bad blocks"

I have this in my smartd.conf:

/dev/hda -H -m root -S on -o on -I 194 -s (S/../.././02|L/../../6/03)
/dev/hdc -H -m root -S on -o on -I 194 -s (S/../.././02|L/../../6/03)
/dev/hde -H -m root -S on -o on -I 194 -s (S/../.././02|L/../../6/03)
/dev/hdg -H -m root -S on -o on -I 194 -s (S/../.././02|L/../../6/03)
/dev/sda -d ata -H -m root -S on -o on -I 194 -s (S/../.././02|L/../../6/03)
/dev/sdb -d ata -H -m root -S on -o on -I 194 -s (S/../.././02|L/../../6/03)
/dev/sdc -d ata -H -m root -S on -o on -I 194 -s (S/../.././02|L/../../6/03)

My Fedora Core box has this in /etc/init.d/mdmonitor:
daemon --check --user=root mdadm ${OPTIONS}
where OPTIONS="--monitor --scan -f --pid-file=/var/run/mdadm/mdadm.pid"

I have no mdadm.conf. My entire filesystem consists of md0 (/boot) and md1(/). I figure if I have problems and need the file, it won't be available anyway. If I am mistaken, please do let me know!

Any other suggestions would be welcomed!

Paul

What about bitmaps? Nobody has mentioned them. It is my understanding that you just turn them on with "mdadm /dev/mdX -b internal". Any caveats for this?

Paul

[snip]

[snip]

Maybe I'm missing something, but are these partitions mounted? Here's what I get when I do this on a mounted partition:

[root@paul ~]# badblocks -nsv /dev/md0 /dev/md0 is mounted; it's not safe to run badblocks!

If you are running RAID, is it safe to run badblocks on the underlying partition?

I use the tw_cli program to check up on my 3ware stuff.

It took me quite a bit of time to figure that one out. I don't have any automated monitoring set up, but it'd be simple enough to script. I check on the array every so often and run a verify every few months to see if it kicks a disk out (it hasn't yet).

0 root@datavibe:~# tw_cli
//datavibe> info

Ctl Model Ports Drives Units NotOpt RRate VRate BBU
------------------------------------------------------------------------
c0 8006-2LP 2 2 1 0 2 - -

//datavibe> info c0

Unit UnitType Status %Cmpl Stripe Size(GB) Cache AVerify IgnECC
------------------------------------------------------------------------------
u0 RAID-1 OK - - 232.885 ON - -

Port Status Unit Size Blocks Serial
---------------------------------------------------------------

p0     OK               u0     232.88 GB   488397168     WD-WMAL718611 
p1     OK               u0     232.88 GB   488397168     WD-WMAL718619

//datavibe>

-j

--------------------------------------------------------
 Rev. Jeffrey Paul    -datavibe-     [email protected]
  aim:x736e65616b   pgp:0xD9B3C17D   phone:877-748-3467
   9440 0C7F C598 01CA 2F17  D098 0A3A 4B8F D9B3 C17D
--------------------------------------------------------

Hi Benjamin,

I am checking this out and I see that you are the writer of this script.
I'm getting errors when it comes to lines 76 and 86-90 about the arithmetic symbols. This is on a Fedora Core 5 system with bash version 3.1.7(1). I weeded out the smartctl command and tried it manually with no luck on my SATA /dev/sd? drives.

What do you (or others) recommend for SATA drives?

Thanks,

Steve

Hi Jeffrey,

Thanks. I use tw_cli too and I have scripted a check to see if it degrades but this doesn't help with checking for disk problems before they happen which SMART should help with. As it happens, smartctl works with 3Ware SATA drives. It is my other SATA drives that I'm unable to monitor.

Steve

______________________________________________________________________
 Steve Cousins, Ocean Modeling Group    Email: [email protected]
 Marine Sciences, 452 Aubert Hall       http://rocky.umeoce.maine.edu
 Univ. of Maine, Orono, ME 04469        Phone: (207) 581-4302

This is a timely thread for me, as I am about to setup a software RAID 10 (a striped pair of mirrors), on 4 x 500GB SATA.

Anything to watch for by not partitioning the drives at all? Or is it safer to make one partition, slightly smaller (suggestions of how much welcome), than the full drive, to allow for possible size discrepencies with replacemnets.

Also I am wondering as this is RAID0 on top of RAID1, if there are any special steps that need to be taken when maintaining the array (adding, removing, rebuilding etc), compared with a "single layer" RAID?

Regards,

Richard

nice race :)

that is because smartctl output has changed and the grep above returns no number.

which command?

smartmontools and a recent kernel just work.
also you can schedule smart tests with smartmontools. so you don't need to cron scripts.

L.

Luca Berra -- [email protected]
       Communication Media & Services S.r.l.
/"\
\ /     ASCII RIBBON CAMPAIGN
 X        AGAINST HTML MAIL
/ \

Do not use the -n option... (and -s won't be much use in a cron job, nor -v, probably!) -n will write to the device which might well have issues with the filesystem cache...

By reading the underlying drives you won't trigger a raid array failure should you do see a bad sector, which might give you time to go something about it. There was some emails on this list some time back (year or 2,3?) about badblocking the md? device - I imagine it might not read every block of every device unless it was a raid-0 array...

Gordon

As in race condition? Where?

I'm running smartmontools 5.33 here. When did the output change? It still works fine here.

Interesting. I'll look into that.

Greetings,
Benjamin

The Nethack IdleRPG! Idle to your favorite Nethack messages!
http://pallas.crash-override.net/nethackidle/

tmp="mktemp"
rm -f ${tmp}
touch ${tmp}

the last two lines are unneeded and can be tricked to overwrite arbitrary filenames

it is '-d ata'

# monitor two sata disks, show temperature in degrees,
# do a long test every sunday and a short every other day
# at 1am on sda and at 2am on sdb, YMMV
/dev/sda -d ata -a -R 194 -s (L/../../7|S/../../[123456])/01
/dev/sdb -d ata -a -R 194 -s (L/../../7|S/../../[123456])/02

Luca Berra -- [email protected]
       Communication Media & Services S.r.l.
/"\
\ /     ASCII RIBBON CAMPAIGN
 X        AGAINST HTML MAIL
/ \

Even though we're quite off-topic here with that since it's more SMART-related... this is at least unusual.
I'm also running regular SMART selftests (short daily, long weekly) and usually they don't affect drive's performance very much. However, whenever I experienced massive slow-downs while selftests are running, this did always point to disk problems... too much (and strategic disadvantageous) reallocated sectors (keep an ear on the disks, hear them seek()ing :)), non-reallocatable sectors or just temperature (for example, WD drives tend to show a bad performance when getting too hot).

regards
Mario

But after a while I learned the trick of speaking fast. You don't have
to think any faster; just use twice as many words to say everything.
                                -- Paul Graham

You don't necessarily need one. However, since Neil considers in-kernel RAID-autodetection a bad thing and since mdadm typically relies on mdadm.conf for RAID-assembly and since especially with newer kernels you probably need to auto-create device nodes (2.6 and udev), it's more convenient to have one. Though you could live without one even then.

I did also run without one over a long time and I also don't like it to have one, however, sometimes in the past convenience won :)

regards
Mario

There is nothing more deceptive than an obvious fact.
             -- Sherlock Holmes by Arthur Conan Doyle

mktemp
rm
touch
why do you do that?

i retested now with 5.36 and it seems the output did _not_ change, i don't know what i saw this morning.

but then it errors on the line
IFS=" " read type status online < <( smartctl -d ata -a ${disk} | grep
\#\ 1 | sed 's, \+, ,g' | cut -f 2,3,5 )

L.

Luca Berra -- [email protected]
       Communication Media & Services S.r.l.
/"\
\ /     ASCII RIBBON CAMPAIGN
 X        AGAINST HTML MAIL
/ \

On 6 Sep 2006, Mario Holbe spake:

You can specify the UUID on the command-line too (although I don't).

The advantage of the config file from my POV is that it lets me activate *all* my RAID arrays with one command, and the command doesn't change, no matter how complex the array configuration. (I'll admit that the sheer number of options to mdadm has always overwhelmed me to some degree, despite the excellent documentation, so I prefer approaches that keep a working command-line unchanged, especially for something as critical as boot-time assembly.)

`In typical emacs fashion, it is both absurdly ornate and
 still not really what one wanted.' --- jdev

On 5 Sep 2006, Paul Waldo uttered the following:

Notably, how many additional writes does it incur? I have some RAID arrays using drives which are quiet *until* you access them, and which then make a bloody racket. The superblock updates are bad enough, but bitmap updates, well, I don't really like seeing one write turned into twelve-odd disk hits that much (just a back-of-the-envelope guess for a three-disk RAID-5 array).

`In typical emacs fashion, it is both absurdly ornate and
 still not really what one wanted.' --- jdev

bitmaps have been working great for me on a raid5 and raid1. it makes it that much more tolerable when i accidentally crash the box and don't have to wait forever for a resync.

i don't notice the extra write traffic all that much... under heavy traffic i see about 3 writes/s to the spare disk in the raid5 -- i assume those are all due to the bitmap in the superblock on the spare.

i've considered using an external bitmap, i forget why i didn't do that initially. the filesystem on the raid5 already has an external journal on raid1.

-dean

Probably because of a wrong understanding of mktemp. I fixed this now.

I think I know what you mean. I've seen these errors sometimes, too. But I was too lazy to investigate yet since they only pop up once a month or so.

Greetings,
Benjamin

 _  _     _   _            _   
| \| |___| |_| |_  __ _ __| |__
| .` / -_)  _| ' \/ _` / _| / /
|_|\_\___|\__|_||_\__,_\__|_\_\
| |  (_)_ _ _  ___ __
| |__| | ' \ || \ \ /
|____|_|_||_\_,_/_\_\
Play Nethack anywhere with an x86 computer:
	http://www.crash-override.net/nethacklinux.html