Slow reconstruction

First, thanks to all the authors of software RAID, which works really well.

I have just one problem(!) and as disks get bigger it gets worse:
reconstruction takes a long time. At the moment I've just had a disk fail on a 5x250 GB RAID5 array and it's going to take eight days to recover. Is there anything I can do to speed things up?

FWIW, each disk is the master drive on an IDE card, two of the disks are shared with the RAID1 root partition and one of the 5 devices is a RAID0 stripe. The system isn't doing anything else, it's just used for backup.

Thanks again.

John

That is an astonishingly slow reconstruction time.

Have you looked at hdparm informational output on each of the drives? dmesg, grepping for the hard drive initialization information?

With speeds like that, I suspect you don't have DMA enabled for some reason. Maybe someone else has seen something more exactly matching this, but when speeds are this off, that's what its been for me in the past.

Running hdparm -Tt /dev/hdXX can tell you what the disks can do individually, I'd start there, and look at the hdparm -i (or -I?) output to see what they were set at, and start tuning the interface.

Hopefully that turns something up quickly.

-Mike

Thanks for all the suggestions.

/proc/sys/dev/raid/speed_limit_max had the value "10000", changing it to 200000 didn't make a difference. hdparm looks OK to me:

/dev/hdm1:
Timing buffer-cache reads: 128 MB in 1.19 seconds =107.56 MB/sec
Timing buffered disk reads: 64 MB in 3.13 seconds = 20.45 MB/sec

/dev/hdo1:
Timing buffer-cache reads: 128 MB in 1.17 seconds =109.40 MB/sec
Timing buffered disk reads: 64 MB in 3.85 seconds = 16.62 MB/sec

rsync:/root# hdparm -Tt /dev/hda4

/dev/hda4:
Timing buffer-cache reads: 128 MB in 1.15 seconds =111.30 MB/sec
Timing buffered disk reads: 64 MB in 4.31 seconds = 14.85 MB/sec

/dev/hdc4:
Timing buffer-cache reads: 128 MB in 1.16 seconds =110.34 MB/sec
Timing buffered disk reads: 64 MB in 4.79 seconds = 13.36 MB/sec

/dev/hdi3:
Timing buffer-cache reads: 128 MB in 1.16 seconds =110.34 MB/sec
Timing buffered disk reads: 64 MB in 2.73 seconds = 23.44 MB/sec

/dev/hdk1:
Timing buffer-cache reads: 128 MB in 1.16 seconds =110.34 MB/sec
Timing buffered disk reads: 64 MB in 2.72 seconds = 23.53 MB/sec

DMA is enabled for all six disks.

If it helps, here is my raidtab. I wonder a little about the 64K chunk size on the stripe but a 128K size on the RAID5. Is this good or bad?

raiddev /dev/md0

raid-level                  1
nr-raid-disks               2
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0

device /dev/hda2
raid-disk 0
device /dev/hdc2
raid-disk 1

raiddev /dev/md2

raid-level                  0
nr-raid-disks               2
chunk-size                  64k
persistent-superblock       1
nr-spare-disks              0

device /dev/hdm1
raid-disk 0
device /dev/hdo1
raid-disk 1

raiddev /dev/md3

raid-level                  5
nr-raid-disks               5
chunk-size                  128k

parity-algorithm left-symmetric
persistent-superblock 1
nr-spare-disks 0
device /dev/hda4
raid-disk 0
device /dev/hdc4
raid-disk 1
device /dev/hdi3
raid-disk 2
device /dev/hdk1
raid-disk 3
device /dev/md2
raid-disk 4

Comments gratefully received.

John

you know about /proc/sys/dev/raid/speed_limit_max and speed_limit_min, right?

echo 40000 > /proc/sys/dev/raid/speed_limit_max

see if it goes faster, then.

-sv

did you raise speed_limit_min??

if you didn't then you're probably being hit by the 'only use idle io' problem.

-sv

That's fixed it! It's now going along at around 5.7 MB/s which isn't huge but is around 11 times what it was. Many thanks.

But I'm a bit surprised this makes a difference because there is no other real IO going on.

John

These are very low values. A modern 250G drive should exceed 50MB/sec. What kind of IDE controller do you use?

Peter

    .+'''+.         .+'''+.         .+'''+.         .+'''+.         .+''
 Kelemen Péter     /       \       /       \     [email protected]
.+'         `+...+'         `+...+'         `+...+'         `+...+'
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

None of it is very modern - it's put together from what we had lying around :-(

(Don't even start..) I'll see if it's any different when the array is reconstructed.

John

For 250GB drives, those Buffered disk reads seem rather slow.. they should be in the 35-50 range. I would check your dma settings with hdparm as someone already suggested. What speed of processor is in the server, and what type of IDE cards/controllers? Maybe also check your /proc/interrupts for errors.. maybe you have an IRQ conflict. Also double check your dmesg output. What kernel are you using?

Tyler.

It is most likely that the controllers (and/or respective drivers) are the source of your problem. I'd say the controllers are UDMA(66) only.

Peter

    .+'''+.         .+'''+.         .+'''+.         .+'''+.         .+''
 Kelemen Péter     /       \       /       \     [email protected]
.+'         `+...+'         `+...+'         `+...+'         `+...+'
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html