Adding larger disks to RAID5

Hello,

I have 8x250gig hard drives in a RAID5 configuration. What I'm trying to do is add more storage to the raid by replacing the 250's with larger disks one-by-one over time. I just replaced the first disk with a 750, and the raid resync'd, but now I'm unable to grow it to the additional space. The raid is just using the first 250 gigs and
nothing more on the drive. The command I issued in attempt to grow
it was 'mdadm --grow /dev/md0 -z max', but nothing happens. Is what I'm trying to do possible? I'm running kernel 2.6.21 w/ raid5 expansion enabled.

Thanks,
Christian

SCSI device sda: 1465149168 512-byte hdwr sectors (750156 MB) SCSI device sdb: 488397168 512-byte hdwr sectors (250059 MB) SCSI device sdc: 488397168 512-byte hdwr sectors (250059 MB) SCSI device sdd: 488397168 512-byte hdwr sectors (250059 MB) SCSI device sde: 488397168 512-byte hdwr sectors (250059 MB) SCSI device sdf: 488397168 512-byte hdwr sectors (250059 MB) SCSI device sdg: 488397168 512-byte hdwr sectors (250059 MB) SCSI device sdh: 488397168 512-byte hdwr sectors (250059 MB) SCSI device sdi: 488397168 512-byte hdwr sectors (250059 MB)

Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sda[0] sdb[1] sdh[7] sdg[6] sdf[5] sde[4] sdd[3] sdc[2]

     1709389248 blocks level 5, 64k chunk, algorithm 2 [8/8] [UUUUUUUU]

/dev/md0:

       Version : 00.90.03

Creation Time : Fri May 18 01:49:46 2007
Raid Level : raid5
Array Size : 1709389248 (1630.20 GiB 1750.41 GB)
Used Dev Size : 244198464 (232.89 GiB 250.06 GB)
Raid Devices : 8
Total Devices : 8
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Mon May 21 13:27:34 2007
State : clean
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 64K

UUID : 905a278f:a088d952:fab6e626:2309d439
Events : 0.94

Number Major Minor RaidDevice State

      0       8        0        0      active sync   /dev/sda
      1       8       16        1      active sync   /dev/sdb
      2       8       32        2      active sync   /dev/sdc
      3       8       48        3      active sync   /dev/sdd
      4       8       64        4      active sync   /dev/sde
      5       8       80        5      active sync   /dev/sdf
      6       8       96        6      active sync   /dev/sdg
      7       8      112        7      active sync   /dev/sdh

No, what you are trying to do is not possible.
A raid5 uses equal amounts of space on each device. So if the devices are of different sizes, it will use an amount equivalent to the smallest device from every device.

If you replace all the devices with 750G drives, then you could grow the array (-z max) to use all the space, not before.

What you could do is partition the new drives in a 250G partition and a 500G partition. Add the 250 to the current array. Create a new raid5 with the 500G.
With only one large drive, you could make a degraded 2-device raid5 which would store 500G, but would not have any data redundancy. When you get a second 750G, add that and you will have redundancy. Then as you replace more drives and get more 500G partitions you can add them to the array growing it as you go until you ultimately have 2 arrays, one with 8 250G partitions, and one with 8 500G partitions.

That seems the best fit for your needs as you describe them.

If possible, it would be best to get all the drives up front and just have one large array.

NeilBrown

Neil,

What seems desirable to me is a way to take a new (larger) spare drive and add it to a RAID1 for a particular RAID 4/5/6 component, and then when it's sync'd, replace the now redundant small drive with another larger drive. Wash, rinse, repeat. This way the array is never degraded. Though I imagine that this particular arrangement doesn't have the benefit of the stripe rewrite when encountering a latent error on the drive that is being migrated. [Presumably the failing addresses could be cycled through the check from userland though, by doing a read above the stacked RAID.]

One could start a RAID 4/5/6 array over a degraded RAID1 for each component, (i.e., a degraded RAID1).

I haven't been following the metadata changes closely. Is it possible to do this with external MD metadata? It can also be done with device-mapper, but dm-mirror is very immature compared to MD RAID1.

Comments?

Regards,

Bill Rugolsky

This doesn't really have anything to do with the metadata used - it is primarily an implementation issue (though you would need to be careful picking up the pieces after a crash).

If we could freeze an array (so that all writes block), then we could do much of what you suggest:
- freeze the array
- remove the target device
- create a raid1 of the target and the new
- re-add the raid1
- unfreeze the array.

The issue of dealing with read errors on the target device is much more awkward to deal with. The approach that seems right to me is:

- create a raid1 variant which does a passive resync: When the next-needed block is read or written, write it to the second device and advance the "next-needed" pointer.
- Get this raid1 to simply return read errors (which might be OK already) so that a read-error won't be fatal. But a read request that be behind the "next-needed" pointer gets served from the second device if the first does fail.
- Implement a 'check-one-disk' operation on raid5 (and others) so that instead of reading all devices, it just reads all through one. If this one is really a raid1-variant, doing that read will effect a resync on the raid1, and any read error will be handled correctly.

So it is all quite possible, and I agree that it could be valuable. It just needs someone to do it, and work out all the fine details.

Anyone want to try some coding ????

NeilBrown

That was supposed to say "a degraded RAID5+1".

Right. Doing it all automatically would be great, but involves some kernel work.

What I was suggesting is that one administratively stop an array built conventionally, and manually start a degraded RAID1 for each component. It seems that is possible using "--build". Then one can manually assemble the RAID5+1 from those RAID1s, and proceed with the disk-by-disk replacement while the array is online. Looks as if that ought to be possible. I'll have to give it a try, as soon as I have an afternoon to tinker [say, ... mid-summer :-/].

Many thanks, Neil.

-Bill