Skip to contentSkip to conversation

raid5/lvm setup questions

DISCUSSION David Greaves 5 participants 7 messages Aug 5, 2006 – Aug 7, 2006 3 messages not in the archive MD RAID

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive
Quoted 9 lines · Shane

Hello all,

I'm building a new server which will use a number of disks and am not sure of the best way to go about the setup. There will be 4 320gb SATA drives installed at first. I'm just wondering how to set the system up for upgradability. I'll be using raid5 but not sure whether to use lvm over the raid array.

Quoted 4 lines

By upgradability, I'd like to do several things. Adding another drive of the same size to the array. I understand reshape can be used here to expand the underlying block device.

Yes, it can.

If the block device is the pv of an lvm array,

Quoted 3 lines

would that also automatically expand in which I would create additional lvs in the new space. If this isn't automatic, are there ways to do it manually?

Not automatic AFAIK - but doable.

Quoted 4 lines

What about replacing all four drives with larger units. Say going from 300gbx4 to 500gbx4. Can one replace them one at a time, going through fail/rebuild as appropriate and then expand the array into the unused space

Yes.

or would

Quoted 1 line

one have to reinstall at that point.

No

None of the requirements above drive you to layering lvm over the top.

That's not to say don't do it - but you certainly don't *need* to do it.

Pros:
* allows snapshots (for consistent backups)
* allows various lvm block movements etc...
* Can later grow vg to use discrete additional block devices without raid5 grow limitations (eg same-ish size disks etc)

Cons:
* extra complexity -> risk of bugs/admin errors...
* performance impact

As an example of the cons: I've just set up lvm2 over my raid5 and whilst testing snapshots, the first thing that happened was a kernel BUG and an oops...

David

#5 lore raw
In reply to David Greaves

On 5 Aug 2006, David Greaves prattled cheerily:

Quoted 2 lines

As an example of the cons: I've just set up lvm2 over my raid5 and whilst testing snapshots, the first thing that happened was a kernel BUG and an oops...

I've been backing up using writable snapshots on LVM2 over RAID-5 for some time. No BUGs.

I think the blame here is likely to be layable at the snapshots' door,
anyway: they're still a little wobbly and the implementation is pretty
complex: bugs surface on a regular basis.

Signature · 2 lines
`We're sysadmins. We deal with the inconceivable so often I can clearly 
 see the need to define levels of inconceivability.' --- Rik Steenwinkel
#7 lore raw
In reply to a message not in the archive
Quoted 3 lines

I seem to recall patches to md floating around a couple years back for partitioning of md devices. Are those still available somewhere?

I believe the patches that you are referring to are now included into the standard 2.6+ kernel.

-Chet

#8 lore raw
In reply to a message not in the archive
Quoted 4 lines · Shane
Quoted 3 lines · Nix

On 5 Aug 2006, David Greaves prattled cheerily:

Quoted 2 lines

As an example of the cons: I've just set up lvm2 over my raid5 and whilst testing snapshots, the first thing that happened was a kernel BUG and an oops...

Quoted 8 lines
Quoted 2 lines

I've been backing up using writable snapshots on LVM2 over RAID-5 for some time. No BUGs.

Just performed some basic throughput tests using 4 SATA disks in a raid5 array. The read performance on the /dev/mdx device runs around 180mbps but if lvm is layered over that, reads on the lv are around 130mbps. Not an unsubstantial reduction.

Check the readahead at various block levels
blockdev --setra xxx

I think I found the best throughput (for me) was with 0 readahead for /dev/hdX, 0 for /dev/mdX and lots for /dev/vg/lv

Quoted 4 lines

I seem to recall patches to md floating around a couple years back for partitioning of md devices. Are those still available somewhere?

man mdadm and see --auto...

David

#9 lore raw
In reply to Nix
Quoted 1 line · Nix

On 5 Aug 2006, David Greaves prattled cheerily:

that's me :)

Quoted 5 lines
Quoted 2 lines

As an example of the cons: I've just set up lvm2 over my raid5 and whilst testing snapshots, the first thing that happened was a kernel BUG and an oops...

I've been backing up using writable snapshots on LVM2 over RAID-5 for some time. No BUGs.

I tried but it didn't recurr.
I sent a report to lkml.

Quoted 3 lines

I think the blame here is likely to be layable at the snapshots' door,
anyway: they're still a little wobbly and the implementation is pretty
complex: bugs surface on a regular basis.

Hmmm. Bugs in a backup strategy. Hmmm.

I think I can live with a nightly shutdown of the daemons whilst rsync does it's stuff across the LAN.

David

#4 lore raw
In reply to a message not in the archive
Quoted 10 lines · On Saturday August 5, [email protected] wrote:
Quoted 4 lines · David Greaves
Quoted 3 lines

Say going from 300gbx4 to 500gbx4. Can one replace them one at a time, going through fail/rebuild as appropriate and then expand the array into the unused space

Yes.

I didn't see anything in the mdadm manual on this. Would one just do a --grow /dev/md0 once the disks were changed out? It looks like --grow is used to change the number of devices in the array but not the device size itself.

It does both (and more).

mdadm --grow /dev/md0 --raid-disk=5
changes the number of drives to 5.

mdadm --grow /dev/md0 --size=max

changes the used-size of each drive to the maximum available.

mdadm --grow /dev/md0 --bitmap=internal
adds an internal write-intent bitmap

mdadm --grow /dev/md0 --chunksize=128
might change the chunksize to 128k.. but doesn't yet. Maybe one day :-)

NeilBrown