Flexible storga: LVM setup on top of mdadm sets. Good idea or not?
Hi list,
I'm looking into making a home brewed nas setup. This is not a production, highly critical environment.
The result needs to be 1 continuous physical storage space.
Expandable without reformating, removing data, etc,..
Acceptable redundancy.
I don't mind 50% loss of data capacity because of raid 1. Storage is (relatively) cheap.
I don't have this material to test. So it's more like a theoretical setup.
I was thinking of creating sets of mdadm RAID 1 setups and unify these raid 1 devices with lvm2 into 1 big logical volume which is resizeable. For example:
I have 4 identical 250 GB sata2 disks.
I use mdadm to create:
/dev/md0 raid1 consisting out of /dev/sda1 and /dev/sdb1. /dev/md1 raid1 consisting out of /dev/sdc1 and /dev/sdd1.
Then I would like to:
create LVM physical volumes out of /dev/md0 and /dev/md1.
create a LVM volume group named storage consisting out of LVM physical device /dev/md0 and /dev/md1 create a LVM logical volume that covers the maximum size of the LVM volume group which should be around 500GB. create a filesystem on this LVM logical volume such as ext3.
Now when I need more storage:
I add 2 250GB sata2 disks.
I create a new raid1 /dev/md2 consisting out of /dev/sde1 and /dev/sdf1 Create a LVM physical volume out of /dev/md2.
Add the /dev/md2 LVM physical volume to the LVM volume group. expand the LVM logical volume so it becomes around 750GB in size. resize the EXT3 filesystem.
Now my question to you is. Is this completely bunkers? Or is it reasonable? Any tips, tricks, advice, comments welcome.
Thanks,
js
Quoted 10 lines · js
I'm looking into making a home brewed nas setup. This is not a production, highly critical environment.
The result needs to be 1 continuous physical storage space. Expandable without reformating, removing data, etc,.. Acceptable redundancy.
I don't mind 50% loss of data capacity because of raid 1. Storage is (relatively) cheap.I don't have this material to test. So it's more like a theoretical setup.
...
Quoted 2 lines
Now my question to you is. Is this completely bunkers? Or is it reasonable?
This is how all our systems are setup. It is the only reasonable way to do raid1. Hardware RAID is nasty when you need to add/replace disks and the new models are bigger.
Quoted 1 line
Any tips, tricks, advice, comments welcome.
The main drawback to md raid1 is that it always resynchronizes the *entire* partition when a disk goes offline temporarily. I also like to split a drive into several pieces, say split a pair of 240G into two 120G md partitions. This gives me some flexibility to migrate partitions between physical drives (for performance tuning, or swapping physical drives around). To migrate, set the mirror "faulty" with mdadm, "hot remove" it, then "hot add" the new partition. If you don't like leaving the md drive unmirrored while resyncing to its new location, reserve 3 physical drives in each md device, but use only 2. Then you can a 3rd mirror to migrate, then set faulty and hot remove the old after the new is finished synchronizing.
Of course, I *hate* having to manually manage the md partitions in Linux. I cut my teeth on AIX LVM - which has robust useful raid1 built it. (The latest experimental LVM mirroring in Linux still doesn't cut it.) AIX is probably too expensive for home brew, but I understand Open Solaris has similar features (but haven't had time to try it).
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
Quoted 2 lines · Stuart D. Gathman
The main drawback to md raid1 is that it always resynchronizes the *entire* partition when a disk goes offline temporarily. I also
False. md uses bitmaps to avoid this.
L.
Signature · 6 lines
Luca Berra -- [email protected] Communication Media & Services S.r.l. /"\ \ / ASCII RIBBON CAMPAIGN X AGAINST HTML MAIL / \
Quoted 4 lines · Luca Berra
Quoted 2 lines · Stuart D. Gathman
The main drawback to md raid1 is that it always resynchronizes the *entire* partition when a disk goes offline temporarily. I also
False. md uses bitmaps to avoid this.
I am using md on dozens of systems up through Centos-5. If there are any bitmaps, they don't work. Or maybe they are a new feature not in kernel 2.6.18. The only change I've noticed since RH7.3 is that sync speed is no longer limited to 10K. Or maybe I'm not looking at the right thing.
When replacing a disk, a full resync is required anyway - so no problem. Where it is a problem is when there is an abnormal system shutdown, so that md devices are not closed properly. The system boots immediately, but performance is slow until the mirror have been resynced. This can take hours. And the system is vulnerable to single disk failure in the source drive for those hours.
How would I see these bitmaps in action?
BTW, to upgrade a 7.2 system to Centos5, I plan to install on a disk with matching partition sizes and replace boot drive. Is there any chance that md raid1 will sync from the old system overtop the new? (Because the new wouild have lower sequence numbers?) Or is there some kind of UUID in the raid superblock to prevent this? (Of course I plan to change partition types away from RAID Auto just to be safe...)
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
Quoted 11 lines · Stuart D. Gathman
Quoted 4 lines · Luca Berra
Quoted 2 lines · Stuart D. Gathman
The main drawback to md raid1 is that it always resynchronizes the *entire* partition when a disk goes offline temporarily. I also
False. md uses bitmaps to avoid this.
I am using md on dozens of systems up through Centos-5. If there are any bitmaps, they don't work. Or maybe they are a new feature not in kernel 2.6.18. The only change I've noticed since RH7.3 is that sync speed is no longer limited to 10K. Or maybe I'm not looking at the right thing.
man mdadm
Bitmaps are a feature of md since mdadm-2.0 (kernel 2.6.13 or 2.6.14)
...
Quoted 1 line
How would I see these bitmaps in action?
man mdadm
but basically
mdadm -G /dev/md?? -b internal
Quoted 6 lines
BTW, to upgrade a 7.2 system to Centos5, I plan to install on a disk with matching partition sizes and replace boot drive. Is there any chance that md raid1 will sync from the old system overtop the new? (Because the new wouild have lower sequence numbers?) Or is there some kind of UUID in the raid superblock to prevent this? (Of course I plan to change partition types away from RAID Auto just to be safe...)
unfortunately redhat insisted for a long time on using the in-kernel md auto-detect, which iirc just ignores md uuid. i think this is still true in redhat5.
changing partition type will prevent this
creating the new raid with a different minor will prevent this or change the minor of the old raid.
... but we're getting offtopic.
if you have further questions post them on linux-raid.
L.
Signature · 6 lines
Luca Berra -- [email protected] Communication Media & Services S.r.l. /"\ \ / ASCII RIBBON CAMPAIGN X AGAINST HTML MAIL / \
Quoted 4 lines · Luca Berra
Quoted 2 lines · Stuart D. Gathman
The main drawback to md raid1 is that it always resynchronizes the *entire* partition when a disk goes offline temporarily. I also
False. md uses bitmaps to avoid this.
The LVM mirroring uses bitmaps. Are you sure md raid does? Since what version?
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 10 lines · Stuart D. Gathman
Quoted 4 lines · Luca Berra
Quoted 2 lines · Stuart D. Gathman
The main drawback to md raid1 is that it always resynchronizes the *entire* partition when a disk goes offline temporarily. I also
False. md uses bitmaps to avoid this.
The LVM mirroring uses bitmaps. Are you sure md raid does? Since what version?
drivers/md/md.c:
- persistent bitmap code
Copyright (C) 2003-2004, Paul Clements, SteelEye Technology, Inc.
Cheers,
Bryn.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFGpcKf6YSQoMYUY94RAq4JAKDJYwRqXm3ZsAqLgrhbup4/MEvNNACfauDJ GNLb7n3pHU55pe+aKXenLAw=
=oj8i
-----END PGP SIGNATURE-----
Quoted 5 lines · Luca Berra
Quoted 2 lines · Stuart D. Gathman
The main drawback to md raid1 is that it always resynchronizes the *entire* partition when a disk goes offline temporarily. I also
False. md uses bitmaps to avoid this.
Ok, I just tested this with Centos-5.0, kernel-2.6.18-8.1.8.el5. I shutdown with md1 50% resynced, and when it boots, it starts over at the beginning. Maybe we are talking about different things, but it is a major annoyance that it restarts the sync from the beginning if it misses just one update (e.g. unexpected power failure or reboot).
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
Quoted 6 lines · Stuart D. Gathman
Quoted 3 lines · Luca Berra
Quoted 2 lines · Stuart D. Gathman
The main drawback to md raid1 is that it always resynchronizes the *entire* partition when a disk goes offline temporarily. I also
Quoted 8 lines
Quoted 1 line
False. md uses bitmaps to avoid this.
Ok, I just tested this with Centos-5.0, kernel-2.6.18-8.1.8.el5. I shutdown with md1 50% resynced, and when it boots, it starts over at the beginning. Maybe we are talking about different things, but it is a major annoyance that it restarts the sync from the beginning if it misses just one update (e.g. unexpected power failure or reboot).
did you add the bitmap to your md array?
L.
Signature · 6 lines
Luca Berra -- [email protected] Communication Media & Services S.r.l. /"\ \ / ASCII RIBBON CAMPAIGN X AGAINST HTML MAIL / \
Quoted 9 lines · Luca Berra
Quoted 8 lines
Quoted 1 line
False. md uses bitmaps to avoid this.
Ok, I just tested this with Centos-5.0, kernel-2.6.18-8.1.8.el5. I shutdown with md1 50% resynced, and when it boots, it starts over at the beginning. Maybe we are talking about different things, but it is a major annoyance that it restarts the sync from the beginning if it misses just one update (e.g. unexpected power failure or reboot).
did you add the bitmap to your md array?
I installed Centos5 from the CDs on raid1 devices. I do see a bitmap option in the map page for mdadm. How do I check whether Centos5 install neglected to activate it? Is there any way to add it after the install? (LVM is on md1 and root,swap are on LVM.) If not, what would be the install procedure to actually use a bitmap? Note: special restrictions on the boot partition are acceptable. Special restrictions on the root partition/LV are not.
# mdadm --examine /dev/hde3
/dev/hde3:
Magic : a92b4efc
Version : 00.90.00
UUID : 14c44605:1e3381da:9833dc4f:b4bdc3c0
Creation Time : Mon Aug 20 15:13:47 2007
Raid Level : raid1
Device Size : 78059712 (74.44 GiB 79.93 GB)
Array Size : 78059712 (74.44 GiB 79.93 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 1 Update Time : Tue Aug 21 10:06:36 2007
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Checksum : bb50a9f9 - correct
Events : 0.8
Number Major Minor RaidDevice State
this 0 33 3 0 active sync /dev/hde3
0 0 33 3 0 active sync /dev/hde3 1 1 34 3 1 active sync /dev/hdg3
Signature · 4 lines
Stuart D. Gathman <[email protected]> Business Management Systems Inc. Phone: 703 591-0911 Fax: 703 591-6154 "Confutatis maledictis, flammis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.