Software raid on top of lvm logical volume
Hi,
I would like to know if it's possible (works perfectly) to create a software mirror (md0) on top of 2 LVM logical volumes :
Physical Devices : /dev/sdb1, /dev/sdc1, /dev/sdd1, /dev/sde1, /dev/sdf1, /dev/sdg1 pvcreate /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 vgcreate -s 32 vgdata1 /dev/sdb1 /dev/sdc1 /dev/sdd1
lvcreate -l xxx vgdata1 lvdata1
vgcreate -s 32 vgdata2 /dev/sde1 /dev/sdf1 /dev/sdg1
lvcreate -l xxx vgdata2 lvdata2
/etc/raidtab :
raiddev /dev/md0
raid-level 1
nr-raid-disks 2
nr-spare-disks 0
chunk-size 32
persistant-superblock 1
device /dev/vgdata1 raid-disk 0 device /dev/vgdata2 raid-disk 1
Tks
Eric Monjoin
IT Architect
IBM france
Quoted 2 lines · Eric Monjoin
I would like to know if it's possible (works perfectly) to create a software mirror (md0) on top of 2 LVM logical volumes :
You'd usually want to make your raid devices first, then put LVM on top of it. I can't really think of any benefits of doing it the other way around.
Signature · 4 lines
Randomly Generated Tagline:
Sarchasm: The gulf between the author of sarcastic wit, and the recipient
who doesn't get it.
- Washington PostTheo Van Dinter a �crit :
Well it's because we have problems in this way. We have a server connected to 2 EMC Symmetrix where we assign some 70Gb and 40Gb Luns. We used Powerpath to manage the dual path to the Luns and so I first created mirror as this :
raiddev /dev/md0
raid-level 1
nr-raid-disks 2
nr-spare-disks 0
chunk-size 32
persistent-superblock 1
device /dev/emcpowera1
raid-disk 0
device /dev/emcpowerf1 raid-disk 1
# failed-disk 1
raiddev /dev/md1
raid-level 1
nr-raid-disks 2
nr-spare-disks 0
chunk-size 32
persistent-superblock 1
device /dev/emcpowerb1
raid-disk 0
device /dev/emcpowerg1 raid-disk 1
# failed-disk 1
raiddev /dev/md2
raid-level 1
nr-raid-disks 2
nr-spare-disks 0
chunk-size 32
persistent-superblock 1
device /dev/emcpowerc1
raid-disk 0
device /dev/emcpowerh1 raid-disk 1
# failed-disk 1
raiddev /dev/md3
raid-level 1
nr-raid-disks 2
nr-spare-disks 0
chunk-size 32
persistent-superblock 1
device /dev/emcpowerd1
raid-disk 0
device /dev/emcpoweri1 raid-disk 1
# failed-disk 1
......
up to raiddev /dev/md9
So the /proc/mdstat give :
Personalities : [raid1]read_ahead 1024 sectors
Event: 15 md9 : active raid1 emcpowerd1[1] emcpowero1[0]
42829184 blocks [2/2] [UU]
md8 : active raid1 emcpowerc1[1] emcpowern1[0]
42829184 blocks [2/2] [UU]
md7 : active raid1 emcpowerb1[1] emcpowerm1[0]
42829184 blocks [2/2] [UU]
md6 : active raid1 emcpowera1[1] emcpowerl1[0]
42829184 blocks [2/2] [UU]
md5 : active raid1 emcpowerp1[1] emcpowerk1[0]
42829184 blocks [2/2] [UU]
md4 : active raid1 emcpowerj1[1] emcpowere1[0]
71384704 blocks [2/2] [UU]
md3 : active raid1 emcpoweri1[1] emcpowerd1[0]
71384704 blocks [2/2] [UU]
md2 : active raid1 emcpowerc1[0] emcpowerh1[1]
71384704 blocks [2/2] [UU]
md1 : active raid1 emcpowerg1[1] emcpowerb1[0]
71384704 blocks [2/2] [UU]
md0 : active raid1 emcpowerf1[1] emcpowera1[0]
71384704 blocks [2/2] [UU]
unused devices: <none>
But after a while I obtain that :
Personalities : [raid1]read_ahead 1024 sectors
Event: 10 md9 : active raid1 [dev e9:31][1] [dev e8:e1][0]
42829184 blocks [2/2] [UU]
md8 : active raid1 [dev e9:21][1] [dev e8:d1][0]
42829184 blocks [2/2] [UU]
md7 : active raid1 [dev e9:11][1] [dev e8:c1][0]
42829184 blocks [2/2] [UU]
md6 : active raid1 [dev e9:01][1] [dev e8:b1][0]
42829184 blocks [2/2] [UU]
md5 : active raid1 [dev e8:f1][1] [dev e8:a1][0]
42829184 blocks [2/2] [UU]
md4 : active raid1 [dev e8:91][1] [dev e8:41][0]
71384704 blocks [2/2] [UU]
md3 : active raid1 [dev e8:81][1] [dev e8:31][0]
71384704 blocks [2/2] [UU]
md2 : active raid1 [dev e8:71][1] [dev e8:21][0]
71384704 blocks [2/2] [UU]
md1 : active raid1 [dev e8:61][1] [dev e8:11][0]
71384704 blocks [2/2] [UU]
md0 : active raid1 [dev e8:51][1] [dev e8:01][0]
71384704 blocks [2/2] [UU]
unused devices: <none>
and if we try to rebuild the mirror after after loosing access to one of the EMC, we have really bad result :
Personalities : [raid1]read_ahead 1024 sectors
Event: 26 md9 : active raid1 emcpowerd1[2] [dev e8:e1][0]
42829184 blocks [2/1] [U_]
[>....................] recovery = 1.4% (630168/42829184)
finish=68.1min speed=10315K/sec
md8 : active raid1 emcpowerc1[2] [dev e8:d1][0]
42829184 blocks [2/1] [U_]md7 : active raid1 emcpowerb1[2] [dev e8:c1][0]
42829184 blocks [2/1] [U_]
md6 : active raid1 emcpowera1[2] [dev e8:b1][0]
42829184 blocks [2/1] [U_]
md5 : active raid1 emcpowerp1[2] [dev e8:a1][0]
42829184 blocks [2/1] [U_]
md4 : active raid1 emcpowerj1[2] [dev e8:41][0]
71384704 blocks [2/1] [U_]
md3 : active raid1 emcpoweri1[2] [dev e8:31][0]
71384704 blocks [2/1] [U_]
md2 : active raid1 emcpowerh1[2] [dev e8:21][0]
71384704 blocks [2/1] [U_]
md1 : active raid1 emcpowerg1[2] [dev e8:11][0]
71384704 blocks [2/1] [U_]
md0 : active raid1 emcpowerf1[2] [dev e8:01][0]
71384704 blocks [2/1] [U_]
So may be it will be better to create a raid device on top of the lvm volume.
Quoted 21 lines · Eric Monjoin
Quoted 5 lines
I would like to know if it's possible (works perfectly) to create a software mirror (md0) on top of 2 LVM logical volumes :
You'd usually want to make your raid devices first, then put LVM on top of it. I can't really think of any benefits of doing it the other way around.
------------------------------------------------------------------------
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 11 lines · Eric Monjoin
Theo Van Dinter a �crit :
Well it's because we have problems in this way. We have a server connected to 2 EMC Symmetrix where we assign some 70Gb and 40Gb Luns. We used Powerpath to manage the dual path to the Luns and so I first created mirror as this :
But after a while I obtain that :
Personalities : [raid1]read_ahead1024 sectors
Event: 10 md9 : active raid1 [dev e9:31][1] [dev e8:e1][0]42829184 blocks [2/2] [UU]and if we try to rebuild the mirror after after loosing access to one of
this might be a problem with powerpath, it seems harmless tough
Quoted 7 lines
the EMC, we have really bad result :
Personalities : [raid1]read_ahead1024 sectors
Event: 26 md9 : active raid1 emcpowerd1[2] [dev e8:e1][0]42829184 blocks [2/1] [U_] [>....................] recovery = 1.4% (630168/42829184)finish=68.1min speed=10315K/sec
this is a problem with linux md raid. it does not support fast resync with a bitmap of changed sectors, there is a project to implement it, but it is not yet in the standard kernel.
look on the linux-raid mailing list archives for something called 'fast raid 1'.
i do not believe you would have any advantage in stacking md above lvm anyway.
Regards,
Luca
Signature · 6 lines
Luca Berra -- [email protected] Communication Media & Services S.r.l. /"\ \ / ASCII RIBBON CAMPAIGN X AGAINST HTML MAIL / \
Quoted 12 lines · Theo Van Dinter
Quoted 3 lines · Eric Monjoin
I would like to know if it's possible (works perfectly) to create a software mirror (md0) on top of 2 LVM logical volumes :
You'd usually want to make your raid devices first, then put LVM on top of it. I can't really think of any benefits of doing it the other way around.
I've thought about this numerous times -- there is the distinct resizing advantage. Namely, if I create a software RAID partition, I can't resize it afterward without destroying it. I have for example, on occasion, had three disks set up where 1/3 of each was devoted to a RAID-0 very fast striping set for data transfers that had to be fast but if they were lost it wasn't critical, and 2/3 was set up as RAID-5 for reliability of another set of data.
I understand this may not be optimal in some situations, but some of us don't have thousands of dollars for SCSI HW RAID controllers.
Signature · 1 line
Michael T. Babcock
Quoted 10 lines · Luca Berra
Quoted 4 lines · Eric Monjoin
md9 : active raid1 emcpowerd1[2] [dev e8:e1][0]
42829184 blocks [2/1] [U_] [>....................] recovery = 1.4% (630168/42829184)finish=68.1min speed=10315K/sec
this is a problem with linux md raid. it does not support fast resync with a bitmap of changed sectors, there is a project to implement it, but it is not yet in the standard kernel.
look on the linux-raid mailing list archives for something called 'fast raid 1'.
fr1.sf.net (also fr5.sf.net may be of interest).
and http://www.it.uc3m.es/ptb/fr1
(yes, I am the author). Those are for 2.4 kernels. A version fo fr1 for 2.6 kernels with additional ondisk bitmaps has been done by Paul Clements, the linux nbd maintainer, and patches by him are around.
I should look them up and merge the two .. bids accepted!
Quoted 2 lines
i do not believe you would have any advantage in stacking md above lvm anyway.
That is right. In fact it sounds downright risky to me.
Peter
[linux-lvm] What is the best way to configure LVM + RAID?
In reply to Michael T. BabcockOK, so like what is the best way to configure multiple inconsistently sized disks into a RAID5 LVM? Will you still be able to resize the array by adding and dropping disks?
Just wanting to know.
Thanks,
Erik.
Quoted 7 lines · Michael T. Babcock
Quoted 3 lines
You'd usually want to make your raid devices first, then put LVM on top of it. I can't really think of any benefits of doing it the other way around.
I've thought about this numerous times -- there is the distinct resizing advantage. Namely, if I create a software RAID partition, I can't resize it afterward without destroying it. I have for example, on
There's no resizing advantage at all -- if you add another disk to the LVM, you can just as easily add it to a RAID array instead.
Never mind the fact that if you lose any disk in the LVM, your RAID groups are dead (unless you split up your LVM disk, but that's what RAID does ...)
LVM gives you flexibility in your disk layout, RAID gives you performance
+ reliability. LVM builds on RAID's features, but the other way around, RAID is undercut by LVM.
Quoted 2 lines
I understand this may not be optimal in some situations, but some of us don't have thousands of dollars for SCSI HW RAID controllers.
Neither do I, but this is irrelevent to the conversation.
Signature · 4 lines
Randomly Generated Tagline: "A bride must be carried over this, but a vampire must be invited into it." - Jeopardy Answer "What is a Supermarket Spacer, Alex?" - Theo's Response
Quoted 16 lines · Michael T. Babcock
Quoted 13 lines · Theo Van Dinter
Quoted 4 lines · Eric Monjoin
I would like to know if it's possible (works perfectly) to create a software mirror (md0) on top of 2 LVM logical volumes :
You'd usually want to make your raid devices first, then put LVM on top of it. I can't really think of any benefits of doing it the other way around.
Quoted 7 lines
I've thought about this numerous times -- there is the distinct resizing advantage. Namely, if I create a software RAID partition, I can't resize it afterward without destroying it. I have for example, on occasion, had three disks set up where 1/3 of each was devoted to a RAID-0 very fast striping set for data transfers that had to be fast but if they were lost it wasn't critical, and 2/3 was set up as RAID-5 for reliability of another set of data.
That would be LVM over raid, not the other way round. You make a raid0 out of 1/3 of each disk (set up a partition on each) nd a raid5 out of 2/3 of each disk. Then you combine and/or divvy up the two raid areas using lvm.
If you have lvm under raid, you STILL can't change the size of the raided device without remaking it completely, so what's the point? (or does lvm somehow magically permit raid0 resizing on the fly above it - that would be just about possible, but would require cooperation).
Peter
[linux-lvm] LVM DISK DIE
In reply to Theo Van DinterHi
i have a lvm whit 4 disk on a machine whit lvm2 and kernel 2.6.8.1 . One is dead, i've tried whit pvmove to 2 disk (200 gb -> 120 +80 ) but it die before moving 30 Gb.
Now the disk i think is totaly unusable, cfdisk refuse to open it (like opening a non exitent device) and i have this:
root@Orange:~# vgscan Reading all physical volumes. This may take a while... Couldn't find device with uuid '42cK3V-88Wm-SEMU-1qUb-fAsZ-Fo1V-QFzFvG'. Couldn't find all physical volumes for volume group orange_vg. Couldn't find device with uuid '42cK3V-88Wm-SEMU-1qUb-fAsZ-Fo1V-QFzFvG'. Couldn't find all physical volumes for volume group orange_vg. Couldn't find device with uuid '42cK3V-88Wm-SEMU-1qUb-fAsZ-Fo1V-QFzFvG'. Couldn't find all physical volumes for volume group orange_vg. Couldn't find device with uuid '42cK3V-88Wm-SEMU-1qUb-fAsZ-Fo1V-QFzFvG'. Couldn't find all physical volumes for volume group orange_vg.
and also this:
root@Orange:~# pvscan Couldn't find device with uuid '42cK3V-88Wm-SEMU-1qUb-fAsZ-Fo1V-QFzFvG'. Couldn't find device with uuid '9X3e0u-IqHl-B7Eh-CcGa-7Zbp-Awv8-5dfBnj'. Couldn't find device with uuid '42cK3V-88Wm-SEMU-1qUb-fAsZ-Fo1V-QFzFvG'. Couldn't find device with uuid '9X3e0u-IqHl-B7Eh-CcGa-7Zbp-Awv8-5dfBnj'. Couldn't find device with uuid '42cK3V-88Wm-SEMU-1qUb-fAsZ-Fo1V-QFzFvG'. Couldn't find device with uuid '9X3e0u-IqHl-B7Eh-CcGa-7Zbp-Awv8-5dfBnj'. Couldn't find device with uuid '42cK3V-88Wm-SEMU-1qUb-fAsZ-Fo1V-QFzFvG'. Couldn't find device with uuid '9X3e0u-IqHl-B7Eh-CcGa-7Zbp-Awv8-5dfBnj'. PV /dev/hdb VG orange_vg lvm2 [189.92 GB / 0 free] PV /dev/hdc VG orange_vg lvm2 [189.92 GB / 0 free] PV unknown device VG orange_vg lvm2 [189.92 GB / 0 free] PV unknown device VG orange_vg lvm2 [189.92 GB / 150.89 GB free] PV /dev/hdg VG orange_vg lvm2 [114.49 GB / 956.00 MB free] PV /dev/hdh VG orange_vg lvm2 [74.52 GB / 4.00 MB free] Total: 6 [948.70 GB] / in use: 6 [948.70 GB] / in no VG: 0 [0 ]
There is a way to save something ?
Thanks
Quoted 4 lines · Michael T. Babcock
I've thought about this numerous times -- there is the distinct resizing advantage. Namely, if I create a software RAID partition, I can't resize it afterward without destroying it. I have for example, on occasion, had three disks set up where 1/3 of each was devoted to a RAID-0 very fast striping set for data transfers that had to be fast but if they were lost it wasn't critical, and 2/3 was set up as RAID-5 for reliability of another set of data.
I understand this may not be optimal in some situations, but some of us don't have thousands of dollars for SCSI HW RAID controllers.
You can make life easier by creating several raid arrays on the same disks and tie them together with LVM. For example split your three 160GB drives into 4 partitions each and create a raid over the three disks.
sda -> sda1+sda2+sda3+sda4, etc.
md0 = sda1+sdb1+sdc1, md1=sda2+sdb2+sdc2, etc.
Markus
Signature · 4 lines
Markus Baertschi Phone: ++41 (21) 807 1677 Bas du Ross� 14b Fax : ++41 (21) 807 1678 CH-1163, Etoy Email: [email protected] Switzerland Homepage: www.markus.org
[linux-lvm] Re: What is the best way to configure LVM + RAID?
In reply to Erik OhrnbergerQuoted 3 lines · Erik Ohrnberger
OK, so like what is the best way to configure multiple inconsistently sized disks into a RAID5 LVM? Will you still be able to resize the array by adding and dropping disks?
Well, you'll just have to make a raid5 of the minimum size (of any three of your disks :), and do something else with the rest of the remaining space.
Peter
I've done something like Markus explained. I had this email ready to go a couple days ago, but the compose window was hidden for a while... I hope it's not too stale for the original poster of this thread:
2.4.26 with latest LVM2 works well. I did something crazy, alot of people probably don't understand why.
I have software RAID5.
md0 = hda5 hdb5 hdc5 hdd5 hde5 md1 = hda6 hdb6 hdc6 hdd6 hde6 md2 = hda7 hdb7 hdc7 hdd7 md3 = hda8 hdb8 hdc8 hdd8
Then I have just one big vgdata and one lvdata.
Why would I want to do this? I couldn't find any other way to have redundancy and be flexible. If I replace a physical disk with a larger one, for instance hde needs to be bigger so I can make md2 and md3 stretch across that disk too... then I am able to do pvmove from md2 to a scratch disk, recreate md2 across hda7-hde7, then pvmove back.
-Scott
Quoted 16 lines · Markus Baertschi
Quoted 4 lines · Michael T. Babcock
I've thought about this numerous times -- there is the distinct resizing advantage. Namely, if I create a software RAID partition, I can't resize it afterward without destroying it. I have for example, on occasion, had three disks set up where 1/3 of each was devoted to a RAID-0 very fast striping set for data transfers that had to be fast but if they were lost it wasn't critical, and 2/3 was set up as RAID-5 for reliability of another set of data.
I understand this may not be optimal in some situations, but some of us don't have thousands of dollars for SCSI HW RAID controllers.
You can make life easier by creating several raid arrays on the same disks and tie them together with LVM. For example split your three 160GB drives into 4 partitions each and create a raid over the three disks.
sda -> sda1+sda2+sda3+sda4, etc.
md0 = sda1+sdb1+sdc1, md1=sda2+sdb2+sdc2, etc.Markus
Quoted 5 lines · Markus Baertschi
You can make life easier by creating several raid arrays on the same disks and tie them together with LVM. For example split your three 160GB drives into 4 partitions each and create a raid over the three disks.
sda -> sda1+sda2+sda3+sda4, etc.
md0 = sda1+sdb1+sdc1, md1=sda2+sdb2+sdc2, etc.
And how exactly does that allow me to easily change how much RAID-0 or RAID-5 storage I have?
If I need more RAID-5 storage and I want to steal it from RAID-0 space, I need to dissassemble LVM, then software RAID, then repartition, then rebuild RAID, then restore LVM.
Signature · 2 lines
Michael T. Babcock http://mikebabcock.ca
Quoted 8 lines · Peter T. Breuer
Quoted 4 lines
i do not believe you would have any advantage in stacking md above lvm anyway.
That is right. In fact it sounds downright risky to me.
I've actually considered a few times how one might be able to merge the md functionality into LVM2 (device-mapper) for having software-RAID part of the logical volume system (since a software-RAID partition is a logical volume of another type anyway).
It would require some significant effort, but it seems doable.
Signature · 2 lines
Michael T. Babcock http://mikebabcock.ca
This solution would seem to be as the holy grail in that it would give you the best of all worlds: resizable RAID storage. Or am I wrong about this assumption?
Quoted 25 lines · Michael T. Babcock
Quoted 8 lines · Peter T. Breuer
Quoted 4 lines
i do not believe you would have any advantage in stacking md above lvm anyway.
That is right. In fact it sounds downright risky to me.
I've actually considered a few times how one might be able to merge the md functionality into LVM2 (device-mapper) for having software-RAID part of the logical volume system (since a software-RAID partition is a logical volume of another type anyway).
It would require some significant effort, but it seems doable. --
Michael T. Babcock
http://mikebabcock.ca
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 2 lines · Michael T. Babcock
And how exactly does that allow me to easily change how much RAID-0 or RAID-5 storage I have?
It doesn't.
Quoted 4 lines
If I need more RAID-5 storage and I want to steal it from RAID-0 space, I need to dissassemble LVM, then software RAID, then repartition, then rebuild RAID, then restore LVM.
You have to anyway. You can't change the size of a raid on the fly.
Peter
Quoted 4 lines · Erik Ohrnberger
This solution would seem to be as the holy grail in that it would give you the best of all worlds: resizable RAID storage. Or am I wrong about this assumption?
Isn't that what NetApps do already with their "modified RAID4"?
Quoted 4 lines · Clint Byrum
Quoted 3 lines
This solution would seem to be as the holy grail in that it would give you the best of all worlds: resizable RAID storage. Or am I wrong about this assumption?
Isn't that what NetApps do already with their "modified RAID4"?
Most RAID systems let you relayout the disk. NetApps are just another example, so is Veritas.
Signature · 6 lines
Randomly Generated Tagline:
quit When the quit statement is read, the bc processor
is terminated, regardless of where the quit state-
ment is found. For example, "if (0 == 1) quit"
will cause bc to terminate.
(Seen in the manpage for "bc". Note the "if" statement's logic)Quoted 5 lines · Erik Ohrnberger
This solution would seem to be as the holy grail in that it would give you the best of all worlds: resizable RAID storage. Or am I wrong about this assumption?
I've been trying to figure out a reason why it wouldn't be doable for a few weeks now.
Currently, one can create RAID-0 like performance with LVM2 by specifying stripes for one's LVs. Why not:
lvcreate -L 3GB --raid-level 5 --stripes 3 -n SafeData VG001 /dev/sda /dev/sdb /dev/sdc
PV section 1 on sda and sdb are data, section 1 on sdc is XOR data. PV section 2 on sda and sdc are data, and XOR data on section 2 of sdb, and so on.
One of the major performance implications would be the larger block sizes being dealt with; many people seem to find smaller stripes better for performance in RAID-0/5 situations, and by default one would have 4MB stripes. With some effort, I'm sure this could be rectified as well.
Thoughts? Or should I blog about it instead? ;-)
If I didn't have multiple out-of-town server and firewall installations this week, I'd code instead :-)
Signature · 2 lines
Michael T. Babcock http://mikebabcock.ca