help using LVM & md for RAID10
Hi,
I'm trying to use LVM together with md. I know you guys don't recommend this combination, but I need it to get good performance from our disks plus some protection. This combination used to work with 0.3, but stop working with the new 1.0.7. I think the problem is that after rebooting LVM has problem figuring out that it needs to use md0-3 and not their corresponding part (/dev/sd?).
Have you done this before?
I'm adding the series of commands we used to create the volume, plus the status after we reboot the machine.
CREATING THE VOLUME
# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid5] [multipath]
read_ahead 1024 sectors
md4 : active raid0 md3[3] md2[2] md1[1] md0[0]
143372288 blocks 512k chunks
md3 : active raid1 sdi[1] sdh[0]
35843584 blocks [2/2] [UU]
md2 : active raid1 sdg[1] sdf[0]
35843584 blocks [2/2] [UU]
md1 : active raid1 sde[1] sdd[0]
35843584 blocks [2/2] [UU]
md0 : active raid1 sdc[1] sdb[0]
35843584 blocks [2/2] [UU]
unused devices: <none>#dd bs=256k if=/dev/zero of=/dev/md0 count=1000
#dd bs=256k if=/dev/zero of=/dev/md1 count=1000
#dd bs=256k if=/dev/zero of=/dev/md2 count=1000
#dd bs=256k if=/dev/zero of=/dev/md3 count=1000
# pvcreate /dev/md0 pvcreate -- physical volume "/dev/md0" successfully created
# pvcreate /dev/md1 pvcreate -- physical volume "/dev/md1" successfully created
# pvcreate /dev/md2 pvcreate -- physical volume "/dev/md2" successfully created
# pvcreate /dev/md3 pvcreate -- physical volume "/dev/md3" successfully created
# vgcreate vg01 /dev/md[0-3] vgcreate -- INFO: using default physical extent size 32 MB vgcreate -- INFO: maximum logical volume size is 2 Terabyte vgcreate -- doing automatic backup of volume group "vg01" vgcreate -- volume group "vg01" successfully created and activated
# pvscan pvscan -- reading all physical volumes (this may take a while...) pvscan -- ACTIVE PV "/dev/md0" of VG "vg01" [34.12 GB / 34.12 GB free]
pvscan -- ACTIVE PV "/dev/md1" of VG "vg01" [34.12 GB / 34.12 GB free]
pvscan -- ACTIVE PV "/dev/md2" of VG "vg01" [34.12 GB / 34.12 GB free]
pvscan -- ACTIVE PV "/dev/md3" of VG "vg01" [34.12 GB / 34.12 GB free]
pvscan -- inactive PV "/dev/md4" of VG "vg01" [34.12 GB / 34.12 GB free]
pvscan -- WARNING: physical volume "/dev/sdb" belongs to a meta device pvscan -- WARNING: physical volume "/dev/sdc" belongs to a meta device pvscan -- WARNING: physical volume "/dev/sdd" belongs to a meta device pvscan -- WARNING: physical volume "/dev/sde" belongs to a meta device pvscan -- WARNING: physical volume "/dev/sdf" belongs to a meta device pvscan -- WARNING: physical volume "/dev/sdg" belongs to a meta device pvscan -- WARNING: physical volume "/dev/sdh" belongs to a meta device pvscan -- WARNING: physical volume "/dev/sdi" belongs to a meta device pvscan -- total: 13 [170.92 GB] / in use: 13 [170.92 GB] / in no VG: 0
[0]
#lvcreate -i 4 -I 512 -L 130G -n lv01 vg01
lvcreate -- doing automatic backup of "vg01"
lvcreate -- logical volume "/dev/vg01/lv01" successfully created
# vgdisplay -v --- Volume group --- VG Name vg01 VG Access read/write VG Status available/resizable VG # 0 MAX LV 256 Cur LV 1 Open LV 0 MAX LV Size 2 TB Max PV 256 Cur PV 4 Act PV 4 VG Size 136.50 GB PE Size 32 MB Total PE 4368 Alloc PE / Size 4160 / 130 GB Free PE / Size 208 / 6.50 GB VG UUID tXh4cm-QVz6-bT6a-yLRl-PnHc-cY16-72aPfp
--- Logical volume ---
LV Name /dev/vg01/lv01 VG Name vg01 LV Write Access read/write LV Status available LV # 1 # open 0 LV Size 130 GB Current LE 4160 Allocated LE 4160 Stripes 4 Stripe size (KByte) 512 Allocation next free Read ahead sectors 1024 Block device 58:0
--- Physical volumes ---
PV Name (#) /dev/md0 (1) PV Status available / allocatable Total PE / Free PE 1092 / 52
PV Name (#) /dev/md1 (2) PV Status available / allocatable Total PE / Free PE 1092 / 52
PV Name (#) /dev/md2 (3) PV Status available / allocatable Total PE / Free PE 1092 / 52
PV Name (#) /dev/md3 (4) PV Status available / allocatable Total PE / Free PE 1092 / 52
AFTER REBOOT
# pvscan pvscan -- reading all physical volumes (this may take a while...) pvscan -- inactive PV "/dev/md0" of VG "vg01" [34.12 GB / 1.62 GB free] pvscan -- inactive PV "/dev/md1" of VG "vg01" [34.12 GB / 1.62 GB free] pvscan -- inactive PV "/dev/md2" of VG "vg01" [34.12 GB / 1.62 GB free] pvscan -- inactive PV "/dev/md3" of VG "vg01" [34.12 GB / 1.62 GB free] pvscan -- inactive PV "/dev/md4" of VG "vg01" [34.12 GB / 1.62 GB free] pvscan -- WARNING: physical volume "/dev/sdb" belongs to a meta device pvscan -- WARNING: physical volume "/dev/sdc" belongs to a meta device pvscan -- WARNING: physical volume "/dev/sdd" belongs to a meta device pvscan -- WARNING: physical volume "/dev/sde" belongs to a meta device pvscan -- WARNING: physical volume "/dev/sdf" belongs to a meta device pvscan -- WARNING: physical volume "/dev/sdg" belongs to a meta device pvscan -- WARNING: physical volume "/dev/sdh" belongs to a meta device pvscan -- WARNING: physical volume "/dev/sdi" belongs to a meta device pvscan -- total: 13 [170.92 GB] / in use: 13 [170.92 GB] / in no VG: 0 [0]
[root@sculpin root]# vgdisplay -v
--- Volume group ---
VG Name vg01 VG Access read/write VG Status available/resizable VG # 0 MAX LV 256 Cur LV 1 Open LV 0 MAX LV Size 2 TB Max PV 256 Cur PV 4 Act PV 4 VG Size 136.50 GB PE Size 32 MB Total PE 4368 Alloc PE / Size 4160 / 130 GB Free PE / Size 208 / 6.50 GB VG UUID tXh4cm-QVz6-bT6a-yLRl-PnHc-cY16-72aPfp
--- Logical volume ---
LV Name /dev/vg01/lv01 VG Name vg01 LV Write Access read/write LV Status available LV # 1 # open 0 LV Size 130 GB Current LE 4160 Allocated LE 4160 Stripes 4 Stripe size (KByte) 512 Allocation next free Read ahead sectors 1024 Block device 58:0
--- Physical volumes ---
PV Name (#) /dev/sdc (1) PV Status available / allocatable Total PE / Free PE 1092 / 52
PV Name (#) /dev/sde (2) PV Status available / allocatable Total PE / Free PE 1092 / 52
PV Name (#) /dev/sdg (3) PV Status available / allocatable Total PE / Free PE 1092 / 52
PV Name (#) /dev/sdi (4) PV Status available / allocatable Total PE / Free PE 1092 / 52
Quoted 15 lines · Alberto
# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid5] [multipath]
read_ahead 1024 sectors
md4 : active raid0 md3[3] md2[2] md1[1] md0[0]
143372288 blocks 512k chunks
md3 : active raid1 sdi[1] sdh[0]
35843584 blocks [2/2] [UU]
md2 : active raid1 sdg[1] sdf[0]
35843584 blocks [2/2] [UU]
md1 : active raid1 sde[1] sdd[0]
35843584 blocks [2/2] [UU]
md0 : active raid1 sdc[1] sdb[0]
35843584 blocks [2/2] [UU]
unused devices: <none>Quoted 6 lines
# pvcreate /dev/md0 # pvcreate /dev/md1 # pvcreate /dev/md2 # pvcreate /dev/md3 # vgcreate vg01 /dev/md[0-3]
In short, you set up a RAID1+0, four RAID1 devices bounded together in one big RAID0 (md4). But next, you configure md[0-3] as PVs, instead of just creating a PV from md4, the higher level RAID device.
You should just do:
# pvcreate /dev/md4 # vgcreate vg01 /dev/md4
And then create LV from this new VG (vg01). Should work, and you get high performance (RAID0 md4), protection agains physical disk failures (between one and four disk crashes, depends on luck), and the ability to hot create, modify and destroy "partitions" (LVM).
Hope it helps.
Signature · 2 lines
Jose Luis Domingo Lopez Linux Registered User #189436 Debian Linux Sid (Linux 2.5.69)