Skip to contentSkip to conversation

LV mirror extents on same physical device

linux-lvm Julie Ashworth 5 participants 16 messages Feb 22, 2014 – Feb 25, 2014

lore
Share & tools
lore
#0 lore raw

hi all,

just some background...
I installed a CentOS5 server in 2010 with a mdadm RAID1 containing 2 partitions (/dev/sda1 and /dev/sdb1). /dev/sda1 failed, and unfortuately, the mirror failed to sync with a new drive.

On this RAID1, I have one VG (VolGroup00) with 5 LVs.

I contacted the folks on the mdadm mailing list. The data on /dev/sdb is good, so they recommended that I convert the linear LVs to mirrors (to copy the data to a new drive).

I formatted a new drive (/dev/sde) identical to /dev/sd[ab], added it to a mdadm device (/dev/md2), which I then added to my VG:
mdadm --create --level=1 -n 2 /dev/md2 /dev/sde2 missing
pvcreate /dev/md2
vgextend VolGroup00 /dev/md2

I then converted all my volumes to mirrors:
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/srvlv lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/rootlv lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/swaplv lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/varlv lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/tmplv

(in case you're wondering, when this completed, I added /dev/sda1 to /dev/md2, so /dev/md2 is a fuctional mdadm RAID1).

Now... the problem ;)....

When I run 'lvs' I see that the LV mirror extents were created on the same device (/dev/md1):

# lvs -a -o +devices 
LV                     VG         Attr   LSize  Origin Snap%  Move Log         Copy%  Convert Devices                                        
rootlv                 VolGroup00 mwi-ao 10.00G                    rootlv_mlog 100.00         rootlv_mimage_0(0),rootlv_mimage_1(0)          
[rootlv_mimage_0]      VolGroup00 iwi-ao 10.00G                                               /dev/md1(0)                                    
[rootlv_mimage_1]      VolGroup00 iwi-ao 10.00G                                               /dev/md1(4036)                                 
[rootlv_mlog]          VolGroup00 mwa-ao 32.00M                                100.00         rootlv_mlog_mimage_0(0),rootlv_mlog_mimage_1(0)
[rootlv_mlog_mimage_0] VolGroup00 iwi-ao 32.00M                                               /dev/md2(4)                                    
[rootlv_mlog_mimage_1] VolGroup00 iwi-ao 32.00M                                               /dev/md1(4356)      

My long-term goal is to convert the mirror back to a linear volume using just /dev/md2, i.e. lvconvert -m0 /dev/VolGroup00/rootlv /dev/md1
lvconvert -m0 /dev/VolGroup00/swaplv /dev/md1
lvconvert -m0 /dev/VolGroup00/srvlv /dev/md1
lvconvert -m0 /dev/VolGroup00/varlv /dev/md1
lvconvert -m0 /dev/VolGroup00/tmplv /dev/md1

... so it seems problematic that both extents are on /dev/md1.

I thought lvm was supposed to use both physical devices. Do you know why both extents were created on the same physical device? Should I create another extent on /dev/md2 before I remove the /dev/md1 extent(s)? If so, can you give me an example of how to use lvconvert in this case?

Thanks in advance!
best,
Julie

#1 lore raw

i don't understand why you are using two mirror technologies together, i'm using in lvm mirror without mdadm and never found problems

2014-02-22 8:14 GMT+01:00 Julie Ashworth <[email protected]>:

Quoted 78 lines

hi all,

just some background...
I installed a CentOS5 server in 2010 with a mdadm RAID1 containing 2 partitions (/dev/sda1 and /dev/sdb1). /dev/sda1 failed, and unfortuately, the mirror failed to sync with a new drive.

On this RAID1, I have one VG (VolGroup00) with 5 LVs.

I contacted the folks on the mdadm mailing list. The data on /dev/sdb is good, so they recommended that I convert the linear LVs to mirrors (to copy the data to a new drive).

I formatted a new drive (/dev/sde) identical to /dev/sd[ab], added it to a mdadm device (/dev/md2), which I then added to my VG:
mdadm --create --level=1 -n 2 /dev/md2 /dev/sde2 missing pvcreate /dev/md2
vgextend VolGroup00 /dev/md2

I then converted all my volumes to mirrors:
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/srvlv lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/rootlv lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/swaplv lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/varlv lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/tmplv

(in case you're wondering, when this completed, I added /dev/sda1 to /dev/md2, so /dev/md2 is a fuctional mdadm RAID1).

Now... the problem ;)....

When I run 'lvs' I see that the LV mirror extents were created on the same device (/dev/md1):

# lvs -a -o +devices
LV                     VG         Attr   LSize  Origin Snap%  Move Log
    Copy%  Convert Devices
rootlv                 VolGroup00 mwi-ao 10.00G
 rootlv_mlog 100.00         rootlv_mimage_0(0),rootlv_mimage_1(0)
[rootlv_mimage_0]      VolGroup00 iwi-ao 10.00G
                    /dev/md1(0)
[rootlv_mimage_1]      VolGroup00 iwi-ao 10.00G
                    /dev/md1(4036)
[rootlv_mlog]          VolGroup00 mwa-ao 32.00M
     100.00         rootlv_mlog_mimage_0(0),rootlv_mlog_mimage_1(0)
[rootlv_mlog_mimage_0] VolGroup00 iwi-ao 32.00M
                    /dev/md2(4)
[rootlv_mlog_mimage_1] VolGroup00 iwi-ao 32.00M
                    /dev/md1(4356)

My long-term goal is to convert the mirror back to a linear volume using just /dev/md2, i.e.
lvconvert -m0 /dev/VolGroup00/rootlv /dev/md1
lvconvert -m0 /dev/VolGroup00/swaplv /dev/md1
lvconvert -m0 /dev/VolGroup00/srvlv /dev/md1
lvconvert -m0 /dev/VolGroup00/varlv /dev/md1
lvconvert -m0 /dev/VolGroup00/tmplv /dev/md1

... so it seems problematic that both extents are on /dev/md1.

I thought lvm was supposed to use both physical devices. Do you know why both extents were created on the same physical device? Should I create another extent on /dev/md2 before I remove the /dev/md1 extent(s)? If so, can you give me an example of how to use lvconvert in this case?

Thanks in advance!
best,
Julie

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Signature · 1 line
esta es mi vida e me la vivo hasta que dios quiera
#2 lore raw
Quoted 2 lines · emmanuel segura

i don't understand why you are using two mirror technologies together, i'm using in lvm mirror without mdadm and never found problems

---end quoted text---

hi Emmanuel,
I personally prefer to use mdadm RAID1, and I'm only using lvm mirroring to recover from an mdadm failure. If you want to learn more, I encourage you to join the mailing list for mdadm issues ([email protected]). best,
Julie

#3 lore raw

I doubt anybody else will have this problem, but just in case...
my solution was to upgrade lvm2, convert my mirrors to linear arrays, and recreate the mirrors. After the lvm2 upgrade, I didn't see allocation errors when creating the mirrored log.

lvconvert -m0 /dev/VolGroup00/lvname
lvconvert -m1 --mirrorlog=mirrored /dev/VolGroup00/lvname

best,
Julie

#4 lore raw

Hi Julie,

Thanks for your comment, but you mirror over mirror, i don't need to learn anything like that :)

2014-02-24 22:51 GMT+01:00 Julie Ashworth <[email protected]>:

Quoted 17 lines

I doubt anybody else will have this problem, but just in case... my solution was to upgrade lvm2, convert my mirrors to linear arrays, and recreate the mirrors. After the lvm2 upgrade, I didn't see allocation errors when creating the mirrored log.

lvconvert -m0 /dev/VolGroup00/lvname
lvconvert -m1 --mirrorlog=mirrored /dev/VolGroup00/lvname

best,
Julie

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Signature · 1 line
esta es mi vida e me la vivo hasta que dios quiera
#5 lore raw

Hi Julie,

Thanks for your comment, but you are using mirror over mirror, i don't need to learn anything like that :)

2014-02-25 12:18 GMT+01:00 emmanuel segura <[email protected]>:

Quoted 32 lines

Hi Julie,

Thanks for your comment, but you mirror over mirror, i don't need to learn anything like that :)

2014-02-24 22:51 GMT+01:00 Julie Ashworth <[email protected]>:

I doubt anybody else will have this problem, but just in case...

Quoted 7 lines

my solution was to upgrade lvm2, convert my mirrors to linear arrays, and recreate the mirrors. After the lvm2 upgrade, I didn't see allocation errors when creating the mirrored log.

lvconvert -m0 /dev/VolGroup00/lvname
lvconvert -m1 --mirrorlog=mirrored /dev/VolGroup00/lvname

Quoted 9 lines

best,
Julie

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

--
esta es mi vida e me la vivo hasta que dios quiera

Signature · 1 line
esta es mi vida e me la vivo hasta que dios quiera
#6 lore raw
In reply to Julie Ashworth

Ok I am not really sure what you are trying to do with madm, from what I can see you want to move data from pv on md1 to a pv on md2, this is really easy to do.

#pvcreate /dev/md2
#vgextend VolGroup00 /dev/md2
# Stop the old pv getting lvs created on it
pvchange -x n /dev/md1
# Whole thing in one go
pvmove -v /dev/md1 /dev/md2
# One at a time
pvmove -v -n rootlv /dev/md1 /dev/md2
vgreduce VolGroup00 /dev/md2

Your done.

James

#7 lore raw
Quoted 10 lines · James Hawtin

#pvcreate /dev/md2
#vgextend VolGroup00 /dev/md2
# Stop the old pv getting lvs created on it
pvchange -x n /dev/md1
# Whole thing in one go
pvmove -v /dev/md1 /dev/md2
# One at a time
pvmove -v -n rootlv /dev/md1 /dev/md2
vgreduce VolGroup00 /dev/md2

---end quoted text---

My LVs are already mirrored (w/ two extents on /dev/md1), so is pvmove really the right tool to move one of those extents? best,
Julie

#8 lore raw
Quoted 10 lines · James Hawtin
Quoted 10 lines

#pvcreate /dev/md2
#vgextend VolGroup00 /dev/md2
# Stop the old pv getting lvs created on it
pvchange -x n /dev/md1
# Whole thing in one go
pvmove -v /dev/md1 /dev/md2
# One at a time
pvmove -v -n rootlv /dev/md1 /dev/md2
vgreduce VolGroup00 /dev/md2

Julie> ---end quoted text---

Julie> My LVs are already mirrored (w/ two extents on /dev/md1), so is Julie> pvmove really the right tool to move one of those extents? Julie> best,

Julie,

I think the problem is that you're confusing the Mirroring done at the MD level, which is /dev/md#, and then the mirroring at the LVM level, which is stacked on top of MD. So there's no need to mirror at both levels. But let's double check your setup, so can you please reply with the output of the following commands:

cat /proc/mounts
cat /proc/mdstat
pvs
vgs
lvs

This will clarify exactly what your setup it and how it looks.

John

#9 lore raw
Quoted 6 lines · John Stoffel

I think the problem is that you're confusing the Mirroring done at the MD level, which is /dev/md#, and then the mirroring at the LVM level, which is stacked on top of MD. So there's no need to mirror at both levels. But let's double check your setup, so can you please reply with the output of the following commands:

I'm not confused about mdadm vs. LVM mirror. I regret mentioning mdadm, since it seems to be confusing everyone. I should have only asked why the lvconvert command produced extents on the same physical device. At this point, I will just go ahead and test if lvconvert -m0 /dev/VolGroup00/rootlv /dev/md1
will produce a functional linear volume on /dev/md2

best,
Julie

#10 lore raw
In reply to Julie Ashworth

If I am reading what I can find correctly - -alloc=anywhere defeats the mirror code from attempting to put on 2 different physical devices.

The man pages seem to lack info on what other values there are for that but disables device mirroring.

Quoted 59 lines · Julie Ashworth

hi all,

just some background...
I installed a CentOS5 server in 2010 with a mdadm RAID1 containing 2 partitions (/dev/sda1 and /dev/sdb1). /dev/sda1 failed, and unfortuately, the mirror failed to sync with a new drive.

On this RAID1, I have one VG (VolGroup00) with 5 LVs.

I contacted the folks on the mdadm mailing list. The data on /dev/sdb is good, so they recommended that I convert the linear LVs to mirrors (to copy the data to a new drive).

I formatted a new drive (/dev/sde) identical to /dev/sd[ab], added it to a mdadm device (/dev/md2), which I then added to my VG:
mdadm --create --level=1 -n 2 /dev/md2 /dev/sde2 missing
pvcreate /dev/md2
vgextend VolGroup00 /dev/md2

I then converted all my volumes to mirrors:
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/srvlv lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/rootlv lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/swaplv lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/varlv lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/tmplv

(in case you're wondering, when this completed, I added /dev/sda1 to /dev/md2, so /dev/md2 is a fuctional mdadm RAID1).

Now... the problem ;)....

When I run 'lvs' I see that the LV mirror extents were created on the same device (/dev/md1):

# lvs -a -o +devices
LV                     VG         Attr   LSize  Origin Snap%  Move Log         Copy%  Convert Devices
rootlv                 VolGroup00 mwi-ao 10.00G                    rootlv_mlog 100.00         rootlv_mimage_0(0),rootlv_mimage_1(0)
[rootlv_mimage_0]      VolGroup00 iwi-ao 10.00G                                               /dev/md1(0)
[rootlv_mimage_1]      VolGroup00 iwi-ao 10.00G                                               /dev/md1(4036)
[rootlv_mlog]          VolGroup00 mwa-ao 32.00M                                100.00         rootlv_mlog_mimage_0(0),rootlv_mlog_mimage_1(0)
[rootlv_mlog_mimage_0] VolGroup00 iwi-ao 32.00M                                               /dev/md2(4)
[rootlv_mlog_mimage_1] VolGroup00 iwi-ao 32.00M                                               /dev/md1(4356)

My long-term goal is to convert the mirror back to a linear volume using just /dev/md2, i.e. lvconvert -m0 /dev/VolGroup00/rootlv /dev/md1
lvconvert -m0 /dev/VolGroup00/swaplv /dev/md1
lvconvert -m0 /dev/VolGroup00/srvlv /dev/md1
lvconvert -m0 /dev/VolGroup00/varlv /dev/md1
lvconvert -m0 /dev/VolGroup00/tmplv /dev/md1

... so it seems problematic that both extents are on /dev/md1.

I thought lvm was supposed to use both physical devices. Do you know why both extents were created on the same physical device? Should I create another extent on /dev/md2 before I remove the /dev/md1 extent(s)? If so, can you give me an example of how to use lvconvert in this case?

Thanks in advance!
best,
Julie

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#11 lore raw

without the anywhere option it indicates you need *4* devices, 2 for data, and 2 for mirror log, using anywhere removes the more than 1 device for everything,.

Using this format should allow you to force the move of a copy from one device to another, see the pvmove man page for more details.

pvmove /dev/sdb1:1000-1999 /dev/sdc1:0-999

I have had to do something similar on some other OS (not Linux) LVM when it did incorrect things like this.

Quoted 67 lines · Roger Heflin

If I am reading what I can find correctly - -alloc=anywhere defeats the mirror code from attempting to put on 2 different physical devices.

The man pages seem to lack info on what other values there are for that but disables device mirroring.

Quoted 59 lines · Julie Ashworth

hi all,

just some background...
I installed a CentOS5 server in 2010 with a mdadm RAID1 containing 2 partitions (/dev/sda1 and /dev/sdb1). /dev/sda1 failed, and unfortuately, the mirror failed to sync with a new drive.

On this RAID1, I have one VG (VolGroup00) with 5 LVs.

I contacted the folks on the mdadm mailing list. The data on /dev/sdb is good, so they recommended that I convert the linear LVs to mirrors (to copy the data to a new drive).

I formatted a new drive (/dev/sde) identical to /dev/sd[ab], added it to a mdadm device (/dev/md2), which I then added to my VG:
mdadm --create --level=1 -n 2 /dev/md2 /dev/sde2 missing
pvcreate /dev/md2
vgextend VolGroup00 /dev/md2

I then converted all my volumes to mirrors:
lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/srvlv lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/rootlv lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/swaplv lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/varlv lvconvert -m1 --mirrorlog=mirrored --alloc=anywhere /dev/VolGroup00/tmplv

(in case you're wondering, when this completed, I added /dev/sda1 to /dev/md2, so /dev/md2 is a fuctional mdadm RAID1).

Now... the problem ;)....

When I run 'lvs' I see that the LV mirror extents were created on the same device (/dev/md1):

# lvs -a -o +devices
LV                     VG         Attr   LSize  Origin Snap%  Move Log         Copy%  Convert Devices
rootlv                 VolGroup00 mwi-ao 10.00G                    rootlv_mlog 100.00         rootlv_mimage_0(0),rootlv_mimage_1(0)
[rootlv_mimage_0]      VolGroup00 iwi-ao 10.00G                                               /dev/md1(0)
[rootlv_mimage_1]      VolGroup00 iwi-ao 10.00G                                               /dev/md1(4036)
[rootlv_mlog]          VolGroup00 mwa-ao 32.00M                                100.00         rootlv_mlog_mimage_0(0),rootlv_mlog_mimage_1(0)
[rootlv_mlog_mimage_0] VolGroup00 iwi-ao 32.00M                                               /dev/md2(4)
[rootlv_mlog_mimage_1] VolGroup00 iwi-ao 32.00M                                               /dev/md1(4356)

My long-term goal is to convert the mirror back to a linear volume using just /dev/md2, i.e. lvconvert -m0 /dev/VolGroup00/rootlv /dev/md1
lvconvert -m0 /dev/VolGroup00/swaplv /dev/md1
lvconvert -m0 /dev/VolGroup00/srvlv /dev/md1
lvconvert -m0 /dev/VolGroup00/varlv /dev/md1
lvconvert -m0 /dev/VolGroup00/tmplv /dev/md1

... so it seems problematic that both extents are on /dev/md1.

I thought lvm was supposed to use both physical devices. Do you know why both extents were created on the same physical device? Should I create another extent on /dev/md2 before I remove the /dev/md1 extent(s)? If so, can you give me an example of how to use lvconvert in this case?

Thanks in advance!
best,
Julie

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#12 lore raw
Quoted 3 lines · Roger Heflin

without the anywhere option it indicates you need *4* devices, 2 for data, and 2 for mirror log, using anywhere removes the more than 1 device for everything,.

The -alloc=aywhere simply refers to the log. I'm not concerned about the log, I'm concered about the actual image extents, which seem to be both on /dev/md1.

Quoted 4 lines

Using this format should allow you to force the move of a copy from one device to another, see the pvmove man page for more details.

pvmove /dev/sdb1:1000-1999 /dev/sdc1:0-999

The end goal is to convert the mirror to a linear volume (on /dev/md2).
I'm still not convinced that pvmove is the right utility for mirrored extents. But thanks for the advice - I will try it if lvconvert -m0 fails. (this is a test server).

best,
Julie

#13 lore raw
In reply to Roger Heflin
Quoted 3 lines · Roger Heflin

If I am reading what I can find correctly - -alloc=anywhere defeats the mirror code from attempting to put on 2 different physical devices.

---end quoted text---

-alloc=anywhere just refers to the logs, which isn't important at this point. best,
Julie

#14 lore raw

This indicates that it "worked" that way on some versions rhel5 but does not work that way on rhel6 in a number of versions.

https://bugzilla.redhat.com/show_bug.cgi?id=594525

I would suspect depending on exactly what version you are using that it is a bug.

I know up to at least 5.7 there are some bad LVM bugs when you are doing abnormal things (mirroring, pvmoves with lv with multiple extents), there are probably other bugs I don't know about.

I would suggest unmirroring it, and just doing a straight pvmove to the new device, mirroring it in the middle would just appear to complicate things.

Quoted 16 lines · Julie Ashworth
Quoted 3 lines · Roger Heflin

If I am reading what I can find correctly - -alloc=anywhere defeats the mirror code from attempting to put on 2 different physical devices.

---end quoted text---

-alloc=anywhere just refers to the logs, which isn't important at this point. best,
Julie

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#15 lore raw
Quoted 8 lines · Roger Heflin

This indicates that it "worked" that way on some versions rhel5 but does not work that way on rhel6 in a number of versions.

https://bugzilla.redhat.com/show_bug.cgi?id=594525

I would suspect depending on exactly what version you are using that it is a bug.

Quoted 7 lines

I know up to at least 5.7 there are some bad LVM bugs when you are doing abnormal things (mirroring, pvmoves with lv with multiple extents), there are probably other bugs I don't know about.

I would suggest unmirroring it, and just doing a straight pvmove to the new device, mirroring it in the middle would just appear to complicate things.

---end quoted text---

Thank you.
It's possible to specify where the extents should be located, i.e.

The following command creates a mirrored logical volume with a single mirror. The volume is 500 megabytes in size, it is named mirrorlv, and it is carved out of volume group vg0. The first leg of the mirror is on device /dev/sda1, the second leg of the mirror is on device /dev/sdb1, and the mirror log is on /dev/sdc1. lvcreate -L 500M -m1 -n mirrorlv vg0 /dev/sda1 /dev/sdb1 /dev/sdc1
(from http://linux-satya.blogspot.com/2010/09/logical-volume-manager.html)

I regret not doing this. Oh well. At this point, I haven't decided if I should convert to linear LV, or add aother mirror on /dev/md2.

Thanks again for the heads-up about the bug.
best,
Julie