Auto PV scan w/ LVM
I've got a couple of Clariion disk frames on which we performing cloning of LUNs for backups using HP-UX systems. Basically box A has a LUN or two, we clone it, and present it to box B - It mounts it, backs it up, then umounts it and we restart the cloning process... Nothing magical.
I'm working towards replicating this process for Linux systems, and keep hitting a road block. The clone LUN is always presented to the backup server, however while it is syncing with the source LUN it is not available for IO directly. In this state multipathd marks both paths as failed, which I'm okay with:
[root@rhtsutility02 ~]# multipath -ll 3600601608d211a00f4e21b978136df11 3600601608d211a00f4e21b978136df11
[size=20 GB][features="1 queue_if_no_path"][hwhandler="1 emc"] \_ round-robin 0 [prio=1][enabled]
\_ 15:0:0:12 sdad 65:208 [failed][ready]
\_ round-robin 0 [enabled]
\_ 17:0:0:12 sdac 65:192 [failed][ready]
Unfortunately, LVM seems to automatically pick up on PVs within the cloned LUN as soon as multipathd grabs it, so as soon as I do a 'vgscan' it hangs until I break the clone and present the cloned LUN to my backup box.
So far I've basically excluded these cloned LUNs from lvm with a filter, however I do need to enable the VGs on them and mount LVs for backups. I'm not sure if switching out the filter all the time is the best way to go, but it doesn't look like LVM2's version of vgimport allows you to specify a PV device.
Anyone had to deal with this before, or have an idea of a good way to deal with it? I agree it's a messy situation, but EMC is good at that :-)
Quoted 31 lines · David Coulson
I've got a couple of Clariion disk frames on which we performing cloning of LUNs for backups using HP-UX systems. Basically box A has a LUN or two, we clone it, and present it to box B - It mounts it, backs it up, then umounts it and we restart the cloning process... Nothing magical.
I'm working towards replicating this process for Linux systems, and keep hitting a road block. The clone LUN is always presented to the backup server, however while it is syncing with the source LUN it is not available for IO directly. In this state multipathd marks both paths as failed, which I'm okay with:
[root@rhtsutility02 ~]# multipath -ll
3600601608d211a00f4e21b978136df113600601608d211a00f4e21b978136df11
[size=20 GB][features="1 queue_if_no_path"][hwhandler="1 emc"] \_ round-robin 0 [prio=1][enabled]
\_ 15:0:0:12 sdad 65:208 [failed][ready]
\_ round-robin 0 [enabled]
\_ 17:0:0:12 sdac 65:192 [failed][ready]Unfortunately, LVM seems to automatically pick up on PVs within the cloned LUN as soon as multipathd grabs it, so as soon as I do a 'vgscan' it hangs until I break the clone and present the cloned LUN to my backup box.
So far I've basically excluded these cloned LUNs from lvm with a filter, however I do need to enable the VGs on them and mount LVs for backups. I'm not sure if switching out the filter all the time is the best way to go, but it doesn't look like LVM2's version of vgimport allows you to specify a PV device.
Quoted 3 lines
Anyone had to deal with this before, or have an idea of a good way to deal with it? I agree it's a messy situation, but EMC is good at that :-)
I'm not crystal clear on your use-case (e.g. the mechanics and constraints of when the clone is presented to the backup, etc).
But the 'vgimportclone' command will enable you to make the cloned LUNs distinct from the originals. Does this help you?
Mike
Quoted 3 lines · Mike Snitzer
I'm not crystal clear on your use-case (e.g. the mechanics and constraints of when the clone is presented to the backup, etc).
The process is basically this:
* Clone in 'sync' state so it is replicating block changes from the source. The clone LUN is still visible to the backup server, however both paths are 'failed' to multipathd
* We fracture/break the clone which makes the clone LUN available to the backup server for IO
* We mount LVs from the clone LUN on the backup server, run our backup job, unmount it
* vgexport the clone LUN VG and remove it from our multipath list with 'multipath -f'
* Tell the SAN to start syncing the clone to source again until we need to break it for another backup.
I basically need to find a nice way to make LVM ignore the clone disk on the backup system until I am ready to make use of it for a backup. multipathd sees it as failed during this time, but vgscan/pvscan, etc all hang. Maybe that is normal.
Quoted 3 lines
But the 'vgimportclone' command will enable you to make the cloned LUNs distinct from the originals. Does this help you?
The clone gets mounted on a different box to the source, so it's not important to me. I don't have a vgimportclone command on RHEL4, unless I need to install some additional packages.
David
Quoted 18 lines · David Coulson
Quoted 2 lines · Mike Snitzer
I'm not crystal clear on your use-case (e.g. the mechanics and constraints of when the clone is presented to the backup, etc).
The process is basically this:
* Clone in 'sync' state so it is replicating block changes from the source. The clone LUN is still visible to the backup server, however both paths are 'failed' to multipathd
* We fracture/break the clone which makes the clone LUN available to the backup server for IO
* We mount LVs from the clone LUN on the backup server, run our backup job, unmount it
* vgexport the clone LUN VG and remove it from our multipath list with 'multipath -f'
* Tell the SAN to start syncing the clone to source again until we need to break it for another backup.
Quoted 4 lines
I basically need to find a nice way to make LVM ignore the clone disk on the backup system until I am ready to make use of it for a backup. multipathd sees it as failed during this time, but vgscan/pvscan, etc all hang. Maybe that is normal.
Are you using multipath's 'queue_if_no_path' (assuming RHEL4 has it)? Could be LVM's IO to the mpath devices will queue (block) until the point when you "break the clone" (which makes the paths(s) available).
You could get very specific with the multipath configuration for these clone LUNs to fail IO immediately (in theory). But in practice this is a delicate task that would require special EMC LUN understanding of how to differentiate between a clone LUN that is actively sync'ing and just some other LUN. It also would be challenging to manage these changes at runtime (might require multipathd being restarted with updated config based on WWID, etc).
Quoted 4 lines
I basically need to find a nice way to make LVM ignore the clone disk on the backup system until I am ready to make use of it for a backup. multipathd sees it as failed during this time, but vgscan/pvscan, etc all hang. Maybe that is normal.
Are you aware of LVM_SYSTEM_DIR environment variable? It could be that you have a default lvm.conf that rejects (via filters) the clone LUNs. But when you want to be able to access the clone LUNs for backup you could have a custom lvm.conf that allows (via filters) these clone LUNs. So each shell you'd be doing your backup work in would point to this custom lvm.conf that lives in $LVM_SYSTEM_DIR.
<write custom filters allowing clone LUNs to /etc/backup_lvm/lvm.conf>
export LVM_SYSTEM_DIR=/etc/backup_lvm/
<do backup commands>
unset LVM_SYSTEM_DIR
Quoted 5 lines
Quoted 2 lines
But the 'vgimportclone' command will enable you to make the cloned LUNs distinct from the originals. Does this help you?
The clone gets mounted on a different box to the source, so it's not important to me. I don't have a vgimportclone command on RHEL4, unless I need to install some additional packages.
vgimportclone is not packaged in RHEL4's lvm2. But vgimportclone wouldn't help you anyway; thanks for clarifying.
Mike
Quoted 4 lines · Mike Snitzer
Are you using multipath's '
queue_if_no_path' (assuming RHEL4 has it)? Could be LVM's IO to the mpath devices will queue (block) until the point when you "break the clone" (which makes the paths(s) available).
This seems to be a pretty good solution to the problem - When I set it to fail_if_no_path via dmsetup, it seemed to work nicely. Any idea how to set this within a multipaths { multipath { wwid xxxx }} config?