LVM snapshot with Clustered VG [SOLVED]
Quoted 2 lines
I need to convert lock on a remote node during last stage of ver3 migration in libvirt/qemu
Hi, I wrote and maintain the dlm and have more recently written a new disk-based lock manager called sanlock, https://fedorahosted.org/sanlock/
which operates with only shared storage among nodes (no networking or other cluster manager.)
sanlock was written to allow RHEV to manage leases for vm images on shared storage, including the ability to migrate leases among hosts (which is the most complicated part, as you've mentioned above.) sanlock plugs into the libvirt locking api, which also supports file locks (usable on local or nfs file systems.) (Search for "virtlockd".)
Trying to use/control vm locks via lvm commands is not a good way to solve the vm management/migration problem at the present time (but see below). Instead, I'd suggest doing the locking via the libvirt locking api which was designed for this purpose. As I mentioned, libvirt supports both sanlock and file locks, but another option is to write a new libvirt locking plug-in for dlm/corosync. This would be the best way to use dlm locks to protect vm images on shared storage; I've been hoping someone would do this for some time.
Incidentally, I've recently started a new project which is to replace clvmd with a new "lvmlockd". I'm designing lvmlockd to support both dlm and sanlock on the back side (transparent to lvm itself). With sanlock, you will not need a dlm/corosync cluster to have the benefits of locking vgs and lvs on shared storage. This project is requiring a lot of preliminary work in the lvm code, because the clvmd approach reaches deeply into lvm itself. Relating back to virt environments, lvmlockd will give you direct control of the lock types, modes and objects in each command. This will hopefully make it much easier to use lvm locks in a controlled and programatic way to solve problems like vm management.
So, in preparation for lvmlockd, you should not assume that lvm commands will operate in a dlm/corosync environment. Any new options or capabilities should be considered more generally. Also, the concept of lvm commands executing remote operations in a cluster was a poor design choice for clvm. lvmlockd will probably not support this notion. Executing remote commands should be done at a higher layer.
Dave
Quoted 8 lines · David Teigland
Quoted 2 lines
I need to convert lock on a remote node during last stage of ver3 migration in libvirt/qemu
Hi, I wrote and maintain the dlm and have more recently written a new disk-based lock manager called sanlock, https://fedorahosted.org/sanlock/
which operates with only shared storage among nodes (no networking or other cluster manager.)
Quoted 5 lines
sanlock was written to allow RHEV to manage leases for vm images on shared storage, including the ability to migrate leases among hosts (which is the most complicated part, as you've mentioned above.) sanlock plugs into the libvirt locking api, which also supports file locks (usable on local or nfs file systems.) (Search for "virtlockd".)
Yes, I know, please see below.
Quoted 9 lines
Trying to use/control vm locks via lvm commands is not a good way to solve the vm management/migration problem at the present time (but see below). Instead, I'd suggest doing the locking via the libvirt locking api which was designed for this purpose. As I mentioned, libvirt supports both sanlock and file locks, but another option is to write a new libvirt locking plug-in for dlm/corosync. This would be the best way to use dlm locks to protect vm images on shared storage; I've been hoping someone would do this for some time.
I almost agree. That's why I developed one more locking mech for libvirt which works and solves what I need.
I also thought about dlm one (may be the next step :) ), but I need lvm to activate volumes, because I do not like current libvirt's "-aln everywhere" approach with VLM, so I decided to use clvm-based locking driver (with my additions to LVM). So, I made new "clvm2" pool type (subtype of a "logical" together with "lvm2") which does not activate volumes at all by itself, and "clvm" locking driver. Of course, that was not so easy, because all current locking drivers assume that disk device used by VM is always accessible (file on NFS f.e.), and that is not true for clvm with exclusive activation. But, I really made it all work, so I expect this to be a very strong alternative to all other locking mechs in libvirt (but only for LVM volumes). Local (prevent LV from being opened on the same node) part is not yet fully baked, only a PoC, but that can be easily replaced with some flock-based implementation (not released yet, but I saw some words about it is being written by Daniel) or even allow chaining of another driver (sanlock on a local storage?) for local (per-node) locking.
Please keep an eye on libvirt list, I want to send my work there if my ideas are accepted here on lvm list, there is strong dependency on them.
It is very hard to inject ideas simultaneously to related projects, but I'll try.
Quoted 11 lines
Incidentally, I've recently started a new project which is to replace clvmd with a new "lvmlockd". I'm designing lvmlockd to support both dlm and sanlock on the back side (transparent to lvm itself). With sanlock, you will not need a dlm/corosync cluster to have the benefits of locking vgs and lvs on shared storage. This project is requiring a lot of preliminary work in the lvm code, because the clvmd approach reaches deeply into lvm itself. Relating back to virt environments, lvmlockd will give you direct control of the lock types, modes and objects in each command. This will hopefully make it much easier to use lvm locks in a controlled and programatic way to solve problems like vm management.
Anyways, I decided to go with pacemaker, so what I have now fully fits my needs (and I have it now, not with EL7).
Of course, I fully agree that clvm is just a big hack from the today's point of view and something need to change here.
Unfortunately I do not have a power and fundings to fully redesign clustered LVM (I do what I do for projects which did not give me a one dollar yet), so I decided to just solve my task, being not intrusive.
And I like a state I have it all now in.
Your idea is clean, and it reminds me idea of moving quorum to corosync, so please keep going. ;)
Quoted 7 lines
So, in preparation for lvmlockd, you should not assume that lvm commands will operate in a dlm/corosync environment. Any new options or capabilities should be considered more generally. Also, the concept of lvm commands executing remote operations in a cluster was a poor design choice for clvm. lvmlockd will probably not support this notion. Executing remote commands should be done at a higher layer.
It all takes soooo long, but I need it all to be done yesterday ;)
Anyways, please feel free to add me to CC of lvmlockd discussions.
Vladislav
Quoted 1 line · Vladislav Bogdanov
I also thought about dlm one (may be the next step :) ),
Let me know if you decide to start work on this and I can probably lend some help.
Quoted 2 lines
Of course, I fully agree that clvm is just a big hack from the today's point of view and something need to change here.
Since you're open to hacking components together, here's another possible
hack: you could create a gfs2 file system on the shared storage, and use file locks on gfs2 files. The gfs2 files could be the actual vm images, but they do not have to be. You could still use lv's for vms directly, and create empty files on gfs2 representing each lv. The virtlockd file locks would be acquired on the empty gfs2 files, representing the lvs. This is another indirect way of using dlm locks, since the gfs2 file locks are passed to the dlm.
Dave
Quoted 9 lines · David Teigland
Quoted 1 line · Vladislav Bogdanov
I also thought about dlm one (may be the next step :) ),
Let me know if you decide to start work on this and I can probably lend some help.
Quoted 2 lines
Of course, I fully agree that clvm is just a big hack from the today's point of view and something need to change here.
Quoted 8 lines
Since you're open to hacking components together, here's another possible
hack: you could create a gfs2 file system on the shared storage, and use file locks on gfs2 files. The gfs2 files could be the actual vm images, but they do not have to be. You could still use lv's for vms directly, and create empty files on gfs2 representing each lv. The virtlockd file locks would be acquired on the empty gfs2 files, representing the lvs. This is another indirect way of using dlm locks, since the gfs2 file locks are passed to the dlm.
That would not solve issue with exclusive LV activation which is required to take snapshots (and just for more safety), but prevents migration.
Vladislav