Skip to contentSkip to conversation

How to implement live migration of VMs in thinlv after using lvmlockd

linux-lvm Zhiyong Ye 4 participants 9 messages Nov 1, 2022 – Nov 3, 2022

lore
Share & tools
lore

Zhiyong Ye

author
#0 lore raw

Hi all,

I want to implement live migration of VMs in the lvm + lvmlockd + sanlock environment. There are multiple hosts in the cluster using the same iscsi connection, and the VMs are running on this environment using thinlv volumes. But if want to live migrate the vm, it will be difficult since thinlv which from the same thin pool can only be exclusive active on one host.

I found a previous subject that discussed this issue:

https://lore.kernel.org/all/[email protected]/

The VM in the source host will become suspended after completing the drain IO operation, and no new IO will be issued until the VM in the destination host resumes again during the live migration. Dave recommends to uninstall volumes at the source and activate at the destination within this time window.

However, executing the activate/deactivate command for thinlv volumes during a VM live migration will cause the VM Guest received an acpi message and the Guest will suppose that the disk device has been unmounted.

Or maybe my understanding is off. Can I ask for your help?

Regards,

Zhiyong Ye

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#1 lore raw
Quoted 9 lines · Zhiyong Ye

Hi all,

I want to implement live migration of VMs in the lvm + lvmlockd + sanlock environment. There are multiple hosts in the cluster using the same iscsi connection, and the VMs are running on this environment using thinlv volumes. But if want to live migrate the vm, it will be difficult since thinlv which from the same thin pool can only be exclusive active on one host.

Quoted 3 lines

I found a previous subject that discussed this issue:

https://lore.kernel.org/all/[email protected]/

Hi, in that email I tried to point out that the real problem is not the locking, but rather the inability of dm-thin to share a thin pool among multiple hosts. The locking restrictions just reflect that technical limitation.

Dave
_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

author
#2 lore raw

Hi Dave,

Thank you for your reply!

Does this mean that there is no way to live migrate VMs when using lvmlockd?

As you describe, the granularity of thinlv's sharing/unsharing is per read/write IO, except that lvmlockd reinforces this limitation for the lvm activation command.

Is it possible to modify the code of lvmlockd to break this limitation and let libvirt/qemu guarantee the mutual exclusivity of each read/write IO across hosts when live migration?

Thanks!

Zhiyong

Quoted 12 lines · David Teigland
Quoted 11 lines · Zhiyong Ye

Hi all,

I want to implement live migration of VMs in the lvm + lvmlockd + sanlock environment. There are multiple hosts in the cluster using the same iscsi connection, and the VMs are running on this environment using thinlv volumes. But if want to live migrate the vm, it will be difficult since thinlv which from the same thin pool can only be exclusive active on one host.

I found a previous subject that discussed this issue:

Quoted 9 lines
Quoted 1 line

https://lore.kernel.org/all/[email protected]/

Hi, in that email I tried to point out that the real problem is not the locking, but rather the inability of dm-thin to share a thin pool among multiple hosts. The locking restrictions just reflect that technical limitation.

Dave

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#3 lore raw
Quoted 5 lines · Zhiyong Ye

Hi Dave,

Thank you for your reply!

Does this mean that there is no way to live migrate VMs when using lvmlockd?

You could by using linear LVs, ovirt does this using sanlock directly, since lvmlockd arrived later.

Quoted 7 lines

As you describe, the granularity of thinlv's sharing/unsharing is per read/write IO, except that lvmlockd reinforces this limitation for the lvm activation command.

Is it possible to modify the code of lvmlockd to break this limitation and let libvirt/qemu guarantee the mutual exclusivity of each read/write IO across hosts when live migration?

lvmlockd locking does not apply to the dm i/o layers. The kind of multi-host locking that you seem to be talking about would need to be implemented inside dm-thin to protect on-disk data structures that it modifies. In reality you would need to write a new dm target with locking and data structures designed for that kind of sharing.

Dave
_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#4 lore raw
Quoted 9 lines · David Teigland
Quoted 5 lines · Zhiyong Ye

Hi Dave,

Thank you for your reply!

Does this mean that there is no way to live migrate VMs when using lvmlockd?

You could by using linear LVs, ovirt does this using sanlock directly, since lvmlockd arrived later.

Another approach would be to use thin provisioning on the SAN instead of at the LVM level.

Signature · 3 lines
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

author
#5 lore raw

Hi Demi,

Thank you for your reply!

Using thin provisioning on the storage server (SAN) side would make the problem much easier, but my scenario is to support different types of SAN, which means that the SAN may not support this feature.

Quoted 6 lines · Demi Marie Obenour
Quoted 5 lines · David Teigland
Quoted 4 lines · Zhiyong Ye

Hi Dave,

Thank you for your reply!

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Does this mean that there is no way to live migrate VMs when using lvmlockd?

You could by using linear LVs, ovirt does this using sanlock directly, since lvmlockd arrived later.

Another approach would be to use thin provisioning on the SAN instead of at the LVM level.

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

author
#6 lore raw
In reply to David Teigland
Quoted 10 lines · David Teigland
Quoted 5 lines · Zhiyong Ye

Hi Dave,

Thank you for your reply!

Does this mean that there is no way to live migrate VMs when using lvmlockd?

You could by using linear LVs, ovirt does this using sanlock directly, since lvmlockd arrived later.

Yes, standard LV is theoretically capable of live migration because it supports multiple hosts using the same LV concurrently with a shared lock (lvchange -asy). But I want to support the live migration feature for both LVs (thin LV and standard LV).

Quoted 8 lines
Quoted 7 lines

As you describe, the granularity of thinlv's sharing/unsharing is per read/write IO, except that lvmlockd reinforces this limitation for the lvm activation command.

Is it possible to modify the code of lvmlockd to break this limitation and let libvirt/qemu guarantee the mutual exclusivity of each read/write IO across hosts when live migration?

Quoted 5 lines

lvmlockd locking does not apply to the dm i/o layers. The kind of multi-host locking that you seem to be talking about would need to be implemented inside dm-thin to protect on-disk data structures that it modifies. In reality you would need to write a new dm target with locking and data structures designed for that kind of sharing.

I can try to write a new dm thin target or make some modifications based on the existing dm-thin target to support this feature, if it is technically feasible. But I'm curious why the current dm-thin doesn't support multi-host shared access, just like dm-linear does.

Regards!

Zhiyong

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#7 lore raw
In reply to David Teigland
Quoted 6 lines · Zhiyong Ye
Quoted 6 lines

I want to implement live migration of VMs in the lvm + lvmlockd + sanlock environment. There are multiple hosts in the cluster using the same iscsi connection, and the VMs are running on this environment using thinlv volumes. But if want to live migrate the vm, it will be difficult since thinlv which from the same thin pool can only be exclusive active on one host.

I just expose the LV (thin or not - I prefer not) as an iSCSI target
that the VM boots from. There is only one host that manages a thin pool, and that is a single point of failure, but no locking issues. You issue the LVM commands on the iSCSI server (which I guess they call NAS
these days).

If you need a way for a VM to request enlarging an LV it accesses, or similar interaction, I would make a simple API where each VM gets a token that determines what LVs it has access to and how much total storage it can consume. Maybe someone has already done that. I just issue the commands on the LVM/NAS/iSCSI host.

I haven't done this, but there can be more than one thin pool, each on
it's own NAS/iSCSI server. So if one storage server crashes, then
only the VMs attached to it crash. You can only (simply) migrate a VM to another VM host on the same storage server.

BUT, you can migrate a VM to another host less instantly using DRBD or other remote mirroring driver. I have done this. You get the remote LV mirror mostly synced, suspend the VM (to a file if you need to rsync that to the remote), finish the sync of the LV(s), resume the VM on the new server - in another city. Handy when you have a few hours notice of a natural disaster (hurricane/flood).

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

author
#8 lore raw

Hi Gathman,

Thank you so much for sharing your usage scenario, and I learn a lot from your experience.

Regards!

Zhiyong

Quoted 23 lines · Stuart D Gathman
Quoted 6 lines · Zhiyong Ye
Quoted 6 lines

I want to implement live migration of VMs in the lvm + lvmlockd + sanlock environment. There are multiple hosts in the cluster using the same iscsi connection, and the VMs are running on this environment using thinlv volumes. But if want to live migrate the vm, it will be difficult since thinlv which from the same thin pool can only be exclusive active on one host.

I just expose the LV (thin or not - I prefer not) as an iSCSI target
that the VM boots from.  There is only one host that manages a thin pool, and that is a single point of failure, but no locking issues.  You issue the LVM commands on the iSCSI server (which I guess they call NAS
these days).

If you need a way for a VM to request enlarging an LV it accesses, or similar interaction, I would make a simple API where each VM gets a token that determines what LVs it has access to and how much total storage it can consume.  Maybe someone has already done that. I just issue the commands on the LVM/NAS/iSCSI host.

I haven't done this, but there can be more than one thin pool, each on
it's own NAS/iSCSI server.  So if one storage server crashes, then
only the VMs attached to it crash.  You can only (simply) migrate a VM to another VM host on the same storage server.

Quoted 6 lines

BUT, you can migrate a VM to another host less instantly using DRBD or other remote mirroring driver.  I have done this.  You get the remote LV mirror mostly synced, suspend the VM (to a file if you need to rsync that to the remote), finish the sync of the LV(s), resume the VM on the new server - in another city.  Handy when you have a few hours notice of a natural disaster (hurricane/flood).

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/