LVM snapshots in a iSCSI and XenSource environment
Hi list,
In advance my excusses for this radar long post (although it's easy readable ;), but I want to make sure that I understand it correctly so I don't end up making a very costly mistake.
I have a storage server (Debian Etch) with mutlipath-tools running and on top of that I use IET iscsi-target software to export the multipathed device to a XenSource server.
XenSource creates a PV on the entire exported disk, and then creates a few LV's when I create some virtual machines.
Now I would like to take snapshots of these virtual machines as a backup. So I want to take a snapshot every day, but hold them for only one day. These are quite static machines, so I need want several snapshots per machine. Just one in case something happens or a bad adjustment is made.
I asume that when I drop the snapshot just before creating the new one, all changes are merged back to the original volume. Correct? While this merging is happening, does the disk then becomes unavailable to the virtual machine, or doesn't the virtual machine doesn't notice the merge?
Also I was wondering if it's a smart idea to create a PV and a LV (both spanning the whole disk) on the storage server, and then exporting the LV true iSCSI to the XenSource server. In that way I can take the snapshots on to storage server directly.
Questions that I think of then are if it's not a problem that XenSource then creates a new PV and some LV's in je LV I created adn exported on the storage server. Is that a problem, or should this work fine?
And another question is how I can then restore a single LV Xen created, from the snapshot of the LV that spans the whole disk on the storage server? In that case I can not just revert to the old disk before taking the snapshot, because then all the LV's created by Xen will be set back to that point, and not just the LV that went bad.
Last question... Without creating a PV and a LV on the storage server and just letting XenSource create what it needs to provision the virtual machines, I can still see the PV and the LV's Xen created on the storage server. Could I take the snapshots from there, although the PV and LV's where not created here?
Hope someone can tell me what the best option is, and if all this is possible?
Thanks!!
Sander
Quoted 6 lines · S. J. van Harmelen
Hi list,
In advance my excusses for this radar long post (although it's easy readable ;), but I want to make sure that I understand it correctly so I don't end up making a very costly mistake.
Quoted 6 lines
I have a storage server (Debian Etch) with mutlipath-tools running and on top of that I use IET iscsi-target software to export the multipathed device to a XenSource server.
XenSource creates a PV on the entire exported disk, and then creates a few LV's when I create some virtual machines.
Which IMO is a pity, as logically, LVM exists and is usable on that given Xen server only. This means you can't really use multiple Xen servers, live migration etc.
Quoted 5 lines
Now I would like to take snapshots of these virtual machines as a backup. So I want to take a snapshot every day, but hold them for only one day. These are quite static machines, so I need want several snapshots per machine. Just one in case something happens or a bad adjustment is made.
OK, should work.
Quoted 5 lines
I asume that when I drop the snapshot just before creating the new one, all changes are merged back to the original volume. Correct? While this merging is happening, does the disk then becomes unavailable to the virtual machine, or doesn't the virtual machine doesn't notice the merge?
I believe there is no "merging back", as the changes are being made to the snapshot volume. If you drop the snapshot, it just doesn't exist anymore, and the original volume does exist as it did before. So yes, you can safely drop the snapshots, the process is transparent.
Quoted 4 lines
Also I was wondering if it's a smart idea to create a PV and a LV (both spanning the whole disk) on the storage server, and then exporting the LV true iSCSI to the XenSource server. In that way I can take the snapshots on to storage server directly.
Yes, it is indeed better and more flexible solution.
Quoted 3 lines
Questions that I think of then are if it's not a problem that XenSource then creates a new PV and some LV's in je LV I created adn exported on the storage server. Is that a problem, or should this work fine?
Hmm? I don't think I understand what you mean.
Quoted 5 lines
And another question is how I can then restore a single LV Xen created, from the snapshot of the LV that spans the whole disk on the storage server? In that case I can not just revert to the old disk before taking the snapshot, because then all the LV's created by Xen will be set back to that point, and not just the LV that went bad.
# Will only work if snapshot size is equal or greater than
# the original volume
dd if=/dev/LVM/volume-snapshot of=/dev/LVM/volume
# or, if the allowed snapshot size is smaller, we don't want our
# precious snapshot dropped
dd if=/dev/LVM/volume-snapshot of=/dev/LVM/new-volume dd if=/dev/LVM/new-volume of=/dev/LVM/volume
Quoted 5 lines
Last question... Without creating a PV and a LV on the storage server and just letting XenSource create what it needs to provision the virtual machines, I can still see the PV and the LV's Xen created on the storage server. Could I take the snapshots from there, although the PV and LV's where not created here?
Only when you don't use them on the Xen server at the same time (no iSCSI connection; LVM modules on the Xen server unloaded, LVM modules on the storage server loaded). Which brings us to a point: it is better to manage LVMs on a storage server than on a Xen host.
Signature · 2 lines
Tomasz Chmielewski http://lists.wpkg.org
Thanks for your reaction!
Quoted 11 lines · Tomasz Chmielewski
Quoted 10 lines · S. J. van Harmelen
Hi list,
In advance my excusses for this radar long post (although it's easy readable ;), but I want to make sure that I understand it correctly so I don't end up making a very costly mistake.
I have a storage server (Debian Etch) with mutlipath-tools running and on top of that I use IET iscsi-target software to export the multipathed device to a XenSource server.
Quoted 6 lines
Quoted 2 lines
XenSource creates a PV on the entire exported disk, and then creates a few LV's when I create some virtual machines.
Which IMO is a pity, as logically, LVM exists and is usable on that given Xen server only. This means you can't really use multiple Xen servers, live migration etc.
Are you sure about that? Accoring to Xen lvm over iSCSI is considered as shared storage that can be used for live migration
(http://docs.xensource.com/XenServer/4.0.1/installation/ch03s03.html):
<snip>
3.3.3. XenServer Hosts with shared iSCSI storage
Adding shared storage to the XenServer network enables grouping of XenServer Hosts into Resource Pools, enabling live relocation of VMs and sharing of server resources.
Requirements
* two or more x86 servers with local storage
* one or more Windows workstations, on same network as the XenServer Hosts
* a server providing a shared directory via iSCSI
<snip>
Quoted 27 lines
Quoted 6 lines
Now I would like to take snapshots of these virtual machines as a backup. So I want to take a snapshot every day, but hold them for only one day. These are quite static machines, so I need want several snapshots per machine. Just one in case something happens or a bad adjustment is made.
OK, should work.
Quoted 5 lines
I asume that when I drop the snapshot just before creating the new one, all changes are merged back to the original volume. Correct? While this merging is happening, does the disk then becomes unavailable to the virtual machine, or doesn't the virtual machine doesn't notice the merge?
I believe there is no "merging back", as the changes are being made to the snapshot volume. If you drop the snapshot, it just doesn't exist anymore, and the original volume does exist as it did before. So yes, you can safely drop the snapshots, the process is transparent.
Quoted 4 lines
Also I was wondering if it's a smart idea to create a PV and a LV (both spanning the whole disk) on the storage server, and then exporting the LV true iSCSI to the XenSource server. In that way I can take the snapshots on to storage server directly.
Quoted 8 lines
Yes, it is indeed better and more flexible solution.
Quoted 3 lines
Questions that I think of then are if it's not a problem that XenSource then creates a new PV and some LV's in je LV I created adn exported on the storage server. Is that a problem, or should this work fine?
Hmm? I don't think I understand what you mean.
I mean I will have 2 sets of PV's and LV's. One set on the storage server (that has one PV and one LV that both span the whole disk), and one set op the Xen server (the ones that Xen makes by itself when I add a new vitrual machine).
Now the one LV from the storage server that Xen sees is being exported true iSCSI, so it wouldn't know that it is infact a LV that it's talking to. As far as the Xen server conserns this is just a raw disk. So it will then just create the needed PV and LV's on it to provision virtual mahines.
The question was if this PV/LV in another PV/LV (on another physical machine) can do any harm?
Quoted 11 lines
Quoted 7 lines
And another question is how I can then restore a single LV Xen created, from the snapshot of the LV that spans the whole disk on the storage server? In that case I can not just revert to the old disk before taking the snapshot, because then all the LV's created by Xen will be set back to that point, and not just the LV that went bad.
# Will only work if snapshot size is equal or greater than
# the original volume
Quoted 7 lines
dd if=/dev/LVM/volume-snapshot of=/dev/LVM/volume
# or, if the allowed snapshot size is smaller, we don't want our
# precious snapshot droppeddd if=/dev/LVM/volume-snapshot of=/dev/LVM/new-volume dd if=/dev/LVM/new-volume of=/dev/LVM/volume
Oke, but how about this when using LVM as I just described a few lines above here with the PV/LV in another PV/LV setup. How can I then restore a snapshot of a virtual machines' LV from the snapshot of the LV on the storage server?
Quoted 15 lines
Quoted 7 lines
Last question... Without creating a PV and a LV on the storage server and just letting XenSource create what it needs to provision the virtual machines, I can still see the PV and the LV's Xen created on the storage server. Could I take the snapshots from there, although the PV and LV's where not created here?
Only when you don't use them on the Xen server at the same time (no iSCSI connection; LVM modules on the Xen server unloaded, LVM modules on the storage server loaded).
Which brings us to a point: it is better to manage LVMs on a storage server than on a Xen host.
Quoted 16 lines · S. J. van Harmelen
Thanks for your reaction!
Quoted 13 lines · Tomasz Chmielewski
Quoted 12 lines · S. J. van Harmelen
Hi list,
In advance my excusses for this radar long post (although it's easy readable ;), but I want to make sure that I understand it correctly so I don't end up making a very costly mistake.
I have a storage server (Debian Etch) with mutlipath-tools running and on top of that I use IET iscsi-target software to export the multipathed device to a XenSource server.
XenSource creates a PV on the entire exported disk, and then creates a few LV's when I create some virtual machines.
Quoted 8 lines
Quoted 1 line
Which IMO is a pity, as logically, LVM exists and is usable on that given Xen server only. This means you can't really use multiple Xen servers, live migration etc.
Are you sure about that? Accoring to Xen lvm over iSCSI is considered as shared storage that can be used for live migration
(http://docs.xensource.com/XenServer/4.0.1/installation/ch03s03.html):<snip>
3.3.3. XenServer Hosts with shared iSCSI storage
This implies turning a Xen server into a NAS server, as I understand?
Yes, this should work - I have a custom Xen setup, and I don't connect a NAS and a Xen server on one host. But according to the description, yes, it should work.
(...)
Quoted 10 lines
Quoted 4 lines
Quoted 3 lines
Questions that I think of then are if it's not a problem that XenSource then creates a new PV and some LV's in je LV I created adn exported on the storage server. Is that a problem, or should this work fine?
Hmm? I don't think I understand what you mean.
I mean I will have 2 sets of PV's and LV's. One set on the storage server (that has one PV and one LV that both span the whole disk), and one set op the Xen server (the ones that Xen makes by itself when I add a new vitrual machine).
Quoted 8 lines
Now the one LV from the storage server that Xen sees is being exported true iSCSI, so it wouldn't know that it is infact a LV that it's talking to. As far as the Xen server conserns this is just a raw disk. So it will then just create the needed PV and LV's on it to provision virtual mahines.
The question was if this PV/LV in another PV/LV (on another physical machine) can do any harm?
Should work, as long as you don't use it on two different machines (i.e., target/initiator) as LVM at the same time.
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
And another question is how I can then restore a single LV Xen created, from the snapshot of the LV that spans the whole disk on the storage server? In that case I can not just revert to the old disk before taking the snapshot, because then all the LV's created by Xen will be set back to that point, and not just the LV that went bad.
# Will only work if snapshot size is equal or greater than
# the original volumedd if=/dev/LVM/volume-snapshot of=/dev/LVM/volume
# or, if the allowed snapshot size is smaller, we don't want our
# precious snapshot dropped
Quoted 7 lines
Quoted 2 lines
dd if=/dev/LVM/volume-snapshot of=/dev/LVM/new-volume dd if=/dev/LVM/new-volume of=/dev/LVM/volume
Oke, but how about this when using LVM as I just described a few lines above here with the PV/LV in another PV/LV setup. How can I then restore a snapshot of a virtual machines' LV from the snapshot of the LV on the storage server?
As a prerequisite, you need to "see" that LVM.
But could you be more specific on what you want (write a bit more details in points, write a small diagram etc.)?
Signature · 2 lines
Tomasz Chmielewski http://wpkg.org
Quoted 75 lines · Tomasz Chmielewski
Quoted 20 lines · S. J. van Harmelen
Thanks for your reaction!
Quoted 11 lines · Tomasz Chmielewski
Quoted 10 lines · S. J. van Harmelen
Hi list,
In advance my excusses for this radar long post (although it's easy readable ;), but I want to make sure that I understand it correctly so I don't end up making a very costly mistake.
I have a storage server (Debian Etch) with mutlipath-tools running and on top of that I use IET iscsi-target software to export the multipathed device to a XenSource server.
Quoted 5 lines
Quoted 2 lines
XenSource creates a PV on the entire exported disk, and then creates a few LV's when I create some virtual machines.
Which IMO is a pity, as logically, LVM exists and is usable on that given Xen server only. This means you can't really use multiple Xen servers, live migration etc.
Quoted 6 lines
Are you sure about that? Accoring to Xen lvm over iSCSI is considered as shared storage that can be used for live migration
(http://docs.xensource.com/XenServer/4.0.1/installation/ch03s03.html):<snip>
3.3.3. XenServer Hosts with shared iSCSI storageThis implies turning a Xen server into a NAS server, as I understand? Yes, this should work - I have a custom Xen setup, and I don't connect a NAS and a Xen server on one host.
But according to the description, yes, it should work.(...)
Quoted 10 lines
Quoted 4 lines
Quoted 3 lines
Questions that I think of then are if it's not a problem that XenSource then creates a new PV and some LV's in je LV I created adn exported on the storage server. Is that a problem, or should this work fine?
Hmm? I don't think I understand what you mean.
I mean I will have 2 sets of PV's and LV's. One set on the storage server (that has one PV and one LV that both span the whole disk), and one set op the Xen server (the ones that Xen makes by itself when I add a new vitrual machine).
Quoted 8 lines
Now the one LV from the storage server that Xen sees is being exported true iSCSI, so it wouldn't know that it is infact a LV that it's talking to. As far as the Xen server conserns this is just a raw disk. So it will then just create the needed PV and LV's on it to provision virtual mahines.
The question was if this PV/LV in another PV/LV (on another physical machine) can do any harm?
Should work, as long as you don't use it on two different machines (i.e., target/initiator) as LVM at the same time.
Quoted 16 lines
Quoted 15 lines
Quoted 5 lines
And another question is how I can then restore a single LV Xen created, from the snapshot of the LV that spans the whole disk on the storage server? In that case I can not just revert to the old disk before taking the snapshot, because then all the LV's created by Xen will be set back to that point, and not just the LV that went bad.
# Will only work if snapshot size is equal or greater than
# the original volumedd if=/dev/LVM/volume-snapshot of=/dev/LVM/volume
# or, if the allowed snapshot size is smaller, we don't want our
# precious snapshot droppeddd if=/dev/LVM/volume-snapshot of=/dev/LVM/new-volume dd if=/dev/LVM/new-volume of=/dev/LVM/volume
Quoted 8 lines
Quoted 4 lines
Oke, but how about this when using LVM as I just described a few lines above here with the PV/LV in another PV/LV setup. How can I then restore a snapshot of a virtual machines' LV from the snapshot of the LV on the storage server?
As a prerequisite, you need to "see" that LVM.
But could you be more specific on what you want (write a bit more details in points, write a small diagram etc.)?
Oke... Let's say I have a storage server with one LUN connected to it. The LUN becomes /dev/sdi on the storage server. I will then create a PV on /dev/sdi spanning the whole disk. In the PV I will create one LV also spanning the whole disk. Say we name this LV lv_storage. So the path to this LV gets to be /dev/lvm/lv_storage.
In iSCSI I will then create a target which points
to /dev/lvm/lv_storage. The Xen server will use open-iscsi to connect to that target (/dev/lvm/lvstorage) and on the Xen server it becomes /dev/sdb. At this point Xen has also created a PV that spans the whole disk (/dev/sdb)
Then when I create a virtual machine on the Xen server, it creates a LV on /dev/sdb with the size of the virtual disk. Say this LV gets named lv_virtualmachine1 and the next gets to be named lv_virtualmachine2 and so on.
So far everything clear?
Now my thoughts where to take snapshots on the storage server. So that would mean I would take a snapshot of lv_storage. This should not be a problem. But what if virtual machine 2 crashes and has to be restored?
In that case I cannot restore lv_storage with a dd command, because then all the data will be restored (lv_virtualmachine1 and lv_virtualmachine2, because they are both on lv_storage). So is it possible to "restore" only lv_virtualmachine2 when I took a snapshot of lv_storage?
I can see the PV and LV's Xen created on lv_storage on the storage server when I run pvs by the way...
Hope you understand what I mean now?
Quoted 2 lines
Quoted 10 lines · S. J. van Harmelen [[email protected]] wrote:
Now my thoughts where to take snapshots on the storage server. So that would mean I would take a snapshot of
lv_storage. This should not be a problem. But what if virtual machine 2 crashes and has to be restored?In that case I cannot restore
lv_storagewith a dd command, because then all the data will be restored (lv_virtualmachine1andlv_virtualmachine2, because they are both onlv_storage). So is it possible to "restore" onlylv_virtualmachine2when I took a snapshot oflv_storage?
I see, it is possible but a bit cumbersome. You should be able to run LVM on the storage server on the restored/old data (LVM on top of LVM??) and get the data for the lv_virtualmachine2 from the storage server to XenSource (ftp/tar whatever method). Then, restore that data (tar extract) at XenSource.
If things are NOT changing at the XenSource LVM, you could just "dd" on the crashed LV, but things may go really bad otherwise.
Any better method(s)???
Quoted 2 lines
I can see the PV and LV's Xen created on
lv_storageon the storage server when I run pvs by the way...
Yes, you will see them but you want the storage server's LVM to NOT use them under normal conditions.
BTW, for backup, you may want snapshot-origin target rather than snapshot target directly..
Quoted 18 lines · On Tue, 2007-11-20 at 09:29 -0800, [email protected] wrote:
Quoted 10 lines · S. J. van Harmelen [[email protected]] wrote:
Now my thoughts where to take snapshots on the storage server. So that would mean I would take a snapshot of
lv_storage. This should not be a problem. But what if virtual machine 2 crashes and has to be restored?In that case I cannot restore
lv_storagewith a dd command, because then all the data will be restored (lv_virtualmachine1andlv_virtualmachine2, because they are both onlv_storage). So is it possible to "restore" onlylv_virtualmachine2when I took a snapshot oflv_storage?I see, it is possible but a bit cumbersome. You should be able to run LVM on the storage server on the restored/old data (LVM on top of LVM??) and get the data for the
lv_virtualmachine2from the storage server to XenSource (ftp/tar whatever method). Then, restore that data (tar extract) at XenSource.
Quoted 4 lines
If things are NOT changing at the XenSource LVM, you could just "dd" on the crashed LV, but things may go really bad otherwise.
Any better method(s)???
Well of course this could also be done from the Xen server. Then there is no need for LVM on the storage server and the lv_storage part can be left out of this picture.
I could then create a second iSCSI target on a second LUN to export to the Xen server so that Xen can take snapshots to that second exported disk. If a problem does occur, then a restore is simple because then I could do a dd from the snapshot lv (on the Xen server) to the other lv (also on the Xen server).
But how about network traffic on this one? Everything I do will go across iSCSI so it will go over the network.
And then I will manually have to greate lv-snapshots for each virtual machine instead of just one big lv-snapshot as I could do when lv_storage exists.
Any thoughts?
Quoted 9 lines
Quoted 3 lines
I can see the PV and LV's Xen created on
lv_storageon the storage server when I run pvs by the way...Yes, you will see them but you want the storage server's LVM to NOT use them under normal conditions.
BTW, for backup, you may want snapshot-origin target rather than snapshot target directly..
Quoted 18 lines · On Tue, 2007-11-20 at 09:29 -0800, [email protected] wrote:
Quoted 10 lines · S. J. van Harmelen [[email protected]] wrote:
Now my thoughts where to take snapshots on the storage server. So that would mean I would take a snapshot of
lv_storage. This should not be a problem. But what if virtual machine 2 crashes and has to be restored?In that case I cannot restore
lv_storagewith a dd command, because then all the data will be restored (lv_virtualmachine1andlv_virtualmachine2, because they are both onlv_storage). So is it possible to "restore" onlylv_virtualmachine2when I took a snapshot oflv_storage?I see, it is possible but a bit cumbersome. You should be able to run LVM on the storage server on the restored/old data (LVM on top of LVM??) and get the data for the
lv_virtualmachine2from the storage server to XenSource (ftp/tar whatever method). Then, restore that data (tar extract) at XenSource.
Quoted 4 lines
If things are NOT changing at the XenSource LVM, you could just "dd" on the crashed LV, but things may go really bad otherwise.
Any better method(s)???
Would it also work when I create lv_storage, take a snapshot of it and then when I need to restore a virtual machine I tempararly export the snapshot true iSCSI to the Xen server.
Then on the Xen server I should see the PV and LV's Xen created both on the real iSCSI exported disk, as the exported snapshot from the storage server. So then I should be able to do a dd from one lv to the other, right?
Quoted 9 lines
Quoted 3 lines
I can see the PV and LV's Xen created on
lv_storageon the storage server when I run pvs by the way...Yes, you will see them but you want the storage server's LVM to NOT use them under normal conditions.
BTW, for backup, you may want snapshot-origin target rather than snapshot target directly..