Snapshots and disk re-use

Hi Everyone,

We host many Xen VMs for customers. We use LVM as the fisk storage for these VMs. When a customer cancels, we generally dd their LV with /dev/zero, before removing the LV.

If however we want to create a snapshot of a customer's LV, is there the chance that the data may appear on a new future LV if we were to create one (for a new customer)? Is is my understanding that most filesystems don't actually remove data from a disk when deleting a file, but just set an "ignore" tag of some sort...

Quoted 11 lines · Jonathan Tripathy

Hi Everyone,

We host many Xen VMs for customers. We use LVM as the fisk storage for these VMs. When a customer cancels, we generally dd their LV with /dev/zero, before removing the LV.

If however we want to create a snapshot of a customer's LV, is there the chance that the data may appear on a new future LV if we were to create one (for a new customer)? Is is my understanding that most filesystems don't actually remove data from a disk when deleting a file, but just set an "ignore" tag of some sort...

I'm a bit confused about your scenario. But I think this will help:

The snapshot device is composed of 2 sub devices, the origin and the exception store. Someone who only has visibility of the snapshot (ie. your vm if your scenario is what I think it is) will have visibility of all of the origin. So you _do_ need to worry about data leaks on the origin device. Their view of the exception store is always a subset which has been overwritten by origin data first. So data leaks are not an issue for the exception store.

- Joe

The snapshot will be done in the Xen host, so the Xen host will see the snapshot. The VM will only see the "normal" LV (and continue to use it as normal). My concern is what happens when I remove the snapshot and later on I create a new additional LV for a new customer.

Quoted 4 lines · Jonathan Tripathy

The snapshot will be done in the Xen host, so the Xen host will see the snapshot. The VM will only see the "normal" LV (and continue to use it as normal). My concern is what happens when I remove the snapshot and later on I create a new additional LV for a new customer.

When you remove the snapshot the exception store will have bits of the origin device on it. Zero it before you use the space to provision a volume for a new customer.

- Joe

And how do I zero just exception store?

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Quoted 1 line · Joe Thornber

And how do I zero just exception store?

In this example, the normal lv is c1_s1. It has two snapshots, c1_s1_snap0 and c1_s1_snap1.

List the underlying volumes like so:

[root@clonebox2 ~]# ls -1 /dev/mapper/clones-c1_s1* /dev/mapper/clones-c1_s1 /dev/mapper/clones-c1_s1-real /dev/mapper/clones-c1_s1_snap0 /dev/mapper/clones-c1_s1_snap0-cow /dev/mapper/clones-c1_s1_snap1 /dev/mapper/clones-c1_s1_snap1-cow

Half of those would be hidden to lvdisplay, though the man page will probably tell you how to show hidden LVs. "ls" shows them all.

c1_s1_snap0-cow and c1_s1_snap1-cow are the copy on write volumes, the exception stores, I believe. -real is where the current data lives, and c1_s1, the "main" LV is a basically a proxy which reads directly from -rela, but diverts writes so that old data is copied to -cow. -snap0 is a proxy which reads from either -cow or -real as needed. Hopefully someone will tell me where I'm wrong, if I am.

Zeroing c1_s1 first, then c1_s1_snap0-cow would zero everything,

Quoted 1 line

Zeroing c1_s1 first, then c1_s1_snap0-cow would zero everything,

Thanks for the explanation. But what if I don't want to zero everything? I only want to return everything back to the state it was before I created the snapshot.

Quoted 4 lines · Jonathan Tripathy

Quoted 1 line

Zeroing c1_s1 first, then c1_s1_snap0-cow would zero everything,

Thanks for the explanation. But what if I don't want to zero everything? I only want to return everything back to the state it was before I created the snapshot.

So you're not worried about the security implication of leftovers in free space, and just want a base image to clone for new customers?

Quoted 7 lines · Stuart D. Gathman

Quoted 5 lines · Jonathan Tripathy

Quoted 1 line

Zeroing c1_s1 first, then c1_s1_snap0-cow would zero everything,

Thanks for the explanation. But what if I don't want to zero everything? I only want to return everything back to the state it was before I created the
snapshot.

Quoted 2 lines

So you're not worried about the security implication of leftovers in free space, and just want a base image to clone for new customers?

Yes I am. This is exactly what I'm worried about. I just need to know how to zero out any "left over" bits of data that would be left over after removing a snapshot. I use snapshots for backing up a volume (by running rsync against the snapshot to a remote server)

So you're not worried about the security implication of leftovers in free space, and just want a base image to clone for new customers?

The logical thing to do is to keep the origin volume untouched (except for upgrading now and then), and take a snapshot for each customer. Each snapshot would then be a new clone of the origin. Unfortunately, large numbers of snapshots are inefficient for writes to new data, so you'd likely have to "dd" to an independent LV instead. (This is being worked on, and there are 3rd party products like Zumastor that fix it now.)

Quoted 9 lines · Stuart D. Gathman

So you're not worried about the security implication of leftovers in free space, and just want a base image to clone for new customers?

The logical thing to do is to keep the origin volume untouched (except for upgrading now and then), and take a snapshot for each customer. Each snapshot would then be a new clone of the origin. Unfortunately, large numbers of snapshots are inefficient for writes to new data, so you'd likely have to "dd" to an independent LV instead. (This is being worked on, and there are 3rd party products like Zumastor that fix it now.)

Actually, if you never (or rarely) write to the origin, lots of snapshots should be fine.

But every write to the origin will first copy the
original origin data to every snapshot.

Quoted 13 lines

Quoted 13 lines

So you're not worried about the security implication of leftovers in free
space, and just want a base image to clone for new customers?

The logical thing to do is to keep the origin volume untouched (except for upgrading now and then), and take a snapshot for each customer. Each snapshot would then be a new clone of the origin. Unfortunately, large numbers of snapshots are inefficient for writes to new data, so you'd likely have to "dd" to an independent LV instead. (This is being
worked on, and there are 3rd party products like Zumastor that fix it now.)

Quoted 6 lines

Actually, if you never (or rarely) write to the origin, lots of snapshots should be fine.

But every write to the origin will first copy the
original origin data to every snapshot.

Why would origin data be copied over to the snapshot after the snapshot has been created? Surely the point of a snapshot is to have "frozen" data?

Quoted 6 lines

Quoted 6 lines

Actually, if you never (or rarely) write to the origin, lots of snapshots should be fine.

But every write to the origin will first copy the
original origin data to every snapshot.

Quoted 2 lines

Why would origin data be copied over to the snapshot after the snapshot has been created? Surely the point of a snapshot is to have "frozen" data?

Yes, is the way this actually works explained somewhere? I would have expected the 'copy-on-write' blocks to be copied only on the side where the write is happening and relocated instead of rewriting all the snapshots that might be outstanding with the old data.

Quoted 9 lines · Les Mikesell

Quoted 8 lines

Quoted 6 lines

Actually, if you never (or rarely) write to the origin, lots of snapshots should be fine.

But every write to the origin will first copy the
original origin data to every snapshot.

Why would origin data be copied over to the snapshot after the snapshot has been created? Surely the point of a snapshot is to have "frozen" data?

Quoted 4 lines

Yes, is the way this actually works explained somewhere? I would have expected the 'copy-on-write' blocks to be copied only on the side where the write is happening and relocated instead of rewriting all the snapshots that might be outstanding with the old data.

It should be obvious when you think about it.

You go to write to the origin,
to a block you've never written to since creating the snapshot. Where are you going to save that original data? Why in the *-cow LV of course.

The origin and snapshot LVs do not have data of their own. The origin data is in *-real, and new writes to the origin first read *-real, write *-cow, then finally do the new write to *-real. This read-write-write cycle is tolerable for a single snapshot, but quickly gets out of hand with multiple snapshots.

Writes to the snapshot, on the other hand, simply write to the *-cow with some mapping, no need to update the origin.

Which is why re-writing stuff that was already correct and not being modified doesn't make much sense.

Quoted 2 lines

Writes to the snapshot, on the other hand, simply write to the *-cow with some mapping, no need to update the origin.

I would have guessed that snapshots were read-only. If you can write, it isn't really a snapshot, is it?

But to be useful, you need to write to one or the other. Using the origin as a (mostly) readonly "template" for many VM customers works efficiently with VMs running in snapshots. In that case, writes to the snapshots are just one write. You *could* consider the origin a "snapshot" in that case. But then you are defining "snapshot" to be the readonly branch, which is a different sense than how LVM uses the term.

Give that I currently follow the current procedure for removing and adding customers:

To remove customer: zero out customer LV, then remove LV To add customer: create a new LV

And I want to run backups of a customer's LV using snapshots I just:
create a snapshot of the customer LV, then use rsync, then remove the snapshot. Is there anything I should do to prevent cross-customer data leakage?

If, in addition to "cross-customer data leakage", you are worried about FBI raids finding old data from deleted customers in LVM free space, then zero independent LVs when deleting (as you planned), and zero *-cow for snapshots before deleting with the origin unmounted (but check with an expert whether that is safe while other snapshots for the same origin are mounted - I think it should be).

Quoted 6 lines · Jonathan Tripathy

Give that I currently follow the current procedure for removing and adding customers:

To remove customer: zero out customer LV, then remove LV To add customer: create a new LV

Quoted 4 lines

And I want to run backups of a customer's LV using snapshots I just:
create a snapshot of the customer LV, then use rsync, then remove the snapshot. Is there anything I should do to prevent cross-customer data leakage?

You are still ambiguous. If by "create a new LV", you mean a new LV that is not an LVM snapshot, then just zero it out when you create it (rather than when you delete it).

IDEA - it seems that the device mapper could logically zero an LV by simply returning blocks of zero on reads until the corresponding block it written. Yeah, would require overhead to track which blocks have been written. That overhead could be 1 bit for each of fairly large blocks, and be fairly small, fit into ram easily, and be stored in a logically zeroed block and discarded when the last block is written. So effectively it only requires storing a pointer in meta-data to the current block where the bitmap is stored. I can see that compared to the simplicity of simply writing zeroes on allocation, it might not be worth it.

So I'm guessing then, that when a snapshot is created for an origin, then there are 2 physical copies of the data on disk? (Albeit only one is accessible at the regular filesystem level)

Quoted 3 lines · Jonathan Tripathy

So I'm guessing then, that when a snapshot is created for an origin, then there are 2 physical copies of the data on disk? (Albeit only one is accessible at the regular filesystem level)

NO, NO, NO. There is still only *one* physical copy of the data after creating a snapshot. You have simply created a "branch point" which can now diverge as each branch is written to.

Quoted 2 lines · Stuart D. Gathman

On Thu, 24 Feb 2011, Jonathan Tripathy wrote:

Quoted 7 lines

Quoted 3 lines

So I'm guessing then, that when a snapshot is created for an origin, then there are 2 physical copies of the data on disk? (Albeit only one is accessible at the regular filesystem level)

NO, NO, NO. There is still only *one* physical copy of the data after creating a snapshot. You have simply created a "branch point" which can now diverge as each branch is written to.

Then why was it suggested that I should zero my new customer LVs upon creation? Please remember that my snapshots will not be written to

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

So I'm guessing then, that when a snapshot is created for an origin, then
there are 2 physical copies of the data on disk? (Albeit only one is accessible at the regular filesystem level)

Quoted 7 lines

Quoted 4 lines

NO, NO, NO. There is still only *one* physical copy of the data after creating a snapshot. You have simply created a "branch point" which can now diverge as each branch is written to.

Then why was it suggested that I should zero my new customer LVs upon creation? Please remember that my snapshots will not be written to

However the origin will be written to as it will be in use...

Quoted 11 lines · Jonathan Tripathy

Quoted 11 lines

Quoted 8 lines

Quoted 4 lines

So I'm guessing then, that when a snapshot is created for an origin, then
there are 2 physical copies of the data on disk? (Albeit only one is accessible at the regular filesystem level)

NO, NO, NO. There is still only *one* physical copy of the data after creating a snapshot. You have simply created a "branch point" which can now diverge as each branch is written to.

Then why was it suggested that I should zero my new customer LVs upon creation? Please remember that my snapshots will not be written to

Quoted 1 line

However the origin will be written to as it will be in use...

There were 3 cases of what you might be asking. One of the 3 cases was:
If you are taking snapshots for backup, then it was suggested to zero the *-cow (which will have any blocks written to the origin since the snapshot was taken) before deleting the snapshot. However, I wasn't sure if this was safe to while the origin is mounted, since writes to the origin consult the *-cow to see whether origin blocks need to be copied before begin overwritten.

Quoted 8 lines

Quoted 1 line

However the origin will be written to as it will be in use...

There were 3 cases of what you might be asking. One of the 3 cases was:
If you are taking snapshots for backup, then it was suggested to zero the *-cow (which will have any blocks written to the origin since the snapshot was taken) before deleting the snapshot. However, I wasn't sure if this was safe to while the origin is mounted, since writes to the origin consult the *-cow to see whether origin blocks need to be copied before begin overwritten.

But I want to keep this data. As the data written to the origin while a respective snapshot is in use would be data during normal operation of a customer VPS.

All I want to do is make sure that my use of snapshot for backup purposes (to rsync the snapshot to a remote server) doesn't leave loose data lying anywhere on physical disk.

Quoted 12 lines · Jonathan Tripathy

Quoted 8 lines

Quoted 1 line

However the origin will be written to as it will be in use...

There were 3 cases of what you might be asking. One of the 3 cases was:
If you are taking snapshots for backup, then it was suggested to zero the *-cow (which will have any blocks written to the origin since the snapshot was taken) before deleting the snapshot. However, I wasn't sure if this was safe to while the origin is mounted, since writes to the origin consult the *-cow to see whether origin blocks need to be copied before begin overwritten.

But I want to keep this data. As the data written to the origin while a respective snapshot is in use would be data during normal operation of a customer VPS.

Quoted 3 lines

All I want to do is make sure that my use of snapshot for backup purposes (to rsync the snapshot to a remote server) doesn't leave loose data lying anywhere on physical disk.

Yes. To be more pedantic, the COW has copies of the original contents of blocks written to the origin since the snapshot. That is why you need to clear it to achieve your stated purpose. The origin blocks are written normally to the *-real volume (you can see these in the /dev/mapper directory).

I am only concerned that since the COW has both the saved origin blocks, and a table of which blocks from the origin have been saved, that clearing the table might cause problems for an active origin. It should be safe with the origin unmounted (but having to shutdown the VM defeats the purpose of a snapshot!) I am hoping an expert will weigh in.

Quoted 5 lines

Yes. To be more pedantic, the COW has copies of the original contents of blocks written to the origin since the snapshot. That is why you need to clear it to achieve your stated purpose. The origin blocks are written normally to the *-real volume (you can see these in the /dev/mapper directory).

But didn't you say that there is only one copy of the files stored physically on disk?

Quoted 5 lines · Jonathan Tripathy

Quoted 5 lines

Yes. To be more pedantic, the COW has copies of the original contents of blocks written to the origin since the snapshot. That is why you need to clear it to achieve your stated purpose. The origin blocks are written normally to the *-real volume (you can see these in the /dev/mapper directory).

Quoted 2 lines

But didn't you say that there is only one copy of the files stored physically on disk?

Yes. When you make the snapshot, there is only one copy, and the COW table is empty. AS YOU WRITE to the origin, each chunk written is saved to *-cow first before being written to *-real.

Quoted 7 lines · Stuart D. Gathman

Quoted 5 lines · Jonathan Tripathy

Quoted 5 lines

Yes. To be more pedantic, the COW has copies of the original contents of blocks written to the origin since the snapshot. That is why you need to clear it to achieve your stated purpose. The origin blocks are written normally to the *-real volume (you can see these in the /dev/mapper directory).

Quoted 5 lines

Quoted 2 lines

But didn't you say that there is only one copy of the files stored physically on disk?

Yes. When you make the snapshot, there is only one copy, and the COW table is empty. AS YOU WRITE to the origin, each chunk written is saved to *-cow first before being written to *-real.

Got ya. So data that is being written to the origin, while the snapshot exists, is the data that may leak, as it's saved to the COW first, then copied over to real.

Hopefully an expert will let me know weather its safe to zero the COW after I�ve finished with the snapshot.

Quoted 20 lines · Jonathan Tripathy

Quoted 7 lines · Stuart D. Gathman

Quoted 5 lines · Jonathan Tripathy

Quoted 5 lines

Yes. To be more pedantic, the COW has copies of the original contents of blocks written to the origin since the snapshot. That is why you need to clear it to achieve your stated purpose. The origin blocks are written normally to the *-real volume (you can see these in the /dev/mapper directory).

Quoted 7 lines

Quoted 3 lines

But didn't you say that there is only one copy of the files stored physically
on disk?

Yes. When you make the snapshot, there is only one copy, and the COW table
is empty. AS YOU WRITE to the origin, each chunk written is saved to *-cow first before being written to *-real.

Got ya. So data that is being written to the origin, while the snapshot exists, is the data that may leak, as it's saved to the COW first, then copied over to real.

Quoted 4 lines

Hopefully an expert will let me know weather its safe to zero the COW after I�ve finished with the snapshot.

Oh, I forgot to mention. My LVM sits on top of linux software RAID1. Does that complicate things at all?

One other thing that wasn't mentioned here.... As far as I understand, if these are lvms on the host and the snapshots are being taken on the host, there is no guaranteed integrity of the filesystems unless you shutdown the guest while the snapshot is taken.

It's too bad they don't implement a system call that could do something like sync filesystem and sleep until I tell you to continue. This would be perfect for snapshot backups of virtual hosts.

One other thing that wasn't mentioned here.... As far as I understand, if these are lvms on the host and the snapshots are being taken on the host, there is no guaranteed integrity of the filesystems unless you shutdown the guest while the snapshot is taken.

It's too bad they don't implement a system call that could do something like sync filesystem and sleep until I tell you to continue. This would be perfect for snapshot backups of virtual hosts.

Oh, I forgot to mention. My LVM sits on top of linux software RAID1. Does that complicate things at all?

Quoted 3 lines

Yes. When you make the snapshot, there is only one copy, and the COW table is empty. AS YOU WRITE to the origin, each chunk written is saved to *-cow first before being written to *-real.

Got ya. So data that is being written to the origin, while the snapshot exists, is the data that may leak, as it's saved to the COW first, then copied over to real.

Hopefully an expert will let me know weather its safe to zero the COW after I've finished with the snapshot.

Quoted 9 lines · Jonathan Tripathy

Quoted 3 lines

Yes. When you make the snapshot, there is only one copy, and the COW table is empty. AS YOU WRITE to the origin, each chunk written is saved to *-cow first before being written to *-real.

Got ya. So data that is being written to the origin, while the snapshot exists, is the data that may leak, as it's saved to the COW first, then copied over to real.

Hopefully an expert will let me know weather its safe to zero the COW after I've finished with the snapshot.

What *is* safe is to zero the snapshot. This will overwrite any blocks in the COW copied from the origin. The problem is that if the snapshot runs out of room, it is invalidated, and you may or may not have overwritten all blocks copied from the origin.

So if you don't hear from an expert, a safe prodecure is to allocate snapshots for backup that are as big as the origin + 1 PP (which should be enough for the COW table as well unless we are talking terabytes). Then you can zero the snapshot (not the COW) after making a backup. That will overwrite any data copied from the origin. The only leftover data will just be the COW table which is a bunch of block #s, so shouldn't be a security problem.

This procedure is less efficient than zeroing LVs on allocation, and takes extra space for worst case snapshot allocation. But if you want allocation to be "instant", and can pay for it when recycling, that should solve your problem. You should still zero all free space (by allocating a huge LV with all remaining space and zeroing it) periodically in case anything got missed.

IDEA, since you are on raid1, reads are faster than writes (twice as fast), and your snapshots will be mostly unused (the COW will only have a few blocks copied from the origin). So you can write a "clear" utility that scans a block device for non-zero chunks, and only writes over those with zeros. This might be a good application for mmap().

Quoted 7 lines · Stuart D. Gathman

Quoted 5 lines · Jonathan Tripathy

Quoted 5 lines

Yes. To be more pedantic, the COW has copies of the original contents of blocks written to the origin since the snapshot. That is why you need to clear it to achieve your stated purpose. The origin blocks are written normally to the *-real volume (you can see these in the /dev/mapper directory).

Quoted 5 lines

Quoted 2 lines

But didn't you say that there is only one copy of the files stored physically on disk?

Yes. When you make the snapshot, there is only one copy, and the COW table is empty. AS YOU WRITE to the origin, each chunk written is saved to *-cow first before being written to *-real.

Got ya. So data that is being written to the origin, while the snapshot exists, is the data that may leak, as it's saved to the COW first, then copied over to real.

Hopefully an expert will let me know weather its safe to zero the COW after I�ve finished with the snapshot.

Quoted 1 line

Hopefully an expert will let me know weather its safe to zero the COW after I?ve finished with the snapshot.

Are any "experts" available to help me answer the above question? I feel that this is a really important issue for those of us in the hosting industry.

Just to sum up my question: when a customer leaves our service, we zero their drive before removing the LV. This hopefully ensures that there is no data "leakage" when we create a new LV for a new customer. However, we need to take into consideration about what happens when we create snapshots of LVs to perform backups (using rsync).

Any help would be appreciated.

Quoted 6 lines · Jonathan Tripathy

Quoted 1 line

Hopefully an expert will let me know weather its safe to zero the COW after I?ve finished with the snapshot.

Are any "experts" available to help me answer the above question? I feel that this is a really important issue for those of us in the hosting industry.

Just to sum up my question: when a customer leaves our service, we zero their drive before removing the LV. This hopefully ensures that there is no data "leakage" when we create a new LV for a new customer. However, we need to take into consideration about what happens when we create snapshots of LVs to perform backups (using rsync).

Any help would be appreciated.

At this point, we'll just have to try it on a non-production server. Hopefully, worst case the kernel crashes. I run Fedora (14 currently) and CentOS-5.5. My guess as an amateur is that zeroing the COW while
the origin is open is a problem. I would suggest this for backup:

1) make snapshot
2) make backup
3) pause VM (check that this closes the origin LV, if not save the VM)
4) with both origin and snapshot not active, zero COW
5) remove snapshot
6) unpause VM (or restore if saved)

Signature · 11 lines

          Stuart D. Gathman <[email protected]>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 
591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Yeah, I'll have to try this on a non-production server. Which kernel do you expect to crash? The Dom0 (Xen + LVM host) or the VM? Anyway, just thinking about it, it seems that pausing/saving/shutting down the VM is a must, as the VM may be writing to disk at the time of zeroing the cow (!!).

In the hosting industy, what does everyone else do? Do they just ignore the issue???

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 10.0.1209 / Virus Database: 1500/3552 - Release Date: 04/05/11

Quoted 12 lines · Jonathan Tripathy

Yeah, I'll have to try this on a non-production server. Which kernel do you expect to crash? The Dom0 (Xen + LVM host) or the VM? Anyway, just thinking about it, it seems that pausing/saving/shutting down the VM is a must, as the VM may be writing to disk at the time of zeroing the cow (!!).

In the hosting industy, what does everyone else do? Do they just ignore the issue???

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Quoted 4 lines

-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 10.0.1209 / Virus Database: 1500/3552 - Release Date: 04/05/11

As an alturnative could you not?

1) Create snapshot lv using specific physical extents. lvcreate -s -l 100 ..... /dev/sdx1:0-99
2) Backup VM
3) Delete snapshot lv
4) create a normal lv using same physical extents lvcreate -l 100 ..... /dev/sdx1:0-99
5) zero normal lv
6) delete normal lv

James

Hi James,

Interesting, didn't know you could do that! However, how do I know that the PEs aren't being used by LVs? Also, could you please explain the syntax? Normally to create a snapshot, I would do:

lvcreate -L20G -s -n backup /dev/vg0/customerID

Quoted 6 lines · Jonathan Tripathy

Hi James,

Interesting, didn't know you could do that! However, how do I know that the PEs aren't being used by LVs? Also, could you please explain the syntax? Normally to create a snapshot, I would do:

lvcreate -L20G -s -n backup /dev/vg0/customerID

Hmmm well you have two options, you could use pvdisplay --map or lvdisplay --map to work out exactly which PEs have been used to build you snapshot cow and then use that information to allow you to create a blanking PV in the same place or you could do it the easy way :-

1 hog the space to specific PEs 2 delete the hog
3 create the snapshot on same PEs 4 backup
5 delete the snapshot
6 create the hog on the same PEs 7 zero the hog

This has the advantage that the creation commands will fail if the PEs you want are not available the problem with it is you probably need more space for snapshots. As its less flexible in space use. Below i have illustrated all the commands, you need to do this. you don;t need all the display commands but they are there to prove to you this has worked, and the lvs are in the same place.

#pvdisplay --map /dev/cciss/c0d1p1
--- Physical volume ---

  PV Name               /dev/cciss/c0d1p1
  VG Name               test_vg
  PV Size               683.51 GB / not usable 5.97 MB
  Allocatable           yes
  PE Size (KByte)       131072
  Total PE              5468
  Free PE               4332
  Allocated PE          1136
  PV UUID               YXjplf-EfLh-8Jkr-2utT-gmi5-gjAH-UOCcN0

--- Physical Segments --- Physical extent 0 to 15:
Logical volume /dev/test_vg/test_lv
Logical extents 0 to 15
Physical extent 16 to 815:
Logical volume /dev/test_vg/mail_lv
Logical extents 0 to 799
Physical extent 816 to 975:
Logical volume /dev/test_vg/data_lv
Logical extents 0 to 159
Physical extent 976 to 2255:
FREE
Physical extent 2256 to 2335:
Logical volume /dev/test_vg/srv_lv
Logical extents 0 to 79
Physical extent 2336 to 2415:
Logical volume /dev/test_vg/data_lv
Logical extents 160 to 239
Physical extent 2416 to 5467:
FREE

#lvcreate -l 20 -n hog_lv test_vg /dev/cciss/c0d1p1:5448-5467

#pvdisplay --map /dev/cciss/c0d1p1
--- Physical volume ---

  PV Name               /dev/cciss/c0d1p1
  VG Name               test_vg
  PV Size               683.51 GB / not usable 5.97 MB
  Allocatable           yes
  PE Size (KByte)       131072
  Total PE              5468
  Free PE               4312
  Allocated PE          1156
  PV UUID               YXjplf-EfLh-8Jkr-2utT-gmi5-gjAH-UOCcN0

--- Physical Segments --- Physical extent 0 to 15:
Logical volume /dev/test_vg/test_lv
Logical extents 0 to 15
Physical extent 16 to 815:
Logical volume /dev/test_vg/mail_lv
Logical extents 0 to 799
Physical extent 816 to 975:
Logical volume /dev/test_vg/data_lv
Logical extents 0 to 159
Physical extent 976 to 2255:
FREE
Physical extent 2256 to 2335:
Logical volume /dev/test_vg/srv_lv
Logical extents 0 to 79
Physical extent 2336 to 2415:
Logical volume /dev/test_vg/data_lv
Logical extents 160 to 239
Physical extent 2416 to 5447:
FREE
Physical extent 5448 to 5467:
Logical volume /dev/test_vg/hog_lv
Logical extents 0 to 19

#lvremove /dev/test_vg/hog_lv
Do you really want to remove active logical volume hog_lv? [y/n]: y
Logical volume "hog_lv" successfully removed
#lvcreate -l 20 -s -n data_snap /dev/test_vg/data_lv /dev/cciss/c0d1p1:5448-5467
Logical volume "data_snap" created
#pvdisplay --map /dev/cciss/c0d1p1

                                   --- Physical volume ---
  PV Name               /dev/cciss/c0d1p1
  VG Name               test_vg
  PV Size               683.51 GB / not usable 5.97 MB
  Allocatable           yes
  PE Size (KByte)       131072
  Total PE              5468
  Free PE               4312
  Allocated PE          1156
  PV UUID               YXjplf-EfLh-8Jkr-2utT-gmi5-gjAH-UOCcN0

--- Physical Segments --- Physical extent 0 to 15:
Logical volume /dev/test_vg/restricted_lv
Logical extents 0 to 15
Physical extent 16 to 815:
Logical volume /dev/test_vg/mail_lv
Logical extents 0 to 799
Physical extent 816 to 975:
Logical volume /dev/test_vg/data_lv
Logical extents 0 to 159
Physical extent 976 to 2255:
FREE
Physical extent 2256 to 2335:
Logical volume /dev/test_vg/srv_lv
Logical extents 0 to 79
Physical extent 2336 to 2415:
Logical volume /dev/test_vg/data_lv
Logical extents 160 to 239
Physical extent 2416 to 5447:
FREE
Physical extent 5448 to 5467:
Logical volume /dev/test_vg/data_snap
Logical extents 0 to 19

#lvdisplay /dev/test_vg/data_snap
--- Logical volume ---

  LV Name                /dev/test_vg/data_snap
  VG Name                test_vg
  LV UUID                bdqB77-f0vb-ZucS-Ka1l-pCr3-Ebeq-kOchmk
  LV Write Access        read/write
  LV snapshot status     active destination for /dev/test_vg/data_lv
  LV Status              available
  # open                 0
  LV Size                30.00 GB
  Current LE             240
  COW-table size         2.50 GB
  COW-table LE           20
  Allocated to snapshot  0.00%
  Snapshot chunk size    4.00 KB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:5

#lvdisplay --map /dev/test_vg/data_snap
--- Logical volume ---

  LV Name                /dev/test_vg/data_snap
  VG Name                test_vg
  LV UUID                IBBvOq-Bg0U-c69v-p7fQ-tR63-T8UV-gM1Ncu
  LV Write Access        read/write
  LV snapshot status     active destination for /dev/test_vg/data_lv
  LV Status              available
  # open                 0
  LV Size                30.00 GB
  Current LE             240
  COW-table size         2.50 GB
  COW-table LE           20
  Allocated to snapshot  0.00%
  Snapshot chunk size    4.00 KB
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:5

--- Segments ---
Logical extent 0 to 19:

    Type                linear
    Physical volume     /dev/cciss/c0d1p1
    Physical extents    5448 to 5467

<NOW BACKUP>

#lvremove /dev/test_vg/data_snap
Do you really want to remove active logical volume data_snap? [y/n]: y
Logical volume "data_snap" successfully removed

#lvcreate -l 20 -n hog_lv test_vg /dev/cciss/c0d1p1:5448-5467
Logical volume "hog_lv" created

#pvdisplay --map /dev/cciss/c0d1p1 --- Physical volume ---

  PV Name               /dev/cciss/c0d1p1
  VG Name               test_vg
  PV Size               683.51 GB / not usable 5.97 MB
  Allocatable           yes
  PE Size (KByte)       131072
  Total PE              5468
  Free PE               4312
  Allocated PE          1156
  PV UUID               YXjplf-EfLh-8Jkr-2utT-gmi5-gjAH-UOCcN0

--- Physical Segments --- Physical extent 0 to 15:
Logical volume /dev/test_vg/restricted_lv
Logical extents 0 to 15
Physical extent 16 to 815:
Logical volume /dev/test_vg/mail_lv
Logical extents 0 to 799
Physical extent 816 to 975:
Logical volume /dev/test_vg/data_lv
Logical extents 0 to 159
Physical extent 976 to 2255:
FREE
Physical extent 2256 to 2335:
Logical volume /dev/test_vg/srv_lv
Logical extents 0 to 79
Physical extent 2336 to 2415:
Logical volume /dev/test_vg/data_lv
Logical extents 160 to 239
Physical extent 2416 to 5447:
FREE
Physical extent 5448 to 5467:
Logical volume /dev/test_vg/hog_lv
Logical extents 0 to 19

#dd if=/dev/zero of=/dev/hog_lv

#lvremove /dev/test_vg/hog_lv
Do you really want to remove active logical volume hog_lv? [y/n]: y
Logical volume "hog_lv" successfully removed

Enjoy

James

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

James,

That's fantastic! Thanks very much! I have a couple of questions:

1) If I wanted to create a script that backed up lots of customer-data LVs, could I just do one zero at the end (and still have no data leakage)?

2) On average, each of my data LVs are 20GB each, and if I were to create a snapshot of 20GB, this would take about 20 mins to erase. If I made the snapshot only 1GB, that means it would be quick to erase at the end (however only 1GB of data could be created on the respect origin, correct?)

Quoted 6 lines

James,

That's fantastic! Thanks very much! I have a couple of questions:

1) If I wanted to create a script that backed up lots of customer-data LVs, could I just do one zero at the end (and still have no data leakage)?

Yes you could, because COW means COPY ON WRITE, so the original block is copied onto the COW with the data from the original disk overwritting any data currently on it. Before that point any data on it was not addressable from the snapshot lv (* see my final point)

Quoted 1 line

2) On average, each of my data LVs are 20GB each, and if I were to create a snapshot of 20GB, this would take about 20 mins to erase. If I made the snapshot only 1GB, that means it would be quick to erase at the end (however only 1GB of data could be created on the respect origin, correct?)

You are right you only have to erase the snapshot cow space, which is normally only 10-15% of the whole original disk. 2GB is pretty fast to over right on any system I have used these days. To be sure though you do need to overwrite the whole cow even if only a few percent was used as you cannot tell which few percent that was.

* I do wonder why you are so worried, leakage is only a problem if the COW is assigned to a future customer LV. If you always used the same space for backups perhaps had a PE just for backups it would never be used in a customer lv therefore you could argue that you never have to erase it. If its on a pe you only use for snapshotting you also don't need to hog the space as any bit of that disk is ok.

Excellent point! As long as I use the same PEs for making the snapshot everytime, I don't need to ever erase it (And it can be a nice big size like 50GB, so even my largest customers won't outgrow the snapshot). However though, wouldn't I need to keep the "hog" around just to make sure that the snapshot PEs don't get assigned to a new customer LV in the future (Currently, we don't specify PEs to use when creating normal LVs)?

An even better question: Does the snapshot have to be on the same physical disk as the LV its mirroring?

Quoted 2 lines · Jonathan Tripathy

Excellent point! As long as I use the same PEs for making the snapshot everytime, I don't need to ever erase it (And it can be a nice big size like 50GB, so even my largest customers won't outgrow the snapshot). However though, wouldn't I need to keep the "hog" around just to make sure that the snapshot PEs don't get assigned to a new customer LV in the future (Currently, we don't specify PEs to use when creating normal LVs)?

I think you missed the point of why I suggested using a separate PV, the space could be divided using fdisk it does not have to separate physical disk (This is ok as you will never use this space for mirroring). If snapshots are created on a separate PV you can use pvchange - x n and pvchange -x y to change if it is allocatable and only when you are creating snaps do you make in allocatable that will prevent accidental reuse in customer lvs without lots of hassle. If you don't use pvchange you will need to specify the PVs when ever you create a customer LV.

Do I get cake now?

lvcreate -nNewCustomerLV -L20G vg0

where vg0 is /dev/vg0
and vg0 is a volume group which uses an entire physical partition (Which I guess is called a PV).

Now, if I were to create my snapshots on a seperate vg, eg:

Quoted 6 lines · Jonathan Tripathy

lvcreate -nNewCustomerLV -L20G vg0

where vg0 is /dev/vg0
and vg0 is a volume group which uses an entire physical partition (Which I guess is called a PV).

Now, if I were to create my snapshots on a seperate vg, eg:

You can't do that. Snapshots must be on the same VG. James is suggesting you create them on the same VG, but a different PV reserved for backup snapshots.

Ok now things are really getting interesting!

Actually, when I create new customer LVs, I always specify which volume group I want to add it to. E.g:

lvcreate -nNewCustomerLV -L20G vg0

where vg0 is /dev/vg0
and vg0 is a volume group which uses an entire physical partition (Which I guess is called a PV).

Now, if I were to create my snapshots on a seperate vg, eg:

lvcreate -L 20G -s -n data_snap /dev/vg0/NewCustomerLV /dev/vg1

does that mean I never need to use pvchange to "switch off" vg1? And I never need to zero or create a "hog"? And no leakage will ever occur?

Quoted 2 lines

Do I get cake now?

Only if it's not a lie.... :)

No you are confused pvs are created on disk partitions, one or more pvs then make you vg.

LVs are created on top of VGs, VGs are created on top of PVs, PVs are created on partitions on block devices or whole block devices.

(Ok we shall stop here however blocks devices could be loop back, meta devices like raid 0/1/5 etc hardware/software or real disks however that is not LVM any more.)

pvcreate /dev/hdb7
vgextend /dev/hdb7

Then, every time I create a new customer LV, I would do:

lvcreate -nNewCustomerLV -L20G vg0 /dev/md3

Then, every time I wanted to create a snapshot:

lvcreate -L20G -s -n data_snap /dev/vg0/NewCustomerLV /dev/hdb7

Is that correct? No Leakage? And no zeroing needed?

Quoted 3 lines

lvcreate -L20G -s -n data_snap /dev/vg0/NewCustomerLV /dev/hdb7

Is that correct? No Leakage? And no zeroing needed?

Correct. Except vgextend needs to specify a VG. And you still need to zero a customer LV before removing it. The PV reserved for snapshots approach only avoids the need to zero snapshots.

In fact, since you still have to zero an LV before removing, you might as well zero an LV when you allocate. Same overhead. Then you don't need to mess with any of these schemes to deal with snapshots. Did we already cover that obvious solution? We did discuss a utility to optimize zeroing an already mostly zero LV.

Quoted 6 lines · Stuart D. Gathman

In fact, since you still have to zero an LV before removing, you might as well zero an LV when you allocate. Same overhead. Then you don't need to mess with any of these schemes to deal with snapshots. Did we already cover that obvious solution? We did discuss a utility to optimize zeroing an already mostly zero LV.

I think you mean and lv of a virtual machine rather than snap shot, not sure how you could zero the cow. Stuart does have a good point here, there is another source of leak outside of snapshots (and probably worse), if you delete an lv the create a new one over the same PEs the data from then old lv will be there, save a few k that is zeroed at the front (to help with fs/label creation). This is worse as it is in order data that the cow from a snap is not. Geting a while fs back is very possible.

Currently, as part of our procedure, we zero LVs once a customer has left our service. The reason why we don't zero on allocation is that customer usually expect quick setup times (under 10 minutes) and zeroing gigabytes worth of space can take too long. Getting new zeroed LVs ready before sign ups also isn't an option but for other reasons.

Quoted 9 lines · Jonathan Tripathy

pvcreate /dev/hdb7
vgextend /dev/hdb7

Then, every time I create a new customer LV, I would do:

lvcreate -nNewCustomerLV -L20G vg0 /dev/md3

Then, every time I wanted to create a snapshot:

Quoted 3 lines

lvcreate -L20G -s -n data_snap /dev/vg0/NewCustomerLV /dev/hdb7

Is that correct? No Leakage? And no zeroing needed?

Correct. Except vgextend needs to specify a VG. And you still need to zero a customer LV before removing it. The PV reserved for snapshots approach only avoids the need to zero snapshots.

I wonder if there is an elegant way to automate things like this via "allocation policies" or similar in the metadata.

Ok, i think I get it now. At the minute, my vg (vg0) only has on PV in it (/dev/md3 which you can tell is a mdadm RAID device). I wasn't aware you could add more PVs (that's pretty cool!). So, let's say I had a spare partition (/dev/hdb7 as an example). To my vg0 volume group, I would firstly:

pvcreate /dev/hdb7
vgextend /dev/hdb7

Then, every time I create a new customer LV, I would do:

lvcreate -nNewCustomerLV -L20G vg0 /dev/md3

Then, every time I wanted to create a snapshot:

lvcreate -L20G -s -n data_snap /dev/vg0/NewCustomerLV /dev/hdb7

Is that correct? No Leakage? And no zeroing needed?

Side note: Since I didn't partition my servers with this in mind, my new PV will probably have to be an iSCSI device located on a remote target :( Either that or use a loopback device with an image, but I'd be scared that the system would not boot properly. Can you give me any tips on how to use an image file as a PV just for snapshots?

Quoted 4 lines · Jonathan Tripathy

Ok, i think I get it now. At the minute, my vg (vg0) only has on PV in it (/dev/md3 which you can tell is a mdadm RAID device). I wasn't aware you could add more PVs (that's pretty cool!). So, let's say I had a spare partition (/dev/hdb7 as an example). To my vg0 volume group, I would firstly:

pvcreate /dev/hdb7
vgextend /dev/hdb7

Right however danger warnings are going off in my mind now!

Quoted 3 lines

Then, every time I create a new customer LV, I would do:

lvcreate -nNewCustomerLV -L20G vg0 /dev/md3

yes that would work

Quoted 5 lines

Then, every time I wanted to create a snapshot:

lvcreate -L20G -s -n data_snap /dev/vg0/NewCustomerLV /dev/hdb7

Yes

Quoted 1 line

Is that correct? No Leakage? And no zeroing needed?

Indeed

Quoted 2 lines

Side note: Since I didn't partition my servers with this in mind, my new PV will probably have to be an iSCSI device located on a remote target :( Either that or use a loopback device with an image, but I'd be scared that the system would not boot properly. Can you give me any tips on how to use an image file as a PV just for snapshots?

Ok, there has been alot of dangerous talk here, i assume you are using an md device so you can mirror things.

If you added a single disk to that, and that disk failed you would have a major problem. Likewise if you rebooted with open snaps and iscsi you would need that iscsi device available during computer boot. I REALLY hope you do not have your local FS on the same vg as your data. As this would result in a non booting machine.

If the PV used for snapshots were to fail while the snapshot was open, or the server rebooted and the PV wasn't available at boot, what would happen? I ask these questions because a loopback device or iSCSI is really my only feasible option right now for the snapshot PV...

Quoted 2 lines · Jonathan Tripathy

If the PV used for snapshots were to fail while the snapshot was open, or the server rebooted and the PV wasn't available at boot, what would happen? I ask these questions because a loopback device or iSCSI is really my only feasible option right now for the snapshot PV...

What would happen is... if the file systems are mounted at boot time (in the fstab) it will fail the fsck because the device is not there. and drop to single user mode, you could then edit the fstab to to remove those file systems that would bring the system online, at which point you could fix what stopped the iscsi from working, and mount the file systems.

At one place I worked they never mounted the data file systems at "boot" but in a rc.local so the system always got to be interactive before any problems so it was easy to go in and fix.

DO NOT... create a loop back device on the same on a file system that the loopback that will then form a pv of, if you do your system is DOOOMMMED! to get it to boot again your have to mount a part volume group and block copy the devices to a new one, even worse if you extended the file system with the loop back on it onto the pv of the loop back it will NEVER work again. So the only place you can create a loopback device is outside of a vg it is to be a part of and frankley better that its in NO vg as you may get recursion problems.

The problem with a loop back is that you need to do a the loopback setup to enable the device before the vgscan and vgchange can bring it online in the volume, very hard to get right at boot time. If you have partitioned it you will also need to do kpartx.

If you use loopbacks i would extend the volume group onto the disk only during backups then nreduce it out afterwards to reduce risks.

Quoted 17 lines · Jonathan Tripathy

Quoted 16 lines

Quoted 5 lines

So you're not worried about the security implication of leftovers in free
space, and just want a base image to clone for new customers?

Quoted 8 lines

The logical thing to do is to keep the origin volume untouched (except for upgrading now and then), and take a snapshot for each customer. Each snapshot would then be a new clone of the origin. Unfortunately, large numbers of snapshots are inefficient for writes to new data, so you'd likely have to "dd" to an independent LV instead. (This is being
worked on, and there are 3rd party products like Zumastor that fix it now.)

Actually, if you never (or rarely) write to the origin, lots of snapshots
should be fine.

Quoted 6 lines

Quoted 3 lines

But every write to the origin will first copy the
original origin data to every snapshot.

Why would origin data be copied over to the snapshot after the snapshot has been created? Surely the point of a snapshot is to have "frozen" data?

Yes, is the way this actually works explained somewhere? I would have expected the 'copy-on-write' blocks to be copied only on the side where the write is happening and relocated instead of rewriting all the snapshots that might be outstanding with the old data.

Quoted 6 lines

Actually, if you never (or rarely) write to the origin, lots of snapshots should be fine.

But every write to the origin will first copy the
original origin data to every snapshot.

Why would origin data be copied over to the snapshot after the snapshot has been created? Surely the point of a snapshot is to have "frozen" data?

Or maybe you want to *revert* the origin back to the state saved in a snapshot. A specific revert command is in the works, but to do it manually, unmount the origin, and copy the snapshot over

Quoted 3 lines · Stuart D. Gathman

Or maybe you want to *revert* the origin back to the state saved in a snapshot. A specific revert command is in the works, but to do it manually, unmount the origin, and copy the snapshot over

That is lvconvert --merge, and it has been around for a while now. It is quite handy for testing a dist-upgrade and then rolling back.

Quoted 6 lines · Phillip Susi

Quoted 3 lines · Stuart D. Gathman

Or maybe you want to *revert* the origin back to the state saved in a snapshot. A specific revert command is in the works, but to do it manually, unmount the origin, and copy the snapshot over

That is lvconvert --merge, and it has been around for a while now. It is quite handy for testing a dist-upgrade and then rolling back.

Yes, I see that it is in Fedora, but not EL5.5.