Skip to contentSkip to conversation

Any way to speed up activation of volumes with snapshots?

linux-lvm Chris Friesen 2 participants 13 messages Sep 14, 2015 – Sep 16, 2015

lore
Share & tools
lore
#0 lore raw

Hi,

I'm running a 3.10 kernel with LVM 2.02.95.

I'm running into a problem where activating snapshots can take quite a long time, roughly one minute per 25GB of delta between the snapshot and the origin volume. (See below for my test procedure.)

I realize that my kernel/LVM aren't exactly bleeding edge, and I wondering whether more recent versions have done anything to speed up the activation process (like maybe making it more lazy-loaded rather than reading in a bunch of data up-front).

If anyone is aware of such improvements, I'd appreciate it if you could point me at the appropriate changes.

For those that are interested, the test that I did was as follows:

lvcreate -L100G -n test1 vg
lvcreate -L 100G -s -n test1snapshot /dev/vg/test1
dd if=/dev/zero of=/dev/cinder-volumes/test1snapshot bs=1M count=95000 lvchange -an /dev/cinder-volumes/test1snapshot
time lvchange -ay /dev/cinder-volumes/test1snapshot

The final "lvchange" command took about 4m22s. This works out to 362MB/sec, which is almost exactly the speed I got doing a "dd" from the volume to /dev/null.

Thanks,
Chris

#1 lore raw

My apologies, I got the version wrong. The tests below were actually from the 3.4 kernel and LVM 2.2.95. I'm doing tests on the newer versions now.

Chris

Quoted 36 lines · Chris Friesen

Hi,

I'm running a 3.10 kernel with LVM 2.02.95.

I'm running into a problem where activating snapshots can take quite a long time, roughly one minute per 25GB of delta between the snapshot and the origin volume. (See below for my test procedure.)

I realize that my kernel/LVM aren't exactly bleeding edge, and I wondering whether more recent versions have done anything to speed up the activation process (like maybe making it more lazy-loaded rather than reading in a bunch of data up-front).

If anyone is aware of such improvements, I'd appreciate it if you could point me at the appropriate changes.

For those that are interested, the test that I did was as follows:

lvcreate -L100G -n test1 vg
lvcreate -L 100G -s -n test1snapshot /dev/vg/test1
dd if=/dev/zero of=/dev/cinder-volumes/test1snapshot bs=1M count=95000 lvchange -an /dev/cinder-volumes/test1snapshot
time lvchange -ay /dev/cinder-volumes/test1snapshot

The final "lvchange" command took about 4m22s. This works out to 362MB/sec, which is almost exactly the speed I got doing a "dd" from the volume to /dev/null.

Thanks,
Chris

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#2 lore raw
In reply to Chris Friesen

Dne 14.9.2015 v 20:05 Chris Friesen napsal(a):

Quoted 8 lines

Hi,

I'm running a 3.10 kernel with LVM 2.02.95.

I'm running into a problem where activating snapshots can take quite a long time, roughly one minute per 25GB of delta between the snapshot and the origin volume. (See below for my test procedure.)

Quoted 7 lines

I realize that my kernel/LVM aren't exactly bleeding edge, and I wondering whether more recent versions have done anything to speed up the activation process (like maybe making it more lazy-loaded rather than reading in a bunch of data up-front).

If anyone is aware of such improvements, I'd appreciate it if you could point me at the appropriate changes.

Hi

There is NO way to accelerate your existing setup, other then placing delta on SSD - the format of snapshot was meant to be used 'temporarily' - i.e. until you take backup of LV - but not for long-term multi-gigabyte case.

This is major mis-use of this 'snapshot' feature - which repeats over and over...

If you want to use long-living snapshots - you need to use thin-provisioning, .

Quoted 1 line

The final "lvchange" command took about 4m22s. This works out to 362MB/sec,

I've seen users waiting even 1/2 hour - so you are still 'lucky' user ;)

Zdenek

#3 lore raw
Quoted 19 lines · Zdenek Kabelac

Dne 14.9.2015 v 20:05 Chris Friesen napsal(a):

Quoted 8 lines

Hi,

I'm running a 3.10 kernel with LVM 2.02.95.

I'm running into a problem where activating snapshots can take quite a long time, roughly one minute per 25GB of delta between the snapshot and the origin volume. (See below for my test procedure.)

Quoted 7 lines

I realize that my kernel/LVM aren't exactly bleeding edge, and I wondering whether more recent versions have done anything to speed up the activation process (like maybe making it more lazy-loaded rather than reading in a bunch of data up-front).

If anyone is aware of such improvements, I'd appreciate it if you could point me at the appropriate changes.

Hi

Quoted 7 lines

There is NO way to accelerate your existing setup, other then placing delta on SSD - the format of snapshot was meant to be used 'temporarily' - i.e. until you take backup of LV - but not for long-term multi-gigabyte case.

This is major mis-use of this 'snapshot' feature - which repeats over and over...

If you want to use long-living snapshots - you need to use thin-provisioning,

Thanks for the quick response.

Presumably we would still have multi-GB of delta between the original volume and the snapshot, so what would make the activation of a thin-provisioned snapshot faster? Is the metadata stored differently?

Chris

#4 lore raw

Dne 14.9.2015 v 21:16 Chris Friesen napsal(a):

Quoted 28 lines · Zdenek Kabelac
Quoted 27 lines

Dne 14.9.2015 v 20:05 Chris Friesen napsal(a):

Quoted 8 lines

Hi,

I'm running a 3.10 kernel with LVM 2.02.95.

I'm running into a problem where activating snapshots can take quite a long time, roughly one minute per 25GB of delta between the snapshot and the origin volume. (See below for my test procedure.)

Quoted 7 lines

I realize that my kernel/LVM aren't exactly bleeding edge, and I wondering whether more recent versions have done anything to speed up the activation process (like maybe making it more lazy-loaded rather than reading in a bunch of data up-front).

If anyone is aware of such improvements, I'd appreciate it if you could point me at the appropriate changes.

Hi

There is NO way to accelerate your existing setup, other then placing delta on SSD - the format of snapshot was meant to be used 'temporarily' - i.e. until you
take backup of LV - but not for long-term multi-gigabyte case.

This is major mis-use of this 'snapshot' feature - which repeats over and over...

Quoted 7 lines
Quoted 1 line

If you want to use long-living snapshots - you need to use thin-provisioning,

Thanks for the quick response.

Presumably we would still have multi-GB of delta between the original volume and the snapshot, so what would make the activation of a thin-provisioned snapshot faster? Is the metadata stored differently?

Yes - metadata are on separate device(LV) and are using b-trees. So now you could i.e. chain snap-of-snap-of-snap...
Also snaps do not need to be active with origin and many other goodies.

Drawbacks - all data have to be in same pool and share same space, so you can't run 'just' 'snapshot' out-of-space - whole pool is out-of-space. Also min chunk size is 64K (instead of 4K for old-snap)

Anyway - try and play and see what performs better and suit your needs.

Zdenek

#5 lore raw
In reply to Chris Friesen

Dne 14.9.2015 v 21:16 Chris Friesen napsal(a):

Quoted 28 lines · Zdenek Kabelac
Quoted 27 lines

Dne 14.9.2015 v 20:05 Chris Friesen napsal(a):

Quoted 8 lines

Hi,

I'm running a 3.10 kernel with LVM 2.02.95.

I'm running into a problem where activating snapshots can take quite a long time, roughly one minute per 25GB of delta between the snapshot and the origin volume. (See below for my test procedure.)

Quoted 7 lines

I realize that my kernel/LVM aren't exactly bleeding edge, and I wondering whether more recent versions have done anything to speed up the activation process (like maybe making it more lazy-loaded rather than reading in a bunch of data up-front).

If anyone is aware of such improvements, I'd appreciate it if you could point me at the appropriate changes.

Hi

There is NO way to accelerate your existing setup, other then placing delta on SSD - the format of snapshot was meant to be used 'temporarily' - i.e. until you
take backup of LV - but not for long-term multi-gigabyte case.

This is major mis-use of this 'snapshot' feature - which repeats over and over...

Quoted 7 lines
Quoted 1 line

If you want to use long-living snapshots - you need to use thin-provisioning,

Thanks for the quick response.

Presumably we would still have multi-GB of delta between the original volume and the snapshot, so what would make the activation of a thin-provisioned snapshot faster? Is the metadata stored differently?

If you don't mind dropping existing snapshot and creating new 'old-snapshot' - using larger block size (i.e. 32K (or even more) instead of 4K) may improve speed of disk reading noticeable (at the price of copying larger chunks when modifying disk blocks - depends on use-case)....

And if you don't mind replacing your ancient kernel and trying some newer one - there have been some 'read' order improvement for this old-snapshot target made as well - so you may at least try if there will be any noticeable change (actually would be interesting just to get report how much faster it boots when you only exchange kernel for a boot - you don't have to use it - just measure boot speed)

Zdenek

#6 lore raw
Quoted 4 lines · Zdenek Kabelac

If you don't mind dropping existing snapshot and creating new 'old-snapshot' - using larger block size (i.e. 32K (or even more) instead of 4K) may improve speed of disk reading noticeable (at the price of copying larger chunks when modifying disk blocks - depends on use-case)....

Okay, good info to have.

Quoted 6 lines

And if you don't mind replacing your ancient kernel and trying some newer one - there have been some 'read' order improvement for this old-snapshot target made as well - so you may at least try if there will be any noticeable change (actually would be interesting just to get report how much faster it boots when you only exchange kernel for a boot - you don't have to use it - just measure boot speed)

This is essentially a custom distro, so changing just the kernel is a bit complicated. However, I have a newer version that I want to test with which has a newer kernel but the same version of LVM, so that should give some good data.

Chris

#7 lore raw
In reply to Chris Friesen
Quoted 7 lines · Chris Friesen

Hi,

I'm running a 3.10 kernel with LVM 2.02.95.

I'm running into a problem where activating snapshots can take quite a long time, roughly one minute per 25GB of delta between the snapshot and the origin volume. (See below for my test procedure.)

As a side question, currently we have a script which runs "vgchange -ay <vg>", which will activate the VG.

This script appears to activate the LVs within the VG serially rather than in parallel. Does it do any processing to determine in which order to activate the LVs?

The reason I ask is that we have an overall timeout on the activate phase, after which we'll put any un-activated LVs into an error state. It would be a shame to try to activate a snapshotted LV first, and then timeout without activating any of the non-snapshotted LVs.

Chris

#8 lore raw

Dne 16.9.2015 v 01:03 Chris Friesen napsal(a):

Quoted 12 lines · Chris Friesen
Quoted 7 lines

Hi,

I'm running a 3.10 kernel with LVM 2.02.95.

I'm running into a problem where activating snapshots can take quite a long time, roughly one minute per 25GB of delta between the snapshot and the origin volume. (See below for my test procedure.)

As a side question, currently we have a script which runs "vgchange -ay <vg>", which will activate the VG.

Quoted 9 lines

This script appears to activate the LVs within the VG serially rather than in parallel. Does it do any processing to determine in which order to activate the LVs?

The reason I ask is that we have an overall timeout on the activate phase, after which we'll put any un-activated LVs into an error state. It would be a shame to try to activate a snapshotted LV first, and then timeout without activating any of the non-snapshotted LVs.

Hi

Sorry, but there is no 'parallel' activation from lvm2 command - as lvm2 is not multithreadded app (and is quite far away from that).

What you can do with recent versions of lvm2 - you could 'mark' (with lvchange --setskipactivation) some LVs to be skipped from normal activation. Then you could activate such skipped LVs with 'lvchange -ay -K'.

As said lvm2 has not been designed for this case.
(btw during whole activation - your access to VG is actually limited only for 'read-only')

Zdenek

#9 lore raw
Quoted 9 lines · Zdenek Kabelac

Dne 16.9.2015 v 01:03 Chris Friesen napsal(a):

Quoted 6 lines

As a side question, currently we have a script which runs "vgchange -ay <vg>", which will activate the VG.

This script appears to activate the LVs within the VG serially rather than in parallel. Does it do any processing to determine in which order to activate the LVs?

Quoted 7 lines

What you can do with recent versions of lvm2 - you could 'mark' (with lvchange --setskipactivation) some LVs to be skipped from normal activation. Then you could activate such skipped LVs with 'lvchange -ay -K'.

As said lvm2 has not been designed for this case.
(btw during whole activation - your access to VG is actually limited only for 'read-only')

Right...we're planning on testing out thin provisioning but I have to figure out a band-aid solution for products already in use.

Just to clarify...what does your "access to VG is actually limited only for 'read-only'" mean exactly?

Does it mean:

1) I only have read-only access to the LVs within the VG until all LVs in the VG are activated?

or

2) I only have readonly access to the VG (so I can't add/remove/modify LVs) but I have read/write access to the LVs within the VG.

Thanks,
Chris

#11 lore raw
In reply to Zdenek Kabelac
Quoted 2 lines · Zdenek Kabelac

Sorry, but there is no 'parallel' activation from lvm2 command - as lvm2 is not multithreadded app (and is quite far away from that).

Just had a weird thought. What would happen if I ran the "vgchange -ay" command, and then a few seconds later ran a bunch of "lvchange -ay" commands in parallel?

Would that maybe let me activate non-snapshotted volumes even if the snapshotted volumes took a long time?

Chris

#12 lore raw

Dne 16.9.2015 v 21:19 Chris Friesen napsal(a):

Quoted 9 lines · Zdenek Kabelac
Quoted 3 lines

Sorry, but there is no 'parallel' activation from lvm2 command - as lvm2 is not multithreadded app (and is quite far away from that).

Just had a weird thought. What would happen if I ran the "vgchange -ay" command, and then a few seconds later ran a bunch of "lvchange -ay" commands in parallel?

Since all of them are 'activation' requests - you just end with success and everything should be activated.

Much more interesting case is - if you run in parallel activation and deactivation :) - then the result depends on last executed command.

Quoted 2 lines

Would that maybe let me activate non-snapshotted volumes even if the snapshotted volumes took a long time?

Yes - you could activate things in parallel this way - since every command has separate udev cookie - so it waits on its set of LVs.

Zdenek

#10 lore raw
In reply to Chris Friesen

Dne 16.9.2015 v 17:30 Chris Friesen napsal(a):

Quoted 26 lines · Zdenek Kabelac
Quoted 9 lines

Dne 16.9.2015 v 01:03 Chris Friesen napsal(a):

Quoted 6 lines

As a side question, currently we have a script which runs "vgchange -ay <vg>", which will activate the VG.

This script appears to activate the LVs within the VG serially rather than in parallel. Does it do any processing to determine in which order to activate the LVs?

Quoted 7 lines

What you can do with recent versions of lvm2 - you could 'mark' (with lvchange --setskipactivation) some LVs to be skipped from normal activation. Then you could activate such skipped LVs with 'lvchange -ay -K'.

As said lvm2 has not been designed for this case.
(btw during whole activation - your access to VG is actually limited only for 'read-only')

Right...we're planning on testing out thin provisioning but I have to figure out a band-aid solution for products already in use.

Just to clarify...what does your "access to VG is actually limited only for 'read-only'" mean exactly?

Does it mean:

Quoted 7 lines

1) I only have read-only access to the LVs within the VG until all LVs in the VG are activated?

or

2) I only have readonly access to the VG (so I can't add/remove/modify LVs) but I have read/write access to the LVs within the VG.

2. is correct - you will not be able to modify VG content (as read-lock is held for VG - and you need write-lock for updates)

Zdenek