Sparse LVs, --virtualsize equal to --size
Hello,
I have been experimenting with LVM's sparse LVs feature. If I understand correctly, a sparse device of --virtualsize of V GBs and --size of S GBs is implemented as a snapshot over two devices:
a) a virtual device of V GBs returning all zero blocks, and
b) a COW device of length S GBs.
Reads from the sparse LV return all zeroes, writes allocate a new exception and store the newly written data inside the COW device.
If the COW device runs out of space, the snapshot is in trouble:
[11285507.534891] device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
For my use case, I don't care about the potential space savings of having S << V and still see a huge device of size V GB. I'd like to have S == V, since I only care about having a virtual device that returns all zeroes for yet-unwritten blocks.
Setting S == V causes the snapshot to die miserably when the virtual device is fully overwritten, because it seems the COW device runs out of space. The manpage for lvcreate(8) does state that:
"Note that a small amount of the space you allocate to the snapshot is used to track the locations of the chunks of data, so you should allocate slightly more space than you actually need and monitor the rate@which the snapshot data is growing so you can avoid running out of space."
So, what I'd like to know is:
a) What is the relation between S, V and the snapshot chunk size? How much is "slightly more", how larger should S be compared to V, to ensure the COW device *never* runs out of space?
b) How are blocks allocated inside the COW device when establishing mappings? My guess is they are allocated sequentially, but in my case, when S == V, I don't really need a full mapping. All LVM has to do, is remember whether a chunk has already been written or not, it doesn't need to translate the chunk number.
Not having to do any translation of chunk numbers from snapshot to COW would be great, because the I/O patterns on the COW device would follow the I/O patterns of the sparse device.
Is there any way I can do that, only exploit the tracking of writes, without messing up the ordering of chunks between the virtual and COW devices?
Thank you,
Vangelis.
Signature · 9 lines
Vangelis Koukis [email protected] OpenPGP public key ID: pub 1024D/1D038E97 2003-07-13 Vangelis Koukis <[email protected]> Key fingerprint = C5CD E02E 2C78 7C10 8A00 53D8 FBFC 3799 1D03 8E97 Only those who will risk going too far can possibly find out how far one can go. -- T.S. Eliot
If you've got the capacity to test, create one that's quite a bit larger and see how much space it actually uses up when full, then use that size in future. There's a header, and then the blocks are written sequentially with an index block inserted ahead of each group of data blocks.
Or try out thin LVs available in recent LVM releases/kernels for an alternative 'sparse' approach.
Alasdair
Long ago, Nostradamus foresaw that on 01/24/2013 01:08 PM, Alasdair G Kergon would write:
Quoted 8 lines
If you've got the capacity to test, create one that's quite a bit larger and see how much space it actually uses up when full, then use that size in future. There's a header, and then the blocks are written sequentially with an index block inserted ahead of each group of data blocks.
Or try out thin LVs available in recent LVM releases/kernels for an alternative 'sparse' approach.
He doesn't actually want sparse, he wants "instant zero" via copy on write. This is useful for virtual machines by preventing information
leaks between customers. Currently, you can create a "template" LV,
and have a snapshot for each VM. Since you (almost) never write to the
template, you don't run into the performance issue. He proposed a new
type of LV that is a normal LV plus a small bitmap (1 bit per chunk) that tracks which chunks have been written to. When a chunk is written to, the remainder of that chunk is set to zero. Reads from unwritten chunks always return zeroes.
So look at thin provisioning with its zeroing option. External origin. (support currently being added to lvm)
Or this not-yet-upstream target:
http://people.redhat.com/agk/patches/linux/editing/dm-add-zeroed-target.patch
Alasdair
Quoted 7 lines · Alasdair G Kergon
So look at thin provisioning with its zeroing option. External origin. (support currently being added to lvm)
Or this not-yet-upstream target:
http://people.redhat.com/agk/patches/linux/editing/dm-add-zeroed-target.patchAlasdair
Thanks Alasdair,
this seems to fit the bill perfectly, it's a shame it's not yet merged upstream.
Until then, if we are to go with the "snapshot-over-the-zero-target" route, can you comment on quantifying the space overhead of tracking chunks in the snapshot?
Thanks again,
Vangelis.
Signature · 9 lines
Vangelis Koukis [email protected] OpenPGP public key ID: pub 1024D/1D038E97 2003-07-13 Vangelis Koukis <[email protected]> Key fingerprint = C5CD E02E 2C78 7C10 8A00 53D8 FBFC 3799 1D03 8E97 Only those who will risk going too far can possibly find out how far one can go. -- T.S. Eliot
Quoted 2 lines · Vangelis Koukis
this seems to fit the bill perfectly, it's a shame it's not yet merged upstream.
One of the reasons the zeroed target is not upstream is because, although we thought it would be useful, we had nobody actually asking to test and use it.
If we bring it up-to-date and prepare to merge it upstream, would you test it?
Alasdair
Quoted 4 lines · Alasdair G Kergon
Quoted 2 lines · Vangelis Koukis
this seems to fit the bill perfectly, it's a shame it's not yet merged upstream.
Quoted 8 lines
One of the reasons the zeroed target is not upstream is because, although we thought it would be useful, we had nobody actually asking to test and use it.
If we bring it up-to-date and prepare to merge it upstream, would you test it?
Alasdair
Dear Alasdair,
this functionality is really useful, especially in VM provisioning scenarios.
The need for this feature came through our work on the Synnefo cloud software [1].
We were already planning to contribute code to Ganeti [2], so it can allocate sparse LVs based on the snapshot feature. If the zeroed DM target makes it upstream, we'll implement support for it inside Ganeti and also propose it for inclusion upstream.
Then I won't just be able to test it manually, we'll be running Ganeti burnin against it, and the number of potential testers will include users currently running Ganeti and Synnefo in production.
Thanks,
Vangelis.
[1] http://www.synnefo.org
[2] http://code.google.com/p/ganeti
Signature · 9 lines
Vangelis Koukis [email protected] OpenPGP public key ID: pub 1024D/1D038E97 2003-07-13 Vangelis Koukis <[email protected]> Key fingerprint = C5CD E02E 2C78 7C10 8A00 53D8 FBFC 3799 1D03 8E97 Only those who will risk going too far can possibly find out how far one can go. -- T.S. Eliot
Quoted 9 lines · Vangelis Koukis
Quoted 7 lines · Alasdair G Kergon
So look at thin provisioning with its zeroing option. External origin. (support currently being added to lvm)
Or this not-yet-upstream target:
http://people.redhat.com/agk/patches/linux/editing/dm-add-zeroed-target.patchAlasdair
Quoted 8 lines
Thanks Alasdair,
this seems to fit the bill perfectly, it's a shame it's not yet merged upstream.
Until then, if we are to go with the "snapshot-over-the-zero-target" route, can you comment on quantifying the space overhead of tracking chunks in the snapshot?
Beware! Large old-style snapshots may take a very long time to activate[1] (reportedly up to few hours) and my guess is many smaller snapshots will behave the same[2], the total amount of chunks written to all snapshots being the key to slow start...
You should definitely try it with your workflow before going too far that direction.
[1]Just search for 'lvm large snapshot'. Someone here may be able to point you to a more complete data.
[2] Not even trying to think about many larger ones as your usecase suggests: just update kernel and java few times and with journal FS you will soon be at few gigabytes per VM.
-- Marian
Quoted 12 lines
Thanks again,
Vangelis._______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 10 lines · Marian Csontos
Quoted 9 lines · Vangelis Koukis
Quoted 7 lines · Alasdair G Kergon
So look at thin provisioning with its zeroing option. External origin. (support currently being added to lvm)
Or this not-yet-upstream target:
http://people.redhat.com/agk/patches/linux/editing/dm-add-zeroed-target.patchAlasdair
Quoted 9 lines
Quoted 8 lines
Thanks Alasdair,
this seems to fit the bill perfectly, it's a shame it's not yet merged upstream.
Until then, if we are to go with the "snapshot-over-the-zero-target" route, can you comment on quantifying the space overhead of tracking chunks in the snapshot?
Hello Marian,
thank you for your answer, here are some points I'm not sure I have understood completely:
Quoted 5 lines
Beware! Large old-style snapshots may take a very long time to activate[1] (reportedly up to few hours) and my guess is many smaller snapshots will behave the same[2], the total amount of chunks written to all snapshots being the key to slow start...
What is old-style snapshots? Old-style compared to what, thin LVs? By "activate", do you refer to the problem of very slow VG activation e.g., on boot, as described here?
http://unix.stackexchange.com/questions/20152/why-does-activation-of-lvm-partition-with-large-snapshot-takes-long-time
https://bugs.launchpad.net/lvm2/+bug/360237
If yes, then the question still remains:
Can you please comment on the exact on-disk format used when doing LVM snapshots? What is the exact format of the blocks being written to the COW volume?
I think this would make clear both a) what the space overhead per snapshot chunk is, b) why it takes so long to activate an LVM VG if the snapshot has grown too large (i.e., if too much information has found its way into the COW volume).
Quoted 10 lines
You should definitely try it with your workflow before going too far that direction.
[1]Just search for 'lvm large snapshot'. Someone here may be able to point you to a more complete data.
[2] Not even trying to think about many larger ones as your usecase
suggests: just update kernel and java few times and with journal FS you will soon be at few gigabytes per VM.
I understand this may well be the case, but I'd like to know the *reason* for the symptom you describe.
Could you point me to information on the exact way snapshots are implemented, a design doc, a specification? Is the process specified only by the code that implements it?
Thanks again,
Vangelis.
Signature · 9 lines
Vangelis Koukis [email protected] OpenPGP public key ID: pub 1024D/1D038E97 2003-07-13 Vangelis Koukis <[email protected]> Key fingerprint = C5CD E02E 2C78 7C10 8A00 53D8 FBFC 3799 1D03 8E97 Only those who will risk going too far can possibly find out how far one can go. -- T.S. Eliot
Quoted 21 lines · Vangelis Koukis
Quoted 10 lines · Marian Csontos
Quoted 9 lines · Vangelis Koukis
Quoted 7 lines · Alasdair G Kergon
So look at thin provisioning with its zeroing option. External origin. (support currently being added to lvm)
Or this not-yet-upstream target:
http://people.redhat.com/agk/patches/linux/editing/dm-add-zeroed-target.patchAlasdair
Quoted 9 lines
Quoted 8 lines
Thanks Alasdair,
this seems to fit the bill perfectly, it's a shame it's not yet merged upstream.
Until then, if we are to go with the "snapshot-over-the-zero-target" route, can you comment on quantifying the space overhead of tracking chunks in the snapshot?
Quoted 7 lines
Hello Marian,
thank you for your answer, here are some points I'm not sure I have understood completely:
Quoted 2 lines
Beware! Large old-style snapshots may take a very long time to activate[1] (reportedly up to few hours)
BTW,
I've seen activation of "thin" snapshots take "ages" as well. Maybe not "few" hours, but close.
That's because the implicitly started thin_check is 100% cpu on a single core, we have 64k "chunk size",
and thus the tree is *huge*.
Also lvremove of a thin snapshot in that setup is single core cpu bound, takes >= 20 minutes or so, locks out any other lvm command for that time.
That's with
LVM version: 2.02.95(2)-RHEL6 (2012-10-16)
Library version: 1.02.74-RHEL6 (2012-10-16) Driver version: 4.22.6
[ "tech preview", so that may still improve ;-) ]
Back on topic:
Quoted 6 lines
Quoted 4 lines
and my guess is many
smaller snapshots will behave the same[2], the total amount of chunks written to all snapshots being the key to slow start...What is old-style snapshots? Old-style compared to what, thin LVs?
Yes.
Quoted 1 line
By "activate", do you refer to the problem of very slow VG activation
Yes.
Quoted 5 lines
If yes, then the question still remains:
Can you please comment on the exact on-disk format used when doing LVM snapshots? What is the exact format of the blocks being written to the COW volume?
I'm pasting parts of an older post to this list
(from 2008, Restore LVM snapshot without creating a full dump to an "external" device?)
"old style snapsthots", aka
dm-snap and dm-exception-store are implemented in a way that for a single snapshot, you get
(mapping only) snapshot-origin
(real storage) origin-real
(mapping only) snapshot
(real storage) COW (or exception store)
COW on disk format is pretty simple (as of now).
its all fixed size chunks.
it starts with a 4x32bit header,
[SnAp][valid][version][chunk_size in sectors]
so any valid snapshot looks
"SnAp" 1 1 [power of two]
chunk_size it what you set with the lvcreate "-c" option.
the rest of the (just as well chunk_size'ed) header block is unused.
expressed in chunks, the COW storage looks like:
[header chunk][exception area 0][data chunks][....][exception area 1][...] where each exception area is one "chunk" itself.
each exception area holds a mapping table of
"logic chunk number" to "in COW storage chunk number", both 64bit. "logic number" is called "old", "in COW" address is called "new". byte number
1 [old][new] 2 [old][new] 3 ... (chunk_size*512/16) [old][new]
following are as many data chunks.
this whole thing is append only.
On activation,
it needs to scan all those [exception area ...] blocks, until it find the "terminating" zeroed one.
It reads in and stores this mapping in core memory.
Hope that helps,
Signature · 3 lines
: Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com
Quoted 43 lines · Lars Ellenberg
Quoted 1 line
What is old-style snapshots? Old-style compared to what, thin LVs?
Yes.
Quoted 1 line
By "activate", do you refer to the problem of very slow VG activation
Yes.
Quoted 5 lines
If yes, then the question still remains:
Can you please comment on the exact on-disk format used when doing LVM snapshots? What is the exact format of the blocks being written to the COW volume?
I'm pasting parts of an older post to this list
(from 2008, Restore LVM snapshot without creating a full dump to an "external" device?)"old style snapsthots", aka
dm-snap and dm-exception-store are implemented in a way that for a single snapshot, you get
(mapping only) snapshot-origin
(real storage) origin-real
(mapping only) snapshot
(real storage) COW (or exception store)COW on disk format is pretty simple (as of now).
its all fixed size chunks.
it starts with a 4x32bit header,
[SnAp][valid][version][chunk_size in sectors]
so any valid snapshot looks
"SnAp" 1 1 [power of two]
chunk_sizeit what you set with the lvcreate "-c" option.the rest of the (just as well chunk_size'ed) header block is unused.
expressed in chunks, the COW storage looks like:
[header chunk][exception area 0][data chunks][....][exception area 1][...] where each exception area is one "chunk" itself.
each exception area holds a mapping table of
"logic chunk number" to "in COW storage chunk number", both 64bit. "logic number" is called "old", "in COW" address is called "new". byte number
Quoted 8 lines
1 [old][new] 2 [old][new] 3 ... (chunk_size*512/16) [old][new]following are as many data chunks.
this whole thing is append only.
Dear Lars,
Thank you very much for the detailed explanation!
This is exactly what I needed, thanks for your time to gather everything into a single answer.
Quoted 6 lines
On activation,
it needs to scan all those [exception area ...] blocks, until it find the "terminating" zeroed one.
It reads in and stores this mapping in core memory.
Yes, so there is no fixed index or bitmap on disk to determine whether a block is in an exception list or not. LVM has to scan all blocks, and build this structure in main memory, on activation time.
This makes old-style snapshots completely unsuitable for our use case. I guess dm-zeroed is what we should go for, from the available options.
Quoted 2 lines
Hope that helps,
It was great help.
Thanks again,
Vangelis.
Signature · 9 lines
Vangelis Koukis [email protected] OpenPGP public key ID: pub 1024D/1D038E97 2003-07-13 Vangelis Koukis <[email protected]> Key fingerprint = C5CD E02E 2C78 7C10 8A00 53D8 FBFC 3799 1D03 8E97 Only those who will risk going too far can possibly find out how far one can go. -- T.S. Eliot
Quoted 10 lines · Stuart D Gathman
Long ago, Nostradamus foresaw that on 01/24/2013 01:08 PM, Alasdair G Kergon would write:
Quoted 8 lines
If you've got the capacity to test, create one that's quite a bit larger and see how much space it actually uses up when full, then use that size in future. There's a header, and then the blocks are written sequentially with an index block inserted ahead of each group of data blocks.
Or try out thin LVs available in recent LVM releases/kernels for an alternative 'sparse' approach.
Quoted 10 lines
He doesn't actually want sparse, he wants "instant zero" via copy on write. This is useful for virtual machines by preventing information
leaks between customers. Currently, you can create a "template" LV,
and have a snapshot for each VM. Since you (almost) never write to the
template, you don't run into the performance issue. He proposed a new
type of LV that is a normal LV plus a small bitmap (1 bit per chunk) that tracks which chunks have been written to. When a chunk is written to, the remainder of that chunk is set to zero. Reads from unwritten chunks always return zeroes.
Spot on!
"Guaranteed initialized to zero" is what I'm looking for. So the LVM snapshot mechanism would work, but it may be a bit of an overkill.
I'd rather not do it in a trial-and-error fashion, trying different values of S just a bit larger than V, until it works. How large is the overhead per chunk? Ideally, we could quantify the overhead as a fraction of the chunk size, and use it as a safe margin for estimating the needed physical size.
Thanks again,
Vangelis.
Signature · 9 lines
Vangelis Koukis [email protected] OpenPGP public key ID: pub 1024D/1D038E97 2003-07-13 Vangelis Koukis <[email protected]> Key fingerprint = C5CD E02E 2C78 7C10 8A00 53D8 FBFC 3799 1D03 8E97 Only those who will risk going too far can possibly find out how far one can go. -- T.S. Eliot