Looking ahead - tiering with LVM?
Hi all
I just wonder how it could be possible some day, some year, to make lvm use tiering. I guess this has been debated numerous times before and I found this lvmts project, but it hasn't been updated for eight years or so.
To detail, and I apologize if this is known stuff. I just want to underline what I'm asking for here
We have lvmcache today and it works well. Warm data moves to SSDs or similar to cache what's hot. This is nice, but all writes will eventually go to the "slow" storage in the back. With tiering, such as that used by things like Dell Compellent and suchlike, all writes go to the fast storage, given there is room, or the logical volume is flagged for only to be used with slow storage. Then, in the night or whenever there's little traffic, a batch job runs and moves the cold-ish data from the fast(er) levels to a slow(er) level based on atime/mtime/something, per block, not file. I guess this will be analogous to extents in the terms of LVM (IIRC Compellent uses 8MB blocks, but that's not really relevant). This goes on, if some data on lower (slower) levels become popular for some reason, they are elevated to a higher (faster) level at runtime.
As far as I can understand (that is, without having read the code, it's beyond me) it seems like LVM has most of this already. Perhaps not timestamps per block (which will take up a wee bit of space and I/O), but still multi-PV storage in a single VG.
So - what would it really take to move LVM to something like a working tiered storage platform?
Kind regards
roy
Signature · 6 lines
Roy Sigurd Karlsbakk (+47) 98013356 http://blogg.karlsbakk.net/ GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt -- Hið góða skaltu í stein höggva, hið illa í snjó rita.
Il 2020-09-02 20:38 Roy Sigurd Karlsbakk ha scritto:
Quoted 6 lines
Hi all
I just wonder how it could be possible some day, some year, to make lvm use tiering. I guess this has been debated numerous times before and I found this lvmts project, but it hasn't been updated for eight years or so.
Hi, having developed and supported file-level form of tiered storage in response to a specific customer request, I have the feeling that tiered storage (both file and block based) is not so useful as it seems. Let me explain why I feel so...
The key difference between caching and tiering is that the former does not increase total available space, while the latter provides as much space as available in each storage tier. For example, 1 TB SSD + 10 TB HDD can be combined for a 11 TB tiered volume.
Tiering is useful when the faster volume provides a significant portion of the total aggregated sum - which is often not the case. In the example above, the SSD only provides a 10% space increase over plain caching. You can argue that one can simple enlarge the performane tier, for example using a 4 TB SSD + 10 TB HDD, but you are now in the ballpark of affording a full-SSD volume - ditching *both* tiering and caching.
That said, LVM already provides the basic building block to provide tiering as you can pvmove between block devices. The difficult thing is how to determine which block to move, and managing them in an automated way.
Regards.
Signature · 5 lines
Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: [email protected] - [email protected] GPG public key ID: FF5F32A8
Quoted 9 lines
Quoted 4 lines
I just wonder how it could be possible some day, some year, to make lvm use tiering. I guess this has been debated numerous times before and I found this lvmts project, but it hasn't been updated for eight years or so.
Hi, having developed and supported file-level form of tiered storage in response to a specific customer request, I have the feeling that tiered storage (both file and block based) is not so useful as it seems. Let me explain why I feel so...
First, filelevel is usually useless. Say you have 50 VMs with Windows server something. A lot of them are bound to have a ton of equal storage in the same areas, but the file size and content will vary over time. With blocklevel tiering, that could work better.
Quoted 4 lines
The key difference between caching and tiering is that the former does not increase total available space, while the latter provides as much space as available in each storage tier. For example, 1 TB SSD + 10 TB HDD can be combined for a 11 TB tiered volume.
This is all known.
Quoted 7 lines
Tiering is useful when the faster volume provides a significant portion of the total aggregated sum - which is often not the case. In the example above, the SSD only provides a 10% space increase over plain caching. You can argue that one can simple enlarge the performane tier, for example using a 4 TB SSD + 10 TB HDD, but you are now in the ballpark of affording a full-SSD volume - ditching *both* tiering and caching.
If you look at IOPS instead of just sequencial speed, you'll see the difference. A set of 10 drives in a RAID-6 will perhaps, maybe, give you 1kIOPS, while a single SSD might give you 50kIOPS or even more. This makes a huge impact.
Quoted 4 lines
That said, LVM already provides the basic building block to provide tiering as you can pvmove between block devices. The difficult thing is how to determine which block to move, and managing them in an automated way.
…which was the reason I asked this question, and which should be quite clear in the original post.
Vennlig hilsen
roy
Signature · 6 lines
Roy Sigurd Karlsbakk (+47) 98013356 http://blogg.karlsbakk.net/ GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt -- Hið góða skaltu í stein höggva, hið illa í snjó rita.
Il 2020-09-09 17:01 Roy Sigurd Karlsbakk ha scritto:
Quoted 4 lines
First, filelevel is usually useless. Say you have 50 VMs with Windows server something. A lot of them are bound to have a ton of equal storage in the same areas, but the file size and content will vary over time. With blocklevel tiering, that could work better.
It really depends on the use case. I applied it to a fileserver, so working at file level was the right choice. For VMs (or big files) it is useless, I agree.
Quoted 1 line
This is all known.
But the only reason to want tiering vs cache is the additional space the former provides. If this additional space is so small (compared to the combined, total volume space), tiering's advantage shrinks to (almost) nothing.
Quoted 4 lines
If you look at IOPS instead of just sequencial speed, you'll see the difference. A set of 10 drives in a RAID-6 will perhaps, maybe, give you 1kIOPS, while a single SSD might give you 50kIOPS or even more. This makes a huge impact.
IOPs are already well server by LVM cache. So, I genuinely ask: what would be tiering advantage here? I'll love to ear a reasonable use case.
Quoted 2 lines
…which was the reason I asked this question, and which should be quite clear in the original post.
Yeah, but this need a direct reply from a core LVM developer, which I wellcome ;) Thanks.
Signature · 5 lines
Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: [email protected] - [email protected] GPG public key ID: FF5F32A8
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Gionatan" == Gionatan Danti <[email protected]> writes:
Gionatan> Il 2020-09-09 17:01 Roy Sigurd Karlsbakk ha scritto:
Quoted 4 lines
Quoted 4 lines
First, filelevel is usually useless. Say you have 50 VMs with Windows server something. A lot of them are bound to have a ton of equal storage in the same areas, but the file size and content will vary over time. With blocklevel tiering, that could work better.
Gionatan> It really depends on the use case. I applied it to a Gionatan> fileserver, so working at file level was the right Gionatan> choice. For VMs (or big files) it is useless, I agree.
This assumes you're tiering whole files, not at the per-block level though, right?
Quoted 1 line
Quoted 1 line
This is all known.
Gionatan> But the only reason to want tiering vs cache is the Gionatan> additional space the former provides. If this additional Gionatan> space is so small (compared to the combined, total volume Gionatan> space), tiering's advantage shrinks to (almost) nothing.
Do you have numbers? I'm using DM_CACHE on my home NAS server box,
and it *does* seem to help, but only in certain cases. I've got a
750gb home directory LV with an 80gb lv_cache writethrough cache setup. So it's not great on write heavy loads, but it's good in read heavy ones, such as kernel compiles where it does make a difference.
So it's not only the caching being per-file or per-block, but how the actual cache is done? writeback is faster, but less reliable if you crash. Writethrough is slower, but much more reliable.
Dne 09. 09. 20 v 20:47 John Stoffel napsal(a):
Quoted 28 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Gionatan" == Gionatan Danti <[email protected]> writes:
Gionatan> Il 2020-09-09 17:01 Roy Sigurd Karlsbakk ha scritto:
Quoted 4 lines
Quoted 4 lines
First, filelevel is usually useless. Say you have 50 VMs with Windows server something. A lot of them are bound to have a ton of equal storage in the same areas, but the file size and content will vary over time. With blocklevel tiering, that could work better.
Gionatan> It really depends on the use case. I applied it to a Gionatan> fileserver, so working at file level was the right Gionatan> choice. For VMs (or big files) it is useless, I agree.
This assumes you're tiering whole files, not at the per-block level though, right?
Quoted 1 line
Quoted 1 line
This is all known.
Gionatan> But the only reason to want tiering vs cache is the Gionatan> additional space the former provides. If this additional Gionatan> space is so small (compared to the combined, total volume Gionatan> space), tiering's advantage shrinks to (almost) nothing.
Do you have numbers? I'm using
DM_CACHEon my home NAS server box,
and it *does* seem to help, but only in certain cases. I've got a
750gb home directory LV with an 80gblv_cachewritethrough cache setup. So it's not great on write heavy loads, but it's good in read heavy ones, such as kernel compiles where it does make a difference.
Quoted 3 lines
So it's not only the caching being per-file or per-block, but how the actual cache is done? writeback is faster, but less reliable if you crash. Writethrough is slower, but much more reliable.
Hi
dm-cache (--type cache) is hotspot cache (most used areas of device)
dm-writecache (--type writecache) is great with write-extensive load (somewhat extends your page cache on your NMVe/SSD/persistent-memory)
We were thinking about layering cached above each other - but so far there
was no big demand and also the complexity of solving problem is rising greatly - aka there is no problem to let users to stack cache on top of another cache on top of 3rd. cache - but what should have when it starts failing...
AFAIK there is no one yet writing driver for combining i.e. SSD + HDD
into a single drive which would be relocating blocks (so you get total size as aproximate sum of both devices) - but there is dm-zoned which solves somewhat similar problem - but I've no experience with that...
Zdenek
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Zdenek" == Zdenek Kabelac <[email protected]> writes:
Zdenek> Dne 09. 09. 20 v 20:47 John Stoffel napsal(a):
Quoted 2 lines
Quoted 2 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Gionatan" == Gionatan Danti <[email protected]> writes:
Gionatan> Il 2020-09-09 17:01 Roy Sigurd Karlsbakk ha scritto:
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Quoted 4 lines
First, filelevel is usually useless. Say you have 50 VMs with Windows server something. A lot of them are bound to have a ton of equal storage in the same areas, but the file size and content will vary over time. With blocklevel tiering, that could work better.
Gionatan> It really depends on the use case. I applied it to a Gionatan> fileserver, so working at file level was the right Gionatan> choice. For VMs (or big files) it is useless, I agree.
Quoted 6 lines
Quoted 6 lines
This assumes you're tiering whole files, not at the per-block level though, right?
Quoted 1 line
Quoted 1 line
This is all known.
Gionatan> But the only reason to want tiering vs cache is the Gionatan> additional space the former provides. If this additional Gionatan> space is so small (compared to the combined, total volume Gionatan> space), tiering's advantage shrinks to (almost) nothing.
Quoted 10 lines
Quoted 10 lines
Do you have numbers? I'm using
DM_CACHEon my home NAS server box,
and it *does* seem to help, but only in certain cases. I've got a
750gb home directory LV with an 80gblv_cachewritethrough cache setup. So it's not great on write heavy loads, but it's good in read heavy ones, such as kernel compiles where it does make a difference.So it's not only the caching being per-file or per-block, but how the actual cache is done? writeback is faster, but less reliable if you crash. Writethrough is slower, but much more reliable.
Zdenek> dm-cache (--type cache) is hotspot cache (most used areas of device)
I assume this is what I'm using on Debian Buster (10.5) right now? I use the crufty tool 'lvcache' to look at and manage my cache devices.
Zdenek> dm-writecache (--type writecache) is great with Zdenek> write-extensive load (somewhat extends your page cache on your Zdenek> NMVe/SSD/persistent-memory)
I don't think I'm using this at all:
sudo lvcache status data/home
+-----------------------+------------------+ | Field | Value | +-----------------------+------------------+ | cached | True | | size | 806380109824 | | cache_lv | home_cache | | cache_lv_size | 85899345920 | | metadata_lv | home_cache_cmeta | | metadata_lv_size | 83886080 | | cache_block_size | 192 | | cache_utilization | 873786/873813 | | cache_utilization_pct | 99.996910094 | | demotions | 43213 | | dirty | 0 | | end | 1574961152 | | features | 1 | | md_block_size | 8 | | md_utilization | 2604/20480 | | md_utilization_pct | 12.71484375 | | promotions | 43208 | | read_hits | 138697828 | | read_misses | 7874434 | | segment_type | cache | | start | 0 | | write_hits | 777455171 | | write_misses | 9841866 | +-----------------------+------------------+
Zdenek> We were thinking about layering cached above each other - but so far there Zdenek> was no big demand and also the complexity of solving problem is rising greatly Zdenek> - aka there is no problem to let users to stack cache on top of another cache Zdenek> on top of 3rd. cache - but what should have when it starts failing...
Zdenek> AFAIK there is no one yet writing driver for combining i.e. SSD + HDD Zdenek> into a single drive which would be relocating blocks (so you get total size as Zdenek> aproximate sum of both devices) - but there is dm-zoned which solves somewhat Zdenek> similar problem - but I've no experience with that...
Zdenek> Zdenek
Il 2020-09-09 20:47 John Stoffel ha scritto:
Quoted 2 lines
This assumes you're tiering whole files, not at the per-block level though, right?
The tiered approach I developed and maintained in the past, yes. For any LVM-based tiering, we are speaking about block-level tiering (as LVM itself has no "files" concept).
Quoted 5 lines
Do you have numbers? I'm using
DM_CACHEon my home NAS server box,
and it *does* seem to help, but only in certain cases. I've got a
750gb home directory LV with an 80gblv_cachewritethrough cache setup. So it's not great on write heavy loads, but it's good in read heavy ones, such as kernel compiles where it does make a difference.
Numbers for available space for tiering vs cache can vary based on your setup. However, storage tiers generally are at least 5-10X apart from each other (ie: 1 TB SSD for 10 TB HDD). Hence my gut fealing that tiering is not drastically better then lvm cache. But hey - I reserve the right to be totally wrong ;)
Quoted 3 lines
So it's not only the caching being per-file or per-block, but how the actual cache is done? writeback is faster, but less reliable if you crash. Writethrough is slower, but much more reliable.
writeback cache surely is more prone to failure vs writethoug cache. The golden rule is that writeback cache should use a mirrored device (with device-level powerloss protected writeback cache if sync write speed is important).
But this is somewhat ortogonal to the original question: block-level tiering itself increases the chances of data loss (ie: losing the SSD component will ruin the entire filesystem), so you should used mirrored (or parity) device for tiering also.
Regards.
Signature · 5 lines
Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: [email protected] - [email protected] GPG public key ID: FF5F32A8
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Gionatan" == Gionatan Danti <[email protected]> writes:
Gionatan> Il 2020-09-09 20:47 John Stoffel ha scritto:
Quoted 2 lines
Quoted 2 lines
This assumes you're tiering whole files, not at the per-block level though, right?
Gionatan> The tiered approach I developed and maintained in the past, yes. For any Gionatan> LVM-based tiering, we are speaking about block-level tiering (as LVM Gionatan> itself has no "files" concept).
Quoted 5 lines
Quoted 5 lines
Do you have numbers? I'm using
DM_CACHEon my home NAS server box,
and it *does* seem to help, but only in certain cases. I've got a
750gb home directory LV with an 80gblv_cachewritethrough cache setup. So it's not great on write heavy loads, but it's good in read heavy ones, such as kernel compiles where it does make a difference.
Gionatan> Numbers for available space for tiering vs cache can vary Gionatan> based on your setup. However, storage tiers generally are at Gionatan> least 5-10X apart from each other (ie: 1 TB SSD for 10 TB Gionatan> HDD). Hence my gut fealing that tiering is not drastically Gionatan> better then lvm cache. But hey - I reserve the right to be Gionatan> totally wrong ;)
Very true, numbers talk, annecdotes walk...
Quoted 3 lines
Quoted 3 lines
So it's not only the caching being per-file or per-block, but how the actual cache is done? writeback is faster, but less reliable if you crash. Writethrough is slower, but much more reliable.
Gionatan> writeback cache surely is more prone to failure vs Gionatan> writethoug cache. The golden rule is that writeback cache Gionatan> should use a mirrored device (with device-level powerloss Gionatan> protected writeback cache if sync write speed is important).
Even in my case I use mirrored SSDs for my cache LVs. It's the only sane thing to do IMHO.
Gionatan> But this is somewhat ortogonal to the original question:
Gionatan> block-level tiering itself increases the chances of data Gionatan> loss (ie: losing the SSD component will ruin the entire Gionatan> filesystem), so you should used mirrored (or parity) device Gionatan> for tiering also.
It does, you really need to have a solid setup in terms of hardware, with known failures modes you can handle, before you start trying to tier blocks.
Though maybe a write through block cache would be ok, since the cache would only be for reads, not writes. Which could help if you have a bunch of VMs (or containers, or whatevers) with alot of duplicated data that are all hitting the disk systems at once.
Il 2020-09-09 21:53 John Stoffel ha scritto:
Quoted 1 line
Very true, numbers talk, annecdotes walk...
Sure - lets try to gather some numbers from the data you posted before...
Quoted 8 lines
sudo lvcache status data/home
+-----------------------+------------------+ | Field | Value | +-----------------------+------------------+ | cached | True | | size | 806380109824 | | cache_lv | home_cache | | cache_lv_size | 85899345920 |
You cache device is squarely in the 10x ballpark (ie: it is ~9.39x smaller than your SSD). Having ~10.64% more space would be nice, but hardly game-changer.
Quoted 2 lines
| read_hits | 138697828 | | read_misses | 7874434 |
You have 94.6% hit rate from your hotspot cache - compare this to an ideally managed tiered storage, with its 100% (ideal, not reasonable in real world) hit rate. Does it really change anything?
Quoted 2 lines
| write_hits | 777455171 | | write_misses | 9841866 |
And you have an even better ~98.7% write hit ratio. As you have a mirrored cache device, you should be able to set LVM for using a writeback cache without risking your data in case of a single cache drive failure. I suspect this would do wonder with your hit ratio - but again, testing is the only method to be sure.
Thanks for sharing your data!
Regards.
Signature · 5 lines
Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: [email protected] - [email protected] GPG public key ID: FF5F32A8
Quoted 9 lines
Quoted 6 lines
First, filelevel is usually useless. Say you have 50 VMs with Windows server something. A lot of them are bound to have a ton of equal If you look at IOPS instead of just sequencial speed, you'll see the difference. A set of 10 drives in a RAID-6 will perhaps, maybe, give you 1kIOPS, while a single SSD might give you 50kIOPS or even more. This makes a huge impact.
IOPs are already well server by LVM cache. So, I genuinely ask: what would be tiering advantage here? I'll love to ear a reasonable use case.
LVMcache only helps if the cache is there in the first place and IIRC it's cleared after a reboot. It help won't that much over time with large storage. It also wastes space.
Vennlig hilsen
roy
Signature · 6 lines
Roy Sigurd Karlsbakk (+47) 98013356 http://blogg.karlsbakk.net/ GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt -- Hið góða skaltu í stein höggva, hið illa í snjó rita.
Il 2020-09-09 21:41 Roy Sigurd Karlsbakk ha scritto:
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
First, filelevel is usually useless. Say you have 50 VMs with Windows server something. A lot of them are bound to have a ton of equal If you look at IOPS instead of just sequencial speed, you'll see the difference. A set of 10 drives in a RAID-6 will perhaps, maybe, give you 1kIOPS, while a single SSD might give you 50kIOPS or even more. This makes a huge impact.
Quoted 5 lines
Quoted 2 lines
IOPs are already well server by LVM cache. So, I genuinely ask: what would be tiering advantage here? I'll love to ear a reasonable use case.
LVMcache only helps if the cache is there in the first place and IIRC it's cleared after a reboot.
I seem to remember that cache is persistent, but a writeback cache must be flushed to the underlying disk in case of unclean shutdown. This, however, should not empty the cache itself.
Quoted 1 line
It help won't that much over time with large storage. It also wastes space.
I tend to disagree: with large storage, you really want an hotspot cache vs a tiered approach unless:
a) the storage tiers are comparable in size, which is quite rare;
b) the slow storage does some sort of offline compression/deduplication, with the faster layer being a landing zone for newly ingested data.
Can you describe a reasonable real-world setup where plain LVM tiering would be useful? Again, this is a genuine question: I am interested in different storage setup than mine.
Thanks.
Signature · 5 lines
Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: [email protected] - [email protected] GPG public key ID: FF5F32A8