dm-cache: block level disk cache target for device mapper
Subsystems (2)
RFC PATCH 1/1 dm-cache: block level disk cache target for device mapper
This is the first cut of a device-mapper target which provides a write-back or write-through block cache. It is intended to be used in conjunction with remote block devices such as iSCSI or ATA-over-Ethernet, particularly in cluster situations.
In performance tests with iSCSI, gave peformance improvements of 2-10x that of iSCSI alone when Postmark or Bonnie loads were applied from 8 clients to a single server. Evidence suggests even greater differences on larger clusters. A detailed performance analysis will be vailable shortly via a technical report on IBM's CyberDigest.
This module was developed during an intership at IBM Research by Ming Zhao. Please direct comments to both Ming and myself.
--- drivers/md/Kconfig | 6 drivers/md/Makefile | 1 drivers/md/dm-cache.c | 1465 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 1472 insertions(+), 0 deletions(-)
drivers/md/Kconfig +6 -0
@@ -261,6 +261,12 @@ config DM_MULTIPATH_EMC ---help--- Multipath support for EMC CX/AX series hardware. +config DM_CACHE + tristate "Cache target support (EXPERIMENTAL)" + depends on BLK_DEV_DM && EXPERIMENTAL + ---help--- + Support for generic cache target for device-mapper. + endmenu endif
drivers/md/Makefile +1 -0
@@ -36,6 +36,7 @@ obj-$(CONFIG_DM_MULTIPATH_EMC) += dm-emc obj-$(CONFIG_DM_SNAPSHOT) += dm-snapshot.o obj-$(CONFIG_DM_MIRROR) += dm-mirror.o obj-$(CONFIG_DM_ZERO) += dm-zero.o +obj-$(CONFIG_DM_CACHE) += dm-cache.o quiet_cmd_unroll = UNROLL $@ cmd_unroll = $(PERL) $(srctree)/$(src)/unroll.pl $(UNROLL) \
drivers/md/dm-cache.c new file +1465 -0
1,465 lines not on this page · raw message
Signature · 1 line
1.4.1
Quoted 4 lines · Eric Van Hensbergen
This is the first cut of a device-mapper target which provides a write-back or write-through block cache. It is intended to be used in conjunction with remote block devices such as iSCSI or ATA-over-Ethernet, particularly in cluster situations.
How does this work in practice? In other words, what is a typical actual configuration?
There is a remote block device, and a local one, and these are kept into sync in some way?
Thanks.
Signature · 2 lines
http://www.PowerDNS.com Open source, database driven DNS Software http://netherlabs.nl Open and Closed source services
Quoted 6 lines · bert hubert
Quoted 4 lines · Eric Van Hensbergen
This is the first cut of a device-mapper target which provides a write-back or write-through block cache. It is intended to be used in conjunction with remote block devices such as iSCSI or ATA-over-Ethernet, particularly in cluster situations.
Quoted 6 lines
How does this work in practice? In other words, what is a typical actual configuration?
There is a remote block device, and a local one, and these are kept into sync in some way?
That's the basic idea. In our testbed, we had a single iSCSI server exporting block devices to several clients -- each maintaining their own local disk cache of the server exported block devices. You can configured either write-through or write-back policies -- write-back has better performance, but somewhat obvious consistency issues in failure cases.
The original intent was to combine this with the dm-cow target (which I posted a few hours before the dm-cache patch) to provide a scalable cluster deployment system based on back-end iSCSI or ATA-over-Ethernet storage.
-eric
Quoted 21 lines · Eric Van Hensbergen
Quoted 6 lines · bert hubert
Quoted 4 lines · Eric Van Hensbergen
This is the first cut of a device-mapper target which provides a write-back or write-through block cache. It is intended to be used in conjunction with remote block devices such as iSCSI or ATA-over-Ethernet, particularly in cluster situations.
Quoted 6 lines
How does this work in practice? In other words, what is a typical actual configuration?
There is a remote block device, and a local one, and these are kept into sync in some way?
That's the basic idea. In our testbed, we had a single iSCSI server exporting block devices to several clients -- each maintaining their own local disk cache of the server exported block devices. You can configured either write-through or write-back policies -- write-back has better performance, but somewhat obvious consistency issues in failure cases.
Quoted 4 lines
The original intent was to combine this with the dm-cow target (which I posted a few hours before the dm-cache patch) to provide a scalable cluster deployment system based on back-end iSCSI or ATA-over-Ethernet storage.
like to see this idea but any similarity with
http://www.ele.uri.edu/Research/hpcl/STICS/stics.pdf?
STICS is patent pending so not sure if kernel can be free to merge this dm-cache.
Quoted 7 lines
-eric
--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
Quoted 5 lines · Ming Zhang
like to see this idea but any similarity with
http://www.ele.uri.edu/Research/hpcl/STICS/stics.pdf?STICS is patent pending so not sure if kernel can be free to merge this dm-cache.
I like the idea of STICS, an efficient bridge between SCSI and IP. But I think its only similarity with dm-cache is the use of nonvolatile caching.
Dm-cache is a generic block-level disk cache that supports a variety of SAN technologies and is not tied to any particular one.
Best regards,
Ming Zhao
Quoted 7 lines · Ming Zhao
Quoted 5 lines · Ming Zhang
like to see this idea but any similarity with
http://www.ele.uri.edu/Research/hpcl/STICS/stics.pdf?STICS is patent pending so not sure if kernel can be free to merge this dm-cache.
Quoted 7 lines
I like the idea of STICS, an efficient bridge between SCSI and IP. But I think its only similarity with dm-cache is the use of nonvolatile caching.
Dm-cache is a generic block-level disk cache that supports a variety of SAN technologies and is not tied to any particular one.
great. as long as you are aware of this and can find the difference, then fine.
Quoted 3 lines
Best regards,
Ming Zhao
Quoted 10 lines · Eric Van Hensbergen
This is the first cut of a device-mapper target which provides a write-back or write-through block cache. It is intended to be used in conjunction with remote block devices such as iSCSI or ATA-over-Ethernet, particularly in cluster situations.
In performance tests with iSCSI, gave peformance improvements of 2-10x that of iSCSI alone when Postmark or Bonnie loads were applied from 8 clients to a single server. Evidence suggests even greater differences on larger clusters. A detailed performance analysis will be vailable shortly via a technical report on IBM's CyberDigest.
Can you give an idea how this relates to the normal block device buffers?
If this is intended to speed up remote disks, is it possible that the cache content can be paged out on local disks in low-mem situations?
Best,
Jens
Quoted 5 lines · Jens Wilke
On Monday 27 November 2006 19:26, Eric Van Hensbergen wrote:
If this is intended to speed up remote disks, is it possible that the cache content can be paged out on local disks in low-mem situations?
The main intent was to use local disks as cache to offload centralized remote disks. The logic was that most systems have local disks, if only for swap -- so why not use them as a cache to help offload centralized storage. While the in-memory page cache works perfectly fine in certain situations -- we were dealing with workloads in which the in-memory page-cache wasn't sufficient to hold all the data.
There are also some additional possibilities we've thought through and have been playing with including allowing the local disk cache to be persistent across reboots (with varying validation schemes).
-eric
Quoted 7 lines · Eric Van Hensbergen
Quoted 5 lines · Jens Wilke
On Monday 27 November 2006 19:26, Eric Van Hensbergen wrote:
If this is intended to speed up remote disks, is it possible that the cache content can be paged out on local disks in low-mem situations?
Quoted 6 lines
The main intent was to use local disks as cache to offload centralized remote disks. The logic was that most systems have local disks, if only for swap -- so why not use them as a cache to help offload centralized storage. While the in-memory page cache works perfectly fine in certain situations -- we were dealing with workloads in which the in-memory page-cache wasn't sufficient to hold all the data.
I derived from the code that the cache is actually another block device and not memory. Maybe you should give a little more description and a sample 'dmsetup create' statement.
Quoted 3 lines
There are also some additional possibilities we've thought through and have been playing with including allowing the local disk cache to be persistent across reboots (with varying validation schemes).
Yes, I like the idea. It would be also possible to use NV-Ram as write back cache, for example in notebooks, to avoid disk spin-ups or to improve transaction performance in enterprise applications.
Maybe it would also make sense to prefill the cache contents on startup with a bulk I/O request?
Comments on the code:
- The data of the write back cache should be flushed on suspend and not in the destructor.
- You don't keep track of I/O on the fly to the cache that is mapped directly in cache_hit(). How do you make sure that this I/O is completed before you replace a cache block?
- The cache block index is hashed, this means the cache data blocks are not clustered. I don't think you can solve this problem with a proper hash function. Perhaps you should consider a (B-)Tree structure for that.
Best,
Jens
Quoted 4 lines · Jens Wilke
- You don't keep track of I/O on the fly to the cache that is mapped directly in
cache_hit(). How do you make sure that this I/O is completed before you replace a cache block?
The previous I/O from cache hit and the later I/O for cache replacement should be queued in order on the cache block device - is this a safe assumption?
- The cache block index is hashed, this means the cache data blocks are not
Quoted 3 lines
clustered. I don't think you can solve this problem with a proper hash function.
Perhaps you should consider a (B-)Tree structure for that.
The current hash algorithm does clustering to some extent by mapping consecutive blocks from the source device to consecutive blocks on the cache device. But I agree that more sophisticated algorithms can be studied.
Thanks a lot for the suggestions. I will try to incorporate them in the next patch.
- Ming
Quoted 3 lines · Jens Wilke
- You don't keep track of I/O on the fly to the cache that is mapped directly in
cache_hit(). How do you make sure that this I/O is completed before you replace a cache block?
The previous I/O from cache hit and the later I/O for cache replacement should be queued in order on the cache block device - is this a safe assumption?
Quoted 3 lines
- The cache block index is hashed, this means the cache data blocks are not clustered. I don't think you can solve this problem with a proper hash function. Perhaps you should consider a (B-)Tree structure for that.
The current hash algorithm does clustering to some extent by mapping consecutive blocks from the source device to consecutive blocks on the cache device. But I agree that more sophisticated algorithms can be studied.
Thanks a lot for the suggestions. I will try to incorporate them in the next patch.
- Ming
Quoted 8 lines · Ming Zhao
Quoted 3 lines · Jens Wilke
- You don't keep track of I/O on the fly to the cache that is mapped directly in
cache_hit(). How do you make sure that this I/O is completed before you replace a cache block?The previous I/O from cache hit and the later I/O for cache replacement should be queued in order on the cache block device - is this a safe assumption?
I don't think you can assume that the request processing in the host is ordered without sychronization. Everything you queue in your worker thread is serialized by that. Everything else is out of your control and each thread may be scheduled or not.
Maybe we have certain guarantees through DM or the block I/O layer, that I don't know. Can somebody else step in here?
Best,
Jens
Quoted 5 lines · Eric Van Hensbergen
This is the first cut of a device-mapper target which provides a write-back or write-through block cache. It is intended to be used in conjunction with remote block devices such as iSCSI or ATA-over-Ethernet, particularly in cluster situations.
Quoted 8 lines
In performance tests with iSCSI, gave peformance improvements of 2-10x that of iSCSI alone when Postmark or Bonnie loads were applied from 8 clients to a single server. Evidence suggests even greater differences on larger clusters. A detailed performance analysis will be vailable shortly via a technical report on IBM's CyberDigest.
This module was developed during an intership at IBM Research by Ming Zhao. Please direct comments to both Ming and myself.
I don't quite understand the need for this. Do you need this module because your iscsi target doesn't use the pagecache on the server?
Quoted 5 lines · Eric Van Hensbergen
This is the first cut of a device-mapper target which provides a write-back or write-through block cache. It is intended to be used in conjunction with remote block devices such as iSCSI or ATA-over-Ethernet, particularly in cluster situations.
The technical paper describing our motivations and some performance results has finally made it through IBM's clearance process. It is available here:
-eric
Hrm,
I finally got onto one of my TODO items a while ago which is an HSM (IOW: cache) target for device-mapper ;)
The design of my HSM target covers remote storage as well as arbitrary slow local storage hierachies.
Let's compare our code and see if there's any mutual stuff to merge or pros and cons of the different approaches when I post something shortly.
Heinz
Quoted 12 lines · Eric Van Hensbergen
Quoted 6 lines · Eric Van Hensbergen
This is the first cut of a device-mapper target which provides a write-back or write-through block cache. It is intended to be used in conjunction with
remote block devices such as iSCSI or ATA-over-Ethernet, particularly in cluster situations.The technical paper describing our motivations and some performance results has finally made it through IBM's clearance process. It is available here:
Quoted 8 lines
-eric
--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Heinz Mauelshagen Red Hat GmbH
Consulting Development Engineer Am Sonnenhang 11
Storage Development 56242 Marienrachdorf
Germany
[email protected] PHONE +49 171 7803392
FAX +49 2626 924446=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-