Skip to contentSkip to conversation

dm-cache: block level disk cache target for device mapper

RFC PATCH In review Eric Van Hensbergen 8 participants 15 messages Nov 27, 2006 – Dec 11, 2006

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 dm-cache: block level disk cache target for device mapper

This is the first cut of a device-mapper target which provides a write-back or write-through block cache. It is intended to be used in conjunction with remote block devices such as iSCSI or ATA-over-Ethernet, particularly in cluster situations.

In performance tests with iSCSI, gave peformance improvements of 2-10x that of iSCSI alone when Postmark or Bonnie loads were applied from 8 clients to a single server. Evidence suggests even greater differences on larger clusters. A detailed performance analysis will be vailable shortly via a technical report on IBM's CyberDigest.

This module was developed during an intership at IBM Research by Ming Zhao. Please direct comments to both Ming and myself.

---
 drivers/md/Kconfig    |    6 
 drivers/md/Makefile   |    1 
 drivers/md/dm-cache.c | 1465 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 1472 insertions(+), 0 deletions(-)
drivers/md/Kconfig +6 -0
index c92c152..0f23a15 100644
@@ -261,6 +261,12 @@ config DM_MULTIPATH_EMC
 	---help---
 	  Multipath support for EMC CX/AX series hardware.
 
+config DM_CACHE
+	tristate "Cache target support (EXPERIMENTAL)"
+	depends on BLK_DEV_DM && EXPERIMENTAL
+	---help---
+	  Support for generic cache target for device-mapper.
+
 endmenu
 
 endif
drivers/md/Makefile +1 -0
index 34957a6..49f7266 100644
@@ -36,6 +36,7 @@ obj-$(CONFIG_DM_MULTIPATH_EMC)	+= dm-emc
 obj-$(CONFIG_DM_SNAPSHOT)	+= dm-snapshot.o
 obj-$(CONFIG_DM_MIRROR)		+= dm-mirror.o
 obj-$(CONFIG_DM_ZERO)		+= dm-zero.o
+obj-$(CONFIG_DM_CACHE)		+= dm-cache.o
 
 quiet_cmd_unroll = UNROLL  $@
       cmd_unroll = $(PERL) $(srctree)/$(src)/unroll.pl $(UNROLL) \
drivers/md/dm-cache.c new file +1465 -0

1,465 lines not on this page · raw message

Signature · 1 line
1.4.1
#1 lore raw
Quoted 4 lines · Eric Van Hensbergen

This is the first cut of a device-mapper target which provides a write-back or write-through block cache. It is intended to be used in conjunction with remote block devices such as iSCSI or ATA-over-Ethernet, particularly in cluster situations.

How does this work in practice? In other words, what is a typical actual configuration?

There is a remote block device, and a local one, and these are kept into sync in some way?

Thanks.

Signature · 2 lines
http://www.PowerDNS.com      Open source, database driven DNS Software 
http://netherlabs.nl              Open and Closed source services
#2 lore raw
Quoted 6 lines · bert hubert
Quoted 4 lines · Eric Van Hensbergen

This is the first cut of a device-mapper target which provides a write-back or write-through block cache. It is intended to be used in conjunction with remote block devices such as iSCSI or ATA-over-Ethernet, particularly in cluster situations.

Quoted 6 lines

How does this work in practice? In other words, what is a typical actual configuration?

There is a remote block device, and a local one, and these are kept into sync in some way?

That's the basic idea. In our testbed, we had a single iSCSI server exporting block devices to several clients -- each maintaining their own local disk cache of the server exported block devices. You can configured either write-through or write-back policies -- write-back has better performance, but somewhat obvious consistency issues in failure cases.

The original intent was to combine this with the dm-cow target (which I posted a few hours before the dm-cache patch) to provide a scalable cluster deployment system based on back-end iSCSI or ATA-over-Ethernet storage.

-eric

#3 lore raw
Quoted 21 lines · Eric Van Hensbergen
Quoted 6 lines · bert hubert
Quoted 4 lines · Eric Van Hensbergen

This is the first cut of a device-mapper target which provides a write-back or write-through block cache. It is intended to be used in conjunction with remote block devices such as iSCSI or ATA-over-Ethernet, particularly in cluster situations.

Quoted 6 lines

How does this work in practice? In other words, what is a typical actual configuration?

There is a remote block device, and a local one, and these are kept into sync in some way?

That's the basic idea. In our testbed, we had a single iSCSI server exporting block devices to several clients -- each maintaining their own local disk cache of the server exported block devices. You can configured either write-through or write-back policies -- write-back has better performance, but somewhat obvious consistency issues in failure cases.

Quoted 4 lines

The original intent was to combine this with the dm-cow target (which I posted a few hours before the dm-cache patch) to provide a scalable cluster deployment system based on back-end iSCSI or ATA-over-Ethernet storage.

like to see this idea but any similarity with
http://www.ele.uri.edu/Research/hpcl/STICS/stics.pdf?

STICS is patent pending so not sure if kernel can be free to merge this dm-cache.

Quoted 7 lines

-eric

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel

#4 lore raw
Quoted 5 lines · Ming Zhang

like to see this idea but any similarity with
http://www.ele.uri.edu/Research/hpcl/STICS/stics.pdf?

STICS is patent pending so not sure if kernel can be free to merge this dm-cache.

I like the idea of STICS, an efficient bridge between SCSI and IP. But I think its only similarity with dm-cache is the use of nonvolatile caching.

Dm-cache is a generic block-level disk cache that supports a variety of SAN technologies and is not tied to any particular one.

Best regards,
Ming Zhao

#5 lore raw
Quoted 7 lines · Ming Zhao
Quoted 5 lines · Ming Zhang

like to see this idea but any similarity with
http://www.ele.uri.edu/Research/hpcl/STICS/stics.pdf?

STICS is patent pending so not sure if kernel can be free to merge this dm-cache.

Quoted 7 lines

I like the idea of STICS, an efficient bridge between SCSI and IP. But I think its only similarity with dm-cache is the use of nonvolatile caching.

Dm-cache is a generic block-level disk cache that supports a variety of SAN technologies and is not tied to any particular one.

great. as long as you are aware of this and can find the difference, then fine.

Quoted 3 lines

Best regards,
Ming Zhao

#6 lore raw
In reply to Eric Van Hensbergen
Quoted 10 lines · Eric Van Hensbergen

This is the first cut of a device-mapper target which provides a write-back or write-through block cache. It is intended to be used in conjunction with remote block devices such as iSCSI or ATA-over-Ethernet, particularly in cluster situations.

In performance tests with iSCSI, gave peformance improvements of 2-10x that of iSCSI alone when Postmark or Bonnie loads were applied from 8 clients to a single server. Evidence suggests even greater differences on larger clusters. A detailed performance analysis will be vailable shortly via a technical report on IBM's CyberDigest.

Can you give an idea how this relates to the normal block device buffers?

If this is intended to speed up remote disks, is it possible that the cache content can be paged out on local disks in low-mem situations?

Best,

Jens

#7 lore raw
Quoted 5 lines · Jens Wilke

On Monday 27 November 2006 19:26, Eric Van Hensbergen wrote:

If this is intended to speed up remote disks, is it possible that the cache content can be paged out on local disks in low-mem situations?

The main intent was to use local disks as cache to offload centralized remote disks. The logic was that most systems have local disks, if only for swap -- so why not use them as a cache to help offload centralized storage. While the in-memory page cache works perfectly fine in certain situations -- we were dealing with workloads in which the in-memory page-cache wasn't sufficient to hold all the data.

There are also some additional possibilities we've thought through and have been playing with including allowing the local disk cache to be persistent across reboots (with varying validation schemes).

-eric

#8 lore raw
Quoted 7 lines · Eric Van Hensbergen
Quoted 5 lines · Jens Wilke

On Monday 27 November 2006 19:26, Eric Van Hensbergen wrote:

If this is intended to speed up remote disks, is it possible that the cache content can be paged out on local disks in low-mem situations?

Quoted 6 lines

The main intent was to use local disks as cache to offload centralized remote disks. The logic was that most systems have local disks, if only for swap -- so why not use them as a cache to help offload centralized storage. While the in-memory page cache works perfectly fine in certain situations -- we were dealing with workloads in which the in-memory page-cache wasn't sufficient to hold all the data.

I derived from the code that the cache is actually another block device and not memory. Maybe you should give a little more description and a sample 'dmsetup create' statement.

Quoted 3 lines

There are also some additional possibilities we've thought through and have been playing with including allowing the local disk cache to be persistent across reboots (with varying validation schemes).

Yes, I like the idea. It would be also possible to use NV-Ram as write back cache, for example in notebooks, to avoid disk spin-ups or to improve transaction performance in enterprise applications.

Maybe it would also make sense to prefill the cache contents on startup with a bulk I/O request?

Comments on the code:

- The data of the write back cache should be flushed on suspend and not in the destructor.
- You don't keep track of I/O on the fly to the cache that is mapped directly in cache_hit(). How do you make sure that this I/O is completed before you replace a cache block?
- The cache block index is hashed, this means the cache data blocks are not clustered. I don't think you can solve this problem with a proper hash function. Perhaps you should consider a (B-)Tree structure for that.

Best,

Jens

#9 lore raw
Quoted 4 lines · Jens Wilke

- You don't keep track of I/O on the fly to the cache that is mapped directly in cache_hit(). How do you make sure that this I/O is completed before you replace a cache block?

The previous I/O from cache hit and the later I/O for cache replacement should be queued in order on the cache block device - is this a safe assumption?

- The cache block index is hashed, this means the cache data blocks are not

Quoted 3 lines

clustered. I don't think you can solve this problem with a proper hash function.
Perhaps you should consider a (B-)Tree structure for that.

The current hash algorithm does clustering to some extent by mapping consecutive blocks from the source device to consecutive blocks on the cache device. But I agree that more sophisticated algorithms can be studied.

Thanks a lot for the suggestions. I will try to incorporate them in the next patch.

- Ming

#10 lore raw
In reply to Jens Wilke
Quoted 3 lines · Jens Wilke

- You don't keep track of I/O on the fly to the cache that is mapped directly in cache_hit(). How do you make sure that this I/O is completed before you replace a cache block?

The previous I/O from cache hit and the later I/O for cache replacement should be queued in order on the cache block device - is this a safe assumption?

Quoted 3 lines

- The cache block index is hashed, this means the cache data blocks are not clustered. I don't think you can solve this problem with a proper hash function. Perhaps you should consider a (B-)Tree structure for that.

The current hash algorithm does clustering to some extent by mapping consecutive blocks from the source device to consecutive blocks on the cache device. But I agree that more sophisticated algorithms can be studied.

Thanks a lot for the suggestions. I will try to incorporate them in the next patch.

- Ming

#11 lore raw
Quoted 8 lines · Ming Zhao
Quoted 3 lines · Jens Wilke

- You don't keep track of I/O on the fly to the cache that is mapped directly in cache_hit(). How do you make sure that this I/O is completed before you replace a cache block?

The previous I/O from cache hit and the later I/O for cache replacement should be queued in order on the cache block device - is this a safe assumption?

I don't think you can assume that the request processing in the host is ordered without sychronization. Everything you queue in your worker thread is serialized by that. Everything else is out of your control and each thread may be scheduled or not.

Maybe we have certain guarantees through DM or the block I/O layer, that I don't know. Can somebody else step in here?

Best,

Jens

#12 lore raw
In reply to Eric Van Hensbergen
Quoted 5 lines · Eric Van Hensbergen

This is the first cut of a device-mapper target which provides a write-back or write-through block cache. It is intended to be used in conjunction with remote block devices such as iSCSI or ATA-over-Ethernet, particularly in cluster situations.

Quoted 8 lines

In performance tests with iSCSI, gave peformance improvements of 2-10x that of iSCSI alone when Postmark or Bonnie loads were applied from 8 clients to a single server. Evidence suggests even greater differences on larger clusters. A detailed performance analysis will be vailable shortly via a technical report on IBM's CyberDigest.

This module was developed during an intership at IBM Research by Ming Zhao. Please direct comments to both Ming and myself.

I don't quite understand the need for this. Do you need this module because your iscsi target doesn't use the pagecache on the server?

#13 lore raw
In reply to Eric Van Hensbergen
Quoted 5 lines · Eric Van Hensbergen

This is the first cut of a device-mapper target which provides a write-back or write-through block cache. It is intended to be used in conjunction with remote block devices such as iSCSI or ATA-over-Ethernet, particularly in cluster situations.

The technical paper describing our motivations and some performance results has finally made it through IBM's clearance process. It is available here:

http://domino.research.ibm.com/library/cyberdig.nsf/1e4115aea78b6e7c85256b360066f0d4/ba52bef8b940e7438525723c006bafea?OpenDocument

-eric

#14 lore raw

Hrm,

I finally got onto one of my TODO items a while ago which is an HSM (IOW: cache) target for device-mapper ;)
The design of my HSM target covers remote storage as well as arbitrary slow local storage hierachies.

Let's compare our code and see if there's any mutual stuff to merge or pros and cons of the different approaches when I post something shortly.

Heinz

Quoted 12 lines · Eric Van Hensbergen
Quoted 6 lines · Eric Van Hensbergen

This is the first cut of a device-mapper target which provides a write-back or write-through block cache. It is intended to be used in conjunction with
remote block devices such as iSCSI or ATA-over-Ethernet, particularly in cluster situations.

The technical paper describing our motivations and some performance results has finally made it through IBM's clearance process. It is available here:

Quoted 8 lines

http://domino.research.ibm.com/library/cyberdig.nsf/1e4115aea78b6e7c85256b360066f0d4/ba52bef8b940e7438525723c006bafea?OpenDocument

-eric

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Red Hat GmbH
Consulting Development Engineer                   Am Sonnenhang 11
Storage Development                               56242 Marienrachdorf
                                                  Germany
[email protected]                            PHONE +49  171 7803392
                                                  FAX   +49 2626 924446

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-