Skip to contentSkip to conversation

enhanceio: STEC EnhanceIO SSD caching software for Linux kernel

RFC PATCH In review Darrick J. Wong 6 participants 14 messages Feb 2, 2013 – Feb 11, 2013

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 enhanceio: STEC EnhanceIO SSD caching software for Linux kernel

This is a patch to migrate STEC's enhanceio driver out of their github repository and into the staging tree. From their README:

"EnhanceIO driver is based on EnhanceIO SSD caching software product developed by STEC Inc. EnhanceIO was derived from Facebook's open source Flashcache project. EnhanceIO uses SSDs as cache devices for traditional rotating hard disk drives (referred to as source volumes throughout this document). EnhanceIO can work with any block device, be it an entire physical disk, an individual disk partition, a RAIDed DAS device, a SAN volume, a device mapper volume or a software RAID (md) device."

This patch contains absolutely no cleanups whatsoever -- there are hundreds of style violations and other funny business, hence this is an RFC patch. On the other hand, it provides a code snapshot for us to review and critique. As far as I can tell, everything is self-contained inside drivers/staging/enhanceio/; no block layer changes seem to be required.

I will start reviewing the code as soon as this patch posts. Be warned that I haven't even compile tested this yet. :)

Amit: We're going to need a signed-off-by from you (and probably everyone else) for the parts that STEC is contributing.

From: Amit Kale <[email protected]>

---
 drivers/staging/Kconfig                |    2 
 drivers/staging/Makefile               |    1 
 drivers/staging/enhanceio/Kconfig      |   21 
 drivers/staging/enhanceio/Makefile     |   16 
 drivers/staging/enhanceio/eio.h        | 1146 ++++++++++
 drivers/staging/enhanceio/eio_conf.c   | 2537 +++++++++++++++++++++++
 drivers/staging/enhanceio/eio_fifo.c   |  265 ++
 drivers/staging/enhanceio/eio_ioctl.c  |  160 +
 drivers/staging/enhanceio/eio_ioctl.h  |   90 +
 drivers/staging/enhanceio/eio_lru.c    |  342 +++
 drivers/staging/enhanceio/eio_main.c   | 3503 ++++++++++++++++++++++++++++++++
 drivers/staging/enhanceio/eio_mem.c    |  252 ++
 drivers/staging/enhanceio/eio_policy.c |  162 +
 drivers/staging/enhanceio/eio_policy.h |  106 +
 drivers/staging/enhanceio/eio_procfs.c | 1825 +++++++++++++++++
 drivers/staging/enhanceio/eio_setlru.c |  193 ++
 drivers/staging/enhanceio/eio_setlru.h |   49 
 drivers/staging/enhanceio/eio_subr.c   |  472 ++++
 drivers/staging/enhanceio/eio_ttc.c    | 1708 ++++++++++++++++
 drivers/staging/enhanceio/eio_ttc.h    |  153 +
 tools/enhanceio/eio_cli                |  283 +++
 21 files changed, 13286 insertions(+)
 create mode 100644 drivers/staging/enhanceio/Kconfig
 create mode 100644 drivers/staging/enhanceio/Makefile
 create mode 100644 drivers/staging/enhanceio/eio.h
 create mode 100644 drivers/staging/enhanceio/eio_conf.c
 create mode 100644 drivers/staging/enhanceio/eio_fifo.c
 create mode 100644 drivers/staging/enhanceio/eio_ioctl.c
 create mode 100644 drivers/staging/enhanceio/eio_ioctl.h
 create mode 100644 drivers/staging/enhanceio/eio_lru.c
 create mode 100644 drivers/staging/enhanceio/eio_main.c
 create mode 100644 drivers/staging/enhanceio/eio_mem.c
 create mode 100644 drivers/staging/enhanceio/eio_policy.c
 create mode 100644 drivers/staging/enhanceio/eio_policy.h
 create mode 100644 drivers/staging/enhanceio/eio_procfs.c
 create mode 100644 drivers/staging/enhanceio/eio_setlru.c
 create mode 100644 drivers/staging/enhanceio/eio_setlru.h
 create mode 100644 drivers/staging/enhanceio/eio_subr.c
 create mode 100644 drivers/staging/enhanceio/eio_ttc.c
 create mode 100644 drivers/staging/enhanceio/eio_ttc.h
 create mode 100755 tools/enhanceio/eio_cli
drivers/staging/Kconfig +2 -0
index 329bdb4..0e97141 100644
@@ -142,4 +142,6 @@ source "drivers/staging/sb105x/Kconfig"
 
 source "drivers/staging/fwserial/Kconfig"
 
+source "drivers/staging/enhanceio/Kconfig"
+
 endif # STAGING
drivers/staging/Makefile +1 -0
index c7ec486..81656de 100644
@@ -63,3 +63,4 @@ obj-$(CONFIG_DRM_IMX)		+= imx-drm/
 obj-$(CONFIG_DGRP)		+= dgrp/
 obj-$(CONFIG_SB105X)		+= sb105x/
 obj-$(CONFIG_FIREWIRE_SERIAL)	+= fwserial/
+obj-$(CONFIG_ENHANCEIO)		+= enhanceio/
drivers/staging/enhanceio/Kconfig new file +21 -0
new file mode 100644
index 0000000..e153496
@@ -0,0 +1,21 @@
+#
+# EnhanceIO caching solution by STEC INC.
+#
+
+config ENHANCEIO
+	tristate "Enable EnhanceIO"
+	depends on BLK_DEV
+	default n
+	---help---
+	Based on Facebook's open source Flashcache project developed by
+	Mohan Srinivasan and hosted at "http://github.com", EnhanceIO is
+	a collection of (currently three) loadable kernel modules for
+	using SSDs as cache devices for traditional rotating hard disk
+
+	The caching engine is a loadable kernel module ("enhanceio.ko")
+	implemented as a device mapper target.	The cache replacement
+	policies are implemented as loadable kernel modules
+	("enhanceio_fifo.ko", "enhanceio_lru.ko") that register with
+	the caching engine module.
+
+	If unsure, say N.
drivers/staging/enhanceio/Makefile new file +16 -0
new file mode 100644
index 0000000..926aa71
@@ -0,0 +1,16 @@
+#
+# Makefile for EnhanceIO block device caching.
+#
+obj-$(CONFIG_ENHANCEIO)	+= enhanceio.o enhanceio_lru.o enhanceio_fifo.o
+enhanceio-y	+= \
+	eio_conf.o \
+	eio_ioctl.o \
+	eio_main.o \
+	eio_mem.o \
+	eio_policy.o \
+	eio_procfs.o \
+	eio_setlru.o \
+	eio_subr.o \
+	eio_ttc.o
+enhanceio_fifo-y	+= eio_fifo.o
+enhanceio_lru-y	+= eio_lru.o
drivers/staging/enhanceio/eio.h new file +1146 -0

1,146 lines not on this page · raw message

drivers/staging/enhanceio/eio_conf.c new file +2537 -0

1,807 lines not on this page · raw message

drivers/staging/enhanceio/eio_fifo.c new file +265 -0

265 lines not on this page · raw message

drivers/staging/enhanceio/eio_ioctl.c new file +160 -0

160 lines not on this page · raw message

drivers/staging/enhanceio/eio_ioctl.h new file +90 -0

90 lines not on this page · raw message

drivers/staging/enhanceio/eio_lru.c new file +342 -0

342 lines not on this page · raw message

drivers/staging/enhanceio/eio_main.c new file +3503 -0

3,503 lines not on this page · raw message

drivers/staging/enhanceio/eio_mem.c new file +252 -0

252 lines not on this page · raw message

drivers/staging/enhanceio/eio_policy.c new file +162 -0

162 lines not on this page · raw message

drivers/staging/enhanceio/eio_policy.h new file +106 -0

106 lines not on this page · raw message

drivers/staging/enhanceio/eio_procfs.c new file +1825 -0

1,825 lines not on this page · raw message

drivers/staging/enhanceio/eio_setlru.c new file +193 -0

193 lines not on this page · raw message

drivers/staging/enhanceio/eio_setlru.h new file +49 -0

49 lines not on this page · raw message

drivers/staging/enhanceio/eio_subr.c new file +472 -0

472 lines not on this page · raw message

drivers/staging/enhanceio/eio_ttc.c new file +1708 -0

1,708 lines not on this page · raw message

drivers/staging/enhanceio/eio_ttc.h new file +153 -0

153 lines not on this page · raw message

tools/enhanceio/eio_cli new file +283 -0

283 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Part of the diff is not shown here. View the raw message.

#1 lore raw
Quoted 10 lines · Darrick J. Wong

This is a patch to migrate STEC's enhanceio driver out of their github repository and into the staging tree. From their README:

"EnhanceIO driver is based on EnhanceIO SSD caching software product developed by STEC Inc. EnhanceIO was derived from Facebook's open source Flashcache project. EnhanceIO uses SSDs as cache devices for traditional rotating hard disk drives (referred to as source volumes throughout this document). EnhanceIO can work with any block device, be it an entire physical disk, an individual disk partition, a RAIDed DAS device, a SAN volume, a device mapper volume or a software RAID (md) device."

What's your take on the benefits of this vs bcache ?

Signature · 2 lines
Michel "Walken" Lespinasse
A program is never fully debugged until the last user dies.
#2 lore raw
Quoted 12 lines · Michel Lespinasse
Quoted 10 lines · Darrick J. Wong

This is a patch to migrate STEC's enhanceio driver out of their github repository and into the staging tree. From their README:

"EnhanceIO driver is based on EnhanceIO SSD caching software product developed by STEC Inc. EnhanceIO was derived from Facebook's open source Flashcache project. EnhanceIO uses SSDs as cache devices for traditional rotating hard disk drives (referred to as source volumes throughout this document). EnhanceIO can work with any block device, be it an entire physical disk, an individual disk partition, a RAIDed DAS device, a SAN volume, a device mapper volume or a software RAID (md) device."

Quoted 1 line

What's your take on the benefits of this vs bcache ?

I've been trying to review it it for cache coherency and consistency wrt. power failures, but the complete lack of high level documentation is making that difficult.

I saw some benchmarks the other day where on writes EIO was faster than the raw device, which was... suspicious.

#3 lore raw

Also, where's the end user documentation?

Quoted 21 lines · Kent Overstreet
Quoted 12 lines · Michel Lespinasse
Quoted 10 lines · Darrick J. Wong

This is a patch to migrate STEC's enhanceio driver out of their github repository and into the staging tree. From their README:

"EnhanceIO driver is based on EnhanceIO SSD caching software product developed by STEC Inc. EnhanceIO was derived from Facebook's open source Flashcache project. EnhanceIO uses SSDs as cache devices for traditional rotating hard disk drives (referred to as source volumes throughout this document). EnhanceIO can work with any block device, be it an entire physical disk, an individual disk partition, a RAIDed DAS device, a SAN volume, a device mapper volume or a software RAID (md) device."

Quoted 1 line

What's your take on the benefits of this vs bcache ?

I've been trying to review it it for cache coherency and consistency wrt. power failures, but the complete lack of high level documentation is making that difficult.

I saw some benchmarks the other day where on writes EIO was faster than the raw device, which was... suspicious.

#4 lore raw

End user documentation is on our github. I think it should go with the user level utilities, rather than the kernel driver. -Amit

Quoted 31 lines

-----Original Message-----
From: Kent Overstreet [mailto:[email protected]]
Sent: Friday, February 01, 2013 5:25 PM
To: Michel Lespinasse
Cc: Darrick J. Wong; Amit Kale; linux-bcache; device-mapper development; Kent Overstreet; Mike Snitzer; LKML; Jason Warr; [email protected]
Subject: Re: [RFC] [DONOTAPPLY] [PATCH] enhanceio: STEC EnhanceIO SSD caching software for Linux kernel

Also, where's the end user documentation?

Quoted 1 line · Kent Overstreet

On Fri, Feb 1, 2013 at 4:57 PM, Michel Lespinasse <[email protected]>

Quoted 1 line · wrote:
Quoted 1 line

On Fri, Feb 1, 2013 at 4:44 PM, Darrick J. Wong

Quoted 4 lines · <[email protected]> wrote:
Quoted 4 lines
Quoted 4 lines

This is a patch to migrate STEC's enhanceio driver out of their github repository and into the staging tree. From their README:

"EnhanceIO driver is based on EnhanceIO SSD caching software

product

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

developed by STEC Inc. EnhanceIO was derived from Facebook's open source Flashcache project. EnhanceIO uses SSDs as cache devices for traditional rotating hard disk drives (referred to as source

volumes throughout this document).

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

EnhanceIO can work with any block device, be it an entire physical disk, an individual disk partition, a RAIDed DAS device, a SAN volume, a device mapper volume or a software RAID (md) device."

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

What's your take on the benefits of this vs bcache ?

I've been trying to review it it for cache coherency and consistency wrt. power failures, but the complete lack of high level

documentation

Quoted 4 lines

is making that difficult.

I saw some benchmarks the other day where on writes EIO was faster than the raw device, which was... suspicious.

PROPRIETARY-CONFIDENTIAL INFORMATION INCLUDED

This electronic transmission, and any documents attached hereto, may contain confidential, proprietary and/or legally privileged information. The information is intended only for use by the recipient named above. If you received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of information received in error is strictly prohibited, and violators will be pursued legally.

#5 lore raw
In reply to Kent Overstreet
Quoted 1 line · Kent Overstreet

Also, where's the end user documentation?

I was trying not to explode this patch's size, since the doc can go straight into Documentation/ without as much fuss as, say, the ~1600 checkpatch warnings and all the other comments we'll have about the source code.

Quoted 14 lines · Kent Overstreet
Quoted 13 lines · Michel Lespinasse
Quoted 12 lines · Darrick J. Wong
Quoted 10 lines

This is a patch to migrate STEC's enhanceio driver out of their github repository and into the staging tree. From their README:

"EnhanceIO driver is based on EnhanceIO SSD caching software product developed by STEC Inc. EnhanceIO was derived from Facebook's open source Flashcache project. EnhanceIO uses SSDs as cache devices for traditional rotating hard disk drives (referred to as source volumes throughout this document). EnhanceIO can work with any block device, be it an entire physical disk, an individual disk partition, a RAIDed DAS device, a SAN volume, a device mapper volume or a software RAID (md) device."

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

What's your take on the benefits of this vs bcache ?

I've been trying to review it it for cache coherency and consistency wrt. power failures, but the complete lack of high level documentation is making that difficult.

Agreed. :/

Quoted 2 lines
Quoted 2 lines

I saw some benchmarks the other day where on writes EIO was faster than the raw device, which was... suspicious.

Hmm, I haven't seen /that/, but then I wasn't specifically looking for it either.

--D

#6 lore raw
In reply to Michel Lespinasse
Quoted 19 lines

-----Original Message-----
From: Michel Lespinasse [mailto:[email protected]]
Sent: Friday, February 01, 2013 4:58 PM
To: Darrick J. Wong
Cc: Amit Kale; linux-bcache; device-mapper development; Kent Overstreet; Mike Snitzer; LKML; Jason Warr; [email protected]
Subject: Re: [RFC] [DONOTAPPLY] [PATCH] enhanceio: STEC EnhanceIO SSD caching software for Linux kernel

Quoted 1 line · Darrick J. Wong

This is a patch to migrate STEC's enhanceio driver out of their

github

Quoted 6 lines

repository and into the staging tree. From their README:

"EnhanceIO driver is based on EnhanceIO SSD caching software product developed by STEC Inc. EnhanceIO was derived from Facebook's open source Flashcache project. EnhanceIO uses SSDs as cache devices for traditional rotating hard disk drives (referred to as source volumes

Quoted 6 lines

throughout this document).

Quoted 3 lines

EnhanceIO can work with any block device, be it an entire physical disk, an individual disk partition, a RAIDed DAS device, a SAN volume, a device mapper volume or a software RAID (md) device."

What's your take on the benefits of this vs bcache ?

EnhanceIO was designed for and has been validated in enterprise environments. The important benefits are -
1. There is no downtime for cache creation, deletion, editing properties, writeback/readonly/writethrough mode change.
2. Wb mode comes with an option to control whether dirty data should be clean-up across reboots, which prevents SSD/HDD going out of sync.
3. Our in-house testing was done for large setups with 500GB+SSDs and proportionately large HDDs, on 24CPU machines with plenty of RAM. It's survived heavy IO loads without any locking or corruption problems.
4. Error handling is exactly what enterprises look for - writethrough/readonly modes work seamlessly regardless of SSD failures. In all the three caching modes, the guarantees of completion in presence of IO errors or shutdowns, in terms of granularity and persistence of data written, is identical to underlying HDDs.
5. It works for all known block devices - Software RAIDs, full block devices with or without partitions, individual partitions, various intelligent block devices.

-Amit

PROPRIETARY-CONFIDENTIAL INFORMATION INCLUDED

This electronic transmission, and any documents attached hereto, may contain confidential, proprietary and/or legally privileged information. The information is intended only for use by the recipient named above. If you received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of information received in error is strictly prohibited, and violators will be pursued legally.

#7 lore raw
Quoted 20 lines · Amit Kale
Quoted 20 lines

-----Original Message-----
From: Michel Lespinasse [mailto:[email protected]]
Sent: Friday, February 01, 2013 4:58 PM
To: Darrick J. Wong
Cc: Amit Kale; linux-bcache; device-mapper development; Kent
Overstreet; Mike Snitzer; LKML; Jason Warr; [email protected]
Subject: Re: [RFC] [DONOTAPPLY] [PATCH] enhanceio: STEC EnhanceIO SSD
caching software for Linux kernel

Quoted 1 line · Darrick J. Wong

This is a patch to migrate STEC's enhanceio driver out of their

github

Quoted 6 lines

repository and into the staging tree. From their README:

"EnhanceIO driver is based on EnhanceIO SSD caching software product developed by STEC Inc. EnhanceIO was derived from Facebook's open source Flashcache project. EnhanceIO uses SSDs as cache devices for traditional rotating hard disk drives (referred to as source volumes

throughout this document).

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

EnhanceIO can work with any block device, be it an entire physical disk, an individual disk partition, a RAIDed DAS device, a SAN volume, a device mapper volume or a software RAID (md) device."

What's your take on the benefits of this vs bcache ?

EnhanceIO was designed for and has been validated in enterprise environments. The important benefits are -
1. There is no downtime for cache creation, deletion, editing properties, writeback/readonly/writethrough mode change.

True of bcache.

Quoted 1 line

2. Wb mode comes with an option to control whether dirty data should be clean-up across reboots, which prevents SSD/HDD going out of sync.

No idea why you'd want this. You have to be able to handle a rebooting with a dirty cache, if you hope to handle unclean shutdown - bcache does this. And if you're running in writeback mode there's probably going to be many gigabytes of dirty data in your cache, at least sometimes, and you aren't going to want to wait for that to flush before you reboot.

If you want to flush dirty data with bcache, you can switch back to writethrough mode whenever you want.

Quoted 1 line

3. Our in-house testing was done for large setups with 500GB+SSDs and proportionately large HDDs, on 24CPU machines with plenty of RAM. It's survived heavy IO loads without any locking or corruption problems.

True of bcache.

Quoted 1 line

4. Error handling is exactly what enterprises look for - writethrough/readonly modes work seamlessly regardless of SSD failures. In all the three caching modes, the guarantees of completion in presence of IO errors or shutdowns, in terms of granularity and persistence of data written, is identical to underlying HDDs.

True of bcache.

Quoted 1 line

5. It works for all known block devices - Software RAIDs, full block devices with or without partitions, individual partitions, various intelligent block devices.

True of bcache.

Quoted 5 lines

-Amit

PROPRIETARY-CONFIDENTIAL INFORMATION INCLUDED

This electronic transmission, and any documents attached hereto, may contain confidential, proprietary and/or legally privileged information. The information is intended only for use by the recipient named above. If you received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of information received in error is strictly prohibited, and violators will be pursued legally.

Which of the above information was proprietary, and should I not be resending this to lkml or - who?

...

#8 lore raw
Quoted 56 lines

-----Original Message-----
From: Kent Overstreet [mailto:[email protected]]
Sent: Wednesday, February 06, 2013 4:01 PM
To: Amit Kale
Cc: Michel Lespinasse; Darrick J. Wong; linux-bcache; device-mapper development; Kent Overstreet; Mike Snitzer; LKML; Jason Warr; [email protected]
Subject: Re: [RFC] [DONOTAPPLY] [PATCH] enhanceio: STEC EnhanceIO SSD caching software for Linux kernel

Quoted 15 lines · Amit Kale
Quoted 15 lines

-----Original Message-----
From: Michel Lespinasse [mailto:[email protected]]
Sent: Friday, February 01, 2013 4:58 PM
To: Darrick J. Wong
Cc: Amit Kale; linux-bcache; device-mapper development; Kent
Overstreet; Mike Snitzer; LKML; Jason Warr; [email protected]
Subject: Re: [RFC] [DONOTAPPLY] [PATCH] enhanceio: STEC EnhanceIO
SSD caching software for Linux kernel

Quoted 1 line · Darrick J. Wong

This is a patch to migrate STEC's enhanceio driver out of their

github

Quoted 2 lines

repository and into the staging tree. From their README:

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

"EnhanceIO driver is based on EnhanceIO SSD caching software product developed by STEC Inc. EnhanceIO was derived from Facebook's open source Flashcache project. EnhanceIO uses SSDs as cache devices for traditional rotating hard disk drives (referred to as source volumes

throughout this document).

Quoted 1 line

EnhanceIO can work with any block device, be it an entire

physical

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

disk, an individual disk partition, a RAIDed DAS device, a SAN volume, a device mapper volume or a software RAID (md) device."

What's your take on the benefits of this vs bcache ?

EnhanceIO was designed for and has been validated in enterprise environments. The important benefits are - 1. There is no downtime

for cache creation, deletion, editing properties,
writeback/readonly/writethrough mode change.

True of bcache.

Quoted 1 line

2. Wb mode comes with an option to control whether dirty data should

be clean-up across reboots, which prevents SSD/HDD going out of sync.

No idea why you'd want this. You have to be able to handle a rebooting with a dirty cache, if you hope to handle unclean shutdown - bcache does this. And if you're running in writeback mode there's probably going to be many gigabytes of dirty data in your cache, at least sometimes, and you aren't going to want to wait for that to flush before you reboot.

Quoted 2 lines

If you want to flush dirty data with bcache, you can switch back to writethrough mode whenever you want.

We have two modes of operation with writeback

1. Warm restart mode (default) - Reboots are quick. Dirty data is kept as it is across reboots. So is clean data. Abrupt reboots are handled correctly by throwing away clean data and retaining dirty data. The throwing away clean data is a fallout of an optimization done to skip metadata updates for clean data.

2. Cold restart mode - A reboot or a shutdown involves clean-up of dirty data. This is for paranoid sysads who are afraid that iSCSI HDD setup may cause them to be reattached. Switching to writethrough mode will achieve the same thing, however with a persistent cache configuration in place, the cache will be write-through after a reboot.

I am happy to see bcache being compliant with the rest of the items I had noted.

-Amit

Quoted 33 lines
Quoted 2 lines

3. Our in-house testing was done for large setups with 500GB+SSDs and

proportionately large HDDs, on 24CPU machines with plenty of RAM. It's survived heavy IO loads without any locking or corruption problems.

True of bcache.

Quoted 1 line

4. Error handling is exactly what enterprises look for -

writethrough/readonly modes work seamlessly regardless of SSD failures. In all the three caching modes, the guarantees of completion in presence of IO errors or shutdowns, in terms of granularity and persistence of data written, is identical to underlying HDDs.

True of bcache.

Quoted 1 line

5. It works for all known block devices - Software RAIDs, full block

devices with or without partitions, individual partitions, various intelligent block devices.

True of bcache.

Quoted 5 lines

-Amit

PROPRIETARY-CONFIDENTIAL INFORMATION INCLUDED

This electronic transmission, and any documents attached hereto, may

contain confidential, proprietary and/or legally privileged information. The information is intended only for use by the recipient named above. If you received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of information received in error is strictly prohibited, and violators will be pursued legally.

Quoted 4 lines

Which of the above information was proprietary, and should I not be resending this to lkml or - who?

...

PROPRIETARY-CONFIDENTIAL INFORMATION INCLUDED

This electronic transmission, and any documents attached hereto, may contain confidential, proprietary and/or legally privileged information. The information is intended only for use by the recipient named above. If you received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of information received in error is strictly prohibited, and violators will be pursued legally.

#9 lore raw
Quoted 56 lines · Amit Kale
Quoted 56 lines

-----Original Message-----
From: Kent Overstreet [mailto:[email protected]]
Sent: Wednesday, February 06, 2013 4:01 PM
To: Amit Kale
Cc: Michel Lespinasse; Darrick J. Wong; linux-bcache; device-mapper development; Kent Overstreet; Mike Snitzer; LKML; Jason Warr; [email protected]
Subject: Re: [RFC] [DONOTAPPLY] [PATCH] enhanceio: STEC EnhanceIO SSD caching software for Linux kernel

Quoted 15 lines · Amit Kale
Quoted 15 lines

-----Original Message-----
From: Michel Lespinasse [mailto:[email protected]]
Sent: Friday, February 01, 2013 4:58 PM
To: Darrick J. Wong
Cc: Amit Kale; linux-bcache; device-mapper development; Kent Overstreet; Mike Snitzer; LKML; Jason Warr; [email protected]
Subject: Re: [RFC] [DONOTAPPLY] [PATCH] enhanceio: STEC EnhanceIO SSD caching software for Linux kernel

Quoted 1 line · Darrick J. Wong

This is a patch to migrate STEC's enhanceio driver out of their

github

Quoted 2 lines

repository and into the staging tree. From their README:

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

"EnhanceIO driver is based on EnhanceIO SSD caching software product developed by STEC Inc. EnhanceIO was derived from Facebook's open source Flashcache project. EnhanceIO uses SSDs as cache devices for traditional rotating hard disk drives (referred to as source volumes

throughout this document).

Quoted 1 line

EnhanceIO can work with any block device, be it an entire

physical

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

disk, an individual disk partition, a RAIDed DAS device, a SAN volume, a device mapper volume or a software RAID (md) device."

What's your take on the benefits of this vs bcache ?

EnhanceIO was designed for and has been validated in enterprise environments. The important benefits are - 1. There is no downtime

for cache creation, deletion, editing properties,
writeback/readonly/writethrough mode change.

True of bcache.

Quoted 1 line

2. Wb mode comes with an option to control whether dirty data should

be clean-up across reboots, which prevents SSD/HDD going out of sync.

No idea why you'd want this. You have to be able to handle a rebooting with a dirty cache, if you hope to handle unclean shutdown - bcache does this. And if you're running in writeback mode there's probably going to be many gigabytes of dirty data in your cache, at least sometimes, and you aren't going to want to wait for that to flush before you reboot.

Quoted 8 lines
Quoted 2 lines

If you want to flush dirty data with bcache, you can switch back to writethrough mode whenever you want.

We have two modes of operation with writeback

1. Warm restart mode (default) - Reboots are quick. Dirty data is kept as it is across reboots. So is clean data. Abrupt reboots are handled correctly by throwing away clean data and retaining dirty data. The throwing away clean data is a fallout of an optimization done to skip metadata updates for clean data.

2. Cold restart mode - A reboot or a shutdown involves clean-up of dirty data. This is for paranoid sysads who are afraid that iSCSI HDD setup may cause them to be reattached. Switching to writethrough mode will achieve the same thing, however with a persistent cache configuration in place, the cache will be write-through after a reboot.

How does Enhance-IO prevent mounting of the backing device if the cache device doesn't show up due to failure etc?
I was reviewing the setup/tear-down code but I couldn't find where EIO writes any superblock etc to the backing device to prevent udev etc from mounting the filesystem.

Quoted 50 lines

I am happy to see bcache being compliant with the rest of the items I had noted.

-Amit

Quoted 33 lines
Quoted 2 lines

3. Our in-house testing was done for large setups with 500GB+SSDs and

proportionately large HDDs, on 24CPU machines with plenty of RAM. It's survived heavy IO loads without any locking or corruption problems.

True of bcache.

Quoted 1 line

4. Error handling is exactly what enterprises look for -

writethrough/readonly modes work seamlessly regardless of SSD failures. In all the three caching modes, the guarantees of completion in presence of IO errors or shutdowns, in terms of granularity and persistence of data written, is identical to underlying HDDs.

True of bcache.

Quoted 1 line

5. It works for all known block devices - Software RAIDs, full block

devices with or without partitions, individual partitions, various intelligent block devices.

True of bcache.

Quoted 5 lines

-Amit

PROPRIETARY-CONFIDENTIAL INFORMATION INCLUDED

This electronic transmission, and any documents attached hereto, may

contain confidential, proprietary and/or legally privileged information. The information is intended only for use by the recipient named above. If you received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of information received in error is strictly prohibited, and violators will be pursued legally.

Quoted 4 lines

Which of the above information was proprietary, and should I not be resending this to lkml or - who?

...

PROPRIETARY-CONFIDENTIAL INFORMATION INCLUDED

This electronic transmission, and any documents attached hereto, may contain confidential, proprietary and/or legally privileged information. The information is intended only for use by the recipient named above. If you received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of information received in error is strictly prohibited, and violators will be pursued legally. --
To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines
CTO | Orion Virtualisation Solutions | www.orionvm.com.au
Phone: 1300 56 99 52 | Mobile: 0428 754 846
#10 lore raw
Quoted 91 lines

-----Original Message-----
From: Joseph Glanville [mailto:[email protected]]
Sent: Saturday, February 09, 2013 1:10 AM
To: Amit Kale
Cc: Kent Overstreet; Michel Lespinasse; Darrick J. Wong; linux-bcache; device-mapper development; Kent Overstreet; Mike Snitzer; LKML; Jason Warr; [email protected]
Subject: Re: [RFC] [DONOTAPPLY] [PATCH] enhanceio: STEC EnhanceIO SSD caching software for Linux kernel

Quoted 8 lines · Amit Kale
Quoted 8 lines

-----Original Message-----
From: Kent Overstreet [mailto:[email protected]]
Sent: Wednesday, February 06, 2013 4:01 PM
To: Amit Kale
Cc: Michel Lespinasse; Darrick J. Wong; linux-bcache; device-mapper development; Kent Overstreet; Mike Snitzer; LKML; Jason Warr; [email protected]
Subject: Re: [RFC] [DONOTAPPLY] [PATCH] enhanceio: STEC EnhanceIO

SSD

Quoted 10 lines
Quoted 10 lines

caching software for Linux kernel

Quoted 7 lines · Amit Kale
Quoted 7 lines

-----Original Message-----
From: Michel Lespinasse [mailto:[email protected]]
Sent: Friday, February 01, 2013 4:58 PM
To: Darrick J. Wong
Cc: Amit Kale; linux-bcache; device-mapper development; Kent
Overstreet; Mike Snitzer; LKML; Jason Warr; [email protected]
Subject: Re: [RFC] [DONOTAPPLY] [PATCH] enhanceio: STEC

EnhanceIO

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

SSD caching software for Linux kernel

Quoted 1 line · Darrick J. Wong

This is a patch to migrate STEC's enhanceio driver out of

their

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

github

Quoted 7 lines

repository and into the staging tree. From their README:

"EnhanceIO driver is based on EnhanceIO SSD caching software product developed by STEC Inc. EnhanceIO was derived from Facebook's open source Flashcache project. EnhanceIO uses SSDs as cache devices for traditional rotating hard disk drives (referred to as source volumes

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

throughout this document).

Quoted 1 line

EnhanceIO can work with any block device, be it an entire

physical

Quoted 1 line
Quoted 1 line
Quoted 1 line

disk, an individual disk partition, a RAIDed DAS device, a

SAN

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

volume, a device mapper volume or a software RAID (md)

device."

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

What's your take on the benefits of this vs bcache ?

EnhanceIO was designed for and has been validated in enterprise environments. The important benefits are - 1. There is no downtime

Quoted 8 lines
Quoted 8 lines

for cache creation, deletion, editing properties,
writeback/readonly/writethrough mode change.

True of bcache.

Quoted 2 lines

2. Wb mode comes with an option to control whether dirty data should

be clean-up across reboots, which prevents SSD/HDD going out of

sync.

Quoted 8 lines
Quoted 8 lines

No idea why you'd want this. You have to be able to handle a rebooting with a dirty cache, if you hope to handle unclean shutdown
- bcache does this. And if you're running in writeback mode there's probably going to be many gigabytes of dirty data in your cache, at least sometimes, and you aren't going to want to wait for that to flush before you reboot.

Quoted 6 lines
Quoted 2 lines

If you want to flush dirty data with bcache, you can switch back to writethrough mode whenever you want.

We have two modes of operation with writeback

1. Warm restart mode (default) - Reboots are quick. Dirty data is

kept as it is across reboots. So is clean data. Abrupt reboots are handled correctly by throwing away clean data and retaining dirty data. The throwing away clean data is a fallout of an optimization done to skip metadata updates for clean data.

Quoted 2 lines

2. Cold restart mode - A reboot or a shutdown involves clean-up of

dirty data. This is for paranoid sysads who are afraid that iSCSI HDD setup may cause them to be reattached. Switching to writethrough mode will achieve the same thing, however with a persistent cache configuration in place, the cache will be write-through after a reboot.

Quoted 5 lines

How does Enhance-IO prevent mounting of the backing device if the cache device doesn't show up due to failure etc?
I was reviewing the setup/tear-down code but I couldn't find where EIO writes any superblock etc to the backing device to prevent udev etc from mounting the filesystem.

EnhanceIO kernel driver doesn't do anything to prevent it. User level scripts have to do that. Kindly check the documentation for an example of how to do this using udev. We have tested several failure scenarios and udev scripts to handle them correctly.

-Amit

Quoted 67 lines
Quoted 3 lines

I am happy to see bcache being compliant with the rest of the items I

had noted.

Quoted 8 lines

-Amit

Quoted 5 lines
Quoted 3 lines

3. Our in-house testing was done for large setups with 500GB+SSDs and

proportionately large HDDs, on 24CPU machines with plenty of RAM. It's survived heavy IO loads without any locking or corruption

problems.

Quoted 5 lines
Quoted 5 lines

True of bcache.

Quoted 1 line

4. Error handling is exactly what enterprises look for -

writethrough/readonly modes work seamlessly regardless of SSD

failures.

Quoted 17 lines
Quoted 17 lines

In all the three caching modes, the guarantees of completion in presence of IO errors or shutdowns, in terms of granularity and persistence of data written, is identical to underlying HDDs.

True of bcache.

Quoted 2 lines

5. It works for all known block devices - Software RAIDs, full block

devices with or without partitions, individual partitions, various intelligent block devices.

True of bcache.

Quoted 4 lines

-Amit

PROPRIETARY-CONFIDENTIAL INFORMATION INCLUDED

Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

This electronic transmission, and any documents attached hereto, may

contain confidential, proprietary and/or legally privileged information. The information is intended only for use by the recipient named above. If you received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of information received in error is strictly prohibited, and violators

will be pursued legally.

Quoted 9 lines
Quoted 5 lines

Which of the above information was proprietary, and should I not be resending this to lkml or - who?

...

PROPRIETARY-CONFIDENTIAL INFORMATION INCLUDED

This electronic transmission, and any documents attached hereto, may

contain confidential, proprietary and/or legally privileged information. The information is intended only for use by the recipient named above. If you received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of information received in error is strictly prohibited, and violators will be pursued legally.

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe
linux-bcache" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 3 lines

--
CTO | Orion Virtualisation Solutions | www.orionvm.com.au
Phone: 1300 56 99 52 | Mobile: 0428 754 846

PROPRIETARY-CONFIDENTIAL INFORMATION INCLUDED

This electronic transmission, and any documents attached hereto, may contain confidential, proprietary and/or legally privileged information. The information is intended only for use by the recipient named above. If you received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of information received in error is strictly prohibited, and violators will be pursued legally.

#11 lore raw
In reply to Darrick J. Wong
Quoted 2 lines · Darrick J. Wong

I will start reviewing the code as soon as this patch posts. Be warned that I haven't even compile tested this yet. :)

Has the code been updated since I looked at it a couple of weeks ago? For example addressing all the 'sparse' warnings?

If you've not compiled or reviewed this code, what do you mean by the 'signed-off by' tag?

- Joe

#12 lore raw
Quoted 9 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]]
Sent: Monday, February 04, 2013 6:40 AM
To: Darrick J. Wong
Cc: Amit Kale; linux-bcache; device-mapper development; Kent
Overstreet; Mike Snitzer; LKML; Jason Warr
Subject: Re: [RFC] [DONOTAPPLY] [PATCH] enhanceio: STEC EnhanceIO SSD
caching software for Linux kernel

Quoted 6 lines
Quoted 2 lines · Darrick J. Wong

I will start reviewing the code as soon as this patch posts. Be warned that I haven't even compile tested this yet. :)

Has the code been updated since I looked at it a couple of weeks ago? For example addressing all the 'sparse' warnings?

Yes. Several warning were fixed and one kernel bug was fixed. It's better in terms of sparse warning, but not there yet.

-Amit

Quoted 5 lines

If you've not compiled or reviewed this code, what do you mean by the 'signed-off by' tag?

- Joe

PROPRIETARY-CONFIDENTIAL INFORMATION INCLUDED

This electronic transmission, and any documents attached hereto, may contain confidential, proprietary and/or legally privileged information. The information is intended only for use by the recipient named above. If you received this electronic message in error, please notify the sender and delete the electronic message. Any disclosure, copying, distribution, or use of the contents of information received in error is strictly prohibited, and violators will be pursued legally.

#13 lore raw
In reply to [email protected]
Quoted 9 lines · On Mon, Feb 04, 2013 at 02:40:17PM +0000…
Quoted 2 lines · Darrick J. Wong

I will start reviewing the code as soon as this patch posts. Be warned that I haven't even compile tested this yet. :)

Has the code been updated since I looked at it a couple of weeks ago? For example addressing all the 'sparse' warnings?

If you've not compiled or reviewed this code, what do you mean by the 'signed-off by' tag?

I changed a couple of things in the Kconfig file, most notably setting default to n and depending on BLK_DEV.

--D

Quoted 2 lines

- Joe