staging: writeboost: Add dm-writeboost

drivers/staging/writeboost/TODO new file +36 -0

@@ -0,0 +1,47 @@
+TODO:
+
+- Expose bugs and fix them.
+- Clean up the interfaces. Some input values should be limited in case user may choose too big values.
+- Wait for proper review by Mike Snitzer.
+- Fix up documents.
+
+-------------------------------------------------------------------------
+
+Project Ideas:
+
+1. Add read caching feature
+Device-Mapper maintainer Mike Snitzer said that
+a single target should provide caching for both reads and writes.
+(cf. http://www.redhat.com/archives/dm-devel/2014-January/msg00078.html)
+
+To follow Mike's guide, I will implement read-caching for Writeboost.
+The great idea is, at least conceptually,
+sending back the read data to the write entrance.
+I didn't notice this simple idea at that time of previous discussion.
+
+This should be done background and
+some threshold to determine
+which sequence of data should be staged.
+
+This feature will be implemented step-by-step.
+I don't think this feature will change then interface drastically.
+
+2. Improve initialization time
+Some user complains that the initialization is too slow.
+
+Any ideas?
+
+3. Reduce metadata footprint
+Writeboost has a hash table in RAM for cache management.
+This isn't cheap.

Greg Kroah-Hartman <[email protected]> please, and also the driverdevel mailing list, which is where the rest of the staging patches go.

And, I'll need an ack from the dm maintainers before I can accept this as something they think is ok and will get merged to the "real" part of the kernel someday.

Quoted 39 lines · Akira Hayakawa

drivers/staging/writeboost/TODO new file +36 -0

@@ -0,0 +1,47 @@
+TODO:
+
+- Expose bugs and fix them.
+- Clean up the interfaces. Some input values should be limited in case user may choose too big values.
+- Wait for proper review by Mike Snitzer.
+- Fix up documents.
+
+-------------------------------------------------------------------------
+
+Project Ideas:
+
+1. Add read caching feature
+Device-Mapper maintainer Mike Snitzer said that
+a single target should provide caching for both reads and writes.
+(cf. http://www.redhat.com/archives/dm-devel/2014-January/msg00078.html)
+
+To follow Mike's guide, I will implement read-caching for Writeboost.
+The great idea is, at least conceptually,
+sending back the read data to the write entrance.
+I didn't notice this simple idea at that time of previous discussion.
+
+This should be done background and
+some threshold to determine
+which sequence of data should be staged.
+
+This feature will be implemented step-by-step.
+I don't think this feature will change then interface drastically.
+
+2. Improve initialization time
+Some user complains that the initialization is too slow.
+
+Any ideas?
+
+3. Reduce metadata footprint
+Writeboost has a hash table in RAM for cache management.
+This isn't cheap.

drivers/staging/writeboost/TODO line 46

Quoted 10 lines

drivers/staging/writeboost/TODO new file +10 -0

+
+Any ideas?
+
+-------------------------------------------------------------------------
+
+Please send any patches
+To:
+Akira Hayakawa <[email protected]>
+Cc:
+Greg Kroah-Hartman <[email protected]>

Greg Kroah-Hartman <[email protected]> please, and also the driverdevel mailing list, which is where the rest of the staging patches go.

And, I'll need an ack from the dm maintainers before I can accept this as something they think is ok and will get merged to the "real" part of the kernel someday.

thanks,

greg k-h

Mike and Alasdair,
I need your ack

Quoted 2 lines · Akira Hayakawa

Mike and Alasdair,
I need your ack

Hi Akira,

I just spent some time playing with your latest code. On the positive side I am seeing some good performance with the fio tests. Which is great, we know your design should outperform dm-cache with small random io.

However I'm still getting v. poor results with the git-extract test, which clones a linux kernel repo, and then checks out 5 revisions, all with drop_caches in between.

It's fine to have different benefits of the caching software depending on the load. But I think the worst case should always be close to the performance of the raw spindle device.

If you get the following work items done I will ack to go upstream:

i) Get this test so it's performance is similar to raw spindle.

ii) Write good documentation in Documentation/device-mapper/. eg. How
do I remove a cache? When should I use dm-writeboost rather than bcache or dm-cache?

iii) Provide an equivalent to the fsck tool to repair a damaged cache.

BUT if you'd still like dm-writeboost to go into staging _without_ any of these 5 work items being completed I'll ack it but to be very clear: dm-writeboost will not migrate out of staging until these items are resolved.

Quoted 1 line

i) Get this test so it's performance is similar to raw spindle.

Quoted 5 lines

ii) Write good documentation in Documentation/device-mapper/. eg. How
do I remove a cache? When should I use dm-writeboost rather than bcache or dm-cache?

iii) Provide an equivalent to the fsck tool to repair a damaged cache.

iv) perform full code review to catch various implementation issues,
any style nits, etc.
v) explore/implement read caching support (could the lack of read caching be contributing to why the git_extract test is so poor?)

Quoted 6 lines · Joe Thornber

Quoted 2 lines · Akira Hayakawa

Mike and Alasdair,
I need your ack

Hi Akira,

Quoted 8 lines

I just spent some time playing with your latest code. On the positive side I am seeing some good performance with the fio tests. Which is great, we know your design should outperform dm-cache with small random io.

However I'm still getting v. poor results with the git-extract test, which clones a linux kernel repo, and then checks out 5 revisions, all with drop_caches in between.

Thanks for re-evaluating dm-writeboost performance Joe.

Quoted 6 lines

It's fine to have different benefits of the caching software depending on the load. But I think the worst case should always be close to the performance of the raw spindle device.

If you get the following work items done I will ack to go upstream:

Quoted 7 lines

i) Get this test so it's performance is similar to raw spindle.

ii) Write good documentation in Documentation/device-mapper/. eg. How
do I remove a cache? When should I use dm-writeboost rather than bcache or dm-cache?

iii) Provide an equivalent to the fsck tool to repair a damaged cache.

I agree with this TODO list. But I'd also add:
iv) perform full code review to catch various implementation issues,
any style nits, etc.

v) explore/implement read caching support (could the lack of read caching be contributing to why the git_extract test is so poor?)

Item iv) would be a task for myself and anyone else interested in getting dm-writeboost ready for inclusion. Akira, I can start working on dm-writeboost code review once I complete review of the dm-dedup target (my current priority) -- but realistically that likely won't be until the new year.

BTW, I'm really not seeing much point putting dm-writeboost in staging. I'd be happy to take dm-writeboost into drivers/md/ once the above list is resolved. BUT if you'd still like dm-writeboost to go into staging _without_ any of these 5 work items being completed I'll ack it but to be very clear: dm-writeboost will not migrate out of staging until these items are resolved.

Thanks,
Mike

Writeboost is significantly slower than the spindle alone for this very simple test. I do not understand what is causing the issue.

Quoted 2 lines · Joe Thornber

Writeboost is significantly slower than the spindle alone for this very simple test. I do not understand what is causing the issue.

I started doing the code review and now understand what's going on, sadly.

You are splitting all bios up into 4k blocks to simplify the metadata layout, and mapping logic. This murders performance. File systems and the block layer try really hard to submit the largest bio possible for a reason.

A simple dd in large chunks across your cache reveals this:

raw spindle:        8.9s
writeboost type 0:  32.2s
writeboost type 1:  71.1s

dm-cache and dm-thin do also split io into blocks, but much larger, user configurable blocks. It's still a performance issue for us, which is why I'm using range locking to move away from this bio splitting (eg, recent cache discard patches).

One of the main advantages of a log based metadata layout is you can cope nicely with arbitrarily sized bios. Unlike dm-cache for instance, which has to do a read from the origin if it wants to cache a write that partially covers a block (or maintain a 'valid' bit for each sector of every cached block).

The writeboost target as it stands will only benefit v. small, random io. It will seriously degrade performance of any other IO profile. I'm NACKing this for upstream, and will not be spending any more time on it at this point.

You've put a lot of effort into this so far, so I suggest you redesign the log metadata, and drop the io splitting; you'll end up with something far better.

Sorry,

- Joe

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel

By the way, what environment are you using for those tests?

Hi, Joe

Thanks for continuous evaluation.

So maybe it's just volume of IO that's causing the problem? What's the difference between Type 0 and Type 1? In the code I notice you have 'rambuf' structures, are you caching IO in memory?

Quoted 3 lines · Akira Hayakawa

Hi, Joe

Thanks for continuous evaluation.

Some more details:

dd, with block size 512b
------------------------

raw spindle : 143
writeboost Type 0: 137
writeboost Type 1: 130

dd, with block size 4k
----------------------

raw spindle : 31
writeboost Type 0: 29
writeboost Type 1: 81

dd, with block size 64m
-----------------------

raw spindle : 8.9
writeboost Type 0: 32
writeboost Type 1: 72

So maybe it's just volume of IO that's causing the problem? What's the difference between Type 0 and Type 1? In the code I notice you have 'rambuf' structures, are you caching IO in memory?

Joe,

I appreciate your continuous work.

Is that read or write?
The difference between Type 0 and 1 should only show up in write path. So is it write test?

And what is the unit of each result?

Quoted 4 lines

So maybe it's just volume of IO that's causing the problem? What's the difference between Type 0 and Type 1? In the code I notice you have 'rambuf' structures, are you caching IO in memory?

"rambuf" is a temporary space that every write data comes in. 127*4KB data are once stored there and 4KB metadata section are added then it becomes a log and flushed to the cache device sequentially (512KB each).

By the way,
I think more clearer discussion can be done if tests are done on physical machines to isolate things relevant to VM. I will also add these tests to dmts later and run on my machine.
But, it will be much better if we have good server with RAID-ed backing store and the newest SSD (How would it be if it's PCI-e SSD)...

So you copy the bio payload to a different block of ram and then complete the bio? Or does the rambuf refer to the bio payload directly?

I generally find it quicker to investigate problems on the machine that are actually exhibiting the problem ;) Seriously though, you're asking us to send this upstream; it needs to work on consumer level hardware.

I've got a big machine with Fusion IO storage that I can run the same tests on later.

Quoted 7 lines · Akira Hayakawa

Joe,

I appreciate your continuous work.

Is that read or write?
The difference between Type 0 and 1 should only show up in write path. So is it write test?

Yes, writing across the whole device using 'dd'.

These are the tests:

  dmtest list --suite writeboost -n /wipe_device/

Quoted 1 line

And what is the unit of each result?

seconds.

Quoted 7 lines

Quoted 4 lines

So maybe it's just volume of IO that's causing the problem? What's the difference between Type 0 and Type 1? In the code I notice you have 'rambuf' structures, are you caching IO in memory?

"rambuf" is a temporary space that every write data comes in. 127*4KB data are once stored there and 4KB metadata section are added then it becomes a log and flushed to the cache device sequentially (512KB each).

So you copy the bio payload to a different block of ram and then complete the bio? Or does the rambuf refer to the bio payload directly?

Quoted 6 lines

By the way,
I think more clearer discussion can be done if tests are done on physical machines to isolate things relevant to VM. I will also add these tests to dmts later and run on my machine.
But, it will be much better if we have good server with RAID-ed backing store and the newest SSD (How would it be if it's PCI-e SSD)...

I generally find it quicker to investigate problems on the machine that are actually exhibiting the problem ;) Seriously though, you're asking us to send this upstream; it needs to work on consumer level hardware.

I've got a big machine with Fusion IO storage that I can run the same tests on later.

- Joe

Joe,

Quoted 3 lines

So you copy the bio payload to a different block of ram and then complete the bio? Or does the rambuf refer to the bio payload directly?

Good question.
The answer is, copy the data (got by bio_data(bio)) to rambuf once and ack if it's not barrier things.
It would be nice if data in rambuf points to bio payload but it now copies because bio payload can be reused after completion. Am I right? Is there a way of eliminate memory copying?

Quoted 11 lines · Akira Hayakawa

Joe,

Quoted 3 lines

So you copy the bio payload to a different block of ram and then complete the bio? Or does the rambuf refer to the bio payload directly?

Good question.
The answer is, copy the data (got by bio_data(bio)) to rambuf once and ack if it's not barrier things.
It would be nice if data in rambuf points to bio payload but it now copies because bio payload can be reused after completion. Am I right? Is there a way of eliminate memory copying?

You *have* to eliminate this memory copying. Remap the bios to the relevant portion of your log, and don't complete them until you log chunk is coherent.

Quoted 2 lines · Joe Thornber

Writeboost is significantly slower than the spindle alone for this very simple test. I do not understand what is causing the issue.

I started doing the code review and now understand what's going on, sadly.

You are splitting all bios up into 4k blocks to simplify the metadata layout, and mapping logic. This murders performance. File systems and the block layer try really hard to submit the largest bio possible for a reason.

A simple dd in large chunks across your cache reveals this:

raw spindle:        8.9s
writeboost type 0:  32.2s
writeboost type 1:  71.1s

dm-cache and dm-thin do also split io into blocks, but much larger, user configurable blocks. It's still a performance issue for us, which is why I'm using range locking to move away from this bio splitting (eg, recent cache discard patches).

One of the main advantages of a log based metadata layout is you can cope nicely with arbitrarily sized bios. Unlike dm-cache for instance, which has to do a read from the origin if it wants to cache a write that partially covers a block (or maintain a 'valid' bit for each sector of every cached block).

The writeboost target as it stands will only benefit v. small, random io. It will seriously degrade performance of any other IO profile. I'm NACKing this for upstream, and will not be spending any more time on it at this point.

You've put a lot of effort into this so far, so I suggest you redesign the log metadata, and drop the io splitting; you'll end up with something far better.

Sorry,

- Joe

--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel