dm-lc: plan to go to staging

Hi,

I am happy to see that dm-lc is picked up by LWN.
http://lwn.net/Articles/562938/

Now I think I should consider the next step forward.

As Joe Thornber said before, what I have to do next are
1. Get some feedback from 3rd party users.
2. Reviewed by Mike Snitzer.
As in
http://www.redhat.com/archives/dm-devel/2013-July/msg00137.html

To achieve the first goal,
I am thinking of going to staging tree
for dm-lc to be widely available to people in community.

dm-lc is now managed in Github repo as a kernel module however, getting available in the tree means a lot to get 3rd party users.

Does this plan sound nice?

If so, what is the next step to go to staging? I have a patch against 3.11-rc7 in linux-next locally. Sending the patch to Greg is the next step? And who and what should be included in CC?

For your information,
there is a precedent that DM target
gets in staging tree. That is enhanceio, also a caching target.

Any comments will be appreciated.

Thanks,
Akira

That's fine with me. For staging drivers, I need a TODO file that lists what needs to be done to the code to get it into a mergable state for the "real" part of the kernel, and an email address to cc: patches on. Also, a MAINTAINERS entry is good as that way the scripts/get_maintainer.pl tool will properly pick up your email address.

You should generate the patch against the linux-next tree, as I have lots of new staging drivers in there, ready for 3.12, and if you make a patch against 3.11-rc7, odds are it will not apply properly.

If you have any other questions, please let me know.

thanks,

greg k-h

Two simple requirements before putting your proof-of-concept into staging if you want to work that way:

1) Drop the major version number to 0. Version 1 is reserved for supported modules.

2) Agree a new and meaningful target name with us so you don't have to change it later. "lc" means nothing, I'm afraid.

Then in general terms, you should continue to compare your device-mapper target with the existing targets and where there are differences, either change your target to be like something that already exists, or be ready to explain why that can't or shouldn't be done.

In particular, the interface and architecture will need substantial changes and working these out should be your highest priority.

For example, you write:

Be careful, you MUST create all the LVs as the destinations of the dirty blocks on the cache device before this operation. Otherwise, the kernel may crash.

I read a statement like that as an indication of an interface or architectural problem. The device-mapper approach is to 'design out' problems, rather than relying on users not doing bad things. Study the existing interfaces used by other targets to understand some approaches that proved successful, then decide which ones come closest to your needs.

(Your code also needs many more comments inline to explain what it does and how it works.)

The documentation file will eventually need rewriting to follow the same format as the other targets recently added to the kernel. We document the kernel interface rather than any particular userspace tools, which just have the status of convenient examples.

Another little thing I noticed: look into using something like __dm_bless_for_disk() for your metadata and clearly segregate your on-disk structures and document the layout.

Alasdair

Another idea:

Make the interface of dm-lc (the arguments to constructor, messages and the status line) the same as dm-cache, so that they can be driven by the same userspace code.

Mikulas

Thank you for your constructive comments,

I want to first settle an agreement on the new name for current dm-lc according to Alasdair's comment:

After we agree on this issue, I will start tackling on the design issues. Choosing this issue first is because it may change the whole codebase and rename the Github repo which can be one of the elements in MAINTAINERS file.

Name is very important.

First of all, let me make a tiny objection:
1) The name dm-lc does means. lc is the initials from Log-structured Caching.
2) LC is also the name of my a favorite anime character from "God only knows". I did want to use her name for my creation.

But,
I can agree on Alasdair's opinion that dm-lc means nothing, for the potential future users.
Abbreviation of some technical word to a software name looks nice but the unfortunate is that LC is not widely common like RAID or LFS.

I have three candidates on my mind.
I am thinking of removing the dm- prefix as bcache and enhanceio do.

The Points are:
1) The name should not be too long.
A name explaining too much about the software is noisy.
2) Making clear the characteristics and the difference from other targets. The characteristics to note are:
a) It writes in-coming writes in log-structured manner.
b) It is extremely fast in write.
c) It is caching software.

Candidates:
1) logcache
Not too long but explaining enough
but looks little bit dull to me.
Futhermore, there still is dm-log
and I feel like duplicating a little bit.

2) lscache
lscache sounds cool to me.
But the problem is that there is a preceding one with the same name as to caching web-server workload.
Unique name makes it easy to search for a web documents.

3) writeboost or wboost
Concentrating on the characteristic b)
but doesn't mention cache.

Up to this point,
which do you think the best for the new name?
or other candidate?

I suppose logcache is an acceptable compromise.

Thanks for reading,
Akira

Yes, that's fine for the general way of referring to this. (Within the codebase, the filename and module name would still need the
prefix: we use this for auto-module loading, function name/error message scoping etc.)

Another candidate: 'lcache' - contrast with 'bcache

I quite like that option.

Alasdair

OK,
I would like to rename from dm-lc to dm-writeboost. Do you agree?

Akira

Descriptive but dull naming is a much underrated quality in software packages.

Consider the package HarfBuzz.

As my daughter put it, "Sounds like a bad coffee name".

rpm -qi harfbuzz says:
Description :
HarfBuzz is an implementation of the OpenType Layout engine.

(And don't get me started on the people who name packages enca, fros, serd, yajl, or yaml. We aren't using ASR-33's anymore, brevity isn't as crucial :)

dm-logcache works just fine for me as a name for this. Or dm-lcache.

Hi.

Progress report for staging dm-writeboost.

(1) Renaming to dm-writeboost
Thanks for comments on renaming.

By a simple democratic,
dm-lc will be renamed to dm-writeboost.
I am really happy with the new name.

(2) Idea for smelling limitation on the kind of backing store Now I am tackling on the design issues.
As Alasdair pointed out
the statement below from my document really smells and seems to be the first priority at this point.

Be careful, you MUST create all the LVs as the destinations of the dirty blocks on the cache device before this operation. Otherwise, the kernel may crash.

Aside from the two small requirements by Alasdair,
I may have to shoot out this problem before sending patch to staging tree.

My investigation today revealed that this statement is too pessimistic. The truth is only that
the backing store should not be some partition out of a disk, say /dev/sdd1 out of /dev/sdd.

With the partition number,
my userland script works badly in attaching a device to a cache. It failed to get the correct device name from sysfs.

I am thinking of addressing this problem by
adding new sysfs member to show device number(major:minor) of the backing store which is likely to maintain backward-compatibility and
doing some modification on the Python scripts.

That's very easy.
I will do this before sending a patch for staging
aside said two requirements.

(3) Draft for TODO and MAINTAINERS files
These are the current drafts for TODO and MAINTAINERS files. Please let me know if you find more to write or simple mistakes.

(3-1) TODO file
TODO:
- Get feedback from 3rd party users.
- Reviewed by Mike Snitzer.
- Audit userspace interfaces to make sure they are sane. Should use the same approach that is proven successful.
- Fix document.
Should document the kernel interfaces
rather than particular userspace tools.
- Add more comments inline
to explain what it does and how it works.

Please send patches to Greg Kroah-Hartman <[email protected]> and Cc:
Akira Hayakawa <[email protected]>

(3-2) part of MAINTAINERS file

M:      Greg Kroah-Hartman <[email protected]>
M:      Akira Hayakawa <[email protected]>
S:      Maintained
L:      [email protected]
W:      https://github.com/akiradeveloper/dm-writeboost.git
F:      drivers/staging/dm-writeboost.git

(4) Next steps
- Fixed all source codes and documents as they are of dm-writeboost but not dm-lc.
- Github repo will be renamed.
- Complete the task said at (2).
- Generating a patch against linux-next
and send it to Greg.

Thanks for your reading,
Akira

Thanks, Alasdair.

I will reply to some of your comments
that is not answered up to now.

dm-writeboost can be used by kicking kernel interfaces but it is not recommended.
My userspace tools take care of all the
implications in using the kernel interfaces.
The fact dm-writeboost supports shared-caching and write-back caching with auto-modulated migration
puts the interfaces and implementations under some constraints. That's simply a tradeoff.

OK, I will document the kernel interfaces
and also mention the userspace tools.

Comments are substantially noise is my philosophy. I will add comments with given feedback clearly mentioning what is clear and what is not clear about the source code. Even in that case,
I'd better polish the code rather than write comments.

I couldn't understand what is __dm_bless_for_disk() for. What does the word "bless" mean in this context?

By the way,
dm-cache and dm-writeboost are definitely different in a way that dm-cache has an another disk for metadata while dm-writeboost doesn't. dm-writeboost packs metadata and data within a log and submit it to the cache device like log-structured filesystem.

So, the annotation checks can not be appropriated
to apply to dm-writeboost.

And what about moving these macros to
include/linux/device-mapper.h?

Akira