pohmelfs: call for inclusion

Hi

I'm please to announce new and completely rewritten distributed filesystem - POHMELFS

It went a long way from parallel NFS design which lived in drivers/staging/pohmelfs for years effectively without usage case - that design was dead.

New pohmelfs uses elliptics network [1] as its storage backend, which was proved as effective distributed system. It is build as key/value storage which by default implements distributed hash table design

Elliptics is used in production in Yandex search company for several years now and clusters range from small (like 6 nodes in 3 datacenters to host 15 billions of small files) to rather large (hundred of nodes to scale to 1 Pb used for streaming).

We start to cook up 2 small clusters for production and real-life pohmelfs testing - one of them is largest in East Europe mirror site, another one is used for internal package storage.

Pohmelfs is just a POSIX frontend to elliptics. It supports hardlinks, symlinks, data checksums, multiple copies and all other goodies provided by the elliptics storage (even multiple columns, although I did not export it to userspace - there is no suitable open/read/write/close interface).
Pohmelfs uses local cache (flushed on timely basis) for all operations, and only sync(2)/fsync(2) (or close(2) with 'sync_on_close' mount option) or writeback will flush data to remote nodes. There is also background work to flush data to storage like commit in ext3.
Directory objects are flushed to the storage when they are created on pohmelfs node, file data is being sent from cache later.

All recovery process is handled by elliptics and is performed without pohmelfs clients ever noticing that. In particular reads are always directed to the replica with the latest data (determined according to ellptics metadata checked at file open time). If some of the nodes or replicas are not available, pohmelfs will try to read them from different copies by itself.

Writes in pohmelfs can be configured to succeed when quorum commits it or when specified in 'successful_write_count=' number of writes returned ok. This is useful when you do not care much about number of active replicas, since you know that your data is safe. Or when you create new storage and want to copy to single datacenter to save bandwidth.

We extended readir/lookup mechanims to prefetch inodes during directory scan, this squizes gap between ls -l on local fs and pohmelfs to the minimum. Included readme file shows all mount options with comments.

But that's enough for advertisement. Here is the code.
I do not know whether it is a good idea to ask it for inclusion into mainline tree right now skipping drivers/staging part.
Greg already removed drivers/staging/pohmelfs, so attached patch adds fs/pohmelfs If it is not the way to go, I will resubmit.

1. Elliptics network
http://www.ioremap.net/projects/elliptics
http://www.elliptics.ru (russian support forum - we can has understand english and will answer pohmelfs questions too)

2. Pohmelfs
http://www.ioremap.net/projects/pohmelfs (old and not yep updated site)
http://www.ioremap.net/taxonomy/term/4 (development blog section)

index d621f02..d7b8308 100644
@@ -261,6 +261,7 @@ config NFS_COMMON
 source "net/sunrpc/Kconfig"
 source "fs/ceph/Kconfig"
 source "fs/cifs/Kconfig"
+source "fs/pohmelfs/Kconfig"
 source "fs/ncpfs/Kconfig"
 source "fs/coda/Kconfig"
 source "fs/afs/Kconfig"

index 93804d4..a2a819f 100644
@@ -124,3 +124,4 @@ obj-$(CONFIG_GFS2_FS)           += gfs2/
 obj-y				+= exofs/ # Multiple modules
 obj-$(CONFIG_CEPH_FS)		+= ceph/
 obj-$(CONFIG_PSTORE)		+= pstore/
+obj-$(CONFIG_POHMELFS)		+= pohmelfs/

new file mode 100644
index 0000000..6358362
@@ -0,0 +1,11 @@
+config POHMELFS
+	tristate "POHMELFS distributed filesystem"
+	depends on INET && EXPERIMENTAL
+	select CRYPTO_HASH
+	help
+	  POHMELFS is a POSIX frontend to Elliptics network
+
+	  Elliptics is a key/value storage, which by default implements
+	  distributed hash table structure.
+
+	  More information can be found at http://www.ioremap.net/projects/elliptics

new file mode 100644
index 0000000..f38002d
@@ -0,0 +1,7 @@
+#
+# Makefile for the linux pohmel filesystem routines.
+#
+
+obj-$(CONFIG_POHMELFS) += pohmelfs.o
+
+pohmelfs-y := dir.o file.o inode.o net.o route.o super.o trans.o symlink.o stat.o pool.o

new file mode 100644
index 0000000..e69de29

new file mode 100644
index 0000000..80a9d87

	Evgeniy Polyakov

Part of the diff is not shown here. View the raw message.

Unfortunately, it doesn't compile against -next.

+static int pohmelfs_fill_super(struct super_block *sb, void *data, int silent) []
+ sb->s_root = d_alloc_root(&psb->root->vfs_inode);

commit 4f7e69f8eb314b1dd187026cca2bba5245caf75a

Author: Al Viro <[email protected]>
Date:   Sun Feb 12 22:15:47 2012 -0500

vfs: d_alloc_root() gone

all callers converted to d_make_root() by now

Documentation/filesystems/porting:
[mandatory]
d_alloc_root() is gone, along with a lot of bugs caused by code
misusing it. Replacement: d_make_root(inode). The difference is, d_make_root() drops the reference to inode if dentry allocation fails.

I will update patch, thanks a lot!

	Evgeniy Polyakov

Here is the fixed version

index d621f02..d7b8308 100644
@@ -261,6 +261,7 @@ config NFS_COMMON
 source "net/sunrpc/Kconfig"
 source "fs/ceph/Kconfig"
 source "fs/cifs/Kconfig"
+source "fs/pohmelfs/Kconfig"
 source "fs/ncpfs/Kconfig"
 source "fs/coda/Kconfig"
 source "fs/afs/Kconfig"

index 93804d4..a2a819f 100644
@@ -124,3 +124,4 @@ obj-$(CONFIG_GFS2_FS)           += gfs2/
 obj-y				+= exofs/ # Multiple modules
 obj-$(CONFIG_CEPH_FS)		+= ceph/
 obj-$(CONFIG_PSTORE)		+= pstore/
+obj-$(CONFIG_POHMELFS)		+= pohmelfs/

new file mode 100644
index 0000000..6358362
@@ -0,0 +1,11 @@
+config POHMELFS
+	tristate "POHMELFS distributed filesystem"
+	depends on INET && EXPERIMENTAL
+	select CRYPTO_HASH
+	help
+	  POHMELFS is a POSIX frontend to Elliptics network
+
+	  Elliptics is a key/value storage, which by default implements
+	  distributed hash table structure.
+
+	  More information can be found at http://www.ioremap.net/projects/elliptics

new file mode 100644
index 0000000..f38002d
@@ -0,0 +1,7 @@
+#
+# Makefile for the linux pohmel filesystem routines.
+#
+
+obj-$(CONFIG_POHMELFS) += pohmelfs.o
+
+pohmelfs-y := dir.o file.o inode.o net.o route.o super.o trans.o symlink.o stat.o pool.o

new file mode 100644
index 0000000..e69de29

new file mode 100644
index 0000000..80a9d87

	Evgeniy Polyakov

Part of the diff is not shown here. View the raw message.

If you haven't already, you could setup a public git tree somewhere to make it easier to track changes and make a request for inclusion in linux-next.

I think you just need to send a request to:
Stephen Rothwell <[email protected]>

Hi

I'm cooking up a tree hosted on ioremap.net for tomorrow, which will be rebased against current tree. It will incorporate your -next pohmelfs patches.

If hosting it at ioremap.net is not suitable, then Stephen, I send a request :)

Then I will send next pull request to Linus "always-keeping-silence" Torvalds

	Evgeniy Polyakov

Please don't rebase. Or if you want to make sure it's in clean form, at least rebase it on top of something *stable*, like the v3.3 release. Not whatever random current state during the merge window.

I mainly just curse at people when they do something horribly bad. It's how I vent.

I can't really say that I tend to care deeply about the innards of some unusual filesystem that hopefully cannot possibly impact anything else. So as long as you keep people like Al happy (or at least not overly unhappy), I'm unlikely to worry too much.

Linus

As far as I can see, it's still suffering the same problems it used to; in particular, the games it plays with d_path() are completely bogus. Take a look at this:

static int pohmelfs_construct_path_string(struct pohmelfs_inode *pi, void *data
int len)

{
	struct path path;
	struct dentry *d;
	char *ptr;
	int err = 0, strlen, reduce = 0;

	d = d_find_alias(&pi->vfs_inode);
	if (!d) {
		err = -ENOENT;
		goto err_out_exit;
	}

	spin_lock(&current->fs->lock);
	path.mnt = mntget(current->fs->root.mnt);
	spin_unlock(&current->fs->lock);

	path.dentry = d;

	if (!IS_ROOT(d) && d_unhashed(d))
		reduce = 1;

	ptr = d_path(&path, data, len);
	if (IS_ERR(ptr)) {
		err = PTR_ERR(ptr);
		goto err_out_put;
	}

	if (reduce && len >= UNHASHED_OBSCURE_STRING_SIZE) {
		char *end = data + len - UNHASHED_OBSCURE_STRING_SIZE;
		*end = '\0';
	}

	strlen = len - (ptr - (char *)data);
	memmove(data, ptr, strlen);
	ptr = data;

	err = strlen - 1; /* no including 0-byte */

	pr_debug("%s: dname: '%s', len: %u, maxlen: %u, name: '%s', strlen: %d.\n",
		__func__, d->d_name.name, d->d_name.len, len, ptr, strlen);

err_out_put:

	dput(d);
	mntput(path.mnt);
err_out_exit:
	return err;
}

For Nth time: what happens to poor sod that happens to be chrooted into that animal? We get pathname relative to process' root or to fs root, whichever's closer. And no, it's used not just for debugging printks...

I think I've asked that question at least 3 times. Never got anything resembling an answer... Is that misspelled dentry_path()? Or is something subtle going on and we really want different strings generated for chrooted processes here?

This is a special case which does bad things intentionally. http-compatibility was added in _this_ POHMELFS on demand from people who do want to access files by handle created from whole path. Not name or inode number, but whole path, since that's only what is available in http (or more generally via rest api)

It is limited, wrong and error-prone. It does not even support rename and hardlinks. But that's what people want.
When I bind-remount part of the tree, things 'dissapear' from the tree. Yes, this is really an uglymoron, but it was created for _some_ limited case which rougly work in sandboxed environment only.

The whole rest of the POHMELFS was completely rewritten with very different model, namely we do _not_ depend on pathes in general case. There are fair inodes which are indexed by 2^512 numbers, they are referenced from direntries and so forth.

So, again, treat http-compat mount option (which turns this path-based-lookups) as a ugly handler for those who /skipped by censorship in editor/ know what they are doing.

	Evgeniy Polyakov

IDGI. Again, you are getting different strings for different processes, so that one inside a chroot generates shorter pathnames. I'm not asking about races with rename() et.al. - it's obviously racy, but that's a separate problem. Details, please - as far as I can tell, that code looks like a reimplementation of dentry_path() in a curiously broken way; what demands that particular breakage? Again, the question of pathname stability, uniqueness, etc. is a separate story; why this specific weirdness? Note that you are passing a to d_path() a vfsmount/dentry pair that violates all kinds of assertions - dentry->d_sb != mnt->mnt_sb more often than not, to start with.

http_compat ugly mount options forbids rename
It cuts fair amount of functionality present for 'normal' pohmelfs

Details are pretty simple - we want to allow external applications to get to filesystem and grab data via single requests, since it is stateless and can not hold dentry structure. They do not connect to server which runs on top of filesystem, but insted directly to storage, which hosts raw data.

Applications know they uploadede data via /whatever/path/was/to/the/file And they want to get that data from server via single 'get'. Obviously they can not store mapping from all filenames to inode number, and they can not request dozen of directory lookups, since it takes time and has to maintain state.

When object was written via remounted path, then it is a problem for those who made a setup - this ugly hack only 'works' in specially crafted environment, which provides its pros and requires fair price of cons.

Generic POHMELFS does not require this compatibility mode.

	Evgeniy Polyakov

Sigh... I wish it hadn't been an English-speaking maillist; mat is hard to translate properly...

OK, let me try for a printable version: suppose we replace that d_path() call with dentry_path() and leave everything else as is; what exactly will be broken and how will it break?

_What_ remounted path? I'm not talking about bindings at all...

Argh, undecency is pretty useful /most of the time/frequently/

I didn't understand you at first. Replacing it with dentry_path() will not break anything. It was not supposed fs should care about chroot for this case - every application (including chrooted) writes into own namespace, so if it changes root, it is on its own...

I believe you will?

Actually if this useful hack is so much a PITA I will drop it. Or fix with dentry_path() instead. It doesn't really deserve _that_ much.

	Evgeniy Polyakov