Skip to contentSkip to conversation

pohmelfs: call for inclusion

take 2 Evgeniy Polyakov 2 participants 2 messages Feb 14, 2012 VFS

lore
Share & tools
lore
#0 lore raw

Hi

I'm please to announce new and completely rewritten distributed filesystem - POHMELFS

It went a long way from parallel NFS design which lived in drivers/staging/pohmelfs for years effectively without usage case - that design was dead.

New pohmelfs uses elliptics network [1] as its storage backend, which was proved as effective distributed system. It is build as key/value storage which by default implements distributed hash table design

Elliptics is used in production in Yandex search company for
several years now and clusters range from small
(like 6 nodes in 3 datacenters to host 15 billions of
small files) to rather large (hundred of nodes to scale to 1 Pb used for streaming).

We start to cook up 2 small clusters for production and real-life pohmelfs testing - one of them is largest in East Europe mirror site, another one is used for internal package storage.

Pohmelfs is just a POSIX frontend to elliptics. It supports hardlinks, symlinks, data checksums, multiple copies and all other goodies provided by the elliptics storage (even multiple columns, although I did not export it to userspace - there is no suitable open/read/write/close interface).
Pohmelfs uses local cache (flushed on timely basis) for all operations, and only sync(2)/fsync(2) (or close(2) with 'sync_on_close' mount option) or writeback will flush data to remote nodes. There is also background work to flush data to storage like commit in ext3.
Directory objects are flushed to the storage when they are created on pohmelfs node, file data is being sent from cache later.

All recovery process is handled by elliptics and is performed without pohmelfs clients ever noticing that. In particular reads are always directed to the replica with the latest data (determined according to ellptics metadata checked at file open time). If some of the nodes or replicas are not available, pohmelfs will try to read them from different copies by itself.

Writes in pohmelfs can be configured to succeed when quorum commits it or when specified in 'successful_write_count=' number of writes returned ok. This is useful when you do not care much about number of active replicas, since you know that your data is safe. Or when you create new storage and want to copy to single datacenter to save bandwidth.

But that's enough for advertisement. Here is the code.
I do not know whether it is a good idea to ask it for inclusion into mainline tree right now skipping drivers/staging part.
Greg already removed drivers/staging/pohmelfs, so attached patch adds fs/pohmelfs If it is not the way to go, I will resubmit.

1. Elliptics network
http://www.ioremap.net/projects/elliptics
http://www.elliptics.ru (russian support forum - we can has understand english and will answer pohmelfs questions too)

2. Pohmelfs
http://www.ioremap.net/projects/pohmelfs (old and not yep updated site)
http://www.ioremap.net/taxonomy/term/4 (development blog section)

fs/Kconfig +1 -0
index d621f02..d7b8308 100644
@@ -261,6 +261,7 @@ config NFS_COMMON
 source "net/sunrpc/Kconfig"
 source "fs/ceph/Kconfig"
 source "fs/cifs/Kconfig"
+source "fs/pohmelfs/Kconfig"
 source "fs/ncpfs/Kconfig"
 source "fs/coda/Kconfig"
 source "fs/afs/Kconfig"
fs/Makefile +1 -0
index 93804d4..a2a819f 100644
@@ -124,3 +124,4 @@ obj-$(CONFIG_GFS2_FS)           += gfs2/
 obj-y				+= exofs/ # Multiple modules
 obj-$(CONFIG_CEPH_FS)		+= ceph/
 obj-$(CONFIG_PSTORE)		+= pstore/
+obj-$(CONFIG_POHMELFS)		+= pohmelfs/
fs/pohmelfs/Kconfig new file +11 -0
new file mode 100644
index 0000000..6358362
@@ -0,0 +1,11 @@
+config POHMELFS
+	tristate "POHMELFS distributed filesystem"
+	depends on INET && EXPERIMENTAL
+	select CRYPTO_HASH
+	help
+	  POHMELFS is a POSIX frontend to Elliptics network
+
+	  Elliptics is a key/value storage, which by default implements
+	  distributed hash table structure.
+
+	  More information can be found at http://www.ioremap.net/projects/elliptics
fs/pohmelfs/Makefile new file +7 -0
new file mode 100644
index 0000000..21b76ce
@@ -0,0 +1,7 @@
+#
+# Makefile for the linux pohmel filesystem routines.
+#
+
+obj-$(CONFIG_POHMELFS) += pohmelfs.o
+
+pohmelfs-y := dir.o file.o inode.o net.o route.o super.o trans.o symlink.o stat.o
fs/pohmelfs/Module.symvers new file +0 -0
new file mode 100644
index 0000000..e69de29
fs/pohmelfs/README new file +71 -0

71 lines not on this page · raw message

fs/pohmelfs/dir.c new file +843 -0

843 lines not on this page · raw message

fs/pohmelfs/file.c new file +473 -0

473 lines not on this page · raw message

fs/pohmelfs/inode.c new file +902 -0

902 lines not on this page · raw message

fs/pohmelfs/net.c new file +611 -0

611 lines not on this page · raw message

fs/pohmelfs/packet.h new file +752 -0

71 lines not on this page · raw message

fs/pohmelfs/pohmelfs.h new file +476 -0

476 lines not on this page · raw message

fs/pohmelfs/route.c new file +317 -0

317 lines not on this page · raw message

fs/pohmelfs/stat.c new file +139 -0

139 lines not on this page · raw message

fs/pohmelfs/super.c new file +968 -0

968 lines not on this page · raw message

fs/pohmelfs/symlink.c new file +13 -0
new file mode 100644
index 0000000..80a9d87

Lines of this file are not shown. View the raw message.

fs/pohmelfs/trans.c new file +384 -0

384 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
	Evgeniy Polyakov

Part of the diff is not shown here. View the raw message.

#1 lore raw
Quoted 4 lines · Evgeniy Polyakov

Hi

I'm please to announce new and completely rewritten distributed filesystem - POHMELFS

<snip>

You also might want to cc: linux-fsdevel on new filesystems, as that's where the developers who can review this properly live.

thanks,

greg k-h