Skip to contentSkip to conversation

pohmelfs: call for inclusion

PATCH Evgeniy Polyakov 3 participants 7 messages Feb 8, 2012 – Feb 10, 2012

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi

I'm please to announce new and completely rewritten distributed filesystem - POHMELFS

It went a long way from parallel NFS design which lived in drivers/staging/pohmelfs for years effectively without usage case - that design was dead.

New pohmelfs uses elliptics network [1] as its storage backend, which was proved as effective distributed system. Elliptics is used in production in Yandex search company for several years now and clusters range from small (like 6 nodes in 3 datacenters to host 15 billions of small files or hundred of nodes to scale to 1 Pb used for streaming).

We start to cook up 2 small clusters for production pohmelfs testing - one of them is largest in East Europe mirror site, another one is used for internal package storage.

Pohmelfs is just a POSIX frontend to elliptics. It supports hardlinks, symlinks, data checksums, multiple copies and so on. Pohmelfs uses local cache (synced on timely basis) for all operations, and only sync (or close with sync_on_close mount option) or writeback will flush data to remote nodes. There is also background work to flush data to storage like commit in ext3
Directory objects are synced to the storage when they are created on pohmelfs node, file data is being sent from cache later.

All recovery process is handled by elliptics and is performed without pohmelfs clients every noticing that. In particular reads are always directed to the replica with the latest data (determined according to ellptics metadata checked at file open time).

Writes in pohmelfs can be configured to succeed when quorum commits it or when specified in 'successful_write_count=' number of writes return ok. This is useful when you do not care much about number of active replicas, since you know that your data is safe. Or when you create new storage and want to copy to single datacenter to save bandwidth.

But that's enough for advertisement. Here is the code.
I do not know whether it is a good idea to ask it for inclusion into mainline tree right now skipping drivers/staging part. But anyway, I post patch to remove drivers/staging/pohmelfs and add fs/pohmelfs If it is not the way to go, I will resubmit.

1. Elliptics network
http://www.ioremap.net/projects/elliptics
http://www.elliptics.ru (russian support forum - we understand english and will answer pohmelfs questions too)

2. Pohmelfs
http://www.ioremap.net/projects/pohmelfs (old and not yep updated site)
http://www.ioremap.net/taxonomy/term/4 (development blog section)

Signature · 1 line
	Evgeniy Polyakov
fs/Kconfig +1 -0
index 9fe0b34..7232749 100644
@@ -259,6 +259,7 @@ config NFS_COMMON
 source "net/sunrpc/Kconfig"
 source "fs/ceph/Kconfig"
 source "fs/cifs/Kconfig"
+source "fs/pohmelfs/Kconfig"
 source "fs/ncpfs/Kconfig"
 source "fs/coda/Kconfig"
 source "fs/afs/Kconfig"
fs/Makefile +1 -0
index afc1096..36664fe 100644
@@ -123,3 +123,4 @@ obj-$(CONFIG_GFS2_FS)           += gfs2/
 obj-$(CONFIG_EXOFS_FS)          += exofs/
 obj-$(CONFIG_CEPH_FS)		+= ceph/
 obj-$(CONFIG_PSTORE)		+= pstore/
+obj-$(CONFIG_POHMELFS)		+= pohmelfs/
fs/pohmelfs/Kconfig new file +11 -0
new file mode 100644
index 0000000..b91e56d
@@ -0,0 +1,11 @@
+config POHMELFS
+	tristate "POHMELFS distributed filesystem"
+	depends on INET && EXPERIMENTAL
+	select CRYPTO_HASH
+	help
+	  POHMELFS is a POSIX frontend to Elliptics network
+
+	  Elliptics is a key/value storage, which by default imlpements
+	  distributed hash table structure.
+
+	  More information can be found at http://www.ioremap.net/projects/elliptics
fs/pohmelfs/Makefile new file +7 -0
new file mode 100644
index 0000000..ad358d7
@@ -0,0 +1,7 @@
+#
+# Makefile for the linux ext2-filesystem routines.
+#
+
+obj-$(CONFIG_POHMELFS) += pohmelfs.o
+
+pohmelfs-y := dir.o file.o inode.o net.o route.o super.o trans.o symlink.o
fs/pohmelfs/Module.symvers new file +0 -0
new file mode 100644
index 0000000..e69de29
fs/pohmelfs/dir.c new file +844 -0

844 lines not on this page · raw message

fs/pohmelfs/file.c new file +470 -0

470 lines not on this page · raw message

fs/pohmelfs/inode.c new file +900 -0

900 lines not on this page · raw message

fs/pohmelfs/net.c new file +611 -0

611 lines not on this page · raw message

fs/pohmelfs/packet.h new file +752 -0

146 lines not on this page · raw message

fs/pohmelfs/pohmelfs.h new file +461 -0

461 lines not on this page · raw message

fs/pohmelfs/route.c new file +279 -0

279 lines not on this page · raw message

fs/pohmelfs/super.c new file +905 -0

905 lines not on this page · raw message

fs/pohmelfs/symlink.c new file +13 -0
new file mode 100644
index 0000000..80a9d87

Lines of this file are not shown. View the raw message.

fs/pohmelfs/trans.c new file +384 -0

384 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

commit 0a1cf67228562021535f3d2d35ff8aa198af0eba

Author: Evgeniy Polyakov <[email protected]>
Date:   Wed Feb 8 23:12:08 2012 +0400

Removed drivers/staging/pohmelfs

drivers/staging/Kconfig +0 -2
index 06c9081..a7bf415 100644

Lines of this file are not shown. View the raw message.

drivers/staging/Makefile +0 -1
index f3c5e33..6dce1c3 100644

Lines of this file are not shown. View the raw message.

drivers/staging/pohmelfs/Kconfig deleted +0 -28
deleted file mode 100644
index 58158b8..0000000

Lines of this file are not shown. View the raw message.

drivers/staging/pohmelfs/Makefile deleted +0 -3
deleted file mode 100644
index 196561c..0000000

Lines of this file are not shown. View the raw message.

drivers/staging/pohmelfs/config.c deleted +0 -611

611 lines not on this page · raw message

drivers/staging/pohmelfs/crypto.c deleted +0 -878

878 lines not on this page · raw message

drivers/staging/pohmelfs/dir.c deleted +0 -1101

1,101 lines not on this page · raw message

drivers/staging/pohmelfs/inode.c deleted +0 -2056

2,056 lines not on this page · raw message

drivers/staging/pohmelfs/lock.c deleted +0 -182

182 lines not on this page · raw message

drivers/staging/pohmelfs/mcache.c deleted +0 -171

171 lines not on this page · raw message

drivers/staging/pohmelfs/net.c deleted +0 -1209

1,209 lines not on this page · raw message

drivers/staging/pohmelfs/netfs.h deleted +0 -919

919 lines not on this page · raw message

drivers/staging/pohmelfs/path_entry.c deleted +0 -120

120 lines not on this page · raw message

drivers/staging/pohmelfs/trans.c deleted +0 -706

706 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Part of the diff is not shown here. View the raw message.

maintainer
#1 lore raw
Quoted 5 lines · Evgeniy Polyakov

But that's enough for advertisement. Here is the code.
I do not know whether it is a good idea to ask it for inclusion into mainline tree right now skipping drivers/staging part. But anyway, I post patch to remove drivers/staging/pohmelfs and add fs/pohmelfs If it is not the way to go, I will resubmit.

It would be easiest to just send a patch deleting the old drivers/staging/pohmelfs, or just ask me to do it, which I can, and then send a new patch just with the new code, as it is so different.

That should make it easier to review, and also small enough to fit through the mailing list size limits :)

thanks,

greg k-h

#2 lore raw
Quoted 9 lines · Greg KH
Quoted 5 lines

But that's enough for advertisement. Here is the code.
I do not know whether it is a good idea to ask it for inclusion into mainline tree right now skipping drivers/staging part. But anyway, I post patch to remove drivers/staging/pohmelfs and add fs/pohmelfs If it is not the way to go, I will resubmit.

It would be easiest to just send a patch deleting the old drivers/staging/pohmelfs, or just ask me to do it, which I can, and then send a new patch just with the new code, as it is so different.

pohmelfs-staging-remove.diff does hust that - it removes drivers/staging/pohmelfs
pohmelfs.diff adds new code into fs/pohmelfs

Or did they fail to leak into maillist because of the size? If so, they are also accessible via
http://www.ioremap.net/tmp/pohmelfs.diff (142 Kb)
http://www.ioremap.net/tmp/pohmelfs-staging-remove.diff (200 Kb)

I also embedd pohmelfs.diff here:

fs/Kconfig +1 -0
index 9fe0b34..7232749 100644
@@ -259,6 +259,7 @@ config NFS_COMMON
 source "net/sunrpc/Kconfig"
 source "fs/ceph/Kconfig"
 source "fs/cifs/Kconfig"
+source "fs/pohmelfs/Kconfig"
 source "fs/ncpfs/Kconfig"
 source "fs/coda/Kconfig"
 source "fs/afs/Kconfig"
fs/Makefile +1 -0
index afc1096..36664fe 100644
@@ -123,3 +123,4 @@ obj-$(CONFIG_GFS2_FS)           += gfs2/
 obj-$(CONFIG_EXOFS_FS)          += exofs/
 obj-$(CONFIG_CEPH_FS)		+= ceph/
 obj-$(CONFIG_PSTORE)		+= pstore/
+obj-$(CONFIG_POHMELFS)		+= pohmelfs/
fs/pohmelfs/Kconfig new file +11 -0
new file mode 100644
index 0000000..b91e56d
@@ -0,0 +1,11 @@
+config POHMELFS
+	tristate "POHMELFS distributed filesystem"
+	depends on INET && EXPERIMENTAL
+	select CRYPTO_HASH
+	help
+	  POHMELFS is a POSIX frontend to Elliptics network
+
+	  Elliptics is a key/value storage, which by default imlpements
+	  distributed hash table structure.
+
+	  More information can be found at http://www.ioremap.net/projects/elliptics
fs/pohmelfs/Makefile new file +7 -0
new file mode 100644
index 0000000..ad358d7
@@ -0,0 +1,7 @@
+#
+# Makefile for the linux ext2-filesystem routines.
+#
+
+obj-$(CONFIG_POHMELFS) += pohmelfs.o
+
+pohmelfs-y := dir.o file.o inode.o net.o route.o super.o trans.o symlink.o
fs/pohmelfs/Module.symvers new file +0 -0
new file mode 100644
index 0000000..e69de29
fs/pohmelfs/dir.c new file +844 -0

844 lines not on this page · raw message

fs/pohmelfs/file.c new file +470 -0

470 lines not on this page · raw message

fs/pohmelfs/inode.c new file +900 -0

900 lines not on this page · raw message

fs/pohmelfs/net.c new file +611 -0

611 lines not on this page · raw message

fs/pohmelfs/packet.h new file +752 -0

146 lines not on this page · raw message

fs/pohmelfs/pohmelfs.h new file +461 -0

461 lines not on this page · raw message

fs/pohmelfs/route.c new file +279 -0

279 lines not on this page · raw message

fs/pohmelfs/super.c new file +905 -0

905 lines not on this page · raw message

fs/pohmelfs/symlink.c new file +13 -0
new file mode 100644
index 0000000..80a9d87

Lines of this file are not shown. View the raw message.

fs/pohmelfs/trans.c new file +384 -0

384 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
	Evgeniy Polyakov

Part of the diff is not shown here. View the raw message.

maintainer
#3 lore raw
Quoted 11 lines · Evgeniy Polyakov
Quoted 9 lines · Greg KH
Quoted 5 lines

But that's enough for advertisement. Here is the code.
I do not know whether it is a good idea to ask it for inclusion into mainline tree right now skipping drivers/staging part. But anyway, I post patch to remove drivers/staging/pohmelfs and add fs/pohmelfs If it is not the way to go, I will resubmit.

It would be easiest to just send a patch deleting the old drivers/staging/pohmelfs, or just ask me to do it, which I can, and then send a new patch just with the new code, as it is so different.

Quoted 8 lines

pohmelfs-staging-remove.diff does hust that - it removes drivers/staging/pohmelfs
pohmelfs.diff adds new code into fs/pohmelfs

Or did they fail to leak into maillist because of the size? If so, they are also accessible via
http://www.ioremap.net/tmp/pohmelfs.diff (142 Kb)
http://www.ioremap.net/tmp/pohmelfs-staging-remove.diff (200 Kb)

I need a signed-off-by: for any patch :)

Care to resend me just the remove patch, with the proper changelog and
signed-off-by: so that I can queue it up?

thanks,

greg k-h

#4 lore raw
Quoted 4 lines · Greg KH

I need a signed-off-by: for any patch :)

Care to resend me just the remove patch, with the proper changelog and
signed-off-by: so that I can queue it up?

Done

Signature · 1 line
	Evgeniy Polyakov
#5 lore raw
In reply to Evgeniy Polyakov

fs/pohmelfs/Kconfig line 8

Quoted 7 lines · Evgeniy Polyakov
fs/pohmelfs/Kconfig new file +7 -0
+	tristate "POHMELFS distributed filesystem"
+	depends on INET && EXPERIMENTAL
+	select CRYPTO_HASH
+	help
+	  POHMELFS is a POSIX frontend to Elliptics network
+
+	  Elliptics is a key/value storage, which by default imlpements

typo fix: implements

Quoted 1 line
fs/pohmelfs/Kconfig new file +1 -0
+	  distributed hash table structure.
#6 lore raw
Quoted 9 lines · On Thu, Feb 09, 2012 at 07:54:15PM -0500, [email protected]…

fs/pohmelfs/Kconfig line 8

Quoted 7 lines
fs/pohmelfs/Kconfig new file +7 -0
+	tristate "POHMELFS distributed filesystem"
+	depends on INET && EXPERIMENTAL
+	select CRYPTO_HASH
+	help
+	  POHMELFS is a POSIX frontend to Elliptics network
+
+	  Elliptics is a key/value storage, which by default imlpements

typo fix: implements

fs/pohmelfs/Kconfig line 9

Quoted 1 line
fs/pohmelfs/Kconfig new file +1 -0
+	  distributed hash table structure.

Fixed, thank you :)

Signature · 1 line
	Evgeniy Polyakov