POHMELFS is back
Hi.
POHMELFS is a distributed filesystem originally created as a cache-coherent pNFS. Practice showed that parallel NFS as well as NFS itself is not a way to go because of scalability issues, thus POHMELFS never got out of staging dir.
So I rewrote POHMELFS from scratch as a frontend to Elliptics network.
Elliptics is a distributed key/value storage, which by default implements hash table system. It has datacenter-aware replica management, does not have single points of failure like any kind of master node/server, column store (with SELECT-like range requests), compression support and so on.
First production elliptics cluster was deployed about 2 years ago, it is close to 1 Pb (around 200 storage nodes in 4 datacenters) now with more that 4 Gb/s of bandwidth from each datacenter,
another one with the biggest number of objects hosts about 400+ millions (not counting that each of them has 3 copies) in 3 different datacenters. Currently there are 5 big clusters in production.
I run a small ask-and-talk presentation today at YaC conference in Moscow.
POHMELFS currently is rather alpha version, since it does not support object removal as well as does not balance reads between multiple groups. Also directory structure is not scalable enough, so it will be changed when elliptics starts supporting server-side scripting (in Lua), which also will be used by elliptics for internal mapreduce implementation.
This patch is aimed for initial review and discussion. Thank you.
1. Elliptics network
http://www.ioremap.net/projects/elliptics
fs/Kconfig +1 -0
@@ -259,6 +259,7 @@ config NFS_COMMON source "net/sunrpc/Kconfig" source "fs/ceph/Kconfig" source "fs/cifs/Kconfig" +source "fs/pohmelfs/Kconfig" source "fs/ncpfs/Kconfig" source "fs/coda/Kconfig" source "fs/afs/Kconfig"
fs/Makefile +1 -0
@@ -123,3 +123,4 @@ obj-$(CONFIG_GFS2_FS) += gfs2/ obj-$(CONFIG_EXOFS_FS) += exofs/ obj-$(CONFIG_CEPH_FS) += ceph/ obj-$(CONFIG_PSTORE) += pstore/ +obj-$(CONFIG_POHMELFS) += pohmelfs/
fs/pohmelfs/Kconfig new file +11 -0
@@ -0,0 +1,11 @@ +config POHMELFS + tristate "POHMELFS distributed filesystem" + depends on INET && EXPERIMENTAL + select CRYPTO_HASH + help + POHMELFS is a POSIX frontend to Elliptics network + + Elliptics is a key/value storage, which by default imlpements + distributed hash table structure. + + More information can be found at http://www.ioremap.net/projects/elliptics
fs/pohmelfs/Makefile new file +7 -0
@@ -0,0 +1,7 @@ +# +# Makefile for the linux ext2-filesystem routines. +# + +obj-$(CONFIG_POHMELFS) += pohmelfs.o + +pohmelfs-y := dir.o file.o inode.o net.o route.o super.o trans.o
fs/pohmelfs/Module.symvers new file +0 -0
fs/pohmelfs/dir.c new file +574 -0
574 lines not on this page · raw message
fs/pohmelfs/file.c new file +103 -0
103 lines not on this page · raw message
fs/pohmelfs/inode.c new file +620 -0
620 lines not on this page · raw message
fs/pohmelfs/net.c new file +527 -0
527 lines not on this page · raw message
fs/pohmelfs/packet.h new file +627 -0
627 lines not on this page · raw message
fs/pohmelfs/pohmelfs.h new file +333 -0
333 lines not on this page · raw message
fs/pohmelfs/route.c new file +274 -0
187 lines not on this page · raw message
fs/pohmelfs/super.c new file +515 -0
515 lines not on this page · raw message
fs/pohmelfs/trans.c new file +319 -0
319 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
Evgeniy Polyakov
Part of the diff is not shown here. View the raw message.
On Mon, 19 Sep 2011 10:13:02 +0400, Evgeniy Polyakov said:
Please don't take the below in the wrong way - I *realize* that POHMELFS is still staging/ caliber, and my official $DAYJOB is provisioning storage for our HPC people. Mostly giving you a feel for what sort of scaling you need to be thinking about as you continue development...
Quoted 3 lines
Elliptics is a distributed key/value storage, which by default implements hash table system. It has datacenter-aware replica management,
Can you please define "datacenter-aware"? I've sat through a few too many buzzword-full but content-free vendor presentations. ;)
Quoted 2 lines
First production elliptics cluster was deployed about 2 years ago, it is close to 1 Pb (around 200 storage nodes in 4 datacenters) now with
Somehow, I'm not terribly thrilled with the idea of provisioning an entire storage node with CPUs and memory and an OS image for every 5Tb of disk. But then, I've currently got about 1Pb of DDN storage behind a 6-node GPFS cluster, and another 1Pb+ of DDN disk currently coming online in a CXFS/DMF configuration..
Quoted 1 line
more that 4 Gb/s of bandwidth from each datacenter,
Also not at all impressive per-node if we're talking an average of 50 nodes per data center. I'm currently waiting for some 10GigE to be provisioned at the moment because we're targeting close to a giga*byte*/sec per server.
Quoted 2 lines
POHMELFS currently is rather alpha version, since it does not support object removal
I'm sure the storage vendors don't mind that. :)
A quick scan of the actual patch:
+ Elliptics is a key/value storage, which by default imlpements
+ distributed hash table structure.
typo - implements.
+ struct kref refcnt;
+
+ /* if set, all received inodes will be attached to dentries in parent dir */
+ int load_all;
+
+ /* currently read object name */
+ u32 namelen;
+
+ /* currently read inode info */
+ long offset;
Ouch. Usual kernel style would be:
int load_all; /* if set, all received inodes will be attached to dentries in parent dir */ u32 namelen; /* currently read object name */
long offset; /* currently read inode info */
I suspect it's just one programmer doing this, as it only happens in a few places and other places it's done the usual kernel way.
+static ssize_t pohmelfs_write(struct file *filp, const char __user *buf, size_t len, loff_t *ppos) +{
+ ssize_t err;
+ struct inode *inode = filp->f_mapping->host;
+#if 0
+ struct inode *inode = filp->f_mapping->host;
Just remove the #if 0'ed code.
in phhmelfs_fill_inode() (and probably other places):
+ pr_info("pohmelfs: %s: ino: %lu inode is regular: %d, dir: %d, link: %d, mode: %o, "
pr_debug please. pr_info per inode reference is just insane.
+void pohmelfs_print_addr(struct sockaddr_storage *addr, const char *fmt, ...)
+ pr_info("pohmelfs: %pI4:%d: %s", &sin->sin_addr.s_addr, ntohs(sin->sin_port), ptr);
Gaak. This apparently gets called *per read*. pr_debug *and* additional "please spam my log" flags please.
+static inline int dnet_id_cmp_str(const unsigned char *id1, const unsigned char *id2) +{
+ unsigned int i = 0;
+
+ for (i*=sizeof(unsigned long); i<DNET_ID_SIZE; ++i) {
strncmp()?
Also, as a general comment - since this is an interface to Elliptics, which as far as I can tell runs in userspace, would this whole thing make more sense using FUSE?
I'm also assuming that Elliptics is responsible for all the *hard* parts of distributed filesystems, like quorum management and re-synching after a partition of the network, and so on? If so, you really need to discuss that some more - in particular how well this all works during failure modes.
Quoted 7 lines · On Mon, Sep 19, 2011 at 02:10:51PM -0400, [email protected]…
Quoted 3 lines · Evgeniy Polyakov
Elliptics is a distributed key/value storage, which by default implements hash table system. It has datacenter-aware replica management,
Can you please define "datacenter-aware"? I've sat through a few too many buzzword-full but content-free vendor presentations. ;)
Elliptics allows to save your replicas in different datacenters and read data either randomly from different groups or from the copy which is closer to you. We also support 'weights' of the servers, so it is possible to read data from the replica, which is hosted on the server with highest weight.
Quoted 9 lines
Quoted 2 lines
First production elliptics cluster was deployed about 2 years ago, it is close to 1 Pb (around 200 storage nodes in 4 datacenters) now with
Somehow, I'm not terribly thrilled with the idea of provisioning an entire storage node with CPUs and memory and an OS image for every 5Tb of disk. But then, I've currently got about 1Pb of DDN storage behind a 6-node GPFS cluster, and another 1Pb+ of DDN disk currently coming online in a CXFS/DMF configuration..
Quoted 5 lines
Quoted 1 line
more that 4 Gb/s of bandwidth from each datacenter,
Also not at all impressive per-node if we're talking an average of 50 nodes per data center. I'm currently waiting for some 10GigE to be provisioned at the moment because we're targeting close to a giga*byte*/sec per server.
If you get 10 times more bandwidth you will not be able to saturate it with 10 times less servers. Scaling to hundreds of server nodes is a good result, since we evenly balance all IO between nodes and no single server is disk or network bound.
Quoted 4 lines
Quoted 2 lines
POHMELFS currently is rather alpha version, since it does not support object removal
I'm sure the storage vendors don't mind that. :)
Actually yes, we use garbage collection to actually remove data and do not really care if it is not removed, since disk volumes and prices allow not to care about disk space. But yet its kinda good idea to remove some objects from listing when logic suggest to remove it :)
Quoted 66 lines
A quick scan of the actual patch:
fs/pohmelfs/Kconfig new file +2
-0+ Elliptics is a key/value storage, which by default imlpements + distributed hash table structure.typo - implements.
fs/pohmelfs/dir.c new file +10
-0+struct kref refcnt; + +/* if set, all received inodes will be attached to dentries in parent dir */ +int load_all; + +/* currently read object name */ +u32 namelen; + +/* currently read inode info */ +long offset;Ouch. Usual kernel style would be:
int
load_all; /* if set, all received inodes will be attached to dentries in parent dir */ u32 namelen; /* currently read object name */
long offset; /* currently read inode info */I suspect it's just one programmer doing this, as it only happens in a few places and other places it's done the usual kernel way.
fs/pohmelfs/file.c new file +6
-0+static ssize_t pohmelfs_write (struct file *filp,const char __user *buf,size_t len,loff_t *ppos) +{ +ssize_t err; +struct inode *inode = filp->f_mapping->host; +#if 0 +struct inode *inode = filp->f_mapping->host;Just remove the #if 0'ed code.
in
phhmelfs_fill_inode()(and probably other places):fs/pohmelfs/inode.c new file +1
-0+ pr_info("pohmelfs: %s: ino: %lu inode is regular: %d, dir: %d, link: %d, mode: %o, "
pr_debugplease.pr_infoper inode reference is just insane.fs/pohmelfs/net.c new file +2
-0+void pohmelfs_print_addr(struct sockaddr_storage *addr, const char *fmt, ...) + pr_info("pohmelfs: %pI4:%d: %s", &sin->sin_addr.s_addr, ntohs(sin->sin_port), ptr);Gaak. This apparently gets called *per read*.
pr_debug*and* additional "please spam my log" flags please.fs/pohmelfs/pohmelfs.h new file +5
-0+static inline int dnet_id_cmp_str (const unsigned char *id1,const unsigned char *id2) +{ +unsigned int i =0 ; + +for (i*=sizeof (unsigned long ); i<DNET_ID_SIZE ; ++i) {
strncmp()?Also, as a general comment - since this is an interface to Elliptics, which as far as I can tell runs in userspace, would this whole thing make more sense using FUSE?
I'm also assuming that Elliptics is responsible for all the *hard* parts of distributed filesystems, like quorum management and re-synching after a partition of the network, and so on? If so, you really need to discuss that some more - in particular how well this all works during failure modes.
Signature · 1 line
Evgeniy Polyakov
Quoted 9 lines · Evgeniy Polyakov
Quoted 5 lines · On Mon, Sep 19, 2011 at 02:10:51PM -0400, [email protected]…
Quoted 1 line · Evgeniy Polyakov
more that 4 Gb/s of bandwidth from each datacenter,
Also not at all impressive per-node if we're talking an average of 50 nodes per data center. I'm currently waiting for some 10GigE to be provisioned at the moment because we're targeting close to a giga*byte*/sec per server.
If you get 10 times more bandwidth you will not be able to saturate it with 10 times less servers.
The point is that the solutions we're looking at are able to drive enough I/O *per server* that we need to look at 10GigE and Infiniband connections. Your numbers currently indicate about 5T of disk and 75 megabit of throughput per node, while current solutions are doing about 100T and pushing a 10GigE per node. So you have a *lot* of per-server scaling work to do still...
Quoted 3 lines
Scaling to hundreds of server nodes is a
good result, since we evenly balance all IO between nodes and no single server is disk or network bound.
You missed the point. Scaling to hundreds of server nodes is a nice *theoretical* result, but one that's not going to get a lot of traction out in the real world, where the *per server* scaling matters too. Which is my boss more likely to be willing to spend money on - a solution that has 50 servers per datacenter to deliver 4 Gb/sec per data center, or one that is delivering that much *per server*? Remember - servers cost money, rack space costs money, hardware maintenance contracts cost money, electricity and cooling cost money , network connectivity costs money, sysadmin time to manage each node costs money
- so in the real world, the solution that uses the smallest number of servers to get to the target aggregate is probably going to win.
Looked at differently - if I'm currently targeting multiple gigabytes/sec throughput to a petabyte of disk from a half-dozen servers, how big and fast a disk farm could I build if I had 50 servers in the room, or 200 across datacenters?
Quoted 8 lines · On Tue, Sep 20, 2011 at 09:41:57AM -0400, [email protected]…
Quoted 2 lines
If you get 10 times more bandwidth you will not be able to saturate it with 10 times less servers.
The point is that the solutions we're looking at are able to drive enough I/O *per server* that we need to look at 10GigE and Infiniband connections. Your numbers currently indicate about 5T of disk and 75 megabit of throughput per node, while current solutions are doing about 100T and pushing a 10GigE per node. So you have a *lot* of per-server scaling work to do still...
Number of server nodes is smaller, and number of physical servers may be even less. There is a fair number of proxy servers for cluster.
But overall of course every server does not saturate own 1gige link, since, well, our uplinks are just gigabits :)
Quoted 10 lines
Quoted 3 lines
Scaling to hundreds of server nodes is a
good result, since we evenly balance all IO between nodes and no single server is disk or network bound.You missed the point. Scaling to hundreds of server nodes is a nice *theoretical* result, but one that's not going to get a lot of traction out in the real world, where the *per server* scaling matters too. Which is my boss more likely to be willing to spend money on - a solution that has 50 servers per datacenter to deliver 4 Gb/sec per data center, or one that is delivering that much *per server*? Remember - servers cost money, rack space costs money,
You are not able to setup 1 server and deliver 4Gb/sec of random IO. If you think this is possible, than you actualyl did not try to do it with existing solutions.
Quoted 3 lines
Looked at differently - if I'm currently targeting multiple gigabytes/sec throughput to a petabyte of disk from a half-dozen servers, how big and fast a disk farm could I build if I had 50 servers in the room, or 200 across datacenters?
A simple question, what RPS rate you got for random reads and writes?
Your solution may scale to bandwifth limits, which is not interesting for us. Huge single-or-small-node solution is random IO limited, but if you read big file, then you will be network limited, and can show nice numbers of Gb/ solution is random IO limited, but if you read big file, then you will be network limited, and can show nice numbers of Gb/s.
As of GPFS you mentioned, then you did not try to setup cluster with weak links (i.e. between physically different datacenters), since it resynchronizes nodes on every glitch and does not scale to RPS, although quite good ad bulk IO.
So, basically, Elliptics was created for low-latency RPS loads and not BULK IO.
Signature · 1 line
Evgeniy Polyakov
Code part comments
Quoted 8 lines · On Mon, Sep 19, 2011 at 02:10:51PM -0400, [email protected]…
fs/pohmelfs/file.c new file +6
-0+static ssize_t pohmelfs_write (struct file *filp,const char __user *buf,size_t len,loff_t *ppos) +{ +ssize_t err; +struct inode *inode = filp->f_mapping->host; +#if 0 +struct inode *inode = filp->f_mapping->host;Just remove the #if 0'ed code.
Actually I need to remove other part, it is kind of debugging part :)
Quoted 4 lines
in
phhmelfs_fill_inode()(and probably other places):fs/pohmelfs/inode.c new file +1
-0+ pr_info("pohmelfs: %s: ino: %lu inode is regular: %d, dir: %d, link: %d, mode: %o, "
pr_debugplease.pr_infoper inode reference is just insane.
Yes, I will clean up all prints for sure
Quoted 5 lines
fs/pohmelfs/net.c new file +2
-0+void pohmelfs_print_addr(struct sockaddr_storage *addr, const char *fmt, ...) + pr_info("pohmelfs: %pI4:%d: %s", &sin->sin_addr.s_addr, ntohs(sin->sin_port), ptr);Gaak. This apparently gets called *per read*.
pr_debug*and* additional "please spam my log" flags please.
Please do not be scared by log prints ,they will be cleanup up for release
Quoted 7 lines
fs/pohmelfs/pohmelfs.h new file +5
-0+static inline int dnet_id_cmp_str (const unsigned char *id1,const unsigned char *id2) +{ +unsigned int i =0 ; + +for (i*=sizeof (unsigned long ); i<DNET_ID_SIZE ; ++i) {
strncmp()?
Actually yes
Quoted 3 lines
Also, as a general comment - since this is an interface to Elliptics, which as far as I can tell runs in userspace, would this whole thing make more sense using FUSE?
I tried FUSE about 3 years ago, it had _serious_ problems with performance. Disk storage can run in userspace, since it is never limited by CPU or copies, but client may not be allowed to waste that much resources.
Quoted 4 lines
I'm also assuming that Elliptics is responsible for all the *hard* parts of distributed filesystems, like quorum management and re-synching after a partition of the network, and so on? If so, you really need to discuss that some more - in particular how well this all works during failure modes.
Yes, POHMELFS is just an another interface.
Elliptics uses eventual consistency model, so when replicas go out of sync, special checking process starts to bring them back. It also recovers missed copy, if we replaced disk/server or added new one.
When some servers go down, all IO is handled by their neighbours according to route tabe generated on the first start. When servers are turned on again, they start getting those IO requests, and eventual consistency process rolls out missed writes. If data is old or missed we will switch to another replica.
Thanks a lot for review!
Signature · 1 line
Evgeniy Polyakov
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 1 line · Rick Morgan
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
Please do not use inline html
Quoted 14 lines
<pre wrap="">+static inline int
dnet_id_cmp_str(const unsigned char *id1, const unsigned char *id2)fs/pohmelfs/pohmelfs.h new file +12
-0+{ + unsigned int <b>i = 0</b>; + + for (<b>i*=sizeof</b>(unsigned long); i<DNET_ID_SIZE; ++i) { + if (id1[i] < id2[i]) +return -1 ; + if (id1[i] > id2[i]) +return 1 ; + } + +return 0 ; +} </pre>
Quoted 2 lines
</blockquote>
if i is initialized to 0, wouldn't i*=sizeof(unsigned long) be zero?<br>
Yes, it counts from zero. This chunk previously checked another parameter first, but since then it was removed, but checking loop remained.
Signature · 1 line
Evgeniy Polyakov