POHMELFS - network filesystem with local coherent cache.

Hi.

POHMELFS stands for Parallel Optimized Host Message Exchange Layered File System. It allows to mount remote servers to local directory via network. This filesystem supports local caching and writeback flushing.
POHMELFS is a brick in a future distributed filesystem.

This set includes two patches:
* network filesystem with write-through cache (slow, but works with remote userspace server)
* hack to show how local cache works and how faster it is compared to async NFS (see below). hack disables writeback flush and performs local allocation of the objects only.

Now, some vaporware aka food for thoughts and your brains.

A small benchmark of the local cached mode (above hack):

$ time tar -xf /home/zbr/threading.tar

POHMELFS NFS v3 (async)
real 0m0.043s 0m1.679s

Which is damn 40 times!

Excited? Now get huge bucket with ice.

Generic problem with writeback cache is a fact, that all local objects has to have IDs in sync with remote side. For example, if remote side is ext3, local one should not overwrite inode with number 0. Contrary write-through cache allows to request remote side about what ID should given data have and be in sync. This one is slow.

Of course this will not be _that_ huge difference in a real world, when tested archives are larger (this one if a git archive of my userspace threading library), which is very small. Since it is so small there is no writeback cache flushing, and thus remote side never receives data.

Actually one can consider this as tmpfs or something like that. Code supports sync, but since inode generation process is very different, files and dirs can not be blindly synced to the ext3. So, this release of POHMELFS consists of two patches: first one is a network filesystem implementation with write-through cache, when object is first created on the remote side and then populated to the local cache. This one is slow.

Second patch is a hack to disable writeback caching and implement local caching only, which is very fast.

Next task is to think about how to generically solve the problem with syncing local changes with remote server, when remote server maintains inodes with completely different numbers.
This, among others, will allow offline work with automatic syncing after reconnect.

This is not intended for inclusion, CRFS by Zach Brown is a bit ahead of POHMELFS, but it is not generic enough (because of above problem), works only with BTRFS, and was closed by Oracle so far :)
So, anyone who managed to read up to this and happend to be at LCA 08 just has to move this Friday to his presentation.

POHMELFS TODO list includes:

* mechanism of keeping it coherent with other users
* unified method of syncing with various remote filesystems

Thank you.

P.S. POHMELFS is about one month old, so do not be so severe with it :)

index f9eed6d..c40f2c5 100644
@@ -1519,6 +1519,8 @@ endmenu
 menu "Network File Systems"
 	depends on NET
 
+source "fs/pohmelfs/Kconfig"
+
 config NFS_FS
 	tristate "NFS file system support"
 	depends on INET

index 720c29d..8fff82a 100644
@@ -118,3 +118,4 @@ obj-$(CONFIG_HPPFS)		+= hppfs/
 obj-$(CONFIG_DEBUG_FS)		+= debugfs/
 obj-$(CONFIG_OCFS2_FS)		+= ocfs2/
 obj-$(CONFIG_GFS2_FS)           += gfs2/
+obj-$(CONFIG_POHMELFS)          += pohmelfs/

new file mode 100644
index 0000000..ac19aac
@@ -0,0 +1,6 @@
+config POHMELFS
+	tristate "POHMELFS filesystem support"
+	help
+	  POHMELFS stands for Parallel Optimized Host Message Exchange Layered File System.
+	  This is a network filesystem which supports coherent caching of data and metadata
+	  on clients.

new file mode 100644
index 0000000..8a87f46
@@ -0,0 +1,3 @@
+obj-$(CONFIG_POHMELFS)	+= pohmelfs.o
+
+pohmelfs-y := inode.o config.o dir.o net.o

	Evgeniy Polyakov

This patch disables writeback in POHMELFS and creates all objects on behalf of its own without sync with remote side. This mode is _very_ fast.

If POHEMLFS would be bound to single remote filesystem, it could use its inode allocation policy and be very happy with write-back cache. By design POHMELFS is a transport layer in distributed filesystem, which will work with some or other remote filesystem (likely completely new one), so instead of stupid algorithm shown here, it will contain correct object creation.

Likely the way to go is to use name hash with parent inode number as unique ID, which can be matched to filesystem path, so that remote side could create objects without _any_ knowledge of inode numbers on the local fs.

index 23f9ecd..5aec593 100644
@@ -80,6 +80,8 @@ static struct pohmelfs_name *pohmelfs_insert_offset(struct pohmelfs_inode *pi,
 	rb_link_node(&new->offset_node, parent, n);
 	rb_insert_color(&new->offset_node, &pi->offset_root);
 
+	pi->total_len += new->len;
+
 	return NULL;
 }
 

@@ -647,6 +649,7 @@ static int pohmelfs_create_entry(struct inode *dir, struct dentry *dentry, u64 s
 	cmd->start = start;
 	netfs_set_cmd_flags(cmd, dentry->d_name.hash, mode);
 
+#if 0
 	netfs_convert_cmd(cmd);
 
 	err = netfs_data_send(st, cmd, sizeof(struct netfs_cmd));

@@ -666,6 +669,30 @@ static int pohmelfs_create_entry(struct inode *dir, struct dentry *dentry, u64 s
 	err = netfs_recv_inode_info(psb, POHMELFS_I(dir), &npi, data);
 	if (err < 0)
 		goto err_out_unlock;
+#else
+	{
+		static u64 pohmelfs_ino = 123;
+
+		st->info.mode = netfs_get_inode_mode(cmd);
+		st->info.ino = pohmelfs_ino++;
+		st->info.nlink = 2;
+		st->info.uid = 2319;
+		st->info.gid = 100;
+
+		cmd->ino = st->info.ino;
+		cmd->start = POHMELFS_I(dir)->total_len;
+	}
+
+	npi = pohmelfs_new_inode(psb, POHMELFS_I(dir), data, cmd, &st->info);
+	if (IS_ERR(npi)) {
+		err = PTR_ERR(npi);
+		if (err != -EEXIST)
+			goto err_out_unlock;
+		npi = NULL;
+	} else
+		err = 0;
+	npi->state = 1;
+#endif
 	mutex_unlock(&st->lock);
 
 	d_add(dentry, &npi->vfs_inode);

index b0ee0b3..6a81bdc 100644
@@ -125,6 +125,16 @@ static int netfs_process_page(struct file *file, struct page *page, __u64 cmd_op
 	int err;
 	void *addr;
 
+	{
+		if (cmd_op == NETFS_READ_PAGE) {
+			if (file)
+				file->f_pos += cmd->size;
+		}
+		SetPageUptodate(page);
+		unlock_page(page);
+		return 0;
+	}
+
 	mutex_lock(&st->lock);
 
 	cmd->ino = inode->i_ino;

@@ -305,6 +315,7 @@ static struct inode *pohmelfs_alloc_inode(struct super_block *sb)
 
 	inode->state = 0;
 	inode->parent = 0;
+	inode->total_len = 0;
 
 	return &inode->vfs_inode;
 }

index 23aa953..b719fbe 100644
@@ -163,6 +163,8 @@ struct pohmelfs_inode
 	u64			ino;
 	u64			parent;
 
+	u64			total_len;
+
 	struct pohmelfs_name	name;
 
 	struct inode		vfs_inode;

	Evgeniy Polyakov

A brick is usually something that is in the way - Or you also say "the user has bricked his machine" when it's quite unusable :)
Hope you did not mean /that/.

Needs a bigger data set to compare. But what is much more
important: does it use a single port for networing, or some firewall-unfriendly-by-default multiple dynamic-port-allocation like NFS?

What will happen when both nodes change an inode in disconnected state? Which inode wins out?

btrfs is all we need :p

Where's the parallelism that is advertised by the "POH" in pohmelfs?

Hi.

No, this brick as a building block :)

It uses single port, configurable at mount time.
POHMELFS client can connect to different addresses (including ipv6) and via different protocols (like sctp). Metadata server will provide that information dynamically, so pohmelfs client will be able to connect to different nodes and perform operations in parallell.

Who will be online first. Second node will be told that there is a merge collision and it has to be resolved by hands.

Well, at least it has some very interesting ideas. Although there are things which are not that good imho, time will show, maybe there will be another state-of-the-art filesystem at the moment...

This was for information.

First, clients work with local caches and sync them either in writeback or via cache coherency algorithm. This work is effectively parallel. Second, pohmelfs as in distributed filesystem is developed as a transport layer to eliminate mount operation for each different node, so that after client asks for data it would be just sent to different server. This allows to make parallel transactions. Essentially it looks like mounting different remote server to virtual directory working with it, except that connection setup should be done not at mount time, but at run time.

	Evgeniy Polyakov