Skip to contentSkip to conversation

FUSE/CUSE: implement direct mmap, take#3

PATCH New Tejun Heo 5 participants 22 messages Jun 18, 2009 – Jul 9, 2009 1 message not in the archive

Subsystems (4)
lore
Share & tools
lore

Tejun Heo

author
#0 lore raw

PATCH 1/1 FUSE/CUSE: implement direct mmap, take#3

Hello,

This is the third take of fuse-implement-direct-mmap patchset. This patchset implements direct mmap support for FUSE (and CUSE). Each direct mmap area is backed by anonymous mapping (shmem_file) and the FUSE server can decide how they are shared.

mmap request is handled in two steps. MMAP first queries the server whether it wants to share the mapping with an existing one or create a new one, and if so, with which flags. MMAP_COMMIT notifies the server the result of mmap and if successful the fd the server can use to access the mmap region.

Changes from the last take[L] are

* mmap-don-t-assume-f_op-mmap-doesn-t-change-vma.patch was unnecessary. Dropped and mmap implementation updated accordingly.

* Updated to the current master which now contains CUSE support.

This patchset contains the following patches.

0001-fdtable-export-alloc_fd.patch 0002-FUSE-make-request_wait_answer-wait-for-end-co.patch 0003-FUSE-implement-fuse_req-prep.patch 0004-FUSE-implement-direct-mmap.patch

0001 update exports fdtable for following FUSE changes. 0002-0003 update fuse_req->end() handling and add ->prep(). 0004 implements direct mmap.

This patchset is on top of

linus#master(1d89b30cc9be41af87881682ec82e2c107849dbe)

and is available in the following git tree.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git fuse-mmap

diffstat follows.

 fs/file.c            |    1 
 fs/fuse/cuse.c       |    1 
 fs/fuse/dev.c        |   70 ++++++--
 fs/fuse/file.c       |  432 +++++++++++++++++++++++++++++++++++++++++++++++++--
 fs/fuse/fuse_i.h     |   19 ++
 include/linux/fuse.h |   47 +++++
 6 files changed, 538 insertions(+), 32 deletions(-)

Thanks.

--
tejun

[L] http://thread.gmane.org/gmane.linux.kernel/821839

author
#1 lore raw

PATCH 1/4 fdtable: export alloc_fd()

Export alloc_fd(). Will be used by FUSE.

---
 fs/file.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)
fs/file.c +1 -0
index f313314..806b3ad 100644
@@ -487,6 +487,7 @@ out:
 	spin_unlock(&files->file_lock);
 	return error;
 }
+EXPORT_SYMBOL_GPL(alloc_fd);
 
 int get_unused_fd(void)
 {
Signature · 1 line
1.6.0.2
#2 lore raw

Re: [PATCH 1/4] fdtable: export alloc_fd()

Quoted 1 line · Tejun Heo

Export alloc_fd(). Will be used by FUSE.

Where and how?

author
#3 lore raw

Re: [PATCH 1/4] fdtable: export alloc_fd()

Hello, Al.

Quoted 4 lines · Al Viro
Quoted 1 line · Tejun Heo

Export alloc_fd(). Will be used by FUSE.

Where and how?

This currently is used via get_unused_fd_flags() in fuse_mmap_commit_prep(). A shmem_file is created in the requesting client's context (for accounting and easier flow of control) and then passed to the server's context where an fd is allocated and assigned to the file so that the fd can be passed to the userland server.

The shmem_file based implementation serves certain device mmap file emulation well but as Miklos pointed out it may not be sufficiently flexible for other purposes and passing file back and forth and wrapping shmem_file is quite ugly, so I think there's pretty good chance it will get reimplemented in some different way which likely won't need alloc_fd() exported. If there's any vfs related changes, I'll cc you.

Thanks.

Signature · 1 line
tejun
#4 lore raw

PATCH 1/1 implement direct mmap support

In reply to Tejun Heo

This patch implements direct mmap. It allows FUSE server to honor each mmap request with anonymous mapping. FUSE server can make multiple mmap requests share a single anonymous mapping or separate mappings as it sees fit.

mmap request is handled in two steps. MMAP first queries the server whether it wants to share the mapping with an existing one or create a new one, and if so, with which flags. MMAP_COMMIT notifies the server the result of mmap and if successful the fd the server can use to access the mmap region.

---
Here's updated direct mmap implementation in libfuse. Thanks.

 example/Makefile.am     |    6 
 example/fmmap.c         |  370 ++++++++++++++++++++++++++++++++++++++++++++++++
 example/fmmapclient.c   |  112 ++++++++++++++
 include/cuse_lowlevel.h |    8 +
 include/fuse.h          |   70 +++++++++
 include/fuse_common.h   |    9 +
 include/fuse_kernel.h   |   47 ++++++
 include/fuse_lowlevel.h |   76 +++++++++
 lib/cuse_lowlevel.c     |   30 +++
 lib/fuse.c              |  129 ++++++++++++++++
 lib/fuse_lowlevel.c     |   65 ++++++++
 lib/fuse_versionscript  |    4 
 12 files changed, 924 insertions(+), 2 deletions(-)
include/fuse.h +70 -0
Index: fuse/include/fuse.h
===================================================================
@@ -493,6 +493,67 @@ struct fuse_operations {
 	 */
 	int (*poll) (const char *, struct fuse_file_info *,
 		     struct fuse_pollhandle *ph, unsigned *reventsp);
+
+	/**
+	 * Direct mmap
+	 *
+	 * Direct mmap is available for direct_io files.  FUSE direct
+	 * mmap is always backed by anonymous mapping which can be
+	 * shared between different maps.
+	 *
+	 * mmap is done in two steps.  mmap() is the first step and
+	 * queries the server whether an existing mmap should be
+	 * reused or a new one should be created and if a new area is
+	 * to be created with which flags.
+	 *
+	 * The server can set *@fdp to an existing fd to share the map
+	 * and set flags on *@flagsp for new map.  Note that setting
+	 * both *@fdp and *@flagsp makes *@flagsp setting ignored.
+	 * The fd set in *@fdp must be from previous FUSE mmaps.
+	 *
+	 * Introduced in version 2.9
+	 */
+	int (*mmap) (const char *, void *addr, size_t len, int prot, int flags,
+		     off_t offset, struct fuse_file_info *fi,
+		     int *fdp, unsigned *flagsp);
+
+	/**
+	 * Direct mmap commit
+	 *
+	 * This method is called to commit a direct mmap.  This method
+	 * is guaranteed to be called after successful return from
+	 * mmap().  If @fd is negative, it contains -errno and
+	 * indicates internal error occurred and the mmap request is
+	 * being failed.  Otherwise, @fd is connected to the anonymous
+	 * mapping which will be used for the mmap.  The server is
+	 * free to resize or fill the area as it likes.  If @fd has
+	 * been created for this mmap, it will be visible to the
+	 * client only after mmap_commit() completes successfully.
+	 *
+	 * On mmap_commit() failure, the kernel will automatically
+	 * close @fd if it has been created for this mmap.
+	 *
+	 * The server can associate MMAP_COMMIT with the respective
+	 * MMAP using @mmap_unique which contains request unique ID of
+	 * the MMAP request.
+	 */
+	int (*mmap_commit) (const char *, void *addr, size_t len,
+			    int prot, int flags, int fd, off_t offset,
+			    struct fuse_file_info *fi, uint64_t mmap_unique);
+
+	/**
+	 * Direct munmap
+	 *
+	 * This method is called when a direct mmap area is being
+	 * unmapped.
+	 *
+	 * The server can associate MUNMAP with the respective MMAP or
+	 * MMAP_COMMIT using @mmap_unique which contains request
+	 * unique ID of the MMAP request.
+	 */
+	void (*munmap) (const char *, void *addr, size_t len,
+			struct fuse_file_info *fi, uint64_t mmap_unique,
+			int fd);
 };
 
 /** Extra context that may be needed by some filesystems
@@ -730,6 +791,15 @@ int fuse_fs_ioctl(struct fuse_fs *fs, co
 int fuse_fs_poll(struct fuse_fs *fs, const char *path,
 		 struct fuse_file_info *fi, struct fuse_pollhandle *ph,
 		 unsigned *reventsp);
+int fuse_fs_mmap(struct fuse_fs *fs, const char *path, void *addr, size_t len,
+		 int prot, int flags, off_t offset, struct fuse_file_info *fi,
+		 int *fdp, unsigned *flagsp);
+int fuse_fs_mmap_commit(struct fuse_fs *fs, const char *path, void *addr,
+			size_t len, int prot, int flags, int fd, off_t offset,
+			struct fuse_file_info *fi, uint64_t mmap_unique);
+void fuse_fs_munmap(struct fuse_fs *fs, const char *path,
+		    void *addr, size_t len, struct fuse_file_info *fi,
+		    uint64_t mmap_unique, int fd);
 void fuse_fs_init(struct fuse_fs *fs, struct fuse_conn_info *conn);
 void fuse_fs_destroy(struct fuse_fs *fs);
 
include/fuse_kernel.h +47 -0

65 lines not on this page · raw message

include/fuse_lowlevel.h +76 -0

88 lines not on this page · raw message

lib/fuse_lowlevel.c +65 -0

84 lines not on this page · raw message

lib/fuse_versionscript +4 -0
Index: fuse/lib/fuse_versionscript
===================================================================
@@ -169,6 +169,10 @@ FUSE_2.8 {
 		fuse_reply_ioctl;
 		fuse_reply_ioctl_iov;
 		fuse_reply_ioctl_retry;
+		fuse_fs_mmap;
+		fuse_fs_mmap_commit;
+		fuse_fs_munmap;
+		fuse_reply_mmap;
 		fuse_reply_poll;
 
 	local:
example/Makefile.am +4 -2
Index: fuse/example/Makefile.am
===================================================================
@@ -3,7 +3,7 @@
 AM_CPPFLAGS = -I$(top_srcdir)/include -D_FILE_OFFSET_BITS=64 -D_REENTRANT
 noinst_HEADERS = fioc.h
 noinst_PROGRAMS = fusexmp fusexmp_fh null hello hello_ll fioc fioclient \
-		  fsel fselclient cusexmp
+		  fsel fselclient fmmap fmmapclient cusexmp
 
 LDADD = ../lib/libfuse.la @libfuse_libs@
 fusexmp_fh_LDADD = ../lib/libfuse.la ../lib/libulockmgr.la @libfuse_libs@
@@ -14,4 +14,6 @@ fioclient_LDADD =
 fselclient_CPPFLAGS =
 fselclient_LDFLAGS =
 fselclient_LDADD =
-
+fmmapclient_CPPFLAGS =
+fmmapclient_LDFLAGS =
+fmmapclient_LDADD =
example/fmmap.c new file +370 -0

370 lines not on this page · raw message

include/fuse_common.h +9 -0
Index: fuse/include/fuse_common.h
===================================================================
@@ -109,6 +109,15 @@ struct fuse_file_info {
 #define FUSE_IOCTL_MAX_IOV	256
 
 /**
+ * Mmap flags
+ *
+ * FUSE_MMAP_DONT_COPY: don't copy the region on fork
+ * FUSE_MMAP_DONT_EXPAND: can't be expanded with mremap()
+ */
+#define FUSE_MMAP_DONT_COPY	(1 << 0)
+#define FUSE_MMAP_DONT_EXPAND	(1 << 1)
+
+/**
  * Connection information, passed to the ->init() method
  *
  * Some of the elements are read-write, these can be changed to
example/fmmapclient.c new file +112 -0

112 lines not on this page · raw message

include/cuse_lowlevel.h +8 -0
Index: fuse/include/cuse_lowlevel.h
===================================================================
@@ -63,6 +63,14 @@ struct cuse_lowlevel_ops {
 		       const void *in_buf, size_t in_bufsz, size_t out_bufsz);
 	void (*poll) (fuse_req_t req, struct fuse_file_info *fi,
 		      struct fuse_pollhandle *ph);
+	void (*mmap) (fuse_req_t req, void *addr, size_t len, int prot,
+		      int flags, off_t offset, struct fuse_file_info *fi);
+	void (*mmap_commit) (fuse_req_t req, void *addr, size_t len, int prot,
+			     int flags, int fd, off_t offset,
+			     struct fuse_file_info *fi, uint64_t mmap_unique);
+	void (*munmap) (fuse_req_t req, void *addr, size_t len,
+			struct fuse_file_info *fi, uint64_t mmap_unique,
+			int fd);
 };
 
 struct fuse_session *cuse_lowlevel_new(struct fuse_args *args,
lib/fuse.c +129 -0

147 lines not on this page · raw message

lib/cuse_lowlevel.c +30 -0
Index: fuse/lib/cuse_lowlevel.c
===================================================================
@@ -93,6 +93,33 @@ static void cuse_fll_poll(fuse_req_t req
 	req_clop(req)->poll(req, fi, ph);
 }
 
+static void cuse_fll_mmap(fuse_req_t req, fuse_ino_t ino, void *addr,
+			  size_t len, int prot, int flags, off_t offset,
+			  struct fuse_file_info *fi)
+{
+	(void)ino;
+	req_clop(req)->mmap(req, addr, len, prot, flags, offset, fi);
+}
+
+static void cuse_fll_mmap_commit(fuse_req_t req, fuse_ino_t ino,
+				 void *addr, size_t len, int prot, int flags,
+				 int fd, off_t offset,
+				 struct fuse_file_info *fi,
+				 uint64_t mmap_unique)
+{
+	(void)ino;
+	req_clop(req)->mmap_commit(req, addr, len, prot, flags, fd, offset, fi,
+				   mmap_unique);
+}
+
+static void cuse_fll_munmap(fuse_req_t req, fuse_ino_t ino,
+			    void *addr, size_t len, struct fuse_file_info *fi,
+			    uint64_t mmap_unique, int fd)
+{
+	(void)ino;
+	req_clop(req)->munmap(req, addr, len, fi, mmap_unique, fd);
+}
+
 static size_t cuse_pack_info(int argc, const char **argv, char *buf)
 {
 	size_t size = 0;
@@ -169,6 +196,9 @@ struct fuse_session *cuse_lowlevel_new(s
 	lop.fsync	= clop->fsync		? cuse_fll_fsync	: NULL;
 	lop.ioctl	= clop->ioctl		? cuse_fll_ioctl	: NULL;
 	lop.poll	= clop->poll		? cuse_fll_poll		: NULL;
+	lop.mmap	= clop->mmap		? cuse_fll_mmap		: NULL;
+	lop.mmap_commit	= clop->mmap_commit	? cuse_fll_mmap_commit	: NULL;
+	lop.munmap	= clop->munmap		? cuse_fll_munmap	: NULL;
 
 	se = fuse_lowlevel_new_common(args, &lop, sizeof(lop), userdata);
 	if (!se) {
author
#5 lore raw

PATCH 2/4 FUSE: make request_wait_answer() wait for ->end() completion

In reply to Tejun Heo

Previously, a request was marked FINISHED before ->end() is executed and thus request_wait_answer() can return before it's done. This patch makes request_wait_answer() wait for ->end() to finish before returning.

Note that no current ->end() user waits for request completion, so this change doesn't cause any behavior difference.

While at it, beef up the comment above ->end() hook and clarify when and where it's called.

---
 fs/fuse/dev.c    |   41 +++++++++++++++++++++++++----------------
 fs/fuse/fuse_i.h |    5 ++++-
 2 files changed, 29 insertions(+), 17 deletions(-)
fs/fuse/dev.c +25 -16
index 8fed2ed..0745728 100644
@@ -278,7 +278,6 @@ __releases(&fc->lock)
 	req->end = NULL;
 	list_del(&req->list);
 	list_del(&req->intr_entry);
-	req->state = FUSE_REQ_FINISHED;
 	if (req->background) {
 		if (fc->num_background == FUSE_MAX_BACKGROUND) {
 			fc->blocked = 0;
@@ -293,10 +292,21 @@ __releases(&fc->lock)
 		fc->active_background--;
 		flush_bg_queue(fc);
 	}
+
 	spin_unlock(&fc->lock);
-	wake_up(&req->waitq);
-	if (end)
+
+	if (end) {
 		end(fc, req);
+		smp_wmb();
+	}
+
+	/*
+	 * We own this request and wake_up() has enough memory
+	 * barrier, no need to grab spin lock to set state.
+	 */
+	req->state = FUSE_REQ_FINISHED;
+
+	wake_up(&req->waitq);
 	fuse_put_request(fc, req);
 }
 
@@ -372,17 +382,16 @@ __acquires(&fc->lock)
 		return;
 
  aborted:
-	BUG_ON(req->state != FUSE_REQ_FINISHED);
-	if (req->locked) {
-		/* This is uninterruptible sleep, because data is
-		   being copied to/from the buffers of req.  During
-		   locked state, there mustn't be any filesystem
-		   operation (e.g. page fault), since that could lead
-		   to deadlock */
-		spin_unlock(&fc->lock);
-		wait_event(req->waitq, !req->locked);
-		spin_lock(&fc->lock);
-	}
+	spin_unlock(&fc->lock);
+	wait_event(req->waitq, req->state == FUSE_REQ_FINISHED);
+	/*
+	 * This is uninterruptible sleep, because data is being copied
+	 * to/from the buffers of req.  During locked state, there
+	 * mustn't be any filesystem operation (e.g. page fault),
+	 * since that could lead to deadlock
+	 */
+	wait_event(req->waitq, !req->locked);
+	spin_lock(&fc->lock);
 }
 
 void fuse_request_send(struct fuse_conn *fc, struct fuse_req *req)
@@ -1062,9 +1071,7 @@ __acquires(&fc->lock)
 
 		req->aborted = 1;
 		req->out.h.error = -ECONNABORTED;
-		req->state = FUSE_REQ_FINISHED;
 		list_del_init(&req->list);
-		wake_up(&req->waitq);
 		if (end) {
 			req->end = NULL;
 			__fuse_get_request(req);
@@ -1074,6 +1081,8 @@ __acquires(&fc->lock)
 			fuse_put_request(fc, req);
 			spin_lock(&fc->lock);
 		}
+		req->state = FUSE_REQ_FINISHED;
+		wake_up(&req->waitq);
 	}
 }
 
fs/fuse/fuse_i.h +4 -1
index aaf2f9f..7b930b6 100644
@@ -292,7 +292,10 @@ struct fuse_req {
 	/** Link on fi->writepages */
 	struct list_head writepages_entry;
 
-	/** Request completion callback */
+	/** Request completion callback.  This function is called from
+	    the kernel context of the FUSE server if the request isn't
+	    being aborted.  If the request is being aborted, it's
+	    called from the kernel context of the aborting process. */
 	void (*end)(struct fuse_conn *, struct fuse_req *);
 
 	/** Request is stolen from fuse_file->reserved_req */
Signature · 1 line
1.6.0.2

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#7 lore raw

Re: [PATCH 2/4] FUSE: make request_wait_answer() wait for ->end() completion

(restoring cc list)

Quoted 13 lines · Andrew Morton

fs/fuse/fuse_i.h struct fuse_req { · line 298

Quoted 5 lines · On Thu, 18 Jun 2009 18:24:31 +0900 Tejun Heo <[email protected]> wrote:
fs/fuse/fuse_i.h +4 -1
-	/** Request completion callback */
+	/** Request completion callback.  This function is called from
+	    the kernel context of the FUSE server if the request isn't
+	    being aborted.  If the request is being aborted, it's
+	    called from the kernel context of the aborting process. */

The comment pretends to be kerneldoc but isn't.

Like this:

Quoted 8 lines

/*
* Request completion callback. This function is called from
* the kernel context of the FUSE server if the request isn't
* being aborted. If the request is being aborted, it's
* called from the kernel context of the aborting process. */

please.

All the sorrounding comments were formatted in false-kerneldoc format, so it will look a bit weird that way. Hmm... I'll add a patch to change the surrounding comments too.

Thanks.

Signature · 1 line
tejun
maintainer
#8 lore raw

Re: [PATCH 2/4] FUSE: make request_wait_answer() wait for ->end() completion

In reply to Tejun Heo
Quoted 4 lines · Tejun Heo

Previously, a request was marked FINISHED before ->end() is executed and thus request_wait_answer() can return before it's done. This patch makes request_wait_answer() wait for ->end() to finish before returning.

Why is this change needed?

Quoted 6 lines

Note that no current ->end() user waits for request completion, so this change doesn't cause any behavior difference.

While at it, beef up the comment above ->end() hook and clarify when and where it's called.

OK.

[snip]

fs/fuse/dev.c __releases(&fc->lock) · line 300

Quoted 12 lines
fs/fuse/dev.c +4 -2
@@ -293,10 +292,21 @@ __releases(&fc->lock)
 		fc->active_background--;
 		flush_bg_queue(fc);
 	}
+
 	spin_unlock(&fc->lock);
-	wake_up(&req->waitq);
-	if (end)
+
+	if (end) {
 		end(fc, req);
+		smp_wmb();

Why is this barrier needed?

Thanks,
Miklos

author
#9 lore raw

Re: [PATCH 2/4] FUSE: make request_wait_answer() wait for ->end() completion

Hello, Miklos.

Quoted 7 lines · Miklos Szeredi
Quoted 4 lines · Tejun Heo

Previously, a request was marked FINISHED before ->end() is executed and thus request_wait_answer() can return before it's done. This patch makes request_wait_answer() wait for ->end() to finish before returning.

Why is this change needed?

Direct mmap implementation manages fds in the server process using the end() callback. After each command which needs to manipulate server fds in some way, the end callback is responsible for doing it and communicating what happened to the client side which is executing fuse_file_direct_mmap(), so it needs to wait for end to complete before proceeding.

Quoted 9 lines

fs/fuse/dev.c __releases(&fc->lock) · line 300

Quoted 7 lines
fs/fuse/dev.c +3 -2
 	spin_unlock(&fc->lock);
-	wake_up(&req->waitq);
-	if (end)
+
+	if (end) {
 		end(fc, req);
+		smp_wmb();

Why is this barrier needed?

To separate the changes end() made from setting FUSE_REQ_FINISHED. The corresponding wait is in wait_answer_interruptible() and the corresponding barrier is rmb() implied by spin_lock(&fc->lock) in that function. ie. the barrier makes sure that when wait_event_interruptible() exits the waiting state because state equals FUSE_REQ_FINISHED, the caller is guaranteed to see the changes made by end().

Thanks.

Signature · 1 line
tejun
author
#10 lore raw

PATCH 3/4 FUSE: implement fuse_req->prep()

In reply to Tejun Heo

Implement ->prep() which is the opposite equivalent of ->end(). It's called right before the request is passed to userland server in the kernel context of the server. ->prep() can fail the request without disrupting the whole channel.

This will be used by direct mmap implementation.

---
 fs/fuse/dev.c    |   29 ++++++++++++++++++++++++++---
 fs/fuse/fuse_i.h |    6 ++++++
 2 files changed, 32 insertions(+), 3 deletions(-)
fs/fuse/dev.c +26 -3
index 0745728..1c56c6e 100644
@@ -758,6 +758,7 @@ static ssize_t fuse_dev_read(struct kiocb *iocb, const struct iovec *iov,
 			      unsigned long nr_segs, loff_t pos)
 {
 	int err;
+	bool restart;
 	struct fuse_req *req;
 	struct fuse_in *in;
 	struct fuse_copy_state cs;
@@ -804,12 +805,32 @@ static ssize_t fuse_dev_read(struct kiocb *iocb, const struct iovec *iov,
 		goto restart;
 	}
 	spin_unlock(&fc->lock);
+
+	restart = false;
 	fuse_copy_init(&cs, fc, 1, req, iov, nr_segs);
+
+	/*
+	 * Execute prep if available.  Failure from prep doesn't
+	 * indicate faulty channel.  On failure, fail the current
+	 * request and proceed to the next one.
+	 */
+	if (req->prep) {
+		err = req->prep(fc, req);
+		if (err) {
+			restart = true;
+			goto finish;
+		}
+	}
+
 	err = fuse_copy_one(&cs, &in->h, sizeof(in->h));
-	if (!err)
-		err = fuse_copy_args(&cs, in->numargs, in->argpages,
-				     (struct fuse_arg *) in->args, 0);
+	if (err)
+		goto finish;
+
+	err = fuse_copy_args(&cs, in->numargs, in->argpages,
+			     (struct fuse_arg *) in->args, 0);
+ finish:
 	fuse_copy_finish(&cs);
+
 	spin_lock(&fc->lock);
 	req->locked = 0;
 	if (req->aborted) {
@@ -819,6 +840,8 @@ static ssize_t fuse_dev_read(struct kiocb *iocb, const struct iovec *iov,
 	if (err) {
 		req->out.h.error = -EIO;
 		request_end(fc, req);
+		if (restart)
+			goto restart;
 		return err;
 	}
 	if (!req->isreply)
fs/fuse/fuse_i.h +6 -0
index 7b930b6..aa112e2 100644
@@ -292,6 +292,12 @@ struct fuse_req {
 	/** Link on fi->writepages */
 	struct list_head writepages_entry;
 
+	/** Request preparation callback.  Called from the kernel
+	    context of the FUSE server before passing the request to
+	    the FUSE server.  Non-zero return from this function will
+	    fail the request. */
+	int (*prep)(struct fuse_conn *, struct fuse_req *);
+
 	/** Request completion callback.  This function is called from
 	    the kernel context of the FUSE server if the request isn't
 	    being aborted.  If the request is being aborted, it's
Signature · 1 line
1.6.0.2
maintainer
#11 lore raw

Re: [PATCH 3/4] FUSE: implement fuse_req->prep()

Quoted 1 line · On Thu, 18 Jun 2009 18:24:32 +0900 Tejun Heo <[email protected]> wrote:

Implement ->prep() which is the opposite equivalent of ->end().

The opposite of "end" is "begin". I'd expect to see a sequence like

	-> prep
	-> begin
	-> end

so the naming choice here is unexpected.

Quoted 14 lines

It's
called right before the request is passed to userland server in the kernel context of the server. ->prep() can fail the request without disrupting the whole channel.

This will be used by direct mmap implementation.

...

fs/fuse/fuse_i.h +0 -0
@@ -292,6 +292,12 @@ struct fuse_req {
 	/** Link on fi->writepages */
 	struct list_head writepages_entry;

fs/fuse/fuse_i.h struct fuse_req { · line 303

Quoted 10 lines
fs/fuse/fuse_i.h +6 -0
 
+	/** Request preparation callback.  Called from the kernel
+	    context of the FUSE server before passing the request to
+	    the FUSE server.  Non-zero return from this function will
+	    fail the request. */
+	int (*prep)(struct fuse_conn *, struct fuse_req *);
+
 	/** Request completion callback.  This function is called from
 	    the kernel context of the FUSE server if the request isn't
 	    being aborted.  If the request is being aborted, it's

Why the strange comment layout? Does kerneldoc actually recognise and appropriately process this text? if not, please do

/*
* Request preparation callback. Called from the kernel
* context of the FUSE server before passing the request to
* the FUSE server. Non-zero return from this function will
* fail the request.
*/

If that looks odd then, well, that wasn't your fault ;)

author
#12 lore raw

Re: [PATCH 3/4] FUSE: implement fuse_req->prep()

Hello,

Quoted 5 lines · Andrew Morton
Quoted 1 line · On Thu, 18 Jun 2009 18:24:32 +0900 Tejun Heo <[email protected]> wrote:

Implement ->prep() which is the opposite equivalent of ->end().

Quoted 7 lines

The opposite of "end" is "begin". I'd expect to see a sequence like

	-> prep
	-> begin
	-> end

so the naming choice here is unexpected.

Yes, indeed. I'll look for a better name.

Quoted 28 lines
Quoted 14 lines

It's
called right before the request is passed to userland server in the kernel context of the server. ->prep() can fail the request without disrupting the whole channel.

This will be used by direct mmap implementation.

...

fs/fuse/fuse_i.h +0 -0
@@ -292,6 +292,12 @@ struct fuse_req {
 	/** Link on fi->writepages */
 	struct list_head writepages_entry;

fs/fuse/fuse_i.h struct fuse_req { · line 303

Quoted 10 lines
fs/fuse/fuse_i.h +6 -0
 
+	/** Request preparation callback.  Called from the kernel
+	    context of the FUSE server before passing the request to
+	    the FUSE server.  Non-zero return from this function will
+	    fail the request. */
+	int (*prep)(struct fuse_conn *, struct fuse_req *);
+
 	/** Request completion callback.  This function is called from
 	    the kernel context of the FUSE server if the request isn't
 	    being aborted.  If the request is being aborted, it's

Why the strange comment layout? Does kerneldoc actually recognise and appropriately process this text? if not, please do

Quoted 8 lines

/*
* Request preparation callback. Called from the kernel
* context of the FUSE server before passing the request to
* the FUSE server. Non-zero return from this function will
* fail the request.
*/

If that looks odd then, well, that wasn't your fault ;)

I usually try to conform to the surrounding style so that new stuff at least is consistent with the rest (and maintainers tend to feel happier that way) but yeah this can cause confusion to kerneldoc handling. I'll add a patch to change the style.

Thanks.

Signature · 1 line
tejun
author
#13 lore raw

PATCH 4/4 FUSE: implement direct mmap

In reply to Tejun Heo

This patch implements direct mmap. It allows FUSE server to honor each mmap request with anonymous mapping. FUSE server can make multiple mmap requests share a single anonymous mapping or separate mappings as it sees fit.

mmap request is handled in two steps. MMAP first queries the server whether it wants to share the mapping with an existing one or create a new one, and if so, with which flags. MMAP_COMMIT notifies the server the result of mmap and if successful the fd the server can use to access the mmap region.

Internally, shmem_file is used to back the mmap areas and vma->vm_file is overridden from the FUSE file to the shmem_file.

For details, please read the comment on top of
fuse_file_direct_mmap().

---
 fs/fuse/cuse.c       |    1 +
 fs/fuse/file.c       |  432 ++++++++++++++++++++++++++++++++++++++++++++++++--
 fs/fuse/fuse_i.h     |    8 +
 include/linux/fuse.h |   47 ++++++
 4 files changed, 476 insertions(+), 12 deletions(-)
fs/fuse/cuse.c +1 -0
index de792dc..0ec447c 100644
@@ -181,6 +181,7 @@ static const struct file_operations cuse_frontend_fops = {
 	.unlocked_ioctl		= cuse_file_ioctl,
 	.compat_ioctl		= cuse_file_compat_ioctl,
 	.poll			= fuse_file_poll,
+	.mmap			= fuse_file_direct_mmap,
 };
 
 
fs/fuse/file.c +420 -12

456 lines not on this page · raw message

fs/fuse/fuse_i.h +8 -0
index aa112e2..0fdaee7 100644
@@ -272,6 +272,13 @@ struct fuse_req {
 			struct fuse_write_out out;
 		} write;
 		struct fuse_lk_in lk_in;
+		struct {
+			/** to move filp for mmap between client and server */
+			struct file *file;
+		} mmap;
+		struct {
+			struct fuse_munmap_in in;
+		} munmap;
 	} misc;
 
 	/** page vector */
@@ -724,6 +731,7 @@ ssize_t fuse_direct_io(struct file *file, const char __user *buf,
 long fuse_do_ioctl(struct file *file, unsigned int cmd, unsigned long arg,
 		   unsigned int flags);
 unsigned fuse_file_poll(struct file *file, poll_table *wait);
+int fuse_file_direct_mmap(struct file *file, struct vm_area_struct *vma);
 int fuse_dev_release(struct inode *inode, struct file *file);
 
 #endif /* _FS_FUSE_I_H */
include/linux/fuse.h +47 -0

65 lines not on this page · raw message

Signature · 1 line
1.6.0.2
maintainer
#14 lore raw

Re: [PATCH 4/4] FUSE: implement direct mmap

Quoted 23 lines · On Thu, 18 Jun 2009 18:24:33 +0900 Tejun Heo <[email protected]> wrote:

This patch implements direct mmap. It allows FUSE server to honor each mmap request with anonymous mapping. FUSE server can make multiple mmap requests share a single anonymous mapping or separate mappings as it sees fit.

mmap request is handled in two steps. MMAP first queries the server whether it wants to share the mapping with an existing one or create a new one, and if so, with which flags. MMAP_COMMIT notifies the server the result of mmap and if successful the fd the server can use to access the mmap region.

Internally, shmem_file is used to back the mmap areas and vma->vm_file is overridden from the FUSE file to the shmem_file.

For details, please read the comment on top of
fuse_file_direct_mmap().

...

fs/fuse/file.c +3 -0
+static int fuse_mmap_commit_prep(struct fuse_conn *fc, struct fuse_req *req)
+{
+	struct fuse_mmap_commit_in *commit_in = (void *)req->in.args[0].value;

fs/fuse/file.c int fuse_notify_poll_wakeup(struct fuse_conn *fc, · line 2133

Quoted 10 lines
fs/fuse/file.c +10 -0
+	struct file *mfile = req->misc.mmap.file;
+	int fd;
+
+	if (!mfile)
+		return 0;
+
+	/* new mmap.file has been created, assign a fd to it */
+	fd = commit_in->fd = get_unused_fd_flags(O_CLOEXEC);
+	if (fd < 0)
+		return 0;

Shouldn't this be `return fd;'?

Quoted 111 lines
fs/fuse/file.c +108 -0
+	get_file(mfile);
+	fd_install(fd, mfile);
+	return 0;
+}
+
 
...
 
+int fuse_file_direct_mmap(struct file *file, struct vm_area_struct *vma)
+{
+	struct fuse_file *ff = file->private_data;
+	struct fuse_conn *fc = ff->fc;
+	struct vm_operations_struct *orig_vm_ops = vma->vm_ops;
+	struct file *orig_vm_file = vma->vm_file;
+	unsigned long orig_vm_flags = vma->vm_flags;
+	struct fuse_mmap *fmmap = NULL;
+	struct file *mfile = NULL;
+	struct fuse_req *req;
+	struct fuse_mmap_in mmap_in;
+	struct fuse_mmap_out mmap_out;
+	struct fuse_mmap_commit_in commit_in;
+	u64 mmap_unique;
+	int err;
+
+	/*
+	 * First, execute FUSE_MMAP which will query the server
+	 * whether this mmap request is valid and which fd it wants to
+	 * use to mmap this request.
+	 */
+	req = fuse_get_req(fc);
+	if (IS_ERR(req)) {
+		err = PTR_ERR(req);
+		goto err;
+	}
+
+	memset(&mmap_in, 0, sizeof(mmap_in));
+	mmap_in.fh = ff->fh;
+	mmap_in.addr = vma->vm_start;
+	mmap_in.len = vma->vm_end - vma->vm_start;
+	mmap_in.prot = ((vma->vm_flags & VM_READ) ? PROT_READ : 0) |
+		       ((vma->vm_flags & VM_WRITE) ? PROT_WRITE : 0) |
+		       ((vma->vm_flags & VM_EXEC) ? PROT_EXEC : 0);
+	mmap_in.flags = ((vma->vm_flags & VM_GROWSDOWN) ? MAP_GROWSDOWN : 0) |
+			((vma->vm_flags & VM_DENYWRITE) ? MAP_DENYWRITE : 0) |
+			((vma->vm_flags & VM_EXECUTABLE) ? MAP_EXECUTABLE : 0) |
+			((vma->vm_flags & VM_LOCKED) ? MAP_LOCKED : 0);
+	mmap_in.offset = (loff_t)vma->vm_pgoff << PAGE_SHIFT;
+
+	req->in.h.opcode = FUSE_MMAP;
+	req->in.h.nodeid = ff->nodeid;
+	req->in.numargs = 1;
+	req->in.args[0].size = sizeof(mmap_in);
+	req->in.args[0].value = &mmap_in;
+	req->out.numargs = 1;
+	req->out.args[0].size = sizeof(mmap_out);
+	req->out.args[0].value = &mmap_out;
+
+	req->end = fuse_mmap_end;
+
+	fuse_request_send(fc, req);
+
+	/* mmap.file is set if server requested to reuse existing mapping */
+	mfile = req->misc.mmap.file;
+	mmap_unique = req->in.h.unique;
+	err = req->out.h.error;
+
+	fuse_put_request(fc, req);
+
+	/* ERR_PTR value in mfile means fget failure, send failure COMMIT */
+	if (IS_ERR(mfile)) {
+		err = PTR_ERR(mfile);
+		goto commit;
+	}
+	/* userland indicated failure, we can just fail */
+	if (err)
+		goto err;
+
+	/*
+	 * Second, create mmap as the server requested.
+	 */
+	fmmap = create_fuse_mmap(fc, file, mfile, mmap_unique, mmap_out.fd,
+				 vma);
+	if (IS_ERR(fmmap)) {
+		err = PTR_ERR(fmmap);
+		goto commit;
+	}
+
+	/* fmmap points to shm_file to mmap, give it to vma */
+	mfile = fmmap->mmap_file;
+	vma->vm_file = mfile;
+
+	/* add flags server requested and mmap the shm_file */
+	if (mmap_out.flags & FUSE_MMAP_DONT_COPY)
+		vma->vm_flags |= VM_DONTCOPY;
+	if (mmap_out.flags & FUSE_MMAP_DONT_EXPAND)
+		vma->vm_flags |= VM_DONTEXPAND;
+
+	err = mfile->f_op->mmap(mfile, vma);
+	if (err)
+		goto commit;
+
+	/*
+	 * Override vm_ops->open and ->close.  This is a bit hacky but
+	 * vma's can't easily be nested and FUSE needs to notify the
+	 * server when to release resources for mmaps.  Both shmem and
+	 * tiny_shmem implementations are okay with this trick but if
+	 * there's a cleaner way to do this, please update it.
+	 */
+	err = -EINVAL;
+	if (vma->vm_ops->open || vma->vm_ops->close || vma->vm_private_data) {
+		printk(KERN_ERR "FUSE: can't do direct mmap. shmem mmap has "

fs/fuse/file.c int fuse_notify_poll_wakeup(struct fuse_conn *fc, · line 2300

Quoted 10 lines
fs/fuse/file.c +10 -0
+		       "open, close or vm_private_data\n");
+		goto commit;
+	}
+
+	fmmap->vm_ops = *vma->vm_ops;
+	vma->vm_ops = &fmmap->vm_ops;
+	vma->vm_ops->open = fuse_vm_open;
+	vma->vm_ops->close = fuse_vm_close;
+	vma->vm_private_data = fmmap;
+	err = 0;

Yes, the vm_operations save/overwrite/restore is the stinky part here.

I'm not completely clear what's going on here, and why, and what the overall implications and risks are. Some discussion in the changelog would help. With all that information we then would be better situated to discuss and perhaps solve <whatever problem this solves> by more pleasing means.

I mean, if the need for this hack indicates that there is some shortcoming in core VM then perhaps we can improve core VM. Dunno.

fs/fuse/file.c int fuse_notify_poll_wakeup(struct fuse_conn *fc, · line 2309

Quoted 8 lines

+ commit:

fs/fuse/file.c +7 -0
+	/*
+	 * Third, either mmap succeeded or failed after MMAP request
+	 * succeeded.  Notify userland what happened.
+	 */
+
+	/* missing commit can cause resource leak on server side, don't fail */
+	req = fuse_get_req_nofail(fc, file);

That's an optimistically-named function.

<looks>

hm, it seems to duplicate mempool.c.

Quoted 54 lines
fs/fuse/file.c +54 -0
+	memset(&commit_in, 0, sizeof(commit_in));
+	commit_in.fh = ff->fh;
+	commit_in.mmap_unique = mmap_unique;
+	commit_in.addr = mmap_in.addr;
+	commit_in.len = mmap_in.len;
+	commit_in.prot = mmap_in.prot;
+	commit_in.flags = mmap_in.flags;
+	commit_in.offset = mmap_in.offset;
+
+	if (!err) {
+		commit_in.fd = fmmap->mmap_fd;
+		/*
+		 * If fmmap->mmap_fd < 0, new fd needs to be created
+		 * when the server reads MMAP_COMMIT.  Pass the file
+		 * pointer.  A fd will be assigned to it by the
+		 * fuse_mmap_commit_prep callback.
+		 */
+		if (fmmap->mmap_fd < 0)
+			req->misc.mmap.file = mfile;
+	} else
+		commit_in.fd = err;
+
+	req->in.h.opcode = FUSE_MMAP_COMMIT;
+	req->in.h.nodeid = ff->nodeid;
+	req->in.numargs = 1;
+	req->in.args[0].size = sizeof(commit_in);
+	req->in.args[0].value = &commit_in;
+
+	req->prep = fuse_mmap_commit_prep;
+	req->end = fuse_mmap_commit_end;
+
+	fuse_request_send(fc, req);
+	if (!err)			/* notified failure to userland */
+		err = req->out.h.error;
+	if (!err && commit_in.fd < 0)	/* failed to allocate fd */
+		err = commit_in.fd;
+	fuse_put_request(fc, req);
+
+	if (!err) {
+		fput(orig_vm_file);
+		fmmap->mmap_fd = commit_in.fd;
+		return 0;
+	}
+
+	/* fall through */
+ err:
+	if (fmmap && !IS_ERR(fmmap))
+		destroy_fuse_mmap(fmmap);
+	if (mfile && !IS_ERR(mfile))
+		fput(mfile);
+
+	/* restore original vm_ops, file and flags */
+	vma->vm_ops = orig_vm_ops;
+	vma->vm_file = orig_vm_file;

fs/fuse/file.c int fuse_notify_poll_wakeup(struct fuse_conn *fc, · line 2374

Quoted 10 lines
fs/fuse/file.c +10 -0
+	vma->vm_flags = orig_vm_flags;
+
+	if (err == -ENOSYS) {
+		/* Can't provide the coherency needed for MAP_SHARED */
+		if (vma->vm_flags & VM_MAYSHARE)
+			return -ENODEV;
+
+		invalidate_inode_pages2(file->f_mapping);
+
+		return generic_file_mmap(file, vma);

Now what's happening on this path?

We seem to have decided to fallback to a standard mmap of some form? Why? If the user's request failed, shouldn't we just fail the syscall? Hasn't the kernel just lied to userspace about what happened?

Also, this code looks like it's duplicating the behaviour of core MM? If so then it needs to be kept in sync as core MM changes - that's a maintenance problem. Can it be improved by changing core MM?

Quoted 11 lines
fs/fuse/file.c +5 -0
+	}
+
+	return err;
+}
+EXPORT_SYMBOL_GPL(fuse_file_direct_mmap);
 
...

fs/fuse/fuse_i.h struct fuse_req { · line 276

Quoted 5 lines
fs/fuse/fuse_i.h +2 -0
@@ -272,6 +272,13 @@ struct fuse_req {
 			struct fuse_write_out out;
 		} write;
 		struct fuse_lk_in lk_in;
+		struct {
+			/** to move filp for mmap between client and server */

Comment pretends to be kerneldoc?

Quoted 13 lines
fs/fuse/fuse_i.h +5 -0
+			struct file *file;
+		} mmap;
+		struct {
+			struct fuse_munmap_in in;
+		} munmap;
 	} misc;
 
 
...

include/linux/fuse.h struct fuse_file_lock { · line 181

Quoted 4 lines
include/linux/fuse.h +1 -0
@@ -178,6 +178,15 @@ struct fuse_file_lock {
  */
 #define FUSE_POLL_SCHEDULE_NOTIFY (1 << 0)
 
+/**

kerneldoc? Does this work?

Quoted 11 lines
include/linux/fuse.h +8 -0
+ * Mmap flags
+ *
+ * FUSE_MMAP_DONT_COPY: don't copy the region on fork
+ * FUSE_MMAP_DONT_EXPAND: can't be expanded with mremap()
+ */
+#define FUSE_MMAP_DONT_COPY	(1 << 0)
+#define FUSE_MMAP_DONT_EXPAND	(1 << 1)
+
 
...
author
#15 lore raw

Re: [PATCH 4/4] FUSE: implement direct mmap

Hello, Andrew.

Quoted 7 lines · Andrew Morton
Quoted 7 lines
fs/fuse/file.c +7 -0
+static int fuse_mmap_commit_prep(struct fuse_conn *fc, struct fuse_req *req)
+{
+	struct fuse_mmap_commit_in *commit_in = (void *)req->in.args[0].value;
+	struct file *mfile = req->misc.mmap.file;
+	int fd;
+
+	if (!mfile)
Quoted 8 lines

fs/fuse/file.c int fuse_notify_poll_wakeup(struct fuse_conn *fc, · line 2133

Quoted 6 lines
fs/fuse/file.c +6 -0
+		return 0;
+
+	/* new mmap.file has been created, assign a fd to it */
+	fd = commit_in->fd = get_unused_fd_flags(O_CLOEXEC);
+	if (fd < 0)
+		return 0;

Shouldn't this be `return fd;'?

Non-zero return from prep aborts the command immediately and starts processing the next command. However, when the fd assigning fails, the failure should be communicated to the userland server so that it can release any resource it might be holding for the mmap. So, the failure state is recorded in commit_in->fd which and the prep callback returns 0 so that the failure status can be transmitted to the userland server.

Quoted 9 lines

fs/fuse/file.c int fuse_notify_poll_wakeup(struct fuse_conn *fc, · line 2300

Quoted 6 lines
fs/fuse/file.c +6 -0
+	fmmap->vm_ops = *vma->vm_ops;
+	vma->vm_ops = &fmmap->vm_ops;
+	vma->vm_ops->open = fuse_vm_open;
+	vma->vm_ops->close = fuse_vm_close;
+	vma->vm_private_data = fmmap;
+	err = 0;

Yes, the vm_operations save/overwrite/restore is the stinky part here.

Quoted 8 lines

I'm not completely clear what's going on here, and why, and what the overall implications and risks are. Some discussion in the changelog would help. With all that information we then would be better situated to discuss and perhaps solve <whatever problem this solves> by more pleasing means.

I mean, if the need for this hack indicates that there is some shortcoming in core VM then perhaps we can improve core VM. Dunno.

Sorry about lack of explanation, in a nutshell, fuse direct mmap implementation wants to use shmem but wants to know when each mmap is opened and closed so that it can determine when the mmap is finally released and tell the userland to release related resources. I think the following two approaches would be more regular for situations like this.

1. Export the necessary parts of shmem operations and build our fuse's own vm_ops using necessary operations.

2. If vma can be nested, create wrapper vma around shmem vma and catch open/close calls from there.

#1 might be the better solution here but with two separate shmem implementations it was a bit awkward. #2 currently can't be done, so I used bastardized form of #1. So, yeap, it's ugly.

Anyways, it looks like the whole fd fiddling and shmem vma trickery might all go away, so please don't invest too much time into it.

Quoted 8 lines

fs/fuse/file.c int fuse_notify_poll_wakeup(struct fuse_conn *fc, · line 2309

Quoted 2 lines
fs/fuse/file.c +2 -0
+	/* missing commit can cause resource leak on server side, don't fail */
+	req = fuse_get_req_nofail(fc, file);

That's an optimistically-named function.

<looks>

hm, it seems to duplicate mempool.c.

It's much more narrowly defined but yeah in a way.

Quoted 11 lines

fs/fuse/file.c int fuse_notify_poll_wakeup(struct fuse_conn *fc, · line 2374

Quoted 8 lines
fs/fuse/file.c +8 -0
+	if (err == -ENOSYS) {
+		/* Can't provide the coherency needed for MAP_SHARED */
+		if (vma->vm_flags & VM_MAYSHARE)
+			return -ENODEV;
+
+		invalidate_inode_pages2(file->f_mapping);
+
+		return generic_file_mmap(file, vma);

Now what's happening on this path?

Quoted 7 lines

We seem to have decided to fallback to a standard mmap of some form? Why? If the user's request failed, shouldn't we just fail the syscall? Hasn't the kernel just lied to userspace about what happened?

Also, this code looks like it's duplicating the behaviour of core MM? If so then it needs to be kept in sync as core MM changes - that's a maintenance problem. Can it be improved by changing core MM?

That's from the original mmap implementation before the direct mmap is added. It's just code movement. No functional change by this patch on this path.

Quoted 10 lines

fs/fuse/fuse_i.h struct fuse_req { · line 276

Quoted 8 lines
fs/fuse/fuse_i.h +2 -0
@@ -272,6 +272,13 @@ struct fuse_req {
 			struct fuse_write_out out;
 		} write;
 		struct fuse_lk_in lk_in;
+		struct {
+			/** to move filp for mmap between client and server */

Comment pretends to be kerneldoc?

Will fix.

Quoted 4 lines

fs/fuse/fuse_i.h struct fuse_req { · line 277

Quoted 2 lines
fs/fuse/fuse_i.h +1 -0
+/**

kerneldoc? Does this work?

I have no idea. Will update.

Quoted 8 lines

fs/fuse/fuse_i.h struct fuse_req {

Quoted 8 lines
fs/fuse/fuse_i.h +8 -0
+ * Mmap flags
+ *
+ * FUSE_MMAP_DONT_COPY: don't copy the region on fork
+ * FUSE_MMAP_DONT_EXPAND: can't be expanded with mremap()
+ */
+#define FUSE_MMAP_DONT_COPY	(1 << 0)
+#define FUSE_MMAP_DONT_EXPAND	(1 << 1)
+

Thanks.

Signature · 1 line
tejun
maintainer
#16 lore raw

Re: [PATCH 4/4] FUSE: implement direct mmap

In reply to Tejun Heo
Quoted 4 lines · Tejun Heo

This patch implements direct mmap. It allows FUSE server to honor each mmap request with anonymous mapping. FUSE server can make multiple mmap requests share a single anonymous mapping or separate mappings as it sees fit.

One thing that worries me is that this isn't generic enough. It may be good for OSS emulation, but if someone would want to use it for normal file based mmap, I don't think it would be usable: there's no synchronization between normal I/O paths.

Now that's not a big problem as long as there's a possility to extend the current interface in that direction. E.g. add the possibility for userspace to handle faults and unmap pages from the address space on demand.

The other problem is that sharing address spaces between filesystems in this way is inherently hackish. CODA manages to get away with it, but CODA doesn't want to reimplement mmap/munmap in exotic ways.

Is there s a possibility to do this without needing to share the pages? E.g. implement explicit calls (notifications) on the fuse interface to read and write the mapped pages. That would get rid of all the ugly problems caused by having to pass and translate file descriptors on the fuse interface.

Quoted 6 lines

mmap request is handled in two steps. MMAP first queries the server whether it wants to share the mapping with an existing one or create a new one, and if so, with which flags. MMAP_COMMIT notifies the server the result of mmap and if successful the fd the server can use to access the mmap region.

And you might have noticed I'm not a big fan of these three way handshake messages ;)

Thanks,
Miklos

author
#17 lore raw

Re: [PATCH 4/4] FUSE: implement direct mmap

Hello, Miklos.

Quoted 10 lines · Miklos Szeredi
Quoted 4 lines · Tejun Heo

This patch implements direct mmap. It allows FUSE server to honor each mmap request with anonymous mapping. FUSE server can make multiple mmap requests share a single anonymous mapping or separate mappings as it sees fit.

One thing that worries me is that this isn't generic enough. It may be good for OSS emulation, but if someone would want to use it for normal file based mmap, I don't think it would be usable: there's no synchronization between normal I/O paths.

I would love if it can be made more elegant and generic. Although I wrote it, I have to agree the whole thing doesn't look too pretty.

Quoted 9 lines

Now that's not a big problem as long as there's a possility to extend the current interface in that direction. E.g. add the possibility for userspace to handle faults and unmap pages from the address space on demand.

The other problem is that sharing address spaces between filesystems in this way is inherently hackish. CODA manages to get away with it, but CODA doesn't want to reimplement mmap/munmap in exotic ways.

Quoted 5 lines

Is there s a possibility to do this without needing to share the pages? E.g. implement explicit calls (notifications) on the fuse interface to read and write the mapped pages. That would get rid of all the ugly problems caused by having to pass and translate file descriptors on the fuse interface.

Hmmm... yeah, it would be great if the fd ugliness can be killed.

One problem to consider is that mmaps of the same file doesn't always share the mapping. For example, on a normal filesystem, shared mmaps of a file will always share the same pages; however, mmaps of a device node might or might not share the mapping. For example, mmaps could represent separate data streams or a shared buffer space which the different mmaps might partially use for data passing between themselves without notifying the server. So, the server should be able to determine whether a mmap is gonna be shared with another mmap or not and that's where the passing fd back and forth came in.

Simlar thing can be done by letting the kernel part of fuse keep track of mapped pages using server provided page IDs and letting the server determine sharing or creation of pages and communicate it to the kernel part. It would require more code but the interface should be less convoluted and more flexible. Is this what you have in mind?

Quoted 8 lines
Quoted 5 lines

mmap request is handled in two steps. MMAP first queries the server whether it wants to share the mapping with an existing one or create a new one, and if so, with which flags. MMAP_COMMIT notifies the server the result of mmap and if successful the fd the server can use to access the mmap region.

And you might have noticed I'm not a big fan of these three way handshake messages ;)

Believe it or not, I'm not particularly big fan of it either. I just couldn't think of anything better at the time. :-)

Thanks.

Signature · 1 line
tejun
maintainer
#18 lore raw

Re: [PATCH 4/4] FUSE: implement direct mmap

Quoted 10 lines · Tejun Heo

One problem to consider is that mmaps of the same file doesn't always share the mapping. For example, on a normal filesystem, shared mmaps of a file will always share the same pages; however, mmaps of a device node might or might not share the mapping. For example, mmaps could represent separate data streams or a shared buffer space which the different mmaps might partially use for data passing between themselves without notifying the server. So, the server should be able to determine whether a mmap is gonna be shared with another mmap or not and that's where the passing fd back and forth came in.

Quoted 5 lines

Simlar thing can be done by letting the kernel part of fuse keep track of mapped pages using server provided page IDs and letting the server determine sharing or creation of pages and communicate it to the kernel part. It would require more code but the interface should be less convoluted and more flexible. Is this what you have in mind?

Something like that. In fact we could possibly allow even sharing the pages with userspace, similarly to how drivers do it with the hardware. Afaics sound drivers now map the dma memory with remap_pfn_range(). Similary we could allocate a chunk of non-swapabble kernel memory on request from the userspace server and map its pages using this trick to both the server's and the client's address space.

This is still sort of OSSP specific, I don't see clearly how it could be made more generic.

Quoted 11 lines
Quoted 8 lines
Quoted 5 lines

mmap request is handled in two steps. MMAP first queries the server whether it wants to share the mapping with an existing one or create a new one, and if so, with which flags. MMAP_COMMIT notifies the server the result of mmap and if successful the fd the server can use to access the mmap region.

And you might have noticed I'm not a big fan of these three way handshake messages ;)

Believe it or not, I'm not particularly big fan of it either. I just couldn't think of anything better at the time. :-)

A better scheme would be to assume that MMAP is successful, and if there's an error, send a MUNMAP request. That way the normal case is simpler and there's no need for an extra message type.

Thanks,
Miklos

author
#19 lore raw

Re: [PATCH 4/4] FUSE: implement direct mmap

Hello, Miklos.

Quoted 9 lines · Miklos Szeredi
Quoted 5 lines

Simlar thing can be done by letting the kernel part of fuse keep track of mapped pages using server provided page IDs and letting the server determine sharing or creation of pages and communicate it to the kernel part. It would require more code but the interface should be less convoluted and more flexible. Is this what you have in mind?

Something like that. In fact we could possibly allow even sharing the pages with userspace, similarly to how drivers do it with the hardware.

This was possible with the original implementation. Hmmm... we'll need some sort of filehandle to represent the address space one way or the other.

Quoted 7 lines

Afaics sound drivers now map the dma memory with remap_pfn_range(). Similary we could allocate a chunk of non-swapabble kernel memory on request from the userspace server and map its pages using this trick to both the server's and the client's address space.

This is still sort of OSSP specific, I don't see clearly how it could be made more generic.

Using non-swappable memory would be fine for most device emulations but mapping large amount of pages would be problematic. Hmmm... this is difficult. It's a compromise among flexibility, scalability and code complexity.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

mmap request is handled in two steps. MMAP first queries the server whether it wants to share the mapping with an existing one or create a new one, and if so, with which flags. MMAP_COMMIT notifies the server the result of mmap and if successful the fd the server can use to access the mmap region.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

And you might have noticed I'm not a big fan of these three way handshake messages ;)

Believe it or not, I'm not particularly big fan of it either. I just couldn't think of anything better at the time. :-)

A better scheme would be to assume that MMAP is successful, and if there's an error, send a MUNMAP request. That way the normal case is simpler and there's no need for an extra message type.

The reason why the second stage was necessary was because the fd needs to be communicated back to the server. The kernel doesn't know whether the server wants to share an existing fd or create a new one so the MMAP queries that and MMAP_COMMIT communicates newly created fd if necessary. What can be done is to always create a new fd and let the server override it if it wants the map to be shared and then the kernel can kill the unused fd.

Thanks.

Signature · 1 line
tejun
maintainer
#20 lore raw

Re: [PATCH 4/4] FUSE: implement direct mmap

Quoted 5 lines · Tejun Heo
Quoted 5 lines

Afaics sound drivers now map the dma memory with remap_pfn_range(). Similary we could allocate a chunk of non-swapabble kernel memory on request from the userspace server and map its pages using this trick to both the server's and the client's address space.

Quoted 7 lines
Quoted 2 lines

This is still sort of OSSP specific, I don't see clearly how it could be made more generic.

Using non-swappable memory would be fine for most device emulations but mapping large amount of pages would be problematic. Hmmm... this is difficult. It's a compromise among flexibility, scalability and code complexity.

What's the difficulty?

Allocating pages, giving them an ID and mapping them into various page tables seems simple in contrast to trying to make a tmpfs file be a fuse file at the same time, which the VM is really not prepared for.

Thanks,
Miklos

author
#21 lore raw

Re: [PATCH 4/4] FUSE: implement direct mmap

Hello, Miklos.

Quoted 6 lines · Miklos Szeredi
Quoted 5 lines · Tejun Heo
Quoted 5 lines

Afaics sound drivers now map the dma memory with remap_pfn_range(). Similary we could allocate a chunk of non-swapabble kernel memory on request from the userspace server and map its pages using this trick to both the server's and the client's address space.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

This is still sort of OSSP specific, I don't see clearly how it could be made more generic.

Using non-swappable memory would be fine for most device emulations but mapping large amount of pages would be problematic. Hmmm... this is difficult. It's a compromise among flexibility, scalability and code complexity.

What's the difficulty?

The desire to avoid pinning all the mapped pages. :-)

Quoted 3 lines

Allocating pages, giving them an ID and mapping them into various page tables seems simple in contrast to trying to make a tmpfs file be a fuse file at the same time, which the VM is really not prepared for.

If pinning all the pages are okay, the above should work fine.

Thanks.

Signature · 1 line
tejun
maintainer
#22 lore raw

Re: [PATCH 4/4] FUSE: implement direct mmap

Quoted 7 lines · Tejun Heo
Quoted 6 lines
Quoted 4 lines

Using non-swappable memory would be fine for most device emulations but mapping large amount of pages would be problematic. Hmmm... this is difficult. It's a compromise among flexibility, scalability and code complexity.

What's the difficulty?

Quoted 7 lines

The desire to avoid pinning all the mapped pages. :-)

Quoted 3 lines

Allocating pages, giving them an ID and mapping them into various page tables seems simple in contrast to trying to make a tmpfs file be a fuse file at the same time, which the VM is really not prepared for.

If pinning all the pages are okay, the above should work fine.

If the page allocation is privileged then I think it's okay.

I have only vague ideas how this "direct mmap" could be used for other things than sound drivers. For example there's a significant push towards making network filesystems coherent. This is currently not well supported by fuse and adding coherency guarantees to mmap is especially tricky. But I'm not sure it really needs a special mmap interface, that's just one of the options.

Thoughts?

Thanks,
Miklos