Skip to contentSkip to conversation

fs io with struct page instead of iovecs

RFC New Zach Brown 3 participants 8 messages Nov 7, 2007 VFS

lore
Share & tools
lore

Zach Brown

author
#0 lore raw

At the FS meeting at LCE there was some talk of doing O_DIRECT writes from the kernel with pages instead of with iovecs. This patch series explores one direction we could head in to achieve this.

We obviously can't just translate user iovecs (which might represent more memory than the machine has) to pinned pages and pass page struct pointers all the way down the stack.

And we don't want to duplicate a lot of the generic FS machinery down a duplicate path which works with pages instead of iovecs.

So this patch series heads in the direction of abstracting out the memory argument to the read and write calls. It's still based on segments, but we hide that it's either iovecs or arrays of page pointers from the callers in the rw stack.

I didn't go too nuts with the syntactic sugar. This is just intended to show the basic mechanics. We can obviously pretty it up if we think this is a sane thing to be doing.

The series builds but has never been run.

What do people (that's you, Christoph) think? I'm flexible.

author
#1 lore raw

PATCH 1/4 struct rwmem: an abstraction of the memory argument to read/write

This adds a structure and interface to represent the segments of memory which are acting as the source or destination for a read or write operation.

Callers would fill this structure and then pass it down the rw path.

The intent is to let stages in the rw path make specific calls against this API and structure instead of working with, say, struct iovec natively.

The main intent of this is to enable kernel calls into the rw path which specify memory with page/offset/len tuples.

Another potential benefit of this is the reduction in iterations over iovecs at various points in the kernel. Each iov_length(iov) call, for example, could be translated into rwm->total_bytes. O_DIRECTs check of memory alignment is changed into a single test against rwm->boundary_bits.

I imagine this might integrate well with the iov_iter interface, though I haven't examined that in any depth.

---
 fs/Makefile           |    2 +-
 fs/rwmem.c            |   92 +++++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/rwmem.h |   29 +++++++++++++++
 3 files changed, 122 insertions(+), 1 deletions(-)
 create mode 100644 fs/rwmem.c
 create mode 100644 include/linux/rwmem.h
fs/Makefile +1 -1
index 500cf15..c342365 100644
@@ -11,7 +11,7 @@ obj-y :=	open.o read_write.o file_table.o super.o \
 		attr.o bad_inode.o file.o filesystems.o namespace.o aio.o \
 		seq_file.o xattr.o libfs.o fs-writeback.o \
 		pnode.o drop_caches.o splice.o sync.o utimes.o \
-		stack.o
+		stack.o rwmem.o
 
 ifeq ($(CONFIG_BLOCK),y)
 obj-y +=	buffer.o bio.o block_dev.o direct-io.o mpage.o ioprio.o
fs/rwmem.c new file +92 -0

92 lines not on this page · raw message

include/linux/rwmem.h new file +29 -0
new file mode 100644
index 0000000..666f9f4
@@ -0,0 +1,29 @@
+#ifndef _LINUX_RWMEM_H
+#define _LINUX_RWMEM_H
+
+struct rwmwm_ops;
+
+struct rwmem {
+	struct rwmem_ops	*ops;
+	size_t			total_bytes;
+	unsigned long		boundary_bits;
+	unsigned long		nr_pages;
+	unsigned short		nr_segs;
+};
+
+struct rwmem_ops {
+	void (*init)(struct rwmem *rwm);
+	unsigned long (*seg_page_offset)(struct rwmem *rwm, unsigned long i);
+	unsigned long (*seg_bytes)(struct rwmem *rwm, unsigned long i);
+	int (*get_seg_pages)(struct rwmem *rwm, unsigned long i,
+			     unsigned long *cursor, struct page **pages,
+			     unsigned long max_pages, int write);
+};
+
+struct rwmem_iovec {
+	struct rwmem		rwmem;
+	const struct iovec	*iov;
+};
+struct rwmem_ops rwmem_iovec_ops;
+
+#endif
Signature · 1 line
1.5.2.2
author
#2 lore raw

PATCH 2/4 dio: use rwmem to work with r/w memory arguments

This switches dio to work with the rwmem api to get memory pages for the IO instead of working with iovecs directly.

It can use direct rwm struct accesses for some static universal properties of a set of memory segments that make up the buffer argument.

It uses helper functions to work with the underlying data structures directly.

---
 fs/direct-io.c |  123 +++++++++++++++++++++++++-------------------------------
 1 files changed, 55 insertions(+), 68 deletions(-)
fs/direct-io.c +55 -68
index acf0da1..0d5ed41 100644
@@ -34,7 +34,7 @@
 #include <linux/blkdev.h>
 #include <linux/buffer_head.h>
 #include <linux/rwsem.h>
-#include <linux/uio.h>
+#include <linux/rwmem.h>
 #include <asm/atomic.h>
 
 /*
@@ -105,11 +105,12 @@ struct dio {
 	sector_t cur_page_block;	/* Where it starts */
 
 	/*
-	 * Page fetching state. These variables belong to dio_refill_pages().
+	 * Page fetching state. direct_io_worker() sets these for
+	 * dio_refill_pages() who modifies them as it fetches.
 	 */
-	int curr_page;			/* changes */
-	int total_pages;		/* doesn't change */
-	unsigned long curr_user_address;/* changes */
+	struct rwmem *rwm;
+	unsigned long cur_seg;
+	unsigned long cur_seg_cursor;
 
 	/*
 	 * Page queue.  These variables belong to dio_refill_pages() and
@@ -146,21 +147,11 @@ static inline unsigned dio_pages_present(struct dio *dio)
  */
 static int dio_refill_pages(struct dio *dio)
 {
+	struct rwmem *rwm = dio->rwm;
 	int ret;
-	int nr_pages;
-
-	nr_pages = min(dio->total_pages - dio->curr_page, DIO_PAGES);
-	down_read(&current->mm->mmap_sem);
-	ret = get_user_pages(
-		current,			/* Task for fault acounting */
-		current->mm,			/* whose pages? */
-		dio->curr_user_address,		/* Where from? */
-		nr_pages,			/* How many pages? */
-		dio->rw == READ,		/* Write to memory? */
-		0,				/* force (?) */
-		&dio->pages[0],
-		NULL);				/* vmas */
-	up_read(&current->mm->mmap_sem);
+
+	ret = rwm->ops->get_seg_pages(rwm, dio->cur_seg, &dio->cur_seg_cursor,
+				      dio->pages, DIO_PAGES, dio->rw == READ);
 
 	if (ret < 0 && dio->blocks_available && (dio->rw & WRITE)) {
 		struct page *page = ZERO_PAGE(0);
@@ -180,8 +171,6 @@ static int dio_refill_pages(struct dio *dio)
 	}
 
 	if (ret >= 0) {
-		dio->curr_user_address += ret * PAGE_SIZE;
-		dio->curr_page += ret;
 		dio->head = 0;
 		dio->tail = ret;
 		ret = 0;
@@ -938,11 +927,9 @@ out:
  */
 static ssize_t
 direct_io_worker(int rw, struct kiocb *iocb, struct inode *inode, 
-	const struct iovec *iov, loff_t offset, unsigned long nr_segs, 
-	unsigned blkbits, get_block_t get_block, dio_iodone_t end_io,
-	struct dio *dio)
+	struct rwmem *rwm, loff_t offset, unsigned blkbits,
+	get_block_t get_block, dio_iodone_t end_io, struct dio *dio)
 {
-	unsigned long user_addr; 
 	unsigned long flags;
 	int seg;
 	ssize_t ret = 0;
@@ -966,44 +953,33 @@ direct_io_worker(int rw, struct kiocb *iocb, struct inode *inode,
 	spin_lock_init(&dio->bio_lock);
 	dio->refcount = 1;
 
+	dio->rwm = rwm;
+
 	/*
 	 * In case of non-aligned buffers, we may need 2 more
 	 * pages since we need to zero out first and last block.
 	 */
+	dio->pages_in_io = rwm->nr_pages;
 	if (unlikely(dio->blkfactor))
-		dio->pages_in_io = 2;
-
-	for (seg = 0; seg < nr_segs; seg++) {
-		user_addr = (unsigned long)iov[seg].iov_base;
-		dio->pages_in_io +=
-			((user_addr+iov[seg].iov_len +PAGE_SIZE-1)/PAGE_SIZE
-				- user_addr/PAGE_SIZE);
-	}
+		dio->pages_in_io += 2;
 
-	for (seg = 0; seg < nr_segs; seg++) {
-		user_addr = (unsigned long)iov[seg].iov_base;
-		dio->size += bytes = iov[seg].iov_len;
+	for (seg = 0; seg < rwm->nr_segs; seg++) {
+		dio->size += bytes = rwm->ops->seg_bytes(rwm, seg);
 
 		/* Index into the first page of the first block */
-		dio->first_block_in_page = (user_addr & ~PAGE_MASK) >> blkbits;
+		dio->first_block_in_page = 
+			rwm->ops->seg_page_offset(rwm, seg) >> blkbits;
 		dio->final_block_in_request = dio->block_in_file +
 						(bytes >> blkbits);
 		/* Page fetching state */
+		dio->cur_seg = seg;
+		dio->cur_seg_cursor = 0;
 		dio->head = 0;
 		dio->tail = 0;
-		dio->curr_page = 0;
 
-		dio->total_pages = 0;
-		if (user_addr & (PAGE_SIZE-1)) {
-			dio->total_pages++;
-			bytes -= PAGE_SIZE - (user_addr & (PAGE_SIZE - 1));
-		}
-		dio->total_pages += (bytes + PAGE_SIZE - 1) / PAGE_SIZE;
-		dio->curr_user_address = user_addr;
-	
 		ret = do_direct_IO(dio);
 
-		dio->result += iov[seg].iov_len -
+		dio->result += bytes -
 			((dio->final_block_in_request - dio->block_in_file) <<
 					blkbits);
 
@@ -1113,15 +1089,11 @@ direct_io_worker(int rw, struct kiocb *iocb, struct inode *inode,
  *
  * Additional i_alloc_sem locking requirements described inline below.
  */
-ssize_t
-__blockdev_direct_IO(int rw, struct kiocb *iocb, struct inode *inode,
-	struct block_device *bdev, const struct iovec *iov, loff_t offset, 
-	unsigned long nr_segs, get_block_t get_block, dio_iodone_t end_io,
-	int dio_lock_type)
+static ssize_t
+blockdev_direct_IO_rwmem(int rw, struct kiocb *iocb, struct inode *inode,
+	struct block_device *bdev, struct rwmem *rwm, loff_t offset, 
+	get_block_t get_block, dio_iodone_t end_io, int dio_lock_type)
 {
-	int seg;
-	size_t size;
-	unsigned long addr;
 	unsigned blkbits = inode->i_blkbits;
 	unsigned bdev_blkbits = 0;
 	unsigned blocksize_mask = (1 << blkbits) - 1;
@@ -1146,17 +1118,12 @@ __blockdev_direct_IO(int rw, struct kiocb *iocb, struct inode *inode,
 	}
 
 	/* Check the memory alignment.  Blocks cannot straddle pages */
-	for (seg = 0; seg < nr_segs; seg++) {
-		addr = (unsigned long)iov[seg].iov_base;
-		size = iov[seg].iov_len;
-		end += size;
-		if ((addr & blocksize_mask) || (size & blocksize_mask))  {
-			if (bdev)
-				 blkbits = bdev_blkbits;
-			blocksize_mask = (1 << blkbits) - 1;
-			if ((addr & blocksize_mask) || (size & blocksize_mask))  
-				goto out;
-		}
+	if (rwm->boundary_bits & blocksize_mask) {
+		if (bdev)
+			 blkbits = bdev_blkbits;
+		blocksize_mask = (1 << blkbits) - 1;
+		if (rwm->boundary_bits & blocksize_mask)
+			goto out;
 	}
 
 	dio = kzalloc(sizeof(*dio), GFP_KERNEL);
@@ -1212,8 +1179,8 @@ __blockdev_direct_IO(int rw, struct kiocb *iocb, struct inode *inode,
 	dio->is_async = !is_sync_kiocb(iocb) && !((rw & WRITE) &&
 		(end > i_size_read(inode)));
 
-	retval = direct_io_worker(rw, iocb, inode, iov, offset,
-				nr_segs, blkbits, get_block, end_io, dio);
+	retval = direct_io_worker(rw, iocb, inode, rwm, offset, blkbits,
+				  get_block, end_io, dio);
 
 	if (rw == READ && dio_lock_type == DIO_LOCKING)
 		release_i_mutex = 0;
@@ -1225,4 +1192,24 @@ out:
 		mutex_lock(&inode->i_mutex);
 	return retval;
 }
+
+ssize_t
+__blockdev_direct_IO(int rw, struct kiocb *iocb, struct inode *inode,
+	struct block_device *bdev, const struct iovec *iov, loff_t offset, 
+	unsigned long nr_segs, get_block_t get_block, dio_iodone_t end_io,
+	int dio_lock_type)
+{
+	struct rwmem_iovec rwi = {
+		.rwmem.ops = &rwmem_iovec_ops,
+		.rwmem.nr_segs = nr_segs,
+		.iov = iov,
+	};
+	struct rwmem *rwm = &rwi.rwmem;
+
+	rwm->ops->init(rwm);
+
+	return blockdev_direct_IO_rwmem(rw, iocb, inode, bdev, rwm, offset,
+					get_block, end_io, dio_lock_type);
+}
+
 EXPORT_SYMBOL(__blockdev_direct_IO);
Signature · 1 line
1.5.2.2
author
#3 lore raw

PATCH 3/4 add rwmem type backed by pages

This lets callers specify a region of memory to read from or write to with an array of page/offset/len tuples. There have been specific requests to do this from servers which want to do O_DIRECT from the kernel. (knfsd?)

This could also be used by places which currently hold a kmap() and call fop->write. ecryptfs_write_lower_page_segment() is one such caller.

---
 fs/rwmem.c            |   66 +++++++++++++++++++++++++++++++++++++++++++++++++
 include/linux/rwmem.h |   16 ++++++++++++
 2 files changed, 82 insertions(+), 0 deletions(-)
fs/rwmem.c +66 -0
index 0433ba4..c87e8a4 100644
@@ -90,3 +90,69 @@ struct rwmem_ops rwmem_iovec_ops = {
 	.seg_bytes		= rwmem_iovec_seg_bytes,
 	.get_seg_pages		= rwmem_iovec_get_seg_pages,
 };
+
+void rwmem_pages_init(struct rwmem *rwm)
+{
+	struct rwmem_pages *rwp = container_of(rwm, struct rwmem_pages, rwmem);
+	struct pgol *pgol;
+	unsigned long i;
+
+	rwm->total_bytes = 0;
+	rwm->nr_pages = rwm->nr_segs;
+	rwm->boundary_bits = 0;
+
+	for (i = 0; i < rwm->nr_segs; i++) {
+		pgol = &rwp->pgol[i];
+
+		rwm->total_bytes += pgol->len;
+		rwm->boundary_bits |= pgol->offset | pgol->len;
+	}
+}
+
+/*
+ * Returns the offset of the start of a segment within its first page.
+ */
+unsigned long rwmem_pages_seg_page_offset(struct rwmem *rwm, unsigned long i)
+{
+	struct rwmem_pages *rwp = container_of(rwm, struct rwmem_pages, rwmem);
+	BUG_ON(i >= rwm->nr_segs);
+	return rwp->pgol[i].offset;
+}
+
+/*
+ * Returns the total bytes in the given segment.
+ */
+unsigned long rwmem_pages_seg_bytes(struct rwmem *rwm, unsigned long i)
+{
+	struct rwmem_pages *rwp = container_of(rwm, struct rwmem_pages, rwmem);
+	BUG_ON(i >= rwm->nr_segs);
+	return rwp->pgol[i].len;
+}
+
+/*
+ * For now each page is its own seg.
+ */
+int rwmem_pages_get_seg_pages(struct rwmem *rwm, unsigned long i,
+			      unsigned long *cursor, struct page **pages,
+			      unsigned long max_pages, int write)
+{
+	struct rwmem_pages *rwp = container_of(rwm, struct rwmem_pages, rwmem);
+	int ret = 0;
+
+	BUG_ON(i >= rwm->nr_segs);
+	BUG_ON(*cursor != 0);
+
+	if (max_pages) {
+		pages[0] = rwp->pgol[i].page;
+		get_page(pages[0]);
+		ret = 1;
+	}
+	return ret;
+}
+
+struct rwmem_ops rwmem_pages_ops = {
+	.init			= rwmem_pages_init,
+	.seg_page_offset	= rwmem_pages_seg_page_offset,
+	.seg_bytes		= rwmem_pages_seg_bytes,
+	.get_seg_pages		= rwmem_pages_get_seg_pages,
+};
include/linux/rwmem.h +16 -0
index 666f9f4..47019f0 100644
@@ -26,4 +26,20 @@ struct rwmem_iovec {
 };
 struct rwmem_ops rwmem_iovec_ops;
 
+/*
+ * How many times do we need this in subsystems before we make a universal
+ * struct?  (bio_vec, skb_frag_struct, pipe_buffer)
+ */
+struct pgol {
+	struct page *page;
+	unsigned int offset;
+	unsigned int len;
+};
+
+struct rwmem_pages {
+	struct rwmem	rwmem;
+	struct pgol	*pgol;
+};
+struct rwmem_ops rwmem_pages_ops;
+
 #endif
Signature · 1 line
1.5.2.2
author
#4 lore raw

PATCH 4/4 add dio interface for page/offset/len tuples

This is what it might look like to feed pgol in to some part of the fs stack instead of iovecs. I imagine we'd want to do it at a much higher level, perhaps something like vfs_write_pages().

---
 fs/direct-io.c |   21 +++++++++++++++++++++
 1 files changed, 21 insertions(+), 0 deletions(-)
fs/direct-io.c +21 -0
index 0d5ed41..e86bcbc 100644
@@ -1213,3 +1213,24 @@ __blockdev_direct_IO(int rw, struct kiocb *iocb, struct inode *inode,
 }
 
 EXPORT_SYMBOL(__blockdev_direct_IO);
+
+ssize_t
+__blockdev_direct_IO_pages(int rw, struct kiocb *iocb, struct inode *inode,
+	struct block_device *bdev, struct pgol *pgol, loff_t offset, 
+	unsigned long nr_pages, get_block_t get_block, dio_iodone_t end_io,
+	int dio_lock_type)
+{
+	struct rwmem_pages rwp = {
+		.rwmem.ops = &rwmem_pages_ops,
+		.rwmem.nr_segs = nr_pages,
+		.pgol = pgol,
+	};
+	struct rwmem *rwm = &rwp.rwmem;
+
+	rwm->ops->init(rwm);
+
+	return blockdev_direct_IO_rwmem(rw, iocb, inode, bdev, rwm, offset,
+					get_block, end_io, dio_lock_type);
+}
+
+EXPORT_SYMBOL(__blockdev_direct_IO_pages);
Signature · 1 line
1.5.2.2
#5 lore raw
In reply to Zach Brown
Quoted 2 lines · Zach Brown

At the FS meeting at LCE there was some talk of doing O_DIRECT writes from the kernel with pages instead of with iovecs. T

Why ? Whats the use case ?

Thanks,
Badari

author
#6 lore raw
Quoted 5 lines · Badari Pulavarty
Quoted 2 lines · Zach Brown

At the FS meeting at LCE there was some talk of doing O_DIRECT writes from the kernel with pages instead of with iovecs. T

Why ? Whats the use case ?

Well, I think there's a few:

There are existing callers which hold a kmap() across ->write, which isn't great. ecryptfs() does this. That's mentioned in the patch series. Arguably loopback should be using this instead of copying some fs paths and trying to call aop methods directly.

I seem to remember Christoph and David having stories of knfsd folks in SGI wanting to do O_DIRECT writes from knfsd? (If not, *I* kind of want to, after rolling some patches to align net rx descriptors :)).

Lustre shows us that there is a point at which you can't saturate your network and storage if your cpu is copying all the data. I'll be the first to admit that the community might not feel a pressing need to address this for in-kernel file system writers, but the observation remains.

- z

#7 lore raw
Quoted 9 lines · Zach Brown
Quoted 5 lines · Badari Pulavarty
Quoted 2 lines · Zach Brown

At the FS meeting at LCE there was some talk of doing O_DIRECT writes from the kernel with pages instead of with iovecs. T

Why ? Whats the use case ?

Well, I think there's a few:

Quoted 8 lines

There are existing callers which hold a kmap() across ->write, which isn't great. ecryptfs() does this. That's mentioned in the patch series. Arguably loopback should be using this instead of copying some fs paths and trying to call aop methods directly.

I seem to remember Christoph and David having stories of knfsd folks in SGI wanting to do O_DIRECT writes from knfsd? (If not, *I* kind of want to, after rolling some patches to align net rx descriptors :)).

The main reason is to remove the serialised writer problem when multiple clients are writing to the one file. With XFS and direct I/O, we can have multiple concurrent writers to the one file and have it scale rather than be limited to what a single cpu holding the i_mutex can do....

Quoted 2 lines

Lustre shows us that there is a point at which you can't saturate your network and storage if your cpu is copying all the data.

Buy more CPUs ;)

Cheers,

Dave.

Signature · 3 lines
Dave Chinner
Principal Engineer
SGI Australian Software Group