Skip to content

AIO cleanups/performance improvements

Kent Overstreet lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest33 patches 53 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/33mm: remove old aio use_mm() commentmm/mmu_context.c+0 -32 replies
2/33aio: remove dead code from aio.hinclude/linux/aio.h+0 -242 replies
3/33gadget: remove only user of aio retrydrivers/usb/gadget/inode.c+29 -92 replies
4/33aio: remove retry-based AIOfs/aio.c fs/ocfs2/dlmglue.c fs/read_write.c +2 more+29 -3782 replies
5/33char: add aio_{read,write} to /dev/{null,zero}drivers/char/mem.c+35 -02 replies
6/33aio: kill return value of aio_complete()fs/aio.c include/linux/aio.h+11 -182 replies
7/33aio: add kiocb_cancel()fs/aio.c+43 -362 replies
8/33aio: move private stuff out of aio.hdrivers/usb/gadget/inode.c fs/aio.c include/linux/aio.h+62 -612 replies
9/33aio: dprintk() -> pr_debug()fs/aio.c+24 -333 replies
10/33aio: do fget() after aio_get_req()fs/aio.c+9 -132 replies
11/33aio: make aio_put_req() locklessfs/aio.c include/linux/aio.h+35 -542 replies
12/33aio: refcounting cleanupfs/aio.c+119 -1531 reply
13/33wait: add wait_event_hrtimeout()include/linux/wait.h+86 -01 reply
14/33aio: make aio_read_evt() more efficient, convert to hrtimersfs/aio.c+88 -1501 reply
15/33aio: use flush_dcache_page()fs/aio.c+17 -282 replies
16/33aio: use cancellation list lazilydrivers/usb/gadget/inode.c fs/aio.c include/linux/aio.h+81 -551 reply
17/33aio: change reqs_active to include unreaped completionsfs/aio.c+32 -142 replies
18/33aio: kill batch allocationfs/aio.c include/linux/aio.h+15 -1021 reply
19/33aio: kill struct aio_ring_infofs/aio.c+75 -811 reply
20/33aio: give shared kioctx fields their own cachelinesfs/aio.c+15 -121 reply
21/33aio: reqs_active -> reqs_availablefs/aio.c+19 -141 reply
22/33aio: percpu reqs_availablefs/aio.c+99 -71 reply
23/33generic dynamic per cpu refcountinginclude/linux/percpu-refcount.h lib/Makefile lib/percpu-refcount.c+358 -13 replies
24/33aio: percpu ioctx refcountfs/aio.c+12 -151 reply
25/33aio: use xchg() instead of completion_lockfs/aio.c+34 -251 reply
26/33aio: don't include aio.h in sched.harch/s390/hypfs/inode.c block/scsi_ioctl.c drivers/char/mem.c +54 more+59 -51 reply
27/33aio: kill ki_keyfs/aio.c include/linux/aio.h+9 -71 reply
28/33aio: kill ki_retryfs/aio.c include/linux/aio.h+82 -1621 reply
29/33block: Prep work for batch completionblock/blk-flush.c block/blk-lib.c drivers/block/drbd/drbd_bitmap.c +52 more+213 -1251 reply
30/33block, aio: batch completion for bios/kiocbsblock/blk-core.c block/blk-flush.c block/blk.h +10 more+321 -1374 replies
31/33virtio-blk: convert to batch completiondrivers/block/virtio_blk.c+20 -111 reply
32/33mtip32xx: convert to batch completiondrivers/block/mtip32xx/mtip32xx.c drivers/block/mtip32xx/mtip32xx.h+51 -431 reply
33/33aio: fix kioctx not being freed after cancellation at exit timefs/aio.c+3 -122 replies

Reviews trailers per patch

27 of 33 patches have a Reviewed-by or Acked-by.

PatchZach BrownTheodore Ts'oJeff Moyer
1/33R-b A-b
2/33R-b A-b
3/33A-b
4/33R-b A-b
5/33R-b A-b
6/33A-b R-b A-b
7/33R-b A-b
8/33R-b A-b
9/33R-b A-b
10/33R-b A-b
11/33R-b A-b
12/33R-b
13/33R-b
14/33R-b
15/33
16/33R-b
17/33
18/33R-b
19/33R-b
20/33R-b
21/33R-b
22/33R-b
23/33R-b
24/33R-b
25/33R-b
26/33
27/33
28/33R-b
29/33R-b
30/33
31/33R-b
32/33R-b
33/33

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v1 thread

Cover letter [PATCH 00/33] AIO cleanups/performance improvements

This is a respin of the AIO patches that have been in Andrew's tree, with all the various fixes squashed.

Two differences from the code that was in Andrew's tree:

* The "block: Prep work for batch completion" patch is new - previously, the batch completion stuff added a separate bi_batch_end_io, this now adds the struct batch_complete * argument to bi_end_io.

* When I went to squash the "aio: fix ringbuffer calculation so we don't wrap" patch
http://atlas.evilpiepirate.org/git/linux-bcache.git/commit/?h=aio-upstream-v0&id=790a3cec8322c4e07704e9356495acdf6ee6aff4
I realized it unintentionally changed behaviour from upstream - so I redid it correctly, and added some comments.

Here's the output of git diff between the two branches (excluding the "prep work for batch completion" patch)

fs/aio.c +29 -26
index 33e9db3..d2c1a82 100644
@@ -75,20 +75,22 @@ struct kioctx {
 
 	struct __percpu kioctx_cpu *cpu;
 
-	/* Size of ringbuffer, in units of struct io_event */
-	unsigned		nr_events;
-
-	/*
-	 * Maximum number of outstanding requests:
-	 * sys_io_setup currently limits this to an unsigned int
-	 */
-	unsigned		max_reqs;
-
 	/*
 	 * For percpu reqs_available, number of slots we move to/from global
 	 * counter at a time:
 	 */
 	unsigned		req_batch;
+	/*
+	 * This is what userspace passed to io_setup(), it's not used for
+	 * anything but counting against the global max_reqs quota.
+	 *
+	 * The real limit is nr_events - 1, which will be larger (see
+	 * aio_setup_ring())
+	 */
+	unsigned		max_reqs;
+
+	/* Size of ringbuffer, in units of struct io_event */
+	unsigned		nr_events;
 
 	unsigned long		mmap_base;
 	unsigned long		mmap_size;
@@ -121,21 +123,20 @@ struct kioctx {
 		wait_queue_head_t wait;
 
 		/*
-		 * Copy of the real tail, that aio_complete uses - to reduce
-		 * cacheline bouncing. The real tail will tend to be much more
-		 * contended - since typically events are delivered one at a
-		 * time, and then aio_read_events() slurps them up a bunch at a
-		 * time - so it's helpful if aio_read_events() isn't also
-		 * contending for the tail. So, aio_complete() updates
-		 * shadow_tail whenever it updates tail.
-		 *
-		 * Also needed because tail is used as a hacky lock and isn't
-		 * always the real tail.
+		 * Copy of the real tail - to reduce cacheline bouncing. Updated
+		 * by aio_complete() whenever it updates the real tail.
 		 */
 		unsigned	shadow_tail;
 	} ____cacheline_aligned_in_smp;
 
 	struct {
+		/*
+		 * This is the canonical copy of the tail pointer, updated by
+		 * aio_complete(). But aio_complete() also uses it as a lock, so
+		 * other code can't use it; aio_complete() keeps shadow_tail in
+		 * sync with the real value of the tail pointer for other code
+		 * to use.
+		 */
 		unsigned	tail;
 	} ____cacheline_aligned_in_smp;
 
@@ -347,20 +348,20 @@ static void free_ioctx(struct kioctx *ctx)
 	head = ring->head;
 	kunmap_atomic(ring);
 
-	while (atomic_read(&ctx->reqs_available) < ctx->max_reqs) {
+	while (atomic_read(&ctx->reqs_available) < ctx->nr_events - 1) {
 		wait_event(ctx->wait,
 			   (head != ctx->shadow_tail) ||
-			   (atomic_read(&ctx->reqs_available) >= ctx->max_reqs));
+			   (atomic_read(&ctx->reqs_available) >= ctx->nr_events - 1));
 
-		avail = (head <= ctx->shadow_tail ?
-			 ctx->shadow_tail : ctx->nr_events) - head;
+		avail = (head <= ctx->shadow_tail
+			 ? ctx->shadow_tail : ctx->nr_events) - head;
 
 		atomic_add(avail, &ctx->reqs_available);
 		head += avail;
 		head %= ctx->nr_events;
 	}
 
-	WARN_ON(atomic_read(&ctx->reqs_available) > ctx->max_reqs);
+	WARN_ON(atomic_read(&ctx->reqs_available) > ctx->nr_events - 1);
 
 	aio_free_ring(ctx);
 
@@ -423,8 +424,6 @@ static struct kioctx *ioctx_alloc(unsigned nr_events)
 		return ERR_PTR(-ENOMEM);
 
 	ctx->max_reqs = nr_events;
-	atomic_set(&ctx->reqs_available, nr_events);
-	ctx->req_batch = nr_events / (num_possible_cpus() * 4);
 
 	percpu_ref_init(&ctx->users);
 	rcu_read_lock();
@@ -444,6 +443,10 @@ static struct kioctx *ioctx_alloc(unsigned nr_events)
 	if (aio_setup_ring(ctx) < 0)
 		goto out_freepcpu;
 
+	atomic_set(&ctx->reqs_available, ctx->nr_events - 1);
+	ctx->req_batch = (ctx->nr_events - 1) / (num_possible_cpus() * 4);
+	BUG_ON(!ctx->req_batch);
+
 	/* limit the number of system wide aios */
 	spin_lock(&aio_nr_lock);
 	if (aio_nr + nr_events > aio_max_nr ||

Benjamin LaHaise (1):
aio: fix kioctx not being freed after cancellation at exit time

Kent Overstreet (27):
aio: kill return value of aio_complete()
aio: add kiocb_cancel()
aio: move private stuff out of aio.h
aio: dprintk() -> pr_debug()
aio: do fget() after aio_get_req()
aio: make aio_put_req() lockless
aio: refcounting cleanup
wait: add wait_event_hrtimeout()
aio: make aio_read_evt() more efficient, convert to hrtimers
aio: use flush_dcache_page()
aio: use cancellation list lazily
aio: change reqs_active to include unreaped completions
aio: kill batch allocation
aio: kill struct aio_ring_info
aio: give shared kioctx fields their own cachelines
aio: reqs_active -> reqs_available
aio: percpu reqs_available
generic dynamic per cpu refcounting
aio: percpu ioctx refcount
aio: use xchg() instead of completion_lock
aio: don't include aio.h in sched.h
aio: kill ki_key
aio: kill ki_retry
block: Prep work for batch completion
block, aio: batch completion for bios/kiocbs
virtio-blk: convert to batch completion
mtip32xx: convert to batch completion

Zach Brown (5):
mm: remove old aio use_mm() comment
aio: remove dead code from aio.h
gadget: remove only user of aio retry
aio: remove retry-based AIO
char: add aio_{read,write} to /dev/{null,zero}

 arch/s390/hypfs/inode.c                      |    1 +
 block/blk-core.c                             |   34 +-
 block/blk-flush.c                            |    5 +-
 block/blk-lib.c                              |    3 +-
 block/blk.h                                  |    3 +-
 block/scsi_ioctl.c                           |    1 +
 drivers/block/drbd/drbd_bitmap.c             |    2 +-
 drivers/block/drbd/drbd_worker.c             |    6 +-
 drivers/block/drbd/drbd_wrappers.h           |    9 +-
 drivers/block/floppy.c                       |    3 +-
 drivers/block/mtip32xx/mtip32xx.c            |   86 +-
 drivers/block/mtip32xx/mtip32xx.h            |    8 +-
 drivers/block/pktcdvd.c                      |    9 +-
 drivers/block/swim3.c                        |    2 +-
 drivers/block/virtio_blk.c                   |   31 +-
 drivers/block/xen-blkback/blkback.c          |    3 +-
 drivers/char/mem.c                           |   36 +
 drivers/infiniband/hw/ipath/ipath_file_ops.c |    1 +
 drivers/infiniband/hw/qib/qib_file_ops.c     |    2 +-
 drivers/md/dm-bufio.c                        |    9 +-
 drivers/md/dm-crypt.c                        |    3 +-
 drivers/md/dm-io.c                           |    2 +-
 drivers/md/dm-snap.c                         |    3 +-
 drivers/md/dm-thin.c                         |    3 +-
 drivers/md/dm-verity.c                       |    3 +-
 drivers/md/dm.c                              |    8 +-
 drivers/md/faulty.c                          |    3 +-
 drivers/md/md.c                              |    9 +-
 drivers/md/multipath.c                       |    3 +-
 drivers/md/raid1.c                           |   15 +-
 drivers/md/raid10.c                          |   21 +-
 drivers/md/raid5.c                           |   15 +-
 drivers/scsi/sg.c                            |    1 +
 drivers/staging/android/logger.c             |    1 +
 drivers/target/target_core_iblock.c          |    6 +-
 drivers/target/target_core_pscsi.c           |    3 +-
 drivers/usb/gadget/inode.c                   |   42 +-
 fs/9p/vfs_addr.c                             |    1 +
 fs/afs/write.c                               |    1 +
 fs/aio.c                                     | 1811 +++++++++++---------------
 fs/bio-integrity.c                           |    3 +-
 fs/bio.c                                     |   62 +-
 fs/block_dev.c                               |    1 +
 fs/btrfs/check-integrity.c                   |   14 +-
 fs/btrfs/compression.c                       |    6 +-
 fs/btrfs/disk-io.c                           |    6 +-
 fs/btrfs/extent_io.c                         |   12 +-
 fs/btrfs/file.c                              |    1 +
 fs/btrfs/inode.c                             |   14 +-
 fs/btrfs/scrub.c                             |   18 +-
 fs/btrfs/volumes.c                           |    4 +-
 fs/buffer.c                                  |    3 +-
 fs/ceph/file.c                               |    1 +
 fs/compat.c                                  |    1 +
 fs/direct-io.c                               |   21 +-
 fs/ecryptfs/file.c                           |    1 +
 fs/ext2/inode.c                              |    1 +
 fs/ext3/inode.c                              |    1 +
 fs/ext4/file.c                               |    1 +
 fs/ext4/indirect.c                           |    1 +
 fs/ext4/inode.c                              |    1 +
 fs/ext4/page-io.c                            |    4 +-
 fs/f2fs/data.c                               |    3 +-
 fs/f2fs/segment.c                            |    3 +-
 fs/fat/inode.c                               |    1 +
 fs/fuse/cuse.c                               |    1 +
 fs/fuse/dev.c                                |    1 +
 fs/fuse/file.c                               |    1 +
 fs/gfs2/aops.c                               |    1 +
 fs/gfs2/file.c                               |    1 +
 fs/gfs2/lops.c                               |    3 +-
 fs/gfs2/ops_fstype.c                         |    3 +-
 fs/hfs/inode.c                               |    1 +
 fs/hfsplus/inode.c                           |    1 +
 fs/hfsplus/wrapper.c                         |    3 +-
 fs/jfs/inode.c                               |    1 +
 fs/jfs/jfs_logmgr.c                          |    4 +-
 fs/jfs/jfs_metapage.c                        |    6 +-
 fs/logfs/dev_bdev.c                          |    8 +-
 fs/mpage.c                                   |    2 +-
 fs/nfs/blocklayout/blocklayout.c             |   17 +-
 fs/nilfs2/inode.c                            |    2 +-
 fs/nilfs2/segbuf.c                           |    3 +-
 fs/ntfs/file.c                               |    1 +
 fs/ntfs/inode.c                              |    1 +
 fs/ocfs2/aops.h                              |    2 +
 fs/ocfs2/cluster/heartbeat.c                 |    4 +-
 fs/ocfs2/dlmglue.c                           |    2 +-
 fs/ocfs2/inode.h                             |    2 +
 fs/pipe.c                                    |    1 +
 fs/read_write.c                              |   35 +-
 fs/reiserfs/inode.c                          |    1 +
 fs/ubifs/file.c                              |    1 +
 fs/udf/inode.c                               |    1 +
 fs/xfs/xfs_aops.c                            |    4 +-
 fs/xfs/xfs_buf.c                             |    3 +-
 fs/xfs/xfs_file.c                            |    1 +
 include/linux/aio.h                          |  199 +--
 include/linux/batch_complete.h               |   23 +
 include/linux/bio.h                          |   38 +-
 include/linux/blk_types.h                    |    4 +-
 include/linux/blkdev.h                       |   12 +-
 include/linux/cgroup.h                       |    1 +
 include/linux/errno.h                        |    1 -
 include/linux/fs.h                           |    2 +-
 include/linux/percpu-refcount.h              |  114 ++
 include/linux/pid_namespace.h                |    1 +
 include/linux/sched.h                        |    2 -
 include/linux/swap.h                         |    3 +-
 include/linux/wait.h                         |   86 ++
 include/linux/writeback.h                    |    1 +
 kernel/fork.c                                |    1 +
 kernel/printk.c                              |    1 +
 kernel/ptrace.c                              |    1 +
 lib/Makefile                                 |    2 +-
 lib/percpu-refcount.c                        |  243 ++++
 mm/bounce.c                                  |   12 +-
 mm/mmu_context.c                             |    3 -
 mm/page_io.c                                 |    6 +-
 mm/shmem.c                                   |    1 +
 mm/swap.c                                    |    1 +
 security/keys/internal.h                     |    2 +
 security/keys/keyctl.c                       |    1 +
 sound/core/pcm_native.c                      |    2 +-
 124 files changed, 1785 insertions(+), 1488 deletions(-)
 create mode 100644 include/linux/batch_complete.h
 create mode 100644 include/linux/percpu-refcount.h
 create mode 100644 lib/percpu-refcount.c
Signature · 1 line
1.8.1.3