io_uring changes for 5.5-rc

Hi Linus,

Here are the changes for io_uring for 5.5. A lot of stuff has been going on this cycle, with improving the support for networked IO (and hence unbounded request completion times) being one of the major themes. There's been a set of fixes done this week, I'll send those out as well once we're certain we're fully happy with them. This pull request contains:

- Unification of the "normal" submit path and the SQPOLL path (Pavel)

- Support for sparse (and bigger) file sets, and updating of those file sets without needing to unregister/register again.

- Independently sized CQ ring, instead of just making it always 2x the SQ ring size. This makes it more flexible for networked applications.

- Support for overflowed CQ ring, never dropping events but providing backpressure on submits.

- Add support for absolute timeouts, not just relative ones.

- Support for generic cancellations. This divorces io_uring from workqueues as well, which additionally gets us one step closer to generic async system call support.

- With cancellations, we can support grabbing the process file table as well, just like we do mm context. This allows support for system calls that create file descriptors, like accept4() support that's built on top of that.

- Support for io_uring tracing (Dmitrii)

- Support for linked timeouts. These abort an operation if it isn't completed by the time noted in the linke timeout.

- Speedup tracking of poll requests

- Various cleanups making the coder easier to follow (Jackie, Pavel, Bob, YueHaibing, me)

- Update MAINTAINERS with new io_uring list

This will cause a merge conflict with some of the late fixes that went into mainline. I'm attaching my merge commit of pulling master into for-next the other day to resolve it. There are three main conflicts, two first are trivial, the last one not that bad. But figured I'd attach it for reference.

Please pull!

git://git.kernel.dk/linux-block.git tags/for-5.5/io_uring-20191121

----------------------------------------------------------------
Bob Liu (2):
io_uring: clean up io_uring_cancel_files()
io_uring: introduce req_need_defer()

Dmitrii Dolgov (1):
io_uring: add set of tracing events

Jackie Liu (5):
io_uring: replace s->needs_lock with s->in_async
io_uring: set -EINTR directly when a signal wakes up in io_cqring_wait
io_uring: remove passed in 'ctx' function parameter ctx if possible
io_uring: keep io_put_req only responsible for release and put req
io_uring: separate the io_free_req and io_free_req_find_next interface

Jens Axboe (47):
io_uring: run dependent links inline if possible
io_uring: allow sparse fixed file sets
io_uring: add support for IORING_REGISTER_FILES_UPDATE
io_uring: allow application controlled CQ ring size
io_uring: add support for absolute timeouts
io_uring: add support for canceling timeout requests
io-wq: small threadpool implementation for io_uring
io_uring: replace workqueue usage with io-wq
io_uring: io_uring: add support for async work inheriting files
net: add __sys_accept4_file() helper
io_uring: add support for IORING_OP_ACCEPT
io_uring: protect fixed file indexing with array_index_nospec()
io_uring: support for larger fixed file sets
io_uring: fix race with canceling timeouts
io_uring: io_wq_create() returns an error pointer, not NULL
io_uring: support for generic async request cancel
io_uring: remove io_uring_add_to_prev() trace event
io_uring: add completion trace event
MAINTAINERS: update io_uring entry
io-wq: use proper nesting IRQ disabling spinlocks for cancel
io_uring: enable optimized link handling for IORING_OP_POLL_ADD
io_uring: fixup a few spots where link failure isn't flagged
io_uring: kill dead REQ_F_LINK_DONE flag
io_uring: abstract out io_async_cancel_one() helper
io_uring: add support for linked SQE timeouts
io_uring: make io_cqring_events() take 'ctx' as argument
io_uring: pass in io_kiocb to fill/add CQ handlers
io_uring: add support for backlogged CQ ring
io-wq: io_wqe_run_queue() doesn't need to use list_empty_careful()
io-wq: add support for bounded vs unbunded work
io_uring: properly mark async work as bounded vs unbounded
io_uring: reduce/pack size of io_ring_ctx
io_uring: fix error clear of ->file_table in io_sqe_files_register()
io_uring: convert accept4() -ERESTARTSYS into -EINTR
io_uring: provide fallback request for OOM situations
io_uring: make ASYNC_CANCEL work with poll and timeout
io_uring: flag SQPOLL busy condition to userspace
io_uring: don't do flush cancel under inflight_lock
io_uring: fix -ENOENT issue with linked timer with short timeout
io_uring: use correct "is IO worker" helper
io_uring: fix potential deadlock in io_poll_wake()
io_uring: check for validity of ->rings in teardown
io_wq: add get/put_work handlers to io_wq_create()
io-wq: ensure we have a stable view of ->cur_work for cancellations
io-wq: ensure free/busy list browsing see all items
io-wq: remove now redundant struct io_wq_nulls_list
io_uring: make POLL_ADD/POLL_REMOVE scale better

Pavel Begunkov (8):
io_uring: remove index from sqe_submit
io_uring: Fix mm_fault with READ/WRITE_FIXED
io_uring: Merge io_submit_sqes and io_ring_submit
io_uring: io_queue_link*() right after submit
io_uring: allocate io_kiocb upfront
io_uring: Use submit info inlined into req
io_uring: use inlined struct sqe_submit
io_uring: Fix getting file for non-fd opcodes

YueHaibing (1):
io-wq: use kfree_rcu() to simplify the code

 MAINTAINERS                     |    5 +-
 fs/Kconfig                      |    3 +
 fs/Makefile                     |    1 +
 fs/io-wq.c                      | 1065 +++++++++++++++++++
 fs/io-wq.h                      |   74 ++
 fs/io_uring.c                   | 2213 ++++++++++++++++++++++++++-------------
 include/Kbuild                  |    1 +
 include/linux/sched.h           |    1 +
 include/linux/socket.h          |    3 +
 include/trace/events/io_uring.h |  358 +++++++
 include/uapi/linux/io_uring.h   |   24 +-
 init/Kconfig                    |    1 +
 kernel/sched/core.c             |   16 +-
 net/socket.c                    |   65 +-
 14 files changed, 3098 insertions(+), 732 deletions(-)
 create mode 100644 fs/io-wq.c
 create mode 100644 fs/io-wq.h
 create mode 100644 include/trace/events/io_uring.h

commit 22ffc78881bce32ae83dbd315fb15cd7ef8a6e4a
Merge: a3085d8079be b226c9e1f4cb
Author: Jens Axboe <[email protected]>
Date: Fri Nov 15 18:03:21 2019 -0700

Merge branch 'master' into for-next

* master: (98 commits)
afs: Fix race in commit bulk status fetch
KVM: Add a comment describing the /dev/kvm no_compat handling drm/amdgpu: fix null pointer deref in firmware header printing
rsxx: add missed destroy_workqueue calls in remove
iocost: check active_list of all the ancestors in iocg_activate()
rbd: silence bogus uninitialized warning in rbd_object_map_update_finish()
ceph: increment/decrement dio counter on async requests
ceph: take the inode lock before acquiring cap refs
ALSA: usb-audio: Fix incorrect size check for processing/extension units
KVM: x86/mmu: Take slots_lock when using kvm_mmu_zap_all_fast()
kbuild: tell sparse about the $ARCH
sparc: vdso: fix build error of vdso32
block, bfq: deschedule empty bfq_queues not referred by any process
mmc: sdhci-of-at91: fix quirk2 overwrite
ALSA: usb-audio: Fix incorrect NULL check in create_yamaha_midi_quirk()
io_uring: ensure registered buffer import returns the IO length
io_uring: Fix getting file for timeout
drm/i915/tgl: MOCS table update
Revert "drm/i915/ehl: Update MOCS table for EHL"
KVM: Forbid /dev/kvm being opened by a compat task when CONFIG_KVM_COMPAT=n
...

diff --cc fs/io_uring.c
index 25f0e8fd935b,2c819c3c855d..187e1c5021ac
@@@ -340,8 -326,7 +340,9 @@@ struct io_kiocb 
  #define REQ_F_TIMEOUT		1024	/* timeout request */
  #define REQ_F_ISREG		2048	/* regular file */
  #define REQ_F_MUST_PUNT		4096	/* must be punted even for NONBLOCK */
- #define REQ_F_INFLIGHT		8192	/* on inflight list */
- #define REQ_F_COMP_LOCKED	16384	/* completion under lock */
+ #define REQ_F_TIMEOUT_NOSEQ	8192	/* no timeout sequence */
++#define REQ_F_INFLIGHT		16384	/* on inflight list */
++#define REQ_F_COMP_LOCKED	32768	/* completion under lock */
  	u64			user_data;
  	u32			result;
  	u32			sequence;
@@@ -482,9 -454,13 +483,13 @@@ static struct io_kiocb *io_get_timeout_
  	struct io_kiocb *req;
  
  	req = list_first_entry_or_null(&ctx->timeout_list, struct io_kiocb, list);
- 	if (req && !__req_need_defer(req)) {
- 		list_del_init(&req->list);
- 		return req;
+ 	if (req) {
+ 		if (req->flags & REQ_F_TIMEOUT_NOSEQ)
+ 			return NULL;
 -		if (!__io_sequence_defer(ctx, req)) {
++		if (!__req_need_defer(req)) {
+ 			list_del_init(&req->list);
+ 			return req;
+ 		}
  	}
  
  	return NULL;
@@@ -2296,12 -1946,7 +2301,13 @@@ static int io_timeout(struct io_kiocb *
  	if (get_timespec64(&ts, u64_to_user_ptr(sqe->addr)))
  		return -EFAULT;
  
 +	if (flags & IORING_TIMEOUT_ABS)
 +		mode = HRTIMER_MODE_ABS;
 +	else
 +		mode = HRTIMER_MODE_REL;
 +
 +	hrtimer_init(&req->timeout.timer, CLOCK_MONOTONIC, mode);
+ 	req->flags |= REQ_F_TIMEOUT;
  
  	/*
  	 * sqe->off holds how many events that need to occur for this
@@@ -2352,82 -2004,14 +2365,83 @@@
  		nxt->sequence++;
  	}
  	req->sequence -= span;
+ add:
  	list_add(&req->list, entry);
 +	req->timeout.timer.function = io_timeout_fn;
 +	hrtimer_start(&req->timeout.timer, timespec64_to_ktime(ts), mode);
  	spin_unlock_irq(&ctx->completion_lock);
 +	return 0;
 +}
  
 -	hrtimer_init(&req->timeout.timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
 -	req->timeout.timer.function = io_timeout_fn;
 -	hrtimer_start(&req->timeout.timer, timespec64_to_ktime(ts),
 -			HRTIMER_MODE_REL);
 +static bool io_cancel_cb(struct io_wq_work *work, void *data)
 +{
 +	struct io_kiocb *req = container_of(work, struct io_kiocb, work);
 +
 +	return req->user_data == (unsigned long) data;
 +}
 +
 +static int io_async_cancel_one(struct io_ring_ctx *ctx, void *sqe_addr)
 +{
 +	enum io_wq_cancel cancel_ret;
 +	int ret = 0;
 +
 +	cancel_ret = io_wq_cancel_cb(ctx->io_wq, io_cancel_cb, sqe_addr);
 +	switch (cancel_ret) {
 +	case IO_WQ_CANCEL_OK:
 +		ret = 0;
 +		break;
 +	case IO_WQ_CANCEL_RUNNING:
 +		ret = -EALREADY;
 +		break;
 +	case IO_WQ_CANCEL_NOTFOUND:
 +		ret = -ENOENT;
 +		break;
 +	}
 +
 +	return ret;
 +}
 +
 +static void io_async_find_and_cancel(struct io_ring_ctx *ctx,
 +				     struct io_kiocb *req, __u64 sqe_addr,
 +				     struct io_kiocb **nxt)
 +{
 +	unsigned long flags;
 +	int ret;
 +
 +	ret = io_async_cancel_one(ctx, (void *) (unsigned long) sqe_addr);
 +	if (ret != -ENOENT) {
 +		spin_lock_irqsave(&ctx->completion_lock, flags);
 +		goto done;
 +	}
 +
 +	spin_lock_irqsave(&ctx->completion_lock, flags);
 +	ret = io_timeout_cancel(ctx, sqe_addr);
 +	if (ret != -ENOENT)
 +		goto done;
 +	ret = io_poll_cancel(ctx, sqe_addr);
 +done:
 +	io_cqring_fill_event(req, ret);
 +	io_commit_cqring(ctx);
 +	spin_unlock_irqrestore(&ctx->completion_lock, flags);
 +	io_cqring_ev_posted(ctx);
 +
 +	if (ret < 0 && (req->flags & REQ_F_LINK))
 +		req->flags |= REQ_F_FAIL_LINK;
 +	io_put_req_find_next(req, nxt);
 +}
 +
 +static int io_async_cancel(struct io_kiocb *req, const struct io_uring_sqe *sqe,
 +			   struct io_kiocb **nxt)
 +{
 +	struct io_ring_ctx *ctx = req->ctx;
 +
 +	if (unlikely(ctx->flags & IORING_SETUP_IOPOLL))
 +		return -EINVAL;
 +	if (sqe->flags || sqe->ioprio || sqe->off || sqe->len ||
 +	    sqe->cancel_flags)
 +		return -EINVAL;
 +
 +	io_async_find_and_cancel(ctx, req, READ_ONCE(sqe->addr), NULL);
  	return 0;
  }
  

Jens Axboe