io_uring IO interface
Subsystems (11)
Jens Axboe
maintainerPATCH v3 1/1 io_uring IO interface
Here's v3 of the io_uring interface. Since data structures etc have changed since the v1 posting, here's a refresher of what io_uring is and how it works.
io_uring is a submission queue (SQ) and completion queue (CQ) pair that an application can use to communicate with the kernel for doing IO. This isn't aio/libaio, but it provides a similar set of features, as well as some new ones:
- io_uring is a lot more efficient than aio. A lot, and in many ways.
- io_uring supports buffered aio. Not just that, but efficiently as well. Cached data isn't punted to an async context.
- io_uring supports polled IO, it takes advantage of the blk-mq polling work that went into 5.0-rc.
- io_uring supports kernel side submissions for polled IO. This enables IO without ever having to do a system call.
- io_uring supports fixed buffers for O_DIRECT. Buffers can be registered after an io_uring context has been setup, which eliminates the need to do get_user_pages() / put_pages() for each and every IO.
To use io_uring, you must first setup an io_uring context. This is done through the first of three new system calls:
io_uring_setup(entries, params)
Sets up a context for doing async IO. On success, returns a file descriptor that the application can mmap to gain access to the SQ ring, CQ ring, and io_uring_sqe's.
Once the rings are setup, the application then mmap's these rings to communicate with the kernel. See a sample application I wrote that natively does this:
http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
IO is done by filling out an io_uring_sqe, and updating the SQ ring. The format of the sqe is as follows:
struct io_uring_sqe {
__u8 opcode; /* type of operation for this sqe */
__u8 flags; /* IOSQE_ flags */
__u16 ioprio; /* ioprio for the request */
__s32 fd; /* file descriptor to do IO on */
__u64 off; /* offset into file */
union {
void *addr; /* buffer or iovecs */
__u64 __pad;
};
__u32 len; /* buffer size or number of iovecs */
union {
__kernel_rwf_t rw_flags;
__u32 fsync_flags;
};
__u16 buf_index; /* index into fixed buffers, if used */
__u16 __pad2;
__u32 __pad3;
__u64 user_data; /* data to be passed back at completion time */
};Most of this is self explanatory. The ->user_data field is passed back through a completion event, so the application can track IOs individually.
Completions are posted on the CQ ring when an sqe completes, they are a struct io_uring_cqe and the format is as follows:
struct io_uring_cqe {
__u64 user_data; /* sqe->data submission passed back */
__s32 res; /* result code for this event */
__u32 flags;
};To either submit IO or reap completions, there's a 2nd new system call:
io_uring_enter(fd, to_submit, min_complete, flags)
Initiates IO against the rings mapped to this fd, or waits for them to complete, or both The behavior is controlled by the parameters passed in. If 'min_complete' is non-zero, then we'll try and submit new IO. If IORING_ENTER_GETEVENTS is set, the kernel will wait for 'min_complete' events, if they aren't already available.
The sample application mentioned above uses the rings directly, but for most uses cases, I intend to have the necessary support in a liburing library that abstracts it enough for application to use in a performant way, without having to deal with the intricacies of the ring. There's already some basic support there and a few test applications, but that side definitely needs some work. Find that repo here:
git://git.kernel.dk/liburing
io_uring is designed to be fast and scalable. I've demonstrated 1.6M 4k IOPS from a single core on my aging test box, and on the latency front, we're also doing extremely well. It's designed to both be async and batching, if you wish, the application gets to control how to use that side.
If you want to play with io_uring, see the sample app above, the liburing repo, or the fio io_uring engine as well.
Patches are against 5.0-rc1 (ish), and can also be found in my 'io_uring' git branch:
git://git.kernel.dk/linux-block io_uring
Since v2
- Separate fixed buffers from sqe entries. register/unregister them through the new io_uring_register(2) system call
- sqe->index is now sqe->buf_index to make it clearer
- fixed buffers require sqe->flags to have IOSQE_FIXED_BUFFER set
- Add sqe field that is passed back at completion through the cqe, instead of passing back the original sqe index. This is more useful as it allows per-life of IO data, ->index did not.
- Cleanup async IO punting
- Don't punt O_DIRECT writes to async handling
- Make sq thread just for polling (submissions and completions)
- Always enable sq workqueue for async offload
- Use GFP_ATOMIC for req allocation
- Fix bio_vec being an unknown type on some kconfigs
- New IORING_OP_FSYNC implementation
- Add fixed fileset support through io_uring_register(2)
- Integrate workqueue support into main patchset
- Fix io_sq_thread() logic for when to grab current->mm
- Fix io_sq_thread() off-by-one
- Improve polling performance for multiple files in an io_uring context
- Have CONFIG_IO_URING select ANON_INODES
- Don't make io_kiocb->ki_flags atomic
- Be fully consistent in naming, for some reason we used the same mess that aio.c is, where io_kiocb,kiocb,iocb are used interchangably. 'req' is now always io_kiocb, 'kiocb' is always kiocb.
- Rename KIOCB_F_* flags as they are req flags, REQ_F_*.
Documentation/filesystems/vfs.txt | 3 + arch/x86/entry/syscalls/syscall_64.tbl | 3 + block/bio.c | 59 +- fs/Makefile | 1 + fs/block_dev.c | 19 +- fs/file.c | 15 +- fs/file_table.c | 9 +- fs/gfs2/file.c | 2 + fs/io_uring.c | 2023 ++++++++++++++++++++++++ fs/iomap.c | 48 +- fs/xfs/xfs_file.c | 1 + include/linux/bio.h | 14 + include/linux/blk_types.h | 1 + include/linux/file.h | 2 + include/linux/fs.h | 6 +- include/linux/iomap.h | 1 + include/linux/syscalls.h | 7 + include/uapi/linux/io_uring.h | 147 ++ init/Kconfig | 9 + kernel/sys_ni.c | 3 + 20 files changed, 2334 insertions(+), 39 deletions(-)
Signature · 8 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 1/16 fs: add an iopoll method to struct file_operations
From: Christoph Hellwig <[email protected]>
This new methods is used to explicitly poll for I/O completion for an iocb. It must be called for any iocb submitted asynchronously (that is with a non-null ki_complete) which has the IOCB_HIPRI flag set.
The method is assisted by a new ki_cookie field in struct iocb to store the polling cookie.
TODO: we can probably union ki_cookie with the existing hint and I/O priority fields to avoid struct kiocb growth.
--- Documentation/filesystems/vfs.txt | 3 +++ include/linux/fs.h | 2 ++ 2 files changed, 5 insertions(+)
Documentation/filesystems/vfs.txt +3 -0
@@ -857,6 +857,7 @@ struct file_operations { ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *); ssize_t (*read_iter) (struct kiocb *, struct iov_iter *); ssize_t (*write_iter) (struct kiocb *, struct iov_iter *); + int (*iopoll)(struct kiocb *kiocb, bool spin); int (*iterate) (struct file *, struct dir_context *); int (*iterate_shared) (struct file *, struct dir_context *); __poll_t (*poll) (struct file *, struct poll_table_struct *);
@@ -902,6 +903,8 @@ otherwise noted. write_iter: possibly asynchronous write with iov_iter as source + iopoll: called when aio wants to poll for completions on HIPRI iocbs + iterate: called when the VFS needs to read the directory contents iterate_shared: called when the VFS needs to read the directory contents
include/linux/fs.h +2 -0
@@ -310,6 +310,7 @@ struct kiocb {int ki_flags;u16 ki_hint;u16 ki_ioprio;/* See linux/ioprio.h */ +unsigned int ki_cookie;/* for ->iopoll */ } __randomize_layout;static inline bool is_sync_kiocb (struct kiocb *kiocb)
@@ -1786,6 +1787,7 @@ struct file_operations {ssize_t (*write) (struct file *,const char __user *,size_t ,loff_t *);ssize_t (*read_iter) (struct kiocb *,struct iov_iter *);ssize_t (*write_iter) (struct kiocb *,struct iov_iter *); +int (*iopoll)(struct kiocb *kiocb,bool spin);int (*iterate) (struct file *,struct dir_context *);int (*iterate_shared) (struct file *,struct dir_context *);__poll_t (*poll) (struct file *,struct poll_table_struct *);
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 2/16 block: wire up block device iopoll method
In reply to Jens AxboeFrom: Christoph Hellwig <[email protected]>
Just call blk_poll on the iocb cookie, we can derive the block device from the inode trivially.
--- fs/block_dev.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
fs/block_dev.c +10 -0
@@ -279,6 +279,14 @@ struct blkdev_dio {static struct bio_set blkdev_dio_pool; +static int blkdev_iopoll (struct kiocb *kiocb,bool wait) +{ +struct block_device *bdev =I_BDEV (kiocb->ki_filp->f_mapping->host); +struct request_queue *q =bdev_get_queue (bdev); + +return blk_poll (q,READ_ONCE (kiocb->ki_cookie), wait); +} +static void blkdev_bio_end_io (struct bio *bio) {struct blkdev_dio *dio = bio->bi_private;
@@ -396,6 +404,7 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) bio->bi_opf |=REQ_HIPRI ; qc =submit_bio (bio); +WRITE_ONCE (iocb->ki_cookie, qc);break ; }
@@ -2068,6 +2077,7 @@ const struct file_operations def_blk_fops = { .llseek = block_llseek, .read_iter = blkdev_read_iter, .write_iter = blkdev_write_iter, + .iopoll = blkdev_iopoll, .mmap = generic_file_mmap, .fsync = blkdev_fsync, .unlocked_ioctl = block_ioctl,
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 3/16 block: add bio_set_polled() helper
In reply to Jens AxboeFor the upcoming async polled IO, we can't sleep allocating requests. If we do, then we introduce a deadlock where the submitter already has async polled IO in-flight, but can't wait for them to complete since polled requests must be active found and reaped.
Utilize the helper in the blockdev DIRECT_IO code.
--- fs/block_dev.c | 4 ++-- include/linux/bio.h | 14 ++++++++++++++ 2 files changed, 16 insertions(+), 2 deletions(-)
fs/block_dev.c +2 -2
@@ -233,7 +233,7 @@ __blkdev_direct_IO_simple(struct kiocb *iocb, struct iov_iter *iter,task_io_account_write (ret); }if (iocb->ki_flags &IOCB_HIPRI ) - bio.bi_opf |=REQ_HIPRI ; +bio_set_polled (&bio, iocb); qc =submit_bio (&bio);for (;;) {
@@ -401,7 +401,7 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) nr_pages =iov_iter_npages (iter,BIO_MAX_PAGES );if (!nr_pages) {if (iocb->ki_flags &IOCB_HIPRI ) - bio->bi_opf |=REQ_HIPRI ; +bio_set_polled (bio, iocb); qc =submit_bio (bio);WRITE_ONCE (iocb->ki_cookie, qc);
include/linux/bio.h +14 -0
@@ -823,5 +823,19 @@ static inline int bio_integrity_add_page(struct bio *bio, struct page *page,#endif /* CONFIG_BLK_DEV_INTEGRITY */ +/* +* Mark a bio as polled. Note that for async polled IO, the caller must +* expect -EWOULDBLOCK if we cannot allocate a request (or other resources). +* We cannot block waiting for requests on polled IO, as those completions +* must be found by the caller. This is different than IRQ driven IO, where +* it's safe to wait for IO to complete. +*/ +static inline void bio_set_polled (struct bio *bio,struct kiocb *kiocb) +{ + bio->bi_opf |=REQ_HIPRI ; +if (!is_sync_kiocb (kiocb)) + bio->bi_opf |=REQ_NOWAIT ; +} +#endif /* CONFIG_BLOCK */ #endif /* __LINUX_BIO_H */
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 4/16 iomap: wire up the iopoll method
In reply to Jens AxboeFrom: Christoph Hellwig <[email protected]>
Store the request queue the last bio was submitted to in the iocb private data in addition to the cookie so that we find the right block device. Also refactor the common direct I/O bio submission code into a nice little helper.
Modified to use bio_set_polled().
--- fs/gfs2/file.c | 2 ++ fs/iomap.c | 43 ++++++++++++++++++++++++++++--------------- fs/xfs/xfs_file.c | 1 + include/linux/iomap.h | 1 + 4 files changed, 32 insertions(+), 15 deletions(-)
fs/gfs2/file.c +2 -0
@@ -1280,6 +1280,7 @@ const struct file_operations gfs2_file_fops = { .llseek = gfs2_llseek, .read_iter = gfs2_file_read_iter, .write_iter = gfs2_file_write_iter, + .iopoll = iomap_dio_iopoll, .unlocked_ioctl = gfs2_ioctl, .mmap = gfs2_mmap, .open = gfs2_open,
@@ -1310,6 +1311,7 @@ const struct file_operations gfs2_file_fops_nolock = { .llseek = gfs2_llseek, .read_iter = gfs2_file_read_iter, .write_iter = gfs2_file_write_iter, + .iopoll = iomap_dio_iopoll, .unlocked_ioctl = gfs2_ioctl, .mmap = gfs2_mmap, .open = gfs2_open,
fs/iomap.c +28 -15
82 lines not on this page · raw message
fs/xfs/xfs_file.c +1 -0
@@ -1203,6 +1203,7 @@ const struct file_operations xfs_file_operations = { .write_iter = xfs_file_write_iter, .splice_read = generic_file_splice_read, .splice_write = iter_file_splice_write, + .iopoll = iomap_dio_iopoll, .unlocked_ioctl = xfs_file_ioctl, #ifdef CONFIG_COMPAT .compat_ioctl = xfs_file_compat_ioctl,
include/linux/iomap.h +1 -0
@@ -162,6 +162,7 @@ typedef int (iomap_dio_end_io_t)(struct kiocb *iocb, ssize_t ret, unsigned flags);ssize_t iomap_dio_rw (struct kiocb *iocb,struct iov_iter *iter,const struct iomap_ops *ops,iomap_dio_end_io_t end_io); +int iomap_dio_iopoll (struct kiocb *kiocb,bool spin);#ifdef CONFIG_SWAP struct file ;
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 5/16 Add io_uring IO interface
In reply to Jens AxboeThe submission queue (SQ) and completion queue (CQ) rings are shared between the application and the kernel. This eliminates the need to copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions are generated in the form of io_uring_sqe data structures. The SQ ring is an index into the io_uring_sqe array, which makes it possible to submit a batch of IOs without them being contiguous in the ring. The CQ ring is always contiguous, as completion events are inherently unordered and can point to any io_uring_iocb.
Two new system calls are added for this:
io_uring_setup(entries, iovecs, params)
Sets up a context for doing async IO. On success, returns a file descriptor that the application can mmap to gain access to the SQ ring, CQ ring, and io_uring_iocbs.
io_uring_enter(fd, to_submit, min_complete, flags)
Initiates IO against the rings mapped to this fd, or waits for them to complete, or both The behavior is controlled by the parameters passed in. If 'min_complete' is non-zero, then we'll try and submit new IO. If IORING_ENTER_GETEVENTS is set, the kernel will wait for 'min_complete' events, if they aren't already available.
With this setup, it's possible to do async IO with a single system call. Future developments will enable polled IO with this interface, and polled submission as well. The latter will enable an application to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO as well. We will only punt to an async context if the command would need to wait for IO on the device side. Any data that can be accessed directly in the page cache is done inline. This avoids the slowness issue of usual threadpools, since cached data is accessed as quickly as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
--- arch/x86/entry/syscalls/syscall_64.tbl | 2 + fs/Makefile | 1 + fs/io_uring.c | 925 +++++++++++++++++++++++++ include/linux/syscalls.h | 5 + include/uapi/linux/io_uring.h | 96 +++ init/Kconfig | 9 + kernel/sys_ni.c | 2 + 7 files changed, 1040 insertions(+) create mode 100644 fs/io_uring.c create mode 100644 include/uapi/linux/io_uring.h
arch/x86/entry/syscalls/syscall_64.tbl +2 -0
@@ -343,6 +343,8 @@ 332 common statx __x64_sys_statx 333 common io_pgetevents __x64_sys_io_pgetevents 334 common rseq __x64_sys_rseq +335 common io_uring_setup __x64_sys_io_uring_setup +336 common io_uring_enter __x64_sys_io_uring_enter # # x32-specific system call numbers start at 512 to avoid cache impact
fs/Makefile +1 -0
@@ -30,6 +30,7 @@ obj-$(CONFIG_TIMERFD) += timerfd.o obj-$(CONFIG_EVENTFD) += eventfd.o obj-$(CONFIG_USERFAULTFD) += userfaultfd.o obj-$(CONFIG_AIO) += aio.o +obj-$(CONFIG_IO_URING) += io_uring.o obj-$(CONFIG_FS_DAX) += dax.o obj-$(CONFIG_FS_ENCRYPTION) += crypto/ obj-$(CONFIG_FILE_LOCKING) += locks.o
fs/io_uring.c new file +925 -0
925 lines not on this page · raw message
include/linux/syscalls.h +5 -0
@@ -69,6 +69,7 @@ struct file_handle;struct sigaltstack ;struct rseq ;union bpf_attr ; +struct io_uring_params ;#include <linux/types.h> #include <linux/aio_abi.h>
@@ -309,6 +310,10 @@ asmlinkage long sys_io_pgetevents_time32(aio_context_t ctx_id,struct io_event __user *events, struct old_timespec32 __user *timeout, const struct __aio_sigset *sig); +asmlinkage long sys_io_uring_setup (u32 entries, +struct io_uring_params __user *p); +asmlinkage long sys_io_uring_enter (unsigned int fd,u32 to_submit, +u32 min_complete,u32 flags);/* fs/xattr.c */ asmlinkage long sys_setxattr(const char __user *path, const char __user *name,
include/uapi/linux/io_uring.h new file +96 -0
96 lines not on this page · raw message
init/Kconfig +9 -0
@@ -1402,6 +1402,15 @@ config AIO by some high performance threaded applications. Disabling this option saves about 7k. +config IO_URING + bool "Enable IO uring support" if EXPERT + select ANON_INODES + default y + help + This option enables support for the io_uring interface, enabling + applications to submit and completion IO through submission and + completion rings that are shared between the kernel and application. + config ADVISE_SYSCALLS bool "Enable madvise/fadvise syscalls" if EXPERT default y
kernel/sys_ni.c +2 -0
@@ -46,6 +46,8 @@ COND_SYSCALL(io_getevents);COND_SYSCALL (io_pgetevents);COND_SYSCALL_COMPAT (io_getevents);COND_SYSCALL_COMPAT (io_pgetevents); +COND_SYSCALL (io_uring_setup); +COND_SYSCALL (io_uring_enter);/* fs/xattr.c */
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 6/16 io_uring: add fsync support
In reply to Jens AxboeFrom: Christoph Hellwig <[email protected]>
Add a new fsync opcode, which either syncs a range if one is passed, or the whole file if the offset and length fields are both cleared to zero. A flag is provided to use fdatasync semantics, that is only force out metadata which is required to retrieve the file data, but not others like metadata.
--- fs/io_uring.c | 33 +++++++++++++++++++++++++++++++++ include/uapi/linux/io_uring.h | 8 +++++++- 2 files changed, 40 insertions(+), 1 deletion(-)
fs/io_uring.c +33 -0
@@ -419,6 +419,36 @@ static ssize_t io_write(struct io_kiocb *req, const struct io_uring_sqe *sqe,return ret; } +static int io_fsync (struct io_kiocb *req,const struct io_uring_sqe *sqe, +bool force_nonblock) +{ +struct io_ring_ctx *ctx = req->ki_ctx; +loff_t end = sqe->off + sqe->len; +struct file *file; +int ret; + +/* fsync always requires a blocking context */ +if (force_nonblock) +return -EAGAIN ; + +if (unlikely (sqe->addr)) +return -EINVAL ; +if (unlikely (sqe->fsync_flags & ~IORING_FSYNC_DATASYNC )) +return -EINVAL ; + + file =fget (sqe->fd); +if (unlikely (!file)) +return -EBADF ; + + ret =vfs_fsync_range (file, sqe->off, end >0 ? end :LLONG_MAX , + sqe->fsync_flags &IORING_FSYNC_DATASYNC ); + +fput (file); +io_cqring_fill_event (ctx, sqe->user_data, ret,0 ); +io_free_req (req); +return 0 ; +} +static int __io_submit_sqe (struct io_ring_ctx *ctx,struct io_kiocb *req,struct sqe_submit *s,bool force_nonblock) {
@@ -441,6 +471,9 @@ static int __io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,case IORING_OP_WRITEV : ret =io_write (req, sqe, force_nonblock);break ; +case IORING_OP_FSYNC : + ret =io_fsync (req, sqe, force_nonblock); +break ;default : ret = -EINVAL ;break ;
include/uapi/linux/io_uring.h +7 -1
@@ -27,7 +27,7 @@ struct io_uring_sqe {__u32 len;/* buffer size or number of iovecs */ union {__kernel_rwf_t rw_flags; -__u32 __resv; +__u32 fsync_flags; };__u64 __pad2;__u64 user_data;/* data to be passed back at completion time */
@@ -35,6 +35,12 @@ struct io_uring_sqe {#define IORING_OP_READV 1#define IORING_OP_WRITEV 2 +#define IORING_OP_FSYNC 3 + +/* +* sqe->fsync_flags +*/ +#define IORING_FSYNC_DATASYNC (1 << 0)/* * IO completion data structure (Completion Queue Entry)
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 7/16 io_uring: support for IO polling
In reply to Jens AxboeAdd support for a polled io_uring context. When a read or write is submitted to a polled context, the application must poll for completions on the CQ ring through io_uring_enter(2). Polled IO may not generate IRQ completions, hence they need to be actively found by the application itself.
To use polling, io_uring_setup() must be used with the IORING_SETUP_IOPOLL flag being set. It is illegal to mix and match polled and non-polled IO on an io_uring.
--- fs/io_uring.c | 306 ++++++++++++++++++++++++++++++++-- include/uapi/linux/io_uring.h | 5 + 2 files changed, 297 insertions(+), 14 deletions(-)
fs/io_uring.c +292 -14
@@ -55,6 +55,11 @@ struct io_cq_ring {struct io_uring_cqe cqes[]; }; +struct list_multi { +struct list_head list; +unsigned multi; +}; +struct io_ring_ctx {struct percpu_ref refs;
@@ -79,13 +84,17 @@ struct io_ring_ctx {struct completion ctx_done; +/* iopoll submission state */ struct { -struct mutex uring_lock; -wait_queue_head_t wait; +spinlock_t poll_lock; +struct list_multi poll_submitted; }____cacheline_aligned_in_smp ;struct { +struct list_multi poll_completing;spinlock_t completion_lock; +struct mutex uring_lock; +wait_queue_head_t wait; }____cacheline_aligned_in_smp ; };
@@ -109,10 +118,13 @@ struct io_kiocb {struct list_head ki_list;unsigned long ki_flags;#define REQ_F_FORCE_NONBLOCK 1/* inline submission attempt */ +#define REQ_F_IOPOLL_COMPLETED 2/* polled IO has completed */ +#define REQ_F_IOPOLL_EAGAIN 4/* submission got EAGAIN */ u64 ki_user_data; };#define IO_PLUG_THRESHOLD 2 +#define IO_IOPOLL_BATCH 8static struct kmem_cache *req_cachep;
@@ -144,6 +156,9 @@ static struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)spin_lock_init (&ctx->completion_lock);init_waitqueue_head (&ctx->wait); +spin_lock_init (&ctx->poll_lock); +INIT_LIST_HEAD (&ctx->poll_submitted.list); +INIT_LIST_HEAD (&ctx->poll_completing.list);mutex_init (&ctx->uring_lock);return ctx;
@@ -195,12 +210,209 @@ static void io_ring_drop_ctx_refs(struct io_ring_ctx *ctx, unsigned refs)wake_up (&ctx->wait); } +static void io_free_req_many (struct io_ring_ctx *ctx,void **reqs,int *nr) +{ +if (*nr) { +kmem_cache_free_bulk (req_cachep, *nr, reqs); +io_ring_drop_ctx_refs (ctx, *nr); + *nr =0 ; + } +} +static void io_free_req (struct io_kiocb *req) {kmem_cache_free (req_cachep, req);io_ring_drop_ctx_refs (req->ki_ctx,1 ); } +static void io_multi_list_splice (struct list_multi *src,struct list_multi *dst) +{ +list_splice_tail_init (&src->list, &dst->list); + dst->multi |= src->multi; +} + +/* +* Track whether we have multiple files in our lists. This will impact how +* we do polling eventually, not spinning if we're on potentially on different +* devices. +*/ +static void io_multi_list_add (struct io_kiocb *req,struct list_multi *list) +{ +if (list_empty (&list->list)) { + list->multi =0 ; + }else if (!list->multi) { +struct io_kiocb *list_req; + + list_req = list_first_entry(&list->list, struct io_kiocb, + ki_list); +if (list_req->rw.ki_filp != req->rw.ki_filp) + list->multi =1 ; + } + +/* +* For fast devices, IO may have already completed. If it has, add +* it to the front so we find it first. We can't add to the poll_done +* list as that's unlocked from the completion side. +*/ +if (req->ki_flags &REQ_F_IOPOLL_COMPLETED ) +list_add (&req->ki_list, &list->list); +else +list_add_tail (&req->ki_list, &list->list); +} + +/* +* Find and free completed poll iocbs +*/ +static void io_iopoll_reap (struct io_ring_ctx *ctx,unsigned int *nr_events) +{ +void *reqs[IO_IOPOLL_BATCH ]; +struct io_kiocb *req, *n; +int to_free =0 ; + +list_for_each_entry_safe (req, n, &ctx->poll_completing.list, ki_list) { +if (!(req->ki_flags &REQ_F_IOPOLL_COMPLETED )) +continue ; +if (to_free ==ARRAY_SIZE (reqs)) +io_free_req_many (ctx, reqs, &to_free); + +list_del (&req->ki_list); + reqs[to_free++] = req; + +fput (req->rw.ki_filp); + (*nr_events)++; + } + +if (to_free) +io_free_req_many (ctx, reqs, &to_free); +} + +static int io_do_iopoll (struct io_ring_ctx *ctx,unsigned int *nr_events, +long min) +{ +int polled, found, ret; +struct io_kiocb *req; +bool spin; + +/* +* Only spin for completions if we don't have multiple devices hanging +* off our complete list, and we're under the requested amount. +*/ + spin = !ctx->poll_completing.multi && (*nr_events < min); + + polled = found =0 ; +list_for_each_entry (req, &ctx->poll_completing.list, ki_list) { +struct kiocb *kiocb = &req->rw; + +if (req->ki_flags &REQ_F_IOPOLL_COMPLETED ) { +/* force caller reap */ + found =0 ; +break ; + } + + found++; + ret = kiocb->ki_filp->f_op->iopoll (kiocb, spin); +if (ret <0 ) +return ret; + + polled += ret; +if (polled && spin) + spin = false; + } + +return found; +} + +/* +* Poll for a mininum of 'min' events, and a maximum of 'max'. Note that if +* min == 0 we consider that a non-spinning poll check - we'll still enter +* the driver poll loop, but only as a non-spinning completion check. +*/ +static int io_iopoll_getevents (struct io_ring_ctx *ctx,unsigned int *nr_events, +long min) +{ +int found; + +/* +* Check if we already have done events that satisfy what we need +*/ +if (!list_empty (&ctx->poll_completing.list)) { +io_iopoll_reap (ctx, nr_events); +if (min && *nr_events >= min) +return 0 ; + } + +/* +* Take in a new working set from the submitted list, if possible. +*/ +if (!list_empty_careful (&ctx->poll_submitted.list)) { +spin_lock (&ctx->poll_lock); +io_multi_list_splice (&ctx->poll_submitted, + &ctx->poll_completing); +spin_unlock (&ctx->poll_lock); + } + +if (list_empty (&ctx->poll_completing.list)) +return 0 ; + +/* +* Check again now that we have a new batch. +*/ +io_iopoll_reap (ctx, nr_events); +if (min && *nr_events >= min) +return 0 ; + +do { + found =io_do_iopoll (ctx, nr_events, min); +if (found <=0 ) +break ; + }while (ctx->poll_completing.multi && (min && *nr_events >= min)); + +io_iopoll_reap (ctx, nr_events); +if (*nr_events >= min) +return 0 ; + +return found; +} + +/* +* We can't just wait for polled events to come to us, we have to actively +* find and complete them. +*/ +static void io_iopoll_reap_events (struct io_ring_ctx *ctx) +{ +if (!(ctx->flags &IORING_SETUP_IOPOLL )) +return ; + +mutex_lock (&ctx->uring_lock); +while (!list_empty_careful (&ctx->poll_submitted.list) || + !list_empty (&ctx->poll_completing.list)) { +unsigned int nr_events =0 ; + +io_iopoll_getevents (ctx, &nr_events,1 ); + } +mutex_unlock (&ctx->uring_lock); +} + +static int io_iopoll_check (struct io_ring_ctx *ctx,unsigned *nr_events, +long min) +{ +int ret =0 ; + +while (!*nr_events || !need_resched ()) { +int tmin =0 ; + +if (*nr_events < min) + tmin = min - *nr_events; + + ret =io_iopoll_getevents (ctx, nr_events, tmin); +if (ret <=0 ) +break ; + ret =0 ; + } + +return ret; +} +static void kiocb_end_write (struct kiocb *kiocb) {if (kiocb->ki_flags &IOCB_WRITE ) {
@@ -216,18 +428,16 @@ static void kiocb_end_write(struct kiocb *kiocb) } } -static void io_cqring_fill_event (struct io_ring_ctx *ctx,u64 ki_user_data, -long res,unsigned ev_flags) +static void __io_cqring_fill_event (struct io_ring_ctx *ctx,u64 ki_user_data, +long res,unsigned ev_flags) {struct io_uring_cqe *cqe; -unsigned long flags;/* * If we can't get a cq entry, userspace overflowed the * submission (by quite a lot). Increment the overflow count in * the ring. */ -spin_lock_irqsave (&ctx->completion_lock, flags); cqe =io_peek_cqring (ctx);if (cqe) { cqe->user_data = ki_user_data;
@@ -237,6 +447,15 @@ static void io_cqring_fill_event(struct io_ring_ctx *ctx, u64 ki_user_data,io_inc_cqring (ctx); }else ctx->cq_ring->overflow++; +} + +static void io_cqring_fill_event (struct io_ring_ctx *ctx,u64 ki_user_data, +long res,unsigned ev_flags) +{ +unsigned long flags; + +spin_lock_irqsave (&ctx->completion_lock, flags); +__io_cqring_fill_event (ctx, ki_user_data, res, ev_flags);spin_unlock_irqrestore (&ctx->completion_lock, flags); }
@@ -260,9 +479,41 @@ static void io_complete_scqring_rw(struct kiocb *kiocb, long res, long res2)io_free_req (req); } +static void io_complete_scqring_iopoll (struct kiocb *kiocb,long res,long res2) +{ + struct io_kiocb *req = container_of(kiocb, struct io_kiocb, rw); + +kiocb_end_write (kiocb); + +if (unlikely (res == -EAGAIN )) { + req->ki_flags |=REQ_F_IOPOLL_EAGAIN ; + }else { +struct io_ring_ctx *ctx = req->ki_ctx; + +__io_cqring_fill_event (ctx, req->ki_user_data, res,0 ); + req->ki_flags |=REQ_F_IOPOLL_COMPLETED ; + } +} + +/* +* After the iocb has been issued, it's safe to be found on the poll list. +* Adding the kiocb to the list AFTER submission ensures that we don't +* find it from a io_getevents() thread before the issuer is done accessing +* the kiocb cookie. +*/ +static void io_iopoll_req_issued (struct io_kiocb *req) +{ +struct io_ring_ctx *ctx = req->ki_ctx; + +spin_lock (&ctx->poll_lock); +io_multi_list_add (req, &ctx->poll_submitted); +spin_unlock (&ctx->poll_lock); +} +static int io_prep_rw (struct io_kiocb *req,const struct io_uring_sqe *sqe,bool force_nonblock) { +struct io_ring_ctx *ctx = req->ki_ctx;struct kiocb *kiocb = &req->rw;int ret;
@@ -288,12 +539,21 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe, kiocb->ki_flags |=IOCB_NOWAIT ; req->ki_flags |=REQ_F_FORCE_NONBLOCK ; } -if (kiocb->ki_flags &IOCB_HIPRI ) { - ret = -EINVAL ; - goto out_fput; - } +if (ctx->flags &IORING_SETUP_IOPOLL ) { + ret = -EOPNOTSUPP ; +if (!(kiocb->ki_flags &IOCB_DIRECT ) || + !kiocb->ki_filp->f_op->iopoll) + goto out_fput; - kiocb->ki_complete = io_complete_scqring_rw; + kiocb->ki_flags |=IOCB_HIPRI ; + kiocb->ki_complete = io_complete_scqring_iopoll; + }else { +if (kiocb->ki_flags &IOCB_HIPRI ) { + ret = -EINVAL ; + goto out_fput; + } + kiocb->ki_complete = io_complete_scqring_rw; + }return 0 ; out_fput:fput (kiocb->ki_filp);
@@ -431,6 +691,8 @@ static int io_fsync(struct io_kiocb *req, const struct io_uring_sqe *sqe,if (force_nonblock)return -EAGAIN ; +if (unlikely (req->ki_ctx->flags &IORING_SETUP_IOPOLL )) +return -EINVAL ;if (unlikely (sqe->addr))return -EINVAL ;if (unlikely (sqe->fsync_flags & ~IORING_FSYNC_DATASYNC ))
@@ -479,7 +741,16 @@ static int __io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,break ; } -return ret; +if (ret) +return ret; + +if (ctx->flags &IORING_SETUP_IOPOLL ) { +if (req->ki_flags &REQ_F_IOPOLL_EAGAIN ) +return -EAGAIN ; +io_iopoll_req_issued (req); + } + +return 0 ; }static void io_sq_wq_submit_work (struct work_struct *work)
@@ -649,12 +920,17 @@ static int __io_uring_enter(struct io_ring_ctx *ctx, unsigned to_submit,return ret; }if (flags &IORING_ENTER_GETEVENTS ) { +unsigned nr_events =0 ;int get_ret;if (!ret && to_submit) min_complete =0 ; - get_ret =io_cqring_wait (ctx, min_complete); +if (ctx->flags &IORING_SETUP_IOPOLL ) + get_ret =io_iopoll_check (ctx, &nr_events, + min_complete); +else + get_ret =io_cqring_wait (ctx, min_complete);if (get_ret <0 && !ret) ret = get_ret; }
@@ -722,6 +998,7 @@ static void io_free_scq_urings(struct io_ring_ctx *ctx)static void io_ring_ctx_free (struct io_ring_ctx *ctx) {io_sq_offload_stop (ctx); +io_iopoll_reap_events (ctx);io_free_scq_urings (ctx);percpu_ref_exit (&ctx->refs);kfree (ctx);
@@ -730,6 +1007,7 @@ static void io_ring_ctx_free(struct io_ring_ctx *ctx)static void io_ring_ctx_wait_and_kill (struct io_ring_ctx *ctx) {percpu_ref_kill (&ctx->refs); +io_iopoll_reap_events (ctx);wait_for_completion (&ctx->ctx_done);io_ring_ctx_free (ctx); }
@@ -937,7 +1215,7 @@ SYSCALL_DEFINE2(io_uring_setup, u32, entries,return -EINVAL ; } -if (p.flags) +if (p.flags & ~IORING_SETUP_IOPOLL )return -EINVAL ; ret =io_uring_create (entries, &p);
include/uapi/linux/io_uring.h +5 -0
@@ -33,6 +33,11 @@ struct io_uring_sqe {__u64 user_data;/* data to be passed back at completion time */ }; +/* +* io_uring_setup() flags +*/ +#define IORING_SETUP_IOPOLL (1 << 0)/* io_context is polled */ +#define IORING_OP_READV 1#define IORING_OP_WRITEV 2#define IORING_OP_FSYNC 3
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 8/16 io_uring: add submission side request cache
In reply to Jens AxboeWe have to add each submitted polled request to the io_ring_ctx poll_submitted list, which means we have to grab the poll_lock. We already use the block plug to batch submissions if we're doing a batch of IO submissions, extend that to cover the poll requests internally as well.
--- fs/io_uring.c | 122 +++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 105 insertions(+), 17 deletions(-)
fs/io_uring.c +105 -17
@@ -126,6 +126,21 @@ struct io_kiocb {#define IO_PLUG_THRESHOLD 2#define IO_IOPOLL_BATCH 8 +struct io_submit_state { +struct io_ring_ctx *ctx; + +struct blk_plug plug; +#ifdef CONFIG_BLOCK +struct blk_plug_cb plug_cb; +#endif + +/* +* Polled iocbs that have been submitted, but not added to the ctx yet +*/ +struct list_multi req_list; +unsigned int req_count; +}; +static struct kmem_cache *req_cachep;static const struct file_operations io_scqring_fops;
@@ -495,19 +510,51 @@ static void io_complete_scqring_iopoll(struct kiocb *kiocb, long res, long res2) } } +/* +* Called either at the end of IO submission, or through a plug callback +* because we're going to schedule. Moves out local batch of requests to +* the ctx poll list, so they can be found for polling + reaping. +*/ +static void io_flush_state_reqs (struct io_ring_ctx *ctx, +struct io_submit_state *state) +{ +spin_lock (&ctx->poll_lock); +io_multi_list_splice (&state->req_list, &ctx->poll_submitted); +spin_unlock (&ctx->poll_lock); + + state->req_count =0 ; +} + +static void io_iopoll_req_add_list (struct io_kiocb *req) +{ +struct io_ring_ctx *ctx = req->ki_ctx; + +spin_lock (&ctx->poll_lock); +io_multi_list_add (req, &ctx->poll_submitted); +spin_unlock (&ctx->poll_lock); +} + +static void io_iopoll_req_add_state (struct io_submit_state *state, +struct io_kiocb *req) +{ +io_multi_list_add (req, &state->req_list); +if (++state->req_count >=IO_IOPOLL_BATCH ) +io_flush_state_reqs (state->ctx, state); +} +/* * After the iocb has been issued, it's safe to be found on the poll list. * Adding the kiocb to the list AFTER submission ensures that we don't * find it from a io_getevents() thread before the issuer is done accessing * the kiocb cookie. */ -static void io_iopoll_req_issued (struct io_kiocb *req) +static void io_iopoll_req_issued (struct io_submit_state *state, +struct io_kiocb *req) { -struct io_ring_ctx *ctx = req->ki_ctx; - -spin_lock (&ctx->poll_lock); -io_multi_list_add (req, &ctx->poll_submitted); -spin_unlock (&ctx->poll_lock); +if (!state || !IS_ENABLED (CONFIG_BLOCK )) +io_iopoll_req_add_list (req); +else +io_iopoll_req_add_state (state, req); } static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe,
@@ -712,7 +759,8 @@ static int io_fsync(struct io_kiocb *req, const struct io_uring_sqe *sqe, }static int __io_submit_sqe (struct io_ring_ctx *ctx,struct io_kiocb *req, -struct sqe_submit *s,bool force_nonblock) +struct sqe_submit *s,bool force_nonblock, +struct io_submit_state *state) {const struct io_uring_sqe *sqe = s->sqe;ssize_t ret;
@@ -747,7 +795,7 @@ static int __io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,if (ctx->flags &IORING_SETUP_IOPOLL ) {if (req->ki_flags &REQ_F_IOPOLL_EAGAIN )return -EAGAIN ; -io_iopoll_req_issued (req); +io_iopoll_req_issued (state, req); }return 0 ;
@@ -779,7 +827,7 @@ static void io_sq_wq_submit_work(struct work_struct *work)use_mm (ctx->sqo_mm);set_fs (USER_DS ); - ret =__io_submit_sqe (ctx, req, &req->work.submit, false); + ret =__io_submit_sqe (ctx, req, &req->work.submit, false,NULL );set_fs (old_fs);unuse_mm (ctx->sqo_mm);
@@ -792,7 +840,8 @@ static void io_sq_wq_submit_work(struct work_struct *work) current->files = old_files; } -static int io_submit_sqe (struct io_ring_ctx *ctx,struct sqe_submit *s) +static int io_submit_sqe (struct io_ring_ctx *ctx,struct sqe_submit *s, +struct io_submit_state *state) {struct io_kiocb *req;ssize_t ret;
@@ -801,7 +850,7 @@ static int io_submit_sqe(struct io_ring_ctx *ctx, struct sqe_submit *s)if (unlikely (!req))return -EAGAIN ; - ret =__io_submit_sqe (ctx, req, s, true); + ret =__io_submit_sqe (ctx, req, s, true, state);if (ret == -EAGAIN ) {memcpy (&req->work.submit, s,sizeof (*s));INIT_WORK (&req->work.work, io_sq_wq_submit_work);
@@ -814,6 +863,43 @@ static int io_submit_sqe(struct io_ring_ctx *ctx, struct sqe_submit *s)return ret; } +#ifdef CONFIG_BLOCK +static void io_state_unplug (struct blk_plug_cb *cb,bool from_schedule) +{ +struct io_submit_state *state; + + state = container_of(cb, struct io_submit_state, plug_cb); +if (!list_empty (&state->req_list.list)) +io_flush_state_reqs (state->ctx, state); +} +#endif + +/* +* Batched submission is done, ensure local IO is flushed out. +*/ +static void io_submit_state_end (struct io_submit_state *state) +{ +blk_finish_plug (&state->plug); +if (!list_empty (&state->req_list.list)) +io_flush_state_reqs (state->ctx, state); +} + +/* +* Start submission side cache. +*/ +static void io_submit_state_start (struct io_submit_state *state, +struct io_ring_ctx *ctx) +{ + state->ctx = ctx; +INIT_LIST_HEAD (&state->req_list.list); + state->req_count =0 ; +#ifdef CONFIG_BLOCK + state->plug_cb.callback = io_state_unplug; +blk_start_plug (&state->plug); +list_add (&state->plug_cb.list, &state->plug.cb_list); +#endif +} +static void io_inc_sqring (struct io_ring_ctx *ctx) {struct io_sq_ring *ring = ctx->sq_ring;
@@ -848,11 +934,13 @@ static bool io_peek_sqring(struct io_ring_ctx *ctx, struct sqe_submit *s)static int io_ring_submit (struct io_ring_ctx *ctx,unsigned int to_submit) { +struct io_submit_state state, *statep =NULL ;int i, ret =0 , submit =0 ; -struct blk_plug plug; -if (to_submit >IO_PLUG_THRESHOLD ) -blk_start_plug (&plug); +if (to_submit >IO_PLUG_THRESHOLD ) { +io_submit_state_start (&state, ctx); + statep = &state; + }for (i =0 ; i < to_submit; i++) {struct sqe_submit s;
@@ -860,7 +948,7 @@ static int io_ring_submit(struct io_ring_ctx *ctx, unsigned int to_submit)if (!io_peek_sqring (ctx, &s))break ; - ret =io_submit_sqe (ctx, &s); + ret =io_submit_sqe (ctx, &s, statep);if (ret)break ;
@@ -868,8 +956,8 @@ static int io_ring_submit(struct io_ring_ctx *ctx, unsigned int to_submit)io_inc_sqring (ctx); } -if (to_submit >IO_PLUG_THRESHOLD ) -blk_finish_plug (&plug); +if (statep) +io_submit_state_end (statep);return submit ? submit : ret; }
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 9/16 fs: add fget_many() and fput_many()
In reply to Jens AxboeSome uses cases repeatedly get and put references to the same file, but the only exposed interface is doing these one at the time. As each of these entail an atomic inc or dec on a shared structure, that cost can add up.
Add fget_many(), which works just like fget(), except it takes an argument for how many references to get on the file. Ditto fput_many(), which can drop an arbitrary number of references to a file.
--- fs/file.c | 15 ++++++++++----- fs/file_table.c | 9 +++++++-- include/linux/file.h | 2 ++ include/linux/fs.h | 4 +++- 4 files changed, 22 insertions(+), 8 deletions(-)
fs/file.c +10 -5
@@ -705,7 +705,7 @@ void do_close_on_exec(struct files_struct *files)spin_unlock (&files->file_lock); } -static struct file *__fget (unsigned int fd,fmode_t mask) +static struct file *__fget (unsigned int fd,fmode_t mask,unsigned int refs) {struct files_struct *files = current->files;struct file *file;
@@ -720,7 +720,7 @@ static struct file *__fget(unsigned int fd, fmode_t mask)*/ if (file->f_mode & mask) file =NULL ; -else if (!get_file_rcu (file)) +else if (!get_file_rcu_many (file, refs)) goto loop; }rcu_read_unlock ();
@@ -728,15 +728,20 @@ static struct file *__fget(unsigned int fd, fmode_t mask)return file; } +struct file *fget_many (unsigned int fd,unsigned int refs) +{ +return __fget (fd,FMODE_PATH , refs); +} +struct file *fget (unsigned int fd) { -return __fget (fd,FMODE_PATH ); +return fget_many (fd,1 ); }EXPORT_SYMBOL (fget);struct file *fget_raw (unsigned int fd) { -return __fget (fd,0 ); +return __fget (fd,0 ,1 ); }EXPORT_SYMBOL (fget_raw);
@@ -767,7 +772,7 @@ static unsigned long __fget_light(unsigned int fd, fmode_t mask)return 0 ;return (unsigned long )file; }else { - file =__fget (fd, mask); + file =__fget (fd, mask,1 );if (!file)return 0 ;return FDPUT_FPUT | (unsigned long )file;
fs/file_table.c +7 -2
@@ -326,9 +326,9 @@ void flush_delayed_fput(void) static DECLARE_DELAYED_WORK(delayed_fput_work, delayed_fput); -void fput (struct file *file) +void fput_many (struct file *file,unsigned int refs) { -if (atomic_long_dec_and_test (&file->f_count)) { +if (atomic_long_sub_and_test (refs, &file->f_count)) {struct task_struct *task = current;if (likely (!in_interrupt () && !(task->flags &PF_KTHREAD ))) {
@@ -347,6 +347,11 @@ void fput(struct file *file) } } +void fput (struct file *file) +{ +fput_many (file,1 ); +} +/* * synchronous analog of fput(); for kernel threads that might be needed * in some umount() (and thus can't use flush_delayed_fput() without
include/linux/file.h +2 -0
@@ -13,6 +13,7 @@struct file ;extern void fput (struct file *); +extern void fput_many (struct file *,unsigned int );struct file_operations ;struct vfsmount ;
@@ -44,6 +45,7 @@ static inline void fdput(struct fd fd) }extern struct file *fget (unsigned int fd); +extern struct file *fget_many (unsigned int fd,unsigned int refs);extern struct file *fget_raw (unsigned int fd);extern unsigned long __fdget (unsigned int fd);extern unsigned long __fdget_raw (unsigned int fd);
include/linux/fs.h +3 -1
@@ -952,7 +952,9 @@ static inline struct file *get_file(struct file *f)atomic_long_inc (&f->f_count);return f; } -#define get_file_rcu (x) atomic_long_inc_not_zero(&(x)->f_count) +#define get_file_rcu_many (x, cnt) \ + atomic_long_add_unless(&(x)->f_count, (cnt), 0) +#define get_file_rcu (x) get_file_rcu_many((x), 1)#define fput_atomic (x) atomic_long_add_unless(&(x)->f_count, -1, 1)#define file_count (x) atomic_long_read(&(x)->f_count)
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 10/16 io_uring: use fget/fput_many() for file references
In reply to Jens AxboeOn the submission side, add file reference batching to the io_submit_state. We get as many references as the number of iocbs we are submitting, and drop unused ones if we end up switching files. The assumption here is that we're usually only dealing with one fd, and if there are multiple, hopefuly they are at least somewhat ordered. Could trivially be extended to cover multiple fds, if needed.
On the completion side we do the same thing, except this is trivially done just locally in io_iopoll_reap().
--- fs/io_uring.c | 98 ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 85 insertions(+), 13 deletions(-)
fs/io_uring.c +85 -13
@@ -139,6 +139,15 @@ struct io_submit_state {*/ struct list_multi req_list;unsigned int req_count; + +/* +* File reference cache +*/ +struct file *file; +unsigned int fd; +unsigned int has_refs; +unsigned int used_refs; +unsigned int ios_left; };static struct kmem_cache *req_cachep;
@@ -282,8 +291,10 @@ static void io_iopoll_reap(struct io_ring_ctx *ctx, unsigned int *nr_events) {void *reqs[IO_IOPOLL_BATCH ];struct io_kiocb *req, *n; -int to_free =0 ; +int file_count, to_free; +struct file *file =NULL ; + file_count = to_free =0 ;list_for_each_entry_safe (req, n, &ctx->poll_completing.list, ki_list) {if (!(req->ki_flags &REQ_F_IOPOLL_COMPLETED ))continue ;
@@ -293,10 +304,26 @@ static void io_iopoll_reap(struct io_ring_ctx *ctx, unsigned int *nr_events)list_del (&req->ki_list); reqs[to_free++] = req; -fput (req->rw.ki_filp); +/* +* Batched puts of the same file, to avoid dirtying the +* file usage count multiple times, if avoidable. +*/ +if (!file) { + file = req->rw.ki_filp; + file_count =1 ; + }else if (file == req->rw.ki_filp) { + file_count++; + }else { +fput_many (file, file_count); + file = req->rw.ki_filp; + file_count =1 ; + } + (*nr_events)++; } +if (file) +fput_many (file, file_count);if (to_free)io_free_req_many (ctx, reqs, &to_free); }
@@ -557,14 +584,56 @@ static void io_iopoll_req_issued(struct io_submit_state *state,io_iopoll_req_add_state (state, req); } +static void io_file_put (struct io_submit_state *state,struct file *file) +{ +if (!state) { +fput (file); + }else if (state->file) { +int diff = state->has_refs - state->used_refs; + +if (diff) +fput_many (state->file, diff); + state->file =NULL ; + } +} + +/* +* Get as many references to a file as we have IOs left in this submission, +* assuming most submissions are for one file, or at least that each file +* has more than one submission. +*/ +static struct file *io_file_get (struct io_submit_state *state,int fd) +{ +if (!state) +return fget (fd); + +if (state->file) { +if (state->fd == fd) { + state->used_refs++; + state->ios_left--; +return state->file; + } +io_file_put (state,NULL ); + } + state->file =fget_many (fd, state->ios_left); +if (!state->file) +return NULL ; + + state->fd = fd; + state->has_refs = state->ios_left; + state->used_refs =1 ; + state->ios_left--; +return state->file; +} +static int io_prep_rw (struct io_kiocb *req,const struct io_uring_sqe *sqe, -bool force_nonblock) +bool force_nonblock,struct io_submit_state *state) {struct io_ring_ctx *ctx = req->ki_ctx;struct kiocb *kiocb = &req->rw;int ret; - kiocb->ki_filp =fget (sqe->fd); + kiocb->ki_filp =io_file_get (state, sqe->fd);if (unlikely (!kiocb->ki_filp))return -EBADF ; kiocb->ki_pos = sqe->off;
@@ -603,7 +672,7 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe, }return 0 ; out_fput: -fput (kiocb->ki_filp); +io_file_put (state, kiocb->ki_filp);return ret; }
@@ -628,7 +697,7 @@ static inline void io_rw_done(struct kiocb *req, ssize_t ret) }static ssize_t io_read (struct io_kiocb *req,const struct io_uring_sqe *sqe, -bool force_nonblock) +bool force_nonblock,struct io_submit_state *state) {struct iovec inline_vecs[UIO_FASTIOV ], *iovec = inline_vecs;void __user *buf = (void __user *) (uintptr_t ) sqe->addr;
@@ -637,7 +706,7 @@ static ssize_t io_read(struct io_kiocb *req, const struct io_uring_sqe *sqe,struct file *file;ssize_t ret; - ret =io_prep_rw (req, sqe, force_nonblock); + ret =io_prep_rw (req, sqe, force_nonblock, state);if (ret)return ret; file = kiocb->ki_filp;
@@ -672,7 +741,7 @@ static ssize_t io_read(struct io_kiocb *req, const struct io_uring_sqe *sqe, }static ssize_t io_write (struct io_kiocb *req,const struct io_uring_sqe *sqe, -bool force_nonblock) +bool force_nonblock,struct io_submit_state *state) {struct iovec inline_vecs[UIO_FASTIOV ], *iovec = inline_vecs;void __user *buf = (void __user *) (uintptr_t ) sqe->addr;
@@ -681,7 +750,7 @@ static ssize_t io_write(struct io_kiocb *req, const struct io_uring_sqe *sqe,struct file *file;ssize_t ret; - ret =io_prep_rw (req, sqe, force_nonblock); + ret =io_prep_rw (req, sqe, force_nonblock, state);if (ret)return ret; file = kiocb->ki_filp;
@@ -776,10 +845,10 @@ static int __io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req, ret = -EINVAL ;switch (sqe->opcode) {case IORING_OP_READV : - ret =io_read (req, sqe, force_nonblock); + ret =io_read (req, sqe, force_nonblock, state);break ;case IORING_OP_WRITEV : - ret =io_write (req, sqe, force_nonblock); + ret =io_write (req, sqe, force_nonblock, state);break ;case IORING_OP_FSYNC : ret =io_fsync (req, sqe, force_nonblock);
@@ -882,17 +951,20 @@ static void io_submit_state_end(struct io_submit_state *state)blk_finish_plug (&state->plug);if (!list_empty (&state->req_list.list))io_flush_state_reqs (state->ctx, state); +io_file_put (state,NULL ); }/* * Start submission side cache. */ static void io_submit_state_start (struct io_submit_state *state, -struct io_ring_ctx *ctx) +struct io_ring_ctx *ctx,unsigned max_ios) { state->ctx = ctx;INIT_LIST_HEAD (&state->req_list.list); state->req_count =0 ; + state->file =NULL ; + state->ios_left = max_ios;#ifdef CONFIG_BLOCK state->plug_cb.callback = io_state_unplug;blk_start_plug (&state->plug);
@@ -938,7 +1010,7 @@ static int io_ring_submit(struct io_ring_ctx *ctx, unsigned int to_submit)int i, ret =0 , submit =0 ;if (to_submit >IO_PLUG_THRESHOLD ) { -io_submit_state_start (&state, ctx); +io_submit_state_start (&state, ctx, to_submit); statep = &state; }
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 11/16 io_uring: batch io_kiocb allocation
In reply to Jens AxboeSimilarly to how we use the state->ios_left to know how many references to get to a file, we can use it to allocate the io_kiocb's we need in bulk.
--- fs/io_uring.c | 66 ++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 50 insertions(+), 16 deletions(-)
fs/io_uring.c +50 -16
@@ -140,6 +140,13 @@ struct io_submit_state {struct list_multi req_list;unsigned int req_count; +/* +* io_kiocb alloc cache +*/ +void *reqs[IO_IOPOLL_BATCH ]; +unsigned int free_reqs; +unsigned int cur_req; +/* * File reference cache */
@@ -209,29 +216,52 @@ static struct io_uring_cqe *io_peek_cqring(struct io_ring_ctx *ctx)return &ring->cqes[tail & ctx->cq_mask]; } -static struct io_kiocb *io_get_req (struct io_ring_ctx *ctx) +static void io_ring_drop_ctx_refs (struct io_ring_ctx *ctx,unsigned refs) { +percpu_ref_put_many (&ctx->refs, refs); + +if (waitqueue_active (&ctx->wait)) +wake_up (&ctx->wait); +} + +static struct io_kiocb *io_get_req (struct io_ring_ctx *ctx, +struct io_submit_state *state) +{ +gfp_t gfp =GFP_ATOMIC | __GFP_NOWARN;struct io_kiocb *req;if (!percpu_ref_tryget (&ctx->refs))return NULL ; - req =kmem_cache_alloc (req_cachep,GFP_ATOMIC | __GFP_NOWARN); -if (!req) -return NULL ; - - req->ki_ctx = ctx; -INIT_LIST_HEAD (&req->ki_list); - req->ki_flags =0 ; -return req; -} +if (!state) + req =kmem_cache_alloc (req_cachep, gfp); +else if (!state->free_reqs) { +size_t sz; +int ret; + + sz =min_t (size_t, state->ios_left,ARRAY_SIZE (state->reqs)); + ret =kmem_cache_alloc_bulk (req_cachep, gfp, sz, + state->reqs); +if (ret <=0 ) + goto out; + state->free_reqs = ret -1 ; + state->cur_req =1 ; + req = state->reqs[0 ]; + }else { + req = state->reqs[state->cur_req]; + state->free_reqs--; + state->cur_req++; + } -static void io_ring_drop_ctx_refs (struct io_ring_ctx *ctx,unsigned refs) -{ -percpu_ref_put_many (&ctx->refs, refs); +if (req) { + req->ki_ctx = ctx; + req->ki_flags =0 ; +return req; + } -if (waitqueue_active (&ctx->wait)) -wake_up (&ctx->wait); +out: +io_ring_drop_ctx_refs (ctx,1 ); +return NULL ; }static void io_free_req_many (struct io_ring_ctx *ctx,void **reqs,int *nr)
@@ -915,7 +945,7 @@ static int io_submit_sqe(struct io_ring_ctx *ctx, struct sqe_submit *s,struct io_kiocb *req;ssize_t ret; - req =io_get_req (ctx); + req =io_get_req (ctx, state);if (unlikely (!req))return -EAGAIN ;
@@ -952,6 +982,9 @@ static void io_submit_state_end(struct io_submit_state *state)if (!list_empty (&state->req_list.list))io_flush_state_reqs (state->ctx, state);io_file_put (state,NULL ); +if (state->free_reqs) +kmem_cache_free_bulk (req_cachep, state->free_reqs, + &state->reqs[state->cur_req]); }/*
@@ -963,6 +996,7 @@ static void io_submit_state_start(struct io_submit_state *state, state->ctx = ctx;INIT_LIST_HEAD (&state->req_list.list); state->req_count =0 ; + state->free_reqs =0 ; state->file =NULL ; state->ios_left = max_ios;#ifdef CONFIG_BLOCK
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 12/16 block: implement bio helper to add iter bvec pages to bio
In reply to Jens AxboeFor an ITER_BVEC, we can just iterate the iov and add the pages to the bio directly. This requires that the caller doesn't releases the pages on IO completion, we add a BIO_HOLD_PAGES flag for that.
The current two callers of bio_iov_iter_get_pages() are updated to check if they need to release pages on completion. This makes them work with bvecs that contain kernel mapped pages already.
--- block/bio.c | 59 ++++++++++++++++++++++++++++++++------- fs/block_dev.c | 5 ++-- fs/iomap.c | 5 ++-- include/linux/blk_types.h | 1 + 4 files changed, 56 insertions(+), 14 deletions(-)
block/bio.c +49 -10
@@ -828,6 +828,23 @@ int bio_add_page(struct bio *bio, struct page *page, }EXPORT_SYMBOL (bio_add_page); +static int __bio_iov_bvec_add_pages (struct bio *bio,struct iov_iter *iter) +{ +const struct bio_vec *bv = iter->bvec; +unsigned int len; +size_t size; + + len =min_t (size_t, bv->bv_len, iter->count); + size =bio_add_page (bio, bv->bv_page, len, + bv->bv_offset + iter->iov_offset); +if (size == len) { +iov_iter_advance (iter, size); +return 0 ; + } + +return -EINVAL ; +} +#define PAGE_PTRS_PER_BVEC (sizeof(struct bio_vec) / sizeof(struct page *))/**
@@ -876,23 +893,43 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) }/** -* bio_iov_iter_get_pages - pin user or kernel pages and add them to a bio +* bio_iov_iter_get_pages - add user or kernel pages to a bio * @bio: bio to add pages to -* @iter: iov iterator describing the region to be mapped +* @iter: iov iterator describing the region to be added +* +* This takes either an iterator pointing to user memory, or one pointing to +* kernel pages (BVEC iterator). If we're adding user pages, we pin them and +* map them into the kernel. On IO completion, the caller should put those +* pages. If we're adding kernel pages, we just have to add the pages to the +* bio directly. We don't grab an extra reference to those pages (the user +* should already have that), and we don't put the page on IO completion. +* The caller needs to check if the bio is flagged BIO_HOLD_PAGES on IO +* completion. If it isn't, then pages should be released. * -* Pins pages from *iter and appends them to @bio's bvec array. The -* pages will have to be released using put_page() when done. * The function tries, but does not guarantee, to pin as many pages as -* fit into the bio, or are requested in *iter, whatever is smaller. -* If MM encounters an error pinning the requested pages, it stops. -* Error is returned only if 0 pages could be pinned. +* fit into the bio, or are requested in *iter, whatever is smaller. If +* MM encounters an error pinning the requested pages, it stops. Error +* is returned only if 0 pages could be pinned. */ int bio_iov_iter_get_pages (struct bio *bio,struct iov_iter *iter) { +const bool is_bvec =iov_iter_is_bvec (iter);unsigned short orig_vcnt = bio->bi_vcnt; +/* +* If this is a BVEC iter, then the pages are kernel pages. Don't +* release them on IO completion. +*/ +if (is_bvec) +bio_set_flag (bio,BIO_HOLD_PAGES ); + do { - int ret = __bio_iov_iter_get_pages(bio, iter); + int ret; + + if (is_bvec) + ret = __bio_iov_bvec_add_pages(bio, iter); + else + ret = __bio_iov_iter_get_pages(bio, iter); if (unlikely(ret)) return bio->bi_vcnt > orig_vcnt ? 0 : ret;
@@ -1634,7 +1671,8 @@ static void bio_dirty_fn(struct work_struct *work) next = bio->bi_private;bio_set_pages_dirty (bio); -bio_release_pages (bio); +if (!bio_flagged (bio,BIO_HOLD_PAGES )) +bio_release_pages (bio);bio_put (bio); } }
@@ -1650,7 +1688,8 @@ void bio_check_pages_dirty(struct bio *bio) goto defer; } -bio_release_pages (bio); +if (!bio_flagged (bio,BIO_HOLD_PAGES )) +bio_release_pages (bio);bio_put (bio);return ; defer:
fs/block_dev.c +3 -2
@@ -324,8 +324,9 @@ static void blkdev_bio_end_io(struct bio *bio)struct bio_vec *bvec;int i; -bio_for_each_segment_all (bvec, bio, i) -put_page (bvec->bv_page); +if (!bio_flagged (bio,BIO_HOLD_PAGES )) +bio_for_each_segment_all (bvec, bio, i) +put_page (bvec->bv_page);bio_put (bio); } }
fs/iomap.c +3 -2
@@ -1582,8 +1582,9 @@ static void iomap_dio_bio_end_io(struct bio *bio)struct bio_vec *bvec;int i; -bio_for_each_segment_all (bvec, bio, i) -put_page (bvec->bv_page); +if (!bio_flagged (bio,BIO_HOLD_PAGES )) +bio_for_each_segment_all (bvec, bio, i) +put_page (bvec->bv_page);bio_put (bio); } }
include/linux/blk_types.h +1 -0
@@ -215,6 +215,7 @@ struct bio {/* * bio flags */ +#define BIO_HOLD_PAGES 0/* don't put O_DIRECT pages */ #define BIO_SEG_VALID 1/* bi_phys_segments valid */ #define BIO_CLONED 2/* doesn't own data */ #define BIO_BOUNCED 3/* bio is a bounce bio */
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 13/16 io_uring: add support for pre-mapped user IO buffers
In reply to Jens AxboeIf we have fixed user buffers, we can map them into the kernel when we setup the io_context. That avoids the need to do get_user_pages() for each and every IO.
To utilize this feature, the application must call io_uring_register() after having setup an io_uring context, passing in IORING_REGISTER_BUFFERS as the opcode, and the following struct as the argument:
struct io_uring_register_buffers {
struct iovec *iovecs;
__u32 nr_iovecs;
};If successful, these buffers are now mapped into the kernel, eligible for IO. To use these fixed buffers, the application must use the IORING_OP_READ_FIXED and IORING_OP_WRITE_FIXED opcodes, and then set sqe->index to the desired buffer index. sqe->addr..sqe->addr+seq->len must point to somewhere inside the indexed buffer.
The application may register buffers throughout the lifetime of the io_uring context. It can call io_uring_register() with IORING_UNREGISTER_BUFFERS as the opcode to unregister the current set of buffers, and then register a new set. The application need not unregister buffers explicitly before shutting down the io_uring context.
It's perfectly valid to setup a larger buffer, and then sometimes only use parts of it for an IO. As long as the range is within the originally mapped region, it will work just fine.
RLIMIT_MEMLOCK is used to check how much memory we can pin. A somewhat arbitrary 1G per buffer size is also imposed.
--- arch/x86/entry/syscalls/syscall_64.tbl | 1 + fs/io_uring.c | 318 +++++++++++++++++++++++-- include/linux/syscalls.h | 2 + include/uapi/linux/io_uring.h | 17 +- kernel/sys_ni.c | 1 + 5 files changed, 323 insertions(+), 16 deletions(-)
arch/x86/entry/syscalls/syscall_64.tbl +1 -0
@@ -345,6 +345,7 @@ 334 common rseq __x64_sys_rseq 335 common io_uring_setup __x64_sys_io_uring_setup 336 common io_uring_enter __x64_sys_io_uring_enter +337 common io_uring_register __x64_sys_io_uring_register # # x32-specific system call numbers start at 512 to avoid cache impact
fs/io_uring.c +303 -15
417 lines not on this page · raw message
include/linux/syscalls.h +2 -0
@@ -314,6 +314,8 @@ asmlinkage long sys_io_uring_setup(u32 entries, struct io_uring_params __user *p);asmlinkage long sys_io_uring_enter (unsigned int fd,u32 to_submit,u32 min_complete,u32 flags); +asmlinkage long sys_io_uring_register (unsigned int fd,unsigned op, +void __user *arg);/* fs/xattr.c */ asmlinkage long sys_setxattr(const char __user *path, const char __user *name,
include/uapi/linux/io_uring.h +16 -1
@@ -29,7 +29,9 @@ struct io_uring_sqe {__kernel_rwf_t rw_flags;__u32 fsync_flags; }; -__u64 __pad2; +__u16 buf_index;/* index into fixed buffers, if used */ +__u16 __pad2; +__u32 __pad3;__u64 user_data;/* data to be passed back at completion time */ };
@@ -41,6 +43,8 @@ struct io_uring_sqe {#define IORING_OP_READV 1#define IORING_OP_WRITEV 2#define IORING_OP_FSYNC 3 +#define IORING_OP_READ_FIXED 4 +#define IORING_OP_WRITE_FIXED 5/* * sqe->fsync_flags
@@ -104,4 +108,15 @@ struct io_uring_params { struct io_cqring_offsets cq_off; }; +/* +* io_uring_register(2) opcodes and arguments +*/ +#define IORING_REGISTER_BUFFERS 0 +#define IORING_UNREGISTER_BUFFERS 1 + +struct io_uring_register_buffers { +struct iovec *iovecs; +__u32 nr_iovecs; +}; +#endif
kernel/sys_ni.c +1 -0
@@ -48,6 +48,7 @@ COND_SYSCALL_COMPAT(io_getevents);COND_SYSCALL_COMPAT (io_pgetevents);COND_SYSCALL (io_uring_setup);COND_SYSCALL (io_uring_enter); +COND_SYSCALL (io_uring_register);/* fs/xattr.c */
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 14/16 io_uring: add submission polling
In reply to Jens AxboeThis enables an application to do IO, without ever entering the kernel. By using the SQ ring to fill in new sqes and watching for completions on the CQ ring, we can submit and reap IOs without doing a single system call. The kernel side thread will poll for new submissions, and in case of HIPRI/polled IO, it'll also poll for completions.
Proof of concept. If the thread has been idle for 1 second, it will set sq_ring->flags |= IORING_SQ_NEED_WAKEUP. The application will have to call io_uring_enter() to start things back up again. If IO is kept busy, that will never be needed. Basically an application that has this feature enabled will guard it's io_uring_enter(2) call with:
read_barrier();
if (*sq_ring->flags & IORING_SQ_NEED_WAKEUP)
io_uring_enter(fd, to_submit, 0, 0);
instead of calling it unconditionally.
Improvements:
1) Maybe have smarter backoff. Busy loop for X time, then go to monitor/mwait, finally the schedule we have now after an idle second. Might not be worth the complexity.
2) Probably want the application to pass in the appropriate grace period, not hard code it at 1 second.
--- fs/io_uring.c | 215 +++++++++++++++++++++++++++++++++- include/uapi/linux/io_uring.h | 10 +- 2 files changed, 218 insertions(+), 7 deletions(-)
fs/io_uring.c +209 -6
@@ -22,6 +22,7 @@#include <linux/percpu.h> #include <linux/slab.h> #include <linux/workqueue.h> +#include <linux/kthread.h> #include <linux/blkdev.h> #include <linux/bvec.h> #include <linux/anon_inodes.h>
@@ -89,8 +90,10 @@ struct io_ring_ctx {/* IO offload */ struct workqueue_struct *sqo_wq; +struct task_struct *sqo_thread;/* if using sq thread polling */ struct mm_struct *sqo_mm;struct files_struct *sqo_files; +wait_queue_head_t sqo_wait;/* if used, fixed mapped user buffers */ struct io_mapped_ubuf *user_bufs;
@@ -1131,6 +1134,167 @@ static bool io_peek_sqring(struct io_ring_ctx *ctx, struct sqe_submit *s)return false; } +static int io_submit_sqes (struct io_ring_ctx *ctx,struct sqe_submit *sqes, +unsigned int nr,bool mm_fault) +{ +struct io_submit_state state, *statep =NULL ; +int ret, i, submitted =0 ; + +if (nr >IO_PLUG_THRESHOLD ) { +io_submit_state_start (&state, ctx, nr); + statep = &state; + } + +for (i =0 ; i < nr; i++) { +if (unlikely (mm_fault)) + ret = -EFAULT ; +else + ret =io_submit_sqe (ctx, &sqes[i], statep); +if (!ret) { + submitted++; +continue ; + } + +io_fill_cq_error (ctx, &sqes[i], ret); + } + +if (statep) +io_submit_state_end (&state); + +return submitted; +} + +static int io_sq_thread (void *data) +{ +struct sqe_submit sqes[IO_IOPOLL_BATCH ]; +struct io_ring_ctx *ctx = data; +struct mm_struct *cur_mm =NULL ; +struct files_struct *old_files; +mm_segment_t old_fs; +DEFINE_WAIT (wait); +unsigned inflight; +unsigned long timeout; + + old_files = current->files; + current->files = ctx->sqo_files; + + old_fs =get_fs (); +set_fs (USER_DS ); + + timeout = inflight =0 ; +while (!kthread_should_stop ()) { +bool all_fixed, mm_fault = false; +int i; + +if (inflight) { +unsigned int nr_events =0 ; + +/* +* Normal IO, just pretend everything completed. +* We don't have to poll completions for that. +*/ +if (ctx->flags &IORING_SETUP_IOPOLL ) { +/* +* App should not use IORING_ENTER_GETEVENTS +* with thread polling, but if it does, then +* ensure we are mutually exclusive. +*/ +if (mutex_trylock (&ctx->uring_lock)) { +io_iopoll_check (ctx, &nr_events,0 ); +mutex_unlock (&ctx->uring_lock); + } + }else { + nr_events = inflight; + } + + inflight -= nr_events; +if (!inflight) + timeout = jiffies +HZ ; + } + +if (!io_peek_sqring (ctx, &sqes[0 ])) { +/* +* We're polling, let us spin for a second without +* work before going to sleep. +*/ +if (inflight || !time_after (jiffies, timeout)) { +cpu_relax (); +continue ; + } + +/* +* Drop cur_mm before scheduling, we can't hold it for +* long periods (or over schedule()). Do this before +* adding ourselves to the waitqueue, as the unuse/drop +* may sleep. +*/ +if (cur_mm) { +unuse_mm (cur_mm); +mmput (cur_mm); + cur_mm =NULL ; + } + +prepare_to_wait (&ctx->sqo_wait, &wait, +TASK_INTERRUPTIBLE ); + +/* Tell userspace we may need a wakeup call */ + ctx->sq_ring->flags |=IORING_SQ_NEED_WAKEUP ; +smp_wmb (); + +if (!io_peek_sqring (ctx, &sqes[0 ])) { +if (kthread_should_park ()) +kthread_parkme (); +if (kthread_should_stop ()) { +finish_wait (&ctx->sqo_wait, &wait); +break ; + } +if (signal_pending (current)) +flush_signals (current); +schedule (); +finish_wait (&ctx->sqo_wait, &wait); + + ctx->sq_ring->flags &= ~IORING_SQ_NEED_WAKEUP ; +smp_wmb (); +continue ; + } +finish_wait (&ctx->sqo_wait, &wait); + + ctx->sq_ring->flags &= ~IORING_SQ_NEED_WAKEUP ; +smp_wmb (); + } + + i =0 ; + all_fixed = true; +do { +if (sqes[i].sqe->opcode !=IORING_OP_READ_FIXED && + sqes[i].sqe->opcode !=IORING_OP_WRITE_FIXED ) + all_fixed = false; +if (i +1 ==ARRAY_SIZE (sqes)) +break ; + i++; +io_inc_sqring (ctx); + }while (io_peek_sqring (ctx, &sqes[i])); + +/* Unless all new commands are FIXED regions, grab mm */ +if (!all_fixed && !cur_mm) { + mm_fault = !mmget_not_zero (ctx->sqo_mm); +if (!mm_fault) { +use_mm (ctx->sqo_mm); + cur_mm = ctx->sqo_mm; + } + } + + inflight +=io_submit_sqes (ctx, sqes, i, mm_fault); + } + current->files = old_files; +set_fs (old_fs); +if (cur_mm) { +unuse_mm (cur_mm); +mmput (cur_mm); + } +return 0 ; +} +static int io_ring_submit (struct io_ring_ctx *ctx,unsigned int to_submit) {struct io_submit_state state, *statep =NULL ;
@@ -1202,9 +1366,14 @@ static int __io_uring_enter(struct io_ring_ctx *ctx, unsigned to_submit,int ret =0 ;if (to_submit) { - ret =io_ring_submit (ctx, to_submit); -if (ret <0 ) -return ret; +if (ctx->flags &IORING_SETUP_SQPOLL ) { +wake_up (&ctx->sqo_wait); + ret = to_submit; + }else { + ret =io_ring_submit (ctx, to_submit); +if (ret <0 ) +return ret; + } }if (flags &IORING_ENTER_GETEVENTS ) {unsigned nr_events =0 ;
@@ -1225,10 +1394,12 @@ static int __io_uring_enter(struct io_ring_ctx *ctx, unsigned to_submit,return ret; } -static int io_sq_offload_start (struct io_ring_ctx *ctx) +static int io_sq_offload_start (struct io_ring_ctx *ctx, +struct io_uring_params *p) {int ret; +init_waitqueue_head (&ctx->sqo_wait); ctx->sqo_mm = current->mm;/*
@@ -1242,6 +1413,27 @@ static int io_sq_offload_start(struct io_ring_ctx *ctx)if (!ctx->sqo_files) goto err; +if (ctx->flags &IORING_SETUP_SQPOLL ) { +if (p->flags &IORING_SETUP_SQ_AFF ) { + ctx->sqo_thread =kthread_create_on_cpu (io_sq_thread, + ctx, p->sq_thread_cpu, +"io_uring-sq" ); + }else { + ctx->sqo_thread =kthread_create (io_sq_thread, ctx, +"io_uring-sq" ); + } +if (IS_ERR (ctx->sqo_thread)) { + ret =PTR_ERR (ctx->sqo_thread); + ctx->sqo_thread =NULL ; + goto err; + } +wake_up_process (ctx->sqo_thread); + }else if (p->flags &IORING_SETUP_SQ_AFF ) { +/* Can't have SQ_AFF without SQPOLL */ + ret = -EINVAL ; + goto err; + } +/* Do QD, or 2 * CPUS, whatever is smallest */ ctx->sqo_wq =alloc_workqueue ("io_ring-wq" ,WQ_UNBOUND |WQ_FREEZABLE ,min (ctx->sq_entries -1 ,2 *num_online_cpus ()));
@@ -1252,6 +1444,11 @@ static int io_sq_offload_start(struct io_ring_ctx *ctx)return 0 ; err: +if (ctx->sqo_thread) { +kthread_park (ctx->sqo_thread); +kthread_stop (ctx->sqo_thread); + ctx->sqo_thread =NULL ; + }if (ctx->sqo_files) ctx->sqo_files =NULL ; ctx->sqo_mm =NULL ;
@@ -1260,6 +1457,11 @@ static int io_sq_offload_start(struct io_ring_ctx *ctx)static void io_sq_offload_stop (struct io_ring_ctx *ctx) { +if (ctx->sqo_thread) { +kthread_park (ctx->sqo_thread); +kthread_stop (ctx->sqo_thread); + ctx->sqo_thread =NULL ; + }if (ctx->sqo_wq) {destroy_workqueue (ctx->sqo_wq); ctx->sqo_wq =NULL ;
@@ -1594,7 +1796,7 @@ static int io_uring_create(unsigned entries, struct io_uring_params *p)if (ret) goto err; - ret =io_sq_offload_start (ctx); + ret =io_sq_offload_start (ctx, p);if (ret) goto err;
@@ -1629,7 +1831,8 @@ SYSCALL_DEFINE2(io_uring_setup, u32, entries,return -EINVAL ; } -if (p.flags & ~IORING_SETUP_IOPOLL ) +if (p.flags & ~(IORING_SETUP_IOPOLL |IORING_SETUP_SQPOLL | +IORING_SETUP_SQ_AFF ))return -EINVAL ; ret =io_uring_create (entries, &p);
include/uapi/linux/io_uring.h +9 -1
@@ -39,6 +39,8 @@ struct io_uring_sqe {* io_uring_setup() flags */ #define IORING_SETUP_IOPOLL (1 << 0)/* io_context is polled */ +#define IORING_SETUP_SQPOLL (1 << 1)/* SQ poll thread */ +#define IORING_SETUP_SQ_AFF (1 << 2)/* sq_thread_cpu is valid */ #define IORING_OP_READV 1#define IORING_OP_WRITEV 2
@@ -81,6 +83,11 @@ struct io_sqring_offsets {__u32 resv[3 ]; }; +/* +* sq_ring->flags +*/ +#define IORING_SQ_NEED_WAKEUP (1 << 0)/* needs io_uring_enter wakeup */ +struct io_cqring_offsets {__u32 head;__u32 tail;
@@ -103,7 +110,8 @@ struct io_uring_params {__u32 sq_entries;__u32 cq_entries;__u32 flags; -__u16 resv[10 ]; +__u16 sq_thread_cpu; +__u16 resv[9 ];struct io_sqring_offsets sq_off;struct io_cqring_offsets cq_off; };
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 15/16 io_uring: add file registration
In reply to Jens AxboeWe normally have to fget/fput for each IO we do on a file. Even with the batching we do, this atomic inc/dec cost adds up.
This adds IORING_REGISTER_FILES, and IORING_UNREGISTER_FILES opcodes for the io_uring_register(2) system call. Pass in an array of fds that are in use by the application, and we'll fget these for the duration of the io_uring context.
When used, the application must set IOSQE_FIXED_FILE in the sqe->flags member. Then, instead of setting sqe->fd to the real fd, it sets sqe->fd to the index in the array passed in to IORING_REGISTER_FILES.
Files are automatically unregistered when the io_uring context is torn down. An application need only unregister if it wishes to register a few set of fds.
--- fs/io_uring.c | 131 +++++++++++++++++++++++++++++----- include/uapi/linux/io_uring.h | 14 +++- 2 files changed, 127 insertions(+), 18 deletions(-)
fs/io_uring.c +114 -17
@@ -99,6 +99,10 @@ struct io_ring_ctx {struct io_mapped_ubuf *user_bufs;unsigned nr_user_bufs; +/* if used, fixed file set */ +struct file **user_files; +unsigned nr_user_files; +struct completion ctx_done;/* iopoll submission state */
@@ -137,6 +141,7 @@ struct io_kiocb {#define REQ_F_FORCE_NONBLOCK 1/* inline submission attempt */ #define REQ_F_IOPOLL_COMPLETED 2/* polled IO has completed */ #define REQ_F_IOPOLL_EAGAIN 4/* submission got EAGAIN */ +#define REQ_F_FIXED_FILE 8/* ctx owns file */ u64 ki_user_data; };
@@ -355,15 +360,17 @@ static void io_iopoll_reap(struct io_ring_ctx *ctx, unsigned int *nr_events)* Batched puts of the same file, to avoid dirtying the * file usage count multiple times, if avoidable. */ -if (!file) { - file = req->rw.ki_filp; - file_count =1 ; - }else if (file == req->rw.ki_filp) { - file_count++; - }else { -fput_many (file, file_count); - file = req->rw.ki_filp; - file_count =1 ; +if (!(req->ki_flags &REQ_F_FIXED_FILE )) { +if (!file) { + file = req->rw.ki_filp; + file_count =1 ; + }else if (file == req->rw.ki_filp) { + file_count++; + }else { +fput_many (file, file_count); + file = req->rw.ki_filp; + file_count =1 ; + } } (*nr_events)++;
@@ -557,13 +564,19 @@ static void io_fill_cq_error(struct io_ring_ctx *ctx, struct sqe_submit *s,wake_up (&ctx->wait); } +static void io_fput (struct io_kiocb *req) +{ +if (!(req->ki_flags &REQ_F_FIXED_FILE )) +fput (req->rw.ki_filp); +} +static void io_complete_scqring_rw (struct kiocb *kiocb,long res,long res2) { struct io_kiocb *req = container_of(kiocb, struct io_kiocb, rw);kiocb_end_write (kiocb); -fput (kiocb->ki_filp); +io_fput (req);io_cqring_fill_event (req->ki_ctx, req->ki_user_data, res,0 );io_free_req (req); }
@@ -680,7 +693,17 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe,struct kiocb *kiocb = &req->rw;int ret; - kiocb->ki_filp =io_file_get (state, sqe->fd); +if (unlikely (sqe->flags & ~IOSQE_FIXED_FILE )) +return -EINVAL ; + +if (sqe->flags &IOSQE_FIXED_FILE ) { +if (unlikely (!ctx->user_files || sqe->fd >= ctx->nr_user_files)) +return -EBADF ; + kiocb->ki_filp = ctx->user_files[sqe->fd]; + req->ki_flags |=REQ_F_FIXED_FILE ; + }else { + kiocb->ki_filp =io_file_get (state, sqe->fd); + }if (unlikely (!kiocb->ki_filp))return -EBADF ; kiocb->ki_pos = sqe->off;
@@ -719,7 +742,8 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe, }return 0 ; out_fput: -io_file_put (state, kiocb->ki_filp); +if (!(sqe->flags &IOSQE_FIXED_FILE )) +io_file_put (state, kiocb->ki_filp);return ret; }
@@ -822,7 +846,7 @@ static ssize_t io_read(struct io_kiocb *req, const struct io_uring_sqe *sqe,kfree (iovec); out_fput:if (unlikely (ret)) -fput (file); +io_fput (req);return ret; }
@@ -884,7 +908,7 @@ static ssize_t io_write(struct io_kiocb *req, const struct io_uring_sqe *sqe, } out_fput:if (unlikely (ret)) -fput (file); +io_fput (req);return ret; }
@@ -902,19 +926,30 @@ static int io_fsync(struct io_kiocb *req, const struct io_uring_sqe *sqe,if (unlikely (req->ki_ctx->flags &IORING_SETUP_IOPOLL ))return -EINVAL ; +if (unlikely (sqe->flags & ~IOSQE_FIXED_FILE )) +return -EINVAL ;if (unlikely (sqe->addr))return -EINVAL ;if (unlikely (sqe->fsync_flags & ~IORING_FSYNC_DATASYNC ))return -EINVAL ; - file =fget (sqe->fd); +if (sqe->flags &IOSQE_FIXED_FILE ) { +if (unlikely (!ctx->user_files || sqe->fd >= ctx->nr_user_files)) +return -EBADF ; + file = ctx->user_files[sqe->fd]; + }else { + file =fget (sqe->fd); + } +if (unlikely (!file))return -EBADF ; ret =vfs_fsync_range (file, sqe->off, end >0 ? end :LLONG_MAX , sqe->fsync_flags &IORING_FSYNC_DATASYNC ); -fput (file); +if (!(sqe->flags &IOSQE_FIXED_FILE )) +fput (file); +io_cqring_fill_event (ctx, sqe->user_data, ret,0 );io_free_req (req);return 0 ;
@@ -928,7 +963,7 @@ static int __io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,ssize_t ret;/* enforce forwards compatibility on users */ -if (unlikely (sqe->flags || sqe->__pad2 || sqe->__pad3)) +if (unlikely (sqe->__pad2 || sqe->__pad3))return -EINVAL ;if (unlikely (s->index >= ctx->sq_entries))
@@ -1394,6 +1429,52 @@ static int __io_uring_enter(struct io_ring_ctx *ctx, unsigned to_submit,return ret; } +static int io_sqe_files_unregister (struct io_ring_ctx *ctx) +{ +int i; + +if (!ctx->user_files) +return -EINVAL ; + +for (i =0 ; i < ctx->nr_user_files; i++) +fput (ctx->user_files[i]); + +kfree (ctx->user_files); + ctx->user_files =NULL ; + ctx->nr_user_files =0 ; +return 0 ; +} + +static int io_sqe_files_register (struct io_ring_ctx *ctx, +struct io_uring_register_files *reg) +{ +int fd, i, ret =0 ; + + ctx->user_files =kcalloc (reg->nr_fds,sizeof (struct file *), +GFP_KERNEL ); +if (!ctx->user_files) +return -ENOMEM ; + +for (i =0 ; i < reg->nr_fds; i++) { + ret = -EFAULT ; +if (copy_from_user (&fd, ®->fds[i],sizeof (fd))) +break ; + + ctx->user_files[i] =fget (fd); + + ret = -EBADF ; +if (!ctx->user_files[i]) +break ; + ctx->nr_user_files++; + ret =0 ; + } + +if (ret) +io_sqe_files_unregister (ctx); + +return ret; +} +static int io_sq_offload_start (struct io_ring_ctx *ctx,struct io_uring_params *p) {
@@ -1615,6 +1696,7 @@ static void io_ring_ctx_free(struct io_ring_ctx *ctx)io_sq_offload_stop (ctx);io_iopoll_reap_events (ctx);io_free_scq_urings (ctx); +io_sqe_files_unregister (ctx);io_sqe_buffer_unregister (ctx);percpu_ref_exit (&ctx->refs);kfree (ctx);
@@ -1871,6 +1953,21 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,break ; ret =io_sqe_buffer_unregister (ctx);break ; +case IORING_REGISTER_FILES : { +struct io_uring_register_files reg; + + ret = -EFAULT ; +if (copy_from_user (®, arg,sizeof (reg))) +break ; + ret =io_sqe_files_register (ctx, ®); +break ; + } +case IORING_UNREGISTER_FILES : + ret = -EINVAL ; +if (arg) +break ; + ret =io_sqe_files_unregister (ctx); +break ;default : ret = -EINVAL ;break ;
include/uapi/linux/io_uring.h +13 -1
@@ -16,7 +16,7 @@*/ struct io_uring_sqe {__u8 opcode;/* type of operation for this sqe */ -__u8 flags;/* as of now unused */ +__u8 flags;/* IOSQE_ flags */ __u16 ioprio;/* ioprio for the request */ __s32 fd;/* file descriptor to do IO on */ __u64 off;/* offset into file */
@@ -35,6 +35,11 @@ struct io_uring_sqe {__u64 user_data;/* data to be passed back at completion time */ }; +/* +* sqe->flags +*/ +#define IOSQE_FIXED_FILE (1 << 0)/* use fixed fileset */ +/* * io_uring_setup() flags */
@@ -121,10 +126,17 @@ struct io_uring_params {*/ #define IORING_REGISTER_BUFFERS 0#define IORING_UNREGISTER_BUFFERS 1 +#define IORING_REGISTER_FILES 2 +#define IORING_UNREGISTER_FILES 3struct io_uring_register_buffers {struct iovec *iovecs;__u32 nr_iovecs; }; +struct io_uring_register_files { +__s32 *fds; +__u32 nr_fds; +}; +#endif
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 16/16 io_uring: add io_uring_event cache hit information
In reply to Jens AxboeAdd hint on whether a read was served out of the page cache, or if it hit media. This is useful for buffered async IO, O_DIRECT reads would never have this set (for obvious reasons).
If the read hit page cache, cqe->flags will have IOCQE_FLAG_CACHEHIT set.
--- fs/io_uring.c | 7 ++++++- include/uapi/linux/io_uring.h | 5 +++++ 2 files changed, 11 insertions(+), 1 deletion(-)
fs/io_uring.c +6 -1
@@ -573,11 +573,16 @@ static void io_fput(struct io_kiocb *req)static void io_complete_scqring_rw (struct kiocb *kiocb,long res,long res2) { struct io_kiocb *req = container_of(kiocb, struct io_kiocb, rw); +unsigned ev_flags =0 ;kiocb_end_write (kiocb);io_fput (req); -io_cqring_fill_event (req->ki_ctx, req->ki_user_data, res,0 ); + +if (res >0 && (req->ki_flags &REQ_F_FORCE_NONBLOCK )) + ev_flags =IOCQE_FLAG_CACHEHIT ; + +io_cqring_fill_event (req->ki_ctx, req->ki_user_data, res, ev_flags);io_free_req (req); }
include/uapi/linux/io_uring.h +5 -0
@@ -67,6 +67,11 @@ struct io_uring_cqe {__u32 flags; }; +/* +* io_uring_event->flags +*/ +#define IOCQE_FLAG_CACHEHIT (1 << 0)/* IO did not hit media */ +/* * Magic offsets for the application to mmap the data it needs */
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>