io_uring IO interface
Subsystems (14)
Jens Axboe
maintainerPATCH v13 1/1 io_uring IO interface
Here's v13 of the io_uring project, hot on the heels of v12. v12 had a few silly regressions due to flipping things around for the SCM fd passing changes. v13 also decouples the SCM fd management from how many fixed files we support, so instead of a fairly random 253 file limit, we now impose a 1024 file limit for a file set.
The AF_UNIX scm parts have been split into its own file. This was needed to prevent issues with CONFIG_UNIX=m, since io_uring is always builtin.
Nothing major in here apart from that. Go forth and test and review, so we can hopefully get this queued up sooner rather than later.
The liburing git repo has a full set of man pages for this, though they could probably still use a bit of polish. I'd also like to see a io_uring(7) man page to describe the overall design of the project, expect that in the not-so-distant future. You can clone that here:
git://git.kernel.dk/liburing
Patches are against 5.0-rc5, and can also be found in my io_uring branch here:
git://git.kernel.dk/linux-block io_uring
Changes since v12:
- Fix release of uid struct at buffer unregister time
- Fix leak of request for poll command on submission time errors
- Allow huge pages for pre-mapped buffers
- Use alloc_skb() instead of __alloc_skb()
- Make max fixed files independent of SCM_MAX_FD (now 1024)
- Use CONFIG_UNIX instead of CONFIG_NET
- Export io_uring_get_socket for CONFIG_UNIX=m
- Add net/unix/scm.c with the helpers io_uring needs
- Disallow registering the io_uring fd to prevent a deadlock with !CONFIG_UNIX
Documentation/filesystems/vfs.txt | 3 + arch/x86/entry/syscalls/syscall_32.tbl | 3 + arch/x86/entry/syscalls/syscall_64.tbl | 3 + block/bio.c | 59 +- fs/Makefile | 1 + fs/block_dev.c | 19 +- fs/file.c | 15 +- fs/file_table.c | 9 +- fs/gfs2/file.c | 2 + fs/io_uring.c | 2796 ++++++++++++++++++++++++ fs/iomap.c | 48 +- fs/xfs/xfs_file.c | 1 + include/linux/bio.h | 14 + include/linux/blk_types.h | 1 + include/linux/file.h | 2 + include/linux/fs.h | 15 +- include/linux/iomap.h | 1 + include/linux/sched/user.h | 2 +- include/linux/syscalls.h | 8 + include/net/af_unix.h | 1 + include/uapi/asm-generic/unistd.h | 8 +- include/uapi/linux/io_uring.h | 142 ++ init/Kconfig | 9 + kernel/sys_ni.c | 3 + net/unix/Kconfig | 5 + net/unix/Makefile | 2 + net/unix/af_unix.c | 63 +- net/unix/garbage.c | 68 +- net/unix/scm.c | 146 ++ net/unix/scm.h | 10 + 30 files changed, 3291 insertions(+), 168 deletions(-)
Signature · 8 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 1/19 fs: add an iopoll method to struct file_operations
From: Christoph Hellwig <[email protected]>
This new methods is used to explicitly poll for I/O completion for an iocb. It must be called for any iocb submitted asynchronously (that is with a non-null ki_complete) which has the IOCB_HIPRI flag set.
The method is assisted by a new ki_cookie field in struct iocb to store the polling cookie.
--- Documentation/filesystems/vfs.txt | 3 +++ include/linux/fs.h | 2 ++ 2 files changed, 5 insertions(+)
Documentation/filesystems/vfs.txt +3 -0
@@ -857,6 +857,7 @@ struct file_operations { ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *); ssize_t (*read_iter) (struct kiocb *, struct iov_iter *); ssize_t (*write_iter) (struct kiocb *, struct iov_iter *); + int (*iopoll)(struct kiocb *kiocb, bool spin); int (*iterate) (struct file *, struct dir_context *); int (*iterate_shared) (struct file *, struct dir_context *); __poll_t (*poll) (struct file *, struct poll_table_struct *);
@@ -902,6 +903,8 @@ otherwise noted. write_iter: possibly asynchronous write with iov_iter as source + iopoll: called when aio wants to poll for completions on HIPRI iocbs + iterate: called when the VFS needs to read the directory contents iterate_shared: called when the VFS needs to read the directory contents
include/linux/fs.h +2 -0
@@ -310,6 +310,7 @@ struct kiocb {int ki_flags;u16 ki_hint;u16 ki_ioprio;/* See linux/ioprio.h */ +unsigned int ki_cookie;/* for ->iopoll */ } __randomize_layout;static inline bool is_sync_kiocb (struct kiocb *kiocb)
@@ -1787,6 +1788,7 @@ struct file_operations {ssize_t (*write) (struct file *,const char __user *,size_t ,loff_t *);ssize_t (*read_iter) (struct kiocb *,struct iov_iter *);ssize_t (*write_iter) (struct kiocb *,struct iov_iter *); +int (*iopoll)(struct kiocb *kiocb,bool spin);int (*iterate) (struct file *,struct dir_context *);int (*iterate_shared) (struct file *,struct dir_context *);__poll_t (*poll) (struct file *,struct poll_table_struct *);
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 01/19] fs: add an iopoll method to struct file_operations
Quoted 9 lines · Jens Axboe
From: Christoph Hellwig <[email protected]>
This new methods is used to explicitly poll for I/O completion for an iocb. It must be called for any iocb submitted asynchronously (that is with a non-null
ki_complete) which has theIOCB_HIPRIflag set.The method is assisted by a new
ki_cookiefield in struct iocb to store the polling cookie.
Quoted 8 lines
Reviewed-by Johannes Thumshirn <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- Documentation/filesystems/vfs.txt | 3 +++ include/linux/fs.h | 2 ++ 2 files changed, 5 insertions(+)
Cheers,
Hannes
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 2/19 block: wire up block device iopoll method
In reply to Jens AxboeFrom: Christoph Hellwig <[email protected]>
Just call blk_poll on the iocb cookie, we can derive the block device from the inode trivially.
--- fs/block_dev.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
fs/block_dev.c +10 -0
@@ -293,6 +293,14 @@ struct blkdev_dio {static struct bio_set blkdev_dio_pool; +static int blkdev_iopoll (struct kiocb *kiocb,bool wait) +{ +struct block_device *bdev =I_BDEV (kiocb->ki_filp->f_mapping->host); +struct request_queue *q =bdev_get_queue (bdev); + +return blk_poll (q,READ_ONCE (kiocb->ki_cookie), wait); +} +static void blkdev_bio_end_io (struct bio *bio) {struct blkdev_dio *dio = bio->bi_private;
@@ -410,6 +418,7 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) bio->bi_opf |=REQ_HIPRI ; qc =submit_bio (bio); +WRITE_ONCE (iocb->ki_cookie, qc);break ; }
@@ -2076,6 +2085,7 @@ const struct file_operations def_blk_fops = { .llseek = block_llseek, .read_iter = blkdev_read_iter, .write_iter = blkdev_write_iter, + .iopoll = blkdev_iopoll, .mmap = generic_file_mmap, .fsync = blkdev_fsync, .unlocked_ioctl = block_ioctl,
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 02/19] block: wire up block device iopoll method
Quoted 5 lines · Jens Axboe
From: Christoph Hellwig <[email protected]>
Just call
blk_pollon the iocb cookie, we can derive the block device from the inode trivially.
Quoted 7 lines
Reviewed-by Johannes Thumshirn <[email protected]>Signed-off-by Christoph Hellwig <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- fs/block_dev.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
Cheers,
Hannes
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 3/19 block: add bio_set_polled() helper
In reply to Jens AxboeFor the upcoming async polled IO, we can't sleep allocating requests. If we do, then we introduce a deadlock where the submitter already has async polled IO in-flight, but can't wait for them to complete since polled requests must be active found and reaped.
Utilize the helper in the blockdev DIRECT_IO code.
--- fs/block_dev.c | 4 ++-- include/linux/bio.h | 14 ++++++++++++++ 2 files changed, 16 insertions(+), 2 deletions(-)
fs/block_dev.c +2 -2
@@ -247,7 +247,7 @@ __blkdev_direct_IO_simple(struct kiocb *iocb, struct iov_iter *iter,task_io_account_write (ret); }if (iocb->ki_flags &IOCB_HIPRI ) - bio.bi_opf |=REQ_HIPRI ; +bio_set_polled (&bio, iocb); qc =submit_bio (&bio);for (;;) {
@@ -415,7 +415,7 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) nr_pages =iov_iter_npages (iter,BIO_MAX_PAGES );if (!nr_pages) {if (iocb->ki_flags &IOCB_HIPRI ) - bio->bi_opf |=REQ_HIPRI ; +bio_set_polled (bio, iocb); qc =submit_bio (bio);WRITE_ONCE (iocb->ki_cookie, qc);
include/linux/bio.h +14 -0
@@ -823,5 +823,19 @@ static inline int bio_integrity_add_page(struct bio *bio, struct page *page,#endif /* CONFIG_BLK_DEV_INTEGRITY */ +/* +* Mark a bio as polled. Note that for async polled IO, the caller must +* expect -EWOULDBLOCK if we cannot allocate a request (or other resources). +* We cannot block waiting for requests on polled IO, as those completions +* must be found by the caller. This is different than IRQ driven IO, where +* it's safe to wait for IO to complete. +*/ +static inline void bio_set_polled (struct bio *bio,struct kiocb *kiocb) +{ + bio->bi_opf |=REQ_HIPRI ; +if (!is_sync_kiocb (kiocb)) + bio->bi_opf |=REQ_NOWAIT ; +} +#endif /* CONFIG_BLOCK */ #endif /* __LINUX_BIO_H */
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 03/19] block: add bio_set_polled() helper
Quoted 5 lines · Jens Axboe
For the upcoming async polled IO, we can't sleep allocating requests. If we do, then we introduce a deadlock where the submitter already has async polled IO in-flight, but can't wait for them to complete since polled requests must be active found and reaped.
Quoted 9 lines
Utilize the helper in the blockdev
DIRECT_IOcode.Reviewed-by Christoph Hellwig <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- fs/block_dev.c | 4 ++-- include/linux/bio.h | 14 ++++++++++++++ 2 files changed, 16 insertions(+), 2 deletions(-)
Cheers,
Hannes
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 4/19 iomap: wire up the iopoll method
In reply to Jens AxboeFrom: Christoph Hellwig <[email protected]>
Store the request queue the last bio was submitted to in the iocb private data in addition to the cookie so that we find the right block device. Also refactor the common direct I/O bio submission code into a nice little helper.
Modified to use bio_set_polled().
--- fs/gfs2/file.c | 2 ++ fs/iomap.c | 43 ++++++++++++++++++++++++++++--------------- fs/xfs/xfs_file.c | 1 + include/linux/iomap.h | 1 + 4 files changed, 32 insertions(+), 15 deletions(-)
fs/gfs2/file.c +2 -0
@@ -1280,6 +1280,7 @@ const struct file_operations gfs2_file_fops = { .llseek = gfs2_llseek, .read_iter = gfs2_file_read_iter, .write_iter = gfs2_file_write_iter, + .iopoll = iomap_dio_iopoll, .unlocked_ioctl = gfs2_ioctl, .mmap = gfs2_mmap, .open = gfs2_open,
@@ -1310,6 +1311,7 @@ const struct file_operations gfs2_file_fops_nolock = { .llseek = gfs2_llseek, .read_iter = gfs2_file_read_iter, .write_iter = gfs2_file_write_iter, + .iopoll = iomap_dio_iopoll, .unlocked_ioctl = gfs2_ioctl, .mmap = gfs2_mmap, .open = gfs2_open,
fs/iomap.c +28 -15
82 lines not on this page · raw message
fs/xfs/xfs_file.c +1 -0
@@ -1203,6 +1203,7 @@ const struct file_operations xfs_file_operations = { .write_iter = xfs_file_write_iter, .splice_read = generic_file_splice_read, .splice_write = iter_file_splice_write, + .iopoll = iomap_dio_iopoll, .unlocked_ioctl = xfs_file_ioctl, #ifdef CONFIG_COMPAT .compat_ioctl = xfs_file_compat_ioctl,
include/linux/iomap.h +1 -0
@@ -162,6 +162,7 @@ typedef int (iomap_dio_end_io_t)(struct kiocb *iocb, ssize_t ret, unsigned flags);ssize_t iomap_dio_rw (struct kiocb *iocb,struct iov_iter *iter,const struct iomap_ops *ops,iomap_dio_end_io_t end_io); +int iomap_dio_iopoll (struct kiocb *kiocb,bool spin);#ifdef CONFIG_SWAP struct file ;
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 04/19] iomap: wire up the iopoll method
Quoted 11 lines · Jens Axboe
From: Christoph Hellwig <[email protected]>
Store the request queue the last bio was submitted to in the iocb private data in addition to the cookie so that we find the right block device. Also refactor the common direct I/O bio submission code into a nice little helper.
Signed-off-by Christoph Hellwig <[email protected]>Modified to use
bio_set_polled().
Quoted 8 lines
Signed-off-by Jens Axboe <[email protected]>--- fs/gfs2/file.c | 2 ++ fs/iomap.c | 43 ++++++++++++++++++++++++++++--------------- fs/xfs/xfs_file.c | 1 + include/linux/iomap.h | 1 + 4 files changed, 32 insertions(+), 15 deletions(-)
Cheers,
Hannes
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 5/19 Add io_uring IO interface
In reply to Jens AxboeThe submission queue (SQ) and completion queue (CQ) rings are shared between the application and the kernel. This eliminates the need to copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions are generated in the form of io_uring_cqe data structures. The SQ ring is an index into the io_uring_sqe array, which makes it possible to submit a batch of IOs without them being contiguous in the ring. The CQ ring is always contiguous, as completion events are inherently unordered, and hence any io_uring_cqe entry can point back to an arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success, returns a file descriptor that the application can mmap to gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for them to complete, or both. The behavior is controlled by the parameters passed in. If 'to_submit' is non-zero, then we'll try and submit new IO. If IORING_ENTER_GETEVENTS is set, the kernel will wait for 'min_complete' events, if they aren't already available. It's valid to set IORING_ENTER_GETEVENTS and 'min_complete' == 0 at the same time, this allows the kernel to return already completed events without waiting for them. This is useful only for polling, as for IRQ driven IO, the application can just check the CQ ring without entering the kernel.
With this setup, it's possible to do async IO with a single system call. Future developments will enable polled IO with this interface, and polled submission as well. The latter will enable an application to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO as well. We will only punt to an async context if the command would need to wait for IO on the device side. Any data that can be accessed directly in the page cache is done inline. This avoids the slowness issue of usual threadpools, since cached data is accessed as quickly as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
--- arch/x86/entry/syscalls/syscall_32.tbl | 2 + arch/x86/entry/syscalls/syscall_64.tbl | 2 + fs/Makefile | 1 + fs/io_uring.c | 1175 ++++++++++++++++++++++++ include/linux/fs.h | 9 + include/linux/syscalls.h | 6 + include/uapi/asm-generic/unistd.h | 6 +- include/uapi/linux/io_uring.h | 95 ++ init/Kconfig | 9 + kernel/sys_ni.c | 2 + net/unix/garbage.c | 3 + 11 files changed, 1309 insertions(+), 1 deletion(-) create mode 100644 fs/io_uring.c create mode 100644 include/uapi/linux/io_uring.h
arch/x86/entry/syscalls/syscall_32.tbl +2 -0
@@ -398,3 +398,5 @@ 384 i386 arch_prctl sys_arch_prctl __ia32_compat_sys_arch_prctl 385 i386 io_pgetevents sys_io_pgetevents __ia32_compat_sys_io_pgetevents 386 i386 rseq sys_rseq __ia32_sys_rseq +425 i386 io_uring_setup sys_io_uring_setup __ia32_sys_io_uring_setup +426 i386 io_uring_enter sys_io_uring_enter __ia32_sys_io_uring_enter
arch/x86/entry/syscalls/syscall_64.tbl +2 -0
@@ -343,6 +343,8 @@ 332 common statx __x64_sys_statx 333 common io_pgetevents __x64_sys_io_pgetevents 334 common rseq __x64_sys_rseq +425 common io_uring_setup __x64_sys_io_uring_setup +426 common io_uring_enter __x64_sys_io_uring_enter # # x32-specific system call numbers start at 512 to avoid cache impact
fs/Makefile +1 -0
@@ -30,6 +30,7 @@ obj-$(CONFIG_TIMERFD) += timerfd.o obj-$(CONFIG_EVENTFD) += eventfd.o obj-$(CONFIG_USERFAULTFD) += userfaultfd.o obj-$(CONFIG_AIO) += aio.o +obj-$(CONFIG_IO_URING) += io_uring.o obj-$(CONFIG_FS_DAX) += dax.o obj-$(CONFIG_FS_ENCRYPTION) += crypto/ obj-$(CONFIG_FILE_LOCKING) += locks.o
fs/io_uring.c new file +1175 -0
1,175 lines not on this page · raw message
include/linux/fs.h +9 -0
@@ -3517,4 +3517,13 @@ extern void inode_nohighmem(struct inode *inode);extern int vfs_fadvise (struct file *file,loff_t offset,loff_t len,int advice); +#if defined(CONFIG_IO_URING ) +extern struct sock *io_uring_get_socket (struct file *file); +#else +static inline struct sock *io_uring_get_socket (struct file *file) +{ +return NULL ; +} +#endif +#endif /* _LINUX_FS_H */
include/linux/syscalls.h +6 -0
@@ -69,6 +69,7 @@ struct file_handle;struct sigaltstack ;struct rseq ;union bpf_attr ; +struct io_uring_params ;#include <linux/types.h> #include <linux/aio_abi.h>
@@ -309,6 +310,11 @@ asmlinkage long sys_io_pgetevents_time32(aio_context_t ctx_id,struct io_event __user *events, struct old_timespec32 __user *timeout, const struct __aio_sigset *sig); +asmlinkage long sys_io_uring_setup (u32 entries, +struct io_uring_params __user *p); +asmlinkage long sys_io_uring_enter (unsigned int fd,u32 to_submit, +u32 min_complete,u32 flags, +const sigset_t __user *sig,size_t sigsz);/* fs/xattr.c */ asmlinkage long sys_setxattr(const char __user *path, const char __user *name,
include/uapi/asm-generic/unistd.h +5 -1
@@ -740,9 +740,13 @@ __SC_COMP(__NR_io_pgetevents, sys_io_pgetevents, compat_sys_io_pgetevents) __SYSCALL(__NR_rseq, sys_rseq)#define __NR_kexec_file_load 294 __SYSCALL(__NR_kexec_file_load, sys_kexec_file_load) +#define __NR_io_uring_setup 425 +__SYSCALL(__NR_io_uring_setup, sys_io_uring_setup) +#define __NR_io_uring_enter 426 +__SYSCALL(__NR_io_uring_enter, sys_io_uring_enter)#undef __NR_syscalls -#define __NR_syscalls 295 +#define __NR_syscalls 427/* * 32 bit systems traditionally used different
include/uapi/linux/io_uring.h new file +95 -0
95 lines not on this page · raw message
init/Kconfig +9 -0
@@ -1414,6 +1414,15 @@ config AIO by some high performance threaded applications. Disabling this option saves about 7k. +config IO_URING + bool "Enable IO uring support" if EXPERT + select ANON_INODES + default y + help + This option enables support for the io_uring interface, enabling + applications to submit and completion IO through submission and + completion rings that are shared between the kernel and application. + config ADVISE_SYSCALLS bool "Enable madvise/fadvise syscalls" if EXPERT default y
kernel/sys_ni.c +2 -0
@@ -46,6 +46,8 @@ COND_SYSCALL(io_getevents);COND_SYSCALL (io_pgetevents);COND_SYSCALL_COMPAT (io_getevents);COND_SYSCALL_COMPAT (io_pgetevents); +COND_SYSCALL (io_uring_setup); +COND_SYSCALL (io_uring_enter);/* fs/xattr.c */
net/unix/garbage.c +3 -0
@@ -108,6 +108,9 @@ struct sock *unix_get_socket(struct file *filp)/* PF_UNIX ? */ if (s && sock->ops && sock->ops->family ==PF_UNIX ) u_sock = s; + }else { +/* Could be an io_uring instance */ + u_sock =io_uring_get_socket (filp); }return u_sock; }
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 05/19] Add io_uring IO interface
Quoted 46 lines · Jens Axboe
The submission queue (SQ) and completion queue (CQ) rings are shared between the application and the kernel. This eliminates the need to copy data back and forth to submit and complete IO.
IO submissions use the
io_uring_sqedata structure, and completions are generated in the form ofio_uring_cqedata structures. The SQ ring is an index into theio_uring_sqearray, which makes it possible to submit a batch of IOs without them being contiguous in the ring. The CQ ring is always contiguous, as completion events are inherently unordered, and hence anyio_uring_cqeentry can point back to an arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up anio_uringinstance for doing async IO. On success, returns a file descriptor that the application can mmap to gain access to the SQ ring, CQ ring, andio_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for them to complete, or both. The behavior is controlled by the parameters passed in. If 'to_submit' is non-zero, then we'll try and submit new IO. IfIORING_ENTER_GETEVENTSis set, the kernel will wait for 'min_complete' events, if they aren't already available. It's valid to setIORING_ENTER_GETEVENTSand 'min_complete' == 0 at the same time, this allows the kernel to return already completed events without waiting for them. This is useful only for polling, as for IRQ driven IO, the application can just check the CQ ring without entering the kernel.With this setup, it's possible to do async IO with a single system call. Future developments will enable polled IO with this interface, and polled submission as well. The latter will enable an application to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for completions if it wants to wait for them to occur.
Each
io_uringis backed by a workqueue, to support buffered async IO as well. We will only punt to an async context if the command would need to wait for IO on the device side. Any data that can be accessed directly in the page cache is done inline. This avoids the slowness issue of usual threadpools, since cached data is accessed as quickly as a sync interface.
[...]
fs/io_uring.c line 184
Quoted 5 lines
fs/io_uring.c new file +5
-0+static void io_commit_cqring (struct io_ring_ctx *ctx) +{ +struct io_cq_ring *ring = ctx->cq_ring; + +if (ctx->cached_cq_tail != ring->r.tail) {
I know that this is very unlikely to actually matter, but both because I don't feel fuzzy about relying on compiler internals regarding when the compiler might decide to generate dangerous double-reads (if switch() can blow up, why shouldn't the compiler be able to make if() blow up if it wants to, too?), and because I would like it to be as clear as possible to the reader which memory is shared with userspace, can we please have READ_ONCE() on *every* shared memory read, not just the ones in places that look like they might plausibly blow up otherwise? Sorry, shared memory is a bit of a pet peeve of mine.
Quoted 12 lines
fs/io_uring.c new file +12
-0+/* order cqe stores with ring update */ +smp_wmb (); +WRITE_ONCE (ring->r.tail, ctx->cached_cq_tail); +/* write side barrier of tail update, app has read side */ +smp_wmb (); + +if (wq_has_sleeper (&ctx->cq_wait)) { +wake_up_interruptible (&ctx->cq_wait); +kill_fasync (&ctx->cq_fasync,SIGIO ,POLL_IN ); + } + } +}
[...]
Quoted 5 lines
fs/io_uring.c new file +5
-0+static void io_cqring_fill_event (struct io_ring_ctx *ctx,u64 ki_user_data, +long res,unsigned ev_flags) +{ +struct io_uring_cqe *cqe; +
fs/io_uring.c line 226
Quoted 10 lines
fs/io_uring.c new file +10
-0+/* +* If we can't get a cq entry, userspace overflowed the +* submission (by quite a lot). Increment the overflow count in +* the ring. +*/ + cqe =io_get_cqring (ctx); +if (cqe) { + cqe->user_data = ki_user_data; + cqe->res = res; + cqe->flags = ev_flags;
Please use WRITE_ONCE() for stores like these.
Quoted 3 lines
fs/io_uring.c new file +3
-0+ }else + ctx->cq_ring->overflow++; +}
[...]
fs/io_uring.c line 531
Quoted 11 lines
fs/io_uring.c new file +11
-0+static int __io_submit_sqe (struct io_ring_ctx *ctx,struct io_kiocb *req, +const struct sqe_submit *s,bool force_nonblock) +{ +ssize_t ret; +int opcode; + +if (unlikely (s->index >= ctx->sq_entries)) +return -EINVAL ; + req->user_data =READ_ONCE (s->sqe->user_data); + + opcode =READ_ONCE (s->sqe->opcode);
There might be a sneaky bug here. Consider the following scenario:
1. request gets submitted from io_sq_wq_submit_work() with opcode IORING_OP_READV, io_read() is invoked
2. io_read() looks up the file, taking a reference to it
3. call_read_iter() returns -EAGAIN
4. io_read() returns -EAGAIN without dropping its reference on the file (because it expects that it'll be called again)
5. __io_submit_sqe() returns -EAGAIN
6. io_sq_wq_submit_work() loops back and retries __io_submit_sqe()
7. __io_submit_sqe() reads opcode again, this time it's IORING_OP_NOP
8. io_nop() gets called
9. io_nop() uses io_free_req() to delete the request without dropping its reference on the file
So that's a file reference leak, I think?
Quoted 17 lines
fs/io_uring.c new file +17
-0+switch (opcode) { +case IORING_OP_NOP : + ret =io_nop (req, req->user_data); +break ; +case IORING_OP_READV : + ret =io_read (req, s, force_nonblock); +break ; +case IORING_OP_WRITEV : + ret =io_write (req, s, force_nonblock); +break ; +default : + ret = -EINVAL ; +break ; + } + +return ret; +}
[...]
Quoted 26 lines
fs/io_uring.c new file +26
-0+static int io_submit_sqe (struct io_ring_ctx *ctx,const struct sqe_submit *s) +{ +struct io_kiocb *req; +ssize_t ret; + +/* enforce forwards compatibility on users */ +if (unlikely (s->sqe->flags)) +return -EINVAL ; + + req =io_get_req (ctx); +if (unlikely (!req)) +return -EAGAIN ; + + req->rw.ki_filp =NULL ; + + ret =__io_submit_sqe (ctx, req, s, true); +if (ret == -EAGAIN ) { +memcpy (&req->submit, s,sizeof (*s)); +INIT_WORK (&req->work, io_sq_wq_submit_work); +queue_work (ctx->sqo_wq, &req->work); + ret =0 ; + } +if (ret) +io_free_req (req); + +return ret;
fs/io_uring.c line 619
Quoted 10 lines
fs/io_uring.c new file +10
-0+} + +static void io_commit_sqring (struct io_ring_ctx *ctx) +{ +struct io_sq_ring *ring = ctx->sq_ring; + +if (ctx->cached_sq_head != ring->r.head) { +WRITE_ONCE (ring->r.head, ctx->cached_sq_head); +/* write side barrier of head update, app has read side */ +smp_wmb ();
Can you elaborate on what this memory barrier is doing? Don't you need some sort of memory barrier *before* the WRITE_ONCE(), to ensure that nobody sees the updated head before you're done reading the submission queue entry? Or is that barrier elsewhere?
Quoted 10 lines
fs/io_uring.c new file +10
-0+ } +} + +/* +* Undo last io_get_sqring() +*/ +static void io_drop_sqring (struct io_ring_ctx *ctx) +{ + ctx->cached_sq_head--; +}
[...]
fs/io_uring.c line 775
Quoted 4 lines
fs/io_uring.c new file +4
-0+static void io_unaccount_mem (struct user_struct *user,unsigned long nr_pages) +{ +if (capable (CAP_IPC_LOCK )) +return ;
Hrm... what happens if root creates a uring, drops CAP_IPC_LOCK, and then destroys the uring? Will the pages get subtracted from ->locked_vm even though they were never added to it, causing a wraparound?
You might want to make sure that ctx->user is set if and only if the creator didn't have CAP_IPC_LOCK, and then just do a user == NULL check instead of a capable(...) check. Or you could do what BPF is doing (AFAICS) and not treat root specially - root can just bump the rlimit if necessary.
Quoted 23 lines
fs/io_uring.c new file +23
-0+atomic_long_sub (nr_pages, &user->locked_vm); +} + +static int io_account_mem (struct user_struct *user,unsigned long nr_pages) +{ +unsigned long page_limit, cur_pages, new_pages; + +if (capable (CAP_IPC_LOCK )) +return 0 ; + +/* Don't allow more pages than we can safely lock */ + page_limit =rlimit (RLIMIT_MEMLOCK ) >>PAGE_SHIFT ; + +do { + cur_pages =atomic_long_read (&user->locked_vm); + new_pages = cur_pages + nr_pages; +if (new_pages > page_limit) +return -ENOMEM ; + }while (atomic_long_cmpxchg (&user->locked_vm, cur_pages, + new_pages) != cur_pages); + +return 0 ; +}
[...]
init/Kconfig config AIO · line 1424
Quoted 8 lines
init/Kconfig +8
-0+config IO_URING + bool "Enable IO uring support" if EXPERT + select ANON_INODES + default y + help + This option enables support for the io_uring interface, enabling + applications to submit and completion IO through submission and + completion rings that are shared between the kernel and application.
Nit: I can't parse this part: "enabling applications to submit and completion IO"
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 05/19] Add io_uring IO interface
Quoted 48 lines · Jann Horn
Quoted 46 lines · Jens Axboe
The submission queue (SQ) and completion queue (CQ) rings are shared between the application and the kernel. This eliminates the need to copy data back and forth to submit and complete IO.
IO submissions use the
io_uring_sqedata structure, and completions are generated in the form ofio_uring_cqedata structures. The SQ ring is an index into theio_uring_sqearray, which makes it possible to submit a batch of IOs without them being contiguous in the ring. The CQ ring is always contiguous, as completion events are inherently unordered, and hence anyio_uring_cqeentry can point back to an arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up anio_uringinstance for doing async IO. On success, returns a file descriptor that the application can mmap to gain access to the SQ ring, CQ ring, andio_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for them to complete, or both. The behavior is controlled by the parameters passed in. If 'to_submit' is non-zero, then we'll try and submit new IO. IfIORING_ENTER_GETEVENTSis set, the kernel will wait for 'min_complete' events, if they aren't already available. It's valid to setIORING_ENTER_GETEVENTSand 'min_complete' == 0 at the same time, this allows the kernel to return already completed events without waiting for them. This is useful only for polling, as for IRQ driven IO, the application can just check the CQ ring without entering the kernel.With this setup, it's possible to do async IO with a single system call. Future developments will enable polled IO with this interface, and polled submission as well. The latter will enable an application to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for completions if it wants to wait for them to occur.
Each
io_uringis backed by a workqueue, to support buffered async IO as well. We will only punt to an async context if the command would need to wait for IO on the device side. Any data that can be accessed directly in the page cache is done inline. This avoids the slowness issue of usual threadpools, since cached data is accessed as quickly as a sync interface.[...]
Quoted 15 lines
fs/io_uring.c line 184
Quoted 5 lines
fs/io_uring.c new file +5
-0+static void io_commit_cqring (struct io_ring_ctx *ctx) +{ +struct io_cq_ring *ring = ctx->cq_ring; + +if (ctx->cached_cq_tail != ring->r.tail) {I know that this is very unlikely to actually matter, but both because I don't feel fuzzy about relying on compiler internals regarding when the compiler might decide to generate dangerous double-reads (if
switch()can blow up, why shouldn't the compiler be able to makeif()blow up if it wants to, too?), and because I would like it to be as clear as possible to the reader which memory is shared with userspace, can we please haveREAD_ONCE()on *every* shared memory read, not just the ones in places that look like they might plausibly blow up otherwise? Sorry, shared memory is a bit of a pet peeve of mine.
Sure, I've done that now.
Quoted 22 lines
Quoted 12 lines
fs/io_uring.c new file +12
-0+/* order cqe stores with ring update */ +smp_wmb (); +WRITE_ONCE (ring->r.tail, ctx->cached_cq_tail); +/* write side barrier of tail update, app has read side */ +smp_wmb (); + +if (wq_has_sleeper (&ctx->cq_wait)) { +wake_up_interruptible (&ctx->cq_wait); +kill_fasync (&ctx->cq_fasync,SIGIO ,POLL_IN ); + } + } +}[...]
Quoted 9 lines
fs/io_uring.c new file +9
-0+static void io_cqring_fill_event (struct io_ring_ctx *ctx,u64 ki_user_data, +long res,unsigned ev_flags) +{ +struct io_uring_cqe *cqe; + +/* +* If we can't get a cq entry, userspace overflowed the +* submission (by quite a lot). Increment the overflow count in +* the ring.
Quoted 8 lines
fs/io_uring.c line 226
Quoted 6 lines
fs/io_uring.c new file +6
-0+*/ + cqe =io_get_cqring (ctx); +if (cqe) { + cqe->user_data = ki_user_data; + cqe->res = res; + cqe->flags = ev_flags;Please use
WRITE_ONCE()for stores like these.
Done
Quoted 31 lines
Quoted 3 lines
fs/io_uring.c new file +3
-0+ }else + ctx->cq_ring->overflow++; +}[...]
fs/io_uring.c line 531
Quoted 11 lines
fs/io_uring.c new file +11
-0+static int __io_submit_sqe (struct io_ring_ctx *ctx,struct io_kiocb *req, +const struct sqe_submit *s,bool force_nonblock) +{ +ssize_t ret; +int opcode; + +if (unlikely (s->index >= ctx->sq_entries)) +return -EINVAL ; + req->user_data =READ_ONCE (s->sqe->user_data); + + opcode =READ_ONCE (s->sqe->opcode);There might be a sneaky bug here. Consider the following scenario:
1. request gets submitted from
io_sq_wq_submit_work()with opcodeIORING_OP_READV,io_read()is invoked
2.io_read()looks up the file, taking a reference to it
3.call_read_iter()returns-EAGAIN
4.io_read()returns-EAGAINwithout dropping its reference on the file (because it expects that it'll be called again)
5.__io_submit_sqe()returns-EAGAIN
6.io_sq_wq_submit_work()loops back and retries__io_submit_sqe()
7.__io_submit_sqe()reads opcode again, this time it'sIORING_OP_NOP
8.io_nop()gets called
9.io_nop()usesio_free_req()to delete the request without dropping its reference on the file
Quoted 1 line
So that's a file reference leak, I think?
Hmm yes, that could happen with a malicious app.
For non-file using opcodes, I think we should just error the sqe if we have req->rw.ki_filp set. That shouldn't happen unless the app is doing something funky. I'll fix this.
Quoted 51 lines
Quoted 17 lines
fs/io_uring.c new file +17
-0+switch (opcode) { +case IORING_OP_NOP : + ret =io_nop (req, req->user_data); +break ; +case IORING_OP_READV : + ret =io_read (req, s, force_nonblock); +break ; +case IORING_OP_WRITEV : + ret =io_write (req, s, force_nonblock); +break ; +default : + ret = -EINVAL ; +break ; + } + +return ret; +}[...]
Quoted 33 lines
fs/io_uring.c new file +33
-0+static int io_submit_sqe (struct io_ring_ctx *ctx,const struct sqe_submit *s) +{ +struct io_kiocb *req; +ssize_t ret; + +/* enforce forwards compatibility on users */ +if (unlikely (s->sqe->flags)) +return -EINVAL ; + + req =io_get_req (ctx); +if (unlikely (!req)) +return -EAGAIN ; + + req->rw.ki_filp =NULL ; + + ret =__io_submit_sqe (ctx, req, s, true); +if (ret == -EAGAIN ) { +memcpy (&req->submit, s,sizeof (*s)); +INIT_WORK (&req->work, io_sq_wq_submit_work); +queue_work (ctx->sqo_wq, &req->work); + ret =0 ; + } +if (ret) +io_free_req (req); + +return ret; +} + +static void io_commit_sqring (struct io_ring_ctx *ctx) +{ +struct io_sq_ring *ring = ctx->sq_ring; + +if (ctx->cached_sq_head != ring->r.head) {
Quoted 8 lines
fs/io_uring.c line 619
Quoted 3 lines
fs/io_uring.c new file +3
-0+WRITE_ONCE (ring->r.head, ctx->cached_sq_head); +/* write side barrier of head update, app has read side */ +smp_wmb ();Can you elaborate on what this memory barrier is doing? Don't you need some sort of memory barrier *before* the
WRITE_ONCE(), to ensure that nobody sees the updated head before you're done reading the submission queue entry? Or is that barrier elsewhere?
The matching read barrier is in the application, it must do that before reading ->head for the SQ ring.
For the other barrier, since the ring->r.head now has a READ_ONCE(), that should be all we need to ensure that loads are done.
Quoted 21 lines
Quoted 10 lines
fs/io_uring.c new file +10
-0+ } +} + +/* +* Undo last io_get_sqring() +*/ +static void io_drop_sqring (struct io_ring_ctx *ctx) +{ + ctx->cached_sq_head--; +}[...]
fs/io_uring.c line 775
Quoted 4 lines
fs/io_uring.c new file +4
-0+static void io_unaccount_mem (struct user_struct *user,unsigned long nr_pages) +{ +if (capable (CAP_IPC_LOCK )) +return ;Hrm... what happens if root creates a uring, drops
CAP_IPC_LOCK, and then destroys the uring? Will the pages get subtracted from ->locked_vm even though they were never added to it, causing a wraparound?
Quoted 5 lines
You might want to make sure that
ctx->useris set if and only if the creator didn't haveCAP_IPC_LOCK, and then just do auser == NULLcheck instead of acapable(...)check. Or you could do what BPF is doing (AFAICS) and not treat root specially - root can just bump the rlimit if necessary.
That won't work since we use ->user for other items later on. But I can store whether we need it or not, I'll do that.
Quoted 34 lines
Quoted 24 lines
fs/io_uring.c new file +23
-0+atomic_long_sub (nr_pages, &user->locked_vm); +} + +static int io_account_mem (struct user_struct *user,unsigned long nr_pages) +{ +unsigned long page_limit, cur_pages, new_pages; + +if (capable (CAP_IPC_LOCK )) +return 0 ; + +/* Don't allow more pages than we can safely lock */ + page_limit =rlimit (RLIMIT_MEMLOCK ) >>PAGE_SHIFT ; + +do { + cur_pages =atomic_long_read (&user->locked_vm); + new_pages = cur_pages + nr_pages; +if (new_pages > page_limit) +return -ENOMEM ; + }while (atomic_long_cmpxchg (&user->locked_vm, cur_pages, + new_pages) != cur_pages); + +return 0 ; +}[...]
init/Kconfig config AIO · line 1424
Quoted 8 lines
init/Kconfig +8
-0+config IO_URING + bool "Enable IO uring support" if EXPERT + select ANON_INODES + default y + help + This option enables support for the io_uring interface, enabling + applications to submit and completion IO through submission and + completion rings that are shared between the kernel and application.
Quoted 2 lines
Nit: I can't parse this part: "enabling applications to submit and completion IO"
completion -> complete
Fixed it up.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 05/19] Add io_uring IO interface
Quoted 48 lines · Jens Axboe
Quoted 47 lines · Jann Horn
Quoted 46 lines · Jens Axboe
The submission queue (SQ) and completion queue (CQ) rings are shared between the application and the kernel. This eliminates the need to copy data back and forth to submit and complete IO.
IO submissions use the
io_uring_sqedata structure, and completions are generated in the form ofio_uring_cqedata structures. The SQ ring is an index into theio_uring_sqearray, which makes it possible to submit a batch of IOs without them being contiguous in the ring. The CQ ring is always contiguous, as completion events are inherently unordered, and hence anyio_uring_cqeentry can point back to an arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up anio_uringinstance for doing async IO. On success, returns a file descriptor that the application can mmap to gain access to the SQ ring, CQ ring, andio_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for them to complete, or both. The behavior is controlled by the parameters passed in. If 'to_submit' is non-zero, then we'll try and submit new IO. IfIORING_ENTER_GETEVENTSis set, the kernel will wait for 'min_complete' events, if they aren't already available. It's valid to setIORING_ENTER_GETEVENTSand 'min_complete' == 0 at the same time, this allows the kernel to return already completed events without waiting for them. This is useful only for polling, as for IRQ driven IO, the application can just check the CQ ring without entering the kernel.With this setup, it's possible to do async IO with a single system call. Future developments will enable polled IO with this interface, and polled submission as well. The latter will enable an application to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for completions if it wants to wait for them to occur.
Each
io_uringis backed by a workqueue, to support buffered async IO as well. We will only punt to an async context if the command would need to wait for IO on the device side. Any data that can be accessed directly in the page cache is done inline. This avoids the slowness issue of usual threadpools, since cached data is accessed as quickly as a sync interface.
[...]
Quoted 42 lines
Quoted 33 lines
Quoted 33 lines
fs/io_uring.c new file +33
-0+static int io_submit_sqe (struct io_ring_ctx *ctx,const struct sqe_submit *s) +{ +struct io_kiocb *req; +ssize_t ret; + +/* enforce forwards compatibility on users */ +if (unlikely (s->sqe->flags)) +return -EINVAL ; + + req =io_get_req (ctx); +if (unlikely (!req)) +return -EAGAIN ; + + req->rw.ki_filp =NULL ; + + ret =__io_submit_sqe (ctx, req, s, true); +if (ret == -EAGAIN ) { +memcpy (&req->submit, s,sizeof (*s)); +INIT_WORK (&req->work, io_sq_wq_submit_work); +queue_work (ctx->sqo_wq, &req->work); + ret =0 ; + } +if (ret) +io_free_req (req); + +return ret; +} + +static void io_commit_sqring (struct io_ring_ctx *ctx) +{ +struct io_sq_ring *ring = ctx->sq_ring; + +if (ctx->cached_sq_head != ring->r.head) {Quoted 8 lines
fs/io_uring.c line 619
Quoted 3 lines
fs/io_uring.c new file +3
-0+WRITE_ONCE (ring->r.head, ctx->cached_sq_head); +/* write side barrier of head update, app has read side */ +smp_wmb ();Can you elaborate on what this memory barrier is doing? Don't you need some sort of memory barrier *before* the
WRITE_ONCE(), to ensure that nobody sees the updated head before you're done reading the submission queue entry? Or is that barrier elsewhere?
Quoted 5 lines
The matching read barrier is in the application, it must do that before reading ->head for the SQ ring.
For the other barrier, since the
ring->r.headnow has aREAD_ONCE(), that should be all we need to ensure that loads are done.
READ_ONCE() / WRITE_ONCE are not hardware memory barriers that enforce ordering with regard to concurrent execution on other cores. They are only compiler barriers, influencing the order in which the compiler emits things. (Well, unless you're on alpha, where READ_ONCE() implies a memory barrier that prevents reordering of dependent reads.)
As far as I can tell, between the READ_ONCE(ring->array[...]) in io_get_sqring() and the WRITE_ONCE() in io_commit_sqring(), you have no *hardware* memory barrier that prevents reordering against concurrently running userspace code. As far as I can tell, the following could happen:
- The kernel reads from ring->array in io_get_sqring(), then updates
the head in io_commit_sqring(). The CPU reorders the memory accesses such that the write to the head becomes visible before the read from ring->array has completed.
- Userspace observes the write to the head and reuses the array slots
the kernel has freed with the write, clobbering ring->array before the kernel reads from ring->array.
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 05/19] Add io_uring IO interface
Quoted 110 lines · Jann Horn
Quoted 48 lines · Jens Axboe
Quoted 47 lines · Jann Horn
Quoted 46 lines · Jens Axboe
The submission queue (SQ) and completion queue (CQ) rings are shared between the application and the kernel. This eliminates the need to copy data back and forth to submit and complete IO.
IO submissions use the
io_uring_sqedata structure, and completions are generated in the form ofio_uring_cqedata structures. The SQ ring is an index into theio_uring_sqearray, which makes it possible to submit a batch of IOs without them being contiguous in the ring. The CQ ring is always contiguous, as completion events are inherently unordered, and hence anyio_uring_cqeentry can point back to an arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up anio_uringinstance for doing async IO. On success, returns a file descriptor that the application can mmap to gain access to the SQ ring, CQ ring, andio_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for them to complete, or both. The behavior is controlled by the parameters passed in. If 'to_submit' is non-zero, then we'll try and submit new IO. IfIORING_ENTER_GETEVENTSis set, the kernel will wait for 'min_complete' events, if they aren't already available. It's valid to setIORING_ENTER_GETEVENTSand 'min_complete' == 0 at the same time, this allows the kernel to return already completed events without waiting for them. This is useful only for polling, as for IRQ driven IO, the application can just check the CQ ring without entering the kernel.With this setup, it's possible to do async IO with a single system call. Future developments will enable polled IO with this interface, and polled submission as well. The latter will enable an application to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for completions if it wants to wait for them to occur.
Each
io_uringis backed by a workqueue, to support buffered async IO as well. We will only punt to an async context if the command would need to wait for IO on the device side. Any data that can be accessed directly in the page cache is done inline. This avoids the slowness issue of usual threadpools, since cached data is accessed as quickly as a sync interface.[...]
Quoted 42 lines
Quoted 33 lines
Quoted 33 lines
(quoted diff) +33
-0+static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s) +{ + struct io_kiocb *req; + ssize_t ret; + + /* enforce forwards compatibility on users */ + if (unlikely(s->sqe->flags)) + return -EINVAL; + + req = io_get_req(ctx); + if (unlikely(!req)) + return -EAGAIN; + + req->rw.ki_filp = NULL; + + ret = __io_submit_sqe(ctx, req, s, true); + if (ret == -EAGAIN) { + memcpy(&req->submit, s, sizeof(*s)); + INIT_WORK(&req->work, io_sq_wq_submit_work); + queue_work(ctx->sqo_wq, &req->work); + ret = 0; + } + if (ret) + io_free_req(req); + + return ret; +} + +static void io_commit_sqring(struct io_ring_ctx *ctx) +{ + struct io_sq_ring *ring = ctx->sq_ring; + + if (ctx->cached_sq_head != ring->r.head) {Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ WRITE_ONCE(ring->r.head, ctx->cached_sq_head); + /* write side barrier of head update, app has read side */ + smp_wmb();Can you elaborate on what this memory barrier is doing? Don't you need some sort of memory barrier *before* the
WRITE_ONCE(), to ensure that nobody sees the updated head before you're done reading the submission queue entry? Or is that barrier elsewhere?Quoted 5 lines
The matching read barrier is in the application, it must do that before reading ->head for the SQ ring.
For the other barrier, since the
ring->r.headnow has aREAD_ONCE(), that should be all we need to ensure that loads are done.
READ_ONCE()/WRITE_ONCEare not hardware memory barriers that enforce ordering with regard to concurrent execution on other cores. They are only compiler barriers, influencing the order in which the compiler emits things. (Well, unless you're on alpha, whereREAD_ONCE()implies a memory barrier that prevents reordering of dependent reads.)As far as I can tell, between the
READ_ONCE(ring->array[...])inio_get_sqring()and theWRITE_ONCE()inio_commit_sqring(), you have no *hardware* memory barrier that prevents reordering against concurrently running userspace code. As far as I can tell, the following could happen:
Quoted 7 lines
- The kernel reads from
ring->arrayinio_get_sqring(), then updates
the head inio_commit_sqring(). The CPU reorders the memory accesses such that the write to the head becomes visible before the read fromring->arrayhas completed.
- Userspace observes the write to the head and reuses the array slots
the kernel has freed with the write, clobberingring->arraybefore the kernel reads fromring->array.
I'd say this is highly theoretical for the normal use case, as we will have submitted IO in between. Hence the load must have been done. The only case that needs it is the sq thread case, since we bundle those up. This should do it:
fs/io_uring.c +3 -0
@@ -1889,6 +1889,9 @@ static int io_sq_thread(void *data)break ; }while (io_get_sqring (ctx, &sqes[i])); +/* order the below ring head store with the SQE loads */ +smp_mb (); +io_commit_sqring (ctx);/* Unless all new commands are FIXED regions, grab mm */
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 05/19] Add io_uring IO interface
Quoted 119 lines · Jens Axboe
Quoted 110 lines · Jann Horn
Quoted 48 lines · Jens Axboe
Quoted 47 lines · Jann Horn
Quoted 46 lines · Jens Axboe
The submission queue (SQ) and completion queue (CQ) rings are shared between the application and the kernel. This eliminates the need to copy data back and forth to submit and complete IO.
IO submissions use the
io_uring_sqedata structure, and completions are generated in the form ofio_uring_cqedata structures. The SQ ring is an index into theio_uring_sqearray, which makes it possible to submit a batch of IOs without them being contiguous in the ring. The CQ ring is always contiguous, as completion events are inherently unordered, and hence anyio_uring_cqeentry can point back to an arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up anio_uringinstance for doing async IO. On success, returns a file descriptor that the application can mmap to gain access to the SQ ring, CQ ring, andio_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for them to complete, or both. The behavior is controlled by the parameters passed in. If 'to_submit' is non-zero, then we'll try and submit new IO. IfIORING_ENTER_GETEVENTSis set, the kernel will wait for 'min_complete' events, if they aren't already available. It's valid to setIORING_ENTER_GETEVENTSand 'min_complete' == 0 at the same time, this allows the kernel to return already completed events without waiting for them. This is useful only for polling, as for IRQ driven IO, the application can just check the CQ ring without entering the kernel.With this setup, it's possible to do async IO with a single system call. Future developments will enable polled IO with this interface, and polled submission as well. The latter will enable an application to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for completions if it wants to wait for them to occur.
Each
io_uringis backed by a workqueue, to support buffered async IO as well. We will only punt to an async context if the command would need to wait for IO on the device side. Any data that can be accessed directly in the page cache is done inline. This avoids the slowness issue of usual threadpools, since cached data is accessed as quickly as a sync interface.[...]
Quoted 42 lines
Quoted 33 lines
Quoted 33 lines
(quoted diff) +33
-0+static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s) +{ + struct io_kiocb *req; + ssize_t ret; + + /* enforce forwards compatibility on users */ + if (unlikely(s->sqe->flags)) + return -EINVAL; + + req = io_get_req(ctx); + if (unlikely(!req)) + return -EAGAIN; + + req->rw.ki_filp = NULL; + + ret = __io_submit_sqe(ctx, req, s, true); + if (ret == -EAGAIN) { + memcpy(&req->submit, s, sizeof(*s)); + INIT_WORK(&req->work, io_sq_wq_submit_work); + queue_work(ctx->sqo_wq, &req->work); + ret = 0; + } + if (ret) + io_free_req(req); + + return ret; +} + +static void io_commit_sqring(struct io_ring_ctx *ctx) +{ + struct io_sq_ring *ring = ctx->sq_ring; + + if (ctx->cached_sq_head != ring->r.head) {Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ WRITE_ONCE(ring->r.head, ctx->cached_sq_head); + /* write side barrier of head update, app has read side */ + smp_wmb();Can you elaborate on what this memory barrier is doing? Don't you need some sort of memory barrier *before* the
WRITE_ONCE(), to ensure that nobody sees the updated head before you're done reading the submission queue entry? Or is that barrier elsewhere?Quoted 5 lines
The matching read barrier is in the application, it must do that before reading ->head for the SQ ring.
For the other barrier, since the
ring->r.headnow has aREAD_ONCE(), that should be all we need to ensure that loads are done.
READ_ONCE()/WRITE_ONCEare not hardware memory barriers that enforce ordering with regard to concurrent execution on other cores. They are only compiler barriers, influencing the order in which the compiler emits things. (Well, unless you're on alpha, whereREAD_ONCE()implies a memory barrier that prevents reordering of dependent reads.)As far as I can tell, between the
READ_ONCE(ring->array[...])inio_get_sqring()and theWRITE_ONCE()inio_commit_sqring(), you have no *hardware* memory barrier that prevents reordering against concurrently running userspace code. As far as I can tell, the following could happen:Quoted 7 lines
- The kernel reads from
ring->arrayinio_get_sqring(), then updates
the head inio_commit_sqring(). The CPU reorders the memory accesses such that the write to the head becomes visible before the read fromring->arrayhas completed.
- Userspace observes the write to the head and reuses the array slots
the kernel has freed with the write, clobberingring->arraybefore the kernel reads fromring->array.
Quoted 4 lines
I'd say this is highly theoretical for the normal use case, as we will have submitted IO in between. Hence the load must have been done. The only case that needs it is the sq thread case, since we bundle those up. This should do it:
Actually, I take that back, as in this particular case the sq thread is the only one that reads it. Hence it'll have done a full submission of the read SQE entries before reading a new round. Not that it matters for that case, as a preempt would have implied a full barrier anyway.
The non-sq thread case does not need the store-vs-load ordering barrier, as SQEs are either discarded or submitted before we commit the sqring. Since that's the case, by definition all loads are done.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 05/19] Add io_uring IO interface
Quoted 121 lines · Jens Axboe
Quoted 119 lines · Jens Axboe
Quoted 117 lines · Jann Horn
Quoted 48 lines · Jens Axboe
Quoted 47 lines · Jann Horn
Quoted 46 lines · Jens Axboe
The submission queue (SQ) and completion queue (CQ) rings are shared between the application and the kernel. This eliminates the need to copy data back and forth to submit and complete IO.
IO submissions use the
io_uring_sqedata structure, and completions are generated in the form ofio_uring_cqedata structures. The SQ ring is an index into theio_uring_sqearray, which makes it possible to submit a batch of IOs without them being contiguous in the ring. The CQ ring is always contiguous, as completion events are inherently unordered, and hence anyio_uring_cqeentry can point back to an arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up anio_uringinstance for doing async IO. On success, returns a file descriptor that the application can mmap to gain access to the SQ ring, CQ ring, andio_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for them to complete, or both. The behavior is controlled by the parameters passed in. If 'to_submit' is non-zero, then we'll try and submit new IO. IfIORING_ENTER_GETEVENTSis set, the kernel will wait for 'min_complete' events, if they aren't already available. It's valid to setIORING_ENTER_GETEVENTSand 'min_complete' == 0 at the same time, this allows the kernel to return already completed events without waiting for them. This is useful only for polling, as for IRQ driven IO, the application can just check the CQ ring without entering the kernel.With this setup, it's possible to do async IO with a single system call. Future developments will enable polled IO with this interface, and polled submission as well. The latter will enable an application to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for completions if it wants to wait for them to occur.
Each
io_uringis backed by a workqueue, to support buffered async IO as well. We will only punt to an async context if the command would need to wait for IO on the device side. Any data that can be accessed directly in the page cache is done inline. This avoids the slowness issue of usual threadpools, since cached data is accessed as quickly as a sync interface.[...]
Quoted 42 lines
Quoted 33 lines
Quoted 33 lines
(quoted diff) +33
-0+static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s) +{ + struct io_kiocb *req; + ssize_t ret; + + /* enforce forwards compatibility on users */ + if (unlikely(s->sqe->flags)) + return -EINVAL; + + req = io_get_req(ctx); + if (unlikely(!req)) + return -EAGAIN; + + req->rw.ki_filp = NULL; + + ret = __io_submit_sqe(ctx, req, s, true); + if (ret == -EAGAIN) { + memcpy(&req->submit, s, sizeof(*s)); + INIT_WORK(&req->work, io_sq_wq_submit_work); + queue_work(ctx->sqo_wq, &req->work); + ret = 0; + } + if (ret) + io_free_req(req); + + return ret; +} + +static void io_commit_sqring(struct io_ring_ctx *ctx) +{ + struct io_sq_ring *ring = ctx->sq_ring; + + if (ctx->cached_sq_head != ring->r.head) {Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ WRITE_ONCE(ring->r.head, ctx->cached_sq_head); + /* write side barrier of head update, app has read side */ + smp_wmb();Can you elaborate on what this memory barrier is doing? Don't you need some sort of memory barrier *before* the
WRITE_ONCE(), to ensure that nobody sees the updated head before you're done reading the submission queue entry? Or is that barrier elsewhere?Quoted 5 lines
The matching read barrier is in the application, it must do that before reading ->head for the SQ ring.
For the other barrier, since the
ring->r.headnow has aREAD_ONCE(), that should be all we need to ensure that loads are done.
READ_ONCE()/WRITE_ONCEare not hardware memory barriers that enforce ordering with regard to concurrent execution on other cores. They are only compiler barriers, influencing the order in which the compiler emits things. (Well, unless you're on alpha, whereREAD_ONCE()implies a memory barrier that prevents reordering of dependent reads.)As far as I can tell, between the
READ_ONCE(ring->array[...])inio_get_sqring()and theWRITE_ONCE()inio_commit_sqring(), you have no *hardware* memory barrier that prevents reordering against concurrently running userspace code. As far as I can tell, the following could happen:- The kernel reads from
ring->arrayinio_get_sqring(), then updates
the head inio_commit_sqring(). The CPU reorders the memory accesses such that the write to the head becomes visible before the read fromring->arrayhas completed.
- Userspace observes the write to the head and reuses the array slots
the kernel has freed with the write, clobberingring->arraybefore the kernel reads fromring->array.
Quoted 2 lines
Quoted 2 lines
I'd say this is highly theoretical for the normal use case, as we will have submitted IO in between. Hence the load must have been done.
Sorry, I'm confused. Who is "we", and which load are you referring to? io_sq_thread() goes directly from io_get_sqring() to io_commit_sqring(), with only a conditional io_sqe_needs_user() in between, if the i == ARRAY_SIZE(sqes) check triggers. There is no "submitting IO" in the middle.
Quoted 5 lines
Quoted 2 lines
The only case that needs it is the sq thread case, since we bundle those up. This should do it:
Actually, I take that back, as in this particular case the sq thread is the only one that reads it.
What is "it"? The head pointer is written by the sq thread and read by userspace, not the other way around. Are you talking about ring->array? Sorry, I'm lost.
Quoted 7 lines
Hence it'll have done a full submission
of the read SQE entries before reading a new round. Not that it matters for that case, as a preempt would have implied a full barrier anyway.The non-sq thread case does not need the store-vs-load ordering barrier, as SQEs are either discarded or submitted before we commit the sqring. Since that's the case, by definition all loads are done.
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 05/19] Add io_uring IO interface
Quoted 122 lines · Jann Horn
Quoted 121 lines · Jens Axboe
Quoted 119 lines · Jens Axboe
Quoted 117 lines · Jann Horn
Quoted 48 lines · Jens Axboe
Quoted 47 lines · Jann Horn
Quoted 46 lines · Jens Axboe
The submission queue (SQ) and completion queue (CQ) rings are shared between the application and the kernel. This eliminates the need to copy data back and forth to submit and complete IO.
IO submissions use the
io_uring_sqedata structure, and completions are generated in the form ofio_uring_cqedata structures. The SQ ring is an index into theio_uring_sqearray, which makes it possible to submit a batch of IOs without them being contiguous in the ring. The CQ ring is always contiguous, as completion events are inherently unordered, and hence anyio_uring_cqeentry can point back to an arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up anio_uringinstance for doing async IO. On success, returns a file descriptor that the application can mmap to gain access to the SQ ring, CQ ring, andio_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for them to complete, or both. The behavior is controlled by the parameters passed in. If 'to_submit' is non-zero, then we'll try and submit new IO. IfIORING_ENTER_GETEVENTSis set, the kernel will wait for 'min_complete' events, if they aren't already available. It's valid to setIORING_ENTER_GETEVENTSand 'min_complete' == 0 at the same time, this allows the kernel to return already completed events without waiting for them. This is useful only for polling, as for IRQ driven IO, the application can just check the CQ ring without entering the kernel.With this setup, it's possible to do async IO with a single system call. Future developments will enable polled IO with this interface, and polled submission as well. The latter will enable an application to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for completions if it wants to wait for them to occur.
Each
io_uringis backed by a workqueue, to support buffered async IO as well. We will only punt to an async context if the command would need to wait for IO on the device side. Any data that can be accessed directly in the page cache is done inline. This avoids the slowness issue of usual threadpools, since cached data is accessed as quickly as a sync interface.[...]
Quoted 42 lines
Quoted 33 lines
Quoted 33 lines
(quoted diff) +33
-0+static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s) +{ + struct io_kiocb *req; + ssize_t ret; + + /* enforce forwards compatibility on users */ + if (unlikely(s->sqe->flags)) + return -EINVAL; + + req = io_get_req(ctx); + if (unlikely(!req)) + return -EAGAIN; + + req->rw.ki_filp = NULL; + + ret = __io_submit_sqe(ctx, req, s, true); + if (ret == -EAGAIN) { + memcpy(&req->submit, s, sizeof(*s)); + INIT_WORK(&req->work, io_sq_wq_submit_work); + queue_work(ctx->sqo_wq, &req->work); + ret = 0; + } + if (ret) + io_free_req(req); + + return ret; +} + +static void io_commit_sqring(struct io_ring_ctx *ctx) +{ + struct io_sq_ring *ring = ctx->sq_ring; + + if (ctx->cached_sq_head != ring->r.head) {Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ WRITE_ONCE(ring->r.head, ctx->cached_sq_head); + /* write side barrier of head update, app has read side */ + smp_wmb();Can you elaborate on what this memory barrier is doing? Don't you need some sort of memory barrier *before* the
WRITE_ONCE(), to ensure that nobody sees the updated head before you're done reading the submission queue entry? Or is that barrier elsewhere?Quoted 5 lines
The matching read barrier is in the application, it must do that before reading ->head for the SQ ring.
For the other barrier, since the
ring->r.headnow has aREAD_ONCE(), that should be all we need to ensure that loads are done.
READ_ONCE()/WRITE_ONCEare not hardware memory barriers that enforce ordering with regard to concurrent execution on other cores. They are only compiler barriers, influencing the order in which the compiler emits things. (Well, unless you're on alpha, whereREAD_ONCE()implies a memory barrier that prevents reordering of dependent reads.)As far as I can tell, between the
READ_ONCE(ring->array[...])inio_get_sqring()and theWRITE_ONCE()inio_commit_sqring(), you have no *hardware* memory barrier that prevents reordering against concurrently running userspace code. As far as I can tell, the following could happen:- The kernel reads from
ring->arrayinio_get_sqring(), then updates
the head inio_commit_sqring(). The CPU reorders the memory accesses such that the write to the head becomes visible before the read fromring->arrayhas completed.
- Userspace observes the write to the head and reuses the array slots
the kernel has freed with the write, clobberingring->arraybefore the kernel reads fromring->array.
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
I'd say this is highly theoretical for the normal use case, as we will have submitted IO in between. Hence the load must have been done.
Sorry, I'm confused. Who is "we", and which load are you referring to?
io_sq_thread()goes directly fromio_get_sqring()toio_commit_sqring(), with only a conditionalio_sqe_needs_user()in between, if thei == ARRAY_SIZE(sqes)check triggers. There is no "submitting IO" in the middle.
You are right, the patch I sent IS needed for the sq thread case! It's only true for the "normal" case that we don't need the smp_mb() before writing the sq ring head, as sqes are fully consumed at that point.
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
The only case that needs it is the sq thread case, since we bundle those up. This should do it:
Actually, I take that back, as in this particular case the sq thread is the only one that reads it.
What is "it"? The head pointer is written by the sq thread and read by userspace, not the other way around. Are you talking about
ring->array? Sorry, I'm lost.
I think we're on the same page, even if it doesn't necessarily sound like it. We do need the smp_mb() before witing io_commit_sqring() for the thread case.
I guess what confused me is that your commenting on the main patch, the case that needs it isn't introduced until later in the series. I'll fold the fix into that patch.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 05/19] Add io_uring IO interface
Quoted 146 lines · Jens Axboe
Quoted 122 lines · Jann Horn
Quoted 121 lines · Jens Axboe
Quoted 119 lines · Jens Axboe
Quoted 117 lines · Jann Horn
Quoted 48 lines · Jens Axboe
Quoted 47 lines · Jann Horn
Quoted 46 lines · Jens Axboe
The submission queue (SQ) and completion queue (CQ) rings are shared between the application and the kernel. This eliminates the need to copy data back and forth to submit and complete IO.
IO submissions use the
io_uring_sqedata structure, and completions are generated in the form ofio_uring_cqedata structures. The SQ ring is an index into theio_uring_sqearray, which makes it possible to submit a batch of IOs without them being contiguous in the ring. The CQ ring is always contiguous, as completion events are inherently unordered, and hence anyio_uring_cqeentry can point back to an arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up anio_uringinstance for doing async IO. On success, returns a file descriptor that the application can mmap to gain access to the SQ ring, CQ ring, andio_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for them to complete, or both. The behavior is controlled by the parameters passed in. If 'to_submit' is non-zero, then we'll try and submit new IO. IfIORING_ENTER_GETEVENTSis set, the kernel will wait for 'min_complete' events, if they aren't already available. It's valid to setIORING_ENTER_GETEVENTSand 'min_complete' == 0 at the same time, this allows the kernel to return already completed events without waiting for them. This is useful only for polling, as for IRQ driven IO, the application can just check the CQ ring without entering the kernel.With this setup, it's possible to do async IO with a single system call. Future developments will enable polled IO with this interface, and polled submission as well. The latter will enable an application to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for completions if it wants to wait for them to occur.
Each
io_uringis backed by a workqueue, to support buffered async IO as well. We will only punt to an async context if the command would need to wait for IO on the device side. Any data that can be accessed directly in the page cache is done inline. This avoids the slowness issue of usual threadpools, since cached data is accessed as quickly as a sync interface.[...]
Quoted 42 lines
Quoted 33 lines
Quoted 33 lines
(quoted diff) +33
-0+static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s) +{ + struct io_kiocb *req; + ssize_t ret; + + /* enforce forwards compatibility on users */ + if (unlikely(s->sqe->flags)) + return -EINVAL; + + req = io_get_req(ctx); + if (unlikely(!req)) + return -EAGAIN; + + req->rw.ki_filp = NULL; + + ret = __io_submit_sqe(ctx, req, s, true); + if (ret == -EAGAIN) { + memcpy(&req->submit, s, sizeof(*s)); + INIT_WORK(&req->work, io_sq_wq_submit_work); + queue_work(ctx->sqo_wq, &req->work); + ret = 0; + } + if (ret) + io_free_req(req); + + return ret; +} + +static void io_commit_sqring(struct io_ring_ctx *ctx) +{ + struct io_sq_ring *ring = ctx->sq_ring; + + if (ctx->cached_sq_head != ring->r.head) {Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ WRITE_ONCE(ring->r.head, ctx->cached_sq_head); + /* write side barrier of head update, app has read side */ + smp_wmb();Can you elaborate on what this memory barrier is doing? Don't you need some sort of memory barrier *before* the
WRITE_ONCE(), to ensure that nobody sees the updated head before you're done reading the submission queue entry? Or is that barrier elsewhere?Quoted 5 lines
The matching read barrier is in the application, it must do that before reading ->head for the SQ ring.
For the other barrier, since the
ring->r.headnow has aREAD_ONCE(), that should be all we need to ensure that loads are done.
READ_ONCE()/WRITE_ONCEare not hardware memory barriers that enforce ordering with regard to concurrent execution on other cores. They are only compiler barriers, influencing the order in which the compiler emits things. (Well, unless you're on alpha, whereREAD_ONCE()implies a memory barrier that prevents reordering of dependent reads.)As far as I can tell, between the
READ_ONCE(ring->array[...])inio_get_sqring()and theWRITE_ONCE()inio_commit_sqring(), you have no *hardware* memory barrier that prevents reordering against concurrently running userspace code. As far as I can tell, the following could happen:- The kernel reads from
ring->arrayinio_get_sqring(), then updates
the head inio_commit_sqring(). The CPU reorders the memory accesses such that the write to the head becomes visible before the read fromring->arrayhas completed.
- Userspace observes the write to the head and reuses the array slots
the kernel has freed with the write, clobberingring->arraybefore the kernel reads fromring->array.Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
I'd say this is highly theoretical for the normal use case, as we will have submitted IO in between. Hence the load must have been done.
Sorry, I'm confused. Who is "we", and which load are you referring to?
io_sq_thread()goes directly fromio_get_sqring()toio_commit_sqring(), with only a conditionalio_sqe_needs_user()in between, if thei == ARRAY_SIZE(sqes)check triggers. There is no "submitting IO" in the middle.You are right, the patch I sent IS needed for the sq thread case! It's only true for the "normal" case that we don't need the
smp_mb()before writing the sq ring head, as sqes are fully consumed at that point.Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
The only case that needs it is the sq thread case, since we bundle those up. This should do it:
Actually, I take that back, as in this particular case the sq thread is the only one that reads it.
What is "it"? The head pointer is written by the sq thread and read by userspace, not the other way around. Are you talking about
ring->array? Sorry, I'm lost.
Quoted 7 lines
I think we're on the same page, even if it doesn't necessarily sound like it. We do need the
smp_mb()before witingio_commit_sqring()for the thread case.I guess what confused me is that your commenting on the main patch, the case that needs it isn't introduced until later in the series. I'll fold the fix into that patch.
A better fix is to let the sq thread have the same behavior as the application driven path, simply committing the sq ring once we've consumed the sqes instead. That's just moving the io_sqring_commit() below io_submit_sqes().
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 05/19] Add io_uring IO interface
Quoted 134 lines · Jens Axboe
Quoted 132 lines · Jens Axboe
Quoted 130 lines · Jann Horn
Quoted 121 lines · Jens Axboe
Quoted 119 lines · Jens Axboe
Quoted 117 lines · Jann Horn
Quoted 48 lines · Jens Axboe
Quoted 47 lines · Jann Horn
Quoted 46 lines · Jens Axboe
The submission queue (SQ) and completion queue (CQ) rings are shared between the application and the kernel. This eliminates the need to copy data back and forth to submit and complete IO.
IO submissions use the
io_uring_sqedata structure, and completions are generated in the form ofio_uring_cqedata structures. The SQ ring is an index into theio_uring_sqearray, which makes it possible to submit a batch of IOs without them being contiguous in the ring. The CQ ring is always contiguous, as completion events are inherently unordered, and hence anyio_uring_cqeentry can point back to an arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up anio_uringinstance for doing async IO. On success, returns a file descriptor that the application can mmap to gain access to the SQ ring, CQ ring, andio_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for them to complete, or both. The behavior is controlled by the parameters passed in. If 'to_submit' is non-zero, then we'll try and submit new IO. IfIORING_ENTER_GETEVENTSis set, the kernel will wait for 'min_complete' events, if they aren't already available. It's valid to setIORING_ENTER_GETEVENTSand 'min_complete' == 0 at the same time, this allows the kernel to return already completed events without waiting for them. This is useful only for polling, as for IRQ driven IO, the application can just check the CQ ring without entering the kernel.With this setup, it's possible to do async IO with a single system call. Future developments will enable polled IO with this interface, and polled submission as well. The latter will enable an application to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for completions if it wants to wait for them to occur.
Each
io_uringis backed by a workqueue, to support buffered async IO as well. We will only punt to an async context if the command would need to wait for IO on the device side. Any data that can be accessed directly in the page cache is done inline. This avoids the slowness issue of usual threadpools, since cached data is accessed as quickly as a sync interface.[...]
Quoted 42 lines
Quoted 33 lines
Quoted 33 lines
(quoted diff) +33
-0+static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s) +{ + struct io_kiocb *req; + ssize_t ret; + + /* enforce forwards compatibility on users */ + if (unlikely(s->sqe->flags)) + return -EINVAL; + + req = io_get_req(ctx); + if (unlikely(!req)) + return -EAGAIN; + + req->rw.ki_filp = NULL; + + ret = __io_submit_sqe(ctx, req, s, true); + if (ret == -EAGAIN) { + memcpy(&req->submit, s, sizeof(*s)); + INIT_WORK(&req->work, io_sq_wq_submit_work); + queue_work(ctx->sqo_wq, &req->work); + ret = 0; + } + if (ret) + io_free_req(req); + + return ret; +} + +static void io_commit_sqring(struct io_ring_ctx *ctx) +{ + struct io_sq_ring *ring = ctx->sq_ring; + + if (ctx->cached_sq_head != ring->r.head) {Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ WRITE_ONCE(ring->r.head, ctx->cached_sq_head); + /* write side barrier of head update, app has read side */ + smp_wmb();Can you elaborate on what this memory barrier is doing? Don't you need some sort of memory barrier *before* the
WRITE_ONCE(), to ensure that nobody sees the updated head before you're done reading the submission queue entry? Or is that barrier elsewhere?Quoted 5 lines
The matching read barrier is in the application, it must do that before reading ->head for the SQ ring.
For the other barrier, since the
ring->r.headnow has aREAD_ONCE(), that should be all we need to ensure that loads are done.
READ_ONCE()/WRITE_ONCEare not hardware memory barriers that enforce ordering with regard to concurrent execution on other cores. They are only compiler barriers, influencing the order in which the compiler emits things. (Well, unless you're on alpha, whereREAD_ONCE()implies a memory barrier that prevents reordering of dependent reads.)As far as I can tell, between the
READ_ONCE(ring->array[...])inio_get_sqring()and theWRITE_ONCE()inio_commit_sqring(), you have no *hardware* memory barrier that prevents reordering against concurrently running userspace code. As far as I can tell, the following could happen:- The kernel reads from
ring->arrayinio_get_sqring(), then updates
the head inio_commit_sqring(). The CPU reorders the memory accesses such that the write to the head becomes visible before the read fromring->arrayhas completed.
- Userspace observes the write to the head and reuses the array slots
the kernel has freed with the write, clobberingring->arraybefore the kernel reads fromring->array.Quoted 2 lines
Quoted 2 lines
I'd say this is highly theoretical for the normal use case, as we will have submitted IO in between. Hence the load must have been done.
Sorry, I'm confused. Who is "we", and which load are you referring to?
io_sq_thread()goes directly fromio_get_sqring()toio_commit_sqring(), with only a conditionalio_sqe_needs_user()in between, if thei == ARRAY_SIZE(sqes)check triggers. There is no "submitting IO" in the middle.
Quoted 3 lines
Quoted 3 lines
You are right, the patch I sent IS needed for the sq thread case! It's only true for the "normal" case that we don't need the
smp_mb()before writing the sq ring head, as sqes are fully consumed at that point.
Hmm... does that actually matter? As long as you don't have an explicit barrier for this, the CPU could still reorder things, right? Pull the store in front of everything else?
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
Quoted 2 lines
The only case that needs it is the sq thread case, since we bundle those up. This should do it:
Actually, I take that back, as in this particular case the sq thread is the only one that reads it.
What is "it"? The head pointer is written by the sq thread and read by userspace, not the other way around. Are you talking about
ring->array? Sorry, I'm lost.
Quoted 6 lines
Quoted 6 lines
I think we're on the same page, even if it doesn't necessarily sound like it. We do need the
smp_mb()before witingio_commit_sqring()for the thread case.I guess what confused me is that your commenting on the main patch, the case that needs it isn't introduced until later in the series.
Ah, yes, I guess that was a bit confusing.
Quoted 5 lines
Quoted 1 line
I'll fold the fix into that patch.
A better fix is to let the sq thread have the same behavior as the application driven path, simply committing the sq ring once we've consumed the sqes instead. That's just moving the
io_sqring_commit()belowio_submit_sqes().
Hmm. How does that help?
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 05/19] Add io_uring IO interface
Quoted 135 lines · Jann Horn
Quoted 134 lines · Jens Axboe
Quoted 132 lines · Jens Axboe
Quoted 130 lines · Jann Horn
Quoted 121 lines · Jens Axboe
Quoted 119 lines · Jens Axboe
Quoted 117 lines · Jann Horn
Quoted 48 lines · Jens Axboe
Quoted 47 lines · Jann Horn
Quoted 46 lines · Jens Axboe
The submission queue (SQ) and completion queue (CQ) rings are shared between the application and the kernel. This eliminates the need to copy data back and forth to submit and complete IO.
IO submissions use the
io_uring_sqedata structure, and completions are generated in the form ofio_uring_cqedata structures. The SQ ring is an index into theio_uring_sqearray, which makes it possible to submit a batch of IOs without them being contiguous in the ring. The CQ ring is always contiguous, as completion events are inherently unordered, and hence anyio_uring_cqeentry can point back to an arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up anio_uringinstance for doing async IO. On success, returns a file descriptor that the application can mmap to gain access to the SQ ring, CQ ring, andio_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for them to complete, or both. The behavior is controlled by the parameters passed in. If 'to_submit' is non-zero, then we'll try and submit new IO. IfIORING_ENTER_GETEVENTSis set, the kernel will wait for 'min_complete' events, if they aren't already available. It's valid to setIORING_ENTER_GETEVENTSand 'min_complete' == 0 at the same time, this allows the kernel to return already completed events without waiting for them. This is useful only for polling, as for IRQ driven IO, the application can just check the CQ ring without entering the kernel.With this setup, it's possible to do async IO with a single system call. Future developments will enable polled IO with this interface, and polled submission as well. The latter will enable an application to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for completions if it wants to wait for them to occur.
Each
io_uringis backed by a workqueue, to support buffered async IO as well. We will only punt to an async context if the command would need to wait for IO on the device side. Any data that can be accessed directly in the page cache is done inline. This avoids the slowness issue of usual threadpools, since cached data is accessed as quickly as a sync interface.[...]
Quoted 42 lines
Quoted 33 lines
Quoted 33 lines
(quoted diff) +33
-0+static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s) +{ + struct io_kiocb *req; + ssize_t ret; + + /* enforce forwards compatibility on users */ + if (unlikely(s->sqe->flags)) + return -EINVAL; + + req = io_get_req(ctx); + if (unlikely(!req)) + return -EAGAIN; + + req->rw.ki_filp = NULL; + + ret = __io_submit_sqe(ctx, req, s, true); + if (ret == -EAGAIN) { + memcpy(&req->submit, s, sizeof(*s)); + INIT_WORK(&req->work, io_sq_wq_submit_work); + queue_work(ctx->sqo_wq, &req->work); + ret = 0; + } + if (ret) + io_free_req(req); + + return ret; +} + +static void io_commit_sqring(struct io_ring_ctx *ctx) +{ + struct io_sq_ring *ring = ctx->sq_ring; + + if (ctx->cached_sq_head != ring->r.head) {Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ WRITE_ONCE(ring->r.head, ctx->cached_sq_head); + /* write side barrier of head update, app has read side */ + smp_wmb();Can you elaborate on what this memory barrier is doing? Don't you need some sort of memory barrier *before* the
WRITE_ONCE(), to ensure that nobody sees the updated head before you're done reading the submission queue entry? Or is that barrier elsewhere?Quoted 5 lines
The matching read barrier is in the application, it must do that before reading ->head for the SQ ring.
For the other barrier, since the
ring->r.headnow has aREAD_ONCE(), that should be all we need to ensure that loads are done.
READ_ONCE()/WRITE_ONCEare not hardware memory barriers that enforce ordering with regard to concurrent execution on other cores. They are only compiler barriers, influencing the order in which the compiler emits things. (Well, unless you're on alpha, whereREAD_ONCE()implies a memory barrier that prevents reordering of dependent reads.)As far as I can tell, between the
READ_ONCE(ring->array[...])inio_get_sqring()and theWRITE_ONCE()inio_commit_sqring(), you have no *hardware* memory barrier that prevents reordering against concurrently running userspace code. As far as I can tell, the following could happen:- The kernel reads from
ring->arrayinio_get_sqring(), then updates
the head inio_commit_sqring(). The CPU reorders the memory accesses such that the write to the head becomes visible before the read fromring->arrayhas completed.
- Userspace observes the write to the head and reuses the array slots
the kernel has freed with the write, clobberingring->arraybefore the kernel reads fromring->array.Quoted 2 lines
Quoted 2 lines
I'd say this is highly theoretical for the normal use case, as we will have submitted IO in between. Hence the load must have been done.
Sorry, I'm confused. Who is "we", and which load are you referring to?
io_sq_thread()goes directly fromio_get_sqring()toio_commit_sqring(), with only a conditionalio_sqe_needs_user()in between, if thei == ARRAY_SIZE(sqes)check triggers. There is no "submitting IO" in the middle.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
You are right, the patch I sent IS needed for the sq thread case! It's only true for the "normal" case that we don't need the
smp_mb()before writing the sq ring head, as sqes are fully consumed at that point.Hmm... does that actually matter? As long as you don't have an explicit barrier for this, the CPU could still reorder things, right? Pull the store in front of everything else?
If the IO has been submitted, by definition the loads have completed. At that point it should be fine to commit the ring head that the application sees.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
I'll fold the fix into that patch.
A better fix is to let the sq thread have the same behavior as the application driven path, simply committing the sq ring once we've consumed the sqes instead. That's just moving the
io_sqring_commit()belowio_submit_sqes().Hmm. How does that help?
Because then it'll have submitted the IO, and hence loads from the sqes in question must have been done.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 05/19] Add io_uring IO interface
Quoted 141 lines · Jens Axboe
Quoted 139 lines · Jann Horn
Quoted 137 lines · Jens Axboe
Quoted 135 lines · Jens Axboe
Quoted 122 lines · Jann Horn
Quoted 121 lines · Jens Axboe
Quoted 119 lines · Jens Axboe
Quoted 117 lines · Jann Horn
Quoted 48 lines · Jens Axboe
Quoted 47 lines · Jann Horn
Quoted 46 lines · Jens Axboe
The submission queue (SQ) and completion queue (CQ) rings are shared between the application and the kernel. This eliminates the need to copy data back and forth to submit and complete IO.
IO submissions use the
io_uring_sqedata structure, and completions are generated in the form ofio_uring_cqedata structures. The SQ ring is an index into theio_uring_sqearray, which makes it possible to submit a batch of IOs without them being contiguous in the ring. The CQ ring is always contiguous, as completion events are inherently unordered, and hence anyio_uring_cqeentry can point back to an arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up anio_uringinstance for doing async IO. On success, returns a file descriptor that the application can mmap to gain access to the SQ ring, CQ ring, andio_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for them to complete, or both. The behavior is controlled by the parameters passed in. If 'to_submit' is non-zero, then we'll try and submit new IO. IfIORING_ENTER_GETEVENTSis set, the kernel will wait for 'min_complete' events, if they aren't already available. It's valid to setIORING_ENTER_GETEVENTSand 'min_complete' == 0 at the same time, this allows the kernel to return already completed events without waiting for them. This is useful only for polling, as for IRQ driven IO, the application can just check the CQ ring without entering the kernel.With this setup, it's possible to do async IO with a single system call. Future developments will enable polled IO with this interface, and polled submission as well. The latter will enable an application to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for completions if it wants to wait for them to occur.
Each
io_uringis backed by a workqueue, to support buffered async IO as well. We will only punt to an async context if the command would need to wait for IO on the device side. Any data that can be accessed directly in the page cache is done inline. This avoids the slowness issue of usual threadpools, since cached data is accessed as quickly as a sync interface.[...]
Quoted 42 lines
Quoted 33 lines
Quoted 33 lines
(quoted diff) +33
-0+static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s) +{ + struct io_kiocb *req; + ssize_t ret; + + /* enforce forwards compatibility on users */ + if (unlikely(s->sqe->flags)) + return -EINVAL; + + req = io_get_req(ctx); + if (unlikely(!req)) + return -EAGAIN; + + req->rw.ki_filp = NULL; + + ret = __io_submit_sqe(ctx, req, s, true); + if (ret == -EAGAIN) { + memcpy(&req->submit, s, sizeof(*s)); + INIT_WORK(&req->work, io_sq_wq_submit_work); + queue_work(ctx->sqo_wq, &req->work); + ret = 0; + } + if (ret) + io_free_req(req); + + return ret; +} + +static void io_commit_sqring(struct io_ring_ctx *ctx) +{ + struct io_sq_ring *ring = ctx->sq_ring; + + if (ctx->cached_sq_head != ring->r.head) {Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ WRITE_ONCE(ring->r.head, ctx->cached_sq_head); + /* write side barrier of head update, app has read side */ + smp_wmb();Can you elaborate on what this memory barrier is doing? Don't you need some sort of memory barrier *before* the
WRITE_ONCE(), to ensure that nobody sees the updated head before you're done reading the submission queue entry? Or is that barrier elsewhere?Quoted 5 lines
The matching read barrier is in the application, it must do that before reading ->head for the SQ ring.
For the other barrier, since the
ring->r.headnow has aREAD_ONCE(), that should be all we need to ensure that loads are done.
READ_ONCE()/WRITE_ONCEare not hardware memory barriers that enforce ordering with regard to concurrent execution on other cores. They are only compiler barriers, influencing the order in which the compiler emits things. (Well, unless you're on alpha, whereREAD_ONCE()implies a memory barrier that prevents reordering of dependent reads.)As far as I can tell, between the
READ_ONCE(ring->array[...])inio_get_sqring()and theWRITE_ONCE()inio_commit_sqring(), you have no *hardware* memory barrier that prevents reordering against concurrently running userspace code. As far as I can tell, the following could happen:- The kernel reads from
ring->arrayinio_get_sqring(), then updates
the head inio_commit_sqring(). The CPU reorders the memory accesses such that the write to the head becomes visible before the read fromring->arrayhas completed.
- Userspace observes the write to the head and reuses the array slots
the kernel has freed with the write, clobberingring->arraybefore the kernel reads fromring->array.Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
I'd say this is highly theoretical for the normal use case, as we will have submitted IO in between. Hence the load must have been done.
Sorry, I'm confused. Who is "we", and which load are you referring to?
io_sq_thread()goes directly fromio_get_sqring()toio_commit_sqring(), with only a conditionalio_sqe_needs_user()in between, if thei == ARRAY_SIZE(sqes)check triggers. There is no "submitting IO" in the middle.You are right, the patch I sent IS needed for the sq thread case! It's only true for the "normal" case that we don't need the
smp_mb()before writing the sq ring head, as sqes are fully consumed at that point.
Quoted 7 lines
Quoted 3 lines
Hmm... does that actually matter? As long as you don't have an explicit barrier for this, the CPU could still reorder things, right? Pull the store in front of everything else?
If the IO has been submitted, by definition the loads have completed. At that point it should be fine to commit the ring head that the application sees.
What exactly do you mean by "the IO has been submitted"? Are you talking about interaction with hardware, or about the end of the syscall, or something else?
Quoted 10 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
I'll fold the fix into that patch.
A better fix is to let the sq thread have the same behavior as the application driven path, simply committing the sq ring once we've consumed the sqes instead. That's just moving the
io_sqring_commit()belowio_submit_sqes().Hmm. How does that help?
Because then it'll have submitted the IO, and hence loads from the sqes in question must have been done.
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 05/19] Add io_uring IO interface
Quoted 146 lines · Jann Horn
Quoted 145 lines · Jens Axboe
Quoted 142 lines · Jann Horn
Quoted 134 lines · Jens Axboe
Quoted 132 lines · Jens Axboe
Quoted 130 lines · Jann Horn
Quoted 121 lines · Jens Axboe
Quoted 119 lines · Jens Axboe
Quoted 117 lines · Jann Horn
Quoted 48 lines · Jens Axboe
Quoted 47 lines · Jann Horn
Quoted 46 lines · Jens Axboe
The submission queue (SQ) and completion queue (CQ) rings are shared between the application and the kernel. This eliminates the need to copy data back and forth to submit and complete IO.
IO submissions use the
io_uring_sqedata structure, and completions are generated in the form ofio_uring_cqedata structures. The SQ ring is an index into theio_uring_sqearray, which makes it possible to submit a batch of IOs without them being contiguous in the ring. The CQ ring is always contiguous, as completion events are inherently unordered, and hence anyio_uring_cqeentry can point back to an arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up anio_uringinstance for doing async IO. On success, returns a file descriptor that the application can mmap to gain access to the SQ ring, CQ ring, andio_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for them to complete, or both. The behavior is controlled by the parameters passed in. If 'to_submit' is non-zero, then we'll try and submit new IO. IfIORING_ENTER_GETEVENTSis set, the kernel will wait for 'min_complete' events, if they aren't already available. It's valid to setIORING_ENTER_GETEVENTSand 'min_complete' == 0 at the same time, this allows the kernel to return already completed events without waiting for them. This is useful only for polling, as for IRQ driven IO, the application can just check the CQ ring without entering the kernel.With this setup, it's possible to do async IO with a single system call. Future developments will enable polled IO with this interface, and polled submission as well. The latter will enable an application to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for completions if it wants to wait for them to occur.
Each
io_uringis backed by a workqueue, to support buffered async IO as well. We will only punt to an async context if the command would need to wait for IO on the device side. Any data that can be accessed directly in the page cache is done inline. This avoids the slowness issue of usual threadpools, since cached data is accessed as quickly as a sync interface.[...]
Quoted 42 lines
Quoted 33 lines
Quoted 33 lines
(quoted diff) +33
-0+static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s) +{ + struct io_kiocb *req; + ssize_t ret; + + /* enforce forwards compatibility on users */ + if (unlikely(s->sqe->flags)) + return -EINVAL; + + req = io_get_req(ctx); + if (unlikely(!req)) + return -EAGAIN; + + req->rw.ki_filp = NULL; + + ret = __io_submit_sqe(ctx, req, s, true); + if (ret == -EAGAIN) { + memcpy(&req->submit, s, sizeof(*s)); + INIT_WORK(&req->work, io_sq_wq_submit_work); + queue_work(ctx->sqo_wq, &req->work); + ret = 0; + } + if (ret) + io_free_req(req); + + return ret; +} + +static void io_commit_sqring(struct io_ring_ctx *ctx) +{ + struct io_sq_ring *ring = ctx->sq_ring; + + if (ctx->cached_sq_head != ring->r.head) {Quoted 8 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ WRITE_ONCE(ring->r.head, ctx->cached_sq_head); + /* write side barrier of head update, app has read side */ + smp_wmb();Can you elaborate on what this memory barrier is doing? Don't you need some sort of memory barrier *before* the
WRITE_ONCE(), to ensure that nobody sees the updated head before you're done reading the submission queue entry? Or is that barrier elsewhere?Quoted 5 lines
The matching read barrier is in the application, it must do that before reading ->head for the SQ ring.
For the other barrier, since the
ring->r.headnow has aREAD_ONCE(), that should be all we need to ensure that loads are done.
READ_ONCE()/WRITE_ONCEare not hardware memory barriers that enforce ordering with regard to concurrent execution on other cores. They are only compiler barriers, influencing the order in which the compiler emits things. (Well, unless you're on alpha, whereREAD_ONCE()implies a memory barrier that prevents reordering of dependent reads.)As far as I can tell, between the
READ_ONCE(ring->array[...])inio_get_sqring()and theWRITE_ONCE()inio_commit_sqring(), you have no *hardware* memory barrier that prevents reordering against concurrently running userspace code. As far as I can tell, the following could happen:- The kernel reads from
ring->arrayinio_get_sqring(), then updates
the head inio_commit_sqring(). The CPU reorders the memory accesses such that the write to the head becomes visible before the read fromring->arrayhas completed.
- Userspace observes the write to the head and reuses the array slots
the kernel has freed with the write, clobberingring->arraybefore the kernel reads fromring->array.Quoted 2 lines
Quoted 2 lines
I'd say this is highly theoretical for the normal use case, as we will have submitted IO in between. Hence the load must have been done.
Sorry, I'm confused. Who is "we", and which load are you referring to?
io_sq_thread()goes directly fromio_get_sqring()toio_commit_sqring(), with only a conditionalio_sqe_needs_user()in between, if thei == ARRAY_SIZE(sqes)check triggers. There is no "submitting IO" in the middle.Quoted 3 lines
Quoted 3 lines
You are right, the patch I sent IS needed for the sq thread case! It's only true for the "normal" case that we don't need the
smp_mb()before writing the sq ring head, as sqes are fully consumed at that point.Hmm... does that actually matter? As long as you don't have an explicit barrier for this, the CPU could still reorder things, right? Pull the store in front of everything else?
Quoted 7 lines
Quoted 3 lines
If the IO has been submitted, by definition the loads have completed. At that point it should be fine to commit the ring head that the application sees.
What exactly do you mean by "the IO has been submitted"? Are you talking about interaction with hardware, or about the end of the syscall, or something else?
I mean that the loads from the sqe, which the IO is made of, have been done. That's what we care about here, right? The sqe has either been turned into an io request and has been submitted, or it has been copied.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 05/19] Add io_uring IO interface
Quoted 152 lines · Jens Axboe
Quoted 150 lines · Jann Horn
Quoted 148 lines · Jens Axboe
Quoted 135 lines · Jann Horn
Quoted 134 lines · Jens Axboe
Quoted 132 lines · Jens Axboe
Quoted 130 lines · Jann Horn
Quoted 121 lines · Jens Axboe
Quoted 119 lines · Jens Axboe
Quoted 117 lines · Jann Horn
Quoted 48 lines · Jens Axboe
Quoted 47 lines · Jann Horn
Quoted 46 lines · Jens Axboe
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.[...]
Quoted 42 lines
Quoted 33 lines
Quoted 33 lines
+static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s)
+{
+ struct io_kiocb *req;
+ ssize_t ret;
+
+ /* enforce forwards compatibility on users */
+ if (unlikely(s->sqe->flags))
+ return -EINVAL;
+
+ req = io_get_req(ctx);
+ if (unlikely(!req))
+ return -EAGAIN;
+
+ req->rw.ki_filp = NULL;
+
+ ret = __io_submit_sqe(ctx, req, s, true);
+ if (ret == -EAGAIN) {
+ memcpy(&req->submit, s, sizeof(*s));
+ INIT_WORK(&req->work, io_sq_wq_submit_work);
+ queue_work(ctx->sqo_wq, &req->work);
+ ret = 0;
+ }
+ if (ret)
+ io_free_req(req);
+
+ return ret;
+}
+
+static void io_commit_sqring(struct io_ring_ctx *ctx)
+{
+ struct io_sq_ring *ring = ctx->sq_ring;
+
+ if (ctx->cached_sq_head != ring->r.head) {Quoted 8 lines
Quoted 3 lines
+ WRITE_ONCE(ring->r.head, ctx->cached_sq_head);
+ /* write side barrier of head update, app has read side */
+ smp_wmb();Can you elaborate on what this memory barrier is doing? Don't you need some sort of memory barrier *before* the
WRITE_ONCE(), to ensure that nobody sees the updated head before you're done reading the submission queue entry? Or is that barrier elsewhere?Quoted 5 lines
The matching read barrier is in the application, it must do that before reading ->head for the SQ ring.
For the other barrier, since the
ring->r.headnow has aREAD_ONCE(), that should be all we need to ensure that loads are done.
READ_ONCE()/WRITE_ONCEare not hardware memory barriers that enforce ordering with regard to concurrent execution on other cores. They are only compiler barriers, influencing the order in which the compiler emits things. (Well, unless you're on alpha, whereREAD_ONCE()implies a memory barrier that prevents reordering of dependent reads.)As far as I can tell, between the
READ_ONCE(ring->array[...])inio_get_sqring()and theWRITE_ONCE()inio_commit_sqring(), you have no *hardware* memory barrier that prevents reordering against concurrently running userspace code. As far as I can tell, the following could happen:- The kernel reads from
ring->arrayinio_get_sqring(), then updates
the head inio_commit_sqring(). The CPU reorders the memory accesses such that the write to the head becomes visible before the read fromring->arrayhas completed.
- Userspace observes the write to the head and reuses the array slots
the kernel has freed with the write, clobberingring->arraybefore the kernel reads fromring->array.Quoted 2 lines
Quoted 2 lines
I'd say this is highly theoretical for the normal use case, as we will have submitted IO in between. Hence the load must have been done.
Sorry, I'm confused. Who is "we", and which load are you referring to?
io_sq_thread()goes directly fromio_get_sqring()toio_commit_sqring(), with only a conditionalio_sqe_needs_user()in between, if thei == ARRAY_SIZE(sqes)check triggers. There is no "submitting IO" in the middle.Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
You are right, the patch I sent IS needed for the sq thread case! It's only true for the "normal" case that we don't need the
smp_mb()before writing the sq ring head, as sqes are fully consumed at that point.Hmm... does that actually matter? As long as you don't have an explicit barrier for this, the CPU could still reorder things, right? Pull the store in front of everything else?
If the IO has been submitted, by definition the loads have completed. At that point it should be fine to commit the ring head that the application sees.
Quoted 7 lines
Quoted 3 lines
What exactly do you mean by "the IO has been submitted"? Are you talking about interaction with hardware, or about the end of the syscall, or something else?
I mean that the loads from the sqe, which the IO is made of, have been done. That's what we care about here, right? The sqe has either been turned into an io request and has been submitted, or it has been copied.
But they might not actually be done. AFAIU the CPU is allowed to do the WRITE_ONCE of the head before doing any of the reads from the sqe
- loads and stores you do, as observed by a concurrently executing thread, can happen in an order independent of the order in which you write them in your code unless you use memory barriers. So the CPU might decide to first write the new head, then do the read for io_get_sqring(), and then do the __io_submit_sqe(), potentially reading e.g. a IORING_OP_NOP opcode that has been written by concurrently executing userspace after userspace has observed the bumped head.
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 05/19] Add io_uring IO interface
Quoted 161 lines · Jann Horn
Quoted 152 lines · Jens Axboe
Quoted 150 lines · Jann Horn
Quoted 148 lines · Jens Axboe
Quoted 135 lines · Jann Horn
Quoted 134 lines · Jens Axboe
Quoted 132 lines · Jens Axboe
Quoted 130 lines · Jann Horn
Quoted 121 lines · Jens Axboe
Quoted 119 lines · Jens Axboe
Quoted 117 lines · Jann Horn
Quoted 48 lines · Jens Axboe
Quoted 47 lines · Jann Horn
On Fri, Feb 8, 2019 at 6:34 PM Jens Axboe <[email protected]> wrote:
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.[...]
Quoted 42 lines
Quoted 41 lines
+static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s)
+{
+ struct io_kiocb *req;
+ ssize_t ret;
+
+ /* enforce forwards compatibility on users */
+ if (unlikely(s->sqe->flags))
+ return -EINVAL;
+
+ req = io_get_req(ctx);
+ if (unlikely(!req))
+ return -EAGAIN;
+
+ req->rw.ki_filp = NULL;
+
+ ret = __io_submit_sqe(ctx, req, s, true);
+ if (ret == -EAGAIN) {
+ memcpy(&req->submit, s, sizeof(*s));
+ INIT_WORK(&req->work, io_sq_wq_submit_work);
+ queue_work(ctx->sqo_wq, &req->work);
+ ret = 0;
+ }
+ if (ret)
+ io_free_req(req);
+
+ return ret;
+}
+
+static void io_commit_sqring(struct io_ring_ctx *ctx)
+{
+ struct io_sq_ring *ring = ctx->sq_ring;
+
+ if (ctx->cached_sq_head != ring->r.head) {
+ WRITE_ONCE(ring->r.head, ctx->cached_sq_head);
+ /* write side barrier of head update, app has read side */
+ smp_wmb();Can you elaborate on what this memory barrier is doing? Don't you need
some sort of memory barrier *before* the WRITE_ONCE(), to ensure that
nobody sees the updated head before you're done reading the submission
queue entry? Or is that barrier elsewhere?Quoted 5 lines
The matching read barrier is in the application, it must do that before reading ->head for the SQ ring.
For the other barrier, since the
ring->r.headnow has aREAD_ONCE(), that should be all we need to ensure that loads are done.
READ_ONCE()/WRITE_ONCEare not hardware memory barriers that enforce ordering with regard to concurrent execution on other cores. They are only compiler barriers, influencing the order in which the compiler emits things. (Well, unless you're on alpha, whereREAD_ONCE()implies a memory barrier that prevents reordering of dependent reads.)As far as I can tell, between the
READ_ONCE(ring->array[...])inio_get_sqring()and theWRITE_ONCE()inio_commit_sqring(), you have no *hardware* memory barrier that prevents reordering against concurrently running userspace code. As far as I can tell, the following could happen:- The kernel reads from
ring->arrayinio_get_sqring(), then updates
the head inio_commit_sqring(). The CPU reorders the memory accesses such that the write to the head becomes visible before the read fromring->arrayhas completed.
- Userspace observes the write to the head and reuses the array slots
the kernel has freed with the write, clobberingring->arraybefore the kernel reads fromring->array.Quoted 2 lines
Quoted 2 lines
I'd say this is highly theoretical for the normal use case, as we will have submitted IO in between. Hence the load must have been done.
Sorry, I'm confused. Who is "we", and which load are you referring to?
io_sq_thread()goes directly fromio_get_sqring()toio_commit_sqring(), with only a conditionalio_sqe_needs_user()in between, if thei == ARRAY_SIZE(sqes)check triggers. There is no "submitting IO" in the middle.Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
You are right, the patch I sent IS needed for the sq thread case! It's only true for the "normal" case that we don't need the
smp_mb()before writing the sq ring head, as sqes are fully consumed at that point.Hmm... does that actually matter? As long as you don't have an explicit barrier for this, the CPU could still reorder things, right? Pull the store in front of everything else?
If the IO has been submitted, by definition the loads have completed. At that point it should be fine to commit the ring head that the application sees.
Quoted 7 lines
Quoted 3 lines
What exactly do you mean by "the IO has been submitted"? Are you talking about interaction with hardware, or about the end of the syscall, or something else?
I mean that the loads from the sqe, which the IO is made of, have been done. That's what we care about here, right? The sqe has either been turned into an io request and has been submitted, or it has been copied.
Quoted 10 lines
But they might not actually be done. AFAIU the CPU is allowed to do the
WRITE_ONCEof the head before doing any of the reads from the sqe
- loads and stores you do, as observed by a concurrently executing thread, can happen in an order independent of the order in which you write them in your code unless you use memory barriers. So the CPU might decide to first write the new head, then do the read forio_get_sqring(), and then do the__io_submit_sqe(), potentially reading e.g. aIORING_OP_NOPopcode that has been written by concurrently executing userspace after userspace has observed the bumped head.
For that to be possible, we'd need NO ordering in between the IO submission and when we write the sq ring head. A single spin lock should do it, right?
It's not that I'm set against adding an smp_mb() to io_commit_sqring(), but I think we're going off the deep end a little bit here on theoretical vs what can practically happen.
For the regular IO cases, we will have done at least one lock/unlock cycle. This is true for nops as well, and poll. The only case that could potentially NOT have one is the fsync, for the case where we punt and don't add it to existing work, we don't have any locking in between.
I'll add the smp_mb() for peace of mind.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 05/19] Add io_uring IO interface
Quoted 181 lines · Jens Axboe
Quoted 161 lines · Jann Horn
Quoted 152 lines · Jens Axboe
Quoted 150 lines · Jann Horn
Quoted 148 lines · Jens Axboe
Quoted 135 lines · Jann Horn
Quoted 134 lines · Jens Axboe
Quoted 132 lines · Jens Axboe
Quoted 130 lines · Jann Horn
Quoted 121 lines · Jens Axboe
Quoted 119 lines · Jens Axboe
Quoted 117 lines · Jann Horn
Quoted 48 lines · Jens Axboe
On 2/8/19 3:12 PM, Jann Horn wrote:
On Fri, Feb 8, 2019 at 6:34 PM Jens Axboe <[email protected]> wrote:
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.[...]
Quoted 47 lines
+static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s)
+{
+ struct io_kiocb *req;
+ ssize_t ret;
+
+ /* enforce forwards compatibility on users */
+ if (unlikely(s->sqe->flags))
+ return -EINVAL;
+
+ req = io_get_req(ctx);
+ if (unlikely(!req))
+ return -EAGAIN;
+
+ req->rw.ki_filp = NULL;
+
+ ret = __io_submit_sqe(ctx, req, s, true);
+ if (ret == -EAGAIN) {
+ memcpy(&req->submit, s, sizeof(*s));
+ INIT_WORK(&req->work, io_sq_wq_submit_work);
+ queue_work(ctx->sqo_wq, &req->work);
+ ret = 0;
+ }
+ if (ret)
+ io_free_req(req);
+
+ return ret;
+}
+
+static void io_commit_sqring(struct io_ring_ctx *ctx)
+{
+ struct io_sq_ring *ring = ctx->sq_ring;
+
+ if (ctx->cached_sq_head != ring->r.head) {
+ WRITE_ONCE(ring->r.head, ctx->cached_sq_head);
+ /* write side barrier of head update, app has read side */
+ smp_wmb();Can you elaborate on what this memory barrier is doing? Don't you need
some sort of memory barrier *before* the WRITE_ONCE(), to ensure that
nobody sees the updated head before you're done reading the submission
queue entry? Or is that barrier elsewhere?The matching read barrier is in the application, it must do that before
reading ->head for the SQ ring.For the other barrier, since the ring->r.head now has a READ_ONCE(),
that should be all we need to ensure that loads are done.
READ_ONCE()/WRITE_ONCEare not hardware memory barriers that enforce ordering with regard to concurrent execution on other cores. They are only compiler barriers, influencing the order in which the compiler emits things. (Well, unless you're on alpha, whereREAD_ONCE()implies a memory barrier that prevents reordering of dependent reads.)As far as I can tell, between the
READ_ONCE(ring->array[...])inio_get_sqring()and theWRITE_ONCE()inio_commit_sqring(), you have no *hardware* memory barrier that prevents reordering against concurrently running userspace code. As far as I can tell, the following could happen:- The kernel reads from
ring->arrayinio_get_sqring(), then updates
the head inio_commit_sqring(). The CPU reorders the memory accesses such that the write to the head becomes visible before the read fromring->arrayhas completed.
- Userspace observes the write to the head and reuses the array slots
the kernel has freed with the write, clobberingring->arraybefore the kernel reads fromring->array.Quoted 2 lines
Quoted 2 lines
I'd say this is highly theoretical for the normal use case, as we will have submitted IO in between. Hence the load must have been done.
Sorry, I'm confused. Who is "we", and which load are you referring to?
io_sq_thread()goes directly fromio_get_sqring()toio_commit_sqring(), with only a conditionalio_sqe_needs_user()in between, if thei == ARRAY_SIZE(sqes)check triggers. There is no "submitting IO" in the middle.Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
You are right, the patch I sent IS needed for the sq thread case! It's only true for the "normal" case that we don't need the
smp_mb()before writing the sq ring head, as sqes are fully consumed at that point.Hmm... does that actually matter? As long as you don't have an explicit barrier for this, the CPU could still reorder things, right? Pull the store in front of everything else?
If the IO has been submitted, by definition the loads have completed. At that point it should be fine to commit the ring head that the application sees.
Quoted 7 lines
Quoted 3 lines
What exactly do you mean by "the IO has been submitted"? Are you talking about interaction with hardware, or about the end of the syscall, or something else?
I mean that the loads from the sqe, which the IO is made of, have been done. That's what we care about here, right? The sqe has either been turned into an io request and has been submitted, or it has been copied.
Quoted 10 lines
But they might not actually be done. AFAIU the CPU is allowed to do the
WRITE_ONCEof the head before doing any of the reads from the sqe
- loads and stores you do, as observed by a concurrently executing thread, can happen in an order independent of the order in which you write them in your code unless you use memory barriers. So the CPU might decide to first write the new head, then do the read forio_get_sqring(), and then do the__io_submit_sqe(), potentially reading e.g. aIORING_OP_NOPopcode that has been written by concurrently executing userspace after userspace has observed the bumped head.For that to be possible, we'd need NO ordering in between the IO submission and when we write the sq ring head. A single spin lock should do it, right?
It's not that I'm set against adding an
smp_mb()toio_commit_sqring(), but I think we're going off the deep end a little bit here on theoretical vs what can practically happen.
Quoted 6 lines
For the regular IO cases, we will have done at least one lock/unlock cycle. This is true for nops as well, and poll. The only case that could potentially NOT have one is the fsync, for the case where we punt and don't add it to existing work, we don't have any locking in between.
I'll add the
smp_mb()for peace of mind.
For reference, folded in:
fs/io_uring.c +7 -0
@@ -1690,6 +1690,13 @@ static void io_commit_sqring(struct io_ring_ctx *ctx)struct io_sq_ring *ring = ctx->sq_ring;if (ctx->cached_sq_head !=READ_ONCE (ring->r.head)) { +/* +* Ensure any loads from the SQEs are done at this point, +* since once we write the new head, the application could +* write new data to them. +*/ +smp_mb (); +WRITE_ONCE (ring->r.head, ctx->cached_sq_head);/* * write side barrier of head update, app has read side. See
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 05/19] Add io_uring IO interface
Quoted 206 lines · Jens Axboe
Quoted 183 lines · Jens Axboe
Quoted 162 lines · Jann Horn
Quoted 153 lines · Jens Axboe
Quoted 152 lines · Jann Horn
Quoted 149 lines · Jens Axboe
Quoted 137 lines · Jann Horn
Quoted 135 lines · Jens Axboe
Quoted 134 lines · Jens Axboe
Quoted 131 lines · Jann Horn
Quoted 122 lines · Jens Axboe
Quoted 121 lines · Jens Axboe
Quoted 118 lines · Jann Horn
On Sat, Feb 9, 2019 at 5:15 AM Jens Axboe <[email protected]> wrote:
On 2/8/19 3:12 PM, Jann Horn wrote:
On Fri, Feb 8, 2019 at 6:34 PM Jens Axboe <[email protected]> wrote:
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
[...]
+static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s)
+{
+ struct io_kiocb *req;
+ ssize_t ret;
+
+ /* enforce forwards compatibility on users */
+ if (unlikely(s->sqe->flags))
+ return -EINVAL;
+
+ req = io_get_req(ctx);
+ if (unlikely(!req))
+ return -EAGAIN;
+
+ req->rw.ki_filp = NULL;
+
+ ret = __io_submit_sqe(ctx, req, s, true);
+ if (ret == -EAGAIN) {
+ memcpy(&req->submit, s, sizeof(*s));
+ INIT_WORK(&req->work, io_sq_wq_submit_work);
+ queue_work(ctx->sqo_wq, &req->work);
+ ret = 0;
+ }
+ if (ret)
+ io_free_req(req);
+
+ return ret;
+}
+
+static void io_commit_sqring(struct io_ring_ctx *ctx)
+{
+ struct io_sq_ring *ring = ctx->sq_ring;
+
+ if (ctx->cached_sq_head != ring->r.head) {
+ WRITE_ONCE(ring->r.head, ctx->cached_sq_head);
+ /* write side barrier of head update, app has read side */
+ smp_wmb();Can you elaborate on what this memory barrier is doing? Don't you need
some sort of memory barrier *before* the WRITE_ONCE(), to ensure that
nobody sees the updated head before you're done reading the submission
queue entry? Or is that barrier elsewhere?The matching read barrier is in the application, it must do that before
reading ->head for the SQ ring.For the other barrier, since the ring->r.head now has a READ_ONCE(),
that should be all we need to ensure that loads are done.READ_ONCE() / WRITE_ONCE are not hardware memory barriers that enforce
ordering with regard to concurrent execution on other cores. They are
only compiler barriers, influencing the order in which the compiler
emits things. (Well, unless you're on alpha, where READ_ONCE() implies
a memory barrier that prevents reordering of dependent reads.)As far as I can tell, between the READ_ONCE(ring->array[...]) in
io_get_sqring() and the WRITE_ONCE() in io_commit_sqring(), you have
no *hardware* memory barrier that prevents reordering against
concurrently running userspace code. As far as I can tell, the
following could happen:- The kernel reads from ring->array in io_get_sqring(), then updates
the head in io_commit_sqring(). The CPU reorders the memory accesses
such that the write to the head becomes visible before the read from
ring->array has completed.
- Userspace observes the write to the head and reuses the array slots
the kernel has freed with the write, clobbering ring->array before the
kernel reads from ring->array.Quoted 2 lines
Quoted 2 lines
I'd say this is highly theoretical for the normal use case, as we will have submitted IO in between. Hence the load must have been done.
Sorry, I'm confused. Who is "we", and which load are you referring to?
io_sq_thread()goes directly fromio_get_sqring()toio_commit_sqring(), with only a conditionalio_sqe_needs_user()in between, if thei == ARRAY_SIZE(sqes)check triggers. There is no "submitting IO" in the middle.Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
You are right, the patch I sent IS needed for the sq thread case! It's only true for the "normal" case that we don't need the
smp_mb()before writing the sq ring head, as sqes are fully consumed at that point.Hmm... does that actually matter? As long as you don't have an explicit barrier for this, the CPU could still reorder things, right? Pull the store in front of everything else?
If the IO has been submitted, by definition the loads have completed. At that point it should be fine to commit the ring head that the application sees.
Quoted 7 lines
Quoted 3 lines
What exactly do you mean by "the IO has been submitted"? Are you talking about interaction with hardware, or about the end of the syscall, or something else?
I mean that the loads from the sqe, which the IO is made of, have been done. That's what we care about here, right? The sqe has either been turned into an io request and has been submitted, or it has been copied.
Quoted 10 lines
But they might not actually be done. AFAIU the CPU is allowed to do the
WRITE_ONCEof the head before doing any of the reads from the sqe
- loads and stores you do, as observed by a concurrently executing thread, can happen in an order independent of the order in which you write them in your code unless you use memory barriers. So the CPU might decide to first write the new head, then do the read forio_get_sqring(), and then do the__io_submit_sqe(), potentially reading e.g. aIORING_OP_NOPopcode that has been written by concurrently executing userspace after userspace has observed the bumped head.For that to be possible, we'd need NO ordering in between the IO submission and when we write the sq ring head. A single spin lock should do it, right?
It's not that I'm set against adding an
smp_mb()toio_commit_sqring(), but I think we're going off the deep end a little bit here on theoretical vs what can practically happen.Quoted 6 lines
For the regular IO cases, we will have done at least one lock/unlock cycle. This is true for nops as well, and poll. The only case that could potentially NOT have one is the fsync, for the case where we punt and don't add it to existing work, we don't have any locking in between.
I'll add the
smp_mb()for peace of mind.For reference, folded in:
fs/io_uring.c +0
-0@@ -1690,6 +1690,13 @@ static void io_commit_sqring(struct io_ring_ctx *ctx)struct io_sq_ring *ring = ctx->sq_ring;if (ctx->cached_sq_head !=READ_ONCE (ring->r.head)) {
fs/io_uring.c static void io_commit_sqring(struct io_ring_ctx *ctx) · line 1702
Quoted 10 lines
fs/io_uring.c +7
-0+/* +* Ensure any loads from the SQEs are done at this point, +* since once we write the new head, the application could +* write new data to them. +*/ +smp_mb (); +WRITE_ONCE (ring->r.head, ctx->cached_sq_head);/* * write side barrier of head update, app has read side. See
I haven’t followed the full set of machinations here, but would smp_store_release() be sufficient? It is a *lot* faster on some architectures. --
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 05/19] Add io_uring IO interface
Quoted 196 lines · Andy Lutomirski
Quoted 196 lines · Jens Axboe
Quoted 178 lines · Jens Axboe
Quoted 158 lines · Jann Horn
Quoted 149 lines · Jens Axboe
Quoted 148 lines · Jann Horn
Quoted 146 lines · Jens Axboe
Quoted 134 lines · Jann Horn
Quoted 133 lines · Jens Axboe
Quoted 132 lines · Jens Axboe
Quoted 130 lines · Jann Horn
Quoted 121 lines · Jens Axboe
Quoted 120 lines · Jens Axboe
On 2/12/19 2:42 PM, Jann Horn wrote:
On Sat, Feb 9, 2019 at 5:15 AM Jens Axboe <[email protected]> wrote:
On 2/8/19 3:12 PM, Jann Horn wrote:
On Fri, Feb 8, 2019 at 6:34 PM Jens Axboe <[email protected]> wrote:
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
[...]
+static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s)
+{
+ struct io_kiocb *req;
+ ssize_t ret;
+
+ /* enforce forwards compatibility on users */
+ if (unlikely(s->sqe->flags))
+ return -EINVAL;
+
+ req = io_get_req(ctx);
+ if (unlikely(!req))
+ return -EAGAIN;
+
+ req->rw.ki_filp = NULL;
+
+ ret = __io_submit_sqe(ctx, req, s, true);
+ if (ret == -EAGAIN) {
+ memcpy(&req->submit, s, sizeof(*s));
+ INIT_WORK(&req->work, io_sq_wq_submit_work);
+ queue_work(ctx->sqo_wq, &req->work);
+ ret = 0;
+ }
+ if (ret)
+ io_free_req(req);
+
+ return ret;
+}
+
+static void io_commit_sqring(struct io_ring_ctx *ctx)
+{
+ struct io_sq_ring *ring = ctx->sq_ring;
+
+ if (ctx->cached_sq_head != ring->r.head) {
+ WRITE_ONCE(ring->r.head, ctx->cached_sq_head);
+ /* write side barrier of head update, app has read side */
+ smp_wmb();Can you elaborate on what this memory barrier is doing? Don't you need
some sort of memory barrier *before* the WRITE_ONCE(), to ensure that
nobody sees the updated head before you're done reading the submission
queue entry? Or is that barrier elsewhere?The matching read barrier is in the application, it must do that before
reading ->head for the SQ ring.For the other barrier, since the ring->r.head now has a READ_ONCE(),
that should be all we need to ensure that loads are done.READ_ONCE() / WRITE_ONCE are not hardware memory barriers that enforce
ordering with regard to concurrent execution on other cores. They are
only compiler barriers, influencing the order in which the compiler
emits things. (Well, unless you're on alpha, where READ_ONCE() implies
a memory barrier that prevents reordering of dependent reads.)As far as I can tell, between the READ_ONCE(ring->array[...]) in
io_get_sqring() and the WRITE_ONCE() in io_commit_sqring(), you have
no *hardware* memory barrier that prevents reordering against
concurrently running userspace code. As far as I can tell, the
following could happen:- The kernel reads from ring->array in io_get_sqring(), then updates
the head in io_commit_sqring(). The CPU reorders the memory accesses
such that the write to the head becomes visible before the read from
ring->array has completed.
- Userspace observes the write to the head and reuses the array slots
the kernel has freed with the write, clobbering ring->array before the
kernel reads from ring->array.Quoted 2 lines
Quoted 2 lines
I'd say this is highly theoretical for the normal use case, as we
will have submitted IO in between. Hence the load must have been done.Sorry, I'm confused. Who is "we", and which load are you referring to?
io_sq_thread()goes directly fromio_get_sqring()toio_commit_sqring(), with only a conditionalio_sqe_needs_user()in between, if thei == ARRAY_SIZE(sqes)check triggers. There is no "submitting IO" in the middle.Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
You are right, the patch I sent IS needed for the sq thread case! It's only true for the "normal" case that we don't need the
smp_mb()before writing the sq ring head, as sqes are fully consumed at that point.Hmm... does that actually matter? As long as you don't have an explicit barrier for this, the CPU could still reorder things, right? Pull the store in front of everything else?
If the IO has been submitted, by definition the loads have completed. At that point it should be fine to commit the ring head that the application sees.
Quoted 7 lines
Quoted 3 lines
What exactly do you mean by "the IO has been submitted"? Are you talking about interaction with hardware, or about the end of the syscall, or something else?
I mean that the loads from the sqe, which the IO is made of, have been done. That's what we care about here, right? The sqe has either been turned into an io request and has been submitted, or it has been copied.
Quoted 10 lines
But they might not actually be done. AFAIU the CPU is allowed to do the
WRITE_ONCEof the head before doing any of the reads from the sqe
- loads and stores you do, as observed by a concurrently executing thread, can happen in an order independent of the order in which you write them in your code unless you use memory barriers. So the CPU might decide to first write the new head, then do the read forio_get_sqring(), and then do the__io_submit_sqe(), potentially reading e.g. aIORING_OP_NOPopcode that has been written by concurrently executing userspace after userspace has observed the bumped head.For that to be possible, we'd need NO ordering in between the IO submission and when we write the sq ring head. A single spin lock should do it, right?
It's not that I'm set against adding an
smp_mb()toio_commit_sqring(), but I think we're going off the deep end a little bit here on theoretical vs what can practically happen.Quoted 6 lines
For the regular IO cases, we will have done at least one lock/unlock cycle. This is true for nops as well, and poll. The only case that could potentially NOT have one is the fsync, for the case where we punt and don't add it to existing work, we don't have any locking in between.
I'll add the
smp_mb()for peace of mind.For reference, folded in:
fs/io_uring.c +0
-0@@ -1690,6 +1690,13 @@ static void io_commit_sqring(struct io_ring_ctx *ctx)struct io_sq_ring *ring = ctx->sq_ring;
Quoted 15 lines
fs/io_uring.c static void io_commit_sqring(struct io_ring_ctx *ctx) · line 1702
Quoted 13 lines
fs/io_uring.c +7
-0if (ctx->cached_sq_head !=READ_ONCE (ring->r.head)) { +/* +* Ensure any loads from the SQEs are done at this point, +* since once we write the new head, the application could +* write new data to them. +*/ +smp_mb (); +WRITE_ONCE (ring->r.head, ctx->cached_sq_head);/* * write side barrier of head update, app has read side. See I haven’t followed the full set of machinations here, but would
smp_store_release()be sufficient? It is a *lot* faster on some architectures.
Ah, yeah, that should work... I forgot that that exists.
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 05/19] Add io_uring IO interface
In reply to Andy LutomirskiQuoted 212 lines · Andy Lutomirski
Quoted 212 lines · Jens Axboe
Quoted 183 lines · Jens Axboe
Quoted 162 lines · Jann Horn
Quoted 153 lines · Jens Axboe
Quoted 152 lines · Jann Horn
Quoted 149 lines · Jens Axboe
Quoted 137 lines · Jann Horn
Quoted 135 lines · Jens Axboe
Quoted 134 lines · Jens Axboe
Quoted 131 lines · Jann Horn
Quoted 122 lines · Jens Axboe
Quoted 121 lines · Jens Axboe
On 2/12/19 2:42 PM, Jann Horn wrote:
On Sat, Feb 9, 2019 at 5:15 AM Jens Axboe <[email protected]> wrote:
On 2/8/19 3:12 PM, Jann Horn wrote:
On Fri, Feb 8, 2019 at 6:34 PM Jens Axboe <[email protected]> wrote:
The submission queue (SQ) and completion queue (CQ) rings are shared
between the application and the kernel. This eliminates the need to
copy data back and forth to submit and complete IO.IO submissions use the io_uring_sqe data structure, and completions
are generated in the form of io_uring_cqe data structures. The SQ
ring is an index into the io_uring_sqe array, which makes it possible
to submit a batch of IOs without them being contiguous in the ring.
The CQ ring is always contiguous, as completion events are inherently
unordered, and hence any io_uring_cqe entry can point back to an
arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up an io_uring instance for doing async IO. On success,
returns a file descriptor that the application can mmap to
gain access to the SQ ring, CQ ring, and io_uring_sqes.io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for
them to complete, or both. The behavior is controlled by the
parameters passed in. If 'to_submit' is non-zero, then we'll
try and submit new IO. If IORING_ENTER_GETEVENTS is set, the
kernel will wait for 'min_complete' events, if they aren't
already available. It's valid to set IORING_ENTER_GETEVENTS
and 'min_complete' == 0 at the same time, this allows the
kernel to return already completed events without waiting
for them. This is useful only for polling, as for IRQ
driven IO, the application can just check the CQ ring
without entering the kernel.With this setup, it's possible to do async IO with a single system
call. Future developments will enable polled IO with this interface,
and polled submission as well. The latter will enable an application
to do IO without doing ANY system calls at all.For IRQ driven IO, an application only needs to enter the kernel for
completions if it wants to wait for them to occur.Each io_uring is backed by a workqueue, to support buffered async IO
as well. We will only punt to an async context if the command would
need to wait for IO on the device side. Any data that can be accessed
directly in the page cache is done inline. This avoids the slowness
issue of usual threadpools, since cached data is accessed as quickly
as a sync interface.
[...]
+static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s)
+{
+ struct io_kiocb *req;
+ ssize_t ret;
+
+ /* enforce forwards compatibility on users */
+ if (unlikely(s->sqe->flags))
+ return -EINVAL;
+
+ req = io_get_req(ctx);
+ if (unlikely(!req))
+ return -EAGAIN;
+
+ req->rw.ki_filp = NULL;
+
+ ret = __io_submit_sqe(ctx, req, s, true);
+ if (ret == -EAGAIN) {
+ memcpy(&req->submit, s, sizeof(*s));
+ INIT_WORK(&req->work, io_sq_wq_submit_work);
+ queue_work(ctx->sqo_wq, &req->work);
+ ret = 0;
+ }
+ if (ret)
+ io_free_req(req);
+
+ return ret;
+}
+
+static void io_commit_sqring(struct io_ring_ctx *ctx)
+{
+ struct io_sq_ring *ring = ctx->sq_ring;
+
+ if (ctx->cached_sq_head != ring->r.head) {
+ WRITE_ONCE(ring->r.head, ctx->cached_sq_head);
+ /* write side barrier of head update, app has read side */
+ smp_wmb();Can you elaborate on what this memory barrier is doing? Don't you need
some sort of memory barrier *before* the WRITE_ONCE(), to ensure that
nobody sees the updated head before you're done reading the submission
queue entry? Or is that barrier elsewhere?The matching read barrier is in the application, it must do that before
reading ->head for the SQ ring.For the other barrier, since the ring->r.head now has a READ_ONCE(),
that should be all we need to ensure that loads are done.READ_ONCE() / WRITE_ONCE are not hardware memory barriers that enforce
ordering with regard to concurrent execution on other cores. They are
only compiler barriers, influencing the order in which the compiler
emits things. (Well, unless you're on alpha, where READ_ONCE() implies
a memory barrier that prevents reordering of dependent reads.)As far as I can tell, between the READ_ONCE(ring->array[...]) in
io_get_sqring() and the WRITE_ONCE() in io_commit_sqring(), you have
no *hardware* memory barrier that prevents reordering against
concurrently running userspace code. As far as I can tell, the
following could happen:- The kernel reads from ring->array in io_get_sqring(), then updates
the head in io_commit_sqring(). The CPU reorders the memory accesses
such that the write to the head becomes visible before the read from
ring->array has completed.
- Userspace observes the write to the head and reuses the array slots
the kernel has freed with the write, clobbering ring->array before the
kernel reads from ring->array.Quoted 2 lines
Quoted 2 lines
I'd say this is highly theoretical for the normal use case, as we
will have submitted IO in between. Hence the load must have been done.Sorry, I'm confused. Who is "we", and which load are you referring to?
io_sq_thread()goes directly fromio_get_sqring()toio_commit_sqring(), with only a conditionalio_sqe_needs_user()in between, if thei == ARRAY_SIZE(sqes)check triggers. There is no "submitting IO" in the middle.Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
You are right, the patch I sent IS needed for the sq thread case! It's only true for the "normal" case that we don't need the
smp_mb()before writing the sq ring head, as sqes are fully consumed at that point.Hmm... does that actually matter? As long as you don't have an explicit barrier for this, the CPU could still reorder things, right? Pull the store in front of everything else?
If the IO has been submitted, by definition the loads have completed. At that point it should be fine to commit the ring head that the application sees.
Quoted 7 lines
Quoted 3 lines
What exactly do you mean by "the IO has been submitted"? Are you talking about interaction with hardware, or about the end of the syscall, or something else?
I mean that the loads from the sqe, which the IO is made of, have been done. That's what we care about here, right? The sqe has either been turned into an io request and has been submitted, or it has been copied.
Quoted 10 lines
But they might not actually be done. AFAIU the CPU is allowed to do the
WRITE_ONCEof the head before doing any of the reads from the sqe
- loads and stores you do, as observed by a concurrently executing thread, can happen in an order independent of the order in which you write them in your code unless you use memory barriers. So the CPU might decide to first write the new head, then do the read forio_get_sqring(), and then do the__io_submit_sqe(), potentially reading e.g. aIORING_OP_NOPopcode that has been written by concurrently executing userspace after userspace has observed the bumped head.For that to be possible, we'd need NO ordering in between the IO submission and when we write the sq ring head. A single spin lock should do it, right?
It's not that I'm set against adding an
smp_mb()toio_commit_sqring(), but I think we're going off the deep end a little bit here on theoretical vs what can practically happen.Quoted 6 lines
For the regular IO cases, we will have done at least one lock/unlock cycle. This is true for nops as well, and poll. The only case that could potentially NOT have one is the fsync, for the case where we punt and don't add it to existing work, we don't have any locking in between.
I'll add the
smp_mb()for peace of mind.For reference, folded in:
fs/io_uring.c +7
-0@@ -1690,6 +1690,13 @@ static void io_commit_sqring(struct io_ring_ctx *ctx)struct io_sq_ring *ring = ctx->sq_ring;if (ctx->cached_sq_head !=READ_ONCE (ring->r.head)) { +/* +* Ensure any loads from the SQEs are done at this point, +* since once we write the new head, the application could +* write new data to them. +*/ +smp_mb (); +WRITE_ONCE (ring->r.head, ctx->cached_sq_head);
Quoted 8 lines
fs/io_uring.c static void io_commit_sqring(struct io_ring_ctx *ctx) · line 1702
Quoted 4 lines
fs/io_uring.c +0
-0/* * write side barrier of head update, app has read side. See I haven’t followed the full set of machinations here, but would
smp_store_release()be sufficient? It is a *lot* faster on some architectures.
Thanks for the hint, yes that looks more appropriate.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 05/19] Add io_uring IO interface
In reply to Jens AxboeQuoted 63 lines · Jens Axboe
The submission queue (SQ) and completion queue (CQ) rings are shared between the application and the kernel. This eliminates the need to copy data back and forth to submit and complete IO.
IO submissions use the
io_uring_sqedata structure, and completions are generated in the form ofio_uring_cqedata structures. The SQ ring is an index into theio_uring_sqearray, which makes it possible to submit a batch of IOs without them being contiguous in the ring. The CQ ring is always contiguous, as completion events are inherently unordered, and hence anyio_uring_cqeentry can point back to an arbitrary submission.Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up anio_uringinstance for doing async IO. On success, returns a file descriptor that the application can mmap to gain access to the SQ ring, CQ ring, andio_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags, sigset, sigsetsize)
Initiates IO against the rings mapped to this fd, or waits for them to complete, or both. The behavior is controlled by the parameters passed in. If 'to_submit' is non-zero, then we'll try and submit new IO. IfIORING_ENTER_GETEVENTSis set, the kernel will wait for 'min_complete' events, if they aren't already available. It's valid to setIORING_ENTER_GETEVENTSand 'min_complete' == 0 at the same time, this allows the kernel to return already completed events without waiting for them. This is useful only for polling, as for IRQ driven IO, the application can just check the CQ ring without entering the kernel.With this setup, it's possible to do async IO with a single system call. Future developments will enable polled IO with this interface, and polled submission as well. The latter will enable an application to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for completions if it wants to wait for them to occur.
Each
io_uringis backed by a workqueue, to support buffered async IO as well. We will only punt to an async context if the command would need to wait for IO on the device side. Any data that can be accessed directly in the page cache is done inline. This avoids the slowness issue of usual threadpools, since cached data is accessed as quickly as a sync interface.Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
Signed-off-by Jens Axboe <[email protected]>--- arch/x86/entry/syscalls/syscall_32.tbl | 2 + arch/x86/entry/syscalls/syscall_64.tbl | 2 + fs/Makefile | 1 + fs/io_uring.c | 1175 ++++++++++++++++++++++++ include/linux/fs.h | 9 + include/linux/syscalls.h | 6 + include/uapi/asm-generic/unistd.h | 6 +- include/uapi/linux/io_uring.h | 95 ++ init/Kconfig | 9 + kernel/sys_ni.c | 2 + net/unix/garbage.c | 3 + 11 files changed, 1309 insertions(+), 1 deletion(-)
Quoted 3 lines
create mode 100644
fs/io_uring.c
create mode 100644include/uapi/linux/io_uring.h
Cheers,
Hannes
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 6/19 io_uring: add fsync support
In reply to Jens AxboeFrom: Christoph Hellwig <[email protected]>
Add a new fsync opcode, which either syncs a range if one is passed, or the whole file if the offset and length fields are both cleared to zero. A flag is provided to use fdatasync semantics, that is only force out metadata which is required to retrieve the file data, but not others like metadata.
--- fs/io_uring.c | 40 +++++++++++++++++++++++++++++++++++ include/uapi/linux/io_uring.h | 8 ++++++- 2 files changed, 47 insertions(+), 1 deletion(-)
fs/io_uring.c +40 -0
@@ -4,6 +4,7 @@* supporting fast/efficient IO. * * Copyright (C) 2018-2019 Jens Axboe +* Copyright (c) 2018-2019 Christoph Hellwig */ #include <linux/kernel.h> #include <linux/init.h>
@@ -518,6 +519,42 @@ static int io_nop(struct io_kiocb *req, u64 user_data)return 0 ; } +static int io_fsync (struct io_kiocb *req,const struct io_uring_sqe *sqe, +bool force_nonblock) +{ +struct io_ring_ctx *ctx = req->ctx; +loff_t sqe_off =READ_ONCE (sqe->off); +loff_t sqe_len =READ_ONCE (sqe->len); +loff_t end = sqe_off + sqe_len; +unsigned fsync_flags; +struct file *file; +int ret, fd; + +/* fsync always requires a blocking context */ +if (force_nonblock) +return -EAGAIN ; + +if (unlikely (sqe->addr || sqe->ioprio)) +return -EINVAL ; + + fsync_flags =READ_ONCE (sqe->fsync_flags); +if (unlikely (fsync_flags & ~IORING_FSYNC_DATASYNC )) +return -EINVAL ; + + fd =READ_ONCE (sqe->fd); + file =fget (fd); +if (unlikely (!file)) +return -EBADF ; + + ret =vfs_fsync_range (file, sqe_off, end >0 ? end :LLONG_MAX , + fsync_flags &IORING_FSYNC_DATASYNC ); + +fput (file); +io_cqring_add_event (ctx, sqe->user_data, ret,0 ); +io_free_req (req); +return 0 ; +} +static int __io_submit_sqe (struct io_ring_ctx *ctx,struct io_kiocb *req,const struct sqe_submit *s,bool force_nonblock) {
@@ -539,6 +576,9 @@ static int __io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,case IORING_OP_WRITEV : ret =io_write (req, s, force_nonblock);break ; +case IORING_OP_FSYNC : + ret =io_fsync (req, s->sqe, force_nonblock); +break ;default : ret = -EINVAL ;break ;
include/uapi/linux/io_uring.h +7 -1
@@ -24,7 +24,7 @@ struct io_uring_sqe {__u32 len;/* buffer size or number of iovecs */ union {__kernel_rwf_t rw_flags; -__u32 __resv; +__u32 fsync_flags; };__u64 user_data;/* data to be passed back at completion time */ __u64 __pad2[3 ];
@@ -33,6 +33,12 @@ struct io_uring_sqe {#define IORING_OP_NOP 0#define IORING_OP_READV 1#define IORING_OP_WRITEV 2 +#define IORING_OP_FSYNC 3 + +/* +* sqe->fsync_flags +*/ +#define IORING_FSYNC_DATASYNC (1U << 0)/* * IO completion data structure (Completion Queue Entry)
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 06/19] io_uring: add fsync support
Quoted 11 lines · Jens Axboe
From: Christoph Hellwig <[email protected]>
Add a new fsync opcode, which either syncs a range if one is passed, or the whole file if the offset and length fields are both cleared to zero. A flag is provided to use fdatasync semantics, that is only force out metadata which is required to retrieve the file data, but not others like metadata.
Signed-off-by Christoph Hellwig <[email protected]>Signed-off-by Jens Axboe <[email protected]>---
[...]
Quoted 14 lines
fs/io_uring.c +14
-0+static int io_fsync (struct io_kiocb *req,const struct io_uring_sqe *sqe, +bool force_nonblock) +{ +struct io_ring_ctx *ctx = req->ctx; +loff_t sqe_off =READ_ONCE (sqe->off); +loff_t sqe_len =READ_ONCE (sqe->len); +loff_t end = sqe_off + sqe_len; +unsigned fsync_flags; +struct file *file; +int ret, fd; + +/* fsync always requires a blocking context */ +if (force_nonblock) +return -EAGAIN ;
fs/io_uring.c static int io_nop(struct io_kiocb *req, u64 user_data) · line 545
Quoted 10 lines
fs/io_uring.c +10
-0+ +if (unlikely (sqe->addr || sqe->ioprio)) +return -EINVAL ; + + fsync_flags =READ_ONCE (sqe->fsync_flags); +if (unlikely (fsync_flags & ~IORING_FSYNC_DATASYNC )) +return -EINVAL ; + + fd =READ_ONCE (sqe->fd); + file =fget (fd);
This always runs on the workqueue, right? Is it possible to call fget() on a workqueue?
fs/io_uring.c static int io_nop(struct io_kiocb *req, u64 user_data) · line 556
Quoted 11 lines
fs/io_uring.c +11
-0+if (unlikely (!file)) +return -EBADF ; + + ret =vfs_fsync_range (file, sqe_off, end >0 ? end :LLONG_MAX , + fsync_flags &IORING_FSYNC_DATASYNC ); + +fput (file); +io_cqring_add_event (ctx, sqe->user_data, ret,0 ); +io_free_req (req); +return 0 ; +}
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 06/19] io_uring: add fsync support
Quoted 32 lines · Jann Horn
Quoted 11 lines · Jens Axboe
From: Christoph Hellwig <[email protected]>
Add a new fsync opcode, which either syncs a range if one is passed, or the whole file if the offset and length fields are both cleared to zero. A flag is provided to use fdatasync semantics, that is only force out metadata which is required to retrieve the file data, but not others like metadata.
Signed-off-by Christoph Hellwig <[email protected]>Signed-off-by Jens Axboe <[email protected]>---
[...]
Quoted 19 lines
fs/io_uring.c +19
-0+static int io_fsync (struct io_kiocb *req,const struct io_uring_sqe *sqe, +bool force_nonblock) +{ +struct io_ring_ctx *ctx = req->ctx; +loff_t sqe_off =READ_ONCE (sqe->off); +loff_t sqe_len =READ_ONCE (sqe->len); +loff_t end = sqe_off + sqe_len; +unsigned fsync_flags; +struct file *file; +int ret, fd; + +/* fsync always requires a blocking context */ +if (force_nonblock) +return -EAGAIN ; + +if (unlikely (sqe->addr || sqe->ioprio)) +return -EINVAL ; + + fsync_flags =READ_ONCE (sqe->fsync_flags);
Quoted 8 lines
fs/io_uring.c static int io_nop(struct io_kiocb *req, u64 user_data) · line 545
Quoted 5 lines
fs/io_uring.c +5
-0+if (unlikely (fsync_flags & ~IORING_FSYNC_DATASYNC )) +return -EINVAL ; + + fd =READ_ONCE (sqe->fd); + file =fget (fd);This always runs on the workqueue, right? Is it possible to call
fget()on a workqueue?
Oops yes, I've now split that into a io_prep_fsync() and io_fsync() part so that it works correctly with this new method. Also added a liburing test for this.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 06/19] io_uring: add fsync support
In reply to Jens AxboeQuoted 8 lines · Jens Axboe
From: Christoph Hellwig <[email protected]>
Add a new fsync opcode, which either syncs a range if one is passed, or the whole file if the offset and length fields are both cleared to zero. A flag is provided to use fdatasync semantics, that is only force out metadata which is required to retrieve the file data, but not others like metadata.
Quoted 7 lines
Signed-off-by Christoph Hellwig <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- fs/io_uring.c | 40 +++++++++++++++++++++++++++++++++++ include/uapi/linux/io_uring.h | 8 ++++++- 2 files changed, 47 insertions(+), 1 deletion(-)
Cheers,
Hannes
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 7/19 io_uring: support for IO polling
In reply to Jens AxboeAdd support for a polled io_uring instance. When a read or write is submitted to a polled io_uring, the application must poll for completions on the CQ ring through io_uring_enter(2). Polled IO may not generate IRQ completions, hence they need to be actively found by the application itself.
To use polling, io_uring_setup() must be used with the IORING_SETUP_IOPOLL flag being set. It is illegal to mix and match polled and non-polled IO on an io_uring.
--- fs/io_uring.c | 274 ++++++++++++++++++++++++++++++++-- include/uapi/linux/io_uring.h | 5 + 2 files changed, 270 insertions(+), 9 deletions(-)
fs/io_uring.c +265 -9
@@ -103,6 +103,14 @@ struct io_ring_ctx {struct {spinlock_t completion_lock; +bool poll_multi_file; +/* +* ->poll_list is protected by the ctx->uring_lock for +* io_uring instances that don't use IORING_SETUP_SQPOLL. +* For SQPOLL, only the single threaded io_sq_thread() will +* manipulate the list, hence no extra locking is needed there. +*/ +struct list_head poll_list; }____cacheline_aligned_in_smp ;#if defined(CONFIG_UNIX )
@@ -114,6 +122,7 @@ struct sqe_submit {const struct io_uring_sqe *sqe;unsigned short index;bool has_user; +bool needs_lock; };struct io_kiocb {
@@ -125,12 +134,15 @@ struct io_kiocb {struct list_head list;unsigned int flags;#define REQ_F_FORCE_NONBLOCK 1/* inline submission attempt */ +#define REQ_F_IOPOLL_COMPLETED 2/* polled IO has completed */ u64 user_data; +u64 error;struct work_struct work; };#define IO_PLUG_THRESHOLD 2 +#define IO_IOPOLL_BATCH 8static struct kmem_cache *req_cachep;
@@ -175,6 +187,7 @@ static struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)mutex_init (&ctx->uring_lock);init_waitqueue_head (&ctx->wait);spin_lock_init (&ctx->completion_lock); +INIT_LIST_HEAD (&ctx->poll_list);return ctx; }
@@ -269,12 +282,153 @@ static struct io_kiocb *io_get_req(struct io_ring_ctx *ctx)return NULL ; } +static void io_free_req_many (struct io_ring_ctx *ctx,void **reqs,int *nr) +{ +if (*nr) { +kmem_cache_free_bulk (req_cachep, *nr, reqs); +io_ring_drop_ctx_refs (ctx, *nr); + *nr =0 ; + } +} +static void io_free_req (struct io_kiocb *req) {io_ring_drop_ctx_refs (req->ctx,1 );kmem_cache_free (req_cachep, req); } +/* +* Find and free completed poll iocbs +*/ +static void io_iopoll_complete (struct io_ring_ctx *ctx,unsigned int *nr_events, +struct list_head *done) +{ +void *reqs[IO_IOPOLL_BATCH ]; +struct io_kiocb *req; +int to_free =0 ; + +while (!list_empty (done)) { + req = list_first_entry(done, struct io_kiocb, list); +list_del (&req->list); + +io_cqring_fill_event (ctx, req->user_data, req->error,0 ); + + reqs[to_free++] = req; + (*nr_events)++; + +fput (req->rw.ki_filp); +if (to_free ==ARRAY_SIZE (reqs)) +io_free_req_many (ctx, reqs, &to_free); + } +io_commit_cqring (ctx); + +io_free_req_many (ctx, reqs, &to_free); +} + +static int io_do_iopoll (struct io_ring_ctx *ctx,unsigned int *nr_events, +long min) +{ +struct io_kiocb *req, *tmp; +LIST_HEAD (done); +bool spin; +int ret; + +/* +* Only spin for completions if we don't have multiple devices hanging +* off our complete list, and we're under the requested amount. +*/ + spin = !ctx->poll_multi_file && *nr_events < min; + + ret =0 ; +list_for_each_entry_safe (req, tmp, &ctx->poll_list, list) { +struct kiocb *kiocb = &req->rw; + +/* +* Move completed entries to our local list. If we find a +* request that requires polling, break out and complete +* the done list first, if we have entries there. +*/ +if (req->flags &REQ_F_IOPOLL_COMPLETED ) { +list_move_tail (&req->list, &done); +continue ; + } +if (!list_empty (&done)) +break ; + + ret = kiocb->ki_filp->f_op->iopoll (kiocb, spin); +if (ret <0 ) +break ; + +if (ret && spin) + spin = false; + ret =0 ; + } + +if (!list_empty (&done)) +io_iopoll_complete (ctx, nr_events, &done); + +return ret; +} + +/* +* Poll for a mininum of 'min' events. Note that if min == 0 we consider that a +* non-spinning poll check - we'll still enter the driver poll loop, but only +* as a non-spinning completion check. +*/ +static int io_iopoll_getevents (struct io_ring_ctx *ctx,unsigned int *nr_events, +long min) +{ +while (!list_empty (&ctx->poll_list)) { +int ret; + + ret =io_do_iopoll (ctx, nr_events, min); +if (ret <0 ) +return ret; +if (!min || *nr_events >= min) +return 0 ; + } + +return 1 ; +} + +/* +* We can't just wait for polled events to come to us, we have to actively +* find and complete them. +*/ +static void io_iopoll_reap_events (struct io_ring_ctx *ctx) +{ +if (!(ctx->flags &IORING_SETUP_IOPOLL )) +return ; + +mutex_lock (&ctx->uring_lock); +while (!list_empty (&ctx->poll_list)) { +unsigned int nr_events =0 ; + +io_iopoll_getevents (ctx, &nr_events,1 ); + } +mutex_unlock (&ctx->uring_lock); +} + +static int io_iopoll_check (struct io_ring_ctx *ctx,unsigned *nr_events, +long min) +{ +int ret =0 ; + +do { +int tmin =0 ; + +if (*nr_events < min) + tmin = min - *nr_events; + + ret =io_iopoll_getevents (ctx, nr_events, tmin); +if (ret <=0 ) +break ; + ret =0 ; + }while (min && !*nr_events && !need_resched ()); + +return ret; +} +static void kiocb_end_write (struct kiocb *kiocb) {if (kiocb->ki_flags &IOCB_WRITE ) {
@@ -301,6 +455,53 @@ static void io_complete_rw(struct kiocb *kiocb, long res, long res2)io_free_req (req); } +static void io_complete_rw_iopoll (struct kiocb *kiocb,long res,long res2) +{ + struct io_kiocb *req = container_of(kiocb, struct io_kiocb, rw); + +kiocb_end_write (kiocb); + + req->error = res; +if (res != -EAGAIN ) + req->flags |=REQ_F_IOPOLL_COMPLETED ; +} + +/* +* After the iocb has been issued, it's safe to be found on the poll list. +* Adding the kiocb to the list AFTER submission ensures that we don't +* find it from a io_iopoll_getevents() thread before the issuer is done +* accessing the kiocb cookie. +*/ +static void io_iopoll_req_issued (struct io_kiocb *req) +{ +struct io_ring_ctx *ctx = req->ctx; + +/* +* Track whether we have multiple files in our lists. This will impact +* how we do polling eventually, not spinning if we're on potentially +* different devices. +*/ +if (list_empty (&ctx->poll_list)) { + ctx->poll_multi_file = false; + }else if (!ctx->poll_multi_file) { +struct io_kiocb *list_req; + + list_req = list_first_entry(&ctx->poll_list, struct io_kiocb, + list); +if (list_req->rw.ki_filp != req->rw.ki_filp) + ctx->poll_multi_file = true; + } + +/* +* For fast devices, IO may have already completed. If it has, add +* it to the front so we find it first. +*/ +if (req->flags &REQ_F_IOPOLL_COMPLETED ) +list_add (&req->list, &ctx->poll_list); +else +list_add_tail (&req->list, &ctx->poll_list); +} +/* * If we tracked the file through the SCM inflight mechanism, we could support * any file. For now, just ensure that anything potentially problematic is done
@@ -321,6 +522,7 @@ static bool io_file_supports_async(struct file *file)static int io_prep_rw (struct io_kiocb *req,const struct io_uring_sqe *sqe,bool force_nonblock) { +struct io_ring_ctx *ctx = req->ctx;struct kiocb *kiocb = &req->rw;unsigned ioprio;int fd, ret;
@@ -356,12 +558,22 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe, kiocb->ki_flags |=IOCB_NOWAIT ; req->flags |=REQ_F_FORCE_NONBLOCK ; } -if (kiocb->ki_flags &IOCB_HIPRI ) { - ret = -EINVAL ; - goto out_fput; - } +if (ctx->flags &IORING_SETUP_IOPOLL ) { + ret = -EOPNOTSUPP ; +if (!(kiocb->ki_flags &IOCB_DIRECT ) || + !kiocb->ki_filp->f_op->iopoll) + goto out_fput; - kiocb->ki_complete = io_complete_rw; + req->error =0 ; + kiocb->ki_flags |=IOCB_HIPRI ; + kiocb->ki_complete = io_complete_rw_iopoll; + }else { +if (kiocb->ki_flags &IOCB_HIPRI ) { + ret = -EINVAL ; + goto out_fput; + } + kiocb->ki_complete = io_complete_rw; + }return 0 ; out_fput:fput (kiocb->ki_filp);
@@ -514,6 +726,9 @@ static int io_nop(struct io_kiocb *req, u64 user_data) {struct io_ring_ctx *ctx = req->ctx; +if (unlikely (ctx->flags &IORING_SETUP_IOPOLL )) +return -EINVAL ; +io_cqring_add_event (ctx, user_data,0 ,0 );io_free_req (req);return 0 ;
@@ -534,6 +749,8 @@ static int io_fsync(struct io_kiocb *req, const struct io_uring_sqe *sqe,if (force_nonblock)return -EAGAIN ; +if (unlikely (req->ctx->flags &IORING_SETUP_IOPOLL )) +return -EINVAL ;if (unlikely (sqe->addr || sqe->ioprio))return -EINVAL ;
@@ -584,7 +801,22 @@ static int __io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,break ; } -return ret; +if (ret) +return ret; + +if (ctx->flags &IORING_SETUP_IOPOLL ) { +if (req->error == -EAGAIN ) +return -EAGAIN ; + +/* workqueue context doesn't hold uring_lock, grab it now */ +if (s->needs_lock) +mutex_lock (&ctx->uring_lock); +io_iopoll_req_issued (req); +if (s->needs_lock) +mutex_unlock (&ctx->uring_lock); + } + +return 0 ; }static void io_sq_wq_submit_work (struct work_struct *work)
@@ -608,8 +840,19 @@ static void io_sq_wq_submit_work(struct work_struct *work)use_mm (ctx->sqo_mm);set_fs (USER_DS ); s->has_user = true; + s->needs_lock = true; - ret =__io_submit_sqe (ctx, req, s, false); +do { + ret =__io_submit_sqe (ctx, req, s, false); +/* +* We can get EAGAIN for polled IO even though we're forcing +* a sync submission from here, since we can't wait for +* request slots on the block side. +*/ +if (ret != -EAGAIN ) +break ; +cond_resched (); + }while (1 );set_fs (old_fs);unuse_mm (ctx->sqo_mm);
@@ -724,6 +967,8 @@ static int io_ring_submit(struct io_ring_ctx *ctx, unsigned int to_submit)break ; s.has_user = true; + s.needs_lock = false; + ret =io_submit_sqe (ctx, &s);if (ret) {io_drop_sqring (ctx);
@@ -877,6 +1122,8 @@ static void io_ring_ctx_free(struct io_ring_ctx *ctx)sock_release (ctx->ring_sock);#endif +io_iopoll_reap_events (ctx); +io_mem_free (ctx->sq_ring);io_mem_free (ctx->sq_sqes);io_mem_free (ctx->cq_ring);
@@ -916,6 +1163,7 @@ static void io_ring_ctx_wait_and_kill(struct io_ring_ctx *ctx)percpu_ref_kill (&ctx->refs);mutex_unlock (&ctx->uring_lock); +io_iopoll_reap_events (ctx);wait_for_completion (&ctx->ctx_done);io_ring_ctx_free (ctx); }
@@ -996,6 +1244,8 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, goto out_ctx; }if (flags &IORING_ENTER_GETEVENTS ) { +unsigned nr_events =0 ; +/* * The application could have included the 'to_submit' count * in how many events it wanted to wait for. If we failed to
@@ -1005,7 +1255,13 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit,if (submitted < to_submit) min_complete =min_t (unsigned, submitted, min_complete); - ret =io_cqring_wait (ctx, min_complete, sig, sigsz); +if (ctx->flags &IORING_SETUP_IOPOLL ) { +mutex_lock (&ctx->uring_lock); + ret =io_iopoll_check (ctx, &nr_events, min_complete); +mutex_unlock (&ctx->uring_lock); + }else { + ret =io_cqring_wait (ctx, min_complete, sig, sigsz); + } } out_ctx:
@@ -1188,7 +1444,7 @@ static long io_uring_setup(u32 entries, struct io_uring_params __user *params)return -EINVAL ; } -if (p.flags) +if (p.flags & ~IORING_SETUP_IOPOLL )return -EINVAL ; ret =io_uring_create (entries, &p);
include/uapi/linux/io_uring.h +5 -0
@@ -30,6 +30,11 @@ struct io_uring_sqe {__u64 __pad2[3 ]; }; +/* +* io_uring_setup() flags +*/ +#define IORING_SETUP_IOPOLL (1U << 0)/* io_context is polled */ +#define IORING_OP_NOP 0#define IORING_OP_READV 1#define IORING_OP_WRITEV 2
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 07/19] io_uring: support for IO polling
Quoted 10 lines · Jens Axboe
Add support for a polled
io_uringinstance. When a read or write is submitted to a polledio_uring, the application must poll for completions on the CQ ring throughio_uring_enter(2). Polled IO may not generate IRQ completions, hence they need to be actively found by the application itself.To use polling,
io_uring_setup()must be used with theIORING_SETUP_IOPOLLflag being set. It is illegal to mix and match polled and non-polled IO on anio_uring.
Quoted 6 lines
Signed-off-by Jens Axboe <[email protected]>--- fs/io_uring.c | 274 ++++++++++++++++++++++++++++++++-- include/uapi/linux/io_uring.h | 5 + 2 files changed, 270 insertions(+), 9 deletions(-)
Cheers,
Hannes
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 8/19 fs: add fget_many() and fput_many()
In reply to Jens AxboeSome uses cases repeatedly get and put references to the same file, but the only exposed interface is doing these one at the time. As each of these entail an atomic inc or dec on a shared structure, that cost can add up.
Add fget_many(), which works just like fget(), except it takes an argument for how many references to get on the file. Ditto fput_many(), which can drop an arbitrary number of references to a file.
--- fs/file.c | 15 ++++++++++----- fs/file_table.c | 9 +++++++-- include/linux/file.h | 2 ++ include/linux/fs.h | 4 +++- 4 files changed, 22 insertions(+), 8 deletions(-)
fs/file.c +10 -5
@@ -705,7 +705,7 @@ void do_close_on_exec(struct files_struct *files)spin_unlock (&files->file_lock); } -static struct file *__fget (unsigned int fd,fmode_t mask) +static struct file *__fget (unsigned int fd,fmode_t mask,unsigned int refs) {struct files_struct *files = current->files;struct file *file;
@@ -720,7 +720,7 @@ static struct file *__fget(unsigned int fd, fmode_t mask)*/ if (file->f_mode & mask) file =NULL ; -else if (!get_file_rcu (file)) +else if (!get_file_rcu_many (file, refs)) goto loop; }rcu_read_unlock ();
@@ -728,15 +728,20 @@ static struct file *__fget(unsigned int fd, fmode_t mask)return file; } +struct file *fget_many (unsigned int fd,unsigned int refs) +{ +return __fget (fd,FMODE_PATH , refs); +} +struct file *fget (unsigned int fd) { -return __fget (fd,FMODE_PATH ); +return __fget (fd,FMODE_PATH ,1 ); }EXPORT_SYMBOL (fget);struct file *fget_raw (unsigned int fd) { -return __fget (fd,0 ); +return __fget (fd,0 ,1 ); }EXPORT_SYMBOL (fget_raw);
@@ -767,7 +772,7 @@ static unsigned long __fget_light(unsigned int fd, fmode_t mask)return 0 ;return (unsigned long )file; }else { - file =__fget (fd, mask); + file =__fget (fd, mask,1 );if (!file)return 0 ;return FDPUT_FPUT | (unsigned long )file;
fs/file_table.c +7 -2
@@ -326,9 +326,9 @@ void flush_delayed_fput(void) static DECLARE_DELAYED_WORK(delayed_fput_work, delayed_fput); -void fput (struct file *file) +void fput_many (struct file *file,unsigned int refs) { -if (atomic_long_dec_and_test (&file->f_count)) { +if (atomic_long_sub_and_test (refs, &file->f_count)) {struct task_struct *task = current;if (likely (!in_interrupt () && !(task->flags &PF_KTHREAD ))) {
@@ -347,6 +347,11 @@ void fput(struct file *file) } } +void fput (struct file *file) +{ +fput_many (file,1 ); +} +/* * synchronous analog of fput(); for kernel threads that might be needed * in some umount() (and thus can't use flush_delayed_fput() without
include/linux/file.h +2 -0
@@ -13,6 +13,7 @@struct file ;extern void fput (struct file *); +extern void fput_many (struct file *,unsigned int );struct file_operations ;struct vfsmount ;
@@ -44,6 +45,7 @@ static inline void fdput(struct fd fd) }extern struct file *fget (unsigned int fd); +extern struct file *fget_many (unsigned int fd,unsigned int refs);extern struct file *fget_raw (unsigned int fd);extern unsigned long __fdget (unsigned int fd);extern unsigned long __fdget_raw (unsigned int fd);
include/linux/fs.h +3 -1
@@ -952,7 +952,9 @@ static inline struct file *get_file(struct file *f)atomic_long_inc (&f->f_count);return f; } -#define get_file_rcu (x) atomic_long_inc_not_zero(&(x)->f_count) +#define get_file_rcu_many (x, cnt) \ + atomic_long_add_unless(&(x)->f_count, (cnt), 0) +#define get_file_rcu (x) get_file_rcu_many((x), 1)#define fput_atomic (x) atomic_long_add_unless(&(x)->f_count, -1, 1)#define file_count (x) atomic_long_read(&(x)->f_count)
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 08/19] fs: add fget_many() and fput_many()
Quoted 9 lines · Jens Axboe
Some uses cases repeatedly get and put references to the same file, but the only exposed interface is doing these one at the time. As each of these entail an atomic inc or dec on a shared structure, that cost can add up.
Add
fget_many(), which works just likefget(), except it takes an argument for how many references to get on the file. Dittofput_many(), which can drop an arbitrary number of references to a file.
Quoted 9 lines
Reviewed-by Christoph Hellwig <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- fs/file.c | 15 ++++++++++----- fs/file_table.c | 9 +++++++-- include/linux/file.h | 2 ++ include/linux/fs.h | 4 +++- 4 files changed, 22 insertions(+), 8 deletions(-)
Cheers,
Hannes
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 9/19 io_uring: use fget/fput_many() for file references
In reply to Jens AxboeAdd a separate io_submit_state structure, to cache some of the things we need for IO submission.
One such example is file reference batching. io_submit_state. We get as many references as the number of sqes we are submitting, and drop unused ones if we end up switching files. The assumption here is that we're usually only dealing with one fd, and if there are multiple, hopefuly they are at least somewhat ordered. Could trivially be extended to cover multiple fds, if needed.
On the completion side we do the same thing, except this is trivially done just locally in io_iopoll_reap().
--- fs/io_uring.c | 142 ++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 121 insertions(+), 21 deletions(-)
fs/io_uring.c +121 -21
@@ -144,6 +144,19 @@ struct io_kiocb {#define IO_PLUG_THRESHOLD 2#define IO_IOPOLL_BATCH 8 +struct io_submit_state { +struct blk_plug plug; + +/* +* File reference cache +*/ +struct file *file; +unsigned int fd; +unsigned int has_refs; +unsigned int used_refs; +unsigned int ios_left; +}; +static struct kmem_cache *req_cachep;static const struct file_operations io_uring_fops;
@@ -304,9 +317,11 @@ static void io_iopoll_complete(struct io_ring_ctx *ctx, unsigned int *nr_events, struct list_head *done) {void *reqs[IO_IOPOLL_BATCH ]; +int file_count, to_free; +struct file *file =NULL ;struct io_kiocb *req; -int to_free =0 ; + file_count = to_free =0 ;while (!list_empty (done)) { req = list_first_entry(done, struct io_kiocb, list);list_del (&req->list);
@@ -316,12 +331,28 @@ static void io_iopoll_complete(struct io_ring_ctx *ctx, unsigned int *nr_events, reqs[to_free++] = req; (*nr_events)++; -fput (req->rw.ki_filp); +/* +* Batched puts of the same file, to avoid dirtying the +* file usage count multiple times, if avoidable. +*/ +if (!file) { + file = req->rw.ki_filp; + file_count =1 ; + }else if (file == req->rw.ki_filp) { + file_count++; + }else { +fput_many (file, file_count); + file = req->rw.ki_filp; + file_count =1 ; + } +if (to_free ==ARRAY_SIZE (reqs))io_free_req_many (ctx, reqs, &to_free); }io_commit_cqring (ctx); +if (file) +fput_many (file, file_count);io_free_req_many (ctx, reqs, &to_free); }
@@ -502,6 +533,48 @@ static void io_iopoll_req_issued(struct io_kiocb *req)list_add_tail (&req->list, &ctx->poll_list); } +static void io_file_put (struct io_submit_state *state,struct file *file) +{ +if (!state) { +fput (file); + }else if (state->file) { +int diff = state->has_refs - state->used_refs; + +if (diff) +fput_many (state->file, diff); + state->file =NULL ; + } +} + +/* +* Get as many references to a file as we have IOs left in this submission, +* assuming most submissions are for one file, or at least that each file +* has more than one submission. +*/ +static struct file *io_file_get (struct io_submit_state *state,int fd) +{ +if (!state) +return fget (fd); + +if (state->file) { +if (state->fd == fd) { + state->used_refs++; + state->ios_left--; +return state->file; + } +io_file_put (state,NULL ); + } + state->file =fget_many (fd, state->ios_left); +if (!state->file) +return NULL ; + + state->fd = fd; + state->has_refs = state->ios_left; + state->used_refs =1 ; + state->ios_left--; +return state->file; +} +/* * If we tracked the file through the SCM inflight mechanism, we could support * any file. For now, just ensure that anything potentially problematic is done
@@ -520,7 +593,7 @@ static bool io_file_supports_async(struct file *file) }static int io_prep_rw (struct io_kiocb *req,const struct io_uring_sqe *sqe, -bool force_nonblock) +bool force_nonblock,struct io_submit_state *state) {struct io_ring_ctx *ctx = req->ctx;struct kiocb *kiocb = &req->rw;
@@ -532,7 +605,7 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe,return 0 ; fd =READ_ONCE (sqe->fd); - kiocb->ki_filp =fget (fd); + kiocb->ki_filp =io_file_get (state, fd);if (unlikely (!kiocb->ki_filp))return -EBADF ;if (force_nonblock && !io_file_supports_async (kiocb->ki_filp))
@@ -576,7 +649,10 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe, }return 0 ; out_fput: -fput (kiocb->ki_filp); +/* in case of error, we didn't use this file reference. drop it. */ +if (state) + state->used_refs--; +io_file_put (state, kiocb->ki_filp);return ret; }
@@ -622,7 +698,7 @@ static int io_import_iovec(struct io_ring_ctx *ctx, int rw, }static ssize_t io_read (struct io_kiocb *req,const struct sqe_submit *s, -bool force_nonblock) +bool force_nonblock,struct io_submit_state *state) {struct iovec inline_vecs[UIO_FASTIOV ], *iovec = inline_vecs;struct kiocb *kiocb = &req->rw;
@@ -630,7 +706,7 @@ static ssize_t io_read(struct io_kiocb *req, const struct sqe_submit *s,struct file *file;ssize_t ret; - ret =io_prep_rw (req, s->sqe, force_nonblock); + ret =io_prep_rw (req, s->sqe, force_nonblock, state);if (ret)return ret; file = kiocb->ki_filp;
@@ -666,7 +742,7 @@ static ssize_t io_read(struct io_kiocb *req, const struct sqe_submit *s, }static ssize_t io_write (struct io_kiocb *req,const struct sqe_submit *s, -bool force_nonblock) +bool force_nonblock,struct io_submit_state *state) {struct iovec inline_vecs[UIO_FASTIOV ], *iovec = inline_vecs;struct kiocb *kiocb = &req->rw;
@@ -674,7 +750,7 @@ static ssize_t io_write(struct io_kiocb *req, const struct sqe_submit *s,struct file *file;ssize_t ret; - ret =io_prep_rw (req, s->sqe, force_nonblock); + ret =io_prep_rw (req, s->sqe, force_nonblock, state);if (ret)return ret;/* Hold on to the file for -EAGAIN */
@@ -773,7 +849,8 @@ static int io_fsync(struct io_kiocb *req, const struct io_uring_sqe *sqe, }static int __io_submit_sqe (struct io_ring_ctx *ctx,struct io_kiocb *req, -const struct sqe_submit *s,bool force_nonblock) +const struct sqe_submit *s,bool force_nonblock, +struct io_submit_state *state) {ssize_t ret;int opcode;
@@ -788,10 +865,10 @@ static int __io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req, ret =io_nop (req, req->user_data);break ;case IORING_OP_READV : - ret =io_read (req, s, force_nonblock); + ret =io_read (req, s, force_nonblock, state);break ;case IORING_OP_WRITEV : - ret =io_write (req, s, force_nonblock); + ret =io_write (req, s, force_nonblock, state);break ;case IORING_OP_FSYNC : ret =io_fsync (req, s->sqe, force_nonblock);
@@ -843,7 +920,7 @@ static void io_sq_wq_submit_work(struct work_struct *work) s->needs_lock = true; do { - ret = __io_submit_sqe(ctx, req, s, false); + ret = __io_submit_sqe(ctx, req, s, false, NULL); /* * We can get EAGAIN for polled IO even though we're forcing * a sync submission from here, since we can't wait for
@@ -864,7 +941,8 @@ static void io_sq_wq_submit_work(struct work_struct *work) } } -static int io_submit_sqe (struct io_ring_ctx *ctx,const struct sqe_submit *s) +static int io_submit_sqe (struct io_ring_ctx *ctx,const struct sqe_submit *s, +struct io_submit_state *state) {struct io_kiocb *req;ssize_t ret;
@@ -879,7 +957,7 @@ static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s) req->rw.ki_filp =NULL ; - ret =__io_submit_sqe (ctx, req, s, true); + ret =__io_submit_sqe (ctx, req, s, true, state);if (ret == -EAGAIN ) {memcpy (&req->submit, s,sizeof (*s));INIT_WORK (&req->work, io_sq_wq_submit_work);
@@ -892,6 +970,26 @@ static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s)return ret; } +/* +* Batched submission is done, ensure local IO is flushed out. +*/ +static void io_submit_state_end (struct io_submit_state *state) +{ +blk_finish_plug (&state->plug); +io_file_put (state,NULL ); +} + +/* +* Start submission side cache. +*/ +static void io_submit_state_start (struct io_submit_state *state, +struct io_ring_ctx *ctx,unsigned max_ios) +{ +blk_start_plug (&state->plug); + state->file =NULL ; + state->ios_left = max_ios; +} +static void io_commit_sqring (struct io_ring_ctx *ctx) {struct io_sq_ring *ring = ctx->sq_ring;
@@ -954,11 +1052,13 @@ static bool io_get_sqring(struct io_ring_ctx *ctx, struct sqe_submit *s)static int io_ring_submit (struct io_ring_ctx *ctx,unsigned int to_submit) { +struct io_submit_state state, *statep =NULL ;int i, ret =0 , submit =0 ; -struct blk_plug plug; -if (to_submit >IO_PLUG_THRESHOLD ) -blk_start_plug (&plug); +if (to_submit >IO_PLUG_THRESHOLD ) { +io_submit_state_start (&state, ctx, to_submit); + statep = &state; + }for (i =0 ; i < to_submit; i++) {struct sqe_submit s;
@@ -969,7 +1069,7 @@ static int io_ring_submit(struct io_ring_ctx *ctx, unsigned int to_submit) s.has_user = true; s.needs_lock = false; - ret =io_submit_sqe (ctx, &s); + ret =io_submit_sqe (ctx, &s, statep);if (ret) {io_drop_sqring (ctx);break ;
@@ -979,8 +1079,8 @@ static int io_ring_submit(struct io_ring_ctx *ctx, unsigned int to_submit) }io_commit_sqring (ctx); -if (to_submit >IO_PLUG_THRESHOLD ) -blk_finish_plug (&plug); +if (statep) +io_submit_state_end (statep);return submit ? submit : ret; }
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 09/19] io_uring: use fget/fput_many() for file references
Quoted 10 lines · Jens Axboe
Add a separate
io_submit_statestructure, to cache some of the things we need for IO submission.One such example is file reference batching.
io_submit_state. We get as many references as the number of sqes we are submitting, and drop unused ones if we end up switching files. The assumption here is that we're usually only dealing with one fd, and if there are multiple, hopefuly they are at least somewhat ordered. Could trivially be extended to cover multiple fds, if needed.
Quoted 8 lines
On the completion side we do the same thing, except this is trivially done just locally in
io_iopoll_reap().Signed-off-by Jens Axboe <[email protected]>--- fs/io_uring.c | 142 ++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 121 insertions(+), 21 deletions(-)
Cheers,
Hannes
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 10/19 io_uring: batch io_kiocb allocation
In reply to Jens AxboeSimilarly to how we use the state->ios_left to know how many references to get to a file, we can use it to allocate the io_kiocb's we need in bulk.
--- fs/io_uring.c | 45 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 38 insertions(+), 7 deletions(-)
fs/io_uring.c +38 -7
@@ -147,6 +147,13 @@ struct io_kiocb {struct io_submit_state {struct blk_plug plug; +/* +* io_kiocb alloc cache +*/ +void *reqs[IO_IOPOLL_BATCH ]; +unsigned int free_reqs; +unsigned int cur_req; +/* * File reference cache */
@@ -277,20 +284,40 @@ static void io_ring_drop_ctx_refs(struct io_ring_ctx *ctx, unsigned refs)wake_up (&ctx->wait); } -static struct io_kiocb *io_get_req (struct io_ring_ctx *ctx) +static struct io_kiocb *io_get_req (struct io_ring_ctx *ctx, +struct io_submit_state *state) {struct io_kiocb *req;if (!percpu_ref_tryget (&ctx->refs))return NULL ; - req =kmem_cache_alloc (req_cachep, __GFP_NOWARN); -if (req) { - req->ctx = ctx; - req->flags =0 ; -return req; +if (!state) { + req =kmem_cache_alloc (req_cachep, __GFP_NOWARN); +if (unlikely (!req)) + goto out; + }else if (!state->free_reqs) { +size_t sz; +int ret; + + sz =min_t (size_t, state->ios_left,ARRAY_SIZE (state->reqs)); + ret =kmem_cache_alloc_bulk (req_cachep, __GFP_NOWARN, sz, + state->reqs); +if (unlikely (ret <=0 )) + goto out; + state->free_reqs = ret -1 ; + state->cur_req =1 ; + req = state->reqs[0 ]; + }else { + req = state->reqs[state->cur_req]; + state->free_reqs--; + state->cur_req++; } + req->ctx = ctx; + req->flags =0 ; +return req; +out:io_ring_drop_ctx_refs (ctx,1 );return NULL ; }
@@ -951,7 +978,7 @@ static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s,if (unlikely (s->sqe->flags))return -EINVAL ; - req =io_get_req (ctx); + req =io_get_req (ctx, state);if (unlikely (!req))return -EAGAIN ;
@@ -977,6 +1004,9 @@ static void io_submit_state_end(struct io_submit_state *state) {blk_finish_plug (&state->plug);io_file_put (state,NULL ); +if (state->free_reqs) +kmem_cache_free_bulk (req_cachep, state->free_reqs, + &state->reqs[state->cur_req]); }/*
@@ -986,6 +1016,7 @@ static void io_submit_state_start(struct io_submit_state *state, struct io_ring_ctx *ctx, unsigned max_ios) {blk_start_plug (&state->plug); + state->free_reqs =0 ; state->file =NULL ; state->ios_left = max_ios; }
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 10/19] io_uring: batch io_kiocb allocation
Quoted 9 lines · Jens Axboe
Similarly to how we use the
state->ios_leftto know how many references to get to a file, we can use it to allocate the io_kiocb's we need in bulk.Signed-off-by Jens Axboe <[email protected]>--- fs/io_uring.c | 45 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 38 insertions(+), 7 deletions(-)
Cheers,
Hannes
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 11/19 block: implement bio helper to add iter bvec pages to bio
In reply to Jens AxboeFor an ITER_BVEC, we can just iterate the iov and add the pages to the bio directly. This requires that the caller doesn't releases the pages on IO completion, we add a BIO_NO_PAGE_REF flag for that.
The current two callers of bio_iov_iter_get_pages() are updated to check if they need to release pages on completion. This makes them work with bvecs that contain kernel mapped pages already.
--- block/bio.c | 59 ++++++++++++++++++++++++++++++++------- fs/block_dev.c | 5 ++-- fs/iomap.c | 5 ++-- include/linux/blk_types.h | 1 + 4 files changed, 56 insertions(+), 14 deletions(-)
block/bio.c +49 -10
@@ -828,6 +828,23 @@ int bio_add_page(struct bio *bio, struct page *page, }EXPORT_SYMBOL (bio_add_page); +static int __bio_iov_bvec_add_pages (struct bio *bio,struct iov_iter *iter) +{ +const struct bio_vec *bv = iter->bvec; +unsigned int len; +size_t size; + + len =min_t (size_t, bv->bv_len, iter->count); + size =bio_add_page (bio, bv->bv_page, len, + bv->bv_offset + iter->iov_offset); +if (size == len) { +iov_iter_advance (iter, size); +return 0 ; + } + +return -EINVAL ; +} +#define PAGE_PTRS_PER_BVEC (sizeof(struct bio_vec) / sizeof(struct page *))/**
@@ -876,23 +893,43 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) }/** -* bio_iov_iter_get_pages - pin user or kernel pages and add them to a bio +* bio_iov_iter_get_pages - add user or kernel pages to a bio * @bio: bio to add pages to -* @iter: iov iterator describing the region to be mapped +* @iter: iov iterator describing the region to be added +* +* This takes either an iterator pointing to user memory, or one pointing to +* kernel pages (BVEC iterator). If we're adding user pages, we pin them and +* map them into the kernel. On IO completion, the caller should put those +* pages. If we're adding kernel pages, we just have to add the pages to the +* bio directly. We don't grab an extra reference to those pages (the user +* should already have that), and we don't put the page on IO completion. +* The caller needs to check if the bio is flagged BIO_NO_PAGE_REF on IO +* completion. If it isn't, then pages should be released. * -* Pins pages from *iter and appends them to @bio's bvec array. The -* pages will have to be released using put_page() when done. * The function tries, but does not guarantee, to pin as many pages as -* fit into the bio, or are requested in *iter, whatever is smaller. -* If MM encounters an error pinning the requested pages, it stops. -* Error is returned only if 0 pages could be pinned. +* fit into the bio, or are requested in *iter, whatever is smaller. If +* MM encounters an error pinning the requested pages, it stops. Error +* is returned only if 0 pages could be pinned. */ int bio_iov_iter_get_pages (struct bio *bio,struct iov_iter *iter) { +const bool is_bvec =iov_iter_is_bvec (iter);unsigned short orig_vcnt = bio->bi_vcnt; +/* +* If this is a BVEC iter, then the pages are kernel pages. Don't +* release them on IO completion. +*/ +if (is_bvec) +bio_set_flag (bio,BIO_NO_PAGE_REF ); + do { - int ret = __bio_iov_iter_get_pages(bio, iter); + int ret; + + if (is_bvec) + ret = __bio_iov_bvec_add_pages(bio, iter); + else + ret = __bio_iov_iter_get_pages(bio, iter); if (unlikely(ret)) return bio->bi_vcnt > orig_vcnt ? 0 : ret;
@@ -1634,7 +1671,8 @@ static void bio_dirty_fn(struct work_struct *work) next = bio->bi_private;bio_set_pages_dirty (bio); -bio_release_pages (bio); +if (!bio_flagged (bio,BIO_NO_PAGE_REF )) +bio_release_pages (bio);bio_put (bio); } }
@@ -1650,7 +1688,8 @@ void bio_check_pages_dirty(struct bio *bio) goto defer; } -bio_release_pages (bio); +if (!bio_flagged (bio,BIO_NO_PAGE_REF )) +bio_release_pages (bio);bio_put (bio);return ; defer:
fs/block_dev.c +3 -2
@@ -338,8 +338,9 @@ static void blkdev_bio_end_io(struct bio *bio)struct bio_vec *bvec;int i; -bio_for_each_segment_all (bvec, bio, i) -put_page (bvec->bv_page); +if (!bio_flagged (bio,BIO_NO_PAGE_REF )) +bio_for_each_segment_all (bvec, bio, i) +put_page (bvec->bv_page);bio_put (bio); } }
fs/iomap.c +3 -2
@@ -1591,8 +1591,9 @@ static void iomap_dio_bio_end_io(struct bio *bio)struct bio_vec *bvec;int i; -bio_for_each_segment_all (bvec, bio, i) -put_page (bvec->bv_page); +if (!bio_flagged (bio,BIO_NO_PAGE_REF )) +bio_for_each_segment_all (bvec, bio, i) +put_page (bvec->bv_page);bio_put (bio); } }
include/linux/blk_types.h +1 -0
@@ -215,6 +215,7 @@ struct bio {/* * bio flags */ +#define BIO_NO_PAGE_REF 0/* don't put release vec pages */ #define BIO_SEG_VALID 1/* bi_phys_segments valid */ #define BIO_CLONED 2/* doesn't own data */ #define BIO_BOUNCED 3/* bio is a bounce bio */
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 11/19] block: implement bio helper to add iter bvec pages to bio
Quoted 8 lines · Jens Axboe
For an
ITER_BVEC, we can just iterate the iov and add the pages to the bio directly. This requires that the caller doesn't releases the pages on IO completion, we add aBIO_NO_PAGE_REFflag for that.The current two callers of
bio_iov_iter_get_pages()are updated to check if they need to release pages on completion. This makes them work with bvecs that contain kernel mapped pages already.
Quoted 9 lines
Reviewed-by Christoph Hellwig <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- block/bio.c | 59 ++++++++++++++++++++++++++++++++------- fs/block_dev.c | 5 ++-- fs/iomap.c | 5 ++-- include/linux/blk_types.h | 1 + 4 files changed, 56 insertions(+), 14 deletions(-)
Cheers,
Hannes
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 12/19 io_uring: add support for pre-mapped user IO buffers
In reply to Jens AxboeIf we have fixed user buffers, we can map them into the kernel when we setup the io_uring. That avoids the need to do get_user_pages() for each and every IO.
To utilize this feature, the application must call io_uring_register() after having setup an io_uring instance, passing in IORING_REGISTER_BUFFERS as the opcode. The argument must be a pointer to an iovec array, and the nr_args should contain how many iovecs the application wishes to map.
If successful, these buffers are now mapped into the kernel, eligible for IO. To use these fixed buffers, the application must use the IORING_OP_READ_FIXED and IORING_OP_WRITE_FIXED opcodes, and then set sqe->index to the desired buffer index. sqe->addr..sqe->addr+seq->len must point to somewhere inside the indexed buffer.
The application may register buffers throughout the lifetime of the io_uring instance. It can call io_uring_register() with IORING_UNREGISTER_BUFFERS as the opcode to unregister the current set of buffers, and then register a new set. The application need not unregister buffers explicitly before shutting down the io_uring instance.
It's perfectly valid to setup a larger buffer, and then sometimes only use parts of it for an IO. As long as the range is within the originally mapped region, it will work just fine.
For now, buffers must not be file backed. If file backed buffers are passed in, the registration will fail with -1/EOPNOTSUPP. This restriction may be relaxed in the future.
RLIMIT_MEMLOCK is used to check how much memory we can pin. A somewhat arbitrary 1G per buffer size is also imposed.
--- arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + fs/io_uring.c | 356 ++++++++++++++++++++++++- include/linux/sched/user.h | 2 +- include/linux/syscalls.h | 2 + include/uapi/asm-generic/unistd.h | 4 +- include/uapi/linux/io_uring.h | 13 +- kernel/sys_ni.c | 1 + 8 files changed, 363 insertions(+), 17 deletions(-)
arch/x86/entry/syscalls/syscall_32.tbl +1 -0
@@ -400,3 +400,4 @@ 386 i386 rseq sys_rseq __ia32_sys_rseq 425 i386 io_uring_setup sys_io_uring_setup __ia32_sys_io_uring_setup 426 i386 io_uring_enter sys_io_uring_enter __ia32_sys_io_uring_enter +427 i386 io_uring_register sys_io_uring_register __ia32_sys_io_uring_register
arch/x86/entry/syscalls/syscall_64.tbl +1 -0
@@ -345,6 +345,7 @@ 334 common rseq __x64_sys_rseq 425 common io_uring_setup __x64_sys_io_uring_setup 426 common io_uring_enter __x64_sys_io_uring_enter +427 common io_uring_register __x64_sys_io_uring_register # # x32-specific system call numbers start at 512 to avoid cache impact
fs/io_uring.c +342 -14
449 lines not on this page · raw message
include/linux/sched/user.h +1 -1
@@ -40,7 +40,7 @@ struct user_struct {kuid_t uid;#if defined(CONFIG_PERF_EVENTS ) || defined(CONFIG_BPF_SYSCALL ) || \ - defined(CONFIG_NET ) + defined(CONFIG_NET ) || defined(CONFIG_IO_URING )atomic_long_t locked_vm;#endif
include/linux/syscalls.h +2 -0
@@ -315,6 +315,8 @@ asmlinkage long sys_io_uring_setup(u32 entries,asmlinkage long sys_io_uring_enter (unsigned int fd,u32 to_submit,u32 min_complete,u32 flags,const sigset_t __user *sig,size_t sigsz); +asmlinkage long sys_io_uring_register (unsigned int fd,unsigned int op, +void __user *arg,unsigned int nr_args);/* fs/xattr.c */ asmlinkage long sys_setxattr(const char __user *path, const char __user *name,
include/uapi/asm-generic/unistd.h +3 -1
@@ -744,9 +744,11 @@ __SYSCALL(__NR_kexec_file_load, sys_kexec_file_load) __SYSCALL(__NR_io_uring_setup, sys_io_uring_setup)#define __NR_io_uring_enter 426 __SYSCALL(__NR_io_uring_enter, sys_io_uring_enter) +#define __NR_io_uring_register 427 +__SYSCALL(__NR_io_uring_register, sys_io_uring_register)#undef __NR_syscalls -#define __NR_syscalls 427 +#define __NR_syscalls 428/* * 32 bit systems traditionally used different
include/uapi/linux/io_uring.h +12 -1
@@ -27,7 +27,10 @@ struct io_uring_sqe {__u32 fsync_flags; };__u64 user_data;/* data to be passed back at completion time */ -__u64 __pad2[3 ]; +union { +__u16 buf_index;/* index into fixed buffers, if used */ +__u64 __pad2[3 ]; + }; };/*
@@ -39,6 +42,8 @@ struct io_uring_sqe {#define IORING_OP_READV 1#define IORING_OP_WRITEV 2#define IORING_OP_FSYNC 3 +#define IORING_OP_READ_FIXED 4 +#define IORING_OP_WRITE_FIXED 5/* * sqe->fsync_flags
@@ -103,4 +108,10 @@ struct io_uring_params { struct io_cqring_offsets cq_off; }; +/* +* io_uring_register(2) opcodes and arguments +*/ +#define IORING_REGISTER_BUFFERS 0 +#define IORING_UNREGISTER_BUFFERS 1 +#endif
kernel/sys_ni.c +1 -0
@@ -48,6 +48,7 @@ COND_SYSCALL_COMPAT(io_getevents);COND_SYSCALL_COMPAT (io_pgetevents);COND_SYSCALL (io_uring_setup);COND_SYSCALL (io_uring_enter); +COND_SYSCALL (io_uring_register);/* fs/xattr.c */
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 12/19] io_uring: add support for pre-mapped user IO buffers
Quoted 33 lines · Jens Axboe
If we have fixed user buffers, we can map them into the kernel when we setup the
io_uring. That avoids the need to doget_user_pages()for each and every IO.To utilize this feature, the application must call
io_uring_register()after having setup anio_uringinstance, passing inIORING_REGISTER_BUFFERSas the opcode. The argument must be a pointer to an iovec array, and thenr_argsshould contain how many iovecs the application wishes to map.If successful, these buffers are now mapped into the kernel, eligible for IO. To use these fixed buffers, the application must use the
IORING_OP_READ_FIXEDandIORING_OP_WRITE_FIXEDopcodes, and then setsqe->indexto the desired buffer index. sqe->addr..sqe->addr+seq->len must point to somewhere inside the indexed buffer.The application may register buffers throughout the lifetime of the
io_uringinstance. It can callio_uring_register()withIORING_UNREGISTER_BUFFERSas the opcode to unregister the current set of buffers, and then register a new set. The application need not unregister buffers explicitly before shutting down theio_uringinstance.It's perfectly valid to setup a larger buffer, and then sometimes only use parts of it for an IO. As long as the range is within the originally mapped region, it will work just fine.
For now, buffers must not be file backed. If file backed buffers are passed in, the registration will fail with -1/EOPNOTSUPP. This restriction may be relaxed in the future.
RLIMIT_MEMLOCKis used to check how much memory we can pin. A somewhat arbitrary 1G per buffer size is also imposed.
[...]
fs/io_uring.c static int io_import_iovec(struct io_ring_ctx *ctx, int rw, · line 766
Quoted 8 lines
static int io_import_iovec(struct
io_ring_ctx*ctx, int rw,
const structsqe_submit*s, struct iovec **iovec, structiov_iter*iter)fs/io_uring.c +1
-0@@ -711,6 +763,15 @@ static int io_import_iovec(struct io_ring_ctx *ctx, int rw,const struct io_uring_sqe *sqe = s->sqe;void __user *buf =u64_to_user_ptr (READ_ONCE (sqe->addr));size_t sqe_len =READ_ONCE (sqe->len); +u8 opcode;
(You could add a comment here if you want, something like "We're reading ->opcode for the second time, but the first read doesn't care whether it's _FIXED or not, so it doesn't matter whether ->opcode changes concurrently. The first read does care about whether it is a READ or a WRITE, so we don't trust this read for that purpose and instead let the caller pass in the read/write flag.")
Quoted 10 lines
fs/io_uring.c +7
-0+ opcode =READ_ONCE (sqe->opcode); +if (opcode ==IORING_OP_READ_FIXED || + opcode ==IORING_OP_WRITE_FIXED ) { +ssize_t ret =io_import_fixed (ctx, rw, sqe, iter); + *iovec =NULL ; +return ret; + }if (!s->has_user)return EFAULT ;
[...]
fs/io_uring.c static unsigned long ring_pages(unsigned sq_entries, unsigned cq_entries) · line 1344
Quoted 12 lines
fs/io_uring.c +8
-0@@ -1242,6 +1334,187 @@ static unsigned long ring_pages(unsigned sq_entries, unsigned cq_entries)return (bytes +PAGE_SIZE -1 ) /PAGE_SIZE ; } +static int io_sqe_buffer_unregister (struct io_ring_ctx *ctx) +{ +int i, j; + +if (!ctx->user_bufs) +return -ENXIO ; + +for (i =0 ; i < ctx->sq_entries; i++) {
->sq_entries? Shouldn't this be ->nr_user_bufs?
fs/io_uring.c static unsigned long ring_pages(unsigned sq_entries, unsigned cq_entries) · line 1356
Quoted 12 lines
fs/io_uring.c +12
-0+struct io_mapped_ubuf *imu = &ctx->user_bufs[i]; + +for (j =0 ; j < imu->nr_bvecs; j++) +put_page (imu->bvec[j].bv_page); + +io_unaccount_mem (ctx->user, imu->nr_bvecs); +kfree (imu->bvec); + imu->nr_bvecs =0 ; + } + +kfree (ctx->user_bufs); + ctx->user_bufs =NULL ;
(It isn't really necessary, but you could set nr_user_bufs=0 here.)
Quoted 2 lines
(quoted diff) +2
-0+ return 0; +}
[...]
Quoted 38 lines
fs/io_uring.c +38
-0+static int io_sqe_buffer_register (struct io_ring_ctx *ctx,void __user *arg, +unsigned nr_args) +{ +struct vm_area_struct **vmas =NULL ; +struct page **pages =NULL ; +int i, j, got_pages =0 ; +int ret = -EINVAL ; + +if (ctx->user_bufs) +return -EBUSY ; +if (!nr_args || nr_args >UIO_MAXIOV ) +return -EINVAL ; + + ctx->user_bufs =kcalloc (nr_args,sizeof (struct io_mapped_ubuf ), +GFP_KERNEL ); +if (!ctx->user_bufs) +return -ENOMEM ; + +for (i =0 ; i < nr_args; i++) { +struct io_mapped_ubuf *imu = &ctx->user_bufs[i]; +unsigned long off, start, end, ubuf; +int pret, nr_pages; +struct iovec iov; +size_t size; + + ret =io_copy_iov (ctx, &iov, arg, i); +if (ret) +break ; + +/* +* Don't impose further limits on the size and buffer +* constraints here, we'll -EINVAL later when IO is +* submitted if they are wrong. +*/ + ret = -EFAULT ; +if (!iov.iov_base || !iov.iov_len) + goto err; +
fs/io_uring.c static unsigned long ring_pages(unsigned sq_entries, unsigned cq_entries) · line 1432
Quoted 10 lines
fs/io_uring.c +10
-0+/* arbitrary limit, but we need something */ +if (iov.iov_len >SZ_1G ) + goto err; + + ubuf = (unsigned long ) iov.iov_base; + end = (ubuf + iov.iov_len +PAGE_SIZE -1 ) >>PAGE_SHIFT ; + start = ubuf >>PAGE_SHIFT ; + nr_pages = end - start; + + ret =io_account_mem (ctx->user, nr_pages);
Technically, this accounting is probably a bit off; I think if you pass in a vector of 4K areas from 1G hugepages, you're going to pin factor 0x40000 more memory than you think you're pinning. (get_user_pages() counts references against the head page of a compound page; nothing in the kernel can tell afterwards which part of the hugepage you're using.) I'm not sure how much of a problem that is, but it should probably at least be documented. Unless I'm just missing something?
Quoted 17 lines
fs/io_uring.c +17
-0+if (ret) + goto err; + +if (!pages || nr_pages > got_pages) { +kfree (vmas); +kfree (pages); + pages =kmalloc_array (nr_pages,sizeof (struct page *), +GFP_KERNEL ); + vmas =kmalloc_array (nr_pages, +sizeof (struct vma_area_struct *), +GFP_KERNEL ); +if (!pages || !vmas) { + ret = -ENOMEM ; +io_unaccount_mem (ctx->user, nr_pages); + goto err; + } + got_pages = nr_pages;
fs/io_uring.c static unsigned long ring_pages(unsigned sq_entries, unsigned cq_entries) · line 1459
Quoted 10 lines
fs/io_uring.c +10
-0+ } + + imu->bvec =kmalloc_array (nr_pages,sizeof (struct bio_vec ), +GFP_KERNEL ); +if (!imu->bvec) { +io_unaccount_mem (ctx->user, nr_pages); + goto err; + } + +down_write (¤t->mm->mmap_sem);
Weren't you planning to make this down_read()?
Quoted 17 lines
fs/io_uring.c +17
-0+ pret =get_user_pages_longterm (ubuf, nr_pages,FOLL_WRITE , + pages, vmas); +if (pret == nr_pages) { +/* don't support file backed memory */ +for (j =0 ; j < nr_pages; j++) { +struct vm_area_struct *vma = vmas[j]; + +if (vma->vm_file && + !is_file_hugepages (vma->vm_file)) { + ret = -EOPNOTSUPP ; +break ; + } + } + }else { + ret = pret <0 ? pret : -EFAULT ; + } +up_write (¤t->mm->mmap_sem);
[...]
Quoted 1 line
+}
[...]
include/linux/sched/user.h struct user_struct { · line 45
Quoted 12 lines
include/linux/sched/user.h +1
-1@@ -40,7 +40,7 @@ struct user_struct {kuid_t uid;#if defined(CONFIG_PERF_EVENTS ) || defined(CONFIG_BPF_SYSCALL ) || \ - defined(CONFIG_NET ) + defined(CONFIG_NET ) || defined(CONFIG_IO_URING )atomic_long_t locked_vm;#endif
You're already using locked_vm in patch 5, right? I think that means that from patch 5 up to this patch, some kernel configs will fail to build.
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 12/19] io_uring: add support for pre-mapped user IO buffers
Quoted 35 lines · Jann Horn
Quoted 33 lines · Jens Axboe
If we have fixed user buffers, we can map them into the kernel when we setup the
io_uring. That avoids the need to doget_user_pages()for each and every IO.To utilize this feature, the application must call
io_uring_register()after having setup anio_uringinstance, passing inIORING_REGISTER_BUFFERSas the opcode. The argument must be a pointer to an iovec array, and thenr_argsshould contain how many iovecs the application wishes to map.If successful, these buffers are now mapped into the kernel, eligible for IO. To use these fixed buffers, the application must use the
IORING_OP_READ_FIXEDandIORING_OP_WRITE_FIXEDopcodes, and then setsqe->indexto the desired buffer index. sqe->addr..sqe->addr+seq->len must point to somewhere inside the indexed buffer.The application may register buffers throughout the lifetime of the
io_uringinstance. It can callio_uring_register()withIORING_UNREGISTER_BUFFERSas the opcode to unregister the current set of buffers, and then register a new set. The application need not unregister buffers explicitly before shutting down theio_uringinstance.It's perfectly valid to setup a larger buffer, and then sometimes only use parts of it for an IO. As long as the range is within the originally mapped region, it will work just fine.
For now, buffers must not be file backed. If file backed buffers are passed in, the registration will fail with -1/EOPNOTSUPP. This restriction may be relaxed in the future.
RLIMIT_MEMLOCKis used to check how much memory we can pin. A somewhat arbitrary 1G per buffer size is also imposed.[...]
Quoted 15 lines
fs/io_uring.c static int io_import_iovec(struct io_ring_ctx *ctx, int rw, · line 766
Quoted 8 lines
static int io_import_iovec(struct
io_ring_ctx*ctx, int rw,
const structsqe_submit*s, struct iovec **iovec, structiov_iter*iter)fs/io_uring.c +1
-0@@ -711,6 +763,15 @@ static int io_import_iovec(struct io_ring_ctx *ctx, int rw,const struct io_uring_sqe *sqe = s->sqe;void __user *buf =u64_to_user_ptr (READ_ONCE (sqe->addr));size_t sqe_len =READ_ONCE (sqe->len); +u8 opcode;(You could add a comment here if you want, something like "We're reading ->opcode for the second time, but the first read doesn't care whether it's
_FIXEDor not, so it doesn't matter whether ->opcode changes concurrently. The first read does care about whether it is a READ or a WRITE, so we don't trust this read for that purpose and instead let the caller pass in the read/write flag.")
Sure, I can add that.
Quoted 10 lines
fs/io_uring.c static unsigned long ring_pages(unsigned sq_entries, unsigned cq_entries) · line 1344
Quoted 8 lines
fs/io_uring.c +8
-0+static int io_sqe_buffer_unregister (struct io_ring_ctx *ctx) +{ +int i, j; + +if (!ctx->user_bufs) +return -ENXIO ; + +for (i =0 ; i < ctx->sq_entries; i++) {->sq_entries? Shouldn't this be ->nr_user_bufs?
It should! I swear I already fixed that, odd. Maybe that was somewhere else...
Quoted 6 lines
Quoted 6 lines
fs/io_uring.c +6
-0+struct io_mapped_ubuf *imu = &ctx->user_bufs[i]; + +for (j =0 ; j < imu->nr_bvecs; j++) +put_page (imu->bvec[j].bv_page); + +io_unaccount_mem (ctx->user, imu->nr_bvecs);
Quoted 8 lines
fs/io_uring.c static unsigned long ring_pages(unsigned sq_entries, unsigned cq_entries) · line 1356
Quoted 6 lines
fs/io_uring.c +6
-0+kfree (imu->bvec); + imu->nr_bvecs =0 ; + } + +kfree (ctx->user_bufs); + ctx->user_bufs =NULL ;(It isn't really necessary, but you could set nr_user_bufs=0 here.)
Doesn't hurt to be defensive.
Quoted 52 lines
Quoted 2 lines
(quoted diff) +2
-0+ return 0; +}[...]
Quoted 38 lines
fs/io_uring.c +38
-0+static int io_sqe_buffer_register (struct io_ring_ctx *ctx,void __user *arg, +unsigned nr_args) +{ +struct vm_area_struct **vmas =NULL ; +struct page **pages =NULL ; +int i, j, got_pages =0 ; +int ret = -EINVAL ; + +if (ctx->user_bufs) +return -EBUSY ; +if (!nr_args || nr_args >UIO_MAXIOV ) +return -EINVAL ; + + ctx->user_bufs =kcalloc (nr_args,sizeof (struct io_mapped_ubuf ), +GFP_KERNEL ); +if (!ctx->user_bufs) +return -ENOMEM ; + +for (i =0 ; i < nr_args; i++) { +struct io_mapped_ubuf *imu = &ctx->user_bufs[i]; +unsigned long off, start, end, ubuf; +int pret, nr_pages; +struct iovec iov; +size_t size; + + ret =io_copy_iov (ctx, &iov, arg, i); +if (ret) +break ; + +/* +* Don't impose further limits on the size and buffer +* constraints here, we'll -EINVAL later when IO is +* submitted if they are wrong. +*/ + ret = -EFAULT ; +if (!iov.iov_base || !iov.iov_len) + goto err; +fs/io_uring.c static unsigned long ring_pages(unsigned sq_entries, unsigned cq_entries) · line 1432
Quoted 10 lines
fs/io_uring.c +10
-0+/* arbitrary limit, but we need something */ +if (iov.iov_len >SZ_1G ) + goto err; + + ubuf = (unsigned long ) iov.iov_base; + end = (ubuf + iov.iov_len +PAGE_SIZE -1 ) >>PAGE_SHIFT ; + start = ubuf >>PAGE_SHIFT ; + nr_pages = end - start; + + ret =io_account_mem (ctx->user, nr_pages);
Quoted 8 lines
Technically, this accounting is probably a bit off; I think if you pass in a vector of 4K areas from 1G hugepages, you're going to pin factor
0x40000more memory than you think you're pinning. (get_user_pages()counts references against the head page of a compound page; nothing in the kernel can tell afterwards which part of the hugepage you're using.) I'm not sure how much of a problem that is, but it should probably at least be documented. Unless I'm just missing something?
No I think you are right, it doesn't account for the hugepage size if you pass in huge pages. I'll fix that up.
Quoted 21 lines
Quoted 21 lines
fs/io_uring.c +21
-0+if (ret) + goto err; + +if (!pages || nr_pages > got_pages) { +kfree (vmas); +kfree (pages); + pages =kmalloc_array (nr_pages,sizeof (struct page *), +GFP_KERNEL ); + vmas =kmalloc_array (nr_pages, +sizeof (struct vma_area_struct *), +GFP_KERNEL ); +if (!pages || !vmas) { + ret = -ENOMEM ; +io_unaccount_mem (ctx->user, nr_pages); + goto err; + } + got_pages = nr_pages; + } + + imu->bvec =kmalloc_array (nr_pages,sizeof (struct bio_vec ), +GFP_KERNEL );
Quoted 8 lines
fs/io_uring.c static unsigned long ring_pages(unsigned sq_entries, unsigned cq_entries) · line 1459
Quoted 6 lines
fs/io_uring.c +6
-0+if (!imu->bvec) { +io_unaccount_mem (ctx->user, nr_pages); + goto err; + } + +down_write (¤t->mm->mmap_sem);Weren't you planning to make this
down_read()?
I think I accidentally messed that up when going back to not using FOLL_ANON. Fixed (again), thanks.
Quoted 27 lines
Quoted 17 lines
fs/io_uring.c +17
-0+ pret =get_user_pages_longterm (ubuf, nr_pages,FOLL_WRITE , + pages, vmas); +if (pret == nr_pages) { +/* don't support file backed memory */ +for (j =0 ; j < nr_pages; j++) { +struct vm_area_struct *vma = vmas[j]; + +if (vma->vm_file && + !is_file_hugepages (vma->vm_file)) { + ret = -EOPNOTSUPP ; +break ; + } + } + }else { + ret = pret <0 ? pret : -EFAULT ; + } +up_write (¤t->mm->mmap_sem);[...]
Quoted 1 line
+}
[...]
Quoted 7 lines
include/linux/sched/user.h +0
-0@@ -40,7 +40,7 @@ struct user_struct {kuid_t uid;
Quoted 9 lines
include/linux/sched/user.h struct user_struct { · line 45
Quoted 5 lines
include/linux/sched/user.h +1
-1#if defined(CONFIG_PERF_EVENTS ) || defined(CONFIG_BPF_SYSCALL ) || \ - defined(CONFIG_NET ) + defined(CONFIG_NET ) || defined(CONFIG_IO_URING )atomic_long_t locked_vm;#endif You're already using
locked_vmin patch 5, right? I think that means that from patch 5 up to this patch, some kernel configs will fail to build.
Good point, I need to do this earlier now.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 12/19] io_uring: add support for pre-mapped user IO buffers
Quoted 11 lines · Jens Axboe
Quoted 8 lines
Technically, this accounting is probably a bit off; I think if you pass in a vector of 4K areas from 1G hugepages, you're going to pin factor
0x40000more memory than you think you're pinning. (get_user_pages()counts references against the head page of a compound page; nothing in the kernel can tell afterwards which part of the hugepage you're using.) I'm not sure how much of a problem that is, but it should probably at least be documented. Unless I'm just missing something?No I think you are right, it doesn't account for the hugepage size if you pass in huge pages. I'll fix that up.
I made a note of it in the man page, I don't think this is unreasonable (or unexpected) behavior.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 12/19] io_uring: add support for pre-mapped user IO buffers
In reply to Jens AxboeQuoted 35 lines · Jens Axboe
If we have fixed user buffers, we can map them into the kernel when we setup the
io_uring. That avoids the need to doget_user_pages()for each and every IO.To utilize this feature, the application must call
io_uring_register()after having setup anio_uringinstance, passing inIORING_REGISTER_BUFFERSas the opcode. The argument must be a pointer to an iovec array, and thenr_argsshould contain how many iovecs the application wishes to map.If successful, these buffers are now mapped into the kernel, eligible for IO. To use these fixed buffers, the application must use the
IORING_OP_READ_FIXEDandIORING_OP_WRITE_FIXEDopcodes, and then setsqe->indexto the desired buffer index. sqe->addr..sqe->addr+seq->len must point to somewhere inside the indexed buffer.The application may register buffers throughout the lifetime of the
io_uringinstance. It can callio_uring_register()withIORING_UNREGISTER_BUFFERSas the opcode to unregister the current set of buffers, and then register a new set. The application need not unregister buffers explicitly before shutting down theio_uringinstance.It's perfectly valid to setup a larger buffer, and then sometimes only use parts of it for an IO. As long as the range is within the originally mapped region, it will work just fine.
For now, buffers must not be file backed. If file backed buffers are passed in, the registration will fail with -1/EOPNOTSUPP. This restriction may be relaxed in the future.
RLIMIT_MEMLOCKis used to check how much memory we can pin. A somewhat arbitrary 1G per buffer size is also imposed.Signed-off-by Jens Axboe <[email protected]>
Quoted 11 lines
--- arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + fs/io_uring.c | 356 ++++++++++++++++++++++++- include/linux/sched/user.h | 2 +- include/linux/syscalls.h | 2 + include/uapi/asm-generic/unistd.h | 4 +- include/uapi/linux/io_uring.h | 13 +- kernel/sys_ni.c | 1 + 8 files changed, 363 insertions(+), 17 deletions(-)
Cheers,
Hannes
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 13/19 net: split out functions related to registering inflight socket files
In reply to Jens AxboeWe need this functionality for the io_uring file registration, but we cannot rely on it since CONFIG_UNIX can be modular. Move the helpers to a separate file, that's always builtin to the kernel if CONFIG_UNIX is m/y.
No functional changes in this patch, just moving code around.
--- include/net/af_unix.h | 1 + net/unix/Kconfig | 5 ++ net/unix/Makefile | 2 + net/unix/af_unix.c | 63 +----------------- net/unix/garbage.c | 71 +------------------- net/unix/scm.c | 146 ++++++++++++++++++++++++++++++++++++++++++ net/unix/scm.h | 10 +++ 7 files changed, 168 insertions(+), 130 deletions(-) create mode 100644 net/unix/scm.c create mode 100644 net/unix/scm.h
include/net/af_unix.h +1 -0
@@ -10,6 +10,7 @@void unix_inflight (struct user_struct *user,struct file *fp);void unix_notinflight (struct user_struct *user,struct file *fp); +void unix_destruct_scm (struct sk_buff *skb);void unix_gc (void );void wait_for_unix_gc (void );struct sock *unix_get_socket (struct file *filp);
net/unix/Kconfig +5 -0
@@ -19,6 +19,11 @@ config UNIX Say Y unless you know what you are doing. +config UNIX_SCM + bool + depends on UNIX + default y + config UNIX_DIAG tristate "UNIX: socket monitoring interface" depends on UNIX
net/unix/Makefile +2 -0
@@ -10,3 +10,5 @@ unix-$(CONFIG_SYSCTL) += sysctl_net_unix.o obj-$(CONFIG_UNIX_DIAG) += unix_diag.o unix_diag-y := diag.o + +obj-$(CONFIG_UNIX_SCM) += scm.o
net/unix/af_unix.c +2 -61
75 lines not on this page · raw message
net/unix/garbage.c +2 -69
82 lines not on this page · raw message
net/unix/scm.c new file +146 -0
146 lines not on this page · raw message
net/unix/scm.h new file +10 -0
@@ -0,0 +1,10 @@ +#ifndef NET_UNIX_SCM_H +#define NET_UNIX_SCM_H + +extern struct list_head gc_inflight_list; +extern spinlock_t unix_gc_lock; + +int unix_attach_fds (struct scm_cookie *scm,struct sk_buff *skb); +void unix_detach_fds (struct scm_cookie *scm,struct sk_buff *skb); + +#endif
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 13/19] net: split out functions related to registering inflight socket files
From: Jens Axboe <[email protected]>
Date: Fri, 8 Feb 2019 10:34:17 -0700
Quoted 10 lines
We need this functionality for the
io_uringfile registration, but we cannot rely on it sinceCONFIG_UNIXcan be modular. Move the helpers to a separate file, that's always builtin to the kernel ifCONFIG_UNIXis m/y.No functional changes in this patch, just moving code around.
Cc David S. Miller <[email protected]>Signed-off-by Jens Axboe <[email protected]>
No objections on my end, feel free to merge this in with your series.
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 13/19] net: split out functions related to registering inflight socket files
Quoted 10 lines · David Miller
From: Jens Axboe <[email protected]>
Date: Fri, 8 Feb 2019 10:34:17 -0700Quoted 7 lines
We need this functionality for the
io_uringfile registration, but we cannot rely on it sinceCONFIG_UNIXcan be modular. Move the helpers to a separate file, that's always builtin to the kernel ifCONFIG_UNIXis m/y.No functional changes in this patch, just moving code around.
Quoted 7 lines
Quoted 3 lines
Cc David S. Miller <[email protected]>Signed-off-by Jens Axboe <[email protected]>No objections on my end, feel free to merge this in with your series.
Acked-by David S. Miller <[email protected]>
Great, thanks Dave! I'll add your ack.
Signature · 1 line
Jens Axboe
Re: [PATCH 13/19] net: split out functions related to registering inflight socket files
In reply to Jens AxboeQuoted 19 lines · Jens Axboe
We need this functionality for the
io_uringfile registration, but we cannot rely on it sinceCONFIG_UNIXcan be modular. Move the helpers to a separate file, that's always builtin to the kernel ifCONFIG_UNIXis m/y.No functional changes in this patch, just moving code around.
Cc David S. Miller <[email protected]>Signed-off-by Jens Axboe <[email protected]>--- include/net/af_unix.h | 1 + net/unix/Kconfig | 5 ++ net/unix/Makefile | 2 + net/unix/af_unix.c | 63 +----------------- net/unix/garbage.c | 71 +------------------- net/unix/scm.c | 146 ++++++++++++++++++++++++++++++++++++++++++ net/unix/scm.h | 10 +++ 7 files changed, 168 insertions(+), 130 deletions(-)
Quoted 3 lines
create mode 100644
net/unix/scm.c
create mode 100644net/unix/scm.h
Cheers,
Hannes
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 14/19 io_uring: add file set registration
In reply to Jens AxboeWe normally have to fget/fput for each IO we do on a file. Even with the batching we do, the cost of the atomic inc/dec of the file usage count adds up.
This adds IORING_REGISTER_FILES, and IORING_UNREGISTER_FILES opcodes for the io_uring_register(2) system call. The arguments passed in must be an array of __s32 holding file descriptors, and nr_args should hold the number of file descriptors the application wishes to pin for the duration of the io_uring instance (or until IORING_UNREGISTER_FILES is called).
When used, the application must set IOSQE_FIXED_FILE in the sqe->flags member. Then, instead of setting sqe->fd to the real fd, it sets sqe->fd to the index in the array passed in to IORING_REGISTER_FILES.
Files are automatically unregistered when the io_uring instance is torn down. An application need only unregister if it wishes to register a new set of fds.
--- fs/io_uring.c | 256 ++++++++++++++++++++++++++++++---- include/uapi/linux/io_uring.h | 9 +- 2 files changed, 235 insertions(+), 30 deletions(-)
fs/io_uring.c +227 -29
@@ -29,6 +29,7 @@#include <linux/net.h> #include <net/sock.h> #include <net/af_unix.h> +#include <net/scm.h> #include <linux/anon_inodes.h> #include <linux/sched/mm.h> #include <linux/uaccess.h>
@@ -41,6 +42,7 @@#include "internal.h" #define IORING_MAX_ENTRIES 4096 +#define IORING_MAX_FIXED_FILES 1024struct io_uring {u32 head____cacheline_aligned_in_smp ;
@@ -102,6 +104,14 @@ struct io_ring_ctx {struct fasync_struct *cq_fasync; }____cacheline_aligned_in_smp ; +/* +* If used, fixed file set. Writers must ensure that ->refs is dead, +* readers must ensure that ->refs is alive as long as the file* is +* used. Only updated through io_uring_register(2). +*/ +struct file **user_files; +unsigned nr_user_files; +/* if used, fixed mapped user buffers */ unsigned nr_user_bufs;struct io_mapped_ubuf *user_bufs;
@@ -149,6 +159,7 @@ struct io_kiocb {unsigned int flags;#define REQ_F_FORCE_NONBLOCK 1/* inline submission attempt */ #define REQ_F_IOPOLL_COMPLETED 2/* polled IO has completed */ +#define REQ_F_FIXED_FILE 4/* ctx owns file */ u64 user_data;u64 error;
@@ -376,15 +387,17 @@ static void io_iopoll_complete(struct io_ring_ctx *ctx, unsigned int *nr_events,* Batched puts of the same file, to avoid dirtying the * file usage count multiple times, if avoidable. */ -if (!file) { - file = req->rw.ki_filp; - file_count =1 ; - }else if (file == req->rw.ki_filp) { - file_count++; - }else { -fput_many (file, file_count); - file = req->rw.ki_filp; - file_count =1 ; +if (!(req->flags &REQ_F_FIXED_FILE )) { +if (!file) { + file = req->rw.ki_filp; + file_count =1 ; + }else if (file == req->rw.ki_filp) { + file_count++; + }else { +fput_many (file, file_count); + file = req->rw.ki_filp; + file_count =1 ; + } }if (to_free ==ARRAY_SIZE (reqs))
@@ -516,13 +529,19 @@ static void kiocb_end_write(struct kiocb *kiocb) } } +static void io_fput (struct io_kiocb *req) +{ +if (!(req->flags &REQ_F_FIXED_FILE )) +fput (req->rw.ki_filp); +} +static void io_complete_rw (struct kiocb *kiocb,long res,long res2) { struct io_kiocb *req = container_of(kiocb, struct io_kiocb, rw);kiocb_end_write (kiocb); -fput (kiocb->ki_filp); +io_fput (req);io_cqring_add_event (req->ctx, req->user_data, res,0 );io_free_req (req); }
@@ -638,19 +657,29 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe, {struct io_ring_ctx *ctx = req->ctx;struct kiocb *kiocb = &req->rw; -unsigned ioprio; +unsigned ioprio, flags;int fd, ret;/* For -EAGAIN retry, everything is already prepped */ if (kiocb->ki_filp)return 0 ; + flags =READ_ONCE (sqe->flags); fd =READ_ONCE (sqe->fd); - kiocb->ki_filp =io_file_get (state, fd); -if (unlikely (!kiocb->ki_filp)) -return -EBADF ; -if (force_nonblock && !io_file_supports_async (kiocb->ki_filp)) - force_nonblock = false; + +if (flags &IOSQE_FIXED_FILE ) { +if (unlikely (!ctx->user_files || + (unsigned ) fd >= ctx->nr_user_files)) +return -EBADF ; + kiocb->ki_filp = ctx->user_files[fd]; + req->flags |=REQ_F_FIXED_FILE ; + }else { + kiocb->ki_filp =io_file_get (state, fd); +if (unlikely (!kiocb->ki_filp)) +return -EBADF ; +if (force_nonblock && !io_file_supports_async (kiocb->ki_filp)) + force_nonblock = false; + } kiocb->ki_pos =READ_ONCE (sqe->off); kiocb->ki_flags =iocb_flags (kiocb->ki_filp); kiocb->ki_hint =ki_hint_validate (file_write_hint (kiocb->ki_filp));
@@ -690,10 +719,14 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe, }return 0 ; out_fput: -/* in case of error, we didn't use this file reference. drop it. */ -if (state) - state->used_refs--; -io_file_put (state, kiocb->ki_filp); +if (!(flags &IOSQE_FIXED_FILE )) { +/* +* in case of error, we didn't use this file reference. drop it. +*/ +if (state) + state->used_refs--; +io_file_put (state, kiocb->ki_filp); + }return ret; }
@@ -825,7 +858,7 @@ static ssize_t io_read(struct io_kiocb *req, const struct sqe_submit *s, out_fput:/* Hold on to the file for -EAGAIN */ if (unlikely (ret && ret != -EAGAIN )) -fput (file); +io_fput (req);return ret; }
@@ -879,7 +912,7 @@ static ssize_t io_write(struct io_kiocb *req, const struct sqe_submit *s,kfree (iovec); out_fput:if (unlikely (ret)) -fput (file); +io_fput (req);return ret; }
@@ -905,7 +938,7 @@ static int io_fsync(struct io_kiocb *req, const struct io_uring_sqe *sqe,loff_t sqe_off =READ_ONCE (sqe->off);loff_t sqe_len =READ_ONCE (sqe->len);loff_t end = sqe_off + sqe_len; -unsigned fsync_flags; +unsigned fsync_flags, flags;struct file *file;int ret, fd;
@@ -923,14 +956,23 @@ static int io_fsync(struct io_kiocb *req, const struct io_uring_sqe *sqe,return -EINVAL ; fd =READ_ONCE (sqe->fd); - file =fget (fd); + flags =READ_ONCE (sqe->flags); + +if (flags &IOSQE_FIXED_FILE ) { +if (unlikely (!ctx->user_files || fd >= ctx->nr_user_files)) +return -EBADF ; + file = ctx->user_files[fd]; + }else { + file =fget (fd); + }if (unlikely (!file))return -EBADF ; ret =vfs_fsync_range (file, sqe_off, end >0 ? end :LLONG_MAX , fsync_flags &IORING_FSYNC_DATASYNC ); -fput (file); +if (!(flags &IOSQE_FIXED_FILE )) +fput (file);io_cqring_add_event (ctx, sqe->user_data, ret,0 );io_free_req (req);return 0 ;
@@ -1067,7 +1109,7 @@ static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s,ssize_t ret;/* enforce forwards compatibility on users */ -if (unlikely (s->sqe->flags)) +if (unlikely (s->sqe->flags & ~IOSQE_FIXED_FILE ))return -EINVAL ; req =io_get_req (ctx, state);
@@ -1255,6 +1297,151 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events,return ring->r.head == ring->r.tail ? ret :0 ; } +static void __io_sqe_files_unregister (struct io_ring_ctx *ctx) +{ +#if defined(CONFIG_UNIX ) +if (ctx->ring_sock) { +struct sock *sock = ctx->ring_sock->sk; +struct sk_buff *skb; + +while ((skb =skb_dequeue (&sock->sk_receive_queue)) !=NULL ) +kfree_skb (skb); + } +#else +int i; + +for (i =0 ; i < ctx->nr_user_files; i++) +fput (ctx->user_files[i]); +#endif +} + +static int io_sqe_files_unregister (struct io_ring_ctx *ctx) +{ +if (!ctx->user_files) +return -ENXIO ; + +__io_sqe_files_unregister (ctx); +kfree (ctx->user_files); + ctx->user_files =NULL ; +return 0 ; +} + +#if defined(CONFIG_UNIX ) +static int __io_sqe_files_scm (struct io_ring_ctx *ctx,int nr,int offset) +{ +struct scm_fp_list *fpl; +struct sk_buff *skb; +int i; + + fpl =kzalloc (sizeof (*fpl),GFP_KERNEL ); +if (!fpl) +return -ENOMEM ; + + skb =alloc_skb (0 ,GFP_KERNEL ); +if (!skb) { +kfree (fpl); +return -ENOMEM ; + } + + skb->sk = ctx->ring_sock->sk; + skb->destructor = unix_destruct_scm; + + fpl->user =get_uid (ctx->user); +for (i =0 ; i < nr; i++) { + fpl->fp[i] =get_file (ctx->user_files[i + offset]); +unix_inflight (fpl->user, fpl->fp[i]); +fput (fpl->fp[i]); + } + + fpl->max = fpl->count = nr; +UNIXCB (skb).fp = fpl; +skb_queue_head (&ctx->ring_sock->sk->sk_receive_queue, skb); +return 0 ; +} + +/* +* If UNIX sockets are enabled, fd passing can cause a reference cycle which +* causes regular reference counting to break down. We rely on the UNIX +* garbage collection to take care of this problem for us. +*/ +static int io_sqe_files_scm (struct io_ring_ctx *ctx) +{ +unsigned left, total; +int ret =0 ; + + total =0 ; + left = ctx->nr_user_files; +while (left) { +unsigned this_files =min_t (unsigned, left,SCM_MAX_FD ); +int ret; + + ret =__io_sqe_files_scm (ctx, this_files, total); +if (ret) +break ; + left -= this_files; + total += this_files; + } + +return ret; +} +#else +static int io_sqe_files_scm (struct io_ring_ctx *ctx) +{ +return 0 ; +} +#endif + +static int io_sqe_files_register (struct io_ring_ctx *ctx,void __user *arg, +unsigned nr_args) +{ +__s32 __user *fds = (__s32 __user *) arg; +int fd, ret =0 ; +unsigned i; + +if (ctx->user_files) +return -EBUSY ; +if (!nr_args) +return -EINVAL ; +if (nr_args >IORING_MAX_FIXED_FILES ) +return -EMFILE ; + + ctx->user_files =kcalloc (nr_args,sizeof (struct file *),GFP_KERNEL ); +if (!ctx->user_files) +return -ENOMEM ; + +for (i =0 ; i < nr_args; i++) { + ret = -EFAULT ; +if (copy_from_user (&fd, &fds[i],sizeof (fd))) +break ; + + ctx->user_files[i] =fget (fd); + + ret = -EBADF ; +if (!ctx->user_files[i]) +break ; +/* +* Don't allow io_uring instances to be registered. If UNIX +* isn't enabled, then this causes a reference cycle and this +* instance can never get freed. If UNIX is enabled we'll +* handle it just fine, but there's still no point in allowing +* a ring fd as it doesn't suppor regular read/write anyway. +*/ +if (ctx->user_files[i]->f_op == &io_uring_fops) { +fput (ctx->user_files[i]); +break ; + } + ctx->nr_user_files++; + ret =0 ; + } + +if (!ret) + ret =io_sqe_files_scm (ctx); +if (ret) +io_sqe_files_unregister (ctx); + +return ret; +} +static int io_sq_offload_start (struct io_ring_ctx *ctx) {int ret;
@@ -1521,14 +1708,16 @@ static void io_ring_ctx_free(struct io_ring_ctx *ctx)destroy_workqueue (ctx->sqo_wq);if (ctx->sqo_mm)mmdrop (ctx->sqo_mm); + +io_iopoll_reap_events (ctx); +io_sqe_buffer_unregister (ctx); +io_sqe_files_unregister (ctx); +#if defined(CONFIG_UNIX )if (ctx->ring_sock)sock_release (ctx->ring_sock);#endif -io_iopoll_reap_events (ctx); -io_sqe_buffer_unregister (ctx); -io_mem_free (ctx->sq_ring);io_mem_free (ctx->sq_sqes);io_mem_free (ctx->cq_ring);
@@ -1886,6 +2075,15 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,break ; ret =io_sqe_buffer_unregister (ctx);break ; +case IORING_REGISTER_FILES : + ret =io_sqe_files_register (ctx, arg, nr_args); +break ; +case IORING_UNREGISTER_FILES : + ret = -EINVAL ; +if (arg || nr_args) +break ; + ret =io_sqe_files_unregister (ctx); +break ;default : ret = -EINVAL ;break ;
include/uapi/linux/io_uring.h +8 -1
@@ -16,7 +16,7 @@*/ struct io_uring_sqe {__u8 opcode;/* type of operation for this sqe */ -__u8 flags;/* as of now unused */ +__u8 flags;/* IOSQE_ flags */ __u16 ioprio;/* ioprio for the request */ __s32 fd;/* file descriptor to do IO on */ __u64 off;/* offset into file */
@@ -33,6 +33,11 @@ struct io_uring_sqe { }; }; +/* +* sqe->flags +*/ +#define IOSQE_FIXED_FILE (1U << 0)/* use fixed fileset */ +/* * io_uring_setup() flags */
@@ -113,5 +118,7 @@ struct io_uring_params {*/ #define IORING_REGISTER_BUFFERS 0#define IORING_UNREGISTER_BUFFERS 1 +#define IORING_REGISTER_FILES 2 +#define IORING_UNREGISTER_FILES 3#endif
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 14/19] io_uring: add file set registration
Quoted 11 lines · Jens Axboe
We normally have to fget/fput for each IO we do on a file. Even with the batching we do, the cost of the atomic inc/dec of the file usage count adds up.
This adds
IORING_REGISTER_FILES, andIORING_UNREGISTER_FILESopcodes for theio_uring_register(2)system call. The arguments passed in must be an array of__s32holding file descriptors, andnr_argsshould hold the number of file descriptors the application wishes to pin for the duration of theio_uringinstance (or untilIORING_UNREGISTER_FILESis called).
Quoted 7 lines
When used, the application must set
IOSQE_FIXED_FILEin thesqe->flagsmember. Then, instead of settingsqe->fdto the real fd, it setssqe->fdto the index in the array passed in toIORING_REGISTER_FILES.Files are automatically unregistered when the
io_uringinstance is torn down. An application need only unregister if it wishes to register a new set of fds.
I think the overall concept here is still broken: You're giving the user_files to the GC, and I think the GC can drop their refcounts, but I don't see you actually getting feedback from the GC anywhere that would let the GC break your references? E.g. in io_prep_rw() you grab file pointers from ctx->user_files after simply checking ctx->nr_user_files, and there is no path from the GC that touches those fields. As far as I can tell, the GC is just going to go through unix_destruct_scm() and drop references on your files, causing use-after-free.
But the unix GC is complicated, and maybe I'm just missing something...
Quoted 44 lines
fs/io_uring.c +44
-0+static void __io_sqe_files_unregister (struct io_ring_ctx *ctx) +{ +#if defined(CONFIG_UNIX ) +if (ctx->ring_sock) { +struct sock *sock = ctx->ring_sock->sk; +struct sk_buff *skb; + +while ((skb =skb_dequeue (&sock->sk_receive_queue)) !=NULL ) +kfree_skb (skb); + } +#else +int i; + +for (i =0 ; i < ctx->nr_user_files; i++) +fput (ctx->user_files[i]); +#endif +} + +static int io_sqe_files_unregister (struct io_ring_ctx *ctx) +{ +if (!ctx->user_files) +return -ENXIO ; + +__io_sqe_files_unregister (ctx); +kfree (ctx->user_files); + ctx->user_files =NULL ; +return 0 ; +} + +#if defined(CONFIG_UNIX ) +static int __io_sqe_files_scm (struct io_ring_ctx *ctx,int nr,int offset) +{ +struct scm_fp_list *fpl; +struct sk_buff *skb; +int i; + + fpl =kzalloc (sizeof (*fpl),GFP_KERNEL ); +if (!fpl) +return -ENOMEM ; + + skb =alloc_skb (0 ,GFP_KERNEL ); +if (!skb) { +kfree (fpl); +return -ENOMEM ;
fs/io_uring.c static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, · line 1353
Quoted 10 lines
fs/io_uring.c +10
-0+ } + + skb->sk = ctx->ring_sock->sk; + skb->destructor = unix_destruct_scm; + + fpl->user =get_uid (ctx->user); +for (i =0 ; i < nr; i++) { + fpl->fp[i] =get_file (ctx->user_files[i + offset]); +unix_inflight (fpl->user, fpl->fp[i]); +fput (fpl->fp[i]);
This pattern is almost always superfluous. You increment the file's refcount, maybe insert the file into a list (essentially), and drop the file's refcount back down. You're already holding a stable reference, and you're not temporarily lending that to anyone else.
Quoted 64 lines
fs/io_uring.c +64
-0+ } + + fpl->max = fpl->count = nr; +UNIXCB (skb).fp = fpl; +skb_queue_head (&ctx->ring_sock->sk->sk_receive_queue, skb); +return 0 ; +} + +/* +* If UNIX sockets are enabled, fd passing can cause a reference cycle which +* causes regular reference counting to break down. We rely on the UNIX +* garbage collection to take care of this problem for us. +*/ +static int io_sqe_files_scm (struct io_ring_ctx *ctx) +{ +unsigned left, total; +int ret =0 ; + + total =0 ; + left = ctx->nr_user_files; +while (left) { +unsigned this_files =min_t (unsigned, left,SCM_MAX_FD ); +int ret; + + ret =__io_sqe_files_scm (ctx, this_files, total); +if (ret) +break ; + left -= this_files; + total += this_files; + } + +return ret; +} +#else +static int io_sqe_files_scm (struct io_ring_ctx *ctx) +{ +return 0 ; +} +#endif + +static int io_sqe_files_register (struct io_ring_ctx *ctx,void __user *arg, +unsigned nr_args) +{ +__s32 __user *fds = (__s32 __user *) arg; +int fd, ret =0 ; +unsigned i; + +if (ctx->user_files) +return -EBUSY ; +if (!nr_args) +return -EINVAL ; +if (nr_args >IORING_MAX_FIXED_FILES ) +return -EMFILE ; + + ctx->user_files =kcalloc (nr_args,sizeof (struct file *),GFP_KERNEL ); +if (!ctx->user_files) +return -ENOMEM ; + +for (i =0 ; i < nr_args; i++) { + ret = -EFAULT ; +if (copy_from_user (&fd, &fds[i],sizeof (fd))) +break ; + + ctx->user_files[i] =fget (fd);
fs/io_uring.c static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, · line 1427
Quoted 10 lines
fs/io_uring.c +10
-0+ + ret = -EBADF ; +if (!ctx->user_files[i]) +break ; +/* +* Don't allow io_uring instances to be registered. If UNIX +* isn't enabled, then this causes a reference cycle and this +* instance can never get freed. If UNIX is enabled we'll +* handle it just fine, but there's still no point in allowing +* a ring fd as it doesn't suppor regular read/write anyway.
nit: support
Quoted 26 lines
fs/io_uring.c +19
-0+*/ +if (ctx->user_files[i]->f_op == &io_uring_fops) { +fput (ctx->user_files[i]); +break ; + } + ctx->nr_user_files++; + ret =0 ; + } + +if (!ret) + ret =io_sqe_files_scm (ctx); +if (ret) +io_sqe_files_unregister (ctx); + +return ret; +} +static int io_sq_offload_start (struct io_ring_ctx *ctx) {int ret;@@ -1521,14 +1708,16 @@ static void io_ring_ctx_free(struct io_ring_ctx *ctx)destroy_workqueue (ctx->sqo_wq);if (ctx->sqo_mm)mmdrop (ctx->sqo_mm); + +io_iopoll_reap_events (ctx);
fs/io_uring.c static void io_ring_ctx_free(struct io_ring_ctx *ctx) · line 1530
Quoted 10 lines
fs/io_uring.c +3
-2+io_sqe_buffer_unregister (ctx); +io_sqe_files_unregister (ctx); +#if defined(CONFIG_UNIX )if (ctx->ring_sock)sock_release (ctx->ring_sock);#endif -io_iopoll_reap_events (ctx); -io_sqe_buffer_unregister (ctx);
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 14/19] io_uring: add file set registration
Quoted 30 lines · Jann Horn
Quoted 11 lines · Jens Axboe
We normally have to fget/fput for each IO we do on a file. Even with the batching we do, the cost of the atomic inc/dec of the file usage count adds up.
This adds
IORING_REGISTER_FILES, andIORING_UNREGISTER_FILESopcodes for theio_uring_register(2)system call. The arguments passed in must be an array of__s32holding file descriptors, andnr_argsshould hold the number of file descriptors the application wishes to pin for the duration of theio_uringinstance (or untilIORING_UNREGISTER_FILESis called).Quoted 7 lines
When used, the application must set
IOSQE_FIXED_FILEin thesqe->flagsmember. Then, instead of settingsqe->fdto the real fd, it setssqe->fdto the index in the array passed in toIORING_REGISTER_FILES.Files are automatically unregistered when the
io_uringinstance is torn down. An application need only unregister if it wishes to register a new set of fds.I think the overall concept here is still broken: You're giving the
user_filesto the GC, and I think the GC can drop their refcounts, but I don't see you actually getting feedback from the GC anywhere that would let the GC break your references? E.g. inio_prep_rw()you grab file pointers fromctx->user_filesafter simply checkingctx->nr_user_files, and there is no path from the GC that touches those fields. As far as I can tell, the GC is just going to go throughunix_destruct_scm()and drop references on your files, causing use-after-free.
Quoted 1 line
But the unix GC is complicated, and maybe I'm just missing something...
Only when the skb is released, which is either done when the io_uring is torn down (and then definitely safe), or if the socket is released, which is again also at a safe time.
Quoted 51 lines
Quoted 51 lines
fs/io_uring.c +51
-0+static void __io_sqe_files_unregister (struct io_ring_ctx *ctx) +{ +#if defined(CONFIG_UNIX ) +if (ctx->ring_sock) { +struct sock *sock = ctx->ring_sock->sk; +struct sk_buff *skb; + +while ((skb =skb_dequeue (&sock->sk_receive_queue)) !=NULL ) +kfree_skb (skb); + } +#else +int i; + +for (i =0 ; i < ctx->nr_user_files; i++) +fput (ctx->user_files[i]); +#endif +} + +static int io_sqe_files_unregister (struct io_ring_ctx *ctx) +{ +if (!ctx->user_files) +return -ENXIO ; + +__io_sqe_files_unregister (ctx); +kfree (ctx->user_files); + ctx->user_files =NULL ; +return 0 ; +} + +#if defined(CONFIG_UNIX ) +static int __io_sqe_files_scm (struct io_ring_ctx *ctx,int nr,int offset) +{ +struct scm_fp_list *fpl; +struct sk_buff *skb; +int i; + + fpl =kzalloc (sizeof (*fpl),GFP_KERNEL ); +if (!fpl) +return -ENOMEM ; + + skb =alloc_skb (0 ,GFP_KERNEL ); +if (!skb) { +kfree (fpl); +return -ENOMEM ; + } + + skb->sk = ctx->ring_sock->sk; + skb->destructor = unix_destruct_scm; + + fpl->user =get_uid (ctx->user); +for (i =0 ; i < nr; i++) {
Quoted 8 lines
fs/io_uring.c static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, · line 1353
Quoted 3 lines
fs/io_uring.c +3
-0+ fpl->fp[i] =get_file (ctx->user_files[i + offset]); +unix_inflight (fpl->user, fpl->fp[i]); +fput (fpl->fp[i]);This pattern is almost always superfluous. You increment the file's refcount, maybe insert the file into a list (essentially), and drop the file's refcount back down. You're already holding a stable reference, and you're not temporarily lending that to anyone else.
Actually, this is me messing up. The fput() should be done AFTER adding to the socket. I'll fix that.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 14/19] io_uring: add file set registration
In reply to Jens AxboeQuoted 19 lines · Jens Axboe
We normally have to fget/fput for each IO we do on a file. Even with the batching we do, the cost of the atomic inc/dec of the file usage count adds up.
This adds
IORING_REGISTER_FILES, andIORING_UNREGISTER_FILESopcodes for theio_uring_register(2)system call. The arguments passed in must be an array of__s32holding file descriptors, andnr_argsshould hold the number of file descriptors the application wishes to pin for the duration of theio_uringinstance (or untilIORING_UNREGISTER_FILESis called).When used, the application must set
IOSQE_FIXED_FILEin thesqe->flagsmember. Then, instead of settingsqe->fdto the real fd, it setssqe->fdto the index in the array passed in toIORING_REGISTER_FILES.Files are automatically unregistered when the
io_uringinstance is torn down. An application need only unregister if it wishes to register a new set of fds.
Quoted 6 lines
Signed-off-by Jens Axboe <[email protected]>--- fs/io_uring.c | 256 ++++++++++++++++++++++++++++++---- include/uapi/linux/io_uring.h | 9 +- 2 files changed, 235 insertions(+), 30 deletions(-)
Cheers,
Hannes
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 15/19 io_uring: add submission polling
In reply to Jens AxboeThis enables an application to do IO, without ever entering the kernel. By using the SQ ring to fill in new sqes and watching for completions on the CQ ring, we can submit and reap IOs without doing a single system call. The kernel side thread will poll for new submissions, and in case of HIPRI/polled IO, it'll also poll for completions.
By default, we allow 1 second of active spinning. This can by changed by passing in a different grace period at io_uring_register(2) time. If the thread exceeds this idle time without having any work to do, it will set:
sq_ring->flags |= IORING_SQ_NEED_WAKEUP.
The application will have to call io_uring_enter() to start things back up again. If IO is kept busy, that will never be needed. Basically an application that has this feature enabled will guard it's io_uring_enter(2) call with:
read_barrier();
if (*sq_ring->flags & IORING_SQ_NEED_WAKEUP)
io_uring_enter(fd, 0, 0, IORING_ENTER_SQ_WAKEUP);
instead of calling it unconditionally.
It's mandatory to use fixed files with this feature. Failure to do so will result in the application getting an -EBADF CQ entry when submitting IO.
--- fs/io_uring.c | 249 +++++++++++++++++++++++++++++++++- include/uapi/linux/io_uring.h | 12 +- 2 files changed, 253 insertions(+), 8 deletions(-)
fs/io_uring.c +242 -7
@@ -24,6 +24,7 @@#include <linux/percpu.h> #include <linux/slab.h> #include <linux/workqueue.h> +#include <linux/kthread.h> #include <linux/blkdev.h> #include <linux/bvec.h> #include <linux/net.h>
@@ -87,12 +88,16 @@ struct io_ring_ctx {unsigned cached_sq_head;unsigned sq_entries;unsigned sq_mask; +unsigned sq_thread_idle;struct io_uring_sqe *sq_sqes; }____cacheline_aligned_in_smp ;/* IO offload */ struct workqueue_struct *sqo_wq; +struct task_struct *sqo_thread;/* if using sq thread polling */ struct mm_struct *sqo_mm; +wait_queue_head_t sqo_wait; +unsigned sqo_stop;struct {/* CQ ring */
@@ -147,6 +152,7 @@ struct sqe_submit {unsigned short index;bool has_user;bool needs_lock; +bool needs_fixed_file; };struct io_kiocb {
@@ -299,6 +305,8 @@ static void io_cqring_add_event(struct io_ring_ctx *ctx, u64 ki_user_data,if (waitqueue_active (&ctx->wait))wake_up (&ctx->wait); +if (waitqueue_active (&ctx->sqo_wait)) +wake_up (&ctx->sqo_wait); }static void io_ring_drop_ctx_refs (struct io_ring_ctx *ctx,unsigned refs)
@@ -652,9 +660,10 @@ static bool io_file_supports_async(struct file *file)return false; } -static int io_prep_rw (struct io_kiocb *req,const struct io_uring_sqe *sqe, +static int io_prep_rw (struct io_kiocb *req,const struct sqe_submit *s,bool force_nonblock,struct io_submit_state *state) { +const struct io_uring_sqe *sqe = s->sqe;struct io_ring_ctx *ctx = req->ctx;struct kiocb *kiocb = &req->rw;unsigned ioprio, flags;
@@ -674,6 +683,8 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe, kiocb->ki_filp = ctx->user_files[fd]; req->flags |=REQ_F_FIXED_FILE ; }else { +if (s->needs_fixed_file) +return -EBADF ; kiocb->ki_filp =io_file_get (state, fd);if (unlikely (!kiocb->ki_filp))return -EBADF ;
@@ -827,7 +838,7 @@ static ssize_t io_read(struct io_kiocb *req, const struct sqe_submit *s,struct file *file;ssize_t ret; - ret =io_prep_rw (req, s->sqe, force_nonblock, state); + ret =io_prep_rw (req, s, force_nonblock, state);if (ret)return ret; file = kiocb->ki_filp;
@@ -871,7 +882,7 @@ static ssize_t io_write(struct io_kiocb *req, const struct sqe_submit *s,struct file *file;ssize_t ret; - ret =io_prep_rw (req, s->sqe, force_nonblock, state); + ret =io_prep_rw (req, s, force_nonblock, state);if (ret)return ret;/* Hold on to the file for -EAGAIN */
@@ -1061,6 +1072,7 @@ static void io_sq_wq_submit_work(struct work_struct *work) user_data =READ_ONCE (s->sqe->user_data); s->needs_lock = true; s->has_user = false; + s->needs_fixed_file = false;/* * If we're doing IO to fixed buffers, we don't need to get/set
@@ -1215,6 +1227,170 @@ static bool io_get_sqring(struct io_ring_ctx *ctx, struct sqe_submit *s)return false; } +static int io_submit_sqes (struct io_ring_ctx *ctx,struct sqe_submit *sqes, +unsigned int nr,bool has_user,bool mm_fault) +{ +struct io_submit_state state, *statep =NULL ; +int ret, i, submitted =0 ; + +if (nr >IO_PLUG_THRESHOLD ) { +io_submit_state_start (&state, ctx, nr); + statep = &state; + } + +for (i =0 ; i < nr; i++) { +if (unlikely (mm_fault)) { + ret = -EFAULT ; + }else { + sqes[i].has_user = has_user; + sqes[i].needs_lock = true; + sqes[i].needs_fixed_file = true; + ret =io_submit_sqe (ctx, &sqes[i], statep); + } +if (!ret) { + submitted++; +continue ; + } + +io_cqring_add_event (ctx, sqes[i].sqe->user_data, ret,0 ); + } + +if (statep) +io_submit_state_end (&state); + +return submitted; +} + +static int io_sq_thread (void *data) +{ +struct sqe_submit sqes[IO_IOPOLL_BATCH ]; +struct io_ring_ctx *ctx = data; +struct mm_struct *cur_mm =NULL ; +mm_segment_t old_fs; +DEFINE_WAIT (wait); +unsigned inflight; +unsigned long timeout; + + old_fs =get_fs (); +set_fs (USER_DS ); + + timeout = inflight =0 ; +while (!kthread_should_stop () && !ctx->sqo_stop) { +bool all_fixed, mm_fault = false; +int i; + +if (inflight) { +unsigned nr_events =0 ; + +if (ctx->flags &IORING_SETUP_IOPOLL ) { +/* +* We disallow the app entering submit/complete +* with polling, but we still need to lock the +* ring to prevent racing with polled issue +* that got punted to a workqueue. +*/ +mutex_lock (&ctx->uring_lock); +io_iopoll_check (ctx, &nr_events,0 ); +mutex_unlock (&ctx->uring_lock); + }else { +/* +* Normal IO, just pretend everything completed. +* We don't have to poll completions for that. +*/ + nr_events = inflight; + } + + inflight -= nr_events; +if (!inflight) + timeout = jiffies + ctx->sq_thread_idle; + } + +if (!io_get_sqring (ctx, &sqes[0 ])) { +/* +* We're polling. If we're within the defined idle +* period, then let us spin without work before going +* to sleep. +*/ +if (inflight || !time_after (jiffies, timeout)) { +cpu_relax (); +continue ; + } + +/* +* Drop cur_mm before scheduling, we can't hold it for +* long periods (or over schedule()). Do this before +* adding ourselves to the waitqueue, as the unuse/drop +* may sleep. +*/ +if (cur_mm) { +unuse_mm (cur_mm); +mmput (cur_mm); + cur_mm =NULL ; + } + +prepare_to_wait (&ctx->sqo_wait, &wait, +TASK_INTERRUPTIBLE ); + +/* Tell userspace we may need a wakeup call */ + ctx->sq_ring->flags |=IORING_SQ_NEED_WAKEUP ; +smp_wmb (); + +if (!io_get_sqring (ctx, &sqes[0 ])) { +if (kthread_should_stop ()) { +finish_wait (&ctx->sqo_wait, &wait); +break ; + } +if (signal_pending (current)) +flush_signals (current); +schedule (); +finish_wait (&ctx->sqo_wait, &wait); + + ctx->sq_ring->flags &= ~IORING_SQ_NEED_WAKEUP ; +smp_wmb (); +continue ; + } +finish_wait (&ctx->sqo_wait, &wait); + + ctx->sq_ring->flags &= ~IORING_SQ_NEED_WAKEUP ; +smp_wmb (); + } + + i =0 ; + all_fixed = true; +do { +if (all_fixed &&io_sqe_needs_user (sqes[i].sqe)) + all_fixed = false; + + i++; +if (i ==ARRAY_SIZE (sqes)) +break ; + }while (io_get_sqring (ctx, &sqes[i])); + +io_commit_sqring (ctx); + +/* Unless all new commands are FIXED regions, grab mm */ +if (!all_fixed && !cur_mm) { + mm_fault = !mmget_not_zero (ctx->sqo_mm); +if (!mm_fault) { +use_mm (ctx->sqo_mm); + cur_mm = ctx->sqo_mm; + } + } + + inflight +=io_submit_sqes (ctx, sqes, i, cur_mm !=NULL , + mm_fault); + } + +io_iopoll_reap_events (ctx); + +set_fs (old_fs); +if (cur_mm) { +unuse_mm (cur_mm); +mmput (cur_mm); + } +return 0 ; +} +static int io_ring_submit (struct io_ring_ctx *ctx,unsigned int to_submit) {struct io_submit_state state, *statep =NULL ;
@@ -1233,6 +1409,7 @@ static int io_ring_submit(struct io_ring_ctx *ctx, unsigned int to_submit) s.has_user = true; s.needs_lock = false; + s.needs_fixed_file = false; ret =io_submit_sqe (ctx, &s, statep);if (ret) {
@@ -1442,13 +1619,47 @@ static int io_sqe_files_register(struct io_ring_ctx *ctx, void __user *arg,return ret; } -static int io_sq_offload_start (struct io_ring_ctx *ctx) +static int io_sq_offload_start (struct io_ring_ctx *ctx, +struct io_uring_params *p) {int ret; +init_waitqueue_head (&ctx->sqo_wait);mmgrab (current->mm); ctx->sqo_mm = current->mm; + ctx->sq_thread_idle =msecs_to_jiffies (p->sq_thread_idle); +if (!ctx->sq_thread_idle) + ctx->sq_thread_idle =HZ ; + + ret = -EINVAL ; +if (!cpu_possible (p->sq_thread_cpu)) + goto err; + +if (ctx->flags &IORING_SETUP_SQPOLL ) { +if (p->flags &IORING_SETUP_SQ_AFF ) { +int cpu; + + cpu =array_index_nospec (p->sq_thread_cpu,NR_CPUS ); + ctx->sqo_thread =kthread_create_on_cpu (io_sq_thread, + ctx, cpu, +"io_uring-sq" ); + }else { + ctx->sqo_thread =kthread_create (io_sq_thread, ctx, +"io_uring-sq" ); + } +if (IS_ERR (ctx->sqo_thread)) { + ret =PTR_ERR (ctx->sqo_thread); + ctx->sqo_thread =NULL ; + goto err; + } +wake_up_process (ctx->sqo_thread); + }else if (p->flags &IORING_SETUP_SQ_AFF ) { +/* Can't have SQ_AFF without SQPOLL */ + ret = -EINVAL ; + goto err; + } +/* Do QD, or 2 * CPUS, whatever is smallest */ ctx->sqo_wq =alloc_workqueue ("io_ring-wq" ,WQ_UNBOUND |WQ_FREEZABLE ,min (ctx->sq_entries -1 ,2 *num_online_cpus ()));
@@ -1459,6 +1670,12 @@ static int io_sq_offload_start(struct io_ring_ctx *ctx)return 0 ; err: +if (ctx->sqo_thread) { + ctx->sqo_stop =1 ; +mb (); +kthread_stop (ctx->sqo_thread); + ctx->sqo_thread =NULL ; + }mmdrop (ctx->sqo_mm); ctx->sqo_mm =NULL ;return ret;
@@ -1704,6 +1921,11 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, void __user *arg,static void io_ring_ctx_free (struct io_ring_ctx *ctx) { +if (ctx->sqo_thread) { + ctx->sqo_stop =1 ; +mb (); +kthread_stop (ctx->sqo_thread); + }if (ctx->sqo_wq)destroy_workqueue (ctx->sqo_wq);if (ctx->sqo_mm)
@@ -1811,7 +2033,7 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit,int submitted =0 ;struct fd f; -if (flags & ~IORING_ENTER_GETEVENTS ) +if (flags & ~(IORING_ENTER_GETEVENTS |IORING_ENTER_SQ_WAKEUP ))return -EINVAL ; f =fdget (fd);
@@ -1827,6 +2049,18 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit,if (!percpu_ref_tryget (&ctx->refs)) goto out_fput; +/* +* For SQ polling, the thread will do all submissions and completions. +* Just return the requested submit count, and wake the thread if +* we were asked to. +*/ +if (ctx->flags &IORING_SETUP_SQPOLL ) { +if (flags &IORING_ENTER_SQ_WAKEUP ) +wake_up (&ctx->sqo_wait); + submitted = to_submit; + goto out_ctx; + } +if (to_submit) { to_submit =min (to_submit, ctx->sq_entries);
@@ -1990,7 +2224,7 @@ static int io_uring_create(unsigned entries, struct io_uring_params *p)if (ret) goto err; - ret =io_sq_offload_start (ctx); + ret =io_sq_offload_start (ctx, p);if (ret) goto err;
@@ -2038,7 +2272,8 @@ static long io_uring_setup(u32 entries, struct io_uring_params __user *params)return -EINVAL ; } -if (p.flags & ~IORING_SETUP_IOPOLL ) +if (p.flags & ~(IORING_SETUP_IOPOLL |IORING_SETUP_SQPOLL | +IORING_SETUP_SQ_AFF ))return -EINVAL ; ret =io_uring_create (entries, &p);
include/uapi/linux/io_uring.h +11 -1
@@ -42,6 +42,8 @@ struct io_uring_sqe {* io_uring_setup() flags */ #define IORING_SETUP_IOPOLL (1U << 0)/* io_context is polled */ +#define IORING_SETUP_SQPOLL (1U << 1)/* SQ poll thread */ +#define IORING_SETUP_SQ_AFF (1U << 2)/* sq_thread_cpu is valid */ #define IORING_OP_NOP 0#define IORING_OP_READV 1
@@ -86,6 +88,11 @@ struct io_sqring_offsets {__u64 resv2; }; +/* +* sq_ring->flags +*/ +#define IORING_SQ_NEED_WAKEUP (1U << 0)/* needs io_uring_enter wakeup */ +struct io_cqring_offsets {__u32 head;__u32 tail;
@@ -100,6 +107,7 @@ struct io_cqring_offsets {* io_uring_enter(2) flags */ #define IORING_ENTER_GETEVENTS (1U << 0) +#define IORING_ENTER_SQ_WAKEUP (1U << 1)/* * Passed in for io_uring_setup(2). Copied back with updated info on success
@@ -108,7 +116,9 @@ struct io_uring_params {__u32 sq_entries;__u32 cq_entries;__u32 flags; -__u32 resv[7 ]; +__u32 sq_thread_cpu; +__u32 sq_thread_idle; +__u32 resv[5 ];struct io_sqring_offsets sq_off;struct io_cqring_offsets cq_off; };
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 15/19] io_uring: add submission polling
Quoted 28 lines · Jens Axboe
This enables an application to do IO, without ever entering the kernel. By using the SQ ring to fill in new sqes and watching for completions on the CQ ring, we can submit and reap IOs without doing a single system call. The kernel side thread will poll for new submissions, and in case of HIPRI/polled IO, it'll also poll for completions.
By default, we allow 1 second of active spinning. This can by changed by passing in a different grace period at
io_uring_register(2)time. If the thread exceeds this idle time without having any work to do, it will set:
sq_ring->flags|=IORING_SQ_NEED_WAKEUP.The application will have to call
io_uring_enter()to start things back up again. If IO is kept busy, that will never be needed. Basically an application that has this feature enabled will guard it'sio_uring_enter(2)call with:
read_barrier();
if (*sq_ring->flags&IORING_SQ_NEED_WAKEUP)io_uring_enter(fd, 0, 0, IORING_ENTER_SQ_WAKEUP);instead of calling it unconditionally.
It's mandatory to use fixed files with this feature. Failure to do so will result in the application getting an
-EBADFCQ entry when submitting IO.
Quoted 6 lines
Signed-off-by Jens Axboe <[email protected]>--- fs/io_uring.c | 249 +++++++++++++++++++++++++++++++++- include/uapi/linux/io_uring.h | 12 +- 2 files changed, 253 insertions(+), 8 deletions(-)
Cheers,
Hannes
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 16/19 io_uring: add io_kiocb ref count
In reply to Jens AxboeWe'll use this for the POLL implementation. Regular requests will NOT be using references, so initialize it to 0. Any real use of the io_kiocb ref will initialize it to at least 2.
--- fs/io_uring.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
fs/io_uring.c +6 -2
@@ -163,6 +163,7 @@ struct io_kiocb {struct io_ring_ctx *ctx;struct list_head list;unsigned int flags; +refcount_t refs;#define REQ_F_FORCE_NONBLOCK 1/* inline submission attempt */ #define REQ_F_IOPOLL_COMPLETED 2/* polled IO has completed */ #define REQ_F_FIXED_FILE 4/* ctx owns file */
@@ -349,6 +350,7 @@ static struct io_kiocb *io_get_req(struct io_ring_ctx *ctx, req->ctx = ctx; req->flags =0 ; +refcount_set (&req->refs,0 );return req; out:io_ring_drop_ctx_refs (ctx,1 );
@@ -366,8 +368,10 @@ static void io_free_req_many(struct io_ring_ctx *ctx, void **reqs, int *nr)static void io_free_req (struct io_kiocb *req) { -io_ring_drop_ctx_refs (req->ctx,1 ); -kmem_cache_free (req_cachep, req); +if (!refcount_read (&req->refs) ||refcount_dec_and_test (&req->refs)) { +io_ring_drop_ctx_refs (req->ctx,1 ); +kmem_cache_free (req_cachep, req); + } }/*
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 17/19 io_uring: add support for IORING_OP_POLL
In reply to Jens AxboeThis is basically a direct port of bfe4037e722e, which implements a one-shot poll command through aio. Description below is based on that commit as well. However, instead of adding a POLL command and relying on io_cancel(2) to remove it, we mimic the epoll(2) interface of having a command to add a poll notification, IORING_OP_POLL_ADD, and one to remove it again, IORING_OP_POLL_REMOVE.
To poll for a file descriptor the application should submit an sqe of type IORING_OP_POLL. It will poll the fd for the events specified in the poll_events field.
Unlike poll or epoll without EPOLLONESHOT this interface always works in one shot mode, that is once the sqe is completed, it will have to be resubmitted.
Based-on-code-from: Christoph Hellwig <[email protected]>
--- fs/io_uring.c | 261 +++++++++++++++++++++++++++++++++- include/uapi/linux/io_uring.h | 3 + 2 files changed, 263 insertions(+), 1 deletion(-)
fs/io_uring.c +260 -1
@@ -140,6 +140,7 @@ struct io_ring_ctx {* manipulate the list, hence no extra locking is needed there. */ struct list_head poll_list; +struct list_head cancel_list; }____cacheline_aligned_in_smp ;#if defined(CONFIG_UNIX )
@@ -155,8 +156,20 @@ struct sqe_submit {bool needs_fixed_file; }; +struct io_poll_iocb { +struct file *file; +struct wait_queue_head *head; +__poll_t events; +bool woken; +bool canceled; +struct wait_queue_entry wait; +}; +struct io_kiocb { -struct kiocb rw; +union { +struct kiocb rw; +struct io_poll_iocb poll; + };struct sqe_submit submit;
@@ -240,6 +253,7 @@ static struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)init_waitqueue_head (&ctx->wait);spin_lock_init (&ctx->completion_lock);INIT_LIST_HEAD (&ctx->poll_list); +INIT_LIST_HEAD (&ctx->cancel_list);return ctx; }
@@ -993,6 +1007,244 @@ static int io_fsync(struct io_kiocb *req, const struct io_uring_sqe *sqe,return 0 ; } +static void io_poll_remove_one (struct io_kiocb *req) +{ +struct io_poll_iocb *poll = &req->poll; + +spin_lock (&poll->head->lock); +WRITE_ONCE (poll->canceled, true); +if (!list_empty (&poll->wait.entry)) { +list_del_init (&poll->wait.entry); +queue_work (req->ctx->sqo_wq, &req->work); + } +spin_unlock (&poll->head->lock); + +list_del_init (&req->list); +} + +static void io_poll_remove_all (struct io_ring_ctx *ctx) +{ +struct io_kiocb *req; + +spin_lock_irq (&ctx->completion_lock); +while (!list_empty (&ctx->cancel_list)) { + req = list_first_entry(&ctx->cancel_list, struct io_kiocb,list); +io_poll_remove_one (req); + } +spin_unlock_irq (&ctx->completion_lock); +} + +/* +* Find a running poll command that matches one specified in sqe->addr, +* and remove it if found. +*/ +static int io_poll_remove (struct io_kiocb *req,const struct io_uring_sqe *sqe) +{ +struct io_ring_ctx *ctx = req->ctx; +struct io_kiocb *poll_req, *next; +int ret = -ENOENT ; + +if (unlikely (req->ctx->flags &IORING_SETUP_IOPOLL )) +return -EINVAL ; +if (sqe->ioprio || sqe->off || sqe->len || sqe->buf_index || + sqe->poll_events) +return -EINVAL ; + +spin_lock_irq (&ctx->completion_lock); +list_for_each_entry_safe (poll_req, next, &ctx->cancel_list, list) { +if (READ_ONCE (sqe->addr) == poll_req->user_data) { +io_poll_remove_one (poll_req); + ret =0 ; +break ; + } + } +spin_unlock_irq (&ctx->completion_lock); + +io_cqring_add_event (req->ctx, sqe->user_data, ret,0 ); +io_free_req (req); +return 0 ; +} + +static void io_poll_complete (struct io_kiocb *req,__poll_t mask) +{ +io_cqring_add_event (req->ctx, req->user_data,mangle_poll (mask),0 ); +io_fput (req); +io_free_req (req); +} + +static void io_poll_complete_work (struct work_struct *work) +{ + struct io_kiocb *req = container_of(work, struct io_kiocb, work); +struct io_poll_iocb *poll = &req->poll; +struct poll_table_struct pt = { ._key = poll->events }; +struct io_ring_ctx *ctx = req->ctx; +__poll_t mask =0 ; + +if (!READ_ONCE (poll->canceled)) + mask =vfs_poll (poll->file, &pt) & poll->events; + +/* +* Note that ->ki_cancel callers also delete iocb from active_reqs after +* calling ->ki_cancel. We need the ctx_lock roundtrip here to +* synchronize with them. In the cancellation case the list_del_init +* itself is not actually needed, but harmless so we keep it in to +* avoid further branches in the fast path. +*/ +spin_lock_irq (&ctx->completion_lock); +if (!mask && !READ_ONCE (poll->canceled)) { +add_wait_queue (poll->head, &poll->wait); +spin_unlock_irq (&ctx->completion_lock); +return ; + } +list_del_init (&req->list); +spin_unlock_irq (&ctx->completion_lock); + +io_poll_complete (req, mask); +} + +static int io_poll_wake (struct wait_queue_entry *wait,unsigned mode,int sync, +void *key) +{ + struct io_poll_iocb *poll = container_of(wait, struct io_poll_iocb, + wait); + struct io_kiocb *req = container_of(poll, struct io_kiocb, poll); +struct io_ring_ctx *ctx = req->ctx; +__poll_t mask =key_to_poll (key); + + poll->woken = true; + +/* for instances that support it check for an event match first: */ +if (mask) { +if (!(mask & poll->events)) +return 0 ; + +/* try to complete the iocb inline if we can: */ +if (spin_trylock (&ctx->completion_lock)) { +list_del (&req->list); +spin_unlock (&ctx->completion_lock); + +list_del_init (&poll->wait.entry); +io_poll_complete (req, mask); +return 1 ; + } + } + +list_del_init (&poll->wait.entry); +queue_work (ctx->sqo_wq, &req->work); +return 1 ; +} + +struct io_poll_table { +struct poll_table_struct pt; +struct io_kiocb *req; +int error; +}; + +static void io_poll_queue_proc (struct file *file,struct wait_queue_head *head, +struct poll_table_struct *p) +{ + struct io_poll_table *pt = container_of(p, struct io_poll_table, pt); + +if (unlikely (pt->req->poll.head)) { + pt->error = -EINVAL ; +return ; + } + + pt->error =0 ; + pt->req->poll.head = head; +add_wait_queue (head, &pt->req->poll.wait); +} + +static int io_poll_add (struct io_kiocb *req,const struct io_uring_sqe *sqe) +{ +struct io_poll_iocb *poll = &req->poll; +struct io_ring_ctx *ctx = req->ctx; +struct io_poll_table ipt; +unsigned flags; +__poll_t mask; +u16 events; +int fd; + +if (unlikely (req->ctx->flags &IORING_SETUP_IOPOLL )) +return -EINVAL ; +if (sqe->addr || sqe->ioprio || sqe->off || sqe->len || sqe->buf_index) +return -EINVAL ; + +INIT_WORK (&req->work, io_poll_complete_work); + events =READ_ONCE (sqe->poll_events); + poll->events =demangle_poll (events) |EPOLLERR |EPOLLHUP ; + + flags =READ_ONCE (sqe->flags); + fd =READ_ONCE (sqe->fd); + +if (flags &IOSQE_FIXED_FILE ) { +if (unlikely (!ctx->user_files || fd >= ctx->nr_user_files)) +return -EBADF ; + poll->file = ctx->user_files[fd]; + req->flags |=REQ_F_FIXED_FILE ; + }else { + poll->file =fget (fd); + } +if (unlikely (!poll->file)) +return -EBADF ; + + poll->head =NULL ; + poll->woken = false; + poll->canceled = false; + + ipt.pt._qproc = io_poll_queue_proc; + ipt.pt._key = poll->events; + ipt.req = req; + ipt.error = -EINVAL ;/* same as no support for IOCB_CMD_POLL */ + +/* initialized the list so that we can do list_empty checks */ +INIT_LIST_HEAD (&poll->wait.entry); +init_waitqueue_func_entry (&poll->wait, io_poll_wake); + +/* one for removal from waitqueue, one for this function */ +refcount_set (&req->refs,2 ); + + mask =vfs_poll (poll->file, &ipt.pt) & poll->events; +if (unlikely (!poll->head)) { +/* we did not manage to set up a waitqueue, done */ + goto out; + } + +spin_lock_irq (&ctx->completion_lock); +spin_lock (&poll->head->lock); +if (poll->woken) { +/* wake_up context handles the rest */ + mask =0 ; + ipt.error =0 ; + }else if (mask || ipt.error) { +/* if we get an error or a mask we are done */ +WARN_ON_ONCE (list_empty (&poll->wait.entry)); +list_del_init (&poll->wait.entry); + }else { +/* actually waiting for an event */ +list_add_tail (&req->list, &ctx->cancel_list); + } +spin_unlock (&poll->head->lock); +spin_unlock_irq (&ctx->completion_lock); + +out: +if (unlikely (ipt.error)) { +if (!(flags &IOSQE_FIXED_FILE )) +fput (poll->file); +/* +* Drop one of our refs to this req, __io_submit_sqe() will +* drop the other one since we're returning an error. +*/ +io_free_req (req); +return ipt.error; + } + +if (mask) +io_poll_complete (req, mask); +io_free_req (req); +return 0 ; +} +static int __io_submit_sqe (struct io_ring_ctx *ctx,struct io_kiocb *req,const struct sqe_submit *s,bool force_nonblock,struct io_submit_state *state)
@@ -1028,6 +1280,12 @@ static int __io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,case IORING_OP_FSYNC : ret =io_fsync (req, s->sqe, force_nonblock);break ; +case IORING_OP_POLL_ADD : + ret =io_poll_add (req, s->sqe); +break ; +case IORING_OP_POLL_REMOVE : + ret =io_poll_remove (req, s->sqe); +break ;default : ret = -EINVAL ;break ;
@@ -1983,6 +2241,7 @@ static void io_ring_ctx_wait_and_kill(struct io_ring_ctx *ctx)percpu_ref_kill (&ctx->refs);mutex_unlock (&ctx->uring_lock); +io_poll_remove_all (ctx);io_iopoll_reap_events (ctx);wait_for_completion (&ctx->ctx_done);io_ring_ctx_free (ctx);
include/uapi/linux/io_uring.h +3 -0
@@ -25,6 +25,7 @@ struct io_uring_sqe {union {__kernel_rwf_t rw_flags;__u32 fsync_flags; +__u16 poll_events; };__u64 user_data;/* data to be passed back at completion time */ union {
@@ -51,6 +52,8 @@ struct io_uring_sqe {#define IORING_OP_FSYNC 3#define IORING_OP_READ_FIXED 4#define IORING_OP_WRITE_FIXED 5 +#define IORING_OP_POLL_ADD 6 +#define IORING_OP_POLL_REMOVE 7/* * sqe->fsync_flags
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 18/19 io_uring: allow workqueue item to handle multiple buffered requests
In reply to Jens AxboeRight now we punt any buffered request that ends up triggering an -EAGAIN to an async workqueue. This works fine in terms of providing async execution of them, but it also can create quite a lot of work queue items. For sequentially buffered IO, it's advantageous to serialize the issue of them. For reads, the first one will trigger a read-ahead, and subsequent request merely end up waiting on later pages to complete. For writes, devices usually respond better to streamed sequential writes.
Add state to track the last buffered request we punted to a work queue, and if the next one is sequential to the previous, attempt to get the previous work item to handle it. We limit the number of sequential add-ons to the a multiple (8) of the max read-ahead size of the file. This should be a good number for both reads and wries, as it defines the max IO size the device can do directly.
This drastically cuts down on the number of context switches we need to handle buffered sequential IO, and a basic test case of copying a big file with io_uring sees a 5x speedup.
--- fs/io_uring.c | 267 ++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 216 insertions(+), 51 deletions(-)
fs/io_uring.c +216 -51
@@ -74,6 +74,16 @@ struct io_mapped_ubuf {unsigned int nr_bvecs; }; +struct async_list { +spinlock_t lock; +atomic_t cnt; +struct list_head list; + +struct file *file; +off_t io_end; +size_t io_pages; +}; + struct io_ring_ctx { struct { struct percpu_ref refs;
@@ -143,6 +153,8 @@ struct io_ring_ctx {struct list_head cancel_list; }____cacheline_aligned_in_smp ; +struct async_list pending_async[2 ]; +#if defined(CONFIG_UNIX )struct socket *ring_sock;#endif
@@ -180,6 +192,7 @@ struct io_kiocb {#define REQ_F_FORCE_NONBLOCK 1/* inline submission attempt */ #define REQ_F_IOPOLL_COMPLETED 2/* polled IO has completed */ #define REQ_F_FIXED_FILE 4/* ctx owns file */ +#define REQ_F_SEQ_PREV 8/* sequential with previous */ u64 user_data;u64 error;
@@ -236,6 +249,7 @@ static void io_ring_ctx_ref_free(struct percpu_ref *ref)static struct io_ring_ctx *io_ring_ctx_alloc (struct io_uring_params *p) {struct io_ring_ctx *ctx; +int i; ctx =kzalloc (sizeof (*ctx),GFP_KERNEL );if (!ctx)
@@ -251,6 +265,11 @@ static struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)init_completion (&ctx->ctx_done);mutex_init (&ctx->uring_lock);init_waitqueue_head (&ctx->wait); +for (i =0 ; i <ARRAY_SIZE (ctx->pending_async); i++) { +spin_lock_init (&ctx->pending_async[i].lock); +INIT_LIST_HEAD (&ctx->pending_async[i].list); +atomic_set (&ctx->pending_async[i].cnt,0 ); + }spin_lock_init (&ctx->completion_lock);INIT_LIST_HEAD (&ctx->poll_list);INIT_LIST_HEAD (&ctx->cancel_list);
@@ -847,6 +866,39 @@ static int io_import_iovec(struct io_ring_ctx *ctx, int rw,return import_iovec (rw, buf, sqe_len,UIO_FASTIOV , iovec, iter); } +static void io_async_list_note (int rw,struct io_kiocb *req,size_t len) +{ +struct async_list *async_list = &req->ctx->pending_async[rw]; +struct kiocb *kiocb = &req->rw; +struct file *filp = kiocb->ki_filp; +off_t io_end = kiocb->ki_pos + len; + +if (filp == async_list->file && kiocb->ki_pos == async_list->io_end) { +unsigned long max_pages; + +/* Use 8x RA size as a decent limiter for both reads/writes */ + max_pages = filp->f_ra.ra_pages; +if (!max_pages) + max_pages =VM_MAX_READAHEAD >> (PAGE_SHIFT -10 ); + max_pages *=8 ; + + len >>=PAGE_SHIFT ; +if (async_list->io_pages + len <= max_pages) { + req->flags |=REQ_F_SEQ_PREV ; + async_list->io_pages += len; + }else { + io_end =0 ; + async_list->io_pages =0 ; + } + } + +if (async_list->file != filp) { + async_list->io_pages =0 ; + async_list->file = filp; + } + async_list->io_end = io_end; +} +static ssize_t io_read (struct io_kiocb *req,const struct sqe_submit *s,bool force_nonblock,struct io_submit_state *state) {
@@ -854,6 +906,7 @@ static ssize_t io_read(struct io_kiocb *req, const struct sqe_submit *s,struct kiocb *kiocb = &req->rw;struct iov_iter iter;struct file *file; +size_t iov_count;ssize_t ret; ret =io_prep_rw (req, s, force_nonblock, state);
@@ -872,16 +925,24 @@ static ssize_t io_read(struct io_kiocb *req, const struct sqe_submit *s,if (ret) goto out_fput; - ret =rw_verify_area (READ , file, &kiocb->ki_pos,iov_iter_count (&iter)); + iov_count =iov_iter_count (&iter); + ret =rw_verify_area (READ , file, &kiocb->ki_pos, iov_count);if (!ret) {ssize_t ret2;/* Catch -EAGAIN return for forced non-blocking submission */ ret2 =call_read_iter (file, kiocb, &iter); -if (!force_nonblock || ret2 != -EAGAIN ) +if (!force_nonblock || ret2 != -EAGAIN ) {io_rw_done (kiocb, ret2); -else + }else { +/* +* If ->needs_lock is true, we're already in async +* context. +*/ +if (!s->needs_lock) +io_async_list_note (READ , req, iov_count); ret = -EAGAIN ; + } }kfree (iovec); out_fput:
@@ -898,14 +959,12 @@ static ssize_t io_write(struct io_kiocb *req, const struct sqe_submit *s,struct kiocb *kiocb = &req->rw;struct iov_iter iter;struct file *file; +size_t iov_count;ssize_t ret; ret =io_prep_rw (req, s, force_nonblock, state);if (ret)return ret; -/* Hold on to the file for -EAGAIN */ -if (force_nonblock && !(kiocb->ki_flags &IOCB_DIRECT )) -return -EAGAIN ; ret = -EBADF ; file = kiocb->ki_filp;
@@ -919,8 +978,17 @@ static ssize_t io_write(struct io_kiocb *req, const struct sqe_submit *s,if (ret) goto out_fput; - ret =rw_verify_area (WRITE , file, &kiocb->ki_pos, -iov_iter_count (&iter)); + iov_count =iov_iter_count (&iter); + + ret = -EAGAIN ; +if (force_nonblock && !(kiocb->ki_flags &IOCB_DIRECT )) { +/* If ->needs_lock is true, we're already in async context. */ +if (!s->needs_lock) +io_async_list_note (WRITE , req, iov_count); + goto out_free; + } + + ret =rw_verify_area (WRITE , file, &kiocb->ki_pos, iov_count);if (!ret) {/* * Open-code file_start_write here to grab freeze protection,
@@ -938,9 +1006,11 @@ static ssize_t io_write(struct io_kiocb *req, const struct sqe_submit *s, kiocb->ki_flags |=IOCB_WRITE ;io_rw_done (kiocb,call_write_iter (file, kiocb, &iter)); } +out_free:kfree (iovec); out_fput: -if (unlikely (ret)) +/* Hold on to the file for -EAGAIN */ +if (unlikely (ret && ret != -EAGAIN ))io_fput (req);return ret; }
@@ -1309,6 +1379,21 @@ static int __io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,return 0 ; } +static struct async_list *io_async_list_from_sqe (struct io_ring_ctx *ctx, +const struct io_uring_sqe *sqe) +{ +switch (sqe->opcode) { +case IORING_OP_READV : +case IORING_OP_READ_FIXED : +return &ctx->pending_async[READ ]; +case IORING_OP_WRITEV : +case IORING_OP_WRITE_FIXED : +return &ctx->pending_async[WRITE ]; +default : +return NULL ; + } +} +static inline bool io_sqe_needs_user (const struct io_uring_sqe *sqe) {u8 opcode =READ_ONCE (sqe->opcode);
@@ -1320,60 +1405,133 @@ static inline bool io_sqe_needs_user(const struct io_uring_sqe *sqe)static void io_sq_wq_submit_work (struct work_struct *work) { struct io_kiocb *req = container_of(work, struct io_kiocb, work); -struct sqe_submit *s = &req->submit;struct io_ring_ctx *ctx = req->ctx; +struct mm_struct *cur_mm =NULL ; +struct async_list *async_list; +LIST_HEAD (req_list);mm_segment_t old_fs; -bool needs_user; -u64 user_data;int ret; -/* Ensure we clear previously set forced non-block flag */ - req->flags &= ~REQ_F_FORCE_NONBLOCK ; - req->rw.ki_flags &= ~IOCB_NOWAIT ; + async_list =io_async_list_from_sqe (ctx, req->submit.sqe); +restart: +do { +struct sqe_submit *s = &req->submit; +u64 user_data =READ_ONCE (s->sqe->user_data); + +/* Ensure we clear previously set forced non-block flag */ + req->flags &= ~REQ_F_FORCE_NONBLOCK ; + req->rw.ki_flags &= ~IOCB_NOWAIT ; - user_data =READ_ONCE (s->sqe->user_data); - s->needs_lock = true; - s->has_user = false; - s->needs_fixed_file = false; + ret =0 ; +if (io_sqe_needs_user (s->sqe) && !cur_mm) { +if (!mmget_not_zero (ctx->sqo_mm)) { + ret = -EFAULT ; + }else { + cur_mm = ctx->sqo_mm; +use_mm (ctx->sqo_mm); + old_fs =get_fs (); +set_fs (USER_DS ); + } + } + +if (!ret) { + s->has_user = cur_mm !=NULL ; + s->needs_lock = true; + s->needs_fixed_file = false; +do { + ret =__io_submit_sqe (ctx, req, s, false,NULL ); +/* +* We can get EAGAIN for polled IO even though +* we're forcing a sync submission from here, +* since we can't wait for request slots on the +* block side. +*/ +if (ret != -EAGAIN ) +break ; +cond_resched (); + }while (1 ); + } +if (ret) { +io_cqring_add_event (ctx, user_data, ret,0 ); +io_free_req (req); + } +if (!async_list) +break ; +if (!list_empty (&req_list)) { + req = list_first_entry(&req_list, struct io_kiocb, + list); +list_del (&req->list); +continue ; + } +if (list_empty (&async_list->list)) +break ; + + req =NULL ; +spin_lock (&async_list->lock); +if (list_empty (&async_list->list)) { +spin_unlock (&async_list->lock); +break ; + } +list_splice_init (&async_list->list, &req_list); +spin_unlock (&async_list->lock); + + req = list_first_entry(&req_list, struct io_kiocb, list); +list_del (&req->list); + }while (req);/* -* If we're doing IO to fixed buffers, we don't need to get/set -* user context +* Rare case of racing with a submitter. If we find the count has +* dropped to zero AND we have pending work items, then restart +* the processing. This is a tiny race window. */ - needs_user =io_sqe_needs_user (s->sqe); -if (needs_user) { -if (!mmget_not_zero (ctx->sqo_mm)) { - ret = -EFAULT ; - goto err; + ret =atomic_dec_return (&async_list->cnt); +while (!ret && !list_empty (&async_list->list)) { +spin_lock (&async_list->lock); +atomic_inc (&async_list->cnt); +list_splice_init (&async_list->list, &req_list); +spin_unlock (&async_list->lock); + +if (!list_empty (&req_list)) { + req = list_first_entry(&req_list, struct io_kiocb, + list); +list_del (&req->list); + goto restart; } -use_mm (ctx->sqo_mm); - old_fs =get_fs (); -set_fs (USER_DS ); - s->has_user = true; + ret =atomic_dec_return (&async_list->cnt); } -do { - ret =__io_submit_sqe (ctx, req, s, false,NULL ); -/* -* We can get EAGAIN for polled IO even though we're forcing -* a sync submission from here, since we can't wait for -* request slots on the block side. -*/ -if (ret != -EAGAIN ) -break ; -cond_resched (); - }while (1 ); - -if (needs_user) { +if (cur_mm) {set_fs (old_fs); -unuse_mm (ctx->sqo_mm); -mmput (ctx->sqo_mm); +unuse_mm (cur_mm); +mmput (cur_mm); } -err: -if (ret) { -io_cqring_add_event (ctx, user_data, ret,0 ); -io_free_req (req); +} + +/* +* See if we can piggy back onto previously submitted work, that is still +* running. We currently only allow this if the new request is sequential +* to the previous one we punted. +*/ +static bool io_add_to_prev_work (struct async_list *list,struct io_kiocb *req) +{ +bool ret = false; + +if (!list) +return false; +if (!(req->flags &REQ_F_SEQ_PREV )) +return false; +if (!atomic_read (&list->cnt)) +return false; + + ret = true; +spin_lock (&list->lock); +list_add_tail (&req->list, &list->list); +if (!atomic_read (&list->cnt)) { +list_del_init (&req->list); + ret = false; } +spin_unlock (&list->lock); +return ret; } static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s,
@@ -1394,9 +1552,16 @@ static int io_submit_sqe(struct io_ring_ctx *ctx, const struct sqe_submit *s, ret =__io_submit_sqe (ctx, req, s, true, state);if (ret == -EAGAIN ) { +struct async_list *list; + + list =io_async_list_from_sqe (ctx, s->sqe);memcpy (&req->submit, s,sizeof (*s)); -INIT_WORK (&req->work, io_sq_wq_submit_work); -queue_work (ctx->sqo_wq, &req->work); +if (!io_add_to_prev_work (list, req)) { +if (list) +atomic_inc (&list->cnt); +INIT_WORK (&req->work, io_sq_wq_submit_work); +queue_work (ctx->sqo_wq, &req->work); + } ret =0 ; }if (ret)
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 19/19 io_uring: add io_uring_event cache hit information
In reply to Jens AxboeAdd hint on whether a read was served out of the page cache, or if it hit media. This is useful for buffered async IO, O_DIRECT reads would never have this set (for obvious reasons).
If the read hit page cache, cqe->flags will have IOCQE_FLAG_CACHEHIT set.
--- fs/io_uring.c | 7 ++++++- include/uapi/linux/io_uring.h | 5 +++++ 2 files changed, 11 insertions(+), 1 deletion(-)
fs/io_uring.c +6 -1
@@ -583,11 +583,16 @@ static void io_fput(struct io_kiocb *req)static void io_complete_rw (struct kiocb *kiocb,long res,long res2) { struct io_kiocb *req = container_of(kiocb, struct io_kiocb, rw); +unsigned ev_flags =0 ;kiocb_end_write (kiocb);io_fput (req); -io_cqring_add_event (req->ctx, req->user_data, res,0 ); + +if (res >0 && (req->flags &REQ_F_FORCE_NONBLOCK )) + ev_flags =IOCQE_FLAG_CACHEHIT ; + +io_cqring_add_event (req->ctx, req->user_data, res, ev_flags);io_free_req (req); }
include/uapi/linux/io_uring.h +5 -0
@@ -69,6 +69,11 @@ struct io_uring_cqe {__u32 flags; }; +/* +* io_uring_event->flags +*/ +#define IOCQE_FLAG_CACHEHIT (1U << 0)/* IO did not hit media */ +/* * Magic offsets for the application to mmap the data it needs */
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>