io_uring IO interface
Subsystems (11)
Jens Axboe
maintainerPATCH v5 1/1 io_uring IO interface
Here's v5 of the io_uring interface. Mostly feels like putting some finishing touches on top of v4, though we do have a few user interface tweaks because of that.
Arnd was kind enough to review the code with an eye towards 32-bit compatability, and that resulted in a few changes. See changelog below.
I also cleaned up the internal ring handling, enabling us to batch writes to the SQ ring head and CQ ring tail. This reduces the number of write ordering barriers we need.
I also dumped the io_submit_state intermediate poll list handling. This drops a patch, and also cleans up the block flush handling since we no longer have to tie into the deep internal of plug callbacks. The win of this just wasn't enough to warrant the complexity.
LWN did a great write up of the API and internals, see that here:
https://lwn.net/Articles/776703/
In terms of benchmarks, I ran some numbers comparing io_uring to libaio and spdk. The tldr is that io_uring is pretty close to spdk, in some cases faster. Latencies over spdk are generally better. The areas where we are still missing a bit of performance all lie in the block layer, and I'll be working on that to close the gap some more.
Latency tests, 3d xpoint, 4k random read
Interface QD Polled Latency IOPS
--------------------------------------------------------------------------io_uring 1 0 9.5usec 77Kio_uring 2 0 8.2usec 183Kio_uring 4 0 8.4usec 383Kio_uring 8 0 13.3usec 449K
libaio 1 0 9.7usec 74K
libaio 2 0 8.5usec 181K
libaio 4 0 8.5usec 373K
libaio 8 0 15.4usec 402K
io_uring 1 1 6.1usec 139Kio_uring 2 1 6.1usec 272Kio_uring 4 1 6.3usec 519Kio_uring 8 1 11.5usec 592K
spdk 1 1 6.1usec 151K
spdk 2 1 6.2usec 293K
spdk 4 1 6.7usec 536K
spdk 8 1 12.6usec 586K
io_uring vs libaio, non polled, io_uring has a slight lead. spdk slightly faster over io_uring polled, especially a lower queue depths. At QD=8, io_uring is faster.
Peak IOPS, 512b random read
Interface QD Polled Latency IOPS
--------------------------------------------------------------------------io_uring 4 1 6.8usec 513Kio_uring 8 1 8.7usec 829Kio_uring 16 1 13.1usec 1019Kio_uring 32 1 20.6usec 1161Kio_uring 64 1 32.4usec 1244K
spdk 4 1 6.8usec 549K
spdk 8 1 8.6usec 865K
spdk 16 1 14.0usec 1105K
spdk 32 1 25.0usec 1227K
spdk 64 1 47.3usec 1251K
io_uring lags spdk about 7% at lower queue depths, getting to within 1% of spdk at higher queue depths.
Peak per-core, multiple devices, 4k random read
Interface QD Polled IOPS
--------------------------------------------------------------------------io_uring 128 1 1620K
libaio 128 0 608K
spdk 128 1 1739K
This is using multiple devices, all running on the same core, meant to test how much performance we can eke out out a single CPU core. spdk has a slight edge over io_uring, with libaio not able to compete at all.
As usual, patches are against 5.0-rc2, and can also be found in my io_uring branch here:
git://git.kernel.dk/linux-block io_uring
Since v4:
- Update some commit messages
- Update some stale comments
- Tweak polling efficiency
- Avoid multiple SQ/CQ ring inc+barriers for batches of IO
- Cache SQ head and CQ tail in the kernel
- Fix buffered rw/work union issue for punted IO
- Drop submit state request issue cache
- Rework io_uring_register() for buffers and files to be more 32-bit friendly
- Make sqe->addr an __u64 instead of playing padding tricks
- Add compat conditional syscall entry for io_uring_setup()
Documentation/filesystems/vfs.txt | 3 + arch/x86/entry/syscalls/syscall_32.tbl | 3 + arch/x86/entry/syscalls/syscall_64.tbl | 3 + block/bio.c | 59 +- fs/Makefile | 1 + fs/block_dev.c | 19 +- fs/file.c | 15 +- fs/file_table.c | 9 +- fs/gfs2/file.c | 2 + fs/io_uring.c | 2017 ++++++++++++++++++++++++ fs/iomap.c | 48 +- fs/xfs/xfs_file.c | 1 + include/linux/bio.h | 14 + include/linux/blk_types.h | 1 + include/linux/file.h | 2 + include/linux/fs.h | 6 +- include/linux/iomap.h | 1 + include/linux/sched/user.h | 2 +- include/linux/syscalls.h | 7 + include/uapi/linux/io_uring.h | 136 ++ init/Kconfig | 9 + kernel/sys_ni.c | 4 + 22 files changed, 2322 insertions(+), 40 deletions(-)
Signature · 8 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 1/15 fs: add an iopoll method to struct file_operations
From: Christoph Hellwig <[email protected]>
This new methods is used to explicitly poll for I/O completion for an iocb. It must be called for any iocb submitted asynchronously (that is with a non-null ki_complete) which has the IOCB_HIPRI flag set.
The method is assisted by a new ki_cookie field in struct iocb to store the polling cookie.
TODO: we can probably union ki_cookie with the existing hint and I/O priority fields to avoid struct kiocb growth.
--- Documentation/filesystems/vfs.txt | 3 +++ include/linux/fs.h | 2 ++ 2 files changed, 5 insertions(+)
Documentation/filesystems/vfs.txt +3 -0
@@ -857,6 +857,7 @@ struct file_operations { ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *); ssize_t (*read_iter) (struct kiocb *, struct iov_iter *); ssize_t (*write_iter) (struct kiocb *, struct iov_iter *); + int (*iopoll)(struct kiocb *kiocb, bool spin); int (*iterate) (struct file *, struct dir_context *); int (*iterate_shared) (struct file *, struct dir_context *); __poll_t (*poll) (struct file *, struct poll_table_struct *);
@@ -902,6 +903,8 @@ otherwise noted. write_iter: possibly asynchronous write with iov_iter as source + iopoll: called when aio wants to poll for completions on HIPRI iocbs + iterate: called when the VFS needs to read the directory contents iterate_shared: called when the VFS needs to read the directory contents
include/linux/fs.h +2 -0
@@ -310,6 +310,7 @@ struct kiocb {int ki_flags;u16 ki_hint;u16 ki_ioprio;/* See linux/ioprio.h */ +unsigned int ki_cookie;/* for ->iopoll */ } __randomize_layout;static inline bool is_sync_kiocb (struct kiocb *kiocb)
@@ -1786,6 +1787,7 @@ struct file_operations {ssize_t (*write) (struct file *,const char __user *,size_t ,loff_t *);ssize_t (*read_iter) (struct kiocb *,struct iov_iter *);ssize_t (*write_iter) (struct kiocb *,struct iov_iter *); +int (*iopoll)(struct kiocb *kiocb,bool spin);int (*iterate) (struct file *,struct dir_context *);int (*iterate_shared) (struct file *,struct dir_context *);__poll_t (*poll) (struct file *,struct poll_table_struct *);
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 2/15 block: wire up block device iopoll method
In reply to Jens AxboeFrom: Christoph Hellwig <[email protected]>
Just call blk_poll on the iocb cookie, we can derive the block device from the inode trivially.
--- fs/block_dev.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
fs/block_dev.c +10 -0
@@ -279,6 +279,14 @@ struct blkdev_dio {static struct bio_set blkdev_dio_pool; +static int blkdev_iopoll (struct kiocb *kiocb,bool wait) +{ +struct block_device *bdev =I_BDEV (kiocb->ki_filp->f_mapping->host); +struct request_queue *q =bdev_get_queue (bdev); + +return blk_poll (q,READ_ONCE (kiocb->ki_cookie), wait); +} +static void blkdev_bio_end_io (struct bio *bio) {struct blkdev_dio *dio = bio->bi_private;
@@ -396,6 +404,7 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) bio->bi_opf |=REQ_HIPRI ; qc =submit_bio (bio); +WRITE_ONCE (iocb->ki_cookie, qc);break ; }
@@ -2068,6 +2077,7 @@ const struct file_operations def_blk_fops = { .llseek = block_llseek, .read_iter = blkdev_read_iter, .write_iter = blkdev_write_iter, + .iopoll = blkdev_iopoll, .mmap = generic_file_mmap, .fsync = blkdev_fsync, .unlocked_ioctl = block_ioctl,
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 3/15 block: add bio_set_polled() helper
In reply to Jens AxboeFor the upcoming async polled IO, we can't sleep allocating requests. If we do, then we introduce a deadlock where the submitter already has async polled IO in-flight, but can't wait for them to complete since polled requests must be active found and reaped.
Utilize the helper in the blockdev DIRECT_IO code.
--- fs/block_dev.c | 4 ++-- include/linux/bio.h | 14 ++++++++++++++ 2 files changed, 16 insertions(+), 2 deletions(-)
fs/block_dev.c +2 -2
@@ -233,7 +233,7 @@ __blkdev_direct_IO_simple(struct kiocb *iocb, struct iov_iter *iter,task_io_account_write (ret); }if (iocb->ki_flags &IOCB_HIPRI ) - bio.bi_opf |=REQ_HIPRI ; +bio_set_polled (&bio, iocb); qc =submit_bio (&bio);for (;;) {
@@ -401,7 +401,7 @@ __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, int nr_pages) nr_pages =iov_iter_npages (iter,BIO_MAX_PAGES );if (!nr_pages) {if (iocb->ki_flags &IOCB_HIPRI ) - bio->bi_opf |=REQ_HIPRI ; +bio_set_polled (bio, iocb); qc =submit_bio (bio);WRITE_ONCE (iocb->ki_cookie, qc);
include/linux/bio.h +14 -0
@@ -823,5 +823,19 @@ static inline int bio_integrity_add_page(struct bio *bio, struct page *page,#endif /* CONFIG_BLK_DEV_INTEGRITY */ +/* +* Mark a bio as polled. Note that for async polled IO, the caller must +* expect -EWOULDBLOCK if we cannot allocate a request (or other resources). +* We cannot block waiting for requests on polled IO, as those completions +* must be found by the caller. This is different than IRQ driven IO, where +* it's safe to wait for IO to complete. +*/ +static inline void bio_set_polled (struct bio *bio,struct kiocb *kiocb) +{ + bio->bi_opf |=REQ_HIPRI ; +if (!is_sync_kiocb (kiocb)) + bio->bi_opf |=REQ_NOWAIT ; +} +#endif /* CONFIG_BLOCK */ #endif /* __LINUX_BIO_H */
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 4/15 iomap: wire up the iopoll method
In reply to Jens AxboeFrom: Christoph Hellwig <[email protected]>
Store the request queue the last bio was submitted to in the iocb private data in addition to the cookie so that we find the right block device. Also refactor the common direct I/O bio submission code into a nice little helper.
Modified to use bio_set_polled().
--- fs/gfs2/file.c | 2 ++ fs/iomap.c | 43 ++++++++++++++++++++++++++++--------------- fs/xfs/xfs_file.c | 1 + include/linux/iomap.h | 1 + 4 files changed, 32 insertions(+), 15 deletions(-)
fs/gfs2/file.c +2 -0
@@ -1280,6 +1280,7 @@ const struct file_operations gfs2_file_fops = { .llseek = gfs2_llseek, .read_iter = gfs2_file_read_iter, .write_iter = gfs2_file_write_iter, + .iopoll = iomap_dio_iopoll, .unlocked_ioctl = gfs2_ioctl, .mmap = gfs2_mmap, .open = gfs2_open,
@@ -1310,6 +1311,7 @@ const struct file_operations gfs2_file_fops_nolock = { .llseek = gfs2_llseek, .read_iter = gfs2_file_read_iter, .write_iter = gfs2_file_write_iter, + .iopoll = iomap_dio_iopoll, .unlocked_ioctl = gfs2_ioctl, .mmap = gfs2_mmap, .open = gfs2_open,
fs/iomap.c +28 -15
82 lines not on this page · raw message
fs/xfs/xfs_file.c +1 -0
@@ -1203,6 +1203,7 @@ const struct file_operations xfs_file_operations = { .write_iter = xfs_file_write_iter, .splice_read = generic_file_splice_read, .splice_write = iter_file_splice_write, + .iopoll = iomap_dio_iopoll, .unlocked_ioctl = xfs_file_ioctl, #ifdef CONFIG_COMPAT .compat_ioctl = xfs_file_compat_ioctl,
include/linux/iomap.h +1 -0
@@ -162,6 +162,7 @@ typedef int (iomap_dio_end_io_t)(struct kiocb *iocb, ssize_t ret, unsigned flags);ssize_t iomap_dio_rw (struct kiocb *iocb,struct iov_iter *iter,const struct iomap_ops *ops,iomap_dio_end_io_t end_io); +int iomap_dio_iopoll (struct kiocb *kiocb,bool spin);#ifdef CONFIG_SWAP struct file ;
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 5/15 Add io_uring IO interface
In reply to Jens AxboeThe submission queue (SQ) and completion queue (CQ) rings are shared between the application and the kernel. This eliminates the need to copy data back and forth to submit and complete IO.
IO submissions use the io_uring_sqe data structure, and completions are generated in the form of io_uring_sqe data structures. The SQ ring is an index into the io_uring_sqe array, which makes it possible to submit a batch of IOs without them being contiguous in the ring. The CQ ring is always contiguous, as completion events are inherently unordered, and hence any io_uring_cqe entry can point back to an arbitrary submission.
Two new system calls are added for this:
io_uring_setup(entries, params)
Sets up a context for doing async IO. On success, returns a file descriptor that the application can mmap to gain access to the SQ ring, CQ ring, and io_uring_sqes.
io_uring_enter(fd, to_submit, min_complete, flags)
Initiates IO against the rings mapped to this fd, or waits for them to complete, or both The behavior is controlled by the parameters passed in. If 'min_complete' is non-zero, then we'll try and submit new IO. If IORING_ENTER_GETEVENTS is set, the kernel will wait for 'min_complete' events, if they aren't already available. It's valid to set IORING_ENTER_GETEVENTS and 'min_complete' == 0 at the same time, this allows the kernel to return already completed events without waiting for them. This is useful only for polling, as for IRQ driven IO, the application can just check the CQ ring without entering the kernel.
With this setup, it's possible to do async IO with a single system call. Future developments will enable polled IO with this interface, and polled submission as well. The latter will enable an application to do IO without doing ANY system calls at all.
For IRQ driven IO, an application only needs to enter the kernel for completions if it wants to wait for them to occur.
Each io_uring is backed by a workqueue, to support buffered async IO as well. We will only punt to an async context if the command would need to wait for IO on the device side. Any data that can be accessed directly in the page cache is done inline. This avoids the slowness issue of usual threadpools, since cached data is accessed as quickly as a sync interface.
Sample application: http://git.kernel.dk/cgit/fio/plain/t/io_uring.c
--- arch/x86/entry/syscalls/syscall_32.tbl | 2 + arch/x86/entry/syscalls/syscall_64.tbl | 2 + fs/Makefile | 1 + fs/io_uring.c | 994 +++++++++++++++++++++++++ include/linux/syscalls.h | 5 + include/uapi/linux/io_uring.h | 94 +++ init/Kconfig | 9 + kernel/sys_ni.c | 3 + 8 files changed, 1110 insertions(+) create mode 100644 fs/io_uring.c create mode 100644 include/uapi/linux/io_uring.h
arch/x86/entry/syscalls/syscall_32.tbl +2 -0
@@ -398,3 +398,5 @@ 384 i386 arch_prctl sys_arch_prctl __ia32_compat_sys_arch_prctl 385 i386 io_pgetevents sys_io_pgetevents __ia32_compat_sys_io_pgetevents 386 i386 rseq sys_rseq __ia32_sys_rseq +387 i386 io_uring_setup sys_io_uring_setup __ia32_compat_sys_io_uring_setup +388 i386 io_uring_enter sys_io_uring_enter __ia32_sys_io_uring_enter
arch/x86/entry/syscalls/syscall_64.tbl +2 -0
@@ -343,6 +343,8 @@ 332 common statx __x64_sys_statx 333 common io_pgetevents __x64_sys_io_pgetevents 334 common rseq __x64_sys_rseq +335 common io_uring_setup __x64_sys_io_uring_setup +336 common io_uring_enter __x64_sys_io_uring_enter # # x32-specific system call numbers start at 512 to avoid cache impact
fs/Makefile +1 -0
@@ -30,6 +30,7 @@ obj-$(CONFIG_TIMERFD) += timerfd.o obj-$(CONFIG_EVENTFD) += eventfd.o obj-$(CONFIG_USERFAULTFD) += userfaultfd.o obj-$(CONFIG_AIO) += aio.o +obj-$(CONFIG_IO_URING) += io_uring.o obj-$(CONFIG_FS_DAX) += dax.o obj-$(CONFIG_FS_ENCRYPTION) += crypto/ obj-$(CONFIG_FILE_LOCKING) += locks.o
fs/io_uring.c new file +994 -0
994 lines not on this page · raw message
include/linux/syscalls.h +5 -0
@@ -69,6 +69,7 @@ struct file_handle;struct sigaltstack ;struct rseq ;union bpf_attr ; +struct io_uring_params ;#include <linux/types.h> #include <linux/aio_abi.h>
@@ -309,6 +310,10 @@ asmlinkage long sys_io_pgetevents_time32(aio_context_t ctx_id,struct io_event __user *events, struct old_timespec32 __user *timeout, const struct __aio_sigset *sig); +asmlinkage long sys_io_uring_setup (u32 entries, +struct io_uring_params __user *p); +asmlinkage long sys_io_uring_enter (unsigned int fd,u32 to_submit, +u32 min_complete,u32 flags);/* fs/xattr.c */ asmlinkage long sys_setxattr(const char __user *path, const char __user *name,
include/uapi/linux/io_uring.h new file +94 -0
94 lines not on this page · raw message
init/Kconfig +9 -0
@@ -1402,6 +1402,15 @@ config AIO by some high performance threaded applications. Disabling this option saves about 7k. +config IO_URING + bool "Enable IO uring support" if EXPERT + select ANON_INODES + default y + help + This option enables support for the io_uring interface, enabling + applications to submit and completion IO through submission and + completion rings that are shared between the kernel and application. + config ADVISE_SYSCALLS bool "Enable madvise/fadvise syscalls" if EXPERT default y
kernel/sys_ni.c +3 -0
@@ -46,6 +46,9 @@ COND_SYSCALL(io_getevents);COND_SYSCALL (io_pgetevents);COND_SYSCALL_COMPAT (io_getevents);COND_SYSCALL_COMPAT (io_pgetevents); +COND_SYSCALL (io_uring_setup); +COND_SYSCALL_COMPAT (io_uring_setup); +COND_SYSCALL (io_uring_enter);/* fs/xattr.c */
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 05/15] Add io_uring IO interface
Hi Jens,
On 2019-01-16 18:49, Jens Axboe wrote:
[...]
fs/io_uring.c line 838
Quoted 7 lines
fs/io_uring.c new file +3
-0+static void *io_mem_alloc (size_t size) +{ +gfp_t gfp_flags =GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN | __GFP_COMP|
fs/io_uring.c new file +3
-0+ __GFP_NORETRY; + +return (void *)__get_free_pages (gfp_flags,get_order (size));
Since these pages are shared between kernel and userspace, do we need to care about d-cache aliasing on armv6 (or other "strange" archs which I've never seen) with vivt or vipt cpu caches?
E.g. vmalloc_user() targets this problem by aligning kernel address on SHMLBA, so no flush_dcache_page() is required.
--
Roman
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 05/15] Add io_uring IO interface
Quoted 18 lines · Roman Penyaev
Hi Jens,
On 2019-01-16 18:49, Jens Axboe wrote:
[...]
fs/io_uring.c line 838
Quoted 7 lines
fs/io_uring.c new file +3
-0+static void *io_mem_alloc (size_t size) +{ +gfp_t gfp_flags =GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN | __GFP_COMP|
fs/io_uring.c new file +3
-0+ __GFP_NORETRY; + +return (void *)__get_free_pages (gfp_flags,get_order (size));Since these pages are shared between kernel and userspace, do we need to care about d-cache aliasing on armv6 (or other "strange" archs which I've never seen) with vivt or vipt cpu caches?
Quoted 2 lines
E.g.
vmalloc_user()targets this problem by aligning kernel address on SHMLBA, so noflush_dcache_page()is required.
I'm honestly not sure, it'd be trivial enough to stick a flush_dcache_page() into the few areas we'd need it. The rings are already page (SHMLBA) aligned.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 05/15] Add io_uring IO interface
Quoted 20 lines · Jens Axboe
Quoted 19 lines · Roman Penyaev
Hi Jens,
On 2019-01-16 18:49, Jens Axboe wrote:
[...]
fs/io_uring.c line 838
Quoted 8 lines
fs/io_uring.c new file +3
-0+static void *io_mem_alloc (size_t size) +{ + gfp_t gfp_flags = GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN |
__GFP_COMP
|fs/io_uring.c new file +3
-0+ __GFP_NORETRY; + +return (void *)__get_free_pages (gfp_flags,get_order (size));Since these pages are shared between kernel and userspace, do we need to care about d-cache aliasing on armv6 (or other "strange" archs which I've never seen) with vivt or vipt cpu caches?
Quoted 6 lines
Quoted 2 lines
E.g.
vmalloc_user()targets this problem by aligning kernel address on SHMLBA, so noflush_dcache_page()is required.I'm honestly not sure, it'd be trivial enough to stick a
flush_dcache_page()into the few areas we'd need it. The rings are already page (SHMLBA) aligned.
For arm SHMLBA is not a page, it is 4x page. So for userspace vaddr which mmap() returns is aligned, but for kernel not. So indeed flush_dcache_page() should be used.
The other question which I can't answer myself is the order of flush_dcache_page() and smp_wmb(). Does flush_scache_page() implies
flush of the cpu write buffer? Or firstly smp_wmb() should be done
in order to flush everything to cache. Here is what arm spec says about write-back cache:
"Writes that miss in the cache are placed in the write buffer and appear on the AMBA ASB interface. The CPU continues execution as soon as the write is placed in the write buffer."
So if you firstly do flush_dcache_page() will it flush write buffer? Because it seems that firstly smp_wmb() and then flush_dcache_page(), or I am going mad?
--
Roman
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 05/15] Add io_uring IO interface
Quoted 24 lines · Roman Penyaev
Quoted 23 lines · Jens Axboe
Quoted 21 lines · Roman Penyaev
Hi Jens,
On 2019-01-16 18:49, Jens Axboe wrote:
[...]
fs/io_uring.c line 838
Quoted 8 lines
fs/io_uring.c new file +3
-0+static void *io_mem_alloc (size_t size) +{ + gfp_t gfp_flags = GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN |
__GFP_COMP
|fs/io_uring.c new file +3
-0+ __GFP_NORETRY; + +return (void *)__get_free_pages (gfp_flags,get_order (size));Since these pages are shared between kernel and userspace, do we need to care about d-cache aliasing on armv6 (or other "strange" archs which I've never seen) with vivt or vipt cpu caches?
E.g.
vmalloc_user()targets this problem by aligning kernel address on SHMLBA, so noflush_dcache_page()is required.
Quoted 7 lines
Quoted 3 lines
I'm honestly not sure, it'd be trivial enough to stick a
flush_dcache_page()into the few areas we'd need it. The rings are already page (SHMLBA) aligned.For arm SHMLBA is not a page, it is 4x page. So for userspace vaddr which
mmap()returns is aligned, but for kernel not. So indeedflush_dcache_page()should be used.
Oh indeed, my bad.
Quoted 6 lines
The other question which I can't answer myself is the order of
flush_dcache_page()andsmp_wmb(). Doesflush_scache_page()implies
flush of the cpu write buffer? Or firstlysmp_wmb()should be done
in order to flush everything to cache. Here is what arm spec says about write-back cache:
Quoted 7 lines
"Writes that miss in the cache are placed in the write buffer and appear on the AMBA ASB interface. The CPU continues execution as soon as the write is placed in the write buffer."
So if you firstly do
flush_dcache_page()will it flush write buffer? Because it seems that firstlysmp_wmb()and thenflush_dcache_page(), or I am going mad?
I don't think you're going mad! We'd first need smp_wmb() to order the writes, then the flush_dcache_page(). For filling the CQ ring, we'd also need to flush the page the cqe belongs to.
Question is if we care enough about performance on vivt to do something about that. I know what my answer will be... If others care, they can incrementally improve upon that.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 05/15] Add io_uring IO interface
Quoted 45 lines · Jens Axboe
Quoted 24 lines · Roman Penyaev
Quoted 23 lines · Jens Axboe
Quoted 21 lines · Roman Penyaev
Hi Jens,
On 2019-01-16 18:49, Jens Axboe wrote:
[...]
fs/io_uring.c line 838
Quoted 8 lines
fs/io_uring.c new file +3
-0+static void *io_mem_alloc (size_t size) +{ + gfp_t gfp_flags = GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN |
__GFP_COMP
|fs/io_uring.c new file +3
-0+ __GFP_NORETRY; + +return (void *)__get_free_pages (gfp_flags,get_order (size));Since these pages are shared between kernel and userspace, do we need to care about d-cache aliasing on armv6 (or other "strange" archs which I've never seen) with vivt or vipt cpu caches?
E.g.
vmalloc_user()targets this problem by aligning kernel address on SHMLBA, so noflush_dcache_page()is required.Quoted 7 lines
Quoted 3 lines
I'm honestly not sure, it'd be trivial enough to stick a
flush_dcache_page()into the few areas we'd need it. The rings are already page (SHMLBA) aligned.For arm SHMLBA is not a page, it is 4x page. So for userspace vaddr which
mmap()returns is aligned, but for kernel not. So indeedflush_dcache_page()should be used.Oh indeed, my bad.
Quoted 10 lines
The other question which I can't answer myself is the order of
flush_dcache_page()andsmp_wmb(). Doesflush_scache_page()implies
flush of the cpu write buffer? Or firstlysmp_wmb()should be done
in order to flush everything to cache. Here is what arm spec says about write-back cache:"Writes that miss in the cache are placed in the write buffer and appear on the AMBA ASB interface. The CPU continues execution as soon as the write is placed in the write buffer."
Quoted 7 lines
Quoted 3 lines
So if you firstly do
flush_dcache_page()will it flush write buffer? Because it seems that firstlysmp_wmb()and thenflush_dcache_page(), or I am going mad?I don't think you're going mad! We'd first need
smp_wmb()to order the writes, then theflush_dcache_page(). For filling the CQ ring, we'd also need to flush the page the cqe belongs to.
Then this is the issue for aio.c as well.
Quoted 4 lines
Question is if we care enough about performance on vivt to do something about that. I know what my answer will be... If others care, they can incrementally improve upon that.
That's perfect answer! May I reuse it? :) Because I expect same questions (if someone cares) for my attempt to do uring for epoll, where I want rely on vmalloc_user() and not to call flush_dcache_page() at all.
--
Roman
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 05/15] Add io_uring IO interface
In reply to Jens AxboeOn 2019-01-16 18:49, Jens Axboe wrote:
[...]
fs/io_uring.c line 849
Quoted 9 lines
fs/io_uring.c new file +9
-0+static int io_allocate_scq_urings (struct io_ring_ctx *ctx, +struct io_uring_params *p) +{ +struct io_sq_ring *sq_ring; +struct io_cq_ring *cq_ring; +size_t size; +int ret; + + sq_ring =io_mem_alloc (struct_size (sq_ring, array, p->sq_entries));
It seems that sq_entries, cq_entries are not limited at all. Can nasty app consume a lot of kernel pages calling io_setup_uring() from a loop passing random entries number? (or even better: decreasing entries number, in order to consume all pages orders with min number of loops).
--
Roman
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 05/15] Add io_uring IO interface
Quoted 4 lines · Roman Penyaev
On 2019-01-16 18:49, Jens Axboe wrote:
[...]
Quoted 15 lines
fs/io_uring.c line 849
Quoted 9 lines
fs/io_uring.c new file +9
-0+static int io_allocate_scq_urings (struct io_ring_ctx *ctx, +struct io_uring_params *p) +{ +struct io_sq_ring *sq_ring; +struct io_cq_ring *cq_ring; +size_t size; +int ret; + + sq_ring =io_mem_alloc (struct_size (sq_ring, array, p->sq_entries));It seems that
sq_entries,cq_entriesare not limited at all. Can nasty app consume a lot of kernel pages callingio_setup_uring()from a loop passing random entries number? (or even better: decreasing entries number,
in order to consume all pages orders with min number of loops).
Yes, that's an oversight, we should have a limit in place. I'll add that.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 05/15] Add io_uring IO interface
Quoted 21 lines · Jens Axboe
Quoted 4 lines · Roman Penyaev
On 2019-01-16 18:49, Jens Axboe wrote:
[...]
Quoted 15 lines
fs/io_uring.c line 849
Quoted 9 lines
fs/io_uring.c new file +9
-0+static int io_allocate_scq_urings (struct io_ring_ctx *ctx, +struct io_uring_params *p) +{ +struct io_sq_ring *sq_ring; +struct io_cq_ring *cq_ring; +size_t size; +int ret; + + sq_ring =io_mem_alloc (struct_size (sq_ring, array, p->sq_entries));It seems that
sq_entries,cq_entriesare not limited at all. Can nasty app consume a lot of kernel pages callingio_setup_uring()from a loop passing random entries number? (or even better: decreasing entries number,
in order to consume all pages orders with min number of loops).
Quoted 1 line
Yes, that's an oversight, we should have a limit in place. I'll add that.
Can we charge the ring memory to the RLIMIT_MEMLOCK as well? I'd prefer not to repeat the mistake of fs.aio-max-nr.
-Jeff
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 05/15] Add io_uring IO interface
Quoted 23 lines · Jeff Moyer
Quoted 21 lines · Jens Axboe
Quoted 19 lines · Roman Penyaev
On 2019-01-16 18:49, Jens Axboe wrote:
[...]
fs/io_uring.c line 849
Quoted 9 lines
fs/io_uring.c new file +9
-0+static int io_allocate_scq_urings (struct io_ring_ctx *ctx, +struct io_uring_params *p) +{ +struct io_sq_ring *sq_ring; +struct io_cq_ring *cq_ring; +size_t size; +int ret; + + sq_ring =io_mem_alloc (struct_size (sq_ring, array, p->sq_entries));It seems that
sq_entries,cq_entriesare not limited at all. Can nasty app consume a lot of kernel pages callingio_setup_uring()from a loop passing random entries number? (or even better: decreasing entries number,
in order to consume all pages orders with min number of loops).
Quoted 4 lines
Quoted 1 line
Yes, that's an oversight, we should have a limit in place. I'll add that.
Can we charge the ring memory to the
RLIMIT_MEMLOCKas well? I'd prefer not to repeat the mistake of fs.aio-max-nr.
Sure, we can do that. With the ring limited in size (it's now 4k entries at most), the amount of memory gobbled up by that is much smaller than the fixed buffers. A max sized ring is about 256k of memory.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 05/15] Add io_uring IO interface
Quoted 24 lines · Jens Axboe
Quoted 23 lines · Jeff Moyer
Quoted 21 lines · Jens Axboe
Quoted 19 lines · Roman Penyaev
On 2019-01-16 18:49, Jens Axboe wrote:
[...]
fs/io_uring.c line 849
Quoted 9 lines
fs/io_uring.c new file +9
-0+static int io_allocate_scq_urings (struct io_ring_ctx *ctx, +struct io_uring_params *p) +{ +struct io_sq_ring *sq_ring; +struct io_cq_ring *cq_ring; +size_t size; +int ret; + + sq_ring =io_mem_alloc (struct_size (sq_ring, array, p->sq_entries));It seems that
sq_entries,cq_entriesare not limited at all. Can nasty app consume a lot of kernel pages callingio_setup_uring()from a loop passing random entries number? (or even better: decreasing entries number,
in order to consume all pages orders with min number of loops).
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Yes, that's an oversight, we should have a limit in place. I'll add that.
Can we charge the ring memory to the
RLIMIT_MEMLOCKas well? I'd prefer not to repeat the mistake of fs.aio-max-nr.Sure, we can do that. With the ring limited in size (it's now 4k entries at most), the amount of memory gobbled up by that is much smaller than the fixed buffers. A max sized ring is about 256k of memory.
One concern here is that, at least looking at my boxes, the default setting for RLIMIT_MEMLOCK is really low. I'd hate for everyone to run into issues using io_uring just because it seems to require root, because the memlock limit is so low.
That's much less of a concern with the fixed buffers, since it's a more esoteric part of it. But everyone should be able to setup a few io_uring queues and use them without having to worry about failing due to an absurdly low RLIMIT_MEMLOCK.
Comments?
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 05/15] Add io_uring IO interface
Quoted 25 lines · Jens Axboe
Quoted 24 lines · Jens Axboe
Quoted 23 lines · Jeff Moyer
Quoted 21 lines · Jens Axboe
Quoted 19 lines · Roman Penyaev
On 2019-01-16 18:49, Jens Axboe wrote:
[...]
fs/io_uring.c line 849
Quoted 9 lines
fs/io_uring.c new file +9
-0+static int io_allocate_scq_urings (struct io_ring_ctx *ctx, +struct io_uring_params *p) +{ +struct io_sq_ring *sq_ring; +struct io_cq_ring *cq_ring; +size_t size; +int ret; + + sq_ring =io_mem_alloc (struct_size (sq_ring, array, p->sq_entries));It seems that
sq_entries,cq_entriesare not limited at all. Can nasty app consume a lot of kernel pages callingio_setup_uring()from a loop passing random entries number? (or even better: decreasing entries number,
in order to consume all pages orders with min number of loops).
Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Yes, that's an oversight, we should have a limit in place. I'll add that.
Can we charge the ring memory to the
RLIMIT_MEMLOCKas well? I'd prefer not to repeat the mistake of fs.aio-max-nr.Sure, we can do that. With the ring limited in size (it's now 4k entries at most), the amount of memory gobbled up by that is much smaller than the fixed buffers. A max sized ring is about 256k of memory.
Per io_uring. Nothing prevents a user from calling io_uring_setup in a loop and continuing to gobble up memory.
Quoted 11 lines
One concern here is that, at least looking at my boxes, the default setting for
RLIMIT_MEMLOCKis really low. I'd hate for everyone to run into issues usingio_uringjust because it seems to require root, because the memlock limit is so low.That's much less of a concern with the fixed buffers, since it's a more esoteric part of it. But everyone should be able to setup a few
io_uringqueues and use them without having to worry about failing due to an absurdly lowRLIMIT_MEMLOCK.Comments?
Yeah, the default is 64k here. We should probably up that. I'd say we either tackle the ridiculously low rlimits, or I guess we just go the aio route and add a sysctl. :-\ I'll see what's involved in the former.
-Jeff
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 05/15] Add io_uring IO interface
Quoted 49 lines · Jeff Moyer
Quoted 25 lines · Jens Axboe
Quoted 24 lines · Jens Axboe
Quoted 23 lines · Jeff Moyer
Quoted 21 lines · Jens Axboe
Quoted 19 lines · Roman Penyaev
On 2019-01-16 18:49, Jens Axboe wrote:
[...]
Review
Quoted 9 lines
(quoted diff) +9
-0+static int io_allocate_scq_urings(struct io_ring_ctx *ctx, + struct io_uring_params *p) +{ + struct io_sq_ring *sq_ring; + struct io_cq_ring *cq_ring; + size_t size; + int ret; + + sq_ring = io_mem_alloc(struct_size(sq_ring, array, p->sq_entries));It seems that
sq_entries,cq_entriesare not limited at all. Can nasty app consume a lot of kernel pages callingio_setup_uring()from a loop passing random entries number? (or even better: decreasing entries number,
in order to consume all pages orders with min number of loops).Quoted 8 lines
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Yes, that's an oversight, we should have a limit in place. I'll add that.
Can we charge the ring memory to the
RLIMIT_MEMLOCKas well? I'd prefer not to repeat the mistake of fs.aio-max-nr.Sure, we can do that. With the ring limited in size (it's now 4k entries at most), the amount of memory gobbled up by that is much smaller than the fixed buffers. A max sized ring is about 256k of memory.
Per
io_uring. Nothing prevents a user from callingio_uring_setupin a loop and continuing to gobble up memory.Quoted 10 lines
One concern here is that, at least looking at my boxes, the default setting for
RLIMIT_MEMLOCKis really low. I'd hate for everyone to run into issues usingio_uringjust because it seems to require root, because the memlock limit is so low.That's much less of a concern with the fixed buffers, since it's a more esoteric part of it. But everyone should be able to setup a few
io_uringqueues and use them without having to worry about failing due to an absurdly lowRLIMIT_MEMLOCK.
Quoted 6 lines
Quoted 1 line
Comments?
Yeah, the default is 64k here. We should probably up that. I'd say we either tackle the ridiculously low rlimits, or I guess we just go the aio route and add a sysctl. :-\ I'll see what's involved in the former.
After giving it a bit of thought, let's go the rlimit route. It is cleaner, and I don't want a sysctl knob for this either. 64k will enable anyone to set up at least one decently sized ring.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 05/15] Add io_uring IO interface
Quoted 5 lines · Jens Axboe
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
It seems that
sq_entries,cq_entriesare not limited at all. Can nasty app consume a lot of kernel pages callingio_setup_uring()from a loop passing random entries number? (or even better: decreasing entries number,
in order to consume all pages orders with min number of loops).
...
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
One concern here is that, at least looking at my boxes, the default setting for
RLIMIT_MEMLOCKis really low. I'd hate for everyone to run into issues usingio_uringjust because it seems to require root, because the memlock limit is so low.That's much less of a concern with the fixed buffers, since it's a more esoteric part of it. But everyone should be able to setup a few
io_uringqueues and use them without having to worry about failing due to an absurdly lowRLIMIT_MEMLOCK.Comments?
Quoted 8 lines
Quoted 4 lines
Yeah, the default is 64k here. We should probably up that. I'd say we either tackle the ridiculously low rlimits, or I guess we just go the aio route and add a sysctl. :-\ I'll see what's involved in the former.
After giving it a bit of thought, let's go the rlimit route. It is cleaner, and I don't want a sysctl knob for this either. 64k will enable anyone to set up at least one decently sized ring.
OK. Note that the MLOCK_LIMIT size has been dictated by gpg's requirements:
commit f947ff8af30f75cb9cf0e966caf8f4809ad1b92e
Author: Rik van Riel <[email protected]> Date: Sun Aug 22 23:06:58 2004 -0700
[PATCH] increase per-user mlock limit default to 32k
Since various gnupg users have indicated that gpg wants to mlock 32kB of memory, I created the patch below that increases the default mlock ulimit to 32kB.
and then
commit 0833422274ff00729a603b020fac297e69a03e40
Author: Kurt Garloff <[email protected]> Date: Wed Oct 29 14:00:48 2008 -0700
mm: increase the default mlock limit from 32k to 64k
...
However, newer gpg2 needs 64k in various circumstances and otherwise fails miserably, see bnc#329675.
So all we need to do is modify gpg2 so that is requires more locked memory, and we're golden! ;-)
-Jeff
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 05/15] Add io_uring IO interface
Quoted 48 lines · Jeff Moyer
Quoted 5 lines · Jens Axboe
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
It seems that
sq_entries,cq_entriesare not limited at all. Can nasty app consume a lot of kernel pages callingio_setup_uring()from a loop passing random entries number? (or even better: decreasing entries number,
in order to consume all pages orders with min number of loops)....
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
One concern here is that, at least looking at my boxes, the default setting for
RLIMIT_MEMLOCKis really low. I'd hate for everyone to run into issues usingio_uringjust because it seems to require root, because the memlock limit is so low.That's much less of a concern with the fixed buffers, since it's a more esoteric part of it. But everyone should be able to setup a few
io_uringqueues and use them without having to worry about failing due to an absurdly lowRLIMIT_MEMLOCK.Comments?
Quoted 8 lines
Quoted 4 lines
Yeah, the default is 64k here. We should probably up that. I'd say we either tackle the ridiculously low rlimits, or I guess we just go the aio route and add a sysctl. :-\ I'll see what's involved in the former.
After giving it a bit of thought, let's go the rlimit route. It is cleaner, and I don't want a sysctl knob for this either. 64k will enable anyone to set up at least one decently sized ring.
OK. Note that the
MLOCK_LIMITsize has been dictated by gpg's requirements:commit
f947ff8af30f75cb9cf0e966caf8f4809ad1b92eAuthor: Rik van Riel <[email protected]> Date: Sun Aug 22 23:06:58 2004 -0700[PATCH] increase per-user mlock limit default to 32k
Since various gnupg users have indicated that gpg wants to mlock 32kB of memory, I created the patch below that increases the default mlock ulimit to 32kB.
and then
commit
0833422274ff00729a603b020fac297e69a03e40Author: Kurt Garloff <[email protected]> Date: Wed Oct 29 14:00:48 2008 -0700
Quoted 8 lines
mm: increase the default mlock limit from 32k to 64k
...
However, newer gpg2 needs 64k in various circumstances and otherwise fails miserably, see bnc#329675.So all we need to do is modify gpg2 so that is requires more locked memory, and we're golden! ;-)
Haha, that's some nice digging there!
Yes, we could bump it, but with the default, we can get a 512 sized ring per user, that's 13 pages (rounded up). Probably good enough to get things off the ground?
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 05/15] Add io_uring IO interface
Quoted 8 lines · Jens Axboe
Quoted 2 lines
So all we need to do is modify gpg2 so that is requires more locked memory, and we're golden! ;-)
Haha, that's some nice digging there!
Yes, we could bump it, but with the default, we can get a 512 sized ring per user, that's 13 pages (rounded up). Probably good enough to get things off the ground?
Agreed. I'll work on bloating gpg as a background task. =P Or I guess we could add instructions for modifying /etc/security/limits.conf to the man pages.
Cheers,
Jeff
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 05/15] Add io_uring IO interface
Quoted 7 lines · Jeff Moyer
Quoted 5 lines · Jens Axboe
Quoted 2 lines
So all we need to do is modify gpg2 so that is requires more locked memory, and we're golden! ;-)
Haha, that's some nice digging there!
Quoted 7 lines
Quoted 3 lines
Yes, we could bump it, but with the default, we can get a 512 sized ring per user, that's 13 pages (rounded up). Probably good enough to get things off the ground?
Agreed. I'll work on bloating gpg as a background task. =P Or I guess we could add instructions for modifying
/etc/security/limits.confto the man pages.
Pushed out with that. Yes, let's ensure that the man pages tell you how to bump this limit. And we may just consider bumping the default 64k limit to 128k on the side. Or maybe go crazy and make it 256k per user, damn.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 05/15] Add io_uring IO interface
In reply to Jeff MoyerQuoted 33 lines · Jeff Moyer
Quoted 31 lines · Jens Axboe
Quoted 30 lines · Jens Axboe
Quoted 28 lines · Jeff Moyer
Quoted 22 lines · Jens Axboe
Quoted 4 lines · Roman Penyaev
On 2019-01-16 18:49, Jens Axboe wrote:
[...]
Quoted 16 lines
Quoted 10 lines
(quoted diff) +9
-0+static int io_allocate_scq_urings(struct io_ring_ctx *ctx, + struct io_uring_params *p) +{ + struct io_sq_ring *sq_ring; + struct io_cq_ring *cq_ring; + size_t size; + int ret; + + sq_ring = io_mem_alloc(struct_size(sq_ring, array,
p->sq_entries));It seems that
sq_entries,cq_entriesare not limited at all. Can nasty app consume a lot of kernel pages callingio_setup_uring()from a loop passing random entries number? (or even better: decreasing entries number,
in order to consume all pages orders with min number of loops).Quoted 1 line
Yes, that's an oversight, we should have a limit in place. I'll add that.
Can we charge the ring memory to the
RLIMIT_MEMLOCKas well? I'd prefer not to repeat the mistake of fs.aio-max-nr.
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Sure, we can do that. With the ring limited in size (it's now 4k entries at most), the amount of memory gobbled up by that is much smaller than the fixed buffers. A max sized ring is about 256k of memory.
Per
io_uring. Nothing prevents a user from callingio_uring_setupin a loop and continuing to gobble up memory.
What if we set a sane limit for a uring instance (not for the whole io_uring), but allocate rings on mmap? Then greedy / nasty app will be killed by oom.
--
Roman
--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 6/15 io_uring: add fsync support
In reply to Jens AxboeFrom: Christoph Hellwig <[email protected]>
Add a new fsync opcode, which either syncs a range if one is passed, or the whole file if the offset and length fields are both cleared to zero. A flag is provided to use fdatasync semantics, that is only force out metadata which is required to retrieve the file data, but not others like metadata.
--- fs/io_uring.c | 33 +++++++++++++++++++++++++++++++++ include/uapi/linux/io_uring.h | 8 +++++++- 2 files changed, 40 insertions(+), 1 deletion(-)
fs/io_uring.c +33 -0
@@ -457,6 +457,36 @@ static int io_nop(struct io_kiocb *req, const struct io_uring_sqe *sqe)return 0 ; } +static int io_fsync (struct io_kiocb *req,const struct io_uring_sqe *sqe, +bool force_nonblock) +{ +struct io_ring_ctx *ctx = req->ctx; +loff_t end = sqe->off + sqe->len; +struct file *file; +int ret; + +/* fsync always requires a blocking context */ +if (force_nonblock) +return -EAGAIN ; + +if (unlikely (sqe->addr)) +return -EINVAL ; +if (unlikely (sqe->fsync_flags & ~IORING_FSYNC_DATASYNC )) +return -EINVAL ; + + file =fget (sqe->fd); +if (unlikely (!file)) +return -EBADF ; + + ret =vfs_fsync_range (file, sqe->off, end >0 ? end :LLONG_MAX , + sqe->fsync_flags &IORING_FSYNC_DATASYNC ); + +fput (file); +io_cqring_fill_event (ctx, sqe->user_data, ret,0 ); +io_free_req (req); +return 0 ; +} +static int __io_submit_sqe (struct io_ring_ctx *ctx,struct io_kiocb *req,struct sqe_submit *s,bool force_nonblock) {
@@ -482,6 +512,9 @@ static int __io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,case IORING_OP_WRITEV : ret =io_write (req, sqe, force_nonblock);break ; +case IORING_OP_FSYNC : + ret =io_fsync (req, sqe, force_nonblock); +break ;default : ret = -EINVAL ;break ;
include/uapi/linux/io_uring.h +7 -1
@@ -24,7 +24,7 @@ struct io_uring_sqe {__u32 len;/* buffer size or number of iovecs */ union {__kernel_rwf_t rw_flags; -__u32 __resv; +__u32 fsync_flags; };__u64 user_data;/* data to be passed back at completion time */ __u64 __pad2[3 ];
@@ -33,6 +33,12 @@ struct io_uring_sqe {#define IORING_OP_NOP 0#define IORING_OP_READV 1#define IORING_OP_WRITEV 2 +#define IORING_OP_FSYNC 3 + +/* +* sqe->fsync_flags +*/ +#define IORING_FSYNC_DATASYNC (1 << 0)/* * IO completion data structure (Completion Queue Entry)
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 7/15 io_uring: support for IO polling
In reply to Jens AxboeAdd support for a polled io_uring context. When a read or write is submitted to a polled context, the application must poll for completions on the CQ ring through io_uring_enter(2). Polled IO may not generate IRQ completions, hence they need to be actively found by the application itself.
To use polling, io_uring_setup() must be used with the IORING_SETUP_IOPOLL flag being set. It is illegal to mix and match polled and non-polled IO on an io_uring.
--- fs/io_uring.c | 263 ++++++++++++++++++++++++++++++++-- include/uapi/linux/io_uring.h | 5 + 2 files changed, 257 insertions(+), 11 deletions(-)
fs/io_uring.c +252 -11
@@ -90,6 +90,8 @@ struct io_ring_ctx {struct {spinlock_t completion_lock; +struct list_head poll_list; +unsigned poll_multi_file; }____cacheline_aligned_in_smp ; };
@@ -113,10 +115,14 @@ struct io_kiocb {struct list_head list;unsigned long flags;#define REQ_F_FORCE_NONBLOCK 1/* inline submission attempt */ +#define REQ_F_IOPOLL_COMPLETED 2/* polled IO has completed */ +#define REQ_F_IOPOLL_EAGAIN 4/* submission got EAGAIN */ u64 user_data; +u64 res; };#define IO_PLUG_THRESHOLD 2 +#define IO_IOPOLL_BATCH 8static struct kmem_cache *req_cachep;
@@ -146,6 +152,7 @@ static struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)init_completion (&ctx->ctx_done);spin_lock_init (&ctx->completion_lock);init_waitqueue_head (&ctx->wait); +INIT_LIST_HEAD (&ctx->poll_list);mutex_init (&ctx->uring_lock);return ctx; }
@@ -177,8 +184,8 @@ static struct io_uring_cqe *io_get_cqring(struct io_ring_ctx *ctx)return &ring->cqes[tail & ctx->cq_mask]; } -static void __io_cqring_fill_event (struct io_ring_ctx *ctx,u64 ki_user_data, -long res,unsigned ev_flags) +static void io_cqring_add_event (struct io_ring_ctx *ctx,u64 ki_user_data, +long res,unsigned ev_flags) {struct io_uring_cqe *cqe;
@@ -192,11 +199,17 @@ static void __io_cqring_fill_event(struct io_ring_ctx *ctx, u64 ki_user_data, cqe->user_data = ki_user_data; cqe->res = res; cqe->flags = ev_flags; -io_commit_cqring (ctx); }else ctx->cq_ring->overflow++; } +static void __io_cqring_fill_event (struct io_ring_ctx *ctx,u64 ki_user_data, +long res,unsigned ev_flags) +{ +io_cqring_add_event (ctx, ki_user_data, res, ev_flags); +io_commit_cqring (ctx); +} +static void io_cqring_fill_event (struct io_ring_ctx *ctx,u64 ki_user_data,long res,unsigned ev_flags) {
@@ -241,12 +254,158 @@ static void io_ring_drop_ctx_refs(struct io_ring_ctx *ctx, unsigned refs)wake_up (&ctx->wait); } +static void io_free_req_many (struct io_ring_ctx *ctx,void **reqs,int *nr) +{ +if (*nr) { +kmem_cache_free_bulk (req_cachep, *nr, reqs); +io_ring_drop_ctx_refs (ctx, *nr); + *nr =0 ; + } +} +static void io_free_req (struct io_kiocb *req) {kmem_cache_free (req_cachep, req);io_ring_drop_ctx_refs (req->ctx,1 ); } +/* +* Find and free completed poll iocbs +*/ +static void io_iopoll_complete (struct io_ring_ctx *ctx,unsigned int *nr_events, +struct list_head *done) +{ +void *reqs[IO_IOPOLL_BATCH ]; +struct io_kiocb *req; +int to_free =0 ; + +while (!list_empty (done)) { + req = list_first_entry(done, struct io_kiocb, list); +list_del (&req->list); + +io_cqring_add_event (ctx, req->user_data, req->res,0 ); + + reqs[to_free++] = req; + (*nr_events)++; + +fput (req->rw.ki_filp); +if (to_free ==ARRAY_SIZE (reqs)) +io_free_req_many (ctx, reqs, &to_free); + } +io_commit_cqring (ctx); + +if (to_free) +io_free_req_many (ctx, reqs, &to_free); +} + +static int io_do_iopoll (struct io_ring_ctx *ctx,unsigned int *nr_events, +long min) +{ +struct io_kiocb *req, *tmp; +LIST_HEAD (done); +bool spin; +int ret; + +/* +* Only spin for completions if we don't have multiple devices hanging +* off our complete list, and we're under the requested amount. +*/ + spin = !ctx->poll_multi_file && (*nr_events < min); + + ret =0 ; +list_for_each_entry_safe (req, tmp, &ctx->poll_list, list) { +struct kiocb *kiocb = &req->rw; + +/* +* Move completed entries to our local list. If we find a +* request that requires polling, break out and complete +* the done list first, if we have entries there. +*/ +if (req->flags &REQ_F_IOPOLL_COMPLETED ) { +list_move_tail (&req->list, &done); +continue ; + } +if (!list_empty (&done)) +break ; + + ret = kiocb->ki_filp->f_op->iopoll (kiocb, spin); +if (ret <0 ) +break ; + +if (ret && spin) + spin = false; + ret =0 ; + } + +if (!list_empty (&done)) +io_iopoll_complete (ctx, nr_events, &done); + +return ret; +} + +/* +* Poll for a mininum of 'min' events. Note that if min == 0 we consider that a +* non-spinning poll check - we'll still enter the driver poll loop, but only +* as a non-spinning completion check. +*/ +static int io_iopoll_getevents (struct io_ring_ctx *ctx,unsigned int *nr_events, +long min) +{ +int ret; + +do { +if (list_empty (&ctx->poll_list)) +return 0 ; + + ret =io_do_iopoll (ctx, nr_events, min); +if (ret <0 ) +break ; + }while (min && *nr_events < min); + +if (ret <0 ) +return ret; + +return *nr_events < min; +} + +/* +* We can't just wait for polled events to come to us, we have to actively +* find and complete them. +*/ +static void io_iopoll_reap_events (struct io_ring_ctx *ctx) +{ +if (!(ctx->flags &IORING_SETUP_IOPOLL )) +return ; + +mutex_lock (&ctx->uring_lock); +while (!list_empty (&ctx->poll_list)) { +unsigned int nr_events =0 ; + +io_iopoll_getevents (ctx, &nr_events,1 ); + } +mutex_unlock (&ctx->uring_lock); +} + +static int io_iopoll_check (struct io_ring_ctx *ctx,unsigned *nr_events, +long min) +{ +int ret =0 ; + +while (!*nr_events || !need_resched ()) { +int tmin =0 ; + +if (*nr_events < min) + tmin = min - *nr_events; + + ret =io_iopoll_getevents (ctx, nr_events, tmin); +if (ret <=0 ) +break ; + ret =0 ; + } + +return ret; +} +static void kiocb_end_write (struct kiocb *kiocb) {if (kiocb->ki_flags &IOCB_WRITE ) {
@@ -273,9 +432,60 @@ static void io_complete_rw(struct kiocb *kiocb, long res, long res2)io_free_req (req); } +static void io_complete_rw_iopoll (struct kiocb *kiocb,long res,long res2) +{ + struct io_kiocb *req = container_of(kiocb, struct io_kiocb, rw); + +kiocb_end_write (kiocb); + +if (unlikely (res == -EAGAIN )) { + req->flags |=REQ_F_IOPOLL_EAGAIN ; + }else { + req->flags |=REQ_F_IOPOLL_COMPLETED ; + req->res = res; + } +} + +/* +* After the iocb has been issued, it's safe to be found on the poll list. +* Adding the kiocb to the list AFTER submission ensures that we don't +* find it from a io_iopoll_getevents() thread before the issuer is done +* accessing the kiocb cookie. +*/ +static void io_iopoll_req_issued (struct io_kiocb *req) +{ +struct io_ring_ctx *ctx = req->ctx; + +/* +* Track whether we have multiple files in our lists. This will impact +* how we do polling eventually, not spinning if we're on potentially +* different devices. +*/ +if (list_empty (&ctx->poll_list)) { + ctx->poll_multi_file =0 ; + }else if (!ctx->poll_multi_file) { +struct io_kiocb *list_req; + + list_req = list_first_entry(&ctx->poll_list, struct io_kiocb, + list); +if (list_req->rw.ki_filp != req->rw.ki_filp) + ctx->poll_multi_file =1 ; + } + +/* +* For fast devices, IO may have already completed. If it has, add +* it to the front so we find it first. +*/ +if (req->flags &REQ_F_IOPOLL_COMPLETED ) +list_add (&req->list, &ctx->poll_list); +else +list_add_tail (&req->list, &ctx->poll_list); +} +static int io_prep_rw (struct io_kiocb *req,const struct io_uring_sqe *sqe,bool force_nonblock) { +struct io_ring_ctx *ctx = req->ctx;struct kiocb *kiocb = &req->rw;int ret;
@@ -301,12 +511,21 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe, kiocb->ki_flags |=IOCB_NOWAIT ; req->flags |=REQ_F_FORCE_NONBLOCK ; } -if (kiocb->ki_flags &IOCB_HIPRI ) { - ret = -EINVAL ; - goto out_fput; - } +if (ctx->flags &IORING_SETUP_IOPOLL ) { + ret = -EOPNOTSUPP ; +if (!(kiocb->ki_flags &IOCB_DIRECT ) || + !kiocb->ki_filp->f_op->iopoll) + goto out_fput; - kiocb->ki_complete = io_complete_rw; + kiocb->ki_flags |=IOCB_HIPRI ; + kiocb->ki_complete = io_complete_rw_iopoll; + }else { +if (kiocb->ki_flags &IOCB_HIPRI ) { + ret = -EINVAL ; + goto out_fput; + } + kiocb->ki_complete = io_complete_rw; + }return 0 ; out_fput:fput (kiocb->ki_filp);
@@ -452,6 +671,9 @@ static int io_nop(struct io_kiocb *req, const struct io_uring_sqe *sqe) {struct io_ring_ctx *ctx = req->ctx; +if (unlikely (ctx->flags &IORING_SETUP_IOPOLL )) +return -EINVAL ; +__io_cqring_fill_event (ctx, sqe->user_data,0 ,0 );io_free_req (req);return 0 ;
@@ -469,6 +691,8 @@ static int io_fsync(struct io_kiocb *req, const struct io_uring_sqe *sqe,if (force_nonblock)return -EAGAIN ; +if (unlikely (req->ctx->flags &IORING_SETUP_IOPOLL )) +return -EINVAL ;if (unlikely (sqe->addr))return -EINVAL ;if (unlikely (sqe->fsync_flags & ~IORING_FSYNC_DATASYNC ))
@@ -520,7 +744,16 @@ static int __io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,break ; } -return ret; +if (ret) +return ret; + +if (ctx->flags &IORING_SETUP_IOPOLL ) { +if (req->flags &REQ_F_IOPOLL_EAGAIN ) +return -EAGAIN ; +io_iopoll_req_issued (req); + } + +return 0 ; }static void io_sq_wq_submit_work (struct work_struct *work)
@@ -700,12 +933,18 @@ static int __io_uring_enter(struct io_ring_ctx *ctx, unsigned to_submit,return ret; }if (flags &IORING_ENTER_GETEVENTS ) { +unsigned nr_events =0 ;int get_ret;if (!ret && to_submit) min_complete =0 ; - get_ret =io_cqring_wait (ctx, min_complete); +if (ctx->flags &IORING_SETUP_IOPOLL ) + get_ret =io_iopoll_check (ctx, &nr_events, + min_complete); +else + get_ret =io_cqring_wait (ctx, min_complete); +if (get_ret <0 && !ret) ret = get_ret; }
@@ -772,6 +1011,7 @@ static void io_free_scq_urings(struct io_ring_ctx *ctx)static void io_ring_ctx_free (struct io_ring_ctx *ctx) {io_sq_offload_stop (ctx); +io_iopoll_reap_events (ctx);io_free_scq_urings (ctx);percpu_ref_exit (&ctx->refs);kfree (ctx);
@@ -783,6 +1023,7 @@ static void io_ring_ctx_wait_and_kill(struct io_ring_ctx *ctx)percpu_ref_kill (&ctx->refs);mutex_unlock (&ctx->uring_lock); +io_iopoll_reap_events (ctx);wait_for_completion (&ctx->ctx_done);io_ring_ctx_free (ctx); }
@@ -992,7 +1233,7 @@ static long io_uring_setup(u32 entries, struct io_uring_params __user *params,return -EINVAL ; } -if (p.flags) +if (p.flags & ~IORING_SETUP_IOPOLL )return -EINVAL ; ret =io_uring_create (entries, &p, compat);
include/uapi/linux/io_uring.h +5 -0
@@ -30,6 +30,11 @@ struct io_uring_sqe {__u64 __pad2[3 ]; }; +/* +* io_uring_setup() flags +*/ +#define IORING_SETUP_IOPOLL (1 << 0)/* io_context is polled */ +#define IORING_OP_NOP 0#define IORING_OP_READV 1#define IORING_OP_WRITEV 2
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 8/15 fs: add fget_many() and fput_many()
In reply to Jens AxboeSome uses cases repeatedly get and put references to the same file, but the only exposed interface is doing these one at the time. As each of these entail an atomic inc or dec on a shared structure, that cost can add up.
Add fget_many(), which works just like fget(), except it takes an argument for how many references to get on the file. Ditto fput_many(), which can drop an arbitrary number of references to a file.
--- fs/file.c | 15 ++++++++++----- fs/file_table.c | 9 +++++++-- include/linux/file.h | 2 ++ include/linux/fs.h | 4 +++- 4 files changed, 22 insertions(+), 8 deletions(-)
fs/file.c +10 -5
@@ -705,7 +705,7 @@ void do_close_on_exec(struct files_struct *files)spin_unlock (&files->file_lock); } -static struct file *__fget (unsigned int fd,fmode_t mask) +static struct file *__fget (unsigned int fd,fmode_t mask,unsigned int refs) {struct files_struct *files = current->files;struct file *file;
@@ -720,7 +720,7 @@ static struct file *__fget(unsigned int fd, fmode_t mask)*/ if (file->f_mode & mask) file =NULL ; -else if (!get_file_rcu (file)) +else if (!get_file_rcu_many (file, refs)) goto loop; }rcu_read_unlock ();
@@ -728,15 +728,20 @@ static struct file *__fget(unsigned int fd, fmode_t mask)return file; } +struct file *fget_many (unsigned int fd,unsigned int refs) +{ +return __fget (fd,FMODE_PATH , refs); +} +struct file *fget (unsigned int fd) { -return __fget (fd,FMODE_PATH ); +return fget_many (fd,1 ); }EXPORT_SYMBOL (fget);struct file *fget_raw (unsigned int fd) { -return __fget (fd,0 ); +return __fget (fd,0 ,1 ); }EXPORT_SYMBOL (fget_raw);
@@ -767,7 +772,7 @@ static unsigned long __fget_light(unsigned int fd, fmode_t mask)return 0 ;return (unsigned long )file; }else { - file =__fget (fd, mask); + file =__fget (fd, mask,1 );if (!file)return 0 ;return FDPUT_FPUT | (unsigned long )file;
fs/file_table.c +7 -2
@@ -326,9 +326,9 @@ void flush_delayed_fput(void) static DECLARE_DELAYED_WORK(delayed_fput_work, delayed_fput); -void fput (struct file *file) +void fput_many (struct file *file,unsigned int refs) { -if (atomic_long_dec_and_test (&file->f_count)) { +if (atomic_long_sub_and_test (refs, &file->f_count)) {struct task_struct *task = current;if (likely (!in_interrupt () && !(task->flags &PF_KTHREAD ))) {
@@ -347,6 +347,11 @@ void fput(struct file *file) } } +void fput (struct file *file) +{ +fput_many (file,1 ); +} +/* * synchronous analog of fput(); for kernel threads that might be needed * in some umount() (and thus can't use flush_delayed_fput() without
include/linux/file.h +2 -0
@@ -13,6 +13,7 @@struct file ;extern void fput (struct file *); +extern void fput_many (struct file *,unsigned int );struct file_operations ;struct vfsmount ;
@@ -44,6 +45,7 @@ static inline void fdput(struct fd fd) }extern struct file *fget (unsigned int fd); +extern struct file *fget_many (unsigned int fd,unsigned int refs);extern struct file *fget_raw (unsigned int fd);extern unsigned long __fdget (unsigned int fd);extern unsigned long __fdget_raw (unsigned int fd);
include/linux/fs.h +3 -1
@@ -952,7 +952,9 @@ static inline struct file *get_file(struct file *f)atomic_long_inc (&f->f_count);return f; } -#define get_file_rcu (x) atomic_long_inc_not_zero(&(x)->f_count) +#define get_file_rcu_many (x, cnt) \ + atomic_long_add_unless(&(x)->f_count, (cnt), 0) +#define get_file_rcu (x) get_file_rcu_many((x), 1)#define fput_atomic (x) atomic_long_add_unless(&(x)->f_count, -1, 1)#define file_count (x) atomic_long_read(&(x)->f_count)
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 9/15 io_uring: use fget/fput_many() for file references
In reply to Jens AxboeAdd a separate io_submit_state structure, to cache some of the things we need for IO submission.
One such example is file reference batching. io_submit_state. We get as many references as the number of sqes we are submitting, and drop unused ones if we end up switching files. The assumption here is that we're usually only dealing with one fd, and if there are multiple, hopefuly they are at least somewhat ordered. Could trivially be extended to cover multiple fds, if needed.
On the completion side we do the same thing, except this is trivially done just locally in io_iopoll_reap().
--- fs/io_uring.c | 139 ++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 118 insertions(+), 21 deletions(-)
fs/io_uring.c +118 -21
@@ -124,6 +124,19 @@ struct io_kiocb {#define IO_PLUG_THRESHOLD 2#define IO_IOPOLL_BATCH 8 +struct io_submit_state { +struct blk_plug plug; + +/* +* File reference cache +*/ +struct file *file; +unsigned int fd; +unsigned int has_refs; +unsigned int used_refs; +unsigned int ios_left; +}; +static struct kmem_cache *req_cachep;static const struct file_operations io_uring_fops;
@@ -276,9 +289,11 @@ static void io_iopoll_complete(struct io_ring_ctx *ctx, unsigned int *nr_events, struct list_head *done) {void *reqs[IO_IOPOLL_BATCH ]; +int file_count, to_free; +struct file *file =NULL ;struct io_kiocb *req; -int to_free =0 ; + file_count = to_free =0 ;while (!list_empty (done)) { req = list_first_entry(done, struct io_kiocb, list);list_del (&req->list);
@@ -288,12 +303,28 @@ static void io_iopoll_complete(struct io_ring_ctx *ctx, unsigned int *nr_events, reqs[to_free++] = req; (*nr_events)++; -fput (req->rw.ki_filp); +/* +* Batched puts of the same file, to avoid dirtying the +* file usage count multiple times, if avoidable. +*/ +if (!file) { + file = req->rw.ki_filp; + file_count =1 ; + }else if (file == req->rw.ki_filp) { + file_count++; + }else { +fput_many (file, file_count); + file = req->rw.ki_filp; + file_count =1 ; + } +if (to_free ==ARRAY_SIZE (reqs))io_free_req_many (ctx, reqs, &to_free); }io_commit_cqring (ctx); +if (file) +fput_many (file, file_count);if (to_free)io_free_req_many (ctx, reqs, &to_free); }
@@ -482,14 +513,56 @@ static void io_iopoll_req_issued(struct io_kiocb *req)list_add_tail (&req->list, &ctx->poll_list); } +static void io_file_put (struct io_submit_state *state,struct file *file) +{ +if (!state) { +fput (file); + }else if (state->file) { +int diff = state->has_refs - state->used_refs; + +if (diff) +fput_many (state->file, diff); + state->file =NULL ; + } +} + +/* +* Get as many references to a file as we have IOs left in this submission, +* assuming most submissions are for one file, or at least that each file +* has more than one submission. +*/ +static struct file *io_file_get (struct io_submit_state *state,int fd) +{ +if (!state) +return fget (fd); + +if (state->file) { +if (state->fd == fd) { + state->used_refs++; + state->ios_left--; +return state->file; + } +io_file_put (state,NULL ); + } + state->file =fget_many (fd, state->ios_left); +if (!state->file) +return NULL ; + + state->fd = fd; + state->has_refs = state->ios_left; + state->used_refs =1 ; + state->ios_left--; +return state->file; +} +static int io_prep_rw (struct io_kiocb *req,const struct io_uring_sqe *sqe, -bool force_nonblock) +bool force_nonblock,struct io_submit_state *state) {struct io_ring_ctx *ctx = req->ctx;struct kiocb *kiocb = &req->rw;int ret; - kiocb->ki_filp =fget (sqe->fd); + kiocb->ki_filp =io_file_get (state, sqe->fd);if (unlikely (!kiocb->ki_filp))return -EBADF ; kiocb->ki_pos = sqe->off;
@@ -528,7 +601,7 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe, }return 0 ; out_fput: -fput (kiocb->ki_filp); +io_file_put (state, kiocb->ki_filp);return ret; }
@@ -568,7 +641,7 @@ static int io_import_iovec(struct io_ring_ctx *ctx, int rw, }static ssize_t io_read (struct io_kiocb *req,const struct io_uring_sqe *sqe, -bool force_nonblock) +bool force_nonblock,struct io_submit_state *state) {struct iovec inline_vecs[UIO_FASTIOV ], *iovec = inline_vecs;struct kiocb *kiocb = &req->rw;
@@ -576,7 +649,7 @@ static ssize_t io_read(struct io_kiocb *req, const struct io_uring_sqe *sqe,struct file *file;ssize_t ret; - ret =io_prep_rw (req, sqe, force_nonblock); + ret =io_prep_rw (req, sqe, force_nonblock, state);if (ret)return ret; file = kiocb->ki_filp;
@@ -611,7 +684,7 @@ static ssize_t io_read(struct io_kiocb *req, const struct io_uring_sqe *sqe, }static ssize_t io_write (struct io_kiocb *req,const struct io_uring_sqe *sqe, -bool force_nonblock) +bool force_nonblock,struct io_submit_state *state) {struct iovec inline_vecs[UIO_FASTIOV ], *iovec = inline_vecs;struct kiocb *kiocb = &req->rw;
@@ -619,7 +692,7 @@ static ssize_t io_write(struct io_kiocb *req, const struct io_uring_sqe *sqe,struct file *file;ssize_t ret; - ret =io_prep_rw (req, sqe, force_nonblock); + ret =io_prep_rw (req, sqe, force_nonblock, state);if (ret)return ret; file = kiocb->ki_filp;
@@ -712,7 +785,8 @@ static int io_fsync(struct io_kiocb *req, const struct io_uring_sqe *sqe, }static int __io_submit_sqe (struct io_ring_ctx *ctx,struct io_kiocb *req, -struct sqe_submit *s,bool force_nonblock) +struct sqe_submit *s,bool force_nonblock, +struct io_submit_state *state) {const struct io_uring_sqe *sqe = s->sqe;ssize_t ret;
@@ -731,10 +805,10 @@ static int __io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req, ret =io_nop (req, sqe);break ;case IORING_OP_READV : - ret =io_read (req, sqe, force_nonblock); + ret =io_read (req, sqe, force_nonblock, state);break ;case IORING_OP_WRITEV : - ret =io_write (req, sqe, force_nonblock); + ret =io_write (req, sqe, force_nonblock, state);break ;case IORING_OP_FSYNC : ret =io_fsync (req, sqe, force_nonblock);
@@ -778,7 +852,7 @@ static void io_sq_wq_submit_work(struct work_struct *work)use_mm (ctx->sqo_mm);set_fs (USER_DS ); - ret =__io_submit_sqe (ctx, req, &req->work.submit, false); + ret =__io_submit_sqe (ctx, req, &req->work.submit, false,NULL );set_fs (old_fs);unuse_mm (ctx->sqo_mm);
@@ -791,7 +865,8 @@ static void io_sq_wq_submit_work(struct work_struct *work) current->files = old_files; } -static int io_submit_sqe (struct io_ring_ctx *ctx,struct sqe_submit *s) +static int io_submit_sqe (struct io_ring_ctx *ctx,struct sqe_submit *s, +struct io_submit_state *state) {struct io_kiocb *req;ssize_t ret;
@@ -800,7 +875,7 @@ static int io_submit_sqe(struct io_ring_ctx *ctx, struct sqe_submit *s)if (unlikely (!req))return -EAGAIN ; - ret =__io_submit_sqe (ctx, req, s, true); + ret =__io_submit_sqe (ctx, req, s, true, state);if (ret == -EAGAIN ) {memcpy (&req->work.submit, s,sizeof (*s));INIT_WORK (&req->work.work, io_sq_wq_submit_work);
@@ -813,6 +888,26 @@ static int io_submit_sqe(struct io_ring_ctx *ctx, struct sqe_submit *s)return ret; } +/* +* Batched submission is done, ensure local IO is flushed out. +*/ +static void io_submit_state_end (struct io_submit_state *state) +{ +blk_finish_plug (&state->plug); +io_file_put (state,NULL ); +} + +/* +* Start submission side cache. +*/ +static void io_submit_state_start (struct io_submit_state *state, +struct io_ring_ctx *ctx,unsigned max_ios) +{ +blk_start_plug (&state->plug); + state->file =NULL ; + state->ios_left = max_ios; +} +static void io_commit_sqring (struct io_ring_ctx *ctx) {struct io_sq_ring *ring = ctx->sq_ring;
@@ -859,11 +954,13 @@ static bool io_get_sqring(struct io_ring_ctx *ctx, struct sqe_submit *s)static int io_ring_submit (struct io_ring_ctx *ctx,unsigned int to_submit) { +struct io_submit_state state, *statep =NULL ;int i, ret =0 , submit =0 ; -struct blk_plug plug; -if (to_submit >IO_PLUG_THRESHOLD ) -blk_start_plug (&plug); +if (to_submit >IO_PLUG_THRESHOLD ) { +io_submit_state_start (&state, ctx, to_submit); + statep = &state; + }for (i =0 ; i < to_submit; i++) {struct sqe_submit s;
@@ -871,7 +968,7 @@ static int io_ring_submit(struct io_ring_ctx *ctx, unsigned int to_submit)if (!io_get_sqring (ctx, &s))break ; - ret =io_submit_sqe (ctx, &s); + ret =io_submit_sqe (ctx, &s, statep);if (ret) {io_drop_sqring (ctx);break ;
@@ -881,8 +978,8 @@ static int io_ring_submit(struct io_ring_ctx *ctx, unsigned int to_submit) }io_commit_sqring (ctx); -if (to_submit >IO_PLUG_THRESHOLD ) -blk_finish_plug (&plug); +if (statep) +io_submit_state_end (statep);return submit ? submit : ret; }
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 10/15 io_uring: batch io_kiocb allocation
In reply to Jens AxboeSimilarly to how we use the state->ios_left to know how many references to get to a file, we can use it to allocate the io_kiocb's we need in bulk.
--- fs/io_uring.c | 66 ++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 50 insertions(+), 16 deletions(-)
fs/io_uring.c +50 -16
@@ -127,6 +127,13 @@ struct io_kiocb {struct io_submit_state {struct blk_plug plug; +/* +* io_kiocb alloc cache +*/ +void *reqs[IO_IOPOLL_BATCH ]; +unsigned int free_reqs; +unsigned int cur_req; +/* * File reference cache */
@@ -242,29 +249,52 @@ static void io_fill_cq_error(struct io_ring_ctx *ctx, struct sqe_submit *s,wake_up (&ctx->wait); } -static struct io_kiocb *io_get_req (struct io_ring_ctx *ctx) +static void io_ring_drop_ctx_refs (struct io_ring_ctx *ctx,unsigned refs) { +percpu_ref_put_many (&ctx->refs, refs); + +if (waitqueue_active (&ctx->wait)) +wake_up (&ctx->wait); +} + +static struct io_kiocb *io_get_req (struct io_ring_ctx *ctx, +struct io_submit_state *state) +{ +gfp_t gfp =GFP_ATOMIC | __GFP_NOWARN;struct io_kiocb *req;if (!percpu_ref_tryget (&ctx->refs))return NULL ; - req =kmem_cache_alloc (req_cachep,GFP_ATOMIC | __GFP_NOWARN); -if (!req) -return NULL ; - - req->ctx = ctx; -INIT_LIST_HEAD (&req->list); - req->flags =0 ; -return req; -} +if (!state) + req =kmem_cache_alloc (req_cachep, gfp); +else if (!state->free_reqs) { +size_t sz; +int ret; + + sz =min_t (size_t, state->ios_left,ARRAY_SIZE (state->reqs)); + ret =kmem_cache_alloc_bulk (req_cachep, gfp, sz, + state->reqs); +if (ret <=0 ) + goto out; + state->free_reqs = ret -1 ; + state->cur_req =1 ; + req = state->reqs[0 ]; + }else { + req = state->reqs[state->cur_req]; + state->free_reqs--; + state->cur_req++; + } -static void io_ring_drop_ctx_refs (struct io_ring_ctx *ctx,unsigned refs) -{ -percpu_ref_put_many (&ctx->refs, refs); +if (req) { + req->ctx = ctx; + req->flags =0 ; +return req; + } -if (waitqueue_active (&ctx->wait)) -wake_up (&ctx->wait); +out: +io_ring_drop_ctx_refs (ctx,1 ); +return NULL ; }static void io_free_req_many (struct io_ring_ctx *ctx,void **reqs,int *nr)
@@ -871,7 +901,7 @@ static int io_submit_sqe(struct io_ring_ctx *ctx, struct sqe_submit *s,struct io_kiocb *req;ssize_t ret; - req =io_get_req (ctx); + req =io_get_req (ctx, state);if (unlikely (!req))return -EAGAIN ;
@@ -895,6 +925,9 @@ static void io_submit_state_end(struct io_submit_state *state) {blk_finish_plug (&state->plug);io_file_put (state,NULL ); +if (state->free_reqs) +kmem_cache_free_bulk (req_cachep, state->free_reqs, + &state->reqs[state->cur_req]); }/*
@@ -904,6 +937,7 @@ static void io_submit_state_start(struct io_submit_state *state, struct io_ring_ctx *ctx, unsigned max_ios) {blk_start_plug (&state->plug); + state->free_reqs =0 ; state->file =NULL ; state->ios_left = max_ios; }
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 11/15 block: implement bio helper to add iter bvec pages to bio
In reply to Jens AxboeFor an ITER_BVEC, we can just iterate the iov and add the pages to the bio directly. This requires that the caller doesn't releases the pages on IO completion, we add a BIO_HOLD_PAGES flag for that.
The current two callers of bio_iov_iter_get_pages() are updated to check if they need to release pages on completion. This makes them work with bvecs that contain kernel mapped pages already.
--- block/bio.c | 59 ++++++++++++++++++++++++++++++++------- fs/block_dev.c | 5 ++-- fs/iomap.c | 5 ++-- include/linux/blk_types.h | 1 + 4 files changed, 56 insertions(+), 14 deletions(-)
block/bio.c +49 -10
@@ -828,6 +828,23 @@ int bio_add_page(struct bio *bio, struct page *page, }EXPORT_SYMBOL (bio_add_page); +static int __bio_iov_bvec_add_pages (struct bio *bio,struct iov_iter *iter) +{ +const struct bio_vec *bv = iter->bvec; +unsigned int len; +size_t size; + + len =min_t (size_t, bv->bv_len, iter->count); + size =bio_add_page (bio, bv->bv_page, len, + bv->bv_offset + iter->iov_offset); +if (size == len) { +iov_iter_advance (iter, size); +return 0 ; + } + +return -EINVAL ; +} +#define PAGE_PTRS_PER_BVEC (sizeof(struct bio_vec) / sizeof(struct page *))/**
@@ -876,23 +893,43 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) }/** -* bio_iov_iter_get_pages - pin user or kernel pages and add them to a bio +* bio_iov_iter_get_pages - add user or kernel pages to a bio * @bio: bio to add pages to -* @iter: iov iterator describing the region to be mapped +* @iter: iov iterator describing the region to be added +* +* This takes either an iterator pointing to user memory, or one pointing to +* kernel pages (BVEC iterator). If we're adding user pages, we pin them and +* map them into the kernel. On IO completion, the caller should put those +* pages. If we're adding kernel pages, we just have to add the pages to the +* bio directly. We don't grab an extra reference to those pages (the user +* should already have that), and we don't put the page on IO completion. +* The caller needs to check if the bio is flagged BIO_HOLD_PAGES on IO +* completion. If it isn't, then pages should be released. * -* Pins pages from *iter and appends them to @bio's bvec array. The -* pages will have to be released using put_page() when done. * The function tries, but does not guarantee, to pin as many pages as -* fit into the bio, or are requested in *iter, whatever is smaller. -* If MM encounters an error pinning the requested pages, it stops. -* Error is returned only if 0 pages could be pinned. +* fit into the bio, or are requested in *iter, whatever is smaller. If +* MM encounters an error pinning the requested pages, it stops. Error +* is returned only if 0 pages could be pinned. */ int bio_iov_iter_get_pages (struct bio *bio,struct iov_iter *iter) { +const bool is_bvec =iov_iter_is_bvec (iter);unsigned short orig_vcnt = bio->bi_vcnt; +/* +* If this is a BVEC iter, then the pages are kernel pages. Don't +* release them on IO completion. +*/ +if (is_bvec) +bio_set_flag (bio,BIO_HOLD_PAGES ); + do { - int ret = __bio_iov_iter_get_pages(bio, iter); + int ret; + + if (is_bvec) + ret = __bio_iov_bvec_add_pages(bio, iter); + else + ret = __bio_iov_iter_get_pages(bio, iter); if (unlikely(ret)) return bio->bi_vcnt > orig_vcnt ? 0 : ret;
@@ -1634,7 +1671,8 @@ static void bio_dirty_fn(struct work_struct *work) next = bio->bi_private;bio_set_pages_dirty (bio); -bio_release_pages (bio); +if (!bio_flagged (bio,BIO_HOLD_PAGES )) +bio_release_pages (bio);bio_put (bio); } }
@@ -1650,7 +1688,8 @@ void bio_check_pages_dirty(struct bio *bio) goto defer; } -bio_release_pages (bio); +if (!bio_flagged (bio,BIO_HOLD_PAGES )) +bio_release_pages (bio);bio_put (bio);return ; defer:
fs/block_dev.c +3 -2
@@ -324,8 +324,9 @@ static void blkdev_bio_end_io(struct bio *bio)struct bio_vec *bvec;int i; -bio_for_each_segment_all (bvec, bio, i) -put_page (bvec->bv_page); +if (!bio_flagged (bio,BIO_HOLD_PAGES )) +bio_for_each_segment_all (bvec, bio, i) +put_page (bvec->bv_page);bio_put (bio); } }
fs/iomap.c +3 -2
@@ -1582,8 +1582,9 @@ static void iomap_dio_bio_end_io(struct bio *bio)struct bio_vec *bvec;int i; -bio_for_each_segment_all (bvec, bio, i) -put_page (bvec->bv_page); +if (!bio_flagged (bio,BIO_HOLD_PAGES )) +bio_for_each_segment_all (bvec, bio, i) +put_page (bvec->bv_page);bio_put (bio); } }
include/linux/blk_types.h +1 -0
@@ -215,6 +215,7 @@ struct bio {/* * bio flags */ +#define BIO_HOLD_PAGES 0/* don't put O_DIRECT pages */ #define BIO_SEG_VALID 1/* bi_phys_segments valid */ #define BIO_CLONED 2/* doesn't own data */ #define BIO_BOUNCED 3/* bio is a bounce bio */
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 12/15 io_uring: add support for pre-mapped user IO buffers
In reply to Jens AxboeIf we have fixed user buffers, we can map them into the kernel when we setup the io_context. That avoids the need to do get_user_pages() for each and every IO.
To utilize this feature, the application must call io_uring_register() after having setup an io_uring context, passing in IORING_REGISTER_BUFFERS as the opcode. The argument must be a pointer to an iovec array, and the nr_args should contain how many iovecs the application wishes to map.
If successful, these buffers are now mapped into the kernel, eligible for IO. To use these fixed buffers, the application must use the IORING_OP_READ_FIXED and IORING_OP_WRITE_FIXED opcodes, and then set sqe->index to the desired buffer index. sqe->addr..sqe->addr+seq->len must point to somewhere inside the indexed buffer.
The application may register buffers throughout the lifetime of the io_uring context. It can call io_uring_register() with IORING_UNREGISTER_BUFFERS as the opcode to unregister the current set of buffers, and then register a new set. The application need not unregister buffers explicitly before shutting down the io_uring context.
It's perfectly valid to setup a larger buffer, and then sometimes only use parts of it for an IO. As long as the range is within the originally mapped region, it will work just fine.
RLIMIT_MEMLOCK is used to check how much memory we can pin. A somewhat arbitrary 1G per buffer size is also imposed.
--- arch/x86/entry/syscalls/syscall_32.tbl | 1 + arch/x86/entry/syscalls/syscall_64.tbl | 1 + fs/io_uring.c | 339 ++++++++++++++++++++++++- include/linux/sched/user.h | 2 +- include/linux/syscalls.h | 2 + include/uapi/linux/io_uring.h | 13 +- kernel/sys_ni.c | 1 + 7 files changed, 347 insertions(+), 12 deletions(-)
arch/x86/entry/syscalls/syscall_32.tbl +1 -0
@@ -400,3 +400,4 @@ 386 i386 rseq sys_rseq __ia32_sys_rseq 387 i386 io_uring_setup sys_io_uring_setup __ia32_compat_sys_io_uring_setup 388 i386 io_uring_enter sys_io_uring_enter __ia32_sys_io_uring_enter +389 i386 io_uring_register sys_io_uring_register __ia32_sys_io_uring_register
arch/x86/entry/syscalls/syscall_64.tbl +1 -0
@@ -345,6 +345,7 @@ 334 common rseq __x64_sys_rseq 335 common io_uring_setup __x64_sys_io_uring_setup 336 common io_uring_enter __x64_sys_io_uring_enter +337 common io_uring_register __x64_sys_io_uring_register # # x32-specific system call numbers start at 512 to avoid cache impact
fs/io_uring.c +329 -10
416 lines not on this page · raw message
include/linux/sched/user.h +1 -1
@@ -40,7 +40,7 @@ struct user_struct {kuid_t uid;#if defined(CONFIG_PERF_EVENTS ) || defined(CONFIG_BPF_SYSCALL ) || \ - defined(CONFIG_NET ) + defined(CONFIG_NET ) || defined(CONFIG_IO_URING )atomic_long_t locked_vm;#endif
include/linux/syscalls.h +2 -0
@@ -314,6 +314,8 @@ asmlinkage long sys_io_uring_setup(u32 entries, struct io_uring_params __user *p);asmlinkage long sys_io_uring_enter (unsigned int fd,u32 to_submit,u32 min_complete,u32 flags); +asmlinkage long sys_io_uring_register (unsigned int fd,unsigned int op, +void __user *arg,unsigned int nr_args);/* fs/xattr.c */ asmlinkage long sys_setxattr(const char __user *path, const char __user *name,
include/uapi/linux/io_uring.h +12 -1
@@ -27,7 +27,10 @@ struct io_uring_sqe {__u32 fsync_flags; };__u64 user_data;/* data to be passed back at completion time */ -__u64 __pad2[3 ]; +union { +__u16 buf_index;/* index into fixed buffers, if used */ +__u64 __pad2[3 ]; + }; };/*
@@ -39,6 +42,8 @@ struct io_uring_sqe {#define IORING_OP_READV 1#define IORING_OP_WRITEV 2#define IORING_OP_FSYNC 3 +#define IORING_OP_READ_FIXED 4 +#define IORING_OP_WRITE_FIXED 5/* * sqe->fsync_flags
@@ -102,4 +107,10 @@ struct io_uring_params { struct io_cqring_offsets cq_off; }; +/* +* io_uring_register(2) opcodes and arguments +*/ +#define IORING_REGISTER_BUFFERS 0 +#define IORING_UNREGISTER_BUFFERS 1 +#endif
kernel/sys_ni.c +1 -0
@@ -49,6 +49,7 @@ COND_SYSCALL_COMPAT(io_pgetevents);COND_SYSCALL (io_uring_setup);COND_SYSCALL_COMPAT (io_uring_setup);COND_SYSCALL (io_uring_enter); +COND_SYSCALL (io_uring_register);/* fs/xattr.c */
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 12/15] io_uring: add support for pre-mapped user IO buffers
Quoted 3 lines · Jens Axboe
If we have fixed user buffers, we can map them into the kernel when we setup the
io_context. That avoids the need to doget_user_pages()for each and every IO.
.....
Quoted 11 lines
fs/io_uring.c +11
-0+return -ENOMEM ; + }while (atomic_long_cmpxchg (&ctx->user->locked_vm, cur_pages, + new_pages) != cur_pages); + +return 0 ; +} + +static int io_sqe_buffer_unregister (struct io_ring_ctx *ctx) +{ +int i, j; +
fs/io_uring.c static void io_sq_offload_stop(struct io_ring_ctx *ctx) · line 1240
Quoted 10 lines
fs/io_uring.c +10
-0+if (!ctx->user_bufs) +return -EINVAL ; + +for (i =0 ; i < ctx->sq_entries; i++) { +struct io_mapped_ubuf *imu = &ctx->user_bufs[i]; + +for (j =0 ; j < imu->nr_bvecs; j++) { +set_page_dirty_lock (imu->bvec[j].bv_page); +put_page (imu->bvec[j].bv_page); + }
Hmmm, so we call set_page_dirty() when the gup reference is dropped...
.....
Quoted 3 lines
fs/io_uring.c +3
-0+static int io_sqe_buffer_register (struct io_ring_ctx *ctx,void __user *arg, +unsigned nr_args) +{
.....
fs/io_uring.c static void io_sq_offload_stop(struct io_ring_ctx *ctx) · line 1348
Quoted 4 lines
fs/io_uring.c +4
-0+down_write (¤t->mm->mmap_sem); + pret =get_user_pages_longterm (ubuf, nr_pages,FOLL_WRITE , + pages,NULL ); +up_write (¤t->mm->mmap_sem);
Thought so. This has the same problem as RDMA w.r.t. using file-backed mappings for the user buffer. It is not synchronised against truncate, hole punches, async page writeback cleaning the page, etc, and so can lead to data corruption and/or kernel panics.
It also can't be used with DAX because the above problems are actually a user-after-free of storage space, not just a dangling page reference that can be cleaned up after the gup pin is dropped.
Perhaps, at least until we solve the GUP problems w.r.t. file backed pages and/or add and require file layout leases for these reference, we should error out if the user buffer pages are file-backed mappings?
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Jens Axboe
maintainerRe: [PATCH 12/15] io_uring: add support for pre-mapped user IO buffers
Quoted 47 lines · Dave Chinner
Quoted 3 lines · Jens Axboe
If we have fixed user buffers, we can map them into the kernel when we setup the
io_context. That avoids the need to doget_user_pages()for each and every IO......
Quoted 11 lines
fs/io_uring.c +11
-0+return -ENOMEM ; + }while (atomic_long_cmpxchg (&ctx->user->locked_vm, cur_pages, + new_pages) != cur_pages); + +return 0 ; +} + +static int io_sqe_buffer_unregister (struct io_ring_ctx *ctx) +{ +int i, j; +fs/io_uring.c static void io_sq_offload_stop(struct io_ring_ctx *ctx) · line 1240
Quoted 10 lines
fs/io_uring.c +10
-0+if (!ctx->user_bufs) +return -EINVAL ; + +for (i =0 ; i < ctx->sq_entries; i++) { +struct io_mapped_ubuf *imu = &ctx->user_bufs[i]; + +for (j =0 ; j < imu->nr_bvecs; j++) { +set_page_dirty_lock (imu->bvec[j].bv_page); +put_page (imu->bvec[j].bv_page); + }Hmmm, so we call
set_page_dirty()when the gup reference is dropped........
Quoted 3 lines
fs/io_uring.c +3
-0+static int io_sqe_buffer_register (struct io_ring_ctx *ctx,void __user *arg, +unsigned nr_args) +{.....
fs/io_uring.c static void io_sq_offload_stop(struct io_ring_ctx *ctx) · line 1348
Quoted 4 lines
fs/io_uring.c +4
-0+down_write (¤t->mm->mmap_sem); + pret =get_user_pages_longterm (ubuf, nr_pages,FOLL_WRITE , + pages,NULL ); +up_write (¤t->mm->mmap_sem);Thought so. This has the same problem as RDMA w.r.t. using file-backed mappings for the user buffer. It is not synchronised against truncate, hole punches, async page writeback cleaning the page, etc, and so can lead to data corruption and/or kernel panics.
Quoted 8 lines
It also can't be used with DAX because the above problems are actually a user-after-free of storage space, not just a dangling page reference that can be cleaned up after the gup pin is dropped.
Perhaps, at least until we solve the GUP problems w.r.t. file backed pages and/or add and require file layout leases for these reference, we should error out if the user buffer pages are file-backed mappings?
Thanks for taking a look at this.
I'd be fine with that restriction, especially since it can get relaxed down the line. Do we have an appropriate API for this? And why isn't get_user_pages_longterm() that exact API already? Would seem that most (all?) callers of this API is currently broken then.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 12/15] io_uring: add support for pre-mapped user IO buffers
Quoted 57 lines · Jens Axboe
Quoted 47 lines · Dave Chinner
Quoted 3 lines · Jens Axboe
If we have fixed user buffers, we can map them into the kernel when we setup the
io_context. That avoids the need to doget_user_pages()for each and every IO......
Quoted 11 lines
fs/io_uring.c +11
-0+return -ENOMEM ; + }while (atomic_long_cmpxchg (&ctx->user->locked_vm, cur_pages, + new_pages) != cur_pages); + +return 0 ; +} + +static int io_sqe_buffer_unregister (struct io_ring_ctx *ctx) +{ +int i, j; +fs/io_uring.c static void io_sq_offload_stop(struct io_ring_ctx *ctx) · line 1240
Quoted 10 lines
fs/io_uring.c +10
-0+if (!ctx->user_bufs) +return -EINVAL ; + +for (i =0 ; i < ctx->sq_entries; i++) { +struct io_mapped_ubuf *imu = &ctx->user_bufs[i]; + +for (j =0 ; j < imu->nr_bvecs; j++) { +set_page_dirty_lock (imu->bvec[j].bv_page); +put_page (imu->bvec[j].bv_page); + }Hmmm, so we call
set_page_dirty()when the gup reference is dropped........
Quoted 3 lines
fs/io_uring.c +3
-0+static int io_sqe_buffer_register (struct io_ring_ctx *ctx,void __user *arg, +unsigned nr_args) +{.....
fs/io_uring.c static void io_sq_offload_stop(struct io_ring_ctx *ctx) · line 1348
Quoted 4 lines
fs/io_uring.c +4
-0+down_write (¤t->mm->mmap_sem); + pret =get_user_pages_longterm (ubuf, nr_pages,FOLL_WRITE , + pages,NULL ); +up_write (¤t->mm->mmap_sem);Thought so. This has the same problem as RDMA w.r.t. using file-backed mappings for the user buffer. It is not synchronised against truncate, hole punches, async page writeback cleaning the page, etc, and so can lead to data corruption and/or kernel panics.
Quoted 8 lines
It also can't be used with DAX because the above problems are actually a user-after-free of storage space, not just a dangling page reference that can be cleaned up after the gup pin is dropped.
Perhaps, at least until we solve the GUP problems w.r.t. file backed pages and/or add and require file layout leases for these reference, we should error out if the user buffer pages are file-backed mappings?
Quoted 5 lines
Thanks for taking a look at this.
I'd be fine with that restriction, especially since it can get relaxed down the line. Do we have an appropriate API for this? And why isn't
get_user_pages_longterm()that exact API already?
get_user_pages_longterm() is the right thing to use to ensure DAX doesn't trip over this - it's effectively just get_user_pages() with a "if (vma_is_fsdax(vma))" check in it to abort and return -EOPNOTSUPP. IOWs, this is safe on DAX but it's not safe on anything else. :/
Unfortunately, disallowing userspace GUP pins on non-DAX file backed pages will break existing "mostly just work" userspace apps all over the place. And so right now there are discussions ongoing about how to map gup references avoid the writeback races and be able to be seen/tracked by other kernel infrastructure (see the long, long thread "[PATCH 0/2] put_user_page*(): start converting the call sites" on -fsdevel). Progress is slow, but I think we're starting to close on a workable solution.
FWIW, this doesn't solve the "long term user pin will block filesystem operations until unpin" problem, that's what moving to using revocable file layout leases is intended to solve. There have been patches posted some time ago to add this user API for this, but we've got to solve the other problems first....
Quoted 2 lines
Would seem that most
(all?) callers of this API is currently broken then.
Yup, there's a long, long history of machines using userspace RDMA panicing because filesystems have detected or tripped over invalid page cache state during writeback attempts. This is not a new problem....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Jens Axboe
maintainerRe: [PATCH 12/15] io_uring: add support for pre-mapped user IO buffers
Quoted 85 lines · Dave Chinner
Quoted 57 lines · Jens Axboe
Quoted 47 lines · Dave Chinner
Quoted 3 lines · Jens Axboe
If we have fixed user buffers, we can map them into the kernel when we setup the
io_context. That avoids the need to doget_user_pages()for each and every IO......
Quoted 11 lines
fs/io_uring.c +11
-0+return -ENOMEM ; + }while (atomic_long_cmpxchg (&ctx->user->locked_vm, cur_pages, + new_pages) != cur_pages); + +return 0 ; +} + +static int io_sqe_buffer_unregister (struct io_ring_ctx *ctx) +{ +int i, j; +fs/io_uring.c static void io_sq_offload_stop(struct io_ring_ctx *ctx) · line 1240
Quoted 10 lines
fs/io_uring.c +10
-0+if (!ctx->user_bufs) +return -EINVAL ; + +for (i =0 ; i < ctx->sq_entries; i++) { +struct io_mapped_ubuf *imu = &ctx->user_bufs[i]; + +for (j =0 ; j < imu->nr_bvecs; j++) { +set_page_dirty_lock (imu->bvec[j].bv_page); +put_page (imu->bvec[j].bv_page); + }Hmmm, so we call
set_page_dirty()when the gup reference is dropped........
Quoted 3 lines
fs/io_uring.c +3
-0+static int io_sqe_buffer_register (struct io_ring_ctx *ctx,void __user *arg, +unsigned nr_args) +{.....
fs/io_uring.c static void io_sq_offload_stop(struct io_ring_ctx *ctx) · line 1348
Quoted 4 lines
fs/io_uring.c +4
-0+down_write (¤t->mm->mmap_sem); + pret =get_user_pages_longterm (ubuf, nr_pages,FOLL_WRITE , + pages,NULL ); +up_write (¤t->mm->mmap_sem);Thought so. This has the same problem as RDMA w.r.t. using file-backed mappings for the user buffer. It is not synchronised against truncate, hole punches, async page writeback cleaning the page, etc, and so can lead to data corruption and/or kernel panics.
Quoted 8 lines
It also can't be used with DAX because the above problems are actually a user-after-free of storage space, not just a dangling page reference that can be cleaned up after the gup pin is dropped.
Perhaps, at least until we solve the GUP problems w.r.t. file backed pages and/or add and require file layout leases for these reference, we should error out if the user buffer pages are file-backed mappings?
Quoted 5 lines
Thanks for taking a look at this.
I'd be fine with that restriction, especially since it can get relaxed down the line. Do we have an appropriate API for this? And why isn't
get_user_pages_longterm()that exact API already?
get_user_pages_longterm()is the right thing to use to ensure DAX doesn't trip over this - it's effectively justget_user_pages()with a "if (vma_is_fsdax(vma))" check in it to abort and return-EOPNOTSUPP. IOWs, this is safe on DAX but it's not safe on anything else. :/Unfortunately, disallowing userspace GUP pins on non-DAX file backed pages will break existing "mostly just work" userspace apps all over the place. And so right now there are discussions ongoing about how to map gup references avoid the writeback races and be able to be seen/tracked by other kernel infrastructure (see the long, long thread "[PATCH 0/2] put_user_page*(): start converting the call sites" on -fsdevel). Progress is slow, but I think we're starting to close on a workable solution.
FWIW, this doesn't solve the "long term user pin will block filesystem operations until unpin" problem, that's what moving to using revocable file layout leases is intended to solve. There have been patches posted some time ago to add this user API for this, but we've got to solve the other problems first....
Quoted 7 lines
Quoted 2 lines
Would seem that most
(all?) callers of this API is currently broken then.Yup, there's a long, long history of machines using userspace RDMA panicing because filesystems have detected or tripped over invalid page cache state during writeback attempts. This is not a new problem....
Thanks for your detailed answer, Dave! I didn't see it before I sent out the previous email. FWIW, I've updated the patch:
http://git.kernel.dk/cgit/linux-block/commit/?h=io_uring&id=0c8f2299f8069af6b2fa8f99a10d81646d1237a7
Checks for file backed memory, fails the registration with EOPNOTSUPP if the check fails.
That should handle the issue on the io_uring side at least, and it's a restriction that can always be relaxed/lifted, when appropriate solutions to file backed buffers exists.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Re: [PATCH 12/15] io_uring: add support for pre-mapped user IO buffers
Quoted 36 lines · Jens Axboe
Quoted 27 lines · Dave Chinner
Quoted 4 lines · Jens Axboe
On 1/16/19 1:53 PM, Dave Chinner wrote:
I'd be fine with that restriction, especially since it can get relaxed down the line. Do we have an appropriate API for this? And why isn'tget_user_pages_longterm()that exact API already?
get_user_pages_longterm()is the right thing to use to ensure DAX doesn't trip over this - it's effectively justget_user_pages()with a "if (vma_is_fsdax(vma))" check in it to abort and return-EOPNOTSUPP. IOWs, this is safe on DAX but it's not safe on anything else. :/Unfortunately, disallowing userspace GUP pins on non-DAX file backed pages will break existing "mostly just work" userspace apps all over the place. And so right now there are discussions ongoing about how to map gup references avoid the writeback races and be able to be seen/tracked by other kernel infrastructure (see the long, long thread "[PATCH 0/2] put_user_page*(): start converting the call sites" on -fsdevel). Progress is slow, but I think we're starting to close on a workable solution.
FWIW, this doesn't solve the "long term user pin will block filesystem operations until unpin" problem, that's what moving to using revocable file layout leases is intended to solve. There have been patches posted some time ago to add this user API for this, but we've got to solve the other problems first....
Quoted 7 lines
Quoted 2 lines
Would seem that most
(all?) callers of this API is currently broken then.Yup, there's a long, long history of machines using userspace RDMA panicing because filesystems have detected or tripped over invalid page cache state during writeback attempts. This is not a new problem....
Quoted 7 lines
Thanks for your detailed answer, Dave! I didn't see it before I sent out the previous email. FWIW, I've updated the patch:
http://git.kernel.dk/cgit/linux-block/commit/?h=io_uring&id=0c8f2299f8069af6b2fa8f99a10d81646d1237a7
Checks for file backed memory, fails the registration with EOPNOTSUPP if the check fails.
Doesn't it need to call put_pages() on all the pages picked up by get_user_pages_longterm() when it returns -EOPNOTSUPP? They haven't been mapped into the imu->bvec array yet, so AFAICT there's nothing to release the page references on teardown here.
Also, not a vma expert here, but the vma array contents may only be valid while the mmap_sem is held - I think vmas can come and go after it has been dropped and so accessing vmas to check vma->vm_file after the mmap_sem has been dropped may be open to read-after-free races.
Quoted 3 lines
That should handle the issue on the
io_uringside at least, and it's a restriction that can always be relaxed/lifted, when appropriate solutions to file backed buffers exists.
Modulo the issue above, that works for me.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Jens Axboe
maintainerRe: [PATCH 12/15] io_uring: add support for pre-mapped user IO buffers
Quoted 42 lines · Dave Chinner
Quoted 41 lines · Jens Axboe
Quoted 27 lines · Dave Chinner
Quoted 4 lines · Jens Axboe
On 1/16/19 1:53 PM, Dave Chinner wrote:
I'd be fine with that restriction, especially since it can get relaxed down the line. Do we have an appropriate API for this? And why isn'tget_user_pages_longterm()that exact API already?
get_user_pages_longterm()is the right thing to use to ensure DAX doesn't trip over this - it's effectively justget_user_pages()with a "if (vma_is_fsdax(vma))" check in it to abort and return-EOPNOTSUPP. IOWs, this is safe on DAX but it's not safe on anything else. :/Unfortunately, disallowing userspace GUP pins on non-DAX file backed pages will break existing "mostly just work" userspace apps all over the place. And so right now there are discussions ongoing about how to map gup references avoid the writeback races and be able to be seen/tracked by other kernel infrastructure (see the long, long thread "[PATCH 0/2] put_user_page*(): start converting the call sites" on -fsdevel). Progress is slow, but I think we're starting to close on a workable solution.
FWIW, this doesn't solve the "long term user pin will block filesystem operations until unpin" problem, that's what moving to using revocable file layout leases is intended to solve. There have been patches posted some time ago to add this user API for this, but we've got to solve the other problems first....
Quoted 7 lines
Quoted 2 lines
Would seem that most
(all?) callers of this API is currently broken then.Yup, there's a long, long history of machines using userspace RDMA panicing because filesystems have detected or tripped over invalid page cache state during writeback attempts. This is not a new problem....
Thanks for your detailed answer, Dave! I didn't see it before I sent out the previous email. FWIW, I've updated the patch:
http://git.kernel.dk/cgit/linux-block/commit/?h=io_uring&id=0c8f2299f8069af6b2fa8f99a10d81646d1237a7
Quoted 7 lines
Quoted 2 lines
Checks for file backed memory, fails the registration with EOPNOTSUPP if the check fails.
Doesn't it need to call
put_pages()on all the pages picked up byget_user_pages_longterm()when it returns-EOPNOTSUPP? They haven't been mapped into theimu->bvecarray yet, so AFAICT there's nothing to release the page references on teardown here.
Oops, yes good point. The usual error handling won't work for this, need to put them.
Quoted 5 lines
Also, not a vma expert here, but the vma array contents may only be valid while the
mmap_semis held - I think vmas can come and go after it has been dropped and so accessing vmas to checkvma->vm_fileafter themmap_semhas been dropped may be open to read-after-free races.
I did fix that one right after sending out the email:
http://git.kernel.dk/cgit/linux-block/commit/?h=io_uring&id=d2b44723d5bceeb9966c858255a03596ed62929c
I'll fix the missing put_pages() on error and update it.
Quoted 5 lines
Quoted 3 lines
That should handle the issue on the
io_uringside at least, and it's a restriction that can always be relaxed/lifted, when appropriate solutions to file backed buffers exists.Modulo the issue above, that works for me.
Great!
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerRe: [PATCH 12/15] io_uring: add support for pre-mapped user IO buffers
In reply to Jens AxboeQuoted 57 lines · Jens Axboe
Quoted 47 lines · Dave Chinner
Quoted 3 lines · Jens Axboe
If we have fixed user buffers, we can map them into the kernel when we setup the
io_context. That avoids the need to doget_user_pages()for each and every IO......
Quoted 11 lines
fs/io_uring.c +11
-0+return -ENOMEM ; + }while (atomic_long_cmpxchg (&ctx->user->locked_vm, cur_pages, + new_pages) != cur_pages); + +return 0 ; +} + +static int io_sqe_buffer_unregister (struct io_ring_ctx *ctx) +{ +int i, j; +fs/io_uring.c static void io_sq_offload_stop(struct io_ring_ctx *ctx) · line 1240
Quoted 10 lines
fs/io_uring.c +10
-0+if (!ctx->user_bufs) +return -EINVAL ; + +for (i =0 ; i < ctx->sq_entries; i++) { +struct io_mapped_ubuf *imu = &ctx->user_bufs[i]; + +for (j =0 ; j < imu->nr_bvecs; j++) { +set_page_dirty_lock (imu->bvec[j].bv_page); +put_page (imu->bvec[j].bv_page); + }Hmmm, so we call
set_page_dirty()when the gup reference is dropped........
Quoted 3 lines
fs/io_uring.c +3
-0+static int io_sqe_buffer_register (struct io_ring_ctx *ctx,void __user *arg, +unsigned nr_args) +{.....
fs/io_uring.c static void io_sq_offload_stop(struct io_ring_ctx *ctx) · line 1348
Quoted 4 lines
fs/io_uring.c +4
-0+down_write (¤t->mm->mmap_sem); + pret =get_user_pages_longterm (ubuf, nr_pages,FOLL_WRITE , + pages,NULL ); +up_write (¤t->mm->mmap_sem);Thought so. This has the same problem as RDMA w.r.t. using file-backed mappings for the user buffer. It is not synchronised against truncate, hole punches, async page writeback cleaning the page, etc, and so can lead to data corruption and/or kernel panics.
Quoted 8 lines
It also can't be used with DAX because the above problems are actually a user-after-free of storage space, not just a dangling page reference that can be cleaned up after the gup pin is dropped.
Perhaps, at least until we solve the GUP problems w.r.t. file backed pages and/or add and require file layout leases for these reference, we should error out if the user buffer pages are file-backed mappings?
Quoted 6 lines
Thanks for taking a look at this.
I'd be fine with that restriction, especially since it can get relaxed down the line. Do we have an appropriate API for this? And why isn't
get_user_pages_longterm()that exact API already? Would seem that most (all?) callers of this API is currently broken then.
I guess for now I can just pass in a vmas array for get_user_pages_longeterm() and then iterate those and check for vma->vm_file. If it's set, then we fail the buffer registration.
And then drop the set_page_dirty() on release, we don't need that.
Signature · 7 lines
Jens Axboe -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 13/15 io_uring: add submission polling
In reply to Jens AxboeThis enables an application to do IO, without ever entering the kernel. By using the SQ ring to fill in new sqes and watching for completions on the CQ ring, we can submit and reap IOs without doing a single system call. The kernel side thread will poll for new submissions, and in case of HIPRI/polled IO, it'll also poll for completions.
Proof of concept. If the thread has been idle for 1 second, it will set sq_ring->flags |= IORING_SQ_NEED_WAKEUP. The application will have to call io_uring_enter() to start things back up again. If IO is kept busy, that will never be needed. Basically an application that has this feature enabled will guard it's io_uring_enter(2) call with:
read_barrier();
if (*sq_ring->flags & IORING_SQ_NEED_WAKEUP)
io_uring_enter(fd, to_submit, 0, 0);
instead of calling it unconditionally.
Improvements:
1) Maybe have smarter backoff. Busy loop for X time, then go to monitor/mwait, finally the schedule we have now after an idle second. Might not be worth the complexity.
2) Probably want the application to pass in the appropriate grace period, not hard code it at 1 second.
--- fs/io_uring.c | 216 +++++++++++++++++++++++++++++++++- include/uapi/linux/io_uring.h | 10 +- 2 files changed, 219 insertions(+), 7 deletions(-)
fs/io_uring.c +210 -6
@@ -23,6 +23,7 @@#include <linux/percpu.h> #include <linux/slab.h> #include <linux/workqueue.h> +#include <linux/kthread.h> #include <linux/blkdev.h> #include <linux/bvec.h> #include <linux/anon_inodes.h>
@@ -88,8 +89,10 @@ struct io_ring_ctx {/* IO offload */ struct workqueue_struct *sqo_wq; +struct task_struct *sqo_thread;/* if using sq thread polling */ struct mm_struct *sqo_mm;struct files_struct *sqo_files; +wait_queue_head_t sqo_wait;/* if used, fixed mapped user buffers */ unsigned nr_user_bufs;
@@ -1065,6 +1068,168 @@ static bool io_get_sqring(struct io_ring_ctx *ctx, struct sqe_submit *s)return false; } +static int io_submit_sqes (struct io_ring_ctx *ctx,struct sqe_submit *sqes, +unsigned int nr,bool mm_fault) +{ +struct io_submit_state state, *statep =NULL ; +int ret, i, submitted =0 ; + +if (nr >IO_PLUG_THRESHOLD ) { +io_submit_state_start (&state, ctx, nr); + statep = &state; + } + +for (i =0 ; i < nr; i++) { +if (unlikely (mm_fault)) + ret = -EFAULT ; +else + ret =io_submit_sqe (ctx, &sqes[i], statep); +if (!ret) { + submitted++; +continue ; + } + +io_fill_cq_error (ctx, &sqes[i], ret); + } + +if (statep) +io_submit_state_end (&state); + +return submitted; +} + +static int io_sq_thread (void *data) +{ +struct sqe_submit sqes[IO_IOPOLL_BATCH ]; +struct io_ring_ctx *ctx = data; +struct mm_struct *cur_mm =NULL ; +struct files_struct *old_files; +mm_segment_t old_fs; +DEFINE_WAIT (wait); +unsigned inflight; +unsigned long timeout; + + old_files = current->files; + current->files = ctx->sqo_files; + + old_fs =get_fs (); +set_fs (USER_DS ); + + timeout = inflight =0 ; +while (!kthread_should_stop ()) { +bool all_fixed, mm_fault = false; +int i; + +if (inflight) { +unsigned int nr_events =0 ; + +/* +* Normal IO, just pretend everything completed. +* We don't have to poll completions for that. +*/ +if (ctx->flags &IORING_SETUP_IOPOLL ) { +/* +* App should not use IORING_ENTER_GETEVENTS +* with thread polling, but if it does, then +* ensure we are mutually exclusive. +*/ +if (mutex_trylock (&ctx->uring_lock)) { +io_iopoll_check (ctx, &nr_events,0 ); +mutex_unlock (&ctx->uring_lock); + } + }else { + nr_events = inflight; + } + + inflight -= nr_events; +if (!inflight) + timeout = jiffies +HZ ; + } + +if (!io_get_sqring (ctx, &sqes[0 ])) { +/* +* We're polling, let us spin for a second without +* work before going to sleep. +*/ +if (inflight || !time_after (jiffies, timeout)) { +cpu_relax (); +continue ; + } + +/* +* Drop cur_mm before scheduling, we can't hold it for +* long periods (or over schedule()). Do this before +* adding ourselves to the waitqueue, as the unuse/drop +* may sleep. +*/ +if (cur_mm) { +unuse_mm (cur_mm); +mmput (cur_mm); + cur_mm =NULL ; + } + +prepare_to_wait (&ctx->sqo_wait, &wait, +TASK_INTERRUPTIBLE ); + +/* Tell userspace we may need a wakeup call */ + ctx->sq_ring->flags |=IORING_SQ_NEED_WAKEUP ; +smp_wmb (); + +if (!io_get_sqring (ctx, &sqes[0 ])) { +if (kthread_should_park ()) +kthread_parkme (); +if (kthread_should_stop ()) { +finish_wait (&ctx->sqo_wait, &wait); +break ; + } +if (signal_pending (current)) +flush_signals (current); +schedule (); +finish_wait (&ctx->sqo_wait, &wait); + + ctx->sq_ring->flags &= ~IORING_SQ_NEED_WAKEUP ; +smp_wmb (); +continue ; + } +finish_wait (&ctx->sqo_wait, &wait); + + ctx->sq_ring->flags &= ~IORING_SQ_NEED_WAKEUP ; +smp_wmb (); + } + + i =0 ; + all_fixed = true; +do { +if (sqes[i].sqe->opcode !=IORING_OP_READ_FIXED && + sqes[i].sqe->opcode !=IORING_OP_WRITE_FIXED ) + all_fixed = false; +if (i +1 ==ARRAY_SIZE (sqes)) +break ; + i++; + }while (io_get_sqring (ctx, &sqes[i])); + +io_commit_sqring (ctx); + +/* Unless all new commands are FIXED regions, grab mm */ +if (!all_fixed && !cur_mm) { + mm_fault = !mmget_not_zero (ctx->sqo_mm); +if (!mm_fault) { +use_mm (ctx->sqo_mm); + cur_mm = ctx->sqo_mm; + } + } + + inflight +=io_submit_sqes (ctx, sqes, i, mm_fault); + } + current->files = old_files; +set_fs (old_fs); +if (cur_mm) { +unuse_mm (cur_mm); +mmput (cur_mm); + } +return 0 ; +} +static int io_ring_submit (struct io_ring_ctx *ctx,unsigned int to_submit) {struct io_submit_state state, *statep =NULL ;
@@ -1138,9 +1303,14 @@ static int __io_uring_enter(struct io_ring_ctx *ctx, unsigned to_submit,int ret =0 ;if (to_submit) { - ret =io_ring_submit (ctx, to_submit); -if (ret <0 ) -return ret; +if (ctx->flags &IORING_SETUP_SQPOLL ) { +wake_up (&ctx->sqo_wait); + ret = to_submit; + }else { + ret =io_ring_submit (ctx, to_submit); +if (ret <0 ) +return ret; + } }if (flags &IORING_ENTER_GETEVENTS ) {unsigned nr_events =0 ;
@@ -1162,10 +1332,12 @@ static int __io_uring_enter(struct io_ring_ctx *ctx, unsigned to_submit,return ret; } -static int io_sq_offload_start (struct io_ring_ctx *ctx) +static int io_sq_offload_start (struct io_ring_ctx *ctx, +struct io_uring_params *p) {int ret; +init_waitqueue_head (&ctx->sqo_wait); ctx->sqo_mm = current->mm;/*
@@ -1178,6 +1350,27 @@ static int io_sq_offload_start(struct io_ring_ctx *ctx)if (!ctx->sqo_files) goto err; +if (ctx->flags &IORING_SETUP_SQPOLL ) { +if (p->flags &IORING_SETUP_SQ_AFF ) { + ctx->sqo_thread =kthread_create_on_cpu (io_sq_thread, + ctx, p->sq_thread_cpu, +"io_uring-sq" ); + }else { + ctx->sqo_thread =kthread_create (io_sq_thread, ctx, +"io_uring-sq" ); + } +if (IS_ERR (ctx->sqo_thread)) { + ret =PTR_ERR (ctx->sqo_thread); + ctx->sqo_thread =NULL ; + goto err; + } +wake_up_process (ctx->sqo_thread); + }else if (p->flags &IORING_SETUP_SQ_AFF ) { +/* Can't have SQ_AFF without SQPOLL */ + ret = -EINVAL ; + goto err; + } +/* Do QD, or 2 * CPUS, whatever is smallest */ ctx->sqo_wq =alloc_workqueue ("io_ring-wq" ,WQ_UNBOUND |WQ_FREEZABLE ,min (ctx->sq_entries -1 ,2 *num_online_cpus ()));
@@ -1188,6 +1381,11 @@ static int io_sq_offload_start(struct io_ring_ctx *ctx)return 0 ; err: +if (ctx->sqo_thread) { +kthread_park (ctx->sqo_thread); +kthread_stop (ctx->sqo_thread); + ctx->sqo_thread =NULL ; + }if (ctx->sqo_files) ctx->sqo_files =NULL ; ctx->sqo_mm =NULL ;
@@ -1196,6 +1394,11 @@ static int io_sq_offload_start(struct io_ring_ctx *ctx)static void io_sq_offload_stop (struct io_ring_ctx *ctx) { +if (ctx->sqo_thread) { +kthread_park (ctx->sqo_thread); +kthread_stop (ctx->sqo_thread); + ctx->sqo_thread =NULL ; + }if (ctx->sqo_wq) {destroy_workqueue (ctx->sqo_wq); ctx->sqo_wq =NULL ;
@@ -1586,7 +1789,7 @@ static int io_uring_create(unsigned entries, struct io_uring_params *p,if (ret) goto err; - ret =io_sq_offload_start (ctx); + ret =io_sq_offload_start (ctx, p);if (ret) goto err;
@@ -1621,7 +1824,8 @@ static long io_uring_setup(u32 entries, struct io_uring_params __user *params,return -EINVAL ; } -if (p.flags & ~IORING_SETUP_IOPOLL ) +if (p.flags & ~(IORING_SETUP_IOPOLL |IORING_SETUP_SQPOLL | +IORING_SETUP_SQ_AFF ))return -EINVAL ; ret =io_uring_create (entries, &p, compat);
include/uapi/linux/io_uring.h +9 -1
@@ -37,6 +37,8 @@ struct io_uring_sqe {* io_uring_setup() flags */ #define IORING_SETUP_IOPOLL (1 << 0)/* io_context is polled */ +#define IORING_SETUP_SQPOLL (1 << 1)/* SQ poll thread */ +#define IORING_SETUP_SQ_AFF (1 << 2)/* sq_thread_cpu is valid */ #define IORING_OP_NOP 0#define IORING_OP_READV 1
@@ -80,6 +82,11 @@ struct io_sqring_offsets {__u32 resv[3 ]; }; +/* +* sq_ring->flags +*/ +#define IORING_SQ_NEED_WAKEUP (1 << 0)/* needs io_uring_enter wakeup */ +struct io_cqring_offsets {__u32 head;__u32 tail;
@@ -102,7 +109,8 @@ struct io_uring_params {__u32 sq_entries;__u32 cq_entries;__u32 flags; -__u16 resv[10 ]; +__u16 sq_thread_cpu; +__u16 resv[9 ];struct io_sqring_offsets sq_off;struct io_cqring_offsets cq_off; };
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 14/15 io_uring: add file registration
In reply to Jens AxboeWe normally have to fget/fput for each IO we do on a file. Even with the batching we do, this atomic inc/dec cost adds up.
This adds IORING_REGISTER_FILES, and IORING_UNREGISTER_FILES opcodes for the io_uring_register(2) system call. The arguments passed in must be an array of __s32 holding file descriptors, and nr_args should hold the number of file descriptors the application wishes to pin for the duration of the io_uring context (or until IORING_UNREGISTER_FILES is called).
When used, the application must set IOSQE_FIXED_FILE in the sqe->flags member. Then, instead of setting sqe->fd to the real fd, it sets sqe->fd to the index in the array passed in to IORING_REGISTER_FILES.
Files are automatically unregistered when the io_uring context is torn down. An application need only unregister if it wishes to register a few set of fds.
--- fs/io_uring.c | 130 ++++++++++++++++++++++++++++------ include/uapi/linux/io_uring.h | 9 ++- 2 files changed, 118 insertions(+), 21 deletions(-)
fs/io_uring.c +110 -20
@@ -94,6 +94,10 @@ struct io_ring_ctx {struct files_struct *sqo_files;wait_queue_head_t sqo_wait; +/* if used, fixed file set */ +struct file **user_files; +unsigned nr_user_files; +/* if used, fixed mapped user buffers */ unsigned nr_user_bufs;struct io_mapped_ubuf *user_bufs;
@@ -135,6 +139,7 @@ struct io_kiocb {#define REQ_F_FORCE_NONBLOCK 1/* inline submission attempt */ #define REQ_F_IOPOLL_COMPLETED 2/* polled IO has completed */ #define REQ_F_IOPOLL_EAGAIN 4/* submission got EAGAIN */ +#define REQ_F_FIXED_FILE 8/* ctx owns file */ u64 user_data;u64 res; };
@@ -355,15 +360,17 @@ static void io_iopoll_complete(struct io_ring_ctx *ctx, unsigned int *nr_events,* Batched puts of the same file, to avoid dirtying the * file usage count multiple times, if avoidable. */ -if (!file) { - file = req->rw.ki_filp; - file_count =1 ; - }else if (file == req->rw.ki_filp) { - file_count++; - }else { -fput_many (file, file_count); - file = req->rw.ki_filp; - file_count =1 ; +if (!(req->flags &REQ_F_FIXED_FILE )) { +if (!file) { + file = req->rw.ki_filp; + file_count =1 ; + }else if (file == req->rw.ki_filp) { + file_count++; + }else { +fput_many (file, file_count); + file = req->rw.ki_filp; + file_count =1 ; + } }if (to_free ==ARRAY_SIZE (reqs))
@@ -500,13 +507,19 @@ static void kiocb_end_write(struct kiocb *kiocb) } } +static void io_fput (struct io_kiocb *req) +{ +if (!(req->flags &REQ_F_FIXED_FILE )) +fput (req->rw.ki_filp); +} +static void io_complete_rw (struct kiocb *kiocb,long res,long res2) { struct io_kiocb *req = container_of(kiocb, struct io_kiocb, rw);kiocb_end_write (kiocb); -fput (kiocb->ki_filp); +io_fput (req);io_cqring_fill_event (req->ctx, req->user_data, res,0 );io_free_req (req); }
@@ -610,7 +623,17 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe,struct kiocb *kiocb = &req->rw;int ret; - kiocb->ki_filp =io_file_get (state, sqe->fd); +if (unlikely (sqe->flags & ~IOSQE_FIXED_FILE )) +return -EINVAL ; + +if (sqe->flags &IOSQE_FIXED_FILE ) { +if (unlikely (!ctx->user_files || sqe->fd >= ctx->nr_user_files)) +return -EBADF ; + kiocb->ki_filp = ctx->user_files[sqe->fd]; + req->flags |=REQ_F_FIXED_FILE ; + }else { + kiocb->ki_filp =io_file_get (state, sqe->fd); + }if (unlikely (!kiocb->ki_filp))return -EBADF ; kiocb->ki_pos = sqe->off;
@@ -649,7 +672,8 @@ static int io_prep_rw(struct io_kiocb *req, const struct io_uring_sqe *sqe, }return 0 ; out_fput: -io_file_put (state, kiocb->ki_filp); +if (!(sqe->flags &IOSQE_FIXED_FILE )) +io_file_put (state, kiocb->ki_filp);return ret; }
@@ -766,7 +790,7 @@ static ssize_t io_read(struct io_kiocb *req, const struct io_uring_sqe *sqe,kfree (iovec); out_fput:if (unlikely (ret)) -fput (file); +io_fput (req);return ret; }
@@ -820,7 +844,7 @@ static ssize_t io_write(struct io_kiocb *req, const struct io_uring_sqe *sqe, } out_fput:if (unlikely (ret)) -fput (file); +io_fput (req);return ret; }
@@ -853,19 +877,30 @@ static int io_fsync(struct io_kiocb *req, const struct io_uring_sqe *sqe,if (unlikely (req->ctx->flags &IORING_SETUP_IOPOLL ))return -EINVAL ; +if (unlikely (sqe->flags & ~IOSQE_FIXED_FILE )) +return -EINVAL ;if (unlikely (sqe->addr))return -EINVAL ;if (unlikely (sqe->fsync_flags & ~IORING_FSYNC_DATASYNC ))return -EINVAL ; - file =fget (sqe->fd); +if (sqe->flags &IOSQE_FIXED_FILE ) { +if (unlikely (!ctx->user_files || sqe->fd >= ctx->nr_user_files)) +return -EBADF ; + file = ctx->user_files[sqe->fd]; + }else { + file =fget (sqe->fd); + } +if (unlikely (!file))return -EBADF ; ret =vfs_fsync_range (file, sqe->off, end >0 ? end :LLONG_MAX , sqe->fsync_flags &IORING_FSYNC_DATASYNC ); -fput (file); +if (!(sqe->flags &IOSQE_FIXED_FILE )) +fput (file); +io_cqring_fill_event (ctx, sqe->user_data, ret,0 );io_free_req (req);return 0 ;
@@ -878,10 +913,6 @@ static int __io_submit_sqe(struct io_ring_ctx *ctx, struct io_kiocb *req,const struct io_uring_sqe *sqe = s->sqe;ssize_t ret; -/* enforce forwards compatibility on users */ -if (unlikely (sqe->flags)) -return -EINVAL ; -if (unlikely (s->index >= ctx->sq_entries))return -EINVAL ; req->user_data = sqe->user_data;
@@ -1332,6 +1363,55 @@ static int __io_uring_enter(struct io_ring_ctx *ctx, unsigned to_submit,return ret; } +static int io_sqe_files_unregister (struct io_ring_ctx *ctx) +{ +int i; + +if (!ctx->user_files) +return -EINVAL ; + +for (i =0 ; i < ctx->nr_user_files; i++) +fput (ctx->user_files[i]); + +kfree (ctx->user_files); + ctx->user_files =NULL ; + ctx->nr_user_files =0 ; +return 0 ; +} + +static int io_sqe_files_register (struct io_ring_ctx *ctx,void __user *arg, +unsigned nr_args) +{ +__s32 __user *fds = (__s32 __user *) arg; +int fd, i, ret =0 ; + +if (!nr_args) +return -EINVAL ; + + ctx->user_files =kcalloc (nr_args,sizeof (struct file *),GFP_KERNEL ); +if (!ctx->user_files) +return -ENOMEM ; + +for (i =0 ; i < nr_args; i++) { + ret = -EFAULT ; +if (copy_from_user (&fd, &fds[i],sizeof (fd))) +break ; + + ctx->user_files[i] =fget (fd); + + ret = -EBADF ; +if (!ctx->user_files[i]) +break ; + ctx->nr_user_files++; + ret =0 ; + } + +if (ret) +io_sqe_files_unregister (ctx); + +return ret; +} +static int io_sq_offload_start (struct io_ring_ctx *ctx,struct io_uring_params *p) {
@@ -1603,6 +1683,7 @@ static void io_ring_ctx_free(struct io_ring_ctx *ctx)io_sq_offload_stop (ctx);io_iopoll_reap_events (ctx);io_free_scq_urings (ctx); +io_sqe_files_unregister (ctx);io_sqe_buffer_unregister (ctx);percpu_ref_exit (&ctx->refs);kfree (ctx);
@@ -1872,6 +1953,15 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,break ; ret =io_sqe_buffer_unregister (ctx);break ; +case IORING_REGISTER_FILES : + ret =io_sqe_files_register (ctx, arg, nr_args); +break ; +case IORING_UNREGISTER_FILES : + ret = -EINVAL ; +if (arg || nr_args) +break ; + ret =io_sqe_files_unregister (ctx); +break ;default : ret = -EINVAL ;break ;
include/uapi/linux/io_uring.h +8 -1
@@ -16,7 +16,7 @@*/ struct io_uring_sqe {__u8 opcode;/* type of operation for this sqe */ -__u8 flags;/* as of now unused */ +__u8 flags;/* IOSQE_ flags */ __u16 ioprio;/* ioprio for the request */ __s32 fd;/* file descriptor to do IO on */ __u64 off;/* offset into file */
@@ -33,6 +33,11 @@ struct io_uring_sqe { }; }; +/* +* sqe->flags +*/ +#define IOSQE_FIXED_FILE (1 << 0)/* use fixed fileset */ +/* * io_uring_setup() flags */
@@ -120,5 +125,7 @@ struct io_uring_params {*/ #define IORING_REGISTER_BUFFERS 0#define IORING_UNREGISTER_BUFFERS 1 +#define IORING_REGISTER_FILES 2 +#define IORING_UNREGISTER_FILES 3#endif
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>
Jens Axboe
maintainerPATCH 15/15 io_uring: add io_uring_event cache hit information
In reply to Jens AxboeAdd hint on whether a read was served out of the page cache, or if it hit media. This is useful for buffered async IO, O_DIRECT reads would never have this set (for obvious reasons).
If the read hit page cache, cqe->flags will have IOCQE_FLAG_CACHEHIT set.
--- fs/io_uring.c | 7 ++++++- include/uapi/linux/io_uring.h | 5 +++++ 2 files changed, 11 insertions(+), 1 deletion(-)
fs/io_uring.c +6 -1
@@ -516,11 +516,16 @@ static void io_fput(struct io_kiocb *req)static void io_complete_rw (struct kiocb *kiocb,long res,long res2) { struct io_kiocb *req = container_of(kiocb, struct io_kiocb, rw); +unsigned ev_flags =0 ;kiocb_end_write (kiocb);io_fput (req); -io_cqring_fill_event (req->ctx, req->user_data, res,0 ); + +if (res >0 && (req->flags &REQ_F_FORCE_NONBLOCK )) + ev_flags =IOCQE_FLAG_CACHEHIT ; + +io_cqring_fill_event (req->ctx, req->user_data, res, ev_flags);io_free_req (req); }
include/uapi/linux/io_uring.h +5 -0
@@ -66,6 +66,11 @@ struct io_uring_cqe {__u32 flags; }; +/* +* io_uring_event->flags +*/ +#define IOCQE_FLAG_CACHEHIT (1 << 0)/* IO did not hit media */ +/* * Magic offsets for the application to mmap the data it needs */
Signature · 7 lines
2.17.1 -- To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO, see: http://www.kvack.org/aio/ Don't email: <a href=mailto:"[email protected]">[email protected]</a>