io_uring: add support for IORING_OP_GETDENTS
v3 cover io_uring: add support for IORING_OP_GETDENTS
These patches add support for IORING_OP_GETDENTS, which is a new io_uring opcode that more or less does an lseek(sqe->fd, sqe->off, SEEK_SET) followed by a getdents64(sqe->fd, (void *)sqe->addr, sqe->len).
A dumb test program for IORING_OP_GETDENTS is available here:
https://krautbox.wantstofly.org/~buytenh/uringfind-v2.c
This test program does something along the lines of what find(1) does:
it scans recursively through a directory tree and prints the names of all directories and files it encounters along the way -- but then using io_uring. (The io_uring version prints the names of encountered files and directories in an order that's determined by SQE completion order, which is somewhat nondeterministic and likely to differ between runs.)
On a directory tree with 14-odd million files in it that's on a six-drive (spinning disk) btrfs raid, find(1) takes:
# echo 3 > /proc/sys/vm/drop_caches # time find /mnt/repo > /dev/null
real 24m7.815s user 0m15.015s sys 0m48.340s #
And the io_uring version takes:
# echo 3 > /proc/sys/vm/drop_caches # time ./uringfind /mnt/repo > /dev/null
real 10m29.064s user 0m4.347s sys 0m1.677s #
The fully cached case also shows some speedup. find(1):
# time find /mnt/repo > /dev/null
real 0m5.223s user 0m1.926s sys 0m3.268s #
Versus the io_uring version:
# time ./uringfind /mnt/repo > /dev/null
real 0m3.604s user 0m2.417s sys 0m0.793s #
That said, the point of this patch isn't primarily to enable lightning-fast find(1) or du(1), but more to complete the set of filesystem I/O primitives available via io_uring, so that applications can do all of their filesystem I/O using the same mechanism, without having to manually punt some of their work out to worker threads -- and indeed, an object storage backend server that I wrote a while ago can run with a pure io_uring based event loop with this patch.
Changes since v2 RFC:
- Rebase onto io_uring-2021-02-17 plus a manually applied version of the mkdirat patch. The latter is needed because userland (liburing) has already merged the opcode for IORING_OP_MKDIRAT (in commit "io_uring.h: 5.12 pending kernel sync") while this opcode isn't in the kernel yet (as of io_uring-2021-02-17), and this means that this can't be merged until IORING_OP_MKDIRAT is merged.
- Adapt to changes made in "io_uring: replace force_nonblock with flags" that are in io_uring-2021-02-17.
Changes since v1 RFC:
- Drop the trailing '64' from IORING_OP_GETDENTS64 (suggested by Matthew Wilcox).
- Instead of requiring that sqe->off be zero, use this field to pass in a directory offset to start reading from. For the first IORING_OP_GETDENTS call on a directory, this can be set to zero, and for subsequent calls, it can be set to the ->d_off field of the last struct linux_dirent64 returned by the previous call.
Lennert Buytenhek (2):
readdir: split the core of getdents64(2) out into vfs_getdents()
io_uring: add support for IORING_OP_GETDENTS
fs/io_uring.c | 73 ++++++++++++++++++++++++++++++++++++++++++ fs/readdir.c | 25 +++++++++----- include/linux/fs.h | 4 ++ include/uapi/linux/io_uring.h | 1 4 files changed, 95 insertions(+), 8 deletions(-)
PATCH v3 1/2 readdir: split the core of getdents64(2) out into vfs_getdents()
So that IORING_OP_GETDENTS may use it, split out the core of the getdents64(2) syscall into a helper function, vfs_getdents().
vfs_getdents() calls into filesystems' ->iterate{,_shared}() which expect serialization on struct file, which means that callers of vfs_getdents() are responsible for either using fdget_pos() or performing the equivalent serialization by hand.
--- fs/readdir.c | 25 +++++++++++++++++-------- include/linux/fs.h | 4 ++++ 2 files changed, 21 insertions(+), 8 deletions(-)
fs/readdir.c +17 -8
@@ -348,10 +348,9 @@ static int filldir64(struct dir_context *ctx, const char *name, int namlen,return -EFAULT ; } -SYSCALL_DEFINE3(getdents64, unsigned int, fd, - struct linux_dirent64 __user *, dirent, unsigned int, count) +int vfs_getdents (struct file *file,struct linux_dirent64 __user *dirent, +unsigned int count) { -struct fd f;struct getdents_callback64 buf = { .ctx.actor = filldir64, .count = count,
@@ -359,11 +358,7 @@ SYSCALL_DEFINE3(getdents64, unsigned int, fd, };int error; - f =fdget_pos (fd); -if (!f.file) -return -EBADF ; - - error =iterate_dir (f.file, &buf.ctx); + error =iterate_dir (file, &buf.ctx);if (error >=0 ) error = buf.error;if (buf.prev_reclen) {
@@ -376,6 +371,20 @@ SYSCALL_DEFINE3(getdents64, unsigned int, fd,else error = count - buf.count; } +return error; +} + +SYSCALL_DEFINE3(getdents64, unsigned int, fd, + struct linux_dirent64 __user *, dirent, unsigned int, count) +{ +struct fd f; +int error; + + f =fdget_pos (fd); +if (!f.file) +return -EBADF ; + + error =vfs_getdents (f.file, dirent, count);fdput_pos (f);return error; }
include/linux/fs.h +4 -0
@@ -3109,6 +3109,10 @@ extern const struct inode_operations simple_symlink_inode_operations;extern int iterate_dir (struct file *,struct dir_context *); +struct linux_dirent64 ; +int vfs_getdents (struct file *file,struct linux_dirent64 __user *dirent, +unsigned int count); +int vfs_fstatat (int dfd,const char __user *filename,struct kstat *stat,int flags);int vfs_fstat (int fd,struct kstat *stat);
Signature · 1 line
2.29.2
PATCH v3 2/2 io_uring: add support for IORING_OP_GETDENTS
In reply to Lennert BuytenhekIORING_OP_GETDENTS behaves much like getdents64(2) and takes the same arguments, but with a small twist: it takes an additional offset argument, and reading from the specified directory starts at the given offset.
For the first IORING_OP_GETDENTS call on a directory, the offset parameter can be set to zero, and for subsequent calls, it can be set to the ->d_off field of the last struct linux_dirent64 returned by the previous IORING_OP_GETDENTS call.
Internally, if necessary, IORING_OP_GETDENTS will vfs_llseek() to the right directory position before calling vfs_getdents().
IORING_OP_GETDENTS may or may not update the specified directory's file offset, and the file offset should not be relied upon having any particular value during or after an IORING_OP_GETDENTS call.
--- fs/io_uring.c | 73 +++++++++++++++++++++++++++++++++++ include/uapi/linux/io_uring.h | 1 + 2 files changed, 74 insertions(+)
fs/io_uring.c +73 -0
@@ -635,6 +635,13 @@ struct io_mkdir {struct filename *filename; }; +struct io_getdents { +struct file *file; +struct linux_dirent64 __user *dirent; +unsigned int count; +loff_t pos; +}; +struct io_completion {struct file *file;struct list_head list;
@@ -772,6 +779,7 @@ struct io_kiocb {struct io_rename rename;struct io_unlink unlink;struct io_mkdir mkdir; +struct io_getdents getdents;/* use only after cleaning per-op data, see io_clean_op() */ struct io_completion compl; };
@@ -1030,6 +1038,11 @@ static const struct io_op_def io_op_defs[] = { .work_flags = IO_WQ_WORK_MM | IO_WQ_WORK_FILES | IO_WQ_WORK_FS | IO_WQ_WORK_BLKCG, }, + [IORING_OP_GETDENTS] = { + .needs_file = 1, + .work_flags = IO_WQ_WORK_MM | IO_WQ_WORK_FILES | + IO_WQ_WORK_FS | IO_WQ_WORK_BLKCG, + }, }; static void io_uring_try_cancel_requests(struct io_ring_ctx *ctx,
@@ -4677,6 +4690,61 @@ static int io_sync_file_range(struct io_kiocb *req, unsigned int issue_flags)return 0 ; } +static int io_getdents_prep (struct io_kiocb *req, +const struct io_uring_sqe *sqe) +{ +struct io_getdents *getdents = &req->getdents; + +if (unlikely (req->ctx->flags &IORING_SETUP_IOPOLL )) +return -EINVAL ; +if (sqe->ioprio || sqe->rw_flags || sqe->buf_index) +return -EINVAL ; + + getdents->pos =READ_ONCE (sqe->off); + getdents->dirent =u64_to_user_ptr (READ_ONCE (sqe->addr)); + getdents->count =READ_ONCE (sqe->len); +return 0 ; +} + +static int io_getdents (struct io_kiocb *req,unsigned int issue_flags) +{ +struct io_getdents *getdents = &req->getdents; +bool pos_unlock = false; +int ret =0 ; + +/* getdents always requires a blocking context */ +if (issue_flags &IO_URING_F_NONBLOCK ) +return -EAGAIN ; + +/* for vfs_llseek and to serialize ->iterate_shared() on this file */ +if (file_count (req->file) >1 ) { + pos_unlock = true; +mutex_lock (&req->file->f_pos_lock); + } + +if (req->file->f_pos != getdents->pos) { +loff_t res =vfs_llseek (req->file, getdents->pos,SEEK_SET ); +if (res <0 ) + ret = res; + } + +if (ret ==0 ) { + ret =vfs_getdents (req->file, getdents->dirent, + getdents->count); + } + +if (pos_unlock) +mutex_unlock (&req->file->f_pos_lock); + +if (ret <0 ) { +if (ret == -ERESTARTSYS ) + ret = -EINTR ; +req_set_fail_links (req); + } +io_req_complete (req, ret); +return 0 ; +} +#if defined(CONFIG_NET )static int io_setup_async_msg (struct io_kiocb *req,struct io_async_msghdr *kmsg)
@@ -6184,6 +6252,8 @@ static int io_req_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)return io_unlinkat_prep (req, sqe);case IORING_OP_MKDIRAT :return io_mkdirat_prep (req, sqe); +case IORING_OP_GETDENTS : +return io_getdents_prep (req, sqe); } printk_once(KERN_WARNING "io_uring: unhandled opcode %d\n",
@@ -6428,6 +6498,9 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)case IORING_OP_MKDIRAT : ret =io_mkdirat (req, issue_flags);break ; +case IORING_OP_GETDENTS : + ret =io_getdents (req, issue_flags); +break ;default : ret = -EINVAL ;break ;
include/uapi/linux/io_uring.h +1 -0
@@ -138,6 +138,7 @@ enum {IORING_OP_RENAMEAT ,IORING_OP_UNLINKAT ,IORING_OP_MKDIRAT , +IORING_OP_GETDENTS ,/* this goes last, obviously */ IORING_OP_LAST,
Signature · 1 line
2.29.2
Quoted 31 lines · Lennert Buytenhek
IORING_OP_GETDENTSbehaves much like getdents64(2) and takes the same arguments, but with a small twist: it takes an additional offset argument, and reading from the specified directory starts at the given offset.For the first
IORING_OP_GETDENTScall on a directory, the offset parameter can be set to zero, and for subsequent calls, it can be set to the ->d_off field of the last structlinux_dirent64returned by the previousIORING_OP_GETDENTScall.Internally, if necessary,
IORING_OP_GETDENTSwillvfs_llseek()to the right directory position before callingvfs_getdents().
IORING_OP_GETDENTSmay or may not update the specified directory's file offset, and the file offset should not be relied upon having any particular value during or after anIORING_OP_GETDENTScall.Signed-off-by Lennert Buytenhek <[email protected]>--- fs/io_uring.c | 73 +++++++++++++++++++++++++++++++++++ include/uapi/linux/io_uring.h | 1 + 2 files changed, 74 insertions(+)fs/io_uring.c +0
-0@@ -635,6 +635,13 @@ struct io_mkdir {struct filename *filename; };
[...]
fs/io_uring.c static int io_sync_file_range(struct io_kiocb *req, unsigned int issue_flags) · line 4720
Quoted 12 lines
fs/io_uring.c +12
-0+static int io_getdents (struct io_kiocb *req,unsigned int issue_flags) +{ +struct io_getdents *getdents = &req->getdents; +bool pos_unlock = false; +int ret =0 ; + +/* getdents always requires a blocking context */ +if (issue_flags &IO_URING_F_NONBLOCK ) +return -EAGAIN ; + +/* for vfs_llseek and to serialize ->iterate_shared() on this file */ +if (file_count (req->file) >1 ) {
Looks racy, is it safe? E.g. can be concurrently dupped and used, or just several similar IORING_OP_GETDENTS requests.
fs/io_uring.c static int io_sync_file_range(struct io_kiocb *req, unsigned int issue_flags) · line 4726
Quoted 6 lines
fs/io_uring.c +6
-0+ pos_unlock = true; +mutex_lock (&req->file->f_pos_lock); + } + +if (req->file->f_pos != getdents->pos) { +loff_t res =vfs_llseek (req->file, getdents->pos,SEEK_SET );
I may be missing the previous discussions, but can this ever become stateless, like passing an offset? Including readdir.c and beyond.
Quoted 20 lines
fs/io_uring.c +20
-0+if (res <0 ) + ret = res; + } + +if (ret ==0 ) { + ret =vfs_getdents (req->file, getdents->dirent, + getdents->count); + } + +if (pos_unlock) +mutex_unlock (&req->file->f_pos_lock); + +if (ret <0 ) { +if (ret == -ERESTARTSYS ) + ret = -EINTR ; +req_set_fail_links (req); + } +io_req_complete (req, ret); +return 0 ; +}
[...]
Signature · 1 line
Pavel Begunkov
Quoted 49 lines · Pavel Begunkov
Quoted 31 lines · Lennert Buytenhek
IORING_OP_GETDENTSbehaves much like getdents64(2) and takes the same arguments, but with a small twist: it takes an additional offset argument, and reading from the specified directory starts at the given offset.For the first
IORING_OP_GETDENTScall on a directory, the offset parameter can be set to zero, and for subsequent calls, it can be set to the ->d_off field of the last structlinux_dirent64returned by the previousIORING_OP_GETDENTScall.Internally, if necessary,
IORING_OP_GETDENTSwillvfs_llseek()to the right directory position before callingvfs_getdents().
IORING_OP_GETDENTSmay or may not update the specified directory's file offset, and the file offset should not be relied upon having any particular value during or after anIORING_OP_GETDENTScall.Signed-off-by Lennert Buytenhek <[email protected]>--- fs/io_uring.c | 73 +++++++++++++++++++++++++++++++++++ include/uapi/linux/io_uring.h | 1 + 2 files changed, 74 insertions(+)fs/io_uring.c +0
-0@@ -635,6 +635,13 @@ struct io_mkdir {struct filename *filename; };[...]
fs/io_uring.c static int io_sync_file_range(struct io_kiocb *req, unsigned int issue_flags) · line 4720
Quoted 12 lines
fs/io_uring.c +12
-0+static int io_getdents (struct io_kiocb *req,unsigned int issue_flags) +{ +struct io_getdents *getdents = &req->getdents; +bool pos_unlock = false; +int ret =0 ; + +/* getdents always requires a blocking context */ +if (issue_flags &IO_URING_F_NONBLOCK ) +return -EAGAIN ; + +/* for vfs_llseek and to serialize ->iterate_shared() on this file */ +if (file_count (req->file) >1 ) {Looks racy, is it safe? E.g. can be concurrently dupped and used, or just several similar
IORING_OP_GETDENTSrequests.
Quoted 9 lines
fs/io_uring.c static int io_sync_file_range(struct io_kiocb *req, unsigned int issue_flags) · line 4726
Quoted 6 lines
fs/io_uring.c +6
-0+ pos_unlock = true; +mutex_lock (&req->file->f_pos_lock); + } + +if (req->file->f_pos != getdents->pos) { +loff_t res =vfs_llseek (req->file, getdents->pos,SEEK_SET );I may be missing the previous discussions, but can this ever become stateless, like passing an offset? Including
readdir.cand beyond.
I mean without those seeks. An emulation would look like rewinding pos back after vfs_getdents, though might be awful on performance.
Quoted 23 lines
Quoted 21 lines
fs/io_uring.c +20
-0+if (res <0 ) + ret = res; + } + +if (ret ==0 ) { + ret =vfs_getdents (req->file, getdents->dirent, + getdents->count); + } + +if (pos_unlock) +mutex_unlock (&req->file->f_pos_lock); + +if (ret <0 ) { +if (ret == -ERESTARTSYS ) + ret = -EINTR ; +req_set_fail_links (req); + } +io_req_complete (req, ret); +return 0 ; +}[...]
Signature · 1 line
Pavel Begunkov
Quoted 32 lines · Pavel Begunkov
Quoted 31 lines
IORING_OP_GETDENTSbehaves much like getdents64(2) and takes the same arguments, but with a small twist: it takes an additional offset argument, and reading from the specified directory starts at the given offset.For the first
IORING_OP_GETDENTScall on a directory, the offset parameter can be set to zero, and for subsequent calls, it can be set to the ->d_off field of the last structlinux_dirent64returned by the previousIORING_OP_GETDENTScall.Internally, if necessary,
IORING_OP_GETDENTSwillvfs_llseek()to the right directory position before callingvfs_getdents().
IORING_OP_GETDENTSmay or may not update the specified directory's file offset, and the file offset should not be relied upon having any particular value during or after anIORING_OP_GETDENTScall.Signed-off-by Lennert Buytenhek <[email protected]>--- fs/io_uring.c | 73 +++++++++++++++++++++++++++++++++++ include/uapi/linux/io_uring.h | 1 + 2 files changed, 74 insertions(+)fs/io_uring.c +0
-0@@ -635,6 +635,13 @@ struct io_mkdir {struct filename *filename; };[...]
Quoted 15 lines
fs/io_uring.c static int io_sync_file_range(struct io_kiocb *req, unsigned int issue_flags) · line 4720
Quoted 12 lines
fs/io_uring.c +12
-0+static int io_getdents (struct io_kiocb *req,unsigned int issue_flags) +{ +struct io_getdents *getdents = &req->getdents; +bool pos_unlock = false; +int ret =0 ; + +/* getdents always requires a blocking context */ +if (issue_flags &IO_URING_F_NONBLOCK ) +return -EAGAIN ; + +/* for vfs_llseek and to serialize ->iterate_shared() on this file */ +if (file_count (req->file) >1 ) {Looks racy, is it safe? E.g. can be concurrently dupped and used, or just several similar
IORING_OP_GETDENTSrequests.
I thought that it was safe, but I thought about it a bit more, and it seems that it is unsafe -- if you IORING_REGISTER_FILES to register the dirfd and then close the dirfd, you'll get a file_count of 1, while you can submit concurrent operations. So I'll remove the conditional locking. Thanks!
(If not for IORING_REGISTER_FILES, it seems safe, because then io_file_get() will hold a(t least one) reference on the file while the operation is in flight, so then if file_count(req->file) == 1 here, then it means that the file is no longer referenced by any fdtable, and nobody else should be able to get a reference to it -- but that's a bit of a useless optimization.)
(Logic was taken from __fdget_pos, where it is safe for a different reason, i.e. __fget_light will not bump the refcount iff current->files is unshared.)
Quoted 9 lines
fs/io_uring.c static int io_sync_file_range(struct io_kiocb *req, unsigned int issue_flags) · line 4726
Quoted 6 lines
fs/io_uring.c +6
-0+ pos_unlock = true; +mutex_lock (&req->file->f_pos_lock); + } + +if (req->file->f_pos != getdents->pos) { +loff_t res =vfs_llseek (req->file, getdents->pos,SEEK_SET );I may be missing the previous discussions, but can this ever become stateless, like passing an offset? Including
readdir.cand beyond.
My aim was to only make the minimally required change initially, but to make that optimization possible in the future (e.g. by reserving the right to either update or not update the file position) -- but I'll try doing the optimization now.
Quoted 24 lines
Quoted 20 lines
fs/io_uring.c +20
-0+if (res <0 ) + ret = res; + } + +if (ret ==0 ) { + ret =vfs_getdents (req->file, getdents->dirent, + getdents->count); + } + +if (pos_unlock) +mutex_unlock (&req->file->f_pos_lock); + +if (ret <0 ) { +if (ret == -ERESTARTSYS ) + ret = -EINTR ; +req_set_fail_links (req); + } +io_req_complete (req, ret); +return 0 ; +}[...]
Signature · 1 line
Pavel Begunkov
Quoted 3 lines · Lennert Buytenhek
IORING_OP_GETDENTSmay or may not update the specified directory's file offset, and the file offset should not be relied upon having any particular value during or after anIORING_OP_GETDENTScall.
This doesn't give me the warm fuzzies. What I might suggest is either passing a parameter to iterate_dir() or breaking out an iterate_dir_nofpos() to make IORING_OP_GETDENTS more of a READV operation. ie the equivalent of this:
@@ -37,7 +37,7 @@
} while (0)
-int iterate_dir(struct file *file, struct dir_context *ctx) +int iterate_dir(struct file *file, struct dir_context *ctx, bool use_fpos)
{
struct inode *inode = file_inode(file);
bool shared = false;@@ -60,12 +60,14 @@ int iterate_dir(struct file *file, struct dir_context *ctx)
res = -ENOENT;
if (!IS_DEADDIR(inode)) {
- ctx->pos = file->f_pos;
+ if (use_fpos)
+ ctx->pos = file->f_pos;
if (shared)
res = file->f_op->iterate_shared(file, ctx);
else
res = file->f_op->iterate(file, ctx);
- file->f_pos = ctx->pos;
+ if (use_fpos)
+ file->f_pos = ctx->pos;
fsnotify_access(file);
file_accessed(file);
}That way there's no need to play with llseek or take a mutex on the f_pos of the directory.
Quoted 36 lines · Matthew Wilcox
Quoted 3 lines
IORING_OP_GETDENTSmay or may not update the specified directory's file offset, and the file offset should not be relied upon having any particular value during or after anIORING_OP_GETDENTScall.This doesn't give me the warm fuzzies. What I might suggest is either passing a parameter to
iterate_dir()or breaking out aniterate_dir_nofpos()to makeIORING_OP_GETDENTSmore of a READV operation. ie the equivalent of this:(quoted diff) +5
-3@@ -37,7 +37,7 @@ } while (0) -int iterate_dir(struct file *file, struct dir_context *ctx) +int iterate_dir(struct file *file, struct dir_context *ctx, bool use_fpos) { struct inode *inode = file_inode(file); bool shared = false;@@ -60,12 +60,14 @@ int iterate_dir(struct file *file, struct dir_context *ctx) res = -ENOENT; if (!IS_DEADDIR(inode)) { - ctx->pos = file->f_pos; + if (use_fpos) + ctx->pos = file->f_pos; if (shared) res = file->f_op->iterate_shared(file, ctx); else res = file->f_op->iterate(file, ctx); - file->f_pos = ctx->pos; + if (use_fpos) + file->f_pos = ctx->pos; fsnotify_access(file); file_accessed(file); }
Quoted 2 lines
That way there's no need to play with llseek or take a mutex on the
f_posof the directory.
I'll try this!
Quoted 36 lines · Lennert Buytenhek
Quoted 36 lines
Quoted 3 lines
IORING_OP_GETDENTSmay or may not update the specified directory's file offset, and the file offset should not be relied upon having any particular value during or after anIORING_OP_GETDENTScall.This doesn't give me the warm fuzzies. What I might suggest is either passing a parameter to
iterate_dir()or breaking out aniterate_dir_nofpos()to makeIORING_OP_GETDENTSmore of a READV operation. ie the equivalent of this:(quoted diff) +5
-3@@ -37,7 +37,7 @@ } while (0) -int iterate_dir(struct file *file, struct dir_context *ctx) +int iterate_dir(struct file *file, struct dir_context *ctx, bool use_fpos) { struct inode *inode = file_inode(file); bool shared = false;@@ -60,12 +60,14 @@ int iterate_dir(struct file *file, struct dir_context *ctx) res = -ENOENT; if (!IS_DEADDIR(inode)) { - ctx->pos = file->f_pos; + if (use_fpos) + ctx->pos = file->f_pos; if (shared) res = file->f_op->iterate_shared(file, ctx); else res = file->f_op->iterate(file, ctx); - file->f_pos = ctx->pos; + if (use_fpos) + file->f_pos = ctx->pos; fsnotify_access(file); file_accessed(file); }
Quoted 4 lines
Quoted 2 lines
That way there's no need to play with llseek or take a mutex on the
f_posof the directory.I'll try this!
The patch below (on top of v3) does what you suggest, and it removes the vfs_llseek() call, but there's two issues:
- We still need to take some sort of mutex on the directory, because, while ->iterate_shared() can be called concurrently on different struct files that point to the same underlying dir inode, it cannot be called concurrently on the same struct file. From Documentation/filesystems/porting.rst:
->iterate_shared() is added; it's a parallel variant of ->iterate(). Exclusion on struct file level is still provided (as well as that between it and lseek on the same struct file) but if your directory has been opened several times, you can get these called in parallel. [...]
- Calling a filesystem's ->iterate_shared() on the same dir with changing file positions but without calling the directory's ->llseek() in between to notify the filesystem of changes in the file position seems to violate an (unstated?) guarantee. It works on my btrfs root fs, since that uses generic_file_llseek() for directory ->llseek(), but e.g. ceph does:
| static loff_t ceph_dir_llseek(struct file *file, loff_t offset, int whence)
| {
| struct ceph_dir_file_info *dfi = file->private_data;
| struct inode *inode = file->f_mapping->host;
| loff_t retval;
|
| inode_lock(inode);
| retval = -EINVAL;
| switch (whence) {
| case SEEK_CUR:
| offset += file->f_pos;
| case SEEK_SET:
| break;
| case SEEK_END:
| retval = -EOPNOTSUPP;
| default:
| goto out;
| }
|
| if (offset >= 0) {
| if (need_reset_readdir(dfi, offset)) {
| dout("dir_llseek dropping %p content\n", file);
| reset_readdir(dfi);
| } else if (is_hash_order(offset) && offset > file->f_pos) {
| /* for hash offset, we don't know if a forward seek
| * is within same frag */
| dfi->dir_release_count = 0;
| dfi->readdir_cache_idx = -1;
| }
|
| if (offset != file->f_pos) {
| file->f_pos = offset;
| file->f_version = 0;
| dfi->file_info.flags &= ~CEPH_F_ATEND;
| }
| retval = offset;
| }
| out:
| inode_unlock(inode);
| return retval;
| }So I think we probably can't get rid of the conditional vfs_llseek() call for now (and we'll probably have to keep taking the dir's ->f_pos_lock) -- what do you think?
(The caveat about that the file pointer may or may not be updated by IORING_OP_GETDENTS would allow making this optimization in the future, and for now it would mean that you can't mix getdents64() and IORING_OP_GETDENTS calls on the same dirfd, which would seem like an unusual thing to do anyway.)
Thanks!
fs/io_uring.c +4 -12
@@ -4709,25 +4709,17 @@ static int io_getdents_prep(struct io_kiocb *req,static int io_getdents (struct io_kiocb *req,unsigned int issue_flags) {struct io_getdents *getdents = &req->getdents; -int ret =0 ; +int ret;/* getdents always requires a blocking context */ if (issue_flags &IO_URING_F_NONBLOCK )return -EAGAIN ; -/* for vfs_llseek and to serialize ->iterate_shared() on this file */ +/* to serialize ->iterate_shared() on this file */ mutex_lock (&req->file->f_pos_lock); -if (req->file->f_pos != getdents->pos) { -loff_t res =vfs_llseek (req->file, getdents->pos,SEEK_SET ); -if (res <0 ) - ret = res; - } - -if (ret ==0 ) { - ret =vfs_getdents (req->file, getdents->dirent, - getdents->count); - } + ret =vfs_getdents (req->file, getdents->dirent, + getdents->count, &getdents->pos);mutex_unlock (&req->file->f_pos_lock);
fs/readdir.c +19 -6
@@ -37,7 +37,7 @@ }while (0 ) -int iterate_dir(struct file *file, struct dir_context *ctx) +int __iterate_dir(struct file *file, struct dir_context *ctx, bool use_fpos) { struct inode *inode = file_inode(file);bool shared = false;
@@ -60,12 +60,14 @@ int iterate_dir(struct file *file, struct dir_context *ctx) res = -ENOENT ;if (!IS_DEADDIR (inode)) { - ctx->pos = file->f_pos; +if (use_fpos) + ctx->pos = file->f_pos;if (shared) res = file->f_op->iterate_shared (file, ctx);else res = file->f_op->iterate (file, ctx); - file->f_pos = ctx->pos; +if (use_fpos) + file->f_pos = ctx->pos;fsnotify_access (file);file_accessed (file); }
@@ -76,6 +78,11 @@ int iterate_dir(struct file *file, struct dir_context *ctx) out:return res; } + +int iterate_dir (struct file *file,struct dir_context *ctx) +{ +return __iterate_dir (file, ctx, true); +}EXPORT_SYMBOL (iterate_dir);/*
@@ -349,7 +356,7 @@ static int filldir64(struct dir_context *ctx, const char *name, int namlen, }int vfs_getdents (struct file *file,struct linux_dirent64 __user *dirent, -unsigned int count) +unsigned int count,loff_t *f_pos) {struct getdents_callback64 buf = { .ctx.actor = filldir64,
@@ -358,7 +365,13 @@ int vfs_getdents(struct file *file, struct linux_dirent64 __user *dirent, };int error; - error =iterate_dir (file, &buf.ctx); +if (f_pos ==NULL ) { + error =__iterate_dir (file, &buf.ctx, true); + }else { + buf.ctx.pos = *f_pos; + error =__iterate_dir (file, &buf.ctx, false); + } +if (error >=0 ) error = buf.error;if (buf.prev_reclen) {
@@ -384,7 +397,7 @@ SYSCALL_DEFINE3(getdents64, unsigned int, fd,if (!f.file)return -EBADF ; - error =vfs_getdents (f.file, dirent, count); + error =vfs_getdents (f.file, dirent, count,NULL );fdput_pos (f);return error; }
include/linux/fs.h +2 -1
@@ -3107,11 +3107,12 @@ const char *simple_get_link(struct dentry *, struct inode *, struct delayed_call *);extern const struct inode_operations simple_symlink_inode_operations; +extern int __iterate_dir (struct file *,struct dir_context *,bool );extern int iterate_dir (struct file *,struct dir_context *);struct linux_dirent64 ;int vfs_getdents (struct file *file,struct linux_dirent64 __user *dirent, -unsigned int count); +unsigned int count,loff_t *f_pos);int vfs_fstatat (int dfd,const char __user *filename,struct kstat *stat,int flags);
From: Lennert Buytenhek
Quoted 30 lines
Sent: 18 February 2021 12:27
These patches add support for
IORING_OP_GETDENTS, which is a newio_uringopcode that more or less does anlseek(sqe->fd, sqe->off, SEEK_SET)followed by agetdents64(sqe->fd, (void *)sqe->addr, sqe->len).A dumb test program for
IORING_OP_GETDENTSis available here:https://krautbox.wantstofly.org/~buytenh/uringfind-v2.c
This test program does something along the lines of what find(1) does:
it scans recursively through a directory tree and prints the names of all directories and files it encounters along the way -- but then usingio_uring. (Theio_uringversion prints the names of encountered files and directories in an order that's determined by SQE completion order, which is somewhat nondeterministic and likely to differ between runs.)On a directory tree with 14-odd million files in it that's on a six-drive (spinning disk) btrfs raid, find(1) takes:
# echo 3 > /proc/sys/vm/drop_caches # time find /mnt/repo > /dev/nullreal 24m7.815s user 0m15.015s sys 0m48.340s #And the
io_uringversion takes:
Quoted 7 lines
# echo 3 > /proc/sys/vm/drop_caches # time ./uringfind /mnt/repo > /dev/nullreal 10m29.064s user 0m4.347s sys 0m1.677s #
While there may be uses for IORING_OP_GETDENTS are you sure your test is comparing like with like?
The underlying work has to be done in either case, so you are swapping system calls for code complexity.
I suspect that find is actually doing a stat() call on every directory entry and that your io_uring example is just believing the 'directory' flag returned in the directory entry for most modern filesystems.
If you write a program that does openat(), readdir(), close() for each directory and with a long enough buffer (mostly) do one readdir() per directory you'll get a much better comparison.
You could even write a program with 2 threads, one does all the open/readdir/close system calls and the other does the printing and generating the list of directories to process. That should get the equivalent overlapping that io_uring gives without much of the complexity.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Jens Axboe
maintainerQuoted 39 lines · David Laight
From: Lennert Buytenhek
Quoted 30 lines
Sent: 18 February 2021 12:27
These patches add support for
IORING_OP_GETDENTS, which is a newio_uringopcode that more or less does anlseek(sqe->fd, sqe->off, SEEK_SET)followed by agetdents64(sqe->fd, (void *)sqe->addr, sqe->len).A dumb test program for
IORING_OP_GETDENTSis available here:https://krautbox.wantstofly.org/~buytenh/uringfind-v2.c
This test program does something along the lines of what find(1) does:
it scans recursively through a directory tree and prints the names of all directories and files it encounters along the way -- but then usingio_uring. (Theio_uringversion prints the names of encountered files and directories in an order that's determined by SQE completion order, which is somewhat nondeterministic and likely to differ between runs.)On a directory tree with 14-odd million files in it that's on a six-drive (spinning disk) btrfs raid, find(1) takes:
# echo 3 > /proc/sys/vm/drop_caches # time find /mnt/repo > /dev/nullreal 24m7.815s user 0m15.015s sys 0m48.340s #And the
io_uringversion takes:Quoted 7 lines
# echo 3 > /proc/sys/vm/drop_caches # time ./uringfind /mnt/repo > /dev/nullreal 10m29.064s user 0m4.347s sys 0m1.677s #
Quoted 4 lines
While there may be uses for
IORING_OP_GETDENTSare you sure your test is comparing like with like?
The underlying work has to be done in either case, so you are swapping system calls for code complexity.
What complexity?
Quoted 4 lines
I suspect that find is actually doing a
stat()call on every directory entry and that yourio_uringexample is just believing the 'directory' flag returned in the directory entry for most modern filesystems.
While that may be true (find doing stat as well), the runtime is clearly dominated by IO. Adding a stat on top would be an extra copy, but no extra IO.
Quoted 9 lines
If you write a program that does
openat(),readdir(),close()for each directory and with a long enough buffer (mostly) do onereaddir()per directory you'll get a much better comparison.You could even write a program with 2 threads, one does all the open/readdir/close system calls and the other does the printing and generating the list of directories to process. That should get the equivalent overlapping that
io_uringgives without much of the complexity.
But this is what take the most offense to - it's _trivial_ to write that program with io_uring, especially compared to managing threads. Threads are certainly a more known paradigm at this point, but an io_uring submit + reap loop is definitely not "much of the complexity". If you're referring to the kernel change itself, that's trivial, as the diffstat shows.
Signature · 1 line
Jens Axboe
From: Jens Axboe
Quoted 42 lines
Sent: 20 February 2021 18:29
Quoted 39 lines · David Laight
From: Lennert Buytenhek
Quoted 37 lines
Sent: 18 February 2021 12:27
These patches add support for
IORING_OP_GETDENTS, which is a newio_uringopcode that more or less does anlseek(sqe->fd, sqe->off, SEEK_SET)followed by agetdents64(sqe->fd, (void *)sqe->addr, sqe->len).A dumb test program for
IORING_OP_GETDENTSis available here:https://krautbox.wantstofly.org/~buytenh/uringfind-v2.c
This test program does something along the lines of what find(1) does:
it scans recursively through a directory tree and prints the names of all directories and files it encounters along the way -- but then usingio_uring. (Theio_uringversion prints the names of encountered files and directories in an order that's determined by SQE completion order, which is somewhat nondeterministic and likely to differ between runs.)On a directory tree with 14-odd million files in it that's on a six-drive (spinning disk) btrfs raid, find(1) takes:
# echo 3 > /proc/sys/vm/drop_caches # time find /mnt/repo > /dev/nullreal 24m7.815s user 0m15.015s sys 0m48.340s #And the
io_uringversion takes:# echo 3 > /proc/sys/vm/drop_caches # time ./uringfind /mnt/repo > /dev/nullreal 10m29.064s user 0m4.347s sys 0m1.677s #
Quoted 6 lines
Quoted 4 lines
While there may be uses for
IORING_OP_GETDENTSare you sure your test is comparing like with like?
The underlying work has to be done in either case, so you are swapping system calls for code complexity.What complexity?
Evan adding commands to a list to execute later is 'complexity'. As in adding more cpu cycles.
Quoted 8 lines
Quoted 4 lines
I suspect that find is actually doing a
stat()call on every directory entry and that yourio_uringexample is just believing the 'directory' flag returned in the directory entry for most modern filesystems.While that may be true (find doing stat as well), the runtime is clearly dominated by IO. Adding a stat on top would be an extra copy, but no extra IO.
I'd expect stat() to require the disk inode be read into memory. getdents() only requires the data of the directory be read. So calling stat() requires a lot more IO.
The other thing I just realises is that the 'system time' output from time is completely meaningless for the io_uring case. All that processing is done by a kernel thread and I doubt is re-attributed to the user process.
Quoted 10 lines
Quoted 9 lines
If you write a program that does
openat(),readdir(),close()for each directory and with a long enough buffer (mostly) do onereaddir()per directory you'll get a much better comparison.You could even write a program with 2 threads, one does all the open/readdir/close system calls and the other does the printing and generating the list of directories to process. That should get the equivalent overlapping that
io_uringgives without much of the complexity.
Quoted 6 lines
But this is what take the most offense to - it's _trivial_ to write that program with
io_uring, especially compared to managing threads. Threads are certainly a more known paradigm at this point, but anio_uringsubmit + reap loop is definitely not "much of the complexity". If you're referring to the kernel change itself, that's trivial, as the diffstat shows.
I've looked at the kernel code in io_uring.c.
Makes me pull my hair out (what's left of it - mostly beard). Apart from saving system call costs I don't actually understand why it isn't a userspace library?
Anyway, I thought the point of io_uring was to attempt to implement asynchronous IO on a unix system.
If you want async IO you need to go back to the mid 1970s and pick the ancestors of RSM/11M rather than those of K&R's unix. That leads you to Ultrix and then Windows NT.
And yes, I have written code that did async IO under RSM/11M.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Jens Axboe
maintainerQuoted 48 lines · David Laight
From: Jens Axboe
Quoted 47 lines
Sent: 20 February 2021 18:29
Quoted 43 lines · David Laight
From: Lennert Buytenhek
Quoted 30 lines
Sent: 18 February 2021 12:27
These patches add support for
IORING_OP_GETDENTS, which is a newio_uringopcode that more or less does anlseek(sqe->fd, sqe->off, SEEK_SET)followed by agetdents64(sqe->fd, (void *)sqe->addr, sqe->len).A dumb test program for
IORING_OP_GETDENTSis available here:https://krautbox.wantstofly.org/~buytenh/uringfind-v2.c
This test program does something along the lines of what find(1) does:
it scans recursively through a directory tree and prints the names of all directories and files it encounters along the way -- but then usingio_uring. (Theio_uringversion prints the names of encountered files and directories in an order that's determined by SQE completion order, which is somewhat nondeterministic and likely to differ between runs.)On a directory tree with 14-odd million files in it that's on a six-drive (spinning disk) btrfs raid, find(1) takes:
# echo 3 > /proc/sys/vm/drop_caches # time find /mnt/repo > /dev/nullreal 24m7.815s user 0m15.015s sys 0m48.340s #And the
io_uringversion takes:Quoted 7 lines
# echo 3 > /proc/sys/vm/drop_caches # time ./uringfind /mnt/repo > /dev/nullreal 10m29.064s user 0m4.347s sys 0m1.677s #While there may be uses for
IORING_OP_GETDENTSare you sure your test is comparing like with like?
The underlying work has to be done in either case, so you are swapping system calls for code complexity.
Quoted 4 lines
Quoted 1 line
What complexity?
Evan adding commands to a list to execute later is 'complexity'. As in adding more cpu cycles.
That's a pretty blanket statement. If the list is heavily shared, and hence contended, yes that's generally true. But it isn't.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
I suspect that find is actually doing a
stat()call on every directory entry and that yourio_uringexample is just believing the 'directory' flag returned in the directory entry for most modern filesystems.
Quoted 7 lines
Quoted 3 lines
While that may be true (find doing stat as well), the runtime is clearly dominated by IO. Adding a stat on top would be an extra copy, but no extra IO.
I'd expect
stat()to require the disk inode be read into memory.getdents()only requires the data of the directory be read. So callingstat()requires a lot more IO.
I actually went and checked instead of guessing, and find isn't doing a stat by default on the files.
Quoted 4 lines
The other thing I just realises is that the 'system time' output from time is completely meaningless for the
io_uringcase. All that processing is done by a kernel thread and I doubt is re-attributed to the user process.
For sure, you can't directly compare the sys times. But the actual runtime is of course directly comparable. The actual example is btrfs, which heavily offloads to threads as well. So the find case doesn't show you the full picture either. Note that once the reworked worker scheme is adopted, sys time will be directly attributed to the original task and it will be included (for io_uring, not talking about btrfs).
I'm going to ignore the rest of this email as it isn't productive going down that path. Suffice it to say that ideally _no_ operations will be using the async offload, that's really only for regular file IO where you cannot poll for readiness. And even those cases are gradually being improved to not rely on it, like for async buffered reads. We still need to handle writes better, and ideally things like this as well. But that's a bit further out.
Signature · 1 line
Jens Axboe