btrfs: io_uring interface for encoded reads
Subsystems (2)
v4 cover btrfs: io_uring interface for encoded reads
This is version 4 of a patch series to add an io_uring interface for encoded reads. The principal use case for this is to eventually allow btrfs send and receive to operate asynchronously, the lack of io_uring encoded I/O being one of the main blockers for this.
I've written a test program for this, which demonstrates the ioctl and io_uring interface produce identical results: https://github.com/maharmstone/io_uring-encoded
Changelog:
v4:
* Rewritten to avoid taking function pointer
* Removed nowait parameter, as this could be derived from iocb flags
* Fixed structure not getting properly initialized
* Followed ioctl by capping uncompressed reads at EOF
* Rebased against btrfs/for-next
* Formatting fixes
* Rearranged structs to minimize holes
* Published test program
* Fixed potential data race with userspace
* Changed to use io_uring_cmd_to_pdu helper function
* Added comments for potentially confusing parts of the code
v3:
* Redo of previous versions
Mark Harmstone (5):
btrfs: remove pointless addition in btrfs_encoded_read
btrfs: change btrfs_encoded_read so that reading of extent is done by
caller
btrfs: don't sleep in btrfs_encoded_read if IOCB_NOWAIT set
btrfs: move priv off stack in btrfs_encoded_read_regular_fill_pages
btrfs: add io_uring command for encoded reads
fs/btrfs/btrfs_inode.h | 13 +- fs/btrfs/file.c | 1 + fs/btrfs/inode.c | 175 ++++++++++++++------- fs/btrfs/ioctl.c | 342 ++++++++++++++++++++++++++++++++++++++++- fs/btrfs/ioctl.h | 2 + fs/btrfs/send.c | 3 +- 6 files changed, 473 insertions(+), 63 deletions(-)
Signature · 1 line
2.45.2
PATCH v4 1/5 btrfs: remove pointless addition in btrfs_encoded_read
iocb->ki_pos isn't used after this function, so there's no point in changing its value.
--- fs/btrfs/inode.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)
fs/btrfs/inode.c +1 -4
@@ -9252,7 +9252,7 @@ ssize_t btrfs_encoded_read(struct kiocb *iocb, struct iov_iter *iter, ret =btrfs_encoded_read_inline (iocb, iter, start, lockend, &cached_state, extent_start, count, encoded, &unlocked); - goto out; + goto out_em; }/*
@@ -9318,9 +9318,6 @@ ssize_t btrfs_encoded_read(struct kiocb *iocb, struct iov_iter *iter, &unlocked); } -out: -if (ret >=0 ) - iocb->ki_pos += encoded->len; out_em:free_extent_map (em); out_unlock_extent:
Signature · 1 line
2.45.2
Re: [PATCH 1/5] btrfs: remove pointless addition in btrfs_encoded_read
Quoted 13 lines · Mark Harmstone
iocb->ki_posisn't used after this function, so there's no point in changing its value.Signed-off-by Mark Harmstone <[email protected]>--- fs/btrfs/inode.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-)fs/btrfs/inode.c +0
-0
fs/btrfs/inode.c ssize_t btrfs_encoded_read(struct kiocb *iocb, struct iov_iter *iter, · line 9255
Quoted 5 lines
fs/btrfs/inode.c +1
-1@@ -9252,7 +9252,7 @@ ssize_t btrfs_encoded_read(struct kiocb *iocb, struct iov_iter *iter, ret =btrfs_encoded_read_inline (iocb, iter, start, lockend, &cached_state, extent_start, count, encoded, &unlocked); - goto out; + goto out_em;
Proceed to out_unlock_extent; free_extent_map() has already been called two lines above, and %em is now NULL.
Thanks, Anand
Quoted 13 lines
fs/btrfs/inode.c +0
-3}/* @@ -9318,9 +9318,6 @@ ssize_t btrfs_encoded_read(struct kiocb *iocb, struct iov_iter *iter, &unlocked); } -out: -if (ret >=0 ) - iocb->ki_pos += encoded->len; out_em:free_extent_map (em); out_unlock_extent:
Re: [PATCH 1/5] btrfs: remove pointless addition in btrfs_encoded_read
Quoted 25 lines · Anand Jain
Quoted 13 lines · Mark Harmstone
iocb->ki_posisn't used after this function, so there's no point in changing its value.Signed-off-by Mark Harmstone <[email protected]>---
fs/btrfs/inode.c| 5 +----1 file changed, 1 insertion(+), 4 deletions(-)fs/btrfs/inode.c +0
-0fs/btrfs/inode.c ssize_t btrfs_encoded_read(struct kiocb *iocb, · line 9256
Quoted 6 lines
fs/btrfs/inode.c +1
-1@@ -9252,7 +9252,7 @@ ssize_t btrfs_encoded_read(struct kiocb *iocb,struct iov_iter *iter, ret = btrfs_encoded_read_inline(iocb, iter, start, lockend, &cached_state, extent_start, count, encoded, &unlocked); - goto out; + goto out_em;Proceed to
out_unlock_extent;free_extent_map()has already been called two lines above, and %em is now NULL.
Quoted 2 lines
Thanks, Anand
Thanks Anand, I'll send a patch for this.
Mark
PATCH v4 2/5 btrfs: change btrfs_encoded_read so that reading of extent is done by caller
In reply to Mark HarmstoneChange the behaviour of btrfs_encoded_read so that if it needs to read an extent from disk, it leaves the extent and inode locked and returns -EIOCBQUEUED. The caller is then responsible for doing the I/O via btrfs_encoded_read_regular and unlocking the extent and inode.
--- fs/btrfs/btrfs_inode.h | 10 +++++++- fs/btrfs/inode.c | 58 ++++++++++++++++++++---------------------- fs/btrfs/ioctl.c | 33 +++++++++++++++++++++++- 3 files changed, 69 insertions(+), 32 deletions(-)
fs/btrfs/btrfs_inode.h +9 -1
@@ -615,7 +615,15 @@ int btrfs_encoded_read_regular_fill_pages(struct btrfs_inode *inode, u64 disk_bytenr, u64 disk_io_size, struct page **pages);ssize_t btrfs_encoded_read (struct kiocb *iocb,struct iov_iter *iter, -struct btrfs_ioctl_encoded_io_args *encoded); +struct btrfs_ioctl_encoded_io_args *encoded, +struct extent_state **cached_state, +u64 *disk_bytenr,u64 *disk_io_size); +ssize_t btrfs_encoded_read_regular (struct kiocb *iocb,struct iov_iter *iter, +u64 start,u64 lockend, +struct extent_state **cached_state, +u64 disk_bytenr,u64 disk_io_size, +size_t count,bool compressed, +bool *unlocked);ssize_t btrfs_do_encoded_write (struct kiocb *iocb,struct iov_iter *from,const struct btrfs_ioctl_encoded_io_args *encoded);
fs/btrfs/inode.c +28 -30
139 lines not on this page · raw message
fs/btrfs/ioctl.c +32 -1
@@ -4513,12 +4513,17 @@ static int btrfs_ioctl_encoded_read(struct file *file, void __user *argp, size_t copy_end_kernel = offsetofend(struct btrfs_ioctl_encoded_io_args, flags);size_t copy_end; +struct btrfs_inode *inode =BTRFS_I (file_inode (file)); +struct btrfs_fs_info *fs_info = inode->root->fs_info; +struct extent_io_tree *io_tree = &inode->io_tree;struct iovec iovstack[UIO_FASTIOV ];struct iovec *iov = iovstack;struct iov_iter iter;loff_t pos;struct kiocb kiocb;ssize_t ret; +u64 disk_bytenr, disk_io_size; +struct extent_state *cached_state =NULL ;if (!capable (CAP_SYS_ADMIN )) { ret = -EPERM ;
@@ -4571,7 +4576,33 @@ static int btrfs_ioctl_encoded_read(struct file *file, void __user *argp,init_sync_kiocb (&kiocb, file); kiocb.ki_pos = pos; - ret =btrfs_encoded_read (&kiocb, &iter, &args); + ret =btrfs_encoded_read (&kiocb, &iter, &args, &cached_state, + &disk_bytenr, &disk_io_size); + +if (ret == -EIOCBQUEUED ) { +bool unlocked = false; +u64 start, lockend, count; + + start =ALIGN_DOWN (kiocb.ki_pos, fs_info->sectorsize); + lockend = start +BTRFS_MAX_UNCOMPRESSED -1 ; + +if (args.compression) + count = disk_io_size; +else + count = args.len; + + ret =btrfs_encoded_read_regular (&kiocb, &iter, start, lockend, + &cached_state, disk_bytenr, + disk_io_size, count, + args.compression, + &unlocked); + +if (!unlocked) { +unlock_extent (io_tree, start, lockend, &cached_state); +btrfs_inode_unlock (inode,BTRFS_ILOCK_SHARED ); + } + } +if (ret >=0 ) {fsnotify_access (file); if (copy_to_user(argp + copy_end,
Signature · 1 line
2.45.2
PATCH v4 3/5 btrfs: don't sleep in btrfs_encoded_read if IOCB_NOWAIT set
In reply to Mark HarmstoneChange btrfs_encoded_read so that it returns -EAGAIN rather than sleeps if IOCB_NOWAIT is set in iocb->ki_flags.
--- fs/btrfs/inode.c | 54 ++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 43 insertions(+), 11 deletions(-)
fs/btrfs/inode.c +43 -11
@@ -8984,12 +8984,16 @@ static ssize_t btrfs_encoded_read_inline(unsigned long ptr;void *tmp;ssize_t ret; +bool nowait = iocb->ki_flags &IOCB_NOWAIT ; path =btrfs_alloc_path ();if (!path) { ret = -ENOMEM ; goto out; } + + path->nowait = !!nowait; + ret =btrfs_lookup_file_extent (NULL , root, path,btrfs_ino (inode), extent_start,0 );if (ret) {
@@ -9200,11 +9204,15 @@ ssize_t btrfs_encoded_read(struct kiocb *iocb, struct iov_iter *iter,size_t count =iov_iter_count (iter);u64 start, lockend;struct extent_map *em; +bool nowait = iocb->ki_flags &IOCB_NOWAIT ;bool unlocked = false;file_accessed (iocb->ki_filp); -btrfs_inode_lock (inode,BTRFS_ILOCK_SHARED ); + ret =btrfs_inode_lock (inode, +BTRFS_ILOCK_SHARED | (nowait ?BTRFS_ILOCK_TRY :0 )); +if (ret) +return ret;if (iocb->ki_pos >= inode->vfs_inode.i_size) {btrfs_inode_unlock (inode,BTRFS_ILOCK_SHARED );
@@ -9217,21 +9225,45 @@ ssize_t btrfs_encoded_read(struct kiocb *iocb, struct iov_iter *iter,*/ lockend = start +BTRFS_MAX_UNCOMPRESSED -1 ; -for (;;) { +if (nowait) {struct btrfs_ordered_extent *ordered; - ret =btrfs_wait_ordered_range (inode, start, - lockend - start +1 ); -if (ret) +if (filemap_range_needs_writeback (inode->vfs_inode.i_mapping, + start, lockend)) { + ret = -EAGAIN ; goto out_unlock_inode; -lock_extent (io_tree, start, lockend, cached_state); + } + +if (!try_lock_extent (io_tree, start, lockend, cached_state)) { + ret = -EAGAIN ; + goto out_unlock_inode; + } + ordered =btrfs_lookup_ordered_range (inode, start, lockend - start +1 ); -if (!ordered) -break ; -btrfs_put_ordered_extent (ordered); -unlock_extent (io_tree, start, lockend, cached_state); -cond_resched (); +if (ordered) { +btrfs_put_ordered_extent (ordered); +unlock_extent (io_tree, start, lockend, cached_state); + ret = -EAGAIN ; + goto out_unlock_inode; + } + }else { +for (;;) { +struct btrfs_ordered_extent *ordered; + + ret =btrfs_wait_ordered_range (inode, start, + lockend - start +1 ); +if (ret) + goto out_unlock_inode; +lock_extent (io_tree, start, lockend, cached_state); + ordered =btrfs_lookup_ordered_range (inode, start, + lockend - start +1 ); +if (!ordered) +break ; +btrfs_put_ordered_extent (ordered); +unlock_extent (io_tree, start, lockend, cached_state); +cond_resched (); + } } em =btrfs_get_extent (inode,NULL , start, lockend - start +1 );
Signature · 1 line
2.45.2
PATCH v4 4/5 btrfs: move priv off stack in btrfs_encoded_read_regular_fill_pages
In reply to Mark HarmstoneChange btrfs_encoded_read_regular_fill_pages so that the priv struct is allocated rather than stored on the stack, in preparation for adding an asynchronous mode to the function.
--- fs/btrfs/inode.c | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-)
fs/btrfs/inode.c +18 -11
@@ -9086,16 +9086,21 @@ int btrfs_encoded_read_regular_fill_pages(struct btrfs_inode *inode, struct page **pages) {struct btrfs_fs_info *fs_info = inode->root->fs_info; -struct btrfs_encoded_read_private priv = { - .pending =ATOMIC_INIT (1 ), - }; +struct btrfs_encoded_read_private *priv;unsigned long i =0 ;struct btrfs_bio *bbio; +int ret; -init_waitqueue_head (&priv.wait); + priv =kmalloc (sizeof (struct btrfs_encoded_read_private ),GFP_NOFS ); +if (!priv) +return -ENOMEM ; + +init_waitqueue_head (&priv->wait); +atomic_set (&priv->pending,1 ); + priv->status =0 ; bbio =btrfs_bio_alloc (BIO_MAX_VECS ,REQ_OP_READ , fs_info, - btrfs_encoded_read_endio, &priv); + btrfs_encoded_read_endio, priv); bbio->bio.bi_iter.bi_sector = disk_bytenr >>SECTOR_SHIFT ; bbio->inode = inode;
@@ -9103,11 +9108,11 @@ int btrfs_encoded_read_regular_fill_pages(struct btrfs_inode *inode,size_t bytes =min_t (u64, disk_io_size,PAGE_SIZE );if (bio_add_page (&bbio->bio, pages[i], bytes,0 ) < bytes) { -atomic_inc (&priv.pending); +atomic_inc (&priv->pending);btrfs_submit_bbio (bbio,0 ); bbio =btrfs_bio_alloc (BIO_MAX_VECS ,REQ_OP_READ , fs_info, - btrfs_encoded_read_endio, &priv); + btrfs_encoded_read_endio, priv); bbio->bio.bi_iter.bi_sector = disk_bytenr >>SECTOR_SHIFT ; bbio->inode = inode;continue ;
@@ -9118,13 +9123,15 @@ int btrfs_encoded_read_regular_fill_pages(struct btrfs_inode *inode, disk_io_size -= bytes; }while (disk_io_size); -atomic_inc (&priv.pending); +atomic_inc (&priv->pending);btrfs_submit_bbio (bbio,0 ); -if (atomic_dec_return (&priv.pending)) -io_wait_event (priv.wait, !atomic_read (&priv.pending)); +if (atomic_dec_return (&priv->pending)) +io_wait_event (priv->wait, !atomic_read (&priv->pending));/* See btrfs_encoded_read_endio() for ordering. */ -return blk_status_to_errno (READ_ONCE (priv.status)); + ret =blk_status_to_errno (READ_ONCE (priv->status)); +kfree (priv); +return ret; } ssize_t btrfs_encoded_read_regular(struct kiocb *iocb, struct iov_iter *iter,
Signature · 1 line
2.45.2
PATCH v4 5/5 btrfs: add io_uring command for encoded reads
In reply to Mark HarmstoneAdds an io_uring command for encoded reads, using the same interface as the existing BTRFS_IOC_ENCODED_READ ioctl.
btrfs_uring_encoded_read is an io_uring version of btrfs_ioctl_encoded_read, which validates the user input and calls btrfs_encoded_read to read the appropriate metadata. If we determine that we need to read an extent from disk, we call btrfs_encoded_read_regular_fill_pages through btrfs_uring_read_extent to prepare the bio.
The existing btrfs_encoded_read_regular_fill_pages is changed so that if it is passed a uring_ctx value, rather than waking up any waiting threads it calls btrfs_uring_read_extent_endio. This in turn copies the read data back to userspace, and calls io_uring_cmd_done to complete the io_uring command.
Because we're potentially doing a non-blocking read, btrfs_uring_read_extent doesn't clean up after itself if it returns -EIOCBQUEUED. Instead, it allocates a priv struct, populates the fields there that we will need to unlock the inode and free our allocations, and defers this to the btrfs_uring_read_finished that gets called when the bio completes.
--- fs/btrfs/btrfs_inode.h | 3 +- fs/btrfs/file.c | 1 + fs/btrfs/inode.c | 43 ++++-- fs/btrfs/ioctl.c | 309 +++++++++++++++++++++++++++++++++++++++++ fs/btrfs/ioctl.h | 2 + fs/btrfs/send.c | 3 +- 6 files changed, 349 insertions(+), 12 deletions(-)
fs/btrfs/btrfs_inode.h +2 -1
@@ -613,7 +613,8 @@ int btrfs_encoded_io_compression_from_extent(struct btrfs_fs_info *fs_info, int compress_type);int btrfs_encoded_read_regular_fill_pages (struct btrfs_inode *inode,u64 disk_bytenr,u64 disk_io_size, -struct page **pages); +struct page **pages, +void *uring_ctx); ssize_t btrfs_encoded_read(struct kiocb *iocb, struct iov_iter *iter, struct btrfs_ioctl_encoded_io_args *encoded, struct extent_state **cached_state,
fs/btrfs/file.c +1 -0
@@ -3710,6 +3710,7 @@ const struct file_operations btrfs_file_operations = { .compat_ioctl = btrfs_compat_ioctl, #endif .remap_file_range = btrfs_remap_file_range, + .uring_cmd = btrfs_uring_cmd, .fop_flags = FOP_BUFFER_RASYNC | FOP_BUFFER_WASYNC, };
fs/btrfs/inode.c +33 -10
78 lines not on this page · raw message
fs/btrfs/ioctl.c +309 -0
321 lines not on this page · raw message
fs/btrfs/ioctl.h +2 -0
@@ -22,5 +22,7 @@ void btrfs_sync_inode_flags_to_i_flags(struct inode *inode);int __pure btrfs_is_empty_uuid (const u8 *uuid);void btrfs_update_ioctl_balance_args (struct btrfs_fs_info *fs_info,struct btrfs_ioctl_balance_args *bargs); +int btrfs_uring_cmd (struct io_uring_cmd *cmd,unsigned int issue_flags); +void btrfs_uring_read_extent_endio (void *ctx,int err);#endif
fs/btrfs/send.c +2 -1
@@ -5669,7 +5669,8 @@ static int send_encoded_extent(struct send_ctx *sctx, struct btrfs_path *path, ret =btrfs_encoded_read_regular_fill_pages (BTRFS_I (inode), disk_bytenr, disk_num_bytes, sctx->send_buf_pages + - (data_offset >>PAGE_SHIFT )); + (data_offset >>PAGE_SHIFT ), +NULL );if (ret) goto out;
Signature · 1 line
2.45.2
Re: [PATCH 5/5] btrfs: add io_uring command for encoded reads
Quoted 16 lines · Mark Harmstone
fs/btrfs/ioctl.c +16
-0+static int btrfs_uring_encoded_read (struct io_uring_cmd *cmd, +unsigned int issue_flags) +{ + size_t copy_end_kernel = offsetofend(struct btrfs_ioctl_encoded_io_args, + flags); +size_t copy_end; +struct btrfs_ioctl_encoded_io_args args = {0 }; +int ret; +u64 disk_bytenr, disk_io_size; +struct file *file = cmd->file; +struct btrfs_inode *inode =BTRFS_I (file->f_inode); +struct btrfs_fs_info *fs_info = inode->root->fs_info; +struct extent_io_tree *io_tree = &inode->io_tree; +struct iovec iovstack[UIO_FASTIOV ]; +struct iovec *iov = iovstack; +struct iov_iter iter;
fs/btrfs/ioctl.c static int btrfs_ioctl_encoded_write(struct file *file, void __user *argp, bool · line 4901
Quoted 10 lines
fs/btrfs/ioctl.c +10
-0+loff_t pos; +struct kiocb kiocb; +struct extent_state *cached_state =NULL ; +u64 start, lockend; +void __user *sqe_addr =u64_to_user_ptr (READ_ONCE (cmd->sqe->addr)); + +if (!capable (CAP_SYS_ADMIN )) { + ret = -EPERM ; + goto out_acct; + }
Access level check must be done first before any data are read, in this case cmd->file and sqe_addr. Fixed.
Re: [PATCH 5/5] btrfs: add io_uring command for encoded reads
In reply to Mark HarmstoneOn 10/22/24 15:50, Mark Harmstone wrote:
...
Quoted 21 lines
fs/btrfs/ioctl.c +21
-0+static void btrfs_uring_read_finished (struct io_uring_cmd *cmd, +unsigned int issue_flags) +{ +struct btrfs_uring_priv *priv = + *io_uring_cmd_to_pdu(cmd, struct btrfs_uring_priv *); +struct btrfs_inode *inode =BTRFS_I (file_inode (priv->iocb.ki_filp)); +struct extent_io_tree *io_tree = &inode->io_tree; +unsigned long i; +u64 cur; +size_t page_offset; +ssize_t ret; + +if (priv->err) { + ret = priv->err; + goto out; + } + +if (priv->compressed) { + i =0 ; + page_offset =0 ; + }else {
fs/btrfs/ioctl.c static int btrfs_ioctl_encoded_write(struct file *file, void __user *argp, bool · line 4775
Quoted 10 lines
fs/btrfs/ioctl.c +10
-0+ i = (priv->iocb.ki_pos - priv->start) >>PAGE_SHIFT ; + page_offset =offset_in_page (priv->iocb.ki_pos - priv->start); + } + cur =0 ; +while (cur < priv->count) { +size_t bytes =min_t (size_t, priv->count - cur, +PAGE_SIZE - page_offset); + +if (copy_page_to_iter (priv->pages[i], page_offset, bytes, + &priv->iter) != bytes) {
If that's an iovec backed iter that might fail, you'd need to steal this patch
https://lore.kernel.org/all/[email protected]/
and fail if "issue_flags & IO_URING_F_TASK_DEAD", see
fs/btrfs/ioctl.c static int btrfs_ioctl_encoded_write(struct file *file, void __user *argp, bool · line 4788
Quoted 13 lines
fs/btrfs/ioctl.c +13
-0+ ret = -EFAULT ; + goto out; + } + + i++; + cur += bytes; + page_offset =0 ; + } + ret = priv->count; + +out: +unlock_extent (io_tree, priv->start, priv->lockend, &priv->cached_state); +btrfs_inode_unlock (inode,BTRFS_ILOCK_SHARED );
When called via io_uring_cmd_complete_in_task() this function might not get run in any reasonable amount of time. Even worse, a misbehaving user can block it until the task dies.
I don't remember if rwsem allows unlock being executed in a different task than the pairing lock, but blocking it for that long could be a problem. I might not remember it right but I think Boris meantioned that the O_DIRECT path drops the inode lock right after submission without waiting for bios to complete. Is that right? Can we do it here as well?
Quoted 9 lines
fs/btrfs/ioctl.c +9
-0+ +io_uring_cmd_done (cmd, ret,0 , issue_flags); +add_rchar (current, ret); + +for (unsigned long index =0 ; index < priv->nr_pages; index++) +__free_page (priv->pages[index]); + +kfree (priv->pages); +kfree (priv->iov);
fs/btrfs/ioctl.c static int btrfs_ioctl_encoded_write(struct file *file, void __user *argp, bool · line 4807
Quoted 10 lines
fs/btrfs/ioctl.c +10
-0+kfree (priv); +} + +void btrfs_uring_read_extent_endio (void *ctx,int err) +{ +struct btrfs_uring_priv *priv = ctx; + + priv->err = err; + + *io_uring_cmd_to_pdu(priv->cmd, struct btrfs_uring_priv *) = priv;
a nit, I'd suggest to create a temp var, should be easier to read. It'd even be nicer if you wrap it into a structure as suggested last time.
struct io_btrfs_cmd {
struct btrfs_uring_priv *priv;
};struct io_btrfs_cmd *bc = io_uring_cmd_to_pdu(cmd, struct io_btrfs_cmd); bc->priv = priv;
Quoted 42 lines
fs/btrfs/ioctl.c +42
-0+io_uring_cmd_complete_in_task (priv->cmd, btrfs_uring_read_finished); +} + +static int btrfs_uring_read_extent (struct kiocb *iocb,struct iov_iter *iter, +u64 start,u64 lockend, +struct extent_state *cached_state, +u64 disk_bytenr,u64 disk_io_size, +size_t count,bool compressed, +struct iovec *iov, +struct io_uring_cmd *cmd) +{ +struct btrfs_inode *inode =BTRFS_I (file_inode (iocb->ki_filp)); +struct extent_io_tree *io_tree = &inode->io_tree; +struct page **pages; +struct btrfs_uring_priv *priv =NULL ; +unsigned long nr_pages; +int ret; + + nr_pages =DIV_ROUND_UP (disk_io_size,PAGE_SIZE ); + pages =kcalloc (nr_pages,sizeof (struct page *),GFP_NOFS ); +if (!pages) +return -ENOMEM ; + ret =btrfs_alloc_page_array (nr_pages, pages,0 ); +if (ret) { + ret = -ENOMEM ; + goto fail; + } + + priv =kmalloc (sizeof (*priv),GFP_NOFS ); +if (!priv) { + ret = -ENOMEM ; + goto fail; + } + + priv->iocb = *iocb; + priv->iov = iov; + priv->iter = *iter; + priv->count = count; + priv->cmd = cmd; + priv->cached_state = cached_state; + priv->compressed = compressed; + priv->nr_pages = nr_pages;
fs/btrfs/ioctl.c static int btrfs_ioctl_encoded_write(struct file *file, void __user *argp, bool · line 4859
Quoted 10 lines
fs/btrfs/ioctl.c +10
-0+ priv->pages = pages; + priv->start = start; + priv->lockend = lockend; + priv->err =0 ; + + ret =btrfs_encoded_read_regular_fill_pages (inode, disk_bytenr, + disk_io_size, pages, + priv); +if (ret && ret != -EIOCBQUEUED ) + goto fail;
Turning both into return EIOCBQUEUED is a bit suspicious, but I lack context to say. Might make sense to return ret and let the caller handle it.
Quoted 27 lines
fs/btrfs/ioctl.c +27
-0+ +/* +* If we return -EIOCBQUEUED, we're deferring the cleanup to +* btrfs_uring_read_finished, which will handle unlocking the extent +* and inode and freeing the allocations. +*/ + +return -EIOCBQUEUED ; + +fail: +unlock_extent (io_tree, start, lockend, &cached_state); +btrfs_inode_unlock (inode,BTRFS_ILOCK_SHARED ); +kfree (priv); +return ret; +} + +static int btrfs_uring_encoded_read (struct io_uring_cmd *cmd, +unsigned int issue_flags) +{ + size_t copy_end_kernel = offsetofend(struct btrfs_ioctl_encoded_io_args, + flags); +size_t copy_end; +struct btrfs_ioctl_encoded_io_args args = {0 }; +int ret; +u64 disk_bytenr, disk_io_size; +struct file *file = cmd->file; +struct btrfs_inode *inode =BTRFS_I (file->f_inode);
fs/btrfs/ioctl.c static int btrfs_ioctl_encoded_write(struct file *file, void __user *argp, bool · line 4896
Quoted 10 lines
fs/btrfs/ioctl.c +10
-0+struct btrfs_fs_info *fs_info = inode->root->fs_info; +struct extent_io_tree *io_tree = &inode->io_tree; +struct iovec iovstack[UIO_FASTIOV ]; +struct iovec *iov = iovstack; +struct iov_iter iter; +loff_t pos; +struct kiocb kiocb; +struct extent_state *cached_state =NULL ; +u64 start, lockend; +void __user *sqe_addr =u64_to_user_ptr (READ_ONCE (cmd->sqe->addr));
Let's rename it, I was taken aback for a brief second why you're copy_from_user() from an SQE / the ring, which turns out to be a user pointer to a btrfs structure.
...
Quoted 4 lines
fs/btrfs/ioctl.c +4
-0+ ret =btrfs_encoded_read (&kiocb, &iter, &args, &cached_state, + &disk_bytenr, &disk_io_size); +if (ret <0 && ret != -EIOCBQUEUED ) + goto out_free;
fs/btrfs/ioctl.c static int btrfs_ioctl_encoded_write(struct file *file, void __user *argp, bool · line 4969
Quoted 10 lines
fs/btrfs/ioctl.c +10
-0+ +file_accessed (file); + +if (copy_to_user (sqe_addr + copy_end, (char *)&args + copy_end_kernel, +sizeof (args) - copy_end_kernel)) { +if (ret == -EIOCBQUEUED ) { +unlock_extent (io_tree, start, lockend, &cached_state); +btrfs_inode_unlock (inode,BTRFS_ILOCK_SHARED ); + }> + ret = -EFAULT; + goto out_free;
It seems we're saving iov in the priv structure, who can access the iovec after the request is submitted? -EIOCBQUEUED in general means that the request is submitted and will get completed async, e.g. via callback, and if the bio callback can use the iov maybe via the iter, this goto will be a use after free.
Also, you're returning -EFAULT back to io_uring, which will kill the io_uring request / cmd while there might still be in flight bios that can try to access it.
Can you inject errors into the copy and test please?
Quoted 47 lines
fs/btrfs/ioctl.c +47
-0+ } + +if (ret == -EIOCBQUEUED ) { +u64 count; + +/* +* If we've optimized things by storing the iovecs on the stack, +* undo this. + */> + if (!iov) { + iov =kmalloc (sizeof (struct iovec ) * args.iovcnt, +GFP_NOFS ); +if (!iov) { +unlock_extent (io_tree, start, lockend, + &cached_state); +btrfs_inode_unlock (inode,BTRFS_ILOCK_SHARED ); + ret = -ENOMEM ; + goto out_acct; + } + +memcpy (iov, iovstack, +sizeof (struct iovec ) * args.iovcnt); + } + + count =min_t (u64,iov_iter_count (&iter), disk_io_size); + +/* Match ioctl by not returning past EOF if uncompressed */ +if (!args.compression) + count =min_t (u64, count, args.len); + + ret =btrfs_uring_read_extent (&kiocb, &iter, start, lockend, + cached_state, disk_bytenr, + disk_io_size, count, + args.compression, iov, cmd); + + goto out_acct; + } + +out_free: +kfree (iov); + +out_acct: +if (ret >0 ) +add_rchar (current, ret); +inc_syscr (current); + +return ret; +}
Signature · 1 line
Pavel Begunkov
Re: [PATCH 5/5] btrfs: add io_uring command for encoded reads
Quoted 221 lines · Pavel Begunkov
On 10/22/24 15:50, Mark Harmstone wrote:
...Quoted 21 lines
fs/btrfs/ioctl.c +21
-0+static void btrfs_uring_read_finished (struct io_uring_cmd *cmd, +unsigned int issue_flags) +{ +struct btrfs_uring_priv *priv = + *io_uring_cmd_to_pdu(cmd, struct btrfs_uring_priv *); +struct btrfs_inode *inode =BTRFS_I (file_inode (priv->iocb.ki_filp)); +struct extent_io_tree *io_tree = &inode->io_tree; +unsigned long i; +u64 cur; +size_t page_offset; +ssize_t ret; + +if (priv->err) { + ret = priv->err; + goto out; + } + +if (priv->compressed) { + i =0 ; + page_offset =0 ; + }else {fs/btrfs/ioctl.c static int btrfs_ioctl_encoded_write(struct file *file, void __user *argp, bool · line 4775
Quoted 10 lines
fs/btrfs/ioctl.c +10
-0+ i = (priv->iocb.ki_pos - priv->start) >>PAGE_SHIFT ; + page_offset =offset_in_page (priv->iocb.ki_pos - priv->start); + } + cur =0 ; +while (cur < priv->count) { +size_t bytes =min_t (size_t, priv->count - cur, +PAGE_SIZE - page_offset); + +if (copy_page_to_iter (priv->pages[i], page_offset, bytes, + &priv->iter) != bytes) {If that's an iovec backed iter that might fail, you'd need to steal this patch
https://lore.kernel.org/all/[email protected]/
and fail if "
issue_flags&IO_URING_F_TASK_DEAD", seefs/btrfs/ioctl.c static int btrfs_ioctl_encoded_write(struct file *file, void __user *argp, bool · line 4788
Quoted 13 lines
fs/btrfs/ioctl.c +13
-0+ ret = -EFAULT ; + goto out; + } + + i++; + cur += bytes; + page_offset =0 ; + } + ret = priv->count; + +out: +unlock_extent (io_tree, priv->start, priv->lockend, &priv->cached_state); +btrfs_inode_unlock (inode,BTRFS_ILOCK_SHARED );When called via
io_uring_cmd_complete_in_task()this function might not get run in any reasonable amount of time. Even worse, a misbehaving user can block it until the task dies.I don't remember if rwsem allows unlock being executed in a different task than the pairing lock, but blocking it for that long could be a problem. I might not remember it right but I think Boris meantioned that the
O_DIRECTpath drops the inode lock right after submission without waiting for bios to complete. Is that right? Can we do it here as well?Quoted 9 lines
fs/btrfs/ioctl.c +9
-0+ +io_uring_cmd_done (cmd, ret,0 , issue_flags); +add_rchar (current, ret); + +for (unsigned long index =0 ; index < priv->nr_pages; index++) +__free_page (priv->pages[index]); + +kfree (priv->pages); +kfree (priv->iov);fs/btrfs/ioctl.c static int btrfs_ioctl_encoded_write(struct file *file, void __user *argp, bool · line 4807
Quoted 10 lines
fs/btrfs/ioctl.c +10
-0+kfree (priv); +} + +void btrfs_uring_read_extent_endio (void *ctx,int err) +{ +struct btrfs_uring_priv *priv = ctx; + + priv->err = err; + + *io_uring_cmd_to_pdu(priv->cmd, struct btrfs_uring_priv *) = priv;a nit, I'd suggest to create a temp var, should be easier to read. It'd even be nicer if you wrap it into a structure as suggested last time.
struct io_btrfs_cmd { struct btrfs_uring_priv *priv; };struct
io_btrfs_cmd*bc =io_uring_cmd_to_pdu(cmd, struct io_btrfs_cmd);bc->priv= priv;Quoted 42 lines
fs/btrfs/ioctl.c +42
-0+io_uring_cmd_complete_in_task (priv->cmd, btrfs_uring_read_finished); +} + +static int btrfs_uring_read_extent (struct kiocb *iocb,struct iov_iter *iter, +u64 start,u64 lockend, +struct extent_state *cached_state, +u64 disk_bytenr,u64 disk_io_size, +size_t count,bool compressed, +struct iovec *iov, +struct io_uring_cmd *cmd) +{ +struct btrfs_inode *inode =BTRFS_I (file_inode (iocb->ki_filp)); +struct extent_io_tree *io_tree = &inode->io_tree; +struct page **pages; +struct btrfs_uring_priv *priv =NULL ; +unsigned long nr_pages; +int ret; + + nr_pages =DIV_ROUND_UP (disk_io_size,PAGE_SIZE ); + pages =kcalloc (nr_pages,sizeof (struct page *),GFP_NOFS ); +if (!pages) +return -ENOMEM ; + ret =btrfs_alloc_page_array (nr_pages, pages,0 ); +if (ret) { + ret = -ENOMEM ; + goto fail; + } + + priv =kmalloc (sizeof (*priv),GFP_NOFS ); +if (!priv) { + ret = -ENOMEM ; + goto fail; + } + + priv->iocb = *iocb; + priv->iov = iov; + priv->iter = *iter; + priv->count = count; + priv->cmd = cmd; + priv->cached_state = cached_state; + priv->compressed = compressed; + priv->nr_pages = nr_pages;fs/btrfs/ioctl.c static int btrfs_ioctl_encoded_write(struct file *file, void __user *argp, bool · line 4859
Quoted 10 lines
fs/btrfs/ioctl.c +10
-0+ priv->pages = pages; + priv->start = start; + priv->lockend = lockend; + priv->err =0 ; + + ret =btrfs_encoded_read_regular_fill_pages (inode, disk_bytenr, + disk_io_size, pages, + priv); +if (ret && ret != -EIOCBQUEUED ) + goto fail;Turning both into return EIOCBQUEUED is a bit suspicious, but I lack context to say. Might make sense to return ret and let the caller handle it.
Quoted 27 lines
fs/btrfs/ioctl.c +27
-0+ +/* +* If we return -EIOCBQUEUED, we're deferring the cleanup to +* btrfs_uring_read_finished, which will handle unlocking the extent +* and inode and freeing the allocations. +*/ + +return -EIOCBQUEUED ; + +fail: +unlock_extent (io_tree, start, lockend, &cached_state); +btrfs_inode_unlock (inode,BTRFS_ILOCK_SHARED ); +kfree (priv); +return ret; +} + +static int btrfs_uring_encoded_read (struct io_uring_cmd *cmd, +unsigned int issue_flags) +{ + size_t copy_end_kernel = offsetofend(struct btrfs_ioctl_encoded_io_args, + flags); +size_t copy_end; +struct btrfs_ioctl_encoded_io_args args = {0 }; +int ret; +u64 disk_bytenr, disk_io_size; +struct file *file = cmd->file; +struct btrfs_inode *inode =BTRFS_I (file->f_inode);fs/btrfs/ioctl.c static int btrfs_ioctl_encoded_write(struct file *file, void __user *argp, bool · line 4896
Quoted 10 lines
fs/btrfs/ioctl.c +10
-0+struct btrfs_fs_info *fs_info = inode->root->fs_info; +struct extent_io_tree *io_tree = &inode->io_tree; +struct iovec iovstack[UIO_FASTIOV ]; +struct iovec *iov = iovstack; +struct iov_iter iter; +loff_t pos; +struct kiocb kiocb; +struct extent_state *cached_state =NULL ; +u64 start, lockend; +void __user *sqe_addr =u64_to_user_ptr (READ_ONCE (cmd->sqe->addr));Let's rename it, I was taken aback for a brief second why you're
copy_from_user()from an SQE / the ring, which turns out to be a user pointer to a btrfs structure....
Quoted 4 lines
fs/btrfs/ioctl.c +4
-0+ ret =btrfs_encoded_read (&kiocb, &iter, &args, &cached_state, + &disk_bytenr, &disk_io_size); +if (ret <0 && ret != -EIOCBQUEUED ) + goto out_free;fs/btrfs/ioctl.c static int btrfs_ioctl_encoded_write(struct file *file, void __user *argp, bool · line 4969
Quoted 10 lines
fs/btrfs/ioctl.c +10
-0+ +file_accessed (file); + +if (copy_to_user (sqe_addr + copy_end, (char *)&args + copy_end_kernel, +sizeof (args) - copy_end_kernel)) { +if (ret == -EIOCBQUEUED ) { +unlock_extent (io_tree, start, lockend, &cached_state); +btrfs_inode_unlock (inode,BTRFS_ILOCK_SHARED ); + }> + ret = -EFAULT; + goto out_free;It seems we're saving iov in the priv structure, who can access the iovec after the request is submitted?
-EIOCBQUEUEDin general means that the request is submitted and will get completed async, e.g. via callback, and if the bio callback can use the iov maybe via the iter, this goto will be a use after free.
Quoted 5 lines
Also, you're returning
-EFAULTback toio_uring, which will kill theio_uringrequest / cmd while there might still be in flight bios that can try to access it.Can you inject errors into the copy and test please?
Thanks for the comments. I get the impression that there are known problems on the io_uring side, so until that is resolved the btrfs part may be insecure or with known runtime bugs, but in the end it does not need any change. We just need to wait until it's resoved on the interface level.
The patches you point to are from FUSE trying to wire up io_uring so this looks like an interface problem. We recently have gained a config option level gurard for experimental and unstable features so we can add the code but don't have to expose users to the functionality unless they konw there are risks or known problems. The io_uring and encoded read has a performance benefit and I'd like to get the patches in for 6.13 but if there's something serious, one option is not add the code or at least guard it (behind a config option).
I'm open to both and we have at least one -rc kernel to decide.
Re: [PATCH 5/5] btrfs: add io_uring command for encoded reads
Quoted 2 lines · David Sterba
Quoted 1 line · Pavel Begunkov
On 10/22/24 15:50, Mark Harmstone wrote:
...
Quoted 10 lines
Quoted 10 lines
It seems we're saving iov in the priv structure, who can access the iovec after the request is submitted?
-EIOCBQUEUEDin general means that the request is submitted and will get completed async, e.g. via callback, and if the bio callback can use the iov maybe via the iter, this goto will be a use after free.Also, you're returning
-EFAULTback toio_uring, which will kill theio_uringrequest / cmd while there might still be in flight bios that can try to access it.
Quoted 7 lines
Quoted 1 line
Can you inject errors into the copy and test please?
Thanks for the comments. I get the impression that there are known problems on the
io_uringside, so until that is resolved the btrfs part may be insecure or with known runtime bugs, but in the end it does not need any change. We just need to wait until it's resoved on the interface level.
There is nothing wrong with io_uring, it's jumping from synchronous to asynchronous that concerns me, or more specifically how this series handles it and all races. Basic stuff like not freeing / changing without protection memory that the async part might still be using. That's up to this series to do it right.
Quoted 2 lines
The patches you point to are from FUSE trying to wire up
io_uringso this looks like an interface problem. We recently have gained a config
That's the easiest part of all, it can only happen when the task dies and mm becomes unavaliable, sane userspace shouldn't have problems like that. Mark just needs to include the referred patch into the series and handle the request as mentioned.
Quoted 4 lines
option level gurard for experimental and unstable features so we can add the code but don't have to expose users to the functionality unless they konw there are risks or known problems. The
io_uringand encoded read has a performance benefit and
Good to hear that
Quoted 3 lines
I'd like to get the patches in for 6.13
but if there's something serious, one option is not add the code or at least guard it (behind a config option).
Let's see what Mark replies, I might be missing some things, and you and other btrfs folks can help to answer the unlock question.
Quoted 1 line
I'm open to both and we have at least one -rc kernel to decide.
Signature · 1 line
Pavel Begunkov
Re: [PATCH 5/5] btrfs: add io_uring command for encoded reads
In reply to Pavel BegunkovThanks Pavel.
Quoted 38 lines · Pavel Begunkov
Quoted 1 line
On 10/22/24 15:50, Mark Harmstone wrote:
...Quoted 21 lines
fs/btrfs/ioctl.c +21
-0+static void btrfs_uring_read_finished (struct io_uring_cmd *cmd, + unsigned int issue_flags) +{ + struct btrfs_uring_priv *priv = + *io_uring_cmd_to_pdu(cmd, struct btrfs_uring_priv *); + struct btrfs_inode *inode = BTRFS_I(file_inode(priv->iocb.ki_filp)); + struct extent_io_tree *io_tree = &inode->io_tree; + unsigned long i; + u64 cur; + size_t page_offset; + ssize_t ret; + + if (priv->err) { + ret = priv->err; + goto out; + } + + if (priv->compressed) { + i = 0; + page_offset = 0; + } else {fs/btrfs/ioctl.c static int btrfs_ioctl_encoded_write(struct file *file, void __user *argp, bool · line 4775
Quoted 10 lines
fs/btrfs/ioctl.c +10
-0+ i = (priv->iocb.ki_pos - priv->start) >> PAGE_SHIFT; + page_offset = offset_in_page(priv->iocb.ki_pos - priv->start); + } + cur = 0; + while (cur < priv->count) { + size_t bytes = min_t(size_t, priv->count - cur, + PAGE_SIZE - page_offset); + + if (copy_page_to_iter(priv->pages[i], page_offset, bytes, + &priv->iter) != bytes) {If that's an iovec backed iter that might fail, you'd need to steal this patch
Quoted 5 lines
https://lore.kernel.org/all/[email protected]/
and fail if "
issue_flags&IO_URING_F_TASK_DEAD", see
Thanks, I've sent a patchset including your patch. Does it make a difference, though? If the task has died, presumably copy_page_to_iter can't copy to another process' memory...?
Quoted 19 lines
Quoted 12 lines
+ ret =
-EFAULT;
+ goto out;
+ }fs/btrfs/ioctl.c +9
-0+ + i++; + cur += bytes; + page_offset = 0; + } + ret = priv->count; + +out: + unlock_extent(io_tree, priv->start, priv->lockend,Quoted 2 lines
&priv->cached_state);
+btrfs_inode_unlock(inode, BTRFS_ILOCK_SHARED);When called via
io_uring_cmd_complete_in_task()this function might not get run in any reasonable amount of time. Even worse, a misbehaving user can block it until the task dies.
Quoted 6 lines
I don't remember if rwsem allows unlock being executed in a different task than the pairing lock, but blocking it for that long could be a problem. I might not remember it right but I think Boris meantioned that the
O_DIRECTpath drops the inode lock right after submission without waiting for bios to complete. Is that right? Can we do it here as well?
We can't release the inode lock until we've released the extent lock. I do intend to look into reducing the amount of time we hold the extent lock, if we can, but it's not trivial to do this in a safe manner. We could perhaps move the unlocking to btrfs_uring_read_extent_endio instead, but it looks unlocking an rwsem in a different context might cause problems with PREEMPT_RT(?).
Quoted 24 lines
Quoted 9 lines
fs/btrfs/ioctl.c +9
-0+ + io_uring_cmd_done(cmd, ret, 0, issue_flags); + add_rchar(current, ret); + + for (unsigned long index = 0; index < priv->nr_pages; index++) + __free_page(priv->pages[index]); + + kfree(priv->pages); + kfree(priv->iov);fs/btrfs/ioctl.c static int btrfs_ioctl_encoded_write(struct file *file, void __user *argp, bool · line 4807
Quoted 10 lines
fs/btrfs/ioctl.c +10
-0+ kfree(priv); +} + +void btrfs_uring_read_extent_endio (void *ctx,int err) +{ + struct btrfs_uring_priv *priv = ctx; + + priv->err = err; + + *io_uring_cmd_to_pdu(priv->cmd, struct btrfs_uring_priv *) = priv;a nit, I'd suggest to create a temp var, should be easier to read. It'd even be nicer if you wrap it into a structure as suggested last time.
Quoted 6 lines
struct
io_btrfs_cmd{
structbtrfs_uring_priv*priv;
};struct
io_btrfs_cmd*bc =io_uring_cmd_to_pdu(cmd, struct io_btrfs_cmd);bc->priv= priv;
No problem, I've sent a patch for this.
Quoted 49 lines
Quoted 49 lines
+
io_uring_cmd_complete_in_task(priv->cmd, btrfs_uring_read_finished);fs/btrfs/inode.c +3
-0+} + +static int btrfs_uring_read_extent(struct kiocb *iocb, struct
iov_iter*iter,
+ u64 start, u64 lockend,
+ structextent_state*cached_state, + u64disk_bytenr, u64disk_io_size, +size_tcount, bool compressed, + struct iovec *iov,
+ structio_uring_cmd*cmd)fs/btrfs/ioctl.c +38
-0+{ + struct btrfs_inode *inode = BTRFS_I(file_inode(iocb->ki_filp)); + struct extent_io_tree *io_tree = &inode->io_tree; + struct page **pages; + struct btrfs_uring_priv *priv = NULL; + unsigned long nr_pages; + int ret; + + nr_pages = DIV_ROUND_UP(disk_io_size, PAGE_SIZE); + pages = kcalloc(nr_pages, sizeof(struct page *), GFP_NOFS); + if (!pages) + return -ENOMEM; + ret = btrfs_alloc_page_array(nr_pages, pages, 0); + if (ret) { + ret = -ENOMEM; + goto fail; + } + + priv = kmalloc(sizeof(*priv), GFP_NOFS); + if (!priv) { + ret = -ENOMEM; + goto fail; + } + + priv->iocb = *iocb; + priv->iov = iov; + priv->iter = *iter; + priv->count = count; + priv->cmd = cmd; + priv->cached_state = cached_state; + priv->compressed = compressed; + priv->nr_pages = nr_pages; + priv->pages = pages; + priv->start = start; + priv->lockend = lockend; + priv->err = 0; + + ret = btrfs_encoded_read_regular_fill_pages(inode, disk_bytenr,
Quoted 8 lines
Quoted 4 lines
+
disk_io_size, pages,
+ priv);
+ if (ret && ret !=-EIOCBQUEUED)
+ goto fail;Turning both into return EIOCBQUEUED is a bit suspicious, but I lack context to say. Might make sense to return ret and let the caller handle it.
btrfs_encoded_read_regular_fill_pages returns 0 if the bio completes before the function can finish, -EIOCBQUEUED otherwise. In either case the behaviour of the calling function will be the same.
Quoted 34 lines
Quoted 34 lines
fs/btrfs/ioctl.c +20
-0+ + /* + * If we return -EIOCBQUEUED, we're deferring the cleanup to + * btrfs_uring_read_finished, which will handle unlocking the extent + * and inode and freeing the allocations. + */ + + return -EIOCBQUEUED; + +fail: + unlock_extent(io_tree, start, lockend, &cached_state); + btrfs_inode_unlock(inode, BTRFS_ILOCK_SHARED); + kfree(priv); + return ret; +} + +static int btrfs_uring_encoded_read (struct io_uring_cmd *cmd, + unsigned int issue_flags) +{ + size_t copy_end_kernel = offsetofend(struct
btrfs_ioctl_encoded_io_args,
+ flags);
+size_tcopy_end;
+ structbtrfs_ioctl_encoded_io_argsargs = { 0 }; + int ret;
+ u64disk_bytenr,disk_io_size;
+ struct file *file =cmd->file;
+ structbtrfs_inode*inode =BTRFS_I(file->f_inode); + structbtrfs_fs_info*fs_info=inode->root->fs_info; + structextent_io_tree*io_tree=&inode->io_tree; + struct iovec iovstack[UIO_FASTIOV];
+ struct iovec *iov = iovstack;
+ structiov_iteriter;
+loff_tpos;
Quoted 8 lines
Quoted 4 lines
+ struct kiocb kiocb;
+ structextent_state*cached_state= NULL;
+ u64 start, lockend;
+ void__user*sqe_addr= u64_to_user_ptr(READ_ONCE(cmd->sqe->addr));Let's rename it, I was taken aback for a brief second why you're
copy_from_user()from an SQE / the ring, which turns out to be a user pointer to a btrfs structure.
sqe_addr being the addr field in the SQE, not the address of the SQE. I can see how it might be misleading, though.
Quoted 24 lines
...
Quoted 8 lines
+ ret =
btrfs_encoded_read(&kiocb, &iter, &args, &cached_state, + &disk_bytenr, &disk_io_size);
+ if (ret < 0 && ret !=-EIOCBQUEUED)
+ gotoout_free;fs/btrfs/ioctl.c +4
-0+ + file_accessed(file); + + if (copy_to_user(sqe_addr + copy_end, (char *)&args +Quoted 8 lines
copy_end_kernel,
+ sizeof(args) -copy_end_kernel)) {
+ if (ret ==-EIOCBQUEUED) {
+unlock_extent(io_tree, start, lockend, &cached_state); +btrfs_inode_unlock(inode, BTRFS_ILOCK_SHARED); + }
+ ret =-EFAULT;
+ gotoout_free;It seems we're saving iov in the priv structure, who can access the iovec after the request is submitted?
-EIOCBQUEUEDin general means that the request is submitted and will get completed async, e.g. via callback, and if the bio callback can use the iov maybe via the iter, this goto will be a use after free.
Quoted 5 lines
Also, you're returning
-EFAULTback toio_uring, which will kill theio_uringrequest / cmd while there might still be in flight bios that can try to access it.Can you inject errors into the copy and test please?
The bio hasn't been submitted at this point, that happens in btrfs_uring_read_extent. So far all we've done is read the metadata from the page cache. The copy_to_user here is copying the metadata info to the userspace structure.
Quoted 49 lines
Quoted 48 lines
+ }
fs/btrfs/ioctl.c +46
-0+ + if (ret == -EIOCBQUEUED) { + u64 count; + + /* + * If we've optimized things by storing the iovecs on the stack, + * undo this. + */> + if (!iov) { + iov = kmalloc(sizeof(struct iovec) * args.iovcnt, + GFP_NOFS); + if (!iov) { + unlock_extent(io_tree, start, lockend, + &cached_state); + btrfs_inode_unlock(inode, BTRFS_ILOCK_SHARED); + ret = -ENOMEM; + goto out_acct; + } + + memcpy(iov, iovstack, + sizeof(struct iovec) * args.iovcnt); + } + + count = min_t(u64, iov_iter_count(&iter), disk_io_size); + + /* Match ioctl by not returning past EOF if uncompressed */ + if (!args.compression) + count = min_t(u64, count, args.len); + + ret = btrfs_uring_read_extent(&kiocb, &iter, start, lockend, + cached_state, disk_bytenr, + disk_io_size, count, + args.compression, iov, cmd); + + goto out_acct; + } + +out_free: + kfree(iov); + +out_acct: + if (ret > 0) + add_rchar(current, ret); + inc_syscr(current); + + return ret; +}
Re: [PATCH 5/5] btrfs: add io_uring command for encoded reads
Quoted 2 lines · Mark Harmstone
Thanks Pavel.
...
Quoted 5 lines
Quoted 5 lines
If that's an iovec backed iter that might fail, you'd need to steal this patch
Quoted 7 lines
Quoted 3 lines
and fail if "
issue_flags&IO_URING_F_TASK_DEAD", seeThanks, I've sent a patchset including your patch. Does it make a difference, though? If the task has died, presumably
copy_page_to_itercan't copy to another process' memory...?
IIRC copy_to_user will crash without mm set, not sure about copy_page_to_iter(). Regardless, when the original task has dies and it gets run from io_uring's fallback path, you shouldn't make any assumptions about the current task.
Quoted 26 lines
Quoted 19 lines
Quoted 12 lines
+ ret =
-EFAULT;
+ goto out;
+ }fs/btrfs/ioctl.c +9
-0+ + i++; + cur += bytes; + page_offset = 0; + } + ret = priv->count; + +out: + unlock_extent(io_tree, priv->start, priv->lockend,Quoted 2 lines
&priv->cached_state);
+btrfs_inode_unlock(inode, BTRFS_ILOCK_SHARED);When called via
io_uring_cmd_complete_in_task()this function might not get run in any reasonable amount of time. Even worse, a misbehaving user can block it until the task dies.Quoted 6 lines
I don't remember if rwsem allows unlock being executed in a different task than the pairing lock, but blocking it for that long could be a problem. I might not remember it right but I think Boris meantioned that the
O_DIRECTpath drops the inode lock right after submission without waiting for bios to complete. Is that right? Can we do it here as well?
Quoted 3 lines
We can't release the inode lock until we've released the extent lock. I do intend to look into reducing the amount of time we hold the extent lock, if we can, but it's not trivial to do this in a safe manner.
I lack the btrfs knowledge, but sounds like it can be done the same way the async dio path works.
Quoted 3 lines
We could perhaps move the unlocking to
btrfs_uring_read_extent_endioinstead, but it looks unlocking an rwsem in a different context might cause problems withPREEMPT_RT(?).
At least from a quick glance it doesn't seem that locks in __clear_extent_bit are [soft]irq protected. Would be a good idea to give it a run with lockdep enabled.
...
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
+ ret = btrfs_encoded_read_regular_fill_pages(inode,
disk_bytenr, +disk_io_size, pages,
+ priv);
+ if (ret && ret !=-EIOCBQUEUED)
+ goto fail;
Quoted 7 lines
Quoted 3 lines
Turning both into return EIOCBQUEUED is a bit suspicious, but I lack context to say. Might make sense to return ret and let the caller handle it.
btrfs_encoded_read_regular_fill_pagesreturns 0 if the bio completes before the function can finish,-EIOCBQUEUEDotherwise. In either case the behaviour of the calling function will be the same.
Ok
...
Quoted 20 lines
Quoted 20 lines
Quoted 9 lines
+ if (copy_to_user(sqe_addr +
copy_end, (char *)&args +copy_end_kernel,
+ sizeof(args) -copy_end_kernel)) {
+ if (ret ==-EIOCBQUEUED) {
+unlock_extent(io_tree, start, lockend, &cached_state); +btrfs_inode_unlock(inode, BTRFS_ILOCK_SHARED); + }
+ ret =-EFAULT;
+ gotoout_free;It seems we're saving iov in the priv structure, who can access the iovec after the request is submitted?
-EIOCBQUEUEDin general means that the request is submitted and will get completed async, e.g. via callback, and if the bio callback can use the iov maybe via the iter, this goto will be a use after free.Also, you're returning
-EFAULTback toio_uring, which will kill theio_uringrequest / cmd while there might still be in flight bios that can try to access it.
Quoted 6 lines
Quoted 1 line
Can you inject errors into the copy and test please?
The bio hasn't been submitted at this point, that happens in
btrfs_uring_read_extent. So far all we've done is read the metadata from the page cache. Thecopy_to_userhere is copying the metadata info to the userspace structure.
I see, in this case it should be fine, but why is it -EIOCBQUEUED then? It always meant that it queued up the request and will complete it asynchronously, and that's where the confusion sprouted from. Not looking deeper but sounds more like -EAGAIN? Assuming it's returned because we can't block
Quoted 13 lines
Quoted 13 lines
Quoted 13 lines
+ }
fs/btrfs/ioctl.c +12
-0+ + if (ret == -EIOCBQUEUED) { + u64 count; + + /* + * If we've optimized things by storing the iovecs on the stack, + * undo this. + */> + if (!iov) { + iov = kmalloc(sizeof(struct iovec) * args.iovcnt, + GFP_NOFS); + if (!iov) { + unlock_extent(io_tree, start, lockend,
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
+
&cached_state);
+btrfs_inode_unlock(inode, BTRFS_ILOCK_SHARED); + ret =-ENOMEM;
+ gotoout_acct;
+ }
+
+ memcpy(iov, iovstack,
+ sizeof(struct iovec) * args.iovcnt);
As an optimisation in the future you can allocate it together with the btrfs_priv structure.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
+ }
fs/btrfs/ioctl.c +3
-0+ + count = min_t(u64, iov_iter_count(&iter), disk_io_size); +
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
+ /* Match ioctl by not returning past EOF if uncompressed */ + if (!args.compression)
+ count =min_t(u64, count, args.len);
+
+ ret = btrfs_uring_read_extent(&kiocb, &iter, start, lockend, +cached_state,disk_bytenr,
+disk_io_size, count,
+ args.compression, iov, cmd);
So that's the only spot where asynchronous code branches off in this function? Do I read you right?
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines
fs/btrfs/ioctl.c +14
-0+ + goto out_acct; + } + +out_free: + kfree(iov); + +out_acct: + if (ret > 0) + add_rchar(current, ret); + inc_syscr(current); + + return ret; +}
Signature · 1 line
Pavel Begunkov
Quoted 7 lines · Mark Harmstone
This is version 4 of a patch series to add an
io_uringinterface for encoded reads. The principal use case for this is to eventually allow btrfs send and receive to operate asynchronously, the lack ofio_uringencoded I/O being one of the main blockers for this.I've written a test program for this, which demonstrates the ioctl and
io_uringinterface produce identical results: https://github.com/maharmstone/io_uring-encoded
We'll need a test utility for fstests too.
Quoted 5 lines
Changelog:
v4:
* Rewritten to avoid taking function pointer
* Removed nowait parameter, as this could be derived from iocb flags
* Fixed structure not getting properly initialized
Quoted 8 lines
* Followed ioctl by capping uncompressed reads at EOF
* Rebased against btrfs/for-next
* Formatting fixes
* Rearranged structs to minimize holes
* Published test program
* Fixed potential data race with userspace
* Changed to useio_uring_cmd_to_pduhelper function
* Added comments for potentially confusing parts of the code
There are some more style issues and changelog updates but overall looks ok to me. We're now in rc5 so we need to add it to for-next now or postpone for next cycle (but I don't see a reason why).
I've noticed CONFIG_IO_URING is a config option, do we need some ifdef protection or are the definitions provided unconditionally. We may also need to ifdef out the ioctl code if io_uring is not built in.
Thanks David.
Quoted 10 lines · David Sterba
Quoted 7 lines · Mark Harmstone
This is version 4 of a patch series to add an
io_uringinterface for encoded reads. The principal use case for this is to eventually allow btrfs send and receive to operate asynchronously, the lack ofio_uringencoded I/O being one of the main blockers for this.I've written a test program for this, which demonstrates the ioctl and
io_uringinterface produce identical results: https://github.com/maharmstone/io_uring-encodedWe'll need a test utility for fstests too.
Yes, no problem.
Quoted 14 lines
Quoted 5 lines
Changelog:
v4:
* Rewritten to avoid taking function pointer
* Removed nowait parameter, as this could be derived from iocb flags
* Fixed structure not getting properly initializedQuoted 8 lines
* Followed ioctl by capping uncompressed reads at EOF
* Rebased against btrfs/for-next
* Formatting fixes
* Rearranged structs to minimize holes
* Published test program
* Fixed potential data race with userspace
* Changed to useio_uring_cmd_to_pduhelper function
* Added comments for potentially confusing parts of the code
Quoted 7 lines
There are some more style issues and changelog updates but overall looks ok to me. We're now in rc5 so we need to add it to for-next now or postpone for next cycle (but I don't see a reason why).
I've noticed
CONFIG_IO_URINGis a config option, do we need some ifdef protection or are the definitions provided unconditionally. We may also need to ifdef out the ioctl code ifio_uringis not built in.
It compiles fine with CONFIG_IO_URING=n, the uring_cmd field in struct file_operations is there unconditionally. FWIW, the other users of uring_cmd don't ifdef their bits out, presumably because it's also gated behind CONFIG_EXPERT.
Mark
Quoted 4 lines · Mark Harmstone
This is version 4 of a patch series to add an
io_uringinterface for encoded reads. The principal use case for this is to eventually allowbtrfs send and receive to operate asynchronously,
How would you define an asynchronously operated Btrfs send and receive? Are you referring to Btrfs send and receive the leveraging io_uring asynchronous operation?
Quoted 6 lines
the lack of
io_uring
encoded I/O being one of the main blockers for this.I've written a test program for this, which demonstrates the ioctl and
io_uringinterface produce identical results: https://github.com/maharmstone/io_uring-encoded
Thanks, Anand
Quoted 34 lines
Changelog:
v4:
* Rewritten to avoid taking function pointer
* Removed nowait parameter, as this could be derived from iocb flags
* Fixed structure not getting properly initialized
* Followed ioctl by capping uncompressed reads at EOF
* Rebased against btrfs/for-next
* Formatting fixes
* Rearranged structs to minimize holes
* Published test program
* Fixed potential data race with userspace
* Changed to useio_uring_cmd_to_pduhelper function
* Added comments for potentially confusing parts of the codev3:
* Redo of previous versionsMark Harmstone (5):
btrfs: remove pointless addition inbtrfs_encoded_read
btrfs: changebtrfs_encoded_readso that reading of extent is done by
caller
btrfs: don't sleep inbtrfs_encoded_readifIOCB_NOWAITset
btrfs: move priv off stack inbtrfs_encoded_read_regular_fill_pages
btrfs: addio_uringcommand for encoded readsfs/btrfs/btrfs_inode.h | 13 +- fs/btrfs/file.c | 1 + fs/btrfs/inode.c | 175 ++++++++++++++------- fs/btrfs/ioctl.c | 342 ++++++++++++++++++++++++++++++++++++++++- fs/btrfs/ioctl.h | 2 + fs/btrfs/send.c | 3 +- 6 files changed, 473 insertions(+), 63 deletions(-)
Quoted 8 lines · Anand Jain
Quoted 3 lines · Mark Harmstone
This is version 4 of a patch series to add an
io_uringinterface for encoded reads. The principal use case for this is to eventually allow btrfs send and receive to operate asynchronously,How would you define an asynchronously operated Btrfs send and receive? Are you referring to Btrfs send and receive the leveraging
io_uringasynchronous operation?
Yes. The ideal for btrfs receive would be if we could mmap the stream file, loop through the TLV entries translating them into io_uring SQEs, then wait at the end for completion.
Mark