.flush and io_uring_cancel_files
RFC 1/1 .flush and io_uring_cancel_files
It looks like taking ->uring_lock should work like kind of grace period for struct files_struct and io_uring_flush(), and that would solve the race with "fcheck(ctx->ring_fd) == ctx->ring_file".
Can you take a look? If you like it, I'll send a proper patch and a bunch of cleanups on top.
fs/io_uring.c +5 -6
@@ -5557,12 +5557,11 @@ static int io_grab_files(struct io_kiocb *req)* the fd has changed since we started down this path, and disallow * this operation if it has. */ -if (fcheck (ctx->ring_fd) == ctx->ring_file) { -list_add (&req->inflight_entry, &ctx->inflight_list); - req->flags |=REQ_F_INFLIGHT ; - req->work.files = current->files; - ret =0 ; - } +list_add (&req->inflight_entry, &ctx->inflight_list); + req->flags |=REQ_F_INFLIGHT ; + req->work.files = current->files; + ret =0 ; +spin_unlock_irq (&ctx->inflight_lock);rcu_read_unlock ();
@@ -7479,6 +7478,10 @@ static int io_uring_release(struct inode *inode, struct
file *file)
static void io_uring_cancel_files(struct io_ring_ctx *ctx,
struct files_struct *files)
{
+ /* wait all submitters that can race for @files */
+ mutex_lock(&ctx->uring_lock);
+ mutex_unlock(&ctx->uring_lock);
+
while (!list_empty_careful(&ctx->inflight_list)) {
struct io_kiocb *cancel_req = NULL, *req; DEFINE_WAIT(wait);
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerQuoted 6 lines · Pavel Begunkov
It looks like taking ->uring_lock should work like kind of grace period for struct
files_structandio_uring_flush(), and that would solve the race with "fcheck(ctx->ring_fd)==ctx->ring_file".Can you take a look? If you like it, I'll send a proper patch and a bunch of cleanups on top.
Adding Jann.
Quoted 37 lines
fs/io_uring.c +9
-6@@ -5557,12 +5557,11 @@ static int io_grab_files(struct io_kiocb *req)* the fd has changed since we started down this path, and disallow * this operation if it has. */ -if (fcheck (ctx->ring_fd) == ctx->ring_file) { -list_add (&req->inflight_entry, &ctx->inflight_list); - req->flags |=REQ_F_INFLIGHT ; - req->work.files = current->files; - ret =0 ; - } +list_add (&req->inflight_entry, &ctx->inflight_list); + req->flags |=REQ_F_INFLIGHT ; + req->work.files = current->files; + ret =0 ; +spin_unlock_irq (&ctx->inflight_lock);rcu_read_unlock ();@@ -7479,6 +7478,10 @@ static int io_uring_release(struct inode *inode, struct file *file)static void io_uring_cancel_files (struct io_ring_ctx *ctx,struct files_struct *files) { +/* wait all submitters that can race for @files */ +mutex_lock (&ctx->uring_lock); +mutex_unlock (&ctx->uring_lock); +while (!list_empty_careful (&ctx->inflight_list)) {struct io_kiocb *cancel_req =NULL , *req;
DEFINE_WAIT(wait);
Signature · 1 line
Jens Axboe
Quoted 41 lines · Pavel Begunkov
It looks like taking ->uring_lock should work like kind of grace period for struct
files_structandio_uring_flush(), and that would solve the race with "fcheck(ctx->ring_fd)==ctx->ring_file".Can you take a look? If you like it, I'll send a proper patch and a bunch of cleanups on top.
fs/io_uring.c +9
-6@@ -5557,12 +5557,11 @@ static int io_grab_files(struct io_kiocb *req)* the fd has changed since we started down this path, and disallow * this operation if it has. */ -if (fcheck (ctx->ring_fd) == ctx->ring_file) { -list_add (&req->inflight_entry, &ctx->inflight_list); - req->flags |=REQ_F_INFLIGHT ; - req->work.files = current->files; - ret =0 ; - } +list_add (&req->inflight_entry, &ctx->inflight_list); + req->flags |=REQ_F_INFLIGHT ; + req->work.files = current->files; + ret =0 ; +spin_unlock_irq (&ctx->inflight_lock);rcu_read_unlock ();@@ -7479,6 +7478,10 @@ static int io_uring_release(struct inode *inode, struct file *file)static void io_uring_cancel_files (struct io_ring_ctx *ctx,struct files_struct *files) { +/* wait all submitters that can race for @files */ +mutex_lock (&ctx->uring_lock); +mutex_unlock (&ctx->uring_lock); +while (!list_empty_careful (&ctx->inflight_list)) {struct io_kiocb *cancel_req =NULL , *req;
Quoted 1 line
DEFINE_WAIT(wait);
First off: You're removing a check in io_grab_files() without changing the comment that describes the check; and the new comment you're adding in io_uring_cancel_files() is IMO too short to be useful.
I'm trying to figure out how your change is supposed to work, and I don't get it. If a submitter is just past fdget() (at which point no locks are held), the ->flush() caller can instantly take and drop the ->uring_lock, and then later the rest of the submission path will grab an unprotected pointer to the files_struct. Am I missing something?
Quoted 31 lines · Jann Horn
Quoted 30 lines · Pavel Begunkov
It looks like taking ->uring_lock should work like kind of grace period for struct
files_structandio_uring_flush(), and that would solve the race with "fcheck(ctx->ring_fd)==ctx->ring_file".Can you take a look? If you like it, I'll send a proper patch and a bunch of cleanups on top.
fs/io_uring.c +5
-6@@ -5557,12 +5557,11 @@ static int io_grab_files(struct io_kiocb *req)* the fd has changed since we started down this path, and disallow * this operation if it has. */ -if (fcheck (ctx->ring_fd) == ctx->ring_file) { -list_add (&req->inflight_entry, &ctx->inflight_list); - req->flags |=REQ_F_INFLIGHT ; - req->work.files = current->files; - ret =0 ; - } +list_add (&req->inflight_entry, &ctx->inflight_list); + req->flags |=REQ_F_INFLIGHT ; + req->work.files = current->files; + ret =0 ; +spin_unlock_irq (&ctx->inflight_lock);rcu_read_unlock ();
Quoted 16 lines
Quoted 11 lines
fs/io_uring.c +4
-0@@ -7479,6 +7478,10 @@ static int io_uring_release(struct inode *inode, struct file *file)static void io_uring_cancel_files (struct io_ring_ctx *ctx,struct files_struct *files) { +/* wait all submitters that can race for @files */ +mutex_lock (&ctx->uring_lock); +mutex_unlock (&ctx->uring_lock); +while (!list_empty_careful (&ctx->inflight_list)) {struct io_kiocb *cancel_req =NULL , *req;Quoted 1 line
DEFINE_WAIT(wait);First off: You're removing a check in
io_grab_files()without changing the comment that describes the check; and the new comment you're adding inio_uring_cancel_files()is IMO too short to be useful.
Obviously, it was stripped down to show the idea, nobody is talking about commiting it as is. I hoped Jens remembers it well enough to understand. Let me describe it in more details then:
Quoted 6 lines
I'm trying to figure out how your change is supposed to work, and I don't get it. If a submitter is just past
fdget()(at which point no locks are held), the ->flush() caller can instantly take and drop the ->uring_lock, and then later the rest of the submission path will grab an unprotected pointer to thefiles_struct. Am I missing something?
old = tsk->files;task_lock(tsk);tsk->files = files;task_unlock(tsk);put_files_struct(old); (i.e. ->flush(old))
It's from reset_files_struct(), and I presume the whole idea of io_uring->flush() is to protect against racing for similarly going away @old files. I.e. ensuring of not having io_uring requests holding @old files.
The only place, where current->files are accessed and copied by io_uring, is io_grab_files(), which is called in the submission path. And the whole submission path is done under @uring_mtx.
For your case, the submitter will take @uring_mtx only after this lock/unlock happened, so it won't see old files (happens-before by locking mutex).
The idea behind lock/unlock is that
- submitters already locked @uring_mtx (i.e. started submission) before the lock/unlock, are waited for in the flush. These can potentially access @old.
- submitters, that came after the lock/unlock, won't see @old files.
So, no new request associated with @old can appear after that. All's left is to deal with already submitted requests, that's done by the rest of io_uring_cancel_files().
The thing I don't know is why current->files is originally accessed without protection in io_grab_files(), but presumably rcu_read_lock() there is for that reason.
Do you see anything suspicious from the description?
Signature · 1 line
Pavel Begunkov
Quoted 66 lines · Pavel Begunkov
Quoted 31 lines · Jann Horn
Quoted 30 lines · Pavel Begunkov
It looks like taking ->uring_lock should work like kind of grace period for struct
files_structandio_uring_flush(), and that would solve the race with "fcheck(ctx->ring_fd)==ctx->ring_file".Can you take a look? If you like it, I'll send a proper patch and a bunch of cleanups on top.
fs/io_uring.c +5
-6@@ -5557,12 +5557,11 @@ static int io_grab_files(struct io_kiocb *req)* the fd has changed since we started down this path, and disallow * this operation if it has. */ -if (fcheck (ctx->ring_fd) == ctx->ring_file) { -list_add (&req->inflight_entry, &ctx->inflight_list); - req->flags |=REQ_F_INFLIGHT ; - req->work.files = current->files; - ret =0 ; - } +list_add (&req->inflight_entry, &ctx->inflight_list); + req->flags |=REQ_F_INFLIGHT ; + req->work.files = current->files; + ret =0 ; +spin_unlock_irq (&ctx->inflight_lock);rcu_read_unlock ();Quoted 16 lines
Quoted 11 lines
fs/io_uring.c +4
-0@@ -7479,6 +7478,10 @@ static int io_uring_release(struct inode *inode, struct file *file)static void io_uring_cancel_files (struct io_ring_ctx *ctx,struct files_struct *files) { +/* wait all submitters that can race for @files */ +mutex_lock (&ctx->uring_lock); +mutex_unlock (&ctx->uring_lock); +while (!list_empty_careful (&ctx->inflight_list)) {struct io_kiocb *cancel_req =NULL , *req;Quoted 1 line
DEFINE_WAIT(wait);First off: You're removing a check in
io_grab_files()without changing the comment that describes the check; and the new comment you're adding inio_uring_cancel_files()is IMO too short to be useful.Obviously, it was stripped down to show the idea, nobody is talking about commiting it as is. I hoped Jens remembers it well enough to understand. Let me describe it in more details then:
Quoted 6 lines
I'm trying to figure out how your change is supposed to work, and I don't get it. If a submitter is just past
fdget()(at which point no locks are held), the ->flush() caller can instantly take and drop the ->uring_lock, and then later the rest of the submission path will grab an unprotected pointer to thefiles_struct. Am I missing something?old =
tsk->files;task_lock(tsk);tsk->files= files;task_unlock(tsk);put_files_struct(old); (i.e. ->flush(old))
Quoted 3 lines
It's from
reset_files_struct(), and I presume the whole idea ofio_uring->flush()is to protect against racing for similarly going away @old files. I.e. ensuring of not havingio_uringrequests holding @old files.
Kind of. We use the ->flush() handler to be notified when the files_struct goes away, so that instead of holding a reference to the files_struct (which would cause a reference loop), we can clean up our references when it goes away.
Quoted 3 lines
The only place, where
current->filesare accessed and copied byio_uring, isio_grab_files(), which is called in the submission path. And the whole submission path is done under @uring_mtx.
No it isn't. We do fdget(fd) at the start of the io_uring_enter syscall, and at that point we obviously can't hold the uring_mtx yet.
Quoted 2 lines
For your case, the submitter will take @uring_mtx only after this lock/unlock happened, so it won't see old files (happens-before by locking mutex).
No, it will see the old files. The concurrent operation we're worried about is not that the files_struct goes away somehow (that can't happen); what we want to guard against is a concurrent close() or dup2() or so removing the uring fd from the files_struct, because if someone calls close() before we stash a pointer to current->files, that pointer isn't protected anymore.
Quoted 3 lines
The thing I don't know is why
current->filesis originally accessed without protection inio_grab_files(), but presumablyrcu_read_lock()there is for that reason.
No, it's because current->files can never change under you; pretty much the only places where current->files can change are unshare() and execve().
Quoted 76 lines · Jann Horn
Quoted 66 lines · Pavel Begunkov
Quoted 31 lines · Jann Horn
Quoted 30 lines · Pavel Begunkov
It looks like taking ->uring_lock should work like kind of grace period for struct
files_structandio_uring_flush(), and that would solve the race with "fcheck(ctx->ring_fd)==ctx->ring_file".Can you take a look? If you like it, I'll send a proper patch and a bunch of cleanups on top.
fs/io_uring.c +5
-6@@ -5557,12 +5557,11 @@ static int io_grab_files(struct io_kiocb *req)* the fd has changed since we started down this path, and disallow * this operation if it has. */ -if (fcheck (ctx->ring_fd) == ctx->ring_file) { -list_add (&req->inflight_entry, &ctx->inflight_list); - req->flags |=REQ_F_INFLIGHT ; - req->work.files = current->files; - ret =0 ; - } +list_add (&req->inflight_entry, &ctx->inflight_list); + req->flags |=REQ_F_INFLIGHT ; + req->work.files = current->files; + ret =0 ; +spin_unlock_irq (&ctx->inflight_lock);rcu_read_unlock ();Quoted 16 lines
Quoted 11 lines
fs/io_uring.c +4
-0@@ -7479,6 +7478,10 @@ static int io_uring_release(struct inode *inode, struct file *file)static void io_uring_cancel_files (struct io_ring_ctx *ctx,struct files_struct *files) { +/* wait all submitters that can race for @files */ +mutex_lock (&ctx->uring_lock); +mutex_unlock (&ctx->uring_lock); +while (!list_empty_careful (&ctx->inflight_list)) {struct io_kiocb *cancel_req =NULL , *req;Quoted 1 line
DEFINE_WAIT(wait);First off: You're removing a check in
io_grab_files()without changing the comment that describes the check; and the new comment you're adding inio_uring_cancel_files()is IMO too short to be useful.Obviously, it was stripped down to show the idea, nobody is talking about commiting it as is. I hoped Jens remembers it well enough to understand. Let me describe it in more details then:
Quoted 6 lines
I'm trying to figure out how your change is supposed to work, and I don't get it. If a submitter is just past
fdget()(at which point no locks are held), the ->flush() caller can instantly take and drop the ->uring_lock, and then later the rest of the submission path will grab an unprotected pointer to thefiles_struct. Am I missing something?old =
tsk->files;task_lock(tsk);tsk->files= files;task_unlock(tsk);put_files_struct(old); (i.e. ->flush(old))Quoted 3 lines
It's from
reset_files_struct(), and I presume the whole idea ofio_uring->flush()is to protect against racing for similarly going away @old files. I.e. ensuring of not havingio_uringrequests holding @old files.Kind of. We use the ->flush() handler to be notified when the
files_structgoes away, so that instead of holding a reference to thefiles_struct(which would cause a reference loop), we can clean up our references when it goes away.
Quoted 6 lines
Quoted 3 lines
The only place, where
current->filesare accessed and copied byio_uring, isio_grab_files(), which is called in the submission path. And the whole submission path is done under @uring_mtx.No it isn't. We do fdget(fd) at the start of the
io_uring_entersyscall, and at that point we obviously can't hold theuring_mtxyet.
__directly__ accessing ->files, or hand-coded. fdget() by itself shouldn't be a problem.
Quoted 5 lines
Quoted 3 lines
For your case, the submitter will take @uring_mtx only after this lock/unlock happened, so it won't see old files (happens-before by locking mutex).
No, it will see the old files. The concurrent operation we're worried
From what you wrote below, it will see the old files just because nobody would try to replace them. Is that it?
Quoted 5 lines
about is not that the
files_structgoes away somehow (that can't happen); what we want to guard against is a concurrentclose()ordup2()or so removing the uring fd from thefiles_struct, because if someone callsclose()before we stash a pointer tocurrent->files, that pointer isn't protected anymore.
Let me guess, you mean maintenance of ->files itself, such as expand_files()?
Quoted 9 lines
Quoted 5 lines
The thing I don't know is why
current->filesis originally accessed without protection inio_grab_files(), but presumablyrcu_read_lock()there is for that reason.No, it's because
current->filescan never change under you; pretty much the only places wherecurrent->filescan change areunshare()andexecve().
I see, good to know
Signature · 1 line
Pavel Begunkov