Skip to contentSkip to conversation

.flush and io_uring_cancel_files

RFC In review Pavel Begunkov 3 participants 6 messages May 26, 2020 – May 29, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC 1/1 .flush and io_uring_cancel_files

It looks like taking ->uring_lock should work like kind of grace period for struct files_struct and io_uring_flush(), and that would solve the race with "fcheck(ctx->ring_fd) == ctx->ring_file".

Can you take a look? If you like it, I'll send a proper patch and a bunch of cleanups on top.

fs/io_uring.c +5 -6
index a3dbd5f40391..012af200dc72 100644
@@ -5557,12 +5557,11 @@ static int io_grab_files(struct io_kiocb *req)
 	 * the fd has changed since we started down this path, and disallow
 	 * this operation if it has.
 	 */
-	if (fcheck(ctx->ring_fd) == ctx->ring_file) {
-		list_add(&req->inflight_entry, &ctx->inflight_list);
-		req->flags |= REQ_F_INFLIGHT;
-		req->work.files = current->files;
-		ret = 0;
-	}
+	list_add(&req->inflight_entry, &ctx->inflight_list);
+	req->flags |= REQ_F_INFLIGHT;
+	req->work.files = current->files;
+	ret = 0;
+
 	spin_unlock_irq(&ctx->inflight_lock);
 	rcu_read_unlock();
 
@@ -7479,6 +7478,10 @@ static int io_uring_release(struct inode *inode, struct

file *file)
static void io_uring_cancel_files(struct io_ring_ctx *ctx,
struct files_struct *files)
{
+ /* wait all submitters that can race for @files */
+ mutex_lock(&ctx->uring_lock);
+ mutex_unlock(&ctx->uring_lock);
+
while (!list_empty_careful(&ctx->inflight_list)) {
struct io_kiocb *cancel_req = NULL, *req; DEFINE_WAIT(wait);

Signature · 1 line
Pavel Begunkov
maintainer
#1 lore raw
Quoted 6 lines · Pavel Begunkov

It looks like taking ->uring_lock should work like kind of grace period for struct files_struct and io_uring_flush(), and that would solve the race with "fcheck(ctx->ring_fd) == ctx->ring_file".

Can you take a look? If you like it, I'll send a proper patch and a bunch of cleanups on top.

Adding Jann.

Quoted 37 lines
fs/io_uring.c +9 -6
index a3dbd5f40391..012af200dc72 100644
@@ -5557,12 +5557,11 @@ static int io_grab_files(struct io_kiocb *req)
 	 * the fd has changed since we started down this path, and disallow
 	 * this operation if it has.
 	 */
-	if (fcheck(ctx->ring_fd) == ctx->ring_file) {
-		list_add(&req->inflight_entry, &ctx->inflight_list);
-		req->flags |= REQ_F_INFLIGHT;
-		req->work.files = current->files;
-		ret = 0;
-	}
+	list_add(&req->inflight_entry, &ctx->inflight_list);
+	req->flags |= REQ_F_INFLIGHT;
+	req->work.files = current->files;
+	ret = 0;
+
 	spin_unlock_irq(&ctx->inflight_lock);
 	rcu_read_unlock();
 
@@ -7479,6 +7478,10 @@ static int io_uring_release(struct inode *inode, struct
 file *file)
 static void io_uring_cancel_files(struct io_ring_ctx *ctx,
 				  struct files_struct *files)
 {
+	/* wait all submitters that can race for @files */
+	mutex_lock(&ctx->uring_lock);
+	mutex_unlock(&ctx->uring_lock);
+
 	while (!list_empty_careful(&ctx->inflight_list)) {
 		struct io_kiocb *cancel_req = NULL, *req;

DEFINE_WAIT(wait);

Signature · 1 line
Jens Axboe
#2 lore raw
In reply to Pavel Begunkov
Quoted 41 lines · Pavel Begunkov

It looks like taking ->uring_lock should work like kind of grace period for struct files_struct and io_uring_flush(), and that would solve the race with "fcheck(ctx->ring_fd) == ctx->ring_file".

Can you take a look? If you like it, I'll send a proper patch and a bunch of cleanups on top.

fs/io_uring.c +9 -6
index a3dbd5f40391..012af200dc72 100644
@@ -5557,12 +5557,11 @@ static int io_grab_files(struct io_kiocb *req)
         * the fd has changed since we started down this path, and disallow
         * this operation if it has.
         */
-       if (fcheck(ctx->ring_fd) == ctx->ring_file) {
-               list_add(&req->inflight_entry, &ctx->inflight_list);
-               req->flags |= REQ_F_INFLIGHT;
-               req->work.files = current->files;
-               ret = 0;
-       }
+       list_add(&req->inflight_entry, &ctx->inflight_list);
+       req->flags |= REQ_F_INFLIGHT;
+       req->work.files = current->files;
+       ret = 0;
+
        spin_unlock_irq(&ctx->inflight_lock);
        rcu_read_unlock();
 
@@ -7479,6 +7478,10 @@ static int io_uring_release(struct inode *inode, struct
 file *file)
 static void io_uring_cancel_files(struct io_ring_ctx *ctx,
                                  struct files_struct *files)
 {
+       /* wait all submitters that can race for @files */
+       mutex_lock(&ctx->uring_lock);
+       mutex_unlock(&ctx->uring_lock);
+
        while (!list_empty_careful(&ctx->inflight_list)) {
                struct io_kiocb *cancel_req = NULL, *req;
Quoted 1 line
                DEFINE_WAIT(wait);

First off: You're removing a check in io_grab_files() without changing the comment that describes the check; and the new comment you're adding in io_uring_cancel_files() is IMO too short to be useful.

I'm trying to figure out how your change is supposed to work, and I don't get it. If a submitter is just past fdget() (at which point no locks are held), the ->flush() caller can instantly take and drop the ->uring_lock, and then later the rest of the submission path will grab an unprotected pointer to the files_struct. Am I missing something?

#3 lore raw
Quoted 31 lines · Jann Horn
Quoted 30 lines · Pavel Begunkov

It looks like taking ->uring_lock should work like kind of grace period for struct files_struct and io_uring_flush(), and that would solve the race with "fcheck(ctx->ring_fd) == ctx->ring_file".

Can you take a look? If you like it, I'll send a proper patch and a bunch of cleanups on top.

fs/io_uring.c +5 -6
index a3dbd5f40391..012af200dc72 100644
@@ -5557,12 +5557,11 @@ static int io_grab_files(struct io_kiocb *req)
         * the fd has changed since we started down this path, and disallow
         * this operation if it has.
         */
-       if (fcheck(ctx->ring_fd) == ctx->ring_file) {
-               list_add(&req->inflight_entry, &ctx->inflight_list);
-               req->flags |= REQ_F_INFLIGHT;
-               req->work.files = current->files;
-               ret = 0;
-       }
+       list_add(&req->inflight_entry, &ctx->inflight_list);
+       req->flags |= REQ_F_INFLIGHT;
+       req->work.files = current->files;
+       ret = 0;
+
        spin_unlock_irq(&ctx->inflight_lock);
        rcu_read_unlock();
 
Quoted 16 lines
Quoted 11 lines
fs/io_uring.c +4 -0
@@ -7479,6 +7478,10 @@ static int io_uring_release(struct inode *inode, struct
 file *file)
 static void io_uring_cancel_files(struct io_ring_ctx *ctx,
                                  struct files_struct *files)
 {
+       /* wait all submitters that can race for @files */
+       mutex_lock(&ctx->uring_lock);
+       mutex_unlock(&ctx->uring_lock);
+
        while (!list_empty_careful(&ctx->inflight_list)) {
                struct io_kiocb *cancel_req = NULL, *req;
Quoted 1 line
                DEFINE_WAIT(wait);

First off: You're removing a check in io_grab_files() without changing the comment that describes the check; and the new comment you're adding in io_uring_cancel_files() is IMO too short to be useful.

Obviously, it was stripped down to show the idea, nobody is talking about commiting it as is. I hoped Jens remembers it well enough to understand. Let me describe it in more details then:

Quoted 6 lines

I'm trying to figure out how your change is supposed to work, and I don't get it. If a submitter is just past fdget() (at which point no locks are held), the ->flush() caller can instantly take and drop the ->uring_lock, and then later the rest of the submission path will grab an unprotected pointer to the files_struct. Am I missing something?

old = tsk->files;
task_lock(tsk);
tsk->files = files;
task_unlock(tsk);
put_files_struct(old); (i.e. ->flush(old))

It's from reset_files_struct(), and I presume the whole idea of io_uring->flush() is to protect against racing for similarly going away @old files. I.e. ensuring of not having io_uring requests holding @old files.

The only place, where current->files are accessed and copied by io_uring, is io_grab_files(), which is called in the submission path. And the whole submission path is done under @uring_mtx.
For your case, the submitter will take @uring_mtx only after this lock/unlock happened, so it won't see old files (happens-before by locking mutex).

The idea behind lock/unlock is that
- submitters already locked @uring_mtx (i.e. started submission) before the lock/unlock, are waited for in the flush. These can potentially access @old.

- submitters, that came after the lock/unlock, won't see @old files.

So, no new request associated with @old can appear after that. All's left is to deal with already submitted requests, that's done by the rest of io_uring_cancel_files().

The thing I don't know is why current->files is originally accessed without protection in io_grab_files(), but presumably rcu_read_lock() there is for that reason.

Do you see anything suspicious from the description?

Signature · 1 line
Pavel Begunkov
#4 lore raw
Quoted 66 lines · Pavel Begunkov
Quoted 31 lines · Jann Horn
Quoted 30 lines · Pavel Begunkov

It looks like taking ->uring_lock should work like kind of grace period for struct files_struct and io_uring_flush(), and that would solve the race with "fcheck(ctx->ring_fd) == ctx->ring_file".

Can you take a look? If you like it, I'll send a proper patch and a bunch of cleanups on top.

fs/io_uring.c +5 -6
index a3dbd5f40391..012af200dc72 100644
@@ -5557,12 +5557,11 @@ static int io_grab_files(struct io_kiocb *req)
         * the fd has changed since we started down this path, and disallow
         * this operation if it has.
         */
-       if (fcheck(ctx->ring_fd) == ctx->ring_file) {
-               list_add(&req->inflight_entry, &ctx->inflight_list);
-               req->flags |= REQ_F_INFLIGHT;
-               req->work.files = current->files;
-               ret = 0;
-       }
+       list_add(&req->inflight_entry, &ctx->inflight_list);
+       req->flags |= REQ_F_INFLIGHT;
+       req->work.files = current->files;
+       ret = 0;
+
        spin_unlock_irq(&ctx->inflight_lock);
        rcu_read_unlock();
 
Quoted 16 lines
Quoted 11 lines
fs/io_uring.c +4 -0
@@ -7479,6 +7478,10 @@ static int io_uring_release(struct inode *inode, struct
 file *file)
 static void io_uring_cancel_files(struct io_ring_ctx *ctx,
                                  struct files_struct *files)
 {
+       /* wait all submitters that can race for @files */
+       mutex_lock(&ctx->uring_lock);
+       mutex_unlock(&ctx->uring_lock);
+
        while (!list_empty_careful(&ctx->inflight_list)) {
                struct io_kiocb *cancel_req = NULL, *req;
Quoted 1 line
                DEFINE_WAIT(wait);

First off: You're removing a check in io_grab_files() without changing the comment that describes the check; and the new comment you're adding in io_uring_cancel_files() is IMO too short to be useful.

Obviously, it was stripped down to show the idea, nobody is talking about commiting it as is. I hoped Jens remembers it well enough to understand. Let me describe it in more details then:

Quoted 6 lines

I'm trying to figure out how your change is supposed to work, and I don't get it. If a submitter is just past fdget() (at which point no locks are held), the ->flush() caller can instantly take and drop the ->uring_lock, and then later the rest of the submission path will grab an unprotected pointer to the files_struct. Am I missing something?

old = tsk->files;
task_lock(tsk);
tsk->files = files;
task_unlock(tsk);
put_files_struct(old); (i.e. ->flush(old))

Quoted 3 lines

It's from reset_files_struct(), and I presume the whole idea of io_uring->flush() is to protect against racing for similarly going away @old files. I.e. ensuring of not having io_uring requests holding @old files.

Kind of. We use the ->flush() handler to be notified when the files_struct goes away, so that instead of holding a reference to the files_struct (which would cause a reference loop), we can clean up our references when it goes away.

Quoted 3 lines

The only place, where current->files are accessed and copied by io_uring, is io_grab_files(), which is called in the submission path. And the whole submission path is done under @uring_mtx.

No it isn't. We do fdget(fd) at the start of the io_uring_enter syscall, and at that point we obviously can't hold the uring_mtx yet.

Quoted 2 lines

For your case, the submitter will take @uring_mtx only after this lock/unlock happened, so it won't see old files (happens-before by locking mutex).

No, it will see the old files. The concurrent operation we're worried about is not that the files_struct goes away somehow (that can't happen); what we want to guard against is a concurrent close() or dup2() or so removing the uring fd from the files_struct, because if someone calls close() before we stash a pointer to current->files, that pointer isn't protected anymore.

Quoted 3 lines

The thing I don't know is why current->files is originally accessed without protection in io_grab_files(), but presumably rcu_read_lock() there is for that reason.

No, it's because current->files can never change under you; pretty much the only places where current->files can change are unshare() and execve().

#5 lore raw
Quoted 76 lines · Jann Horn
Quoted 66 lines · Pavel Begunkov
Quoted 31 lines · Jann Horn
Quoted 30 lines · Pavel Begunkov

It looks like taking ->uring_lock should work like kind of grace period for struct files_struct and io_uring_flush(), and that would solve the race with "fcheck(ctx->ring_fd) == ctx->ring_file".

Can you take a look? If you like it, I'll send a proper patch and a bunch of cleanups on top.

fs/io_uring.c +5 -6
index a3dbd5f40391..012af200dc72 100644
@@ -5557,12 +5557,11 @@ static int io_grab_files(struct io_kiocb *req)
         * the fd has changed since we started down this path, and disallow
         * this operation if it has.
         */
-       if (fcheck(ctx->ring_fd) == ctx->ring_file) {
-               list_add(&req->inflight_entry, &ctx->inflight_list);
-               req->flags |= REQ_F_INFLIGHT;
-               req->work.files = current->files;
-               ret = 0;
-       }
+       list_add(&req->inflight_entry, &ctx->inflight_list);
+       req->flags |= REQ_F_INFLIGHT;
+       req->work.files = current->files;
+       ret = 0;
+
        spin_unlock_irq(&ctx->inflight_lock);
        rcu_read_unlock();
 
Quoted 16 lines
Quoted 11 lines
fs/io_uring.c +4 -0
@@ -7479,6 +7478,10 @@ static int io_uring_release(struct inode *inode, struct
 file *file)
 static void io_uring_cancel_files(struct io_ring_ctx *ctx,
                                  struct files_struct *files)
 {
+       /* wait all submitters that can race for @files */
+       mutex_lock(&ctx->uring_lock);
+       mutex_unlock(&ctx->uring_lock);
+
        while (!list_empty_careful(&ctx->inflight_list)) {
                struct io_kiocb *cancel_req = NULL, *req;
Quoted 1 line
                DEFINE_WAIT(wait);

First off: You're removing a check in io_grab_files() without changing the comment that describes the check; and the new comment you're adding in io_uring_cancel_files() is IMO too short to be useful.

Obviously, it was stripped down to show the idea, nobody is talking about commiting it as is. I hoped Jens remembers it well enough to understand. Let me describe it in more details then:

Quoted 6 lines

I'm trying to figure out how your change is supposed to work, and I don't get it. If a submitter is just past fdget() (at which point no locks are held), the ->flush() caller can instantly take and drop the ->uring_lock, and then later the rest of the submission path will grab an unprotected pointer to the files_struct. Am I missing something?

old = tsk->files;
task_lock(tsk);
tsk->files = files;
task_unlock(tsk);
put_files_struct(old); (i.e. ->flush(old))

Quoted 3 lines

It's from reset_files_struct(), and I presume the whole idea of io_uring->flush() is to protect against racing for similarly going away @old files. I.e. ensuring of not having io_uring requests holding @old files.

Kind of. We use the ->flush() handler to be notified when the files_struct goes away, so that instead of holding a reference to the files_struct (which would cause a reference loop), we can clean up our references when it goes away.

Quoted 6 lines
Quoted 3 lines

The only place, where current->files are accessed and copied by io_uring, is io_grab_files(), which is called in the submission path. And the whole submission path is done under @uring_mtx.

No it isn't. We do fdget(fd) at the start of the io_uring_enter syscall, and at that point we obviously can't hold the uring_mtx yet.

__directly__ accessing ->files, or hand-coded. fdget() by itself shouldn't be a problem.

Quoted 5 lines
Quoted 3 lines

For your case, the submitter will take @uring_mtx only after this lock/unlock happened, so it won't see old files (happens-before by locking mutex).

No, it will see the old files. The concurrent operation we're worried

From what you wrote below, it will see the old files just because nobody would try to replace them. Is that it?

Quoted 5 lines

about is not that the files_struct goes away somehow (that can't happen); what we want to guard against is a concurrent close() or dup2() or so removing the uring fd from the files_struct, because if someone calls close() before we stash a pointer to current->files, that pointer isn't protected anymore.

Let me guess, you mean maintenance of ->files itself, such as expand_files()?

Quoted 9 lines
Quoted 5 lines

The thing I don't know is why current->files is originally accessed without protection in io_grab_files(), but presumably rcu_read_lock() there is for that reason.

No, it's because current->files can never change under you; pretty much the only places where current->files can change are unshare() and execve().

I see, good to know

Signature · 1 line
Pavel Begunkov