aio: fix oops because of extra IO control block freeing.

PATCH In review Leonid Ananiev 4 participants 4 messages Mar 5, 2007 – Mar 8, 2007

Subsystems (4)

Memory management VFS Page cache Aio

lore

Share & tools

lore

Leonid Ananiev

author

2007-03-05 14:23 UTC #0 lore raw

PATCH 1/3 aio: fix oops because of extra IO control block freeing.

From Leonid Ananiev

The patch fixes oops because of extra IO control block freeing. IO is retried if page could not be invalidated.

Signed-off-by Leonid Ananiev <[email protected]>

The patch fixes oops "Kernel BUG at fs/aio.c:509" archived at http://lkml.org/lkml/2007/2/8/337
The number of IO control block (iocb)users < 0.
If page could not be invalidated by invalidate_inode_pages2_range()
than EIO is returned. It happens if journal_try_to_free_buffers() fails to drop_buffers(). This EIO is not differing from real IO competition with EIO and aio_complete() is called. Later aio_complete() is called from dio_bio_end_aio() and frees iocb once more.

After patch generic_file_direct_IO() sets PgBusy flag in iocb if page could not be invalidated. iocb is retried after IO competition. The process is waked up if IO is SYNC else iocb is kicked. The lines “if (ret != -EIOCBRETRY)” is deleted because nothing set to EIOCBRETRY.

Next patches 2/3 and 3/3 do cleanup only.
The patch is applied and tested with aio-stress on 2.6.20 and 2.6.21-rc2

fs/aio.c +5 ~~-26~~

diff -uprNX linux-2.6.21-rc2/Documentation/dontdiff linux-2.6.21-rc2/fs/aio.c linux-2.6.21-rc2-aio31/fs/aio.c
@@ -723,37 +723,12 @@ static ssize_t aio_run_iocb(struct kiocb
 	ret = retry(iocb);
 	current->io_wait = NULL;
 
-	if (ret != -EIOCBRETRY && ret != -EIOCBQUEUED) {
+	if (!kiocbIsPgBusy(iocb) && ret != -EIOCBQUEUED) {
 		BUG_ON(!list_empty(&iocb->ki_wait.task_list));
 		aio_complete(iocb, ret, 0);
 	}
 out:
 	spin_lock_irq(&ctx->ctx_lock);
-
-	if (-EIOCBRETRY == ret) {
-		/*
-		 * OK, now that we are done with this iteration
-		 * and know that there is more left to go,
-		 * this is where we let go so that a subsequent
-		 * "kick" can start the next iteration
-		 */
-
-		/* will make __queue_kicked_iocb succeed from here on */
-		INIT_LIST_HEAD(&iocb->ki_run_list);
-		/* we must queue the next iteration ourselves, if it
-		 * has already been kicked */
-		if (kiocbIsKicked(iocb)) {
-			__queue_kicked_iocb(iocb);
-
-			/*
-			 * __queue_kicked_iocb will always return 1 here, because
-			 * iocb->ki_run_list is empty at this point so it should
-			 * be safe to unconditionally queue the context into the
-			 * work queue.
-			 */
-			aio_queue_work(ctx);
-		}
-	}
 	return ret;
 }

@@ -945,6 +920,10 @@ int fastcall aio_complete(struct kiocb *
 		wake_up_process(iocb->ki_obj.tsk);
 		return 1;
 	}
+	if (TestClearPgBusy(iocb)) { // page could not be invalidated
+		kick_iocb(iocb);
+		return 1;
+	}
 
 	info = &ctx->ring_info;

fs/read_write.c +3 -3

diff -uprNX linux-2.6.21-rc2/Documentation/dontdiff linux-2.6.21-rc2/fs/read_write.c linux-2.6.21-rc2-aio31/fs/read_write.c
@@ -239,7 +239,7 @@ ssize_t do_sync_read(struct file *filp,
 
 	for (;;) {
 		ret = filp->f_op->aio_read(&kiocb, &iov, 1, kiocb.ki_pos);
-		if (ret != -EIOCBRETRY)
+		if (!TestClearPgBusy(&kiocb))
 			break;
 		wait_on_retry_sync_kiocb(&kiocb);
 	}

@@ -297,7 +297,7 @@ ssize_t do_sync_write(struct file *filp,
 
 	for (;;) {
 		ret = filp->f_op->aio_write(&kiocb, &iov, 1, kiocb.ki_pos);
-		if (ret != -EIOCBRETRY)
+		if (!TestClearPgBusy(&kiocb))
 			break;
 		wait_on_retry_sync_kiocb(&kiocb);
 	}

@@ -463,7 +463,7 @@ ssize_t do_sync_readv_writev(struct file
 
 	for (;;) {
 		ret = fn(&kiocb, iov, nr_segs, kiocb.ki_pos);
-		if (ret != -EIOCBRETRY)
+		if (!TestClearPgBusy(&kiocb))
 			break;
 		wait_on_retry_sync_kiocb(&kiocb);
 	}

include/linux/aio.h +3 -0

diff -uprNX linux-2.6.21-rc2/Documentation/dontdiff linux-2.6.21-rc2/include/linux/aio.h linux-2.6.21-rc2-aio31/include/linux/aio.h
@@ -34,21 +34,26 @@ struct kioctx;
 /* #define KIF_LOCKED		0 */
 #define KIF_KICKED		1
 #define KIF_CANCELLED		2
+#define KIF_PG_BUSY		3
 
 #define kiocbTryLock(iocb)	test_and_set_bit(KIF_LOCKED, &(iocb)->ki_flags)
 #define kiocbTryKick(iocb)	test_and_set_bit(KIF_KICKED, &(iocb)->ki_flags)
+#define TestClearPgBusy(iocb)	test_and_clear_bit(KIF_PG_BUSY, &(iocb)->ki_flags)
 
 #define kiocbSetLocked(iocb)	set_bit(KIF_LOCKED, &(iocb)->ki_flags)
 #define kiocbSetKicked(iocb)	set_bit(KIF_KICKED, &(iocb)->ki_flags)
 #define kiocbSetCancelled(iocb)	set_bit(KIF_CANCELLED, &(iocb)->ki_flags)
+#define kiocbSetPgBusy(iocb)	set_bit(KIF_PG_BUSY, &(iocb)->ki_flags)
 
 #define kiocbClearLocked(iocb)	clear_bit(KIF_LOCKED, &(iocb)->ki_flags)
 #define kiocbClearKicked(iocb)	clear_bit(KIF_KICKED, &(iocb)->ki_flags)
 #define kiocbClearCancelled(iocb)	clear_bit(KIF_CANCELLED,

&(iocb)->ki_flags)
+#define kiocbClearPgBusy(iocb) clear_bit(KIF_PG_BUSY, &(iocb)->ki_flags)

#define kiocbIsLocked(iocb) test_bit(KIF_LOCKED, &(iocb)->ki_flags)
#define kiocbIsKicked(iocb) test_bit(KIF_KICKED, &(iocb)->ki_flags)
#define kiocbIsCancelled(iocb) test_bit(KIF_CANCELLED, &(iocb)->ki_flags)
+#define kiocbIsPgBusy(iocb) test_bit(KIF_PG_BUSY, &(iocb)->ki_flags)

/* is there a better place to document function pointer methods? */ /**

mm/filemap.c +3 -4

diff -upr linux-2.6.20/mm/filemap.c linux-2.6.20-aio2/mm/filemap.c
@@ -2413,10 +2413,9 @@ generic_file_direct_IO(int rw, struct ki
 		if (rw == WRITE && mapping->nrpages) {
 			pgoff_t end = (offset + write_len - 1)
 						>> PAGE_CACHE_SHIFT;
-			int err = invalidate_inode_pages2_range(mapping,
-					offset >> PAGE_CACHE_SHIFT, end);
-			if (err)
-				retval = err;
+			if (invalidate_inode_pages2_range(mapping,
+					offset >> PAGE_CACHE_SHIFT, end))
+				kiocbSetPgBusy(iocb);
 		}
 	}
 	return retval;

Andrew Morton

maintainer

2007-03-07 22:14 UTC #1 lore raw

Quoted 18 lines · On Mon, 05 Mar 2007 17:23:33 +0300 Leonid Ananiev…

From Leonid Ananiev
The patch fixes oops because of extra IO control block freeing. IO is retried if page could not be invalidated.
Signed-off-by Leonid Ananiev <[email protected]>
The patch fixes oops "Kernel BUG at fs/aio.c:509" archived at
http://lkml.org/lkml/2007/2/8/337
The number of IO control block (iocb)users < 0.
If page could not be invalidated by invalidate_inode_pages2_range() than EIO is returned. It happens if journal_try_to_free_buffers() fails to drop_buffers().
This EIO is not differing from real IO competition with EIO and aio_complete() is called.
Later aio_complete() is called from dio_bio_end_aio() and frees iocb once more.

Quoted 5 lines

After patch generic_file_direct_IO() sets PgBusy flag in iocb if page could not be invalidated. iocb is retried after IO competition. The process is waked up if IO is SYNC else iocb is kicked. The lines ___if (ret != -EIOCBRETRY)___ is deleted because nothing set to EIOCBRETRY.

Please copy [email protected] on AIO patches.

Quoted 1 line

Next patches 2/3 and 3/3 do cleanup only.

I cannot find those patches.

Quoted 2 lines

The patch is applied and tested with aio-stress on 2.6.20 and 2.6.21-rc2 diff -uprNX linux-2.6.21-rc2/Documentation/dontdiff

Your email client performs space-stuffing, which makes things harder on the receiving end. http://mbligh.org/linuxdocs/Email/Clients/Thunderbird might help.

Quoted 142 lines

linux-2.6.21-rc2/fs/aio.c linux-2.6.21-rc2-aio31/fs/aio.c

fs/aio.c +5 ~~-26~~

@@ -723,37 +723,12 @@ static ssize_t aio_run_iocb(struct kiocb
  	ret = retry(iocb);
  	current->io_wait = NULL;
 
-	if (ret != -EIOCBRETRY && ret != -EIOCBQUEUED) {
+	if (!kiocbIsPgBusy(iocb) && ret != -EIOCBQUEUED) {
  		BUG_ON(!list_empty(&iocb->ki_wait.task_list));
  		aio_complete(iocb, ret, 0);
  	}
  out:
  	spin_lock_irq(&ctx->ctx_lock);
-
-	if (-EIOCBRETRY == ret) {
-		/*
-		 * OK, now that we are done with this iteration
-		 * and know that there is more left to go,
-		 * this is where we let go so that a subsequent
-		 * "kick" can start the next iteration
-		 */
-
-		/* will make __queue_kicked_iocb succeed from here on */
-		INIT_LIST_HEAD(&iocb->ki_run_list);
-		/* we must queue the next iteration ourselves, if it
-		 * has already been kicked */
-		if (kiocbIsKicked(iocb)) {
-			__queue_kicked_iocb(iocb);
-
-			/*
-			 * __queue_kicked_iocb will always return 1 here, because
-			 * iocb->ki_run_list is empty at this point so it should
-			 * be safe to unconditionally queue the context into the
-			 * work queue.
-			 */
-			aio_queue_work(ctx);
-		}
-	}
  	return ret;
  }

@@ -945,6 +920,10 @@ int fastcall aio_complete(struct kiocb *
  		wake_up_process(iocb->ki_obj.tsk);
  		return 1;
  	}
+	if (TestClearPgBusy(iocb)) { // page could not be invalidated
+		kick_iocb(iocb);
+		return 1;
+	}
 
  	info = &ctx->ring_info;

diff -uprNX linux-2.6.21-rc2/Documentation/dontdiff linux-2.6.21-rc2/fs/read_write.c linux-2.6.21-rc2-aio31/fs/read_write.c

fs/read_write.c +3 -4

-0800

@@ -239,7 +239,7 @@ ssize_t do_sync_read(struct file *filp,
 
  	for (;;) {
  		ret = filp->f_op->aio_read(&kiocb, &iov, 1, kiocb.ki_pos);
-		if (ret != -EIOCBRETRY)
+		if (!TestClearPgBusy(&kiocb))
  			break;
  		wait_on_retry_sync_kiocb(&kiocb);
  	}

@@ -297,7 +297,7 @@ ssize_t do_sync_write(struct file *filp,
 
  	for (;;) {
  		ret = filp->f_op->aio_write(&kiocb, &iov, 1, kiocb.ki_pos);
-		if (ret != -EIOCBRETRY)
+		if (!TestClearPgBusy(&kiocb))
  			break;
  		wait_on_retry_sync_kiocb(&kiocb);
  	}

@@ -463,7 +463,7 @@ ssize_t do_sync_readv_writev(struct file
 
  	for (;;) {
  		ret = fn(&kiocb, iov, nr_segs, kiocb.ki_pos);
-		if (ret != -EIOCBRETRY)
+		if (!TestClearPgBusy(&kiocb))
  			break;
  		wait_on_retry_sync_kiocb(&kiocb);
  	}

diff -uprNX linux-2.6.21-rc2/Documentation/dontdiff linux-2.6.21-rc2/include/linux/aio.h linux-2.6.21-rc2-aio31/include/linux/aio.h

include/linux/aio.h +0 -0

00:38:11.000000000 -0800

include/linux/aio.h +5 -0

@@ -34,21 +34,26 @@ struct kioctx;
  /* #define KIF_LOCKED		0 */
  #define KIF_KICKED		1
  #define KIF_CANCELLED		2
+#define KIF_PG_BUSY		3
 
  #define kiocbTryLock(iocb)	test_and_set_bit(KIF_LOCKED, &(iocb)->ki_flags)
  #define kiocbTryKick(iocb)	test_and_set_bit(KIF_KICKED, &(iocb)->ki_flags)
+#define TestClearPgBusy(iocb)	test_and_clear_bit(KIF_PG_BUSY, 
 &(iocb)->ki_flags)
 
  #define kiocbSetLocked(iocb)	set_bit(KIF_LOCKED, &(iocb)->ki_flags)
  #define kiocbSetKicked(iocb)	set_bit(KIF_KICKED, &(iocb)->ki_flags)
  #define kiocbSetCancelled(iocb)	set_bit(KIF_CANCELLED, &(iocb)->ki_flags)
+#define kiocbSetPgBusy(iocb)	set_bit(KIF_PG_BUSY, &(iocb)->ki_flags)
 
  #define kiocbClearLocked(iocb)	clear_bit(KIF_LOCKED, &(iocb)->ki_flags)
  #define kiocbClearKicked(iocb)	clear_bit(KIF_KICKED, &(iocb)->ki_flags)
  #define kiocbClearCancelled(iocb)	clear_bit(KIF_CANCELLED, 
 &(iocb)->ki_flags)
+#define kiocbClearPgBusy(iocb)	clear_bit(KIF_PG_BUSY, &(iocb)->ki_flags)
 
  #define kiocbIsLocked(iocb)	test_bit(KIF_LOCKED, &(iocb)->ki_flags)
  #define kiocbIsKicked(iocb)	test_bit(KIF_KICKED, &(iocb)->ki_flags)
  #define kiocbIsCancelled(iocb)	test_bit(KIF_CANCELLED, &(iocb)->ki_flags)
+#define kiocbIsPgBusy(iocb)	test_bit(KIF_PG_BUSY, &(iocb)->ki_flags)

/* is there a better place to document function pointer methods? */ /**
diff -upr linux-2.6.20/mm/filemap.c linux-2.6.20-aio2/mm/filemap.c

mm/filemap.c +3 -5

@@ -2413,10 +2413,9 @@ generic_file_direct_IO(int rw, struct ki
  		if (rw == WRITE && mapping->nrpages) {
  			pgoff_t end = (offset + write_len - 1)
  						>> PAGE_CACHE_SHIFT;
-			int err = invalidate_inode_pages2_range(mapping,
-					offset >> PAGE_CACHE_SHIFT, end);
-			if (err)
-				retval = err;
+			if (invalidate_inode_pages2_range(mapping,
+					offset >> PAGE_CACHE_SHIFT, end))
+				kiocbSetPgBusy(iocb);
  		}
  	}
  	return retval;
-

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Zach Brown

2007-03-07 22:48 UTC #2 lore raw

Quoted 7 lines · Andrew Morton

Quoted 4 lines · On Mon, 05 Mar 2007 17:23:33 +0300 Leonid Ananiev…
From Leonid Ananiev
The patch fixes oops because of extra IO control block freeing. IO is retried if page could not be invalidated.

This patch is incorrect and shouldn't be merged.

Quoted 11 lines

Quoted 11 lines
Signed-off-by Leonid Ananiev <[email protected]>
The patch fixes oops "Kernel BUG at fs/aio.c:509" archived at
http://lkml.org/lkml/2007/2/8/337
The number of IO control block (iocb)users < 0.
If page could not be invalidated by invalidate_inode_pages2_range() than EIO is returned. It happens if journal_try_to_free_buffers() fails to drop_buffers().
This EIO is not differing from real IO competition with EIO and aio_complete() is called.

(I'm going to read this as "This EIO is misinterpreted as real IO completion with -EIO and aio_complete() is called.")

Quoted 2 lines

Quoted 2 lines
Later aio_complete() is called from dio_bio_end_aio() and frees iocb once more.

This analysis is correct. Nothing can clobber -EIOCBQUEUED as it is returned up from fs/direct-io.c to fs/aio.c. This -EIO from invalidation is one of the two places that currently break this rule.

The fix I had hoped for, invalidating down in fs/direct-io.c before returning -EIOCBQUEUED, doesn't work because it ends up getting the ordering between the journal lock and the page lock backwards. Sigh. (note to self: help lockdep warn us about that ordering)

Quoted 5 lines

Quoted 5 lines
After patch generic_file_direct_IO() sets PgBusy flag in iocb if page could not be invalidated. iocb is retried after IO competition. The process is waked up if IO is SYNC else iocb is kicked. The lines ___if (ret != -EIOCBRETRY)___ is deleted because nothing set to EIOCBRETRY.

True, but this is gratuitously cruel to external users of -EIOCBRETRY. Silently breaking them doesn't sound like a great plan. If we really decide to remove EIOCBRETRY support we'd get rid of all the retry infrastructure and remove the EIOCBRETRY errno so their builds failed. That's a separate issue that shouldn't be confused with this EIOCBQUEUED clobbering. Just removing some pieces of the infrastructure willy-nilly isn't acceptable.

Quoted 6 lines

Please copy [email protected] on AIO patches.
Quoted 1 line
Next patches 2/3 and 3/3 do cleanup only.
I cannot find those patches.

Me either. I was waiting for them to arrive before responding.

Quoted 7 lines

Quoted 7 lines
diff -upr linux-2.6.20/mm/filemap.c linux-2.6.20-aio2/mm/filemap.c
mm/filemap.c +1 -0
+0300
@@ -2413,10 +2413,9 @@ generic_file_direct_IO(int rw, struct ki
  		if (rw == WRITE && mapping->nrpages) {
  			pgoff_t end = (offset + write_len - 1)

Quoted 8 lines

mm/filemap.c generic_file_direct_IO(int rw, struct ki · line 2418

Quoted 8 lines

mm/filemap.c +3 -4

  						>> PAGE_CACHE_SHIFT;
-			int err = invalidate_inode_pages2_range(mapping,
-					offset >> PAGE_CACHE_SHIFT, end);
-			if (err)
-				retval = err;
+			if (invalidate_inode_pages2_range(mapping,
+					offset >> PAGE_CACHE_SHIFT, end))
+				kiocbSetPgBusy(iocb);

There are two problems behind this bug:

- ext3_releasepage() returns failure if it races with kjournald holding a reference while it waits for a transaction to commit.

- generic_file_direct_IO() causes an oops if it clobbers -EIOCBQUEUED with the return code from invalidate_inode_pages2_range()..->releasepage().

This fix makes the incorrect assertion that *any* failure from invalidate_inode_pages2_range(), which might not have anything to do with this race you're currently seeing, is transitory and should trigger a retry. That's wrong, it should be returning the error.

Now, getting ext3_releasepage() to not fail if this race hits to begin with is another story. Chris has some ideas about reworking the page laundering helper to make that more reliable.

- z

Ananiev, Leonid I

2007-03-08 20:19 UTC #3 lore raw

ZB>If we really decide to remove EIOCBRETRY support we'd get rid of all ZB>the retry infrastructure and remove the EIOCBRETRY errno so their ZB>builds failed.

Originally EIOCBRETRY was used in fs/read_write.c for vector IO. And EIOCBRETRY was deleted from it after.
Now EIOCBRETRY is used in drivers/usb/gadget/inode.c only. And the patch 2/3 proposes to set iocb flag instead of EIOCBRETRY errno in inode.c. Agree that the name kiocbSetPgBusy() could be better chosen if kiocbSetPgBusy() is just instead for vector IO. But why other than DIO developers must continue using EIOCBRETRY? Thay could use the same way as fs/read_write.c uses?

ZB> I was waiting for them to arrive before responding. Sorry. I've listed "[email protected]; [email protected]" in a single (that is incorrect) thunderbird 'To' line; I have got response from Stern; and thought that mailing was OK.

ZB>- ext3_releasepage() returns failure if it races with kjournald ZB>holding a reference while it waits for a transaction to commit.

The patch proposes iterative synchronization way to solve this problem.

ZB>- generic_file_direct_IO() causes an oops if it clobbers -EIOCBQUEUED ZB>with the return code from invalidate_inode_pages2_range()..- ZB>releasepage().

After patching invalidate_inode_pages2_range() does not change return value.

ZB>This fix makes the incorrect assertion that *any* failure from ZB>invalidate_inode_pages2_range(), which might not have anything to do ZB>with this race you're currently seeing, is transitory and should ZB>trigger a retry. That's wrong, it should be returning the error.

Just patch makes to retry iocb if page is transitory busy for any reason.
If busy -> do retry IO operation later.
We will get correct errno if mapping was deleted or retry success if page is dirty or committed.

Leonid