Skip to contentSkip to conversation

next-20130117 - kernel BUG with aio

PATCH In review Valdis Kletnieks 7 participants 26 messages Jan 21, 2013 – Feb 6, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Am seeing a reproducible BUG in the kernel with next-20130117 whenever I fire up VirtualBox. Unfortunately, I hadn't done that in a while, so the last 'known good' kernel was next-20121203.

I'm strongly suspecting one of Kent Overstreet's 32 patches against aio, because 'git blame' shows those landing on Jan 12, and not much else happening to fs/aio.c in ages.

The stack traceback ring any bells before I go to bisect this?

[  327.375581] BUG: sleeping function called from invalid context at arch/x86/mm/fault.c:1138
[  327.375588] in_atomic(): 0, irqs_disabled(): 1, pid: 2096, name: AioMgr0-N
[  327.375590] INFO: lockdep is turned off.
[  327.375593] irq event stamp: 0
[  327.375595] hardirqs last  enabled at (0): [<          (null)>]           (null)
[  327.375599] hardirqs last disabled at (0): [<ffffffff8102d9eb>] copy_process.part.40+0x565/0x14be
[  327.375607] softirqs last  enabled at (0): [<ffffffff8102d9eb>] copy_process.part.40+0x565/0x14be
[  327.375611] softirqs last disabled at (0): [<          (null)>]           (null)
[  327.375616] Pid: 2096, comm: AioMgr0-N Tainted: P           O 3.8.0-rc3-next-20130117-dirty #49
[  327.375618] Call Trace:
[  327.375624]  [<ffffffff810770ba>] ? print_irqtrace_events+0x9d/0xa1
[  327.375630]  [<ffffffff8105a576>] __might_sleep+0x19f/0x1a7
[  327.375635]  [<ffffffff81617ab4>] __do_page_fault+0x2a4/0x57c
[  327.375641]  [<ffffffff810dbb55>] ? invalidate_inode_pages2_range+0x2e0/0x2f8
[  327.375645]  [<ffffffff811843f4>] ? ext4_direct_IO+0x224/0x3c2
[  327.375650]  [<ffffffff81186438>] ? noalloc_get_block_write+0x57/0x57
[  327.375654]  [<ffffffff81182c4d>] ? ext4_readpages+0x41/0x41
[  327.375659]  [<ffffffff810b7caf>] ? time_hardirqs_off+0x1b/0x2f
[  327.375663]  [<ffffffff81615373>] ? error_sti+0x5/0x6
[  327.375667]  [<ffffffff8107522f>] ? trace_hardirqs_off_caller+0x1f/0x9e
[  327.375672]  [<ffffffff8124ad2d>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[  327.375676]  [<ffffffff81617d95>] do_page_fault+0x9/0xb
[  327.375680]  [<ffffffff81615182>] page_fault+0x22/0x30
[  327.375685]  [<ffffffff811522af>] ? kioctx_ring_unlock+0xd/0x5f
[  327.375689]  [<ffffffff811524c7>] batch_complete_aio+0x1c6/0x212
[  327.375694]  [<ffffffff8117fc63>] ? ext4_unwritten_wait+0x98/0x98
[  327.375697]  [<ffffffff81152b3a>] aio_complete_batch+0x125/0x132
[  327.375702]  [<ffffffff8117fc63>] ? ext4_unwritten_wait+0x98/0x98
[  327.375705]  [<ffffffff81153931>] do_io_submit+0x781/0x84b
[  327.375710]  [<ffffffff81153a06>] sys_io_submit+0xb/0xd
[  327.375715]  [<ffffffff8161b0d2>] system_call_fastpath+0x16/0x1b

(and that BUG cascades into a second one:

[  327.375724] BUG: unable to handle kernel NULL pointer dereference at 0000000000000250
[  327.375729] IP: [<ffffffff811522af>] kioctx_ring_unlock+0xd/0x5f
[  327.375733] PGD d0d36067 PUD da749067 PMD 0
[  327.375740] Oops: 0002 [#1] PREEMPT SMP
...
[  327.375829] Call Trace:
[  327.375833]  [<ffffffff811524c7>] batch_complete_aio+0x1c6/0x212
[  327.375838]  [<ffffffff8117fc63>] ? ext4_unwritten_wait+0x98/0x98
[  327.375842]  [<ffffffff81152b3a>] aio_complete_batch+0x125/0x132
[  327.375846]  [<ffffffff8117fc63>] ? ext4_unwritten_wait+0x98/0x98
[  327.375850]  [<ffffffff81153931>] do_io_submit+0x781/0x84b
[  327.375855]  [<ffffffff81153a06>] sys_io_submit+0xb/0xd
[  327.375859]  [<ffffffff8161b0d2>] system_call_fastpath+0x16/0x1b
[  327.375861] Code: 00 50 48 8d 5f 90 48 81 c7 98 01 00 00 e8 0e 9c f0 ff 48 89 df e8 87 fd ff ff 58 5b 5d c3 55 48 89 e5 41 54 41 89 f4 53
48 89 fb <89> b3 50 02 00 00 48 8b 47 50 48 8b 38 e8 37 f9 ff ff 44 89 60
[  327.375937] RIP  [<ffffffff811522af>] kioctx_ring_unlock+0xd/0x5f
[  327.375942]  RSP <ffff8800b8965db8>
[  327.375944] CR2: 0000000000000250
[  327.375949] ---[ end trace b119850056dcfba4 ]---
#1 lore raw
Quoted 8 lines · Valdis Kletnieks

Am seeing a reproducible BUG in the kernel with next-20130117 whenever I fire up VirtualBox. Unfortunately, I hadn't done that in a while, so the last 'known good' kernel was next-20121203.

I'm strongly suspecting one of Kent Overstreet's 32 patches against aio, because 'git blame' shows those landing on Jan 12, and not much else happening to fs/aio.c in ages.

Take a try?
---

fs/aio.c +3 -0
@@ -683,6 +683,9 @@ static inline void kioctx_ring_unlock(st
 {
 	struct aio_ring *ring;
 
+	if (!ctx)
+		return;
+
 	smp_wmb();
 	/* make event visible before updating tail */
 

--

#2 lore raw
Quoted 15 lines · Hillf Danton
Quoted 8 lines · Valdis Kletnieks

Am seeing a reproducible BUG in the kernel with next-20130117 whenever I fire up VirtualBox. Unfortunately, I hadn't done that in a while, so the last 'known good' kernel was next-20121203.

I'm strongly suspecting one of Kent Overstreet's 32 patches against aio, because 'git blame' shows those landing on Jan 12, and not much else happening to fs/aio.c in ages.

Take a try?
---

fs/aio.c static inline void kioctx_ring_unlock(st · line 690

Quoted 8 lines
fs/aio.c +3 -0
@@ -683,6 +683,9 @@ static inline void kioctx_ring_unlock(st
 {
 	struct aio_ring *ring;
 
+	if (!ctx)
+		return;
+
 	smp_wmb();
 	/* make event visible before updating tail */

Well, things are improved - at least now it doesn't BUG :)

[  534.879083] ------------[ cut here ]------------
[  534.879094] WARNING: at fs/aio.c:336 put_ioctx+0x1cb/0x252()
[  534.879121] Call Trace:
[  534.879129]  [<ffffffff8102f5ad>] warn_slowpath_common+0x7e/0x97
[  534.879133]  [<ffffffff8102f5db>] warn_slowpath_null+0x15/0x17
[  534.879137]  [<ffffffff811521f0>] put_ioctx+0x1cb/0x252
[  534.879142]  [<ffffffff8105bee3>] ? __wake_up+0x3f/0x48
[  534.879146]  [<ffffffff8115229e>] ? kill_ioctx_work+0x27/0x2b
[  534.879150]  [<ffffffff811531a5>] sys_io_destroy+0x40/0x50
[  534.879156]  [<ffffffff8161b112>] system_call_fastpath+0x16/0x1b
[  534.879159] ---[ end trace a2c46a8bc9058404 ]---

Hopefully that tells you and Kent something. :)

#3 lore raw
Quoted 19 lines · On Wed, Jan 23, 2013 at 5:28 AM, <[email protected]> wrote:
Quoted 18 lines · Hillf Danton
Quoted 8 lines · Valdis Kletnieks

Am seeing a reproducible BUG in the kernel with next-20130117 whenever I fire up VirtualBox. Unfortunately, I hadn't done that in a while, so the last 'known good' kernel was next-20121203.

I'm strongly suspecting one of Kent Overstreet's 32 patches against aio, because 'git blame' shows those landing on Jan 12, and not much else happening to fs/aio.c in ages.

Take a try?
---

fs/aio.c Tue Jan 22 21:43:58 2013 renamed from fs/aio.c Tue Jan 22 21:37:54 2013 +0 -0
@@ -683,6 +683,9 @@ static inline void kioctx_ring_unlock(st
 {
      struct aio_ring *ring;
 
Quoted 7 lines

fs/aio.c Tue Jan 22 21:43:58 2013 static inline void kioctx_ring_unlock(st · line 690

Quoted 5 lines
fs/aio.c Tue Jan 22 21:43:58 2013 renamed from fs/aio.c Tue Jan 22 21:37:54 2013 +3 -0
+     if (!ctx)
+             return;
+
      smp_wmb();
      /* make event visible before updating tail */

Well, things are improved - at least now it doesn't BUG :)

Good news ;)

Quoted 13 lines
[  534.879083] ------------[ cut here ]------------
[  534.879094] WARNING: at fs/aio.c:336 put_ioctx+0x1cb/0x252()
[  534.879121] Call Trace:
[  534.879129]  [<ffffffff8102f5ad>] warn_slowpath_common+0x7e/0x97
[  534.879133]  [<ffffffff8102f5db>] warn_slowpath_null+0x15/0x17
[  534.879137]  [<ffffffff811521f0>] put_ioctx+0x1cb/0x252
[  534.879142]  [<ffffffff8105bee3>] ? __wake_up+0x3f/0x48
[  534.879146]  [<ffffffff8115229e>] ? kill_ioctx_work+0x27/0x2b
[  534.879150]  [<ffffffff811531a5>] sys_io_destroy+0x40/0x50
[  534.879156]  [<ffffffff8161b112>] system_call_fastpath+0x16/0x1b
[  534.879159] ---[ end trace a2c46a8bc9058404 ]---
Quoted 1 line

Hopefully that tells you and Kent something. :)

Try again?
---

fs/aio.c +13 -8
@@ -683,6 +683,9 @@ static inline void kioctx_ring_unlock(st
 {
 	struct aio_ring *ring;
 
+	if (!ctx)
+		return;
+
 	smp_wmb();
 	/* make event visible before updating tail */
 
@@ -723,6 +726,7 @@ void batch_complete_aio(struct batch_com
 	n = rb_first(&batch->kiocb);
 	while (n) {
 		struct kiocb *req = container_of(n, struct kiocb, ki_node);
+		int cancelled = 0;
 
 		if (n->rb_right) {
 			n->rb_right->__rb_parent_color = n->__rb_parent_color;
@@ -736,13 +740,8 @@ void batch_complete_aio(struct batch_com
 
 		if (unlikely(xchg(&req->ki_cancel,
 				  KIOCB_CANCELLED) == KIOCB_CANCELLED)) {
-			/*
-			 * Can't use the percpu reqs_available here - could race
-			 * with free_ioctx()
-			 */
-			atomic_inc(&req->ki_ctx->reqs_available);
-			aio_put_req(req);
-			continue;
+			cancelled = 1;
+			goto lock;
 		}
 
 		if (unlikely(req->ki_eventfd != eventfd)) {
@@ -759,6 +758,7 @@ void batch_complete_aio(struct batch_com
 			req->ki_eventfd = NULL;
 		}
 
+	lock:
 		if (unlikely(req->ki_ctx != ctx)) {
 			if (ctx)
 				kioctx_ring_unlock(ctx, tail);
@@ -767,7 +767,12 @@ void batch_complete_aio(struct batch_com
 			tail = kioctx_ring_lock(ctx);
 		}
 
-		tail = kioctx_ring_put(ctx, req, tail);
+		if (cancelled) {
+			if (++tail >= ctx->nr)
+				tail = 0;
+		} else
+			tail = kioctx_ring_put(ctx, req, tail);
+
 		aio_put_req(req);
 	}
 

--

#4 lore raw

fs/aio.c

Quoted 5 lines · Hillf Danton

Try again?
---

fs/aio.c +0 -0

Now seeing this:

[ 2941.495370] ------------[ cut here ]------------
[ 2941.495379] WARNING: at fs/aio.c:336 put_ioctx+0x1cb/0x252()

Same warn location, but different traceback?

[ 2941.495407] Call Trace:
[ 2941.495415]  [<ffffffff8102f5ad>] warn_slowpath_common+0x7e/0x97
[ 2941.495419]  [<ffffffff8102f5db>] warn_slowpath_null+0x15/0x17
[ 2941.495423]  [<ffffffff8115247b>] put_ioctx+0x1cb/0x252
[ 2941.495428]  [<ffffffff81617fc3>] ? sub_preempt_count+0x33/0x46
[ 2941.495433]  [<ffffffff81050448>] ? abort_exclusive_wait+0x89/0x89
[ 2941.495438]  [<ffffffff81152529>] kill_ioctx_work+0x27/0x2b
[ 2941.495443]  [<ffffffff81603ded>] process_one_work+0x26f/0x4be
[ 2941.495447]  [<ffffffff81603d27>] ? process_one_work+0x1a9/0x4be
[ 2941.495453]  [<ffffffff810483c5>] ? spin_lock_irq+0x9/0xb
[ 2941.495457]  [<ffffffff8104b0f9>] worker_thread+0x1b6/0x28e
[ 2941.495461]  [<ffffffff8104af43>] ? rescuer_thread+0x1a5/0x1a5
[ 2941.495466]  [<ffffffff8104f7cc>] kthread+0x9d/0xa5
[ 2941.495471]  [<ffffffff8104f72f>] ? __kthread_parkme+0x60/0x60
[ 2941.495475]  [<ffffffff8161b06c>] ret_from_fork+0x7c/0xb0
[ 2941.495479]  [<ffffffff8104f72f>] ? __kthread_parkme+0x60/0x60
[ 2941.495483] ---[ end trace 8e545cd216c853ec ]---

Obviously VirtualBox 4.2.6 on my laptop is going out of its way to get indigestion at this patch series. :)

#5 lore raw
Quoted 10 lines · On Thu, Jan 24, 2013 at 12:22:21PM -0500, [email protected] wrote:

fs/aio.c

Quoted 5 lines · Hillf Danton

Try again?
---

fs/aio.c +0 -0

Now seeing this:

Quoted 4 lines
[ 2941.495370] ------------[ cut here ]------------
[ 2941.495379] WARNING: at fs/aio.c:336 put_ioctx+0x1cb/0x252()

Same warn location, but different traceback?

Finally reproduced it (thanks to Zach Brown for the idea - using a loopback device) - it's triggered when there's outstanding kiocbs when we're trying to tear down the kioctx.

Found a couple bugs once I was able to reproduce it. Besides the null pointer deref that Hillf posted a patch for, cancellation was fubar - kiocb_cancel() shouldn't be marking the kiocb as cancelled if it didn't have a cancel callback.

The other two bugs I found were both a result of the fact that aio_run_iocb() touches the kiocb after passing it off to a method that may call aio_complete() on it - which is something I originally missed.

Digression: When I was refactoring, I was hoping to be able to change the kiocb refcounting so that the refcount's just initialized to one, and the initial refcount is dropped when aio_complete() is called - and since nothing outside of the aio code messes with the kiocb refcount, it might be possible to get rid of the kiocb refcount entirely if I can figure out how to deal with cancellation.

Anyways - that didn't work out, or at least it's going to take more work. The two bugs that resulted from that were:

- The "aio: kill ki_retry" patch dropped the second kiocb ref that io_submit_one drops when it returns. But aio_rw_vect_retry() touches the kiocb after passing it off to f_op->aio_(read|write) which will call aio_complete(), so this is a use after free.

- The "aio: smoosh struct kiocb" patch assumed that some of the fields in struct kiocb aren't needed after aio_complete() has been called. This is _almost_ true, but again aio_rw_vect_retry() looks at those fields which ends up determining whether aio_run_iocb() calls aio_complete() itself.

This seems _ridiculously_ sketchy to me, or at least convoluted... but it'll take awhile to figure out how to clean that up and I'm not in a great hurry to do so.

So, Andrew - that "smoosh struct kiocb" patch should just be dropped, even if I fixed that issue clearly the idea is a lot less safe than I thought. I've got patches for the other stuff I'm going to mail out momentarily.

Ben, Zach - the cancellation stuff (both the fix and the other changes) need more review, and we need a saner way of testing cancellation. Maybe it'd be worth implementing cancellation for regular block devices just so that we have a way of testing it :/

#6 lore raw
Quoted 4 lines · On Thu, 24 Jan 2013 13:18:50 -0800 Kent Overstreet <[email protected]>…

So, Andrew - that "smoosh struct kiocb" patch should just be dropped, even if I fixed that issue clearly the idea is a lot less safe than I thought. I've got patches for the other stuff I'm going to mail out momentarily.

Dropped. Do you expect that this will fix everything?

Please cc linux-fsdevel on the aio stuff - I'm seeing some AIO things over there which aren't cc'ed to lkml or linux-aio.

Please also take a look at Jan's recent
http://www.spinics.net/lists/linux-fsdevel/msg61738.html and have a think about how this plays with your patchset.

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>

#7 lore raw
Quoted 9 lines · Andrew Morton
Quoted 4 lines · On Thu, 24 Jan 2013 13:18:50 -0800 Kent Overstreet <[email protected]>…

So, Andrew - that "smoosh struct kiocb" patch should just be dropped, even if I fixed that issue clearly the idea is a lot less safe than I thought. I've got patches for the other stuff I'm going to mail out momentarily.

Dropped. Do you expect that this will fix everything?

No, I didn't see that bug until after I'd fixed the other three, but as far as I can tell everything's fixed with the patches I'm about to mail out - my test VM has been running for the past two days without errors, it's kill -9'ing a process that's got iocbs in flight to a loopback device every two seconds.

Quoted 2 lines

Please cc linux-fsdevel on the aio stuff - I'm seeing some AIO things over there which aren't cc'ed to lkml or linux-aio.

Will do

Quoted 3 lines

Please also take a look at Jan's recent
http://www.spinics.net/lists/linux-fsdevel/msg61738.html and have a think about how this plays with your patchset.

Oh fun, I've chased a bug or two in that ext4 code, that stuff's scary... I'm sure it'll take me a bit to wrap my head around what's going on there but I'm looking at it now.

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>

#8 lore raw
Quoted 5 lines

No, I didn't see that bug until after I'd fixed the other three, but as far as I can tell everything's fixed with the patches I'm about to mail out - my test VM has been running for the past two days without errors, it's kill -9'ing a process that's got iocbs in flight to a loopback device every two seconds.

I'm really worried that this patch series hasn't seen significant enough testing to justify being queued.

I'll be first in line for blame for not finding the time to finish my review of the series.

What specific tests has this gone through? The aio tests in xfstests / ltp? (And as you discovered while chasing this bug, whatever platform you were running on doesn't seem slow enough to catch some paths.. run all the tests over loop?)

Jeff, can you suggest a more modern testing regime for the aio core? It's been so long since I had to hammer on this stuff..

- z

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>

#9 lore raw
Quoted 12 lines · Zach Brown
Quoted 5 lines

No, I didn't see that bug until after I'd fixed the other three, but as far as I can tell everything's fixed with the patches I'm about to mail out - my test VM has been running for the past two days without errors, it's kill -9'ing a process that's got iocbs in flight to a loopback device every two seconds.

I'm really worried that this patch series hasn't seen significant enough testing to justify being queued.

I'll be first in line for blame for not finding the time to finish my review of the series.

Quoted 7 lines

What specific tests has this gone through? The aio tests in xfstests / ltp? (And as you discovered while chasing this bug, whatever platform you were running on doesn't seem slow enough to catch some paths.. run all the tests over loop?)

Jeff, can you suggest a more modern testing regime for the aio core? It's been so long since I had to hammer on this stuff..

Modern? No. ;-) I usually use xfstests (all of them, not just the aio group), the libaio test harness, and then hand it off to our performance team to stress the code under benchmarking workloads. Oh, and usually targeted testing for the thing that I'm working on.

I'll put a couple of kernels together to hand off to our performance team, though I don't know how much time they have at present.

Cheers,
Jeff

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>

#10 lore raw
In reply to Zach Brown
Quoted 9 lines · Zach Brown
Quoted 5 lines

No, I didn't see that bug until after I'd fixed the other three, but as far as I can tell everything's fixed with the patches I'm about to mail out - my test VM has been running for the past two days without errors, it's kill -9'ing a process that's got iocbs in flight to a loopback device every two seconds.

I'm really worried that this patch series hasn't seen significant enough testing to justify being queued.

Quoted 7 lines

I'll be first in line for blame for not finding the time to finish my review of the series.

What specific tests has this gone through? The aio tests in xfstests / ltp? (And as you discovered while chasing this bug, whatever platform you were running on doesn't seem slow enough to catch some paths.. run all the tests over loop?)

I have run xfstests on real hardware - though that didn't catch this either. These patches are all queued up for our internal tree (I need to bug Ted to review them... and add these latest fixes). And aio is used heavily enough internally that if I screwed anything up, I'll be on the hook...

Quoted 2 lines

Jeff, can you suggest a more modern testing regime for the aio core? It's been so long since I had to hammer on this stuff..

I'm wondering how much it'd buy us to rig up fault injection (I've got some awesome dynamic fault injection code I need to push upstream...). I'll try and test with that in the next day or so, though.

There's some people here that'd been working on code coverage tooling too, I should probably learn how to work that stuff.

#11 lore raw
In reply to Andrew Morton
Quoted 3 lines · Andrew Morton

Please also take a look at Jan's recent
http://www.spinics.net/lists/linux-fsdevel/msg61738.html and have a think about how this plays with your patchset.

I can't think of any possible interactions - none of my aio stuff messes with the way the fput() happens; the aio code does call fput() when the kiocb is freed and my patches do touch _that_ code but none of the behaviour there changes.

Might be worth documenting this though, I can't think of any reason it'd be obvious looking at the aio code that the fput() has to happen after aio_complete(). As with the bugs I just sent you patches for it's not terribly clear who owns what in the kiocb when.

Reading those patches though - the main change is to call inode_dio_done() before calling aio_complete(). All inode_dio_done() does though is issue a wakeup - to whatever called inode_dio_wait().

That means whatever called inode_dio_wait() needs its own ref on the inode, and from a cursory glance at the code it is _not_ at all clear to me that's the case - if inode_dio_wait() is merely finishing things for that specific IO that need to be done in process context, I can easily imagine it not being the case.

Assuming whatever does call inode_dio_wait() does have its own ref, there was only a real use after free when nothing was waiting on the inode.

Similarly for the ext4 code to write unwritten extents - and having seen and helped chase a bug in that code before, that code _definitely_ needs auditing.

reviewer
#12 lore raw
Quoted 10 lines · Kent Overstreet
Quoted 3 lines · Andrew Morton

Please also take a look at Jan's recent
http://www.spinics.net/lists/linux-fsdevel/msg61738.html and have a think about how this plays with your patchset.

I can't think of any possible interactions - none of my aio stuff messes with the way the fput() happens; the aio code does call fput() when the kiocb is freed and my patches do touch _that_ code but none of the behaviour there changes.

Quoted 8 lines

Might be worth documenting this though, I can't think of any reason it'd be obvious looking at the aio code that the fput() has to happen after aio_complete(). As with the bugs I just sent you patches for it's not terribly clear who owns what in the kiocb when.

Reading those patches though - the main change is to call inode_dio_done() before calling aio_complete(). All inode_dio_done() does though is issue a wakeup - to whatever called inode_dio_wait().

inode_dio_done() does a decrement and wakeup.

Quoted 9 lines

That means whatever called inode_dio_wait() needs its own ref on the inode, and from a cursory glance at the code it is _not_ at all clear to me that's the case - if inode_dio_wait() is merely finishing things for that specific IO that need to be done in process context, I can easily imagine it not being the case.

Assuming whatever does call inode_dio_wait() does have its own ref, there was only a real use after free when nothing was waiting on the inode.

Well, but there doesn't have to be any waiter... If there is, it had
better have it's own ref, that's for sure.

Quoted 3 lines

Similarly for the ext4 code to write unwritten extents - and having seen and helped chase a bug in that code before, that code _definitely_ needs auditing.

Agreed. That code is a mess. I'm cleaning up some of it but it's not
easy.

Honza

Signature · 8 lines
Jan Kara <[email protected]>
SUSE Labs, CR

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to [email protected].  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
#13 lore raw

PATCH 1/3 aio: Fix a null pointer deref in batch_complete_aio

In reply to Kent Overstreet

The batch completion code was trying to be a bit too clever, and skip checking ctx where it couldn't be NULL - but that broke if a kiocb had been cancelled. Move the check to kioctx_ring_unlock().

---
 fs/aio.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
fs/aio.c +4 -2
index 62573d3..0d2f39d 100644
@@ -683,6 +683,9 @@ static inline void kioctx_ring_unlock(struct kioctx *ctx, unsigned tail)
 {
 	struct aio_ring *ring;
 
+	if (!ctx)
+		return;
+
 	smp_wmb();
 	/* make event visible before updating tail */
 
@@ -760,8 +763,7 @@ void batch_complete_aio(struct batch_complete *batch)
 		}
 
 		if (unlikely(req->ki_ctx != ctx)) {
-			if (ctx)
-				kioctx_ring_unlock(ctx, tail);
+			kioctx_ring_unlock(ctx, tail);
 
 			ctx = req->ki_ctx;
 			tail = kioctx_ring_lock(ctx);
Signature · 7 lines
1.7.12

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to [email protected].  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
#14 lore raw

PATCH 2/3 aio-kill-ki_retry-fix-fix

The "aio: kill ki-retry" patch was assuming that we didn't touch struct kiocb after passing it off to something that would call aio_complete() - which was wrong. So, revert the refcounting changes.

---
 fs/aio.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
fs/aio.c +4 -2
index 0d2f39d..85d1b38 100644
@@ -592,7 +592,7 @@ static inline struct kiocb *aio_get_req(struct kioctx *ctx)
 
 	memset(req, 0, offsetof(struct kiocb, ki_ctx));
 	req->ki_ctx = ctx;
-	atomic_set(&req->ki_users, 1);
+	atomic_set(&req->ki_users, 2);
 	return req;
 out_put:
 	put_reqs_available(ctx, 1);
@@ -1291,10 +1291,12 @@ static int io_submit_one(struct kioctx *ctx, struct iocb __user *user_iocb,
 	if (ret)
 		goto out_put_req;
 
+	aio_put_req(req);	/* drop extra ref to req */
 	return 0;
 out_put_req:
 	put_reqs_available(ctx, 1);
-	aio_put_req(req);
+	aio_put_req(req);	/* drop extra ref to req */
+	aio_put_req(req);	/* drop i/o ref to req */
 	return ret;
 }
 
Signature · 7 lines
1.7.12

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to [email protected].  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
#15 lore raw

PATCH 3/3 aio-use-cancellation-list-lazily-fix

In reply to Kent Overstreet

The cancellation changes were fubar - we can't cancel a kiocb if it doesn't actually have a cancellation callback.

The use of xchg() in aio_complete() was right - there we're marking the kiocb as completed - but we need to use cmpxchg() in kiocb_cancel() - a lock isn't sufficient since we're synchronizing with aio_complete() which isn't taking any locks.

---
 fs/aio.c            | 32 ++++++++++++++++++++++----------
 include/linux/aio.h | 11 +++++++++++
 2 files changed, 33 insertions(+), 10 deletions(-)
fs/aio.c +22 -10
index 85d1b38..2760180 100644
@@ -244,28 +244,40 @@ static int aio_setup_ring(struct kioctx *ctx)
 
 void kiocb_set_cancel_fn(struct kiocb *req, kiocb_cancel_fn *cancel)
 {
-	if (!req->ki_list.next) {
-		struct kioctx *ctx = req->ki_ctx;
-		unsigned long flags;
+	struct kioctx *ctx = req->ki_ctx;
+	unsigned long flags;
 
-		spin_lock_irqsave(&ctx->ctx_lock, flags);
+	spin_lock_irqsave(&ctx->ctx_lock, flags);
+
+	if (!req->ki_list.next)
 		list_add(&req->ki_list, &ctx->active_reqs);
-		spin_unlock_irqrestore(&ctx->ctx_lock, flags);
-	}
 
 	req->ki_cancel = cancel;
+
+	spin_unlock_irqrestore(&ctx->ctx_lock, flags);
 }
 EXPORT_SYMBOL(kiocb_set_cancel_fn);
 
 static int kiocb_cancel(struct kioctx *ctx, struct kiocb *kiocb,
 			struct io_event *res)
 {
-	kiocb_cancel_fn *cancel;
+	kiocb_cancel_fn *old, *cancel;
 	int ret = -EINVAL;
 
-	cancel = xchg(&kiocb->ki_cancel, KIOCB_CANCELLED);
-	if (!cancel || cancel == KIOCB_CANCELLED)
-		return ret;
+	/*
+	 * Don't want to set kiocb->ki_cancel = KIOCB_CANCELLED unless it
+	 * actually has a cancel function, hence the cmpxchg()
+	 */
+
+	cancel = ACCESS_ONCE(kiocb->ki_cancel);
+	do {
+		if (!cancel || cancel == KIOCB_CANCELLED)
+			return ret;
+
+		BUG();
+		old = cancel;
+		cancel = cmpxchg(&kiocb->ki_cancel, old, KIOCB_CANCELLED);
+	} while (cancel != old);
 
 	atomic_inc(&kiocb->ki_users);
 	spin_unlock_irq(&ctx->ctx_lock);
include/linux/aio.h +11 -0
index 8eaa490..cd4aea0 100644
@@ -15,6 +15,17 @@ struct batch_complete;
 
 #define KIOCB_KEY		0
 
+/*
+ * We use ki_cancel == KIOCB_CANCELLED to indicate that a kiocb has been either
+ * cancelled or completed (this makes a certain amount of sense because
+ * successful cancellation - io_cancel() - does deliver the completion to
+ * userspace).
+ *
+ * And since most things don't implement kiocb cancellation and we'd really like
+ * kiocb completion to be lockless when possible, we use ki_cancel to
+ * synchronize cancellation and completion - we only set it to KIOCB_CANCELLED
+ * with xchg() or cmpxchg(), see batch_complete_aio() and kiocb_cancel().
+ */
 #define KIOCB_CANCELLED		((void *) (~0ULL))
 
 typedef int (kiocb_cancel_fn)(struct kiocb *, struct io_event *);
Signature · 7 lines
1.7.12

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in
the body to [email protected].  For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>
#16 lore raw

Re: [PATCH 3/3] aio-use-cancellation-list-lazily-fix

Quoted 8 lines · Kent Overstreet

The cancellation changes were fubar - we can't cancel a kiocb if it doesn't actually have a cancellation callback.

The use of xchg() in aio_complete() was right - there we're marking the kiocb as completed - but we need to use cmpxchg() in kiocb_cancel() - a lock isn't sufficient since we're synchronizing with aio_complete() which isn't taking any locks.

What is observed without this fix?

Quoted 45 lines
---
 fs/aio.c            | 32 ++++++++++++++++++++++----------
 include/linux/aio.h | 11 +++++++++++
 2 files changed, 33 insertions(+), 10 deletions(-)
fs/aio.c +9 -10
index 85d1b38..2760180 100644
@@ -244,28 +244,40 @@ static int aio_setup_ring(struct kioctx *ctx)
 
 void kiocb_set_cancel_fn(struct kiocb *req, kiocb_cancel_fn *cancel)
 {
-       if (!req->ki_list.next) {
-               struct kioctx *ctx = req->ki_ctx;
-               unsigned long flags;
+       struct kioctx *ctx = req->ki_ctx;
+       unsigned long flags;
 
-               spin_lock_irqsave(&ctx->ctx_lock, flags);
+       spin_lock_irqsave(&ctx->ctx_lock, flags);
+
+       if (!req->ki_list.next)
                list_add(&req->ki_list, &ctx->active_reqs);
-               spin_unlock_irqrestore(&ctx->ctx_lock, flags);
-       }
 
        req->ki_cancel = cancel;
+
+       spin_unlock_irqrestore(&ctx->ctx_lock, flags);
 }
 EXPORT_SYMBOL(kiocb_set_cancel_fn);
 
 static int kiocb_cancel(struct kioctx *ctx, struct kiocb *kiocb,
                        struct io_event *res)
 {
-       kiocb_cancel_fn *cancel;
+       kiocb_cancel_fn *old, *cancel;
        int ret = -EINVAL;
 
-       cancel = xchg(&kiocb->ki_cancel, KIOCB_CANCELLED);
-       if (!cancel || cancel == KIOCB_CANCELLED)
-               return ret;
+       /*

fs/aio.c static int aio_setup_ring(struct kioctx *ctx) · line 277

Quoted 10 lines
fs/aio.c +10 -0
+        * Don't want to set kiocb->ki_cancel = KIOCB_CANCELLED unless it
+        * actually has a cancel function, hence the cmpxchg()
+        */
+
+       cancel = ACCESS_ONCE(kiocb->ki_cancel);
+       do {
+               if (!cancel || cancel == KIOCB_CANCELLED)
+                       return ret;
+
+               BUG();

Hmm, what is trapped?

Quoted 29 lines
fs/aio.c +3 -0
+               old = cancel;
+               cancel = cmpxchg(&kiocb->ki_cancel, old, KIOCB_CANCELLED);
+       } while (cancel != old);
 
        atomic_inc(&kiocb->ki_users);
        spin_unlock_irq(&ctx->ctx_lock);
include/linux/aio.h +11 -1
index 8eaa490..cd4aea0 100644
@@ -15,6 +15,17 @@ struct batch_complete;
 
 #define KIOCB_KEY              0
 
+/*
+ * We use ki_cancel == KIOCB_CANCELLED to indicate that a kiocb has been either
+ * cancelled or completed (this makes a certain amount of sense because
+ * successful cancellation - io_cancel() - does deliver the completion to
+ * userspace).
+ *
+ * And since most things don't implement kiocb cancellation and we'd really like
+ * kiocb completion to be lockless when possible, we use ki_cancel to
+ * synchronize cancellation and completion - we only set it to KIOCB_CANCELLED
+ * with xchg() or cmpxchg(), see batch_complete_aio() and kiocb_cancel().
+ */
 #define KIOCB_CANCELLED                ((void *) (~0ULL))
 
 typedef int (kiocb_cancel_fn)(struct kiocb *, struct io_event *);
--
Quoted 2 lines

1.7.12

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>

#17 lore raw

Re: [PATCH 3/3] aio-use-cancellation-list-lazily-fix

Quoted 16 lines · On Fri, 25 Jan 2013 21:30:32 +0800 Hillf Danton <[email protected]> wrote:
Quoted 15 lines · Kent Overstreet

The cancellation changes were fubar - we can't cancel a kiocb if it doesn't actually have a cancellation callback.

The use of xchg() in aio_complete() was right - there we're marking the kiocb as completed - but we need to use cmpxchg() in kiocb_cancel() - a lock isn't sufficient since we're synchronizing with aio_complete() which isn't taking any locks.

static int kiocb_cancel(struct kioctx *ctx, struct kiocb *kiocb,
struct io_event *res)
{

fs/aio.c +1 -1
-       kiocb_cancel_fn *cancel;
+       kiocb_cancel_fn *old, *cancel;
        int ret = -EINVAL;
Quoted 20 lines
Quoted 4 lines
fs/aio.c +1 -3
-       cancel = xchg(&kiocb->ki_cancel, KIOCB_CANCELLED);
-       if (!cancel || cancel == KIOCB_CANCELLED)
-               return ret;
+       /*

fs/aio.c static int aio_setup_ring(struct kioctx *ctx) · line 277

Quoted 10 lines
fs/aio.c +10 -0
+        * Don't want to set kiocb->ki_cancel = KIOCB_CANCELLED unless it
+        * actually has a cancel function, hence the cmpxchg()
+        */
+
+       cancel = ACCESS_ONCE(kiocb->ki_cancel);
+       do {
+               if (!cancel || cancel == KIOCB_CANCELLED)
+                       return ret;
+
+               BUG();

Hmm, what is trapped?

Quoted 3 lines
fs/aio.c +3 -0
+               old = cancel;
+               cancel = cmpxchg(&kiocb->ki_cancel, old, KIOCB_CANCELLED);
+       } while (cancel != old);

erk, I missed that. What earthly sense is there in putting a BUG() in that place.

I think I'll delete it and pretend I never saw it :(

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>

#18 lore raw

Re: [PATCH 3/3] aio-use-cancellation-list-lazily-fix

Quoted 36 lines · Andrew Morton
Quoted 33 lines · On Fri, 25 Jan 2013 21:30:32 +0800 Hillf Danton <[email protected]> wrote:
Quoted 19 lines · Kent Overstreet

The cancellation changes were fubar - we can't cancel a kiocb if it doesn't actually have a cancellation callback.

The use of xchg() in aio_complete() was right - there we're marking the kiocb as completed - but we need to use cmpxchg() in kiocb_cancel() - a lock isn't sufficient since we're synchronizing with aio_complete() which isn't taking any locks.

static int kiocb_cancel(struct kioctx *ctx, struct kiocb *kiocb,
struct io_event *res)
{

fs/aio.c +2 -4
-       kiocb_cancel_fn *cancel;
+       kiocb_cancel_fn *old, *cancel;
        int ret = -EINVAL;
 
-       cancel = xchg(&kiocb->ki_cancel, KIOCB_CANCELLED);
-       if (!cancel || cancel == KIOCB_CANCELLED)
-               return ret;
+       /*

fs/aio.c static int aio_setup_ring(struct kioctx *ctx) · line 277

Quoted 10 lines
fs/aio.c +10 -0
+        * Don't want to set kiocb->ki_cancel = KIOCB_CANCELLED unless it
+        * actually has a cancel function, hence the cmpxchg()
+        */
+
+       cancel = ACCESS_ONCE(kiocb->ki_cancel);
+       do {
+               if (!cancel || cancel == KIOCB_CANCELLED)
+                       return ret;
+
+               BUG();

Hmm, what is trapped?

Quoted 8 lines
Quoted 3 lines

fs/aio.c static int aio_setup_ring(struct kioctx *ctx) · line 280

Quoted 3 lines
fs/aio.c +3 -0
+               old = cancel;
+               cancel = cmpxchg(&kiocb->ki_cancel, old, KIOCB_CANCELLED);
+       } while (cancel != old);

erk, I missed that. What earthly sense is there in putting a BUG() in that place.

I think I'll delete it and pretend I never saw it :(

Argh. Yeah, sorry about that. Put that in there when I was trying to track down the other bugs :(

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>

Acked-by on v1 1/3
#19 lore raw

Re: [PATCH 1/3] aio: Fix a null pointer deref in batch_complete_aio

In reply to Kent Overstreet
Quoted 7 lines · Kent Overstreet

The batch completion code was trying to be a bit too clever, and skip checking ctx where it couldn't be NULL - but that broke if a kiocb had been cancelled. Move the check to kioctx_ring_unlock().

---

Quoted 28 lines
 fs/aio.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
fs/aio.c +4 -3
index 62573d3..0d2f39d 100644
@@ -683,6 +683,9 @@ static inline void kioctx_ring_unlock(struct kioctx *ctx, unsigned tail)
 {
        struct aio_ring *ring;
 
+       if (!ctx)
+               return;
+
        smp_wmb();
        /* make event visible before updating tail */
 
@@ -760,8 +763,7 @@ void batch_complete_aio(struct batch_complete *batch)
                }
 
                if (unlikely(req->ki_ctx != ctx)) {
-                       if (ctx)
-                               kioctx_ring_unlock(ctx, tail);
+                       kioctx_ring_unlock(ctx, tail);
 
                        ctx = req->ki_ctx;
                        tail = kioctx_ring_lock(ctx);
--
Quoted 2 lines

1.7.12

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>

#20 lore raw
In reply to [email protected]
Quoted 39 lines · On Tue, 22 Jan 2013 16:28:18 -0500 [email protected] wrote:
Quoted 15 lines · Hillf Danton
Quoted 8 lines · Valdis Kletnieks

Am seeing a reproducible BUG in the kernel with next-20130117 whenever I fire up VirtualBox. Unfortunately, I hadn't done that in a while, so the last 'known good' kernel was next-20121203.

I'm strongly suspecting one of Kent Overstreet's 32 patches against aio, because 'git blame' shows those landing on Jan 12, and not much else happening to fs/aio.c in ages.

Take a try?
---

fs/aio.c static inline void kioctx_ring_unlock(st · line 690

Quoted 8 lines
fs/aio.c +3 -0
@@ -683,6 +683,9 @@ static inline void kioctx_ring_unlock(st
 {
 	struct aio_ring *ring;
 
+	if (!ctx)
+		return;
+
 	smp_wmb();
 	/* make event visible before updating tail */

Well, things are improved - at least now it doesn't BUG :)

[  534.879083] ------------[ cut here ]------------
[  534.879094] WARNING: at fs/aio.c:336 put_ioctx+0x1cb/0x252()
[  534.879121] Call Trace:
[  534.879129]  [<ffffffff8102f5ad>] warn_slowpath_common+0x7e/0x97
[  534.879133]  [<ffffffff8102f5db>] warn_slowpath_null+0x15/0x17
[  534.879137]  [<ffffffff811521f0>] put_ioctx+0x1cb/0x252
[  534.879142]  [<ffffffff8105bee3>] ? __wake_up+0x3f/0x48
[  534.879146]  [<ffffffff8115229e>] ? kill_ioctx_work+0x27/0x2b
[  534.879150]  [<ffffffff811531a5>] sys_io_destroy+0x40/0x50
[  534.879156]  [<ffffffff8161b112>] system_call_fastpath+0x16/0x1b
[  534.879159] ---[ end trace a2c46a8bc9058404 ]---
Quoted 1 line

Hopefully that tells you and Kent something. :)

Did this get fixed?

#21 lore raw
Quoted 42 lines · Andrew Morton
Quoted 39 lines · On Tue, 22 Jan 2013 16:28:18 -0500 [email protected] wrote:
Quoted 15 lines · Hillf Danton
Quoted 8 lines · Valdis Kletnieks

Am seeing a reproducible BUG in the kernel with next-20130117 whenever I fire up VirtualBox. Unfortunately, I hadn't done that in a while, so the last 'known good' kernel was next-20121203.

I'm strongly suspecting one of Kent Overstreet's 32 patches against aio, because 'git blame' shows those landing on Jan 12, and not much else happening to fs/aio.c in ages.

Take a try?
---

fs/aio.c static inline void kioctx_ring_unlock(st · line 690

Quoted 8 lines
fs/aio.c +3 -0
@@ -683,6 +683,9 @@ static inline void kioctx_ring_unlock(st
 {
 	struct aio_ring *ring;
 
+	if (!ctx)
+		return;
+
 	smp_wmb();
 	/* make event visible before updating tail */

Well, things are improved - at least now it doesn't BUG :)

[  534.879083] ------------[ cut here ]------------
[  534.879094] WARNING: at fs/aio.c:336 put_ioctx+0x1cb/0x252()
[  534.879121] Call Trace:
[  534.879129]  [<ffffffff8102f5ad>] warn_slowpath_common+0x7e/0x97
[  534.879133]  [<ffffffff8102f5db>] warn_slowpath_null+0x15/0x17
[  534.879137]  [<ffffffff811521f0>] put_ioctx+0x1cb/0x252
[  534.879142]  [<ffffffff8105bee3>] ? __wake_up+0x3f/0x48
[  534.879146]  [<ffffffff8115229e>] ? kill_ioctx_work+0x27/0x2b
[  534.879150]  [<ffffffff811531a5>] sys_io_destroy+0x40/0x50
[  534.879156]  [<ffffffff8161b112>] system_call_fastpath+0x16/0x1b
[  534.879159] ---[ end trace a2c46a8bc9058404 ]---
Quoted 3 lines
Quoted 1 line

Hopefully that tells you and Kent something. :)

Did this get fixed?

With the patches I sent you, yes - not seeing a new linux-next tree yet?

#22 lore raw
Quoted 4 lines · Kent Overstreet
Quoted 1 line · Andrew Morton

Did this get fixed?

With the patches I sent you, yes - not seeing a new linux-next tree yet?

Well, it's a mixed bag at my end. Finally got a chance to do some more testing, and:

1) next-20130128 didn't show anything in dmesg, but my VirtualBox Windows 7 images appear to livelock on the way up - the Windows throbber would keep going, but it never made any actual progress towards booting. (Part of the delay was fixing a next-20121224 environment, and then discovering it took Windows *two* reboot cycles to get its act back together after getting into that hung state).

2_ next-20130128 plus the following 3 patches:

Subject: [PATCH 1/3] aio: Fix a null pointer deref in batch_complete_aio
Subject: [PATCH 3/3] aio-use-cancellation-list-lazily-fix
Subject: [PATCH 2/3] aio-kill-ki_retry-fix-fix

VirtualBox appears to be functional (I did 2 complete boot/shutdown sequences of both a 32-bit and 64-bit Win7 Enterprise image). *HOWEVER*, I saw 3 of these in dmesg:

[  668.278624] WARNING: at fs/aio.c:348 put_ioctx+0x1c0/0x241()
[  668.278652] Call Trace:
[  668.278660]  [<ffffffff8102ed10>] warn_slowpath_common+0x7c/0x96
[  668.278665]  [<ffffffff8102edc9>] warn_slowpath_null+0x15/0x17
[  668.278669]  [<ffffffff8114c562>] put_ioctx+0x1c0/0x241
[  668.278673]  [<ffffffff8114d42a>] sys_io_destroy+0x4c/0x5c
[  668.278679]  [<ffffffff8160c112>] system_call_fastpath+0x16/0x1b

and the code there says:

        WARN_ON(atomic_read(&ctx->reqs_available) > ctx->nr);

which leaves me wondering exactly how we exited the while loop just above - is the intention that it loop until reqs_available == ctx->nr exactly? Looks like if 'avail' is anything other than exactly 1 in that while loop, we can be at a state where reqs_avail == (ctx->nr -1), get 'avail=2', do the atomic_add, fall out of the loop, and trigger the WARN_ON.

Damned if I see how that can happen though....

#23 lore raw
Quoted 16 lines · On Tue, Feb 05, 2013 at 10:53:00AM -0500, [email protected] wrote:
Quoted 4 lines · Kent Overstreet
Quoted 1 line · Andrew Morton

Did this get fixed?

With the patches I sent you, yes - not seeing a new linux-next tree yet?

Well, it's a mixed bag at my end. Finally got a chance to do some more testing, and:

1) next-20130128 didn't show anything in dmesg, but my VirtualBox Windows 7 images appear to livelock on the way up - the Windows throbber would keep going, but it never made any actual progress towards booting. (Part of the delay was fixing a next-20121224 environment, and then discovering it took Windows *two* reboot cycles to get its act back together after getting into that hung state).

Quoted 5 lines

2_ next-20130128 plus the following 3 patches:

Subject: [PATCH 1/3] aio: Fix a null pointer deref in batch_complete_aio
Subject: [PATCH 3/3] aio-use-cancellation-list-lazily-fix
Subject: [PATCH 2/3] aio-kill-ki_retry-fix-fix

The "smoosh struct kiocb" patch also needs to be dropped. That causes aio_rw_vect_retry() to check ki_nbytes/ki_left after they've been overwritten by aio_complete(), which causes it to return an error when it shouldn't have, which causes aio_run_iocb() to double complete the iocb causing put_reqs_available() to be called twice and the count screwed up.

Quoted 32 lines

VirtualBox appears to be functional (I did 2 complete boot/shutdown sequences of both a 32-bit and 64-bit Win7 Enterprise image). *HOWEVER*, I saw 3 of these in dmesg:

[  668.278624] WARNING: at fs/aio.c:348 put_ioctx+0x1c0/0x241()
[  668.278652] Call Trace:
[  668.278660]  [<ffffffff8102ed10>] warn_slowpath_common+0x7c/0x96
[  668.278665]  [<ffffffff8102edc9>] warn_slowpath_null+0x15/0x17
[  668.278669]  [<ffffffff8114c562>] put_ioctx+0x1c0/0x241
[  668.278673]  [<ffffffff8114d42a>] sys_io_destroy+0x4c/0x5c
[  668.278679]  [<ffffffff8160c112>] system_call_fastpath+0x16/0x1b

and the code there says:

        WARN_ON(atomic_read(&ctx->reqs_available) > ctx->nr);

which leaves me wondering exactly how we exited the while loop just above - is the intention that it loop until reqs_available == ctx->nr exactly? Looks like if 'avail' is anything other than exactly 1 in that while loop, we can be at a state where reqs_avail == (ctx->nr -1), get 'avail=2', do the atomic_add, fall out of the loop, and trigger the WARN_ON.

Damned if I see how that can happen though....

#24 lore raw
Quoted 23 lines · Kent Overstreet
Quoted 16 lines · On Tue, Feb 05, 2013 at 10:53:00AM -0500, [email protected] wrote:
Quoted 4 lines · Kent Overstreet
Quoted 1 line · Andrew Morton

Did this get fixed?

With the patches I sent you, yes - not seeing a new linux-next tree yet?

Well, it's a mixed bag at my end. Finally got a chance to do some more testing, and:

1) next-20130128 didn't show anything in dmesg, but my VirtualBox Windows 7 images appear to livelock on the way up - the Windows throbber would keep going, but it never made any actual progress towards booting. (Part of the delay was fixing a next-20121224 environment, and then discovering it took Windows *two* reboot cycles to get its act back together after getting into that hung state).

Quoted 5 lines

2_ next-20130128 plus the following 3 patches:

Subject: [PATCH 1/3] aio: Fix a null pointer deref in batch_complete_aio
Subject: [PATCH 3/3] aio-use-cancellation-list-lazily-fix
Subject: [PATCH 2/3] aio-kill-ki_retry-fix-fix

Quoted 6 lines

The "smoosh struct kiocb" patch also needs to be dropped. That causes aio_rw_vect_retry() to check ki_nbytes/ki_left after they've been overwritten by aio_complete(), which causes it to return an error when it shouldn't have, which causes aio_run_iocb() to double complete the iocb causing put_reqs_available() to be called twice and the count screwed up.

Unfortunately, that's not a clean slam-dunk revert:

[/usr/src/linux-next] patch -p1 -R --dry-run < ~/Downloads/32-32-aio-Smoosh-struct-kiocb.patch checking file fs/aio.c
Hunk #1 FAILED at 570.
Hunk #2 FAILED at 634.
Hunk #3 FAILED at 1246.
3 out of 3 hunks FAILED
checking file include/linux/aio.h
Hunk #1 succeeded at 31 (offset 11 lines).

Looks like the above 3 patches introduce conflicts. Not sure what the proper resolution is for some of it. For the first hunk, the smoosh patch has near line 590:

- atomic_set(&req->ki_users, 1);

+       memset(req, 0, offsetof(struct kiocb, ki_ctx));
        req->ki_ctx = ctx;
+       atomic_set(&req->ki_users, 1);
        return req;

but after the 3 patches, I have:

        memset(req, 0, offsetof(struct kiocb, ki_ctx));
        req->ki_ctx = ctx;
        atomic_set(&req->ki_users, 2);
        return req;

Easy to fix, except that '2' is too magical for me to understand, so I'm not sure I want to smash it to a 1 to make the revert easier. :)

author Tested-by
#25 lore raw
In reply to Kent Overstreet
Quoted 6 lines · Kent Overstreet

The "smoosh struct kiocb" patch also needs to be dropped. That causes aio_rw_vect_retry() to check ki_nbytes/ki_left after they've been overwritten by aio_complete(), which causes it to return an error when it shouldn't have, which causes aio_run_iocb() to double complete the iocb causing put_reqs_available() to be called twice and the count screwed up.

Hooray! linux-next-20130206 builds, boots, and VirtualBox 4.2.6 runs without producing any kernel messages. Feel free to add the following as appropriate to this version of the patch series: