aio: fixes for kernel memory disclosure in aio read events

Please pull the following 2 fixes from my aio-fixes git tree at
git://git.kvack.org/~bcrl/aio-fixes.git . These fix a kernel memory disclosure issue (arbitrary kmap() & copy_to_user()) revealed in CVE-2014-0206 by changes that were introduced in v3.10.

Benjamin LaHaise (2):
aio: fix aio request leak when events are reaped by userspace
aio: fix kernel memory disclosure in io_getevents() introduced in

    v3.10

 fs/aio.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

1.8.2.1


-- 
"Thought is the essence of where you are now."

The aio cleanups and optimizations by kmo that were merged into the 3.10 tree added a regression for userspace event reaping. Specifically, the reference counts are not decremented if the event is reaped in userspace, leading to the application being unable to submit further aio requests. This patch applies to 3.12+. A separate backport is required for 3.10/3.11. This issue was uncovered as part of CVE-2014-0206.

---
 fs/aio.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

index 4f078c0..6a9c7e4 100644
@@ -1021,6 +1021,7 @@ void aio_complete(struct kiocb *iocb, long res, long res2)
 
 	/* everything turned out well, dispose of the aiocb. */
 	kiocb_free(iocb);
+	put_reqs_available(ctx, 1);
 
 	/*
 	 * We have to order our ring_info tail store above and test

@@ -1100,8 +1101,6 @@ static long aio_read_events_ring(struct kioctx *ctx,
 	flush_dcache_page(ctx->ring_pages[0]);
 
 	pr_debug("%li  h%u t%u\n", ret, head, tail);
-
-	put_reqs_available(ctx, ret);
 out:
 	mutex_unlock(&ctx->ring_lock);
 

1.8.2.1


-- 
"Thought is the essence of where you are now."

Some testing I've done today indicates that the original commit broke AIO with regard to users that overflow the maximum number of request per IO context (where -EAGAIN is returned).

In fact, it did worse - the attached C program can easily overrun the ring buffer that is responsible for managing the completed requests, and caused notification about their completion never to be returned.

See attachments to this mail:

aio_bug.c: a program that helps to reproduce the probelm
dmesg-bad.txt: fs/aio.c with dynamic printk on v3.15.10.
The printks show how the ring buffer gets overrun.
dmesg-good.txt: fs/aio.c with dynamic printk on a kernel with the
commit reverted.

For this program to work, we need enough events coming from io_submit() in one go so that the ring buffer overflows. Local file system should support O_DIRECT and it seems to test here correctly on a laptop with an SSD.

I am not an expert on this code, but it looks perhaps that the reason that put_reqs_available() was called in the original context was to prevent this scenario exactly, and to properly manage the maximum number of outstanding requests.

This reverts commit b34e0e1319b31202eb142dcd9688cf7145a30bf6.

---
 fs/aio.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

index 6d68e01dc7ca..45b07e8a1834 100644
@@ -1028,7 +1028,6 @@ void aio_complete(struct kiocb *iocb, long res, long res2)
 
 	/* everything turned out well, dispose of the aiocb. */
 	kiocb_free(iocb);
-	put_reqs_available(ctx, 1);
 
 	/*
 	 * We have to order our ring_info tail store above and test

@@ -1111,6 +1110,8 @@ static long aio_read_events_ring(struct kioctx *ctx,
 	flush_dcache_page(ctx->ring_pages[0]);
 
 	pr_debug("%li  h%u t%u\n", ret, head, tail);
+
+	put_reqs_available(ctx, ret);
 out:
 	mutex_unlock(&ctx->ring_lock);
 

1.9.3

-- 
Dan Aloni

[ 2966.641825] aio_setup_ring: pid(9966) page[0]->count=3
[ 2966.641829] aio_setup_ring: attempting mmap of 4096 bytes
[ 2966.641834] aio_setup_ring: mmap address: 0x7ff021605000
[ 2966.641837] ioctx_alloc: allocated ioctx ffff8800c85fc000[140669328838656]: mm=ffff8801f7f5ec00 mask=0x7f
[ 2966.642645] aio_complete: ffff8800c85fc000[1]: ffff8801f7e5a280: 0000000000603cc0 0 1000 0
[ 2966.642647] aio_complete: added to ring ffff8801f7e5a280 at [1]
[ 2966.642650] aio_complete: ffff8800c85fc000[2]: ffff8801f7e5a280: 0000000000603d00 0 1000 0
[ 2966.642651] aio_complete: added to ring ffff8801f7e5a280 at [2]
[ 2966.642654] aio_complete: ffff8800c85fc000[3]: ffff8801f7e5a280: 0000000000603d40 0 1000 0
[ 2966.642655] aio_complete: added to ring ffff8801f7e5a280 at [3]
[ 2966.642657] aio_complete: ffff8800c85fc000[4]: ffff8801f7e5a280: 0000000000603d80 0 1000 0
[ 2966.642658] aio_complete: added to ring ffff8801f7e5a280 at [4]
[ 2966.642661] aio_complete: ffff8800c85fc000[5]: ffff8801f7e5a280: 0000000000603dc0 0 1000 0
[ 2966.642662] aio_complete: added to ring ffff8801f7e5a280 at [5]
[ 2966.642664] aio_complete: ffff8800c85fc000[6]: ffff8801f7e5a280: 0000000000603e00 0 1000 0
[ 2966.642665] aio_complete: added to ring ffff8801f7e5a280 at [6]
[ 2966.642668] aio_complete: ffff8800c85fc000[7]: ffff8801f7e5a280: 0000000000603e40 0 1000 0
[ 2966.642670] aio_complete: added to ring ffff8801f7e5a280 at [7]
[ 2966.642676] aio_complete: ffff8800c85fc000[8]: ffff8801f7e5a280: 0000000000603e80 0 1000 0
[ 2966.642677] aio_complete: added to ring ffff8801f7e5a280 at [8]
[ 2966.642680] aio_complete: ffff8800c85fc000[9]: ffff8801f7e5a280: 0000000000603ec0 0 1000 0
[ 2966.642680] aio_complete: added to ring ffff8801f7e5a280 at [9]
[ 2966.642683] aio_complete: ffff8800c85fc000[10]: ffff8801f7e5a280: 0000000000603f00 0 1000 0
[ 2966.642684] aio_complete: added to ring ffff8801f7e5a280 at [10]
[ 2966.642686] aio_complete: ffff8800c85fc000[11]: ffff8801f7e5a280: 0000000000603f40 0 1000 0
[ 2966.642687] aio_complete: added to ring ffff8801f7e5a280 at [11]
[ 2966.642690] aio_complete: ffff8800c85fc000[12]: ffff8801f7e5a280: 0000000000603f80 0 1000 0
[ 2966.642691] aio_complete: added to ring ffff8801f7e5a280 at [12]
[ 2966.642693] aio_complete: ffff8800c85fc000[13]: ffff8801f7e5a280: 0000000000603fc0 0 1000 0
[ 2966.642694] aio_complete: added to ring ffff8801f7e5a280 at [13]
[ 2966.642697] aio_complete: ffff8800c85fc000[14]: ffff8801f7e5a280: 0000000000604000 0 1000 0
[ 2966.642698] aio_complete: added to ring ffff8801f7e5a280 at [14]
[ 2966.642700] aio_complete: ffff8800c85fc000[15]: ffff8801f7e5a280: 0000000000604040 0 1000 0
[ 2966.642701] aio_complete: added to ring ffff8801f7e5a280 at [15]
[ 2966.642704] aio_complete: ffff8800c85fc000[16]: ffff8801f7e5a280: 0000000000604080 0 1000 0
[ 2966.642704] aio_complete: added to ring ffff8801f7e5a280 at [16]
[ 2966.642707] aio_complete: ffff8800c85fc000[17]: ffff8801f7e5a280: 00000000006040c0 0 1000 0
[ 2966.642708] aio_complete: added to ring ffff8801f7e5a280 at [17]
[ 2966.642710] aio_complete: ffff8800c85fc000[18]: ffff8801f7e5a280: 0000000000604100 0 1000 0
[ 2966.642711] aio_complete: added to ring ffff8801f7e5a280 at [18]
[ 2966.642714] aio_complete: ffff8800c85fc000[19]: ffff8801f7e5a280: 0000000000604140 0 1000 0
[ 2966.642715] aio_complete: added to ring ffff8801f7e5a280 at [19]
[ 2966.642717] aio_complete: ffff8800c85fc000[20]: ffff8801f7e5a280: 0000000000604180 0 1000 0
[ 2966.642718] aio_complete: added to ring ffff8801f7e5a280 at [20]
[ 2966.642720] aio_complete: ffff8800c85fc000[21]: ffff8801f7e5a280: 00000000006041c0 0 1000 0
[ 2966.642721] aio_complete: added to ring ffff8801f7e5a280 at [21]
[ 2966.642725] aio_complete: ffff8800c85fc000[22]: ffff8801f7e5a280: 0000000000604200 0 1000 0
[ 2966.642725] aio_complete: added to ring ffff8801f7e5a280 at [22]
[ 2966.642728] aio_complete: ffff8800c85fc000[23]: ffff8801f7e5a280: 0000000000604240 0 1000 0
[ 2966.642729] aio_complete: added to ring ffff8801f7e5a280 at [23]
[ 2966.642731] aio_complete: ffff8800c85fc000[24]: ffff8801f7e5a280: 0000000000604280 0 1000 0
[ 2966.642732] aio_complete: added to ring ffff8801f7e5a280 at [24]
[ 2966.642735] aio_complete: ffff8800c85fc000[25]: ffff8801f7e5a280: 00000000006042c0 0 1000 0
[ 2966.642735] aio_complete: added to ring ffff8801f7e5a280 at [25]
[ 2966.642738] aio_complete: ffff8800c85fc000[26]: ffff8801f7e5a280: 0000000000604300 0 1000 0
[ 2966.642739] aio_complete: added to ring ffff8801f7e5a280 at [26]
[ 2966.642741] aio_complete: ffff8800c85fc000[27]: ffff8801f7e5a280: 0000000000604340 0 1000 0
[ 2966.642742] aio_complete: added to ring ffff8801f7e5a280 at [27]
[ 2966.642745] aio_complete: ffff8800c85fc000[28]: ffff8801f7e5a280: 0000000000604380 0 1000 0
[ 2966.642745] aio_complete: added to ring ffff8801f7e5a280 at [28]
[ 2966.642748] aio_complete: ffff8800c85fc000[29]: ffff8801f7e5a280: 00000000006043c0 0 1000 0
[ 2966.642749] aio_complete: added to ring ffff8801f7e5a280 at [29]
[ 2966.642751] aio_complete: ffff8800c85fc000[30]: ffff8801f7e5a280: 0000000000604400 0 1000 0
[ 2966.642752] aio_complete: added to ring ffff8801f7e5a280 at [30]
[ 2966.642755] aio_complete: ffff8800c85fc000[31]: ffff8801f7e5a280: 0000000000604440 0 1000 0
[ 2966.642756] aio_complete: added to ring ffff8801f7e5a280 at [31]
[ 2966.642758] aio_complete: ffff8800c85fc000[32]: ffff8801f7e5a280: 0000000000604480 0 1000 0
[ 2966.642759] aio_complete: added to ring ffff8801f7e5a280 at [32]
[ 2966.642761] aio_complete: ffff8800c85fc000[33]: ffff8801f7e5a280: 00000000006044c0 0 1000 0
[ 2966.642762] aio_complete: added to ring ffff8801f7e5a280 at [33]
[ 2966.642765] aio_complete: ffff8800c85fc000[34]: ffff8801f7e5a280: 0000000000604500 0 1000 0
[ 2966.642766] aio_complete: added to ring ffff8801f7e5a280 at [34]
[ 2966.642768] aio_complete: ffff8800c85fc000[35]: ffff8801f7e5a280: 0000000000604540 0 1000 0
[ 2966.642769] aio_complete: added to ring ffff8801f7e5a280 at [35]
[ 2966.642772] aio_complete: ffff8800c85fc000[36]: ffff8801f7e5a280: 0000000000604580 0 1000 0
[ 2966.642773] aio_complete: added to ring ffff8801f7e5a280 at [36]
[ 2966.642775] aio_complete: ffff8800c85fc000[37]: ffff8801f7e5a280: 00000000006045c0 0 1000 0
[ 2966.642776] aio_complete: added to ring ffff8801f7e5a280 at [37]
[ 2966.642779] aio_complete: ffff8800c85fc000[38]: ffff8801f7e5a280: 0000000000604600 0 1000 0
[ 2966.642780] aio_complete: added to ring ffff8801f7e5a280 at [38]
[ 2966.642784] aio_complete: ffff8800c85fc000[39]: ffff8801f7e5a280: 0000000000604640 0 1000 0
[ 2966.642785] aio_complete: added to ring ffff8801f7e5a280 at [39]
[ 2966.642787] aio_complete: ffff8800c85fc000[40]: ffff8801f7e5a280: 0000000000604680 0 1000 0
[ 2966.642788] aio_complete: added to ring ffff8801f7e5a280 at [40]
[ 2966.642791] aio_complete: ffff8800c85fc000[41]: ffff8801f7e5a280: 00000000006046c0 0 1000 0
[ 2966.642792] aio_complete: added to ring ffff8801f7e5a280 at [41]
[ 2966.642794] aio_complete: ffff8800c85fc000[42]: ffff8801f7e5a280: 0000000000604700 0 1000 0
[ 2966.642795] aio_complete: added to ring ffff8801f7e5a280 at [42]
[ 2966.642797] aio_complete: ffff8800c85fc000[43]: ffff8801f7e5a280: 0000000000604740 0 1000 0
[ 2966.642798] aio_complete: added to ring ffff8801f7e5a280 at [43]
[ 2966.642801] aio_complete: ffff8800c85fc000[44]: ffff8801f7e5a280: 0000000000604780 0 1000 0
[ 2966.642802] aio_complete: added to ring ffff8801f7e5a280 at [44]
[ 2966.642804] aio_complete: ffff8800c85fc000[45]: ffff8801f7e5a280: 00000000006047c0 0 1000 0
[ 2966.642805] aio_complete: added to ring ffff8801f7e5a280 at [45]
[ 2966.642807] aio_complete: ffff8800c85fc000[46]: ffff8801f7e5a280: 0000000000604800 0 1000 0
[ 2966.642808] aio_complete: added to ring ffff8801f7e5a280 at [46]
[ 2966.642811] aio_complete: ffff8800c85fc000[47]: ffff8801f7e5a280: 0000000000604840 0 1000 0
[ 2966.642812] aio_complete: added to ring ffff8801f7e5a280 at [47]
[ 2966.642814] aio_complete: ffff8800c85fc000[48]: ffff8801f7e5a280: 0000000000604880 0 1000 0
[ 2966.642815] aio_complete: added to ring ffff8801f7e5a280 at [48]
[ 2966.642818] aio_complete: ffff8800c85fc000[49]: ffff8801f7e5a280: 00000000006048c0 0 1000 0
[ 2966.642819] aio_complete: added to ring ffff8801f7e5a280 at [49]
[ 2966.642822] aio_complete: ffff8800c85fc000[50]: ffff8801f7e5a280: 0000000000604900 0 1000 0
[ 2966.642823] aio_complete: added to ring ffff8801f7e5a280 at [50]
[ 2966.642825] aio_complete: ffff8800c85fc000[51]: ffff8801f7e5a280: 0000000000604940 0 1000 0
[ 2966.642826] aio_complete: added to ring ffff8801f7e5a280 at [51]
[ 2966.642828] aio_complete: ffff8800c85fc000[52]: ffff8801f7e5a280: 0000000000604980 0 1000 0
[ 2966.642829] aio_complete: added to ring ffff8801f7e5a280 at [52]
[ 2966.642832] aio_complete: ffff8800c85fc000[53]: ffff8801f7e5a280: 00000000006049c0 0 1000 0
[ 2966.642833] aio_complete: added to ring ffff8801f7e5a280 at [53]
[ 2966.642835] aio_complete: ffff8800c85fc000[54]: ffff8801f7e5a280: 0000000000604a00 0 1000 0
[ 2966.642836] aio_complete: added to ring ffff8801f7e5a280 at [54]
[ 2966.642838] aio_complete: ffff8800c85fc000[55]: ffff8801f7e5a280: 0000000000604a40 0 1000 0
[ 2966.642839] aio_complete: added to ring ffff8801f7e5a280 at [55]
[ 2966.642842] aio_complete: ffff8800c85fc000[56]: ffff8801f7e5a280: 0000000000604a80 0 1000 0
[ 2966.642843] aio_complete: added to ring ffff8801f7e5a280 at [56]
[ 2966.642845] aio_complete: ffff8800c85fc000[57]: ffff8801f7e5a280: 0000000000604ac0 0 1000 0
[ 2966.642846] aio_complete: added to ring ffff8801f7e5a280 at [57]
[ 2966.642849] aio_complete: ffff8800c85fc000[58]: ffff8801f7e5a280: 0000000000604b00 0 1000 0
[ 2966.642849] aio_complete: added to ring ffff8801f7e5a280 at [58]
[ 2966.642852] aio_complete: ffff8800c85fc000[59]: ffff8801f7e5a280: 0000000000604b40 0 1000 0
[ 2966.642853] aio_complete: added to ring ffff8801f7e5a280 at [59]
[ 2966.642855] aio_complete: ffff8800c85fc000[60]: ffff8801f7e5a280: 0000000000604b80 0 1000 0
[ 2966.642856] aio_complete: added to ring ffff8801f7e5a280 at [60]
[ 2966.642859] aio_complete: ffff8800c85fc000[61]: ffff8801f7e5a280: 0000000000604bc0 0 1000 0
[ 2966.642860] aio_complete: added to ring ffff8801f7e5a280 at [61]
[ 2966.642862] aio_complete: ffff8800c85fc000[62]: ffff8801f7e5a280: 0000000000604c00 0 1000 0
[ 2966.642863] aio_complete: added to ring ffff8801f7e5a280 at [62]
[ 2966.642865] aio_complete: ffff8800c85fc000[63]: ffff8801f7e5a280: 0000000000604c40 0 1000 0
[ 2966.642866] aio_complete: added to ring ffff8801f7e5a280 at [63]
[ 2966.642869] aio_complete: ffff8800c85fc000[64]: ffff8801f7e5a280: 0000000000604c80 0 1000 0
[ 2966.642870] aio_complete: added to ring ffff8801f7e5a280 at [64]
[ 2966.642873] aio_complete: ffff8800c85fc000[65]: ffff8801f7e5a280: 0000000000604cc0 0 1000 0
[ 2966.642874] aio_complete: added to ring ffff8801f7e5a280 at [65]
[ 2966.642876] aio_complete: ffff8800c85fc000[66]: ffff8801f7e5a280: 0000000000604d00 0 1000 0
[ 2966.642877] aio_complete: added to ring ffff8801f7e5a280 at [66]
[ 2966.642879] aio_complete: ffff8800c85fc000[67]: ffff8801f7e5a280: 0000000000604d40 0 1000 0
[ 2966.642880] aio_complete: added to ring ffff8801f7e5a280 at [67]
[ 2966.642883] aio_complete: ffff8800c85fc000[68]: ffff8801f7e5a280: 0000000000604d80 0 1000 0
[ 2966.642884] aio_complete: added to ring ffff8801f7e5a280 at [68]
[ 2966.642886] aio_complete: ffff8800c85fc000[69]: ffff8801f7e5a280: 0000000000604dc0 0 1000 0
[ 2966.642887] aio_complete: added to ring ffff8801f7e5a280 at [69]
[ 2966.642891] aio_complete: ffff8800c85fc000[70]: ffff8801f7e5a280: 0000000000604e00 0 1000 0
[ 2966.642892] aio_complete: added to ring ffff8801f7e5a280 at [70]
[ 2966.642894] aio_complete: ffff8800c85fc000[71]: ffff8801f7e5a280: 0000000000604e40 0 1000 0
[ 2966.642895] aio_complete: added to ring ffff8801f7e5a280 at [71]
[ 2966.642898] aio_complete: ffff8800c85fc000[72]: ffff8801f7e5a280: 0000000000604e80 0 1000 0
[ 2966.642899] aio_complete: added to ring ffff8801f7e5a280 at [72]
[ 2966.642901] aio_complete: ffff8800c85fc000[73]: ffff8801f7e5a280: 0000000000604ec0 0 1000 0
[ 2966.642902] aio_complete: added to ring ffff8801f7e5a280 at [73]
[ 2966.642904] aio_complete: ffff8800c85fc000[74]: ffff8801f7e5a280: 0000000000604f00 0 1000 0
[ 2966.642905] aio_complete: added to ring ffff8801f7e5a280 at [74]
[ 2966.642908] aio_complete: ffff8800c85fc000[75]: ffff8801f7e5a280: 0000000000604f40 0 1000 0
[ 2966.642909] aio_complete: added to ring ffff8801f7e5a280 at [75]
[ 2966.642911] aio_complete: ffff8800c85fc000[76]: ffff8801f7e5a280: 0000000000604f80 0 1000 0
[ 2966.642912] aio_complete: added to ring ffff8801f7e5a280 at [76]
[ 2966.642914] aio_complete: ffff8800c85fc000[77]: ffff8801f7e5a280: 0000000000604fc0 0 1000 0
[ 2966.642915] aio_complete: added to ring ffff8801f7e5a280 at [77]
[ 2966.642918] aio_complete: ffff8800c85fc000[78]: ffff8801f7e5a280: 0000000000605000 0 1000 0
[ 2966.642919] aio_complete: added to ring ffff8801f7e5a280 at [78]
[ 2966.642922] aio_complete: ffff8800c85fc000[79]: ffff8801f7e5a280: 0000000000605040 0 1000 0
[ 2966.642923] aio_complete: added to ring ffff8801f7e5a280 at [79]
[ 2966.642925] aio_complete: ffff8800c85fc000[80]: ffff8801f7e5a280: 0000000000605080 0 1000 0
[ 2966.642926] aio_complete: added to ring ffff8801f7e5a280 at [80]
[ 2966.642929] aio_complete: ffff8800c85fc000[81]: ffff8801f7e5a280: 00000000006050c0 0 1000 0
[ 2966.642929] aio_complete: added to ring ffff8801f7e5a280 at [81]
[ 2966.642932] aio_complete: ffff8800c85fc000[82]: ffff8801f7e5a280: 0000000000605100 0 1000 0
[ 2966.642933] aio_complete: added to ring ffff8801f7e5a280 at [82]
[ 2966.642935] aio_complete: ffff8800c85fc000[83]: ffff8801f7e5a280: 0000000000605140 0 1000 0
[ 2966.642936] aio_complete: added to ring ffff8801f7e5a280 at [83]
[ 2966.642939] aio_complete: ffff8800c85fc000[84]: ffff8801f7e5a280: 0000000000605180 0 1000 0
[ 2966.642939] aio_complete: added to ring ffff8801f7e5a280 at [84]
[ 2966.642942] aio_complete: ffff8800c85fc000[85]: ffff8801f7e5a280: 00000000006051c0 0 1000 0
[ 2966.642943] aio_complete: added to ring ffff8801f7e5a280 at [85]
[ 2966.642945] aio_complete: ffff8800c85fc000[86]: ffff8801f7e5a280: 0000000000605200 0 1000 0
[ 2966.642946] aio_complete: added to ring ffff8801f7e5a280 at [86]
[ 2966.642949] aio_complete: ffff8800c85fc000[87]: ffff8801f7e5a280: 0000000000605240 0 1000 0
[ 2966.642950] aio_complete: added to ring ffff8801f7e5a280 at [87]
[ 2966.642952] aio_complete: ffff8800c85fc000[88]: ffff8801f7e5a280: 0000000000605280 0 1000 0
[ 2966.642953] aio_complete: added to ring ffff8801f7e5a280 at [88]
[ 2966.642955] aio_complete: ffff8800c85fc000[89]: ffff8801f7e5a280: 00000000006052c0 0 1000 0
[ 2966.642956] aio_complete: added to ring ffff8801f7e5a280 at [89]
[ 2966.642959] aio_complete: ffff8800c85fc000[90]: ffff8801f7e5a280: 0000000000605300 0 1000 0
[ 2966.642960] aio_complete: added to ring ffff8801f7e5a280 at [90]
[ 2966.642962] aio_complete: ffff8800c85fc000[91]: ffff8801f7e5a280: 0000000000605340 0 1000 0
[ 2966.642963] aio_complete: added to ring ffff8801f7e5a280 at [91]
[ 2966.642966] aio_complete: ffff8800c85fc000[92]: ffff8801f7e5a280: 0000000000605380 0 1000 0
[ 2966.642967] aio_complete: added to ring ffff8801f7e5a280 at [92]
[ 2966.642969] aio_complete: ffff8800c85fc000[93]: ffff8801f7e5a280: 00000000006053c0 0 1000 0
[ 2966.642970] aio_complete: added to ring ffff8801f7e5a280 at [93]
[ 2966.642973] aio_complete: ffff8800c85fc000[94]: ffff8801f7e5a280: 0000000000605400 0 1000 0
[ 2966.642974] aio_complete: added to ring ffff8801f7e5a280 at [94]
[ 2966.642976] aio_complete: ffff8800c85fc000[95]: ffff8801f7e5a280: 0000000000605440 0 1000 0
[ 2966.642977] aio_complete: added to ring ffff8801f7e5a280 at [95]
[ 2966.642979] aio_complete: ffff8800c85fc000[96]: ffff8801f7e5a280: 0000000000605480 0 1000 0
[ 2966.642980] aio_complete: added to ring ffff8801f7e5a280 at [96]
[ 2966.642983] aio_complete: ffff8800c85fc000[97]: ffff8801f7e5a280: 00000000006054c0 0 1000 0
[ 2966.642984] aio_complete: added to ring ffff8801f7e5a280 at [97]
[ 2966.642986] aio_complete: ffff8800c85fc000[98]: ffff8801f7e5a280: 0000000000605500 0 1000 0
[ 2966.642987] aio_complete: added to ring ffff8801f7e5a280 at [98]
[ 2966.642989] aio_complete: ffff8800c85fc000[99]: ffff8801f7e5a280: 0000000000605540 0 1000 0
[ 2966.642990] aio_complete: added to ring ffff8801f7e5a280 at [99]
[ 2966.642993] aio_complete: ffff8800c85fc000[100]: ffff8801f7e5a280: 0000000000605580 0 1000 0
[ 2966.642994] aio_complete: added to ring ffff8801f7e5a280 at [100]
[ 2966.642998] aio_complete: ffff8800c85fc000[101]: ffff8801f7e5a280: 00000000006055c0 0 1000 0
[ 2966.642998] aio_complete: added to ring ffff8801f7e5a280 at [101]
[ 2966.643001] aio_complete: ffff8800c85fc000[102]: ffff8801f7e5a280: 0000000000605600 0 1000 0
[ 2966.643002] aio_complete: added to ring ffff8801f7e5a280 at [102]
[ 2966.643004] aio_complete: ffff8800c85fc000[103]: ffff8801f7e5a280: 0000000000605640 0 1000 0
[ 2966.643005] aio_complete: added to ring ffff8801f7e5a280 at [103]
[ 2966.643008] aio_complete: ffff8800c85fc000[104]: ffff8801f7e5a280: 0000000000605680 0 1000 0
[ 2966.643009] aio_complete: added to ring ffff8801f7e5a280 at [104]
[ 2966.643011] aio_complete: ffff8800c85fc000[105]: ffff8801f7e5a280: 00000000006056c0 0 1000 0
[ 2966.643012] aio_complete: added to ring ffff8801f7e5a280 at [105]
[ 2966.643015] aio_complete: ffff8800c85fc000[106]: ffff8801f7e5a280: 0000000000605700 0 1000 0
[ 2966.643016] aio_complete: added to ring ffff8801f7e5a280 at [106]
[ 2966.643018] aio_complete: ffff8800c85fc000[107]: ffff8801f7e5a280: 0000000000605740 0 1000 0
[ 2966.643019] aio_complete: added to ring ffff8801f7e5a280 at [107]
[ 2966.643022] aio_complete: ffff8800c85fc000[108]: ffff8801f7e5a280: 0000000000605780 0 1000 0
[ 2966.643023] aio_complete: added to ring ffff8801f7e5a280 at [108]
[ 2966.643025] aio_complete: ffff8800c85fc000[109]: ffff8801f7e5a280: 00000000006057c0 0 1000 0
[ 2966.643026] aio_complete: added to ring ffff8801f7e5a280 at [109]
[ 2966.643028] aio_complete: ffff8800c85fc000[110]: ffff8801f7e5a280: 0000000000605800 0 1000 0
[ 2966.643029] aio_complete: added to ring ffff8801f7e5a280 at [110]
[ 2966.643032] aio_complete: ffff8800c85fc000[111]: ffff8801f7e5a280: 0000000000605840 0 1000 0
[ 2966.643033] aio_complete: added to ring ffff8801f7e5a280 at [111]
[ 2966.643035] aio_complete: ffff8800c85fc000[112]: ffff8801f7e5a280: 0000000000605880 0 1000 0
[ 2966.643036] aio_complete: added to ring ffff8801f7e5a280 at [112]
[ 2966.643038] aio_complete: ffff8800c85fc000[113]: ffff8801f7e5a280: 00000000006058c0 0 1000 0
[ 2966.643039] aio_complete: added to ring ffff8801f7e5a280 at [113]
[ 2966.643042] aio_complete: ffff8800c85fc000[114]: ffff8801f7e5a280: 0000000000605900 0 1000 0
[ 2966.643043] aio_complete: added to ring ffff8801f7e5a280 at [114]
[ 2966.643045] aio_complete: ffff8800c85fc000[115]: ffff8801f7e5a280: 0000000000605940 0 1000 0
[ 2966.643046] aio_complete: added to ring ffff8801f7e5a280 at [115]
[ 2966.643049] aio_complete: ffff8800c85fc000[116]: ffff8801f7e5a280: 0000000000605980 0 1000 0
[ 2966.643049] aio_complete: added to ring ffff8801f7e5a280 at [116]
[ 2966.643052] aio_complete: ffff8800c85fc000[117]: ffff8801f7e5a280: 00000000006059c0 0 1000 0
[ 2966.643053] aio_complete: added to ring ffff8801f7e5a280 at [117]
[ 2966.643055] aio_complete: ffff8800c85fc000[118]: ffff8801f7e5a280: 0000000000605a00 0 1000 0
[ 2966.643056] aio_complete: added to ring ffff8801f7e5a280 at [118]
[ 2966.643059] aio_complete: ffff8800c85fc000[119]: ffff8801f7e5a280: 0000000000605a40 0 1000 0
[ 2966.643060] aio_complete: added to ring ffff8801f7e5a280 at [119]
[ 2966.643063] aio_complete: ffff8800c85fc000[120]: ffff8801f7e5a280: 0000000000605a80 0 1000 0
[ 2966.643064] aio_complete: added to ring ffff8801f7e5a280 at [120]
[ 2966.643066] aio_complete: ffff8800c85fc000[121]: ffff8801f7e5a280: 0000000000605ac0 0 1000 0
[ 2966.643067] aio_complete: added to ring ffff8801f7e5a280 at [121]
[ 2966.643069] aio_complete: ffff8800c85fc000[122]: ffff8801f7e5a280: 0000000000605b00 0 1000 0
[ 2966.643070] aio_complete: added to ring ffff8801f7e5a280 at [122]
[ 2966.643073] aio_complete: ffff8800c85fc000[123]: ffff8801f7e5a280: 0000000000605b40 0 1000 0
[ 2966.643074] aio_complete: added to ring ffff8801f7e5a280 at [123]
[ 2966.643076] aio_complete: ffff8800c85fc000[124]: ffff8801f7e5a280: 0000000000605b80 0 1000 0
[ 2966.643077] aio_complete: added to ring ffff8801f7e5a280 at [124]
[ 2966.643079] aio_complete: ffff8800c85fc000[125]: ffff8801f7e5a280: 0000000000605bc0 0 1000 0
[ 2966.643080] aio_complete: added to ring ffff8801f7e5a280 at [125]
[ 2966.643083] aio_complete: ffff8800c85fc000[126]: ffff8801f7e5a280: 0000000000605c00 0 1000 0
[ 2966.643084] aio_complete: added to ring ffff8801f7e5a280 at [126]
[ 2966.643086] aio_complete: ffff8800c85fc000[0]: ffff8801f7e5a280: 0000000000605c40 0 1000 0
[ 2966.643087] aio_complete: added to ring ffff8801f7e5a280 at [0]
[ 2966.643089] aio_complete: ffff8800c85fc000[1]: ffff8801f7e5a280: 0000000000605c80 0 1000 0
[ 2966.643090] aio_complete: added to ring ffff8801f7e5a280 at [1]
[ 2966.643147] aio_read_events_ring: h0 t1 m127
[ 2966.643149] aio_read_events_ring: 1  h1 t1
[ 2966.643150] aio_read_events_ring: h1 t1 m127
[ 2966.699376] aio_read_events_ring: h0 t0 m511
[ 2966.699386] aio_read_events_ring: h0 t0 m511
[ 2966.699389] aio_read_events_ring: h0 t0 m511
[ 2966.799574] aio_read_events_ring: h0 t0 m511
[ 3048.065378] aio_setup_ring: pid(1025) page[0]->count=3
[ 3048.065380] aio_setup_ring: attempting mmap of 4096 bytes
[ 3048.065382] aio_setup_ring: mmap address: 0x7f6cae9aa000
[ 3048.065383] ioctx_alloc: allocated ioctx ffff8800b6e4b200[140104762564608]: mm=ffff8800b6da9000 mask=0x7f
[ 3048.065556] aio_complete: ffff8800b6e4b200[1]: ffff8800b6786e00: 0000000000603cc0 0 1000 0
[ 3048.065558] aio_complete: added to ring ffff8800b6786e00 at [1]
[ 3048.065561] aio_complete: ffff8800b6e4b200[2]: ffff8800b6786e00: 0000000000603d00 0 1000 0
[ 3048.065562] aio_complete: added to ring ffff8800b6786e00 at [2]
[ 3048.065565] aio_complete: ffff8800b6e4b200[3]: ffff8800b6786e00: 0000000000603d40 0 1000 0
[ 3048.065566] aio_complete: added to ring ffff8800b6786e00 at [3]
[ 3048.065569] aio_complete: ffff8800b6e4b200[4]: ffff8800b6786e00: 0000000000603d80 0 1000 0
[ 3048.065570] aio_complete: added to ring ffff8800b6786e00 at [4]
[ 3048.065573] aio_complete: ffff8800b6e4b200[5]: ffff8800b6786e00: 0000000000603dc0 0 1000 0
[ 3048.065574] aio_complete: added to ring ffff8800b6786e00 at [5]
[ 3048.065591] aio_complete: ffff8800b6e4b200[6]: ffff8800b6786e00: 0000000000603e00 0 1000 0
[ 3048.065595] aio_complete: added to ring ffff8800b6786e00 at [6]
[ 3048.065602] aio_complete: ffff8800b6e4b200[7]: ffff8800b6786e00: 0000000000603e40 0 1000 0
[ 3048.065603] aio_complete: added to ring ffff8800b6786e00 at [7]
[ 3048.065607] aio_complete: ffff8800b6e4b200[8]: ffff8800b6786e00: 0000000000603e80 0 1000 0
[ 3048.065608] aio_complete: added to ring ffff8800b6786e00 at [8]
[ 3048.065611] aio_complete: ffff8800b6e4b200[9]: ffff8800b6786e00: 0000000000603ec0 0 1000 0
[ 3048.065612] aio_complete: added to ring ffff8800b6786e00 at [9]
[ 3048.065615] aio_complete: ffff8800b6e4b200[10]: ffff8800b6786e00: 0000000000603f00 0 1000 0
[ 3048.065616] aio_complete: added to ring ffff8800b6786e00 at [10]
[ 3048.065619] aio_complete: ffff8800b6e4b200[11]: ffff8800b6786e00: 0000000000603f40 0 1000 0
[ 3048.065620] aio_complete: added to ring ffff8800b6786e00 at [11]
[ 3048.065623] aio_complete: ffff8800b6e4b200[12]: ffff8800b6786e00: 0000000000603f80 0 1000 0
[ 3048.065623] aio_complete: added to ring ffff8800b6786e00 at [12]
[ 3048.065626] aio_complete: ffff8800b6e4b200[13]: ffff8800b6786e00: 0000000000603fc0 0 1000 0
[ 3048.065627] aio_complete: added to ring ffff8800b6786e00 at [13]
[ 3048.065631] aio_complete: ffff8800b6e4b200[14]: ffff8800b6786e00: 0000000000604000 0 1000 0
[ 3048.065632] aio_complete: added to ring ffff8800b6786e00 at [14]
[ 3048.065635] aio_complete: ffff8800b6e4b200[15]: ffff8800b6786e00: 0000000000604040 0 1000 0
[ 3048.065635] aio_complete: added to ring ffff8800b6786e00 at [15]
[ 3048.065638] aio_complete: ffff8800b6e4b200[16]: ffff8800b6786e00: 0000000000604080 0 1000 0
[ 3048.065639] aio_complete: added to ring ffff8800b6786e00 at [16]
[ 3048.065642] aio_complete: ffff8800b6e4b200[17]: ffff8800b6786e00: 00000000006040c0 0 1000 0
[ 3048.065643] aio_complete: added to ring ffff8800b6786e00 at [17]
[ 3048.065646] aio_complete: ffff8800b6e4b200[18]: ffff8800b6786e00: 0000000000604100 0 1000 0
[ 3048.065647] aio_complete: added to ring ffff8800b6786e00 at [18]
[ 3048.065650] aio_complete: ffff8800b6e4b200[19]: ffff8800b6786e00: 0000000000604140 0 1000 0
[ 3048.065651] aio_complete: added to ring ffff8800b6786e00 at [19]
[ 3048.065654] aio_complete: ffff8800b6e4b200[20]: ffff8800b6786e00: 0000000000604180 0 1000 0
[ 3048.065655] aio_complete: added to ring ffff8800b6786e00 at [20]
[ 3048.065658] aio_complete: ffff8800b6e4b200[21]: ffff8800b6786e00: 00000000006041c0 0 1000 0
[ 3048.065659] aio_complete: added to ring ffff8800b6786e00 at [21]
[ 3048.065663] aio_complete: ffff8800b6e4b200[22]: ffff8800b6786e00: 0000000000604200 0 1000 0
[ 3048.065664] aio_complete: added to ring ffff8800b6786e00 at [22]
[ 3048.065668] aio_complete: ffff8800b6e4b200[23]: ffff8800b6786e00: 0000000000604240 0 1000 0
[ 3048.065669] aio_complete: added to ring ffff8800b6786e00 at [23]
[ 3048.065672] aio_complete: ffff8800b6e4b200[24]: ffff8800b6786e00: 0000000000604280 0 1000 0
[ 3048.065672] aio_complete: added to ring ffff8800b6786e00 at [24]
[ 3048.065676] aio_complete: ffff8800b6e4b200[25]: ffff8800b6786e00: 00000000006042c0 0 1000 0
[ 3048.065677] aio_complete: added to ring ffff8800b6786e00 at [25]
[ 3048.065680] aio_complete: ffff8800b6e4b200[26]: ffff8800b6786e00: 0000000000604300 0 1000 0
[ 3048.065681] aio_complete: added to ring ffff8800b6786e00 at [26]
[ 3048.065690] aio_complete: ffff8800b6e4b200[27]: ffff8800b6786e00: 0000000000604340 0 1000 0
[ 3048.065691] aio_complete: added to ring ffff8800b6786e00 at [27]
[ 3048.065694] aio_complete: ffff8800b6e4b200[28]: ffff8800b6786e00: 0000000000604380 0 1000 0
[ 3048.065694] aio_complete: added to ring ffff8800b6786e00 at [28]
[ 3048.065697] aio_complete: ffff8800b6e4b200[29]: ffff8800b6786e00: 00000000006043c0 0 1000 0
[ 3048.065715] aio_complete: added to ring ffff8800b6786e00 at [29]
[ 3048.065721] aio_complete: ffff8800b6e4b200[30]: ffff8800b6786e00: 0000000000604400 0 1000 0
[ 3048.065722] aio_complete: added to ring ffff8800b6786e00 at [30]
[ 3048.065736] aio_complete: ffff8800b6e4b200[31]: ffff8800b6786e00: 0000000000604440 0 1000 0
[ 3048.065738] aio_complete: added to ring ffff8800b6786e00 at [31]
[ 3048.065744] aio_complete: ffff8800b6e4b200[32]: ffff8800b6786e00: 0000000000604480 0 1000 0
[ 3048.065745] aio_complete: added to ring ffff8800b6786e00 at [32]
[ 3048.065748] aio_complete: ffff8800b6e4b200[33]: ffff8800b6786e00: 00000000006044c0 0 1000 0
[ 3048.065749] aio_complete: added to ring ffff8800b6786e00 at [33]
[ 3048.065761] aio_complete: ffff8800b6e4b200[34]: ffff8800b6786e00: 0000000000604500 0 1000 0
[ 3048.065762] aio_complete: added to ring ffff8800b6786e00 at [34]
[ 3048.065765] aio_complete: ffff8800b6e4b200[35]: ffff8800b6786e00: 0000000000604540 0 1000 0
[ 3048.065765] aio_complete: added to ring ffff8800b6786e00 at [35]
[ 3048.065770] aio_complete: ffff8800b6e4b200[36]: ffff8800b6786e00: 0000000000604580 0 1000 0
[ 3048.065771] aio_complete: added to ring ffff8800b6786e00 at [36]
[ 3048.065773] aio_complete: ffff8800b6e4b200[37]: ffff8800b6786e00: 00000000006045c0 0 1000 0
[ 3048.065774] aio_complete: added to ring ffff8800b6786e00 at [37]
[ 3048.065777] aio_complete: ffff8800b6e4b200[38]: ffff8800b6786e00: 0000000000604600 0 1000 0
[ 3048.065778] aio_complete: added to ring ffff8800b6786e00 at [38]
[ 3048.065781] aio_complete: ffff8800b6e4b200[39]: ffff8800b6786e00: 0000000000604640 0 1000 0
[ 3048.065782] aio_complete: added to ring ffff8800b6786e00 at [39]
[ 3048.065785] aio_complete: ffff8800b6e4b200[40]: ffff8800b6786e00: 0000000000604680 0 1000 0
[ 3048.065786] aio_complete: added to ring ffff8800b6786e00 at [40]
[ 3048.065789] aio_complete: ffff8800b6e4b200[41]: ffff8800b6786e00: 00000000006046c0 0 1000 0
[ 3048.065790] aio_complete: added to ring ffff8800b6786e00 at [41]
[ 3048.065793] aio_complete: ffff8800b6e4b200[42]: ffff8800b6786e00: 0000000000604700 0 1000 0
[ 3048.065794] aio_complete: added to ring ffff8800b6786e00 at [42]
[ 3048.065797] aio_complete: ffff8800b6e4b200[43]: ffff8800b6786e00: 0000000000604740 0 1000 0
[ 3048.065798] aio_complete: added to ring ffff8800b6786e00 at [43]
[ 3048.065801] aio_complete: ffff8800b6e4b200[44]: ffff8800b6786e00: 0000000000604780 0 1000 0
[ 3048.065802] aio_complete: added to ring ffff8800b6786e00 at [44]
[ 3048.065805] aio_complete: ffff8800b6e4b200[45]: ffff8800b6786e00: 00000000006047c0 0 1000 0
[ 3048.065806] aio_complete: added to ring ffff8800b6786e00 at [45]
[ 3048.065809] aio_complete: ffff8800b6e4b200[46]: ffff8800b6786e00: 0000000000604800 0 1000 0
[ 3048.065809] aio_complete: added to ring ffff8800b6786e00 at [46]
[ 3048.065812] aio_complete: ffff8800b6e4b200[47]: ffff8800b6786e00: 0000000000604840 0 1000 0
[ 3048.065813] aio_complete: added to ring ffff8800b6786e00 at [47]
[ 3048.065816] aio_complete: ffff8800b6e4b200[48]: ffff8800b6786e00: 0000000000604880 0 1000 0
[ 3048.065817] aio_complete: added to ring ffff8800b6786e00 at [48]
[ 3048.065820] aio_complete: ffff8800b6e4b200[49]: ffff8800b6786e00: 00000000006048c0 0 1000 0
[ 3048.065820] aio_complete: added to ring ffff8800b6786e00 at [49]
[ 3048.065824] aio_complete: ffff8800b6e4b200[50]: ffff8800b6786e00: 0000000000604900 0 1000 0
[ 3048.065825] aio_complete: added to ring ffff8800b6786e00 at [50]
[ 3048.065827] aio_complete: ffff8800b6e4b200[51]: ffff8800b6786e00: 0000000000604940 0 1000 0
[ 3048.065828] aio_complete: added to ring ffff8800b6786e00 at [51]
[ 3048.065831] aio_complete: ffff8800b6e4b200[52]: ffff8800b6786e00: 0000000000604980 0 1000 0
[ 3048.065832] aio_complete: added to ring ffff8800b6786e00 at [52]
[ 3048.065834] aio_complete: ffff8800b6e4b200[53]: ffff8800b6786e00: 00000000006049c0 0 1000 0
[ 3048.065835] aio_complete: added to ring ffff8800b6786e00 at [53]
[ 3048.065838] aio_complete: ffff8800b6e4b200[54]: ffff8800b6786e00: 0000000000604a00 0 1000 0
[ 3048.065839] aio_complete: added to ring ffff8800b6786e00 at [54]
[ 3048.065842] aio_complete: ffff8800b6e4b200[55]: ffff8800b6786e00: 0000000000604a40 0 1000 0
[ 3048.065843] aio_complete: added to ring ffff8800b6786e00 at [55]
[ 3048.065858] aio_complete: ffff8800b6e4b200[56]: ffff8800b6786e00: 0000000000604a80 0 1000 0
[ 3048.065860] aio_complete: added to ring ffff8800b6786e00 at [56]
[ 3048.065865] aio_complete: ffff8800b6e4b200[57]: ffff8800b6786e00: 0000000000604ac0 0 1000 0
[ 3048.065866] aio_complete: added to ring ffff8800b6786e00 at [57]
[ 3048.065870] aio_complete: ffff8800b6e4b200[58]: ffff8800b6786e00: 0000000000604b00 0 1000 0
[ 3048.065870] aio_complete: added to ring ffff8800b6786e00 at [58]
[ 3048.065873] aio_complete: ffff8800b6e4b200[59]: ffff8800b6786e00: 0000000000604b40 0 1000 0
[ 3048.065874] aio_complete: added to ring ffff8800b6786e00 at [59]
[ 3048.065877] aio_complete: ffff8800b6e4b200[60]: ffff8800b6786e00: 0000000000604b80 0 1000 0
[ 3048.065878] aio_complete: added to ring ffff8800b6786e00 at [60]
[ 3048.065881] aio_complete: ffff8800b6e4b200[61]: ffff8800b6786e00: 0000000000604bc0 0 1000 0
[ 3048.065882] aio_complete: added to ring ffff8800b6786e00 at [61]
[ 3048.065884] aio_complete: ffff8800b6e4b200[62]: ffff8800b6786e00: 0000000000604c00 0 1000 0
[ 3048.065885] aio_complete: added to ring ffff8800b6786e00 at [62]
[ 3048.065888] aio_complete: ffff8800b6e4b200[63]: ffff8800b6786e00: 0000000000604c40 0 1000 0
[ 3048.065889] aio_complete: added to ring ffff8800b6786e00 at [63]
[ 3048.065893] aio_complete: ffff8800b6e4b200[64]: ffff8800b6786e00: 0000000000604c80 0 1000 0
[ 3048.065894] aio_complete: added to ring ffff8800b6786e00 at [64]
[ 3048.066254] aio_complete: ffff8800b6e4b200[65]: ffff8800b6786e00: 0000000000604cc0 0 1000 0
[ 3048.066258] aio_complete: added to ring ffff8800b6786e00 at [65]
[ 3048.066269] aio_complete: ffff8800b6e4b200[66]: ffff8800b6786e00: 0000000000604d00 0 1000 0
[ 3048.066274] aio_complete: added to ring ffff8800b6786e00 at [66]
[ 3048.066284] aio_complete: ffff8800b6e4b200[67]: ffff8800b6786e00: 0000000000604d40 0 1000 0
[ 3048.066287] aio_complete: added to ring ffff8800b6786e00 at [67]
[ 3048.066298] aio_complete: ffff8800b6e4b200[68]: ffff8800b6786e00: 0000000000604d80 0 1000 0
[ 3048.066301] aio_complete: added to ring ffff8800b6786e00 at [68]
[ 3048.066312] aio_complete: ffff8800b6e4b200[69]: ffff8800b6786e00: 0000000000604dc0 0 1000 0
[ 3048.066315] aio_complete: added to ring ffff8800b6786e00 at [69]
[ 3048.066326] aio_complete: ffff8800b6e4b200[70]: ffff8800b6786e00: 0000000000604e00 0 1000 0
[ 3048.066329] aio_complete: added to ring ffff8800b6786e00 at [70]
[ 3048.066340] aio_complete: ffff8800b6e4b200[71]: ffff8800b6786e00: 0000000000604e40 0 1000 0
[ 3048.066343] aio_complete: added to ring ffff8800b6786e00 at [71]
[ 3048.066354] aio_complete: ffff8800b6e4b200[72]: ffff8800b6786e00: 0000000000604e80 0 1000 0
[ 3048.066357] aio_complete: added to ring ffff8800b6786e00 at [72]
[ 3048.066370] aio_complete: ffff8800b6e4b200[73]: ffff8800b6786e00: 0000000000604ec0 0 1000 0
[ 3048.066373] aio_complete: added to ring ffff8800b6786e00 at [73]
[ 3048.066383] aio_complete: ffff8800b6e4b200[74]: ffff8800b6786e00: 0000000000604f00 0 1000 0
[ 3048.066387] aio_complete: added to ring ffff8800b6786e00 at [74]
[ 3048.066398] aio_complete: ffff8800b6e4b200[75]: ffff8800b6786e00: 0000000000604f40 0 1000 0
[ 3048.066401] aio_complete: added to ring ffff8800b6786e00 at [75]
[ 3048.066412] aio_complete: ffff8800b6e4b200[76]: ffff8800b6786e00: 0000000000604f80 0 1000 0
[ 3048.066416] aio_complete: added to ring ffff8800b6786e00 at [76]
[ 3048.066426] aio_complete: ffff8800b6e4b200[77]: ffff8800b6786e00: 0000000000604fc0 0 1000 0
[ 3048.066432] aio_complete: added to ring ffff8800b6786e00 at [77]
[ 3048.066446] aio_complete: ffff8800b6e4b200[78]: ffff8800b6786e00: 0000000000605000 0 1000 0
[ 3048.066449] aio_complete: added to ring ffff8800b6786e00 at [78]
[ 3048.066460] aio_complete: ffff8800b6e4b200[79]: ffff8800b6786e00: 0000000000605040 0 1000 0
[ 3048.066463] aio_complete: added to ring ffff8800b6786e00 at [79]
[ 3048.066474] aio_complete: ffff8800b6e4b200[80]: ffff8800b6786e00: 0000000000605080 0 1000 0
[ 3048.066477] aio_complete: added to ring ffff8800b6786e00 at [80]
[ 3048.066488] aio_complete: ffff8800b6e4b200[81]: ffff8800b6786e00: 00000000006050c0 0 1000 0
[ 3048.066492] aio_complete: added to ring ffff8800b6786e00 at [81]
[ 3048.066502] aio_complete: ffff8800b6e4b200[82]: ffff8800b6786e00: 0000000000605100 0 1000 0
[ 3048.066507] aio_complete: added to ring ffff8800b6786e00 at [82]
[ 3048.066518] aio_complete: ffff8800b6e4b200[83]: ffff8800b6786e00: 0000000000605140 0 1000 0
[ 3048.066521] aio_complete: added to ring ffff8800b6786e00 at [83]
[ 3048.066532] aio_complete: ffff8800b6e4b200[84]: ffff8800b6786e00: 0000000000605180 0 1000 0
[ 3048.066536] aio_complete: added to ring ffff8800b6786e00 at [84]
[ 3048.066547] aio_complete: ffff8800b6e4b200[85]: ffff8800b6786e00: 00000000006051c0 0 1000 0
[ 3048.066550] aio_complete: added to ring ffff8800b6786e00 at [85]
[ 3048.066561] aio_complete: ffff8800b6e4b200[86]: ffff8800b6786e00: 0000000000605200 0 1000 0
[ 3048.066565] aio_complete: added to ring ffff8800b6786e00 at [86]
[ 3048.066576] aio_complete: ffff8800b6e4b200[87]: ffff8800b6786e00: 0000000000605240 0 1000 0
[ 3048.066579] aio_complete: added to ring ffff8800b6786e00 at [87]
[ 3048.066590] aio_complete: ffff8800b6e4b200[88]: ffff8800b6786e00: 0000000000605280 0 1000 0
[ 3048.066594] aio_complete: added to ring ffff8800b6786e00 at [88]
[ 3048.066605] aio_complete: ffff8800b6e4b200[89]: ffff8800b6786e00: 00000000006052c0 0 1000 0
[ 3048.066608] aio_complete: added to ring ffff8800b6786e00 at [89]
[ 3048.066619] aio_complete: ffff8800b6e4b200[90]: ffff8800b6786e00: 0000000000605300 0 1000 0
[ 3048.066623] aio_complete: added to ring ffff8800b6786e00 at [90]
[ 3048.066633] aio_complete: ffff8800b6e4b200[91]: ffff8800b6786e00: 0000000000605340 0 1000 0
[ 3048.066638] aio_complete: added to ring ffff8800b6786e00 at [91]
[ 3048.066652] aio_complete: ffff8800b6e4b200[92]: ffff8800b6786e00: 0000000000605380 0 1000 0
[ 3048.066655] aio_complete: added to ring ffff8800b6786e00 at [92]
[ 3048.066666] aio_complete: ffff8800b6e4b200[93]: ffff8800b6786e00: 00000000006053c0 0 1000 0
[ 3048.066670] aio_complete: added to ring ffff8800b6786e00 at [93]
[ 3048.066681] aio_complete: ffff8800b6e4b200[94]: ffff8800b6786e00: 0000000000605400 0 1000 0
[ 3048.066684] aio_complete: added to ring ffff8800b6786e00 at [94]
[ 3048.066696] aio_complete: ffff8800b6e4b200[95]: ffff8800b6786e00: 0000000000605440 0 1000 0
[ 3048.066699] aio_complete: added to ring ffff8800b6786e00 at [95]
[ 3048.066716] aio_complete: ffff8800b6e4b200[96]: ffff8800b6786e00: 0000000000605480 0 1000 0
[ 3048.066720] aio_complete: added to ring ffff8800b6786e00 at [96]
[ 3048.066731] aio_complete: ffff8800b6e4b200[97]: ffff8800b6786e00: 00000000006054c0 0 1000 0
[ 3048.066734] aio_complete: added to ring ffff8800b6786e00 at [97]
[ 3048.066745] aio_complete: ffff8800b6e4b200[98]: ffff8800b6786e00: 0000000000605500 0 1000 0
[ 3048.066750] aio_complete: added to ring ffff8800b6786e00 at [98]
[ 3048.066761] aio_complete: ffff8800b6e4b200[99]: ffff8800b6786e00: 0000000000605540 0 1000 0
[ 3048.066764] aio_complete: added to ring ffff8800b6786e00 at [99]
[ 3048.066775] aio_complete: ffff8800b6e4b200[100]: ffff8800b6786e00: 0000000000605580 0 1000 0
[ 3048.066779] aio_complete: added to ring ffff8800b6786e00 at [100]
[ 3048.066790] aio_complete: ffff8800b6e4b200[101]: ffff8800b6786e00: 00000000006055c0 0 1000 0
[ 3048.066794] aio_complete: added to ring ffff8800b6786e00 at [101]
[ 3048.066805] aio_complete: ffff8800b6e4b200[102]: ffff8800b6786e00: 0000000000605600 0 1000 0
[ 3048.066808] aio_complete: added to ring ffff8800b6786e00 at [102]
[ 3048.066820] aio_complete: ffff8800b6e4b200[103]: ffff8800b6786e00: 0000000000605640 0 1000 0
[ 3048.066823] aio_complete: added to ring ffff8800b6786e00 at [103]
[ 3048.066834] aio_complete: ffff8800b6e4b200[104]: ffff8800b6786e00: 0000000000605680 0 1000 0
[ 3048.066837] aio_complete: added to ring ffff8800b6786e00 at [104]
[ 3048.066848] aio_complete: ffff8800b6e4b200[105]: ffff8800b6786e00: 00000000006056c0 0 1000 0
[ 3048.067233] aio_complete: added to ring ffff8800b6786e00 at [105]
[ 3048.067850] aio_complete: ffff8800b6e4b200[106]: ffff8800b6786e00: 0000000000605700 0 1000 0
[ 3048.067855] aio_complete: added to ring ffff8800b6786e00 at [106]
[ 3048.067868] aio_complete: ffff8800b6e4b200[107]: ffff8800b6786e00: 0000000000605740 0 1000 0
[ 3048.067872] aio_complete: added to ring ffff8800b6786e00 at [107]
[ 3048.067883] aio_complete: ffff8800b6e4b200[108]: ffff8800b6786e00: 0000000000605780 0 1000 0
[ 3048.067886] aio_complete: added to ring ffff8800b6786e00 at [108]
[ 3048.067897] aio_complete: ffff8800b6e4b200[109]: ffff8800b6786e00: 00000000006057c0 0 1000 0
[ 3048.067901] aio_complete: added to ring ffff8800b6786e00 at [109]
[ 3048.067911] aio_complete: ffff8800b6e4b200[110]: ffff8800b6786e00: 0000000000605800 0 1000 0
[ 3048.067915] aio_complete: added to ring ffff8800b6786e00 at [110]
[ 3048.067926] aio_complete: ffff8800b6e4b200[111]: ffff8800b6786e00: 0000000000605840 0 1000 0
[ 3048.067930] aio_complete: added to ring ffff8800b6786e00 at [111]
[ 3048.067941] aio_complete: ffff8800b6e4b200[112]: ffff8800b6786e00: 0000000000605880 0 1000 0
[ 3048.067956] aio_complete: added to ring ffff8800b6786e00 at [112]
[ 3048.067973] aio_complete: ffff8800b6e4b200[113]: ffff8800b6786e00: 00000000006058c0 0 1000 0
[ 3048.067983] aio_complete: added to ring ffff8800b6786e00 at [113]
[ 3048.068000] aio_complete: ffff8800b6e4b200[114]: ffff8800b6786e00: 0000000000605900 0 1000 0
[ 3048.068010] aio_complete: added to ring ffff8800b6786e00 at [114]
[ 3048.068266] aio_complete: ffff8800b6e4b200[115]: ffff8800b6786e00: 0000000000605940 0 1000 0
[ 3048.068275] aio_complete: added to ring ffff8800b6786e00 at [115]
[ 3048.068292] aio_complete: ffff8800b6e4b200[116]: ffff8800b6786e00: 0000000000605980 0 1000 0
[ 3048.068301] aio_complete: added to ring ffff8800b6786e00 at [116]
[ 3048.068320] aio_complete: ffff8800b6e4b200[117]: ffff8800b6786e00: 00000000006059c0 0 1000 0
[ 3048.068323] aio_complete: added to ring ffff8800b6786e00 at [117]
[ 3048.068334] aio_complete: ffff8800b6e4b200[118]: ffff8800b6786e00: 0000000000605a00 0 1000 0
[ 3048.068337] aio_complete: added to ring ffff8800b6786e00 at [118]
[ 3048.068348] aio_complete: ffff8800b6e4b200[119]: ffff8800b6786e00: 0000000000605a40 0 1000 0
[ 3048.068351] aio_complete: added to ring ffff8800b6786e00 at [119]
[ 3048.068366] aio_complete: ffff8800b6e4b200[120]: ffff8800b6786e00: 0000000000605a80 0 1000 0
[ 3048.068370] aio_complete: added to ring ffff8800b6786e00 at [120]
[ 3048.068381] aio_complete: ffff8800b6e4b200[121]: ffff8800b6786e00: 0000000000605ac0 0 1000 0
[ 3048.068384] aio_complete: added to ring ffff8800b6786e00 at [121]
[ 3048.068395] aio_complete: ffff8800b6e4b200[122]: ffff8800b6786e00: 0000000000605b00 0 1000 0
[ 3048.068399] aio_complete: added to ring ffff8800b6786e00 at [122]
[ 3048.068410] aio_complete: ffff8800b6e4b200[123]: ffff8800b6786e00: 0000000000605b40 0 1000 0
[ 3048.068414] aio_complete: added to ring ffff8800b6786e00 at [123]
[ 3048.068436] aio_complete: ffff8800b6e4b200[124]: ffff8800b6786e00: 0000000000605b80 0 1000 0
[ 3048.068444] aio_complete: added to ring ffff8800b6786e00 at [124]
[ 3048.068462] aio_complete: ffff8800b6e4b200[125]: ffff8800b6786e00: 0000000000605bc0 0 1000 0
[ 3048.068472] aio_complete: added to ring ffff8800b6786e00 at [125]
[ 3048.068489] aio_complete: ffff8800b6e4b200[126]: ffff8800b6786e00: 0000000000605c00 0 1000 0
[ 3048.068498] aio_complete: added to ring ffff8800b6786e00 at [126]
[ 3048.068658] aio_read_events_ring: h0 t126 m127
[ 3048.068664] aio_read_events_ring: 126  h126 t126
[ 3048.068717] aio_complete: ffff8800b6e4b200[0]: ffff8800b6786e00: 0000000000605c40 0 1000 0
[ 3048.068721] aio_complete: added to ring ffff8800b6786e00 at [0]
[ 3048.068733] aio_complete: ffff8800b6e4b200[1]: ffff8800b6786e00: 0000000000605c80 0 1000 0
[ 3048.068736] aio_complete: added to ring ffff8800b6786e00 at [1]
[ 3048.068757] aio_read_events_ring: h126 t1 m127
[ 3048.068761] aio_read_events_ring: 2  h1 t1
[ 3048.079517] free_ioctx: freeing ffff8800b6e4b200
[ 3048.079525] aio_free_ring: pid(1005) [0] page->count=2

/*
* aio_bug.c
*
* Copyright (C) 2014, Dan Aloni, Kernelim Ltd.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*/

#include <assert.h>
#include <errno.h>
#include <libaio.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/param.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>

#define MAX_IOS 128

const int max_events = 32;
const int io_size = 0x1000;
io_context_t io_ctx;
struct iocb io[MAX_IOS];
struct iocb *iops[MAX_IOS];
struct iovec iovecs[MAX_IOS];
struct io_event events[MAX_IOS];
char *data;

long submitted = 0;
long completed = 0;
long pending = 0;

void prune(void)

{
	int ret;

        ret = io_getevents(io_ctx, pending, MAX_IOS, events, NULL);
	if (ret > 0) {
		printf("Completed: %d\n", ret);
		completed += ret;
		pending -= ret;
	}
}

int main(int argc, char **argv)

{
	int ret;
	int fd;
	const char *filename = "aio_bug_temp";
	long i, to_submit;
	struct iocb **iocb_sub;

	ret = io_setup(max_events, &io_ctx);
	assert(!ret);

	unlink(filename);
	fd = open(filename, O_CREAT | O_RDWR | O_DIRECT, 0644);
	assert(fd >= 0);

	ret = ftruncate(fd, MAX_IOS * io_size);
	assert(!ret);

	data = mmap(NULL, io_size * MAX_IOS, PROT_READ | PROT_WRITE,
		    MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
	assert(data != MAP_FAILED);

	for (i = 0; i < MAX_IOS; i++) {
		iops[i] = &io[i];
		io[i].data = io;
		iovecs[i].iov_base = &data[io_size * i];
		iovecs[i].iov_len = io_size;
		io_prep_preadv(&io[i], fd, &iovecs[i], 1, 0);
	}

	to_submit = MAX_IOS;
	iocb_sub = iops;

	while (submitted < MAX_IOS) {
		printf("Submitting: %ld\n", to_submit);

		ret = io_submit(io_ctx, to_submit, iocb_sub);
		if (ret >= 0) {
			printf("Submitted: %d\n", ret);
			submitted += ret;
			iocb_sub += ret;
			pending += ret;
			to_submit -= ret;
		} else {
			if (ret == -EAGAIN) {
				printf("Submitted too much, that's okay\n");
				prune();
			}
		}
	}

	prune();
	io_destroy(io_ctx);

	close(fd);

	printf("Verifying...\n");

	assert(completed == submitted);

	printf("OK\n");

	return 0;
}

/*
*
* Good output:
*

 *   Submitting: 128
 *   Submitted: 126
 *   Submitting: 2
 *   Submitted too much, that's okay
 *   Completed: 126
 *   Submitting: 2
 *   Submitted: 2
 *   Completed: 2
 *   Verifying...
 *   OK
 *
 * Bad output:
 *
 *   Submitting: 128
 *   Submitted: 128
 *   <program stuck, IO/s swallowed>

*
*/

Argh, that would be a problem.

...

Reverting isn't okay, as that reintroduces another regression. We need to come up with a fix for this issue that doesn't reintroduce the other regression for events reaped in user space. Let me have a look and see what I can come up with...

-ben

"Thought is the essence of where you are now."

About the original regression you mention, is there a program you can indicate that reproduces it? On my setups, the regression testing in libaio was not able to detect the current regression too.

Dan Aloni

You can trigger the behaviour with fio by using userspace event reaping. Adding a test case for that behaviour to libaio would be a good idea.

I thought about how to fix this, and it isn't actually that hard. Move the put_reqs_available() call back into event consumption, and then add code in the submit path to call put_reqs_available() if the system runs out of events by noticing that there is free space in the event ring. Something along the lines below should do it (please note, this is completely untested!). I'll test and polish this off tomorrow, as it's getting a bit late here.

-ben

"Thought is the essence of where you are now."

index ae63587..749ac22 100644
@@ -142,6 +142,7 @@ struct kioctx {
 	struct {
 		unsigned	tail;
 		spinlock_t	completion_lock;
+		unsigned	completed_events;
 	} ____cacheline_aligned_in_smp;
 
 	struct page		*internal_pages[AIO_RING_PAGES];

@@ -857,6 +858,31 @@ out:
 	return ret;
 }
 
+static void refill_reqs_available(struct kioctx *ctx)
+{
+	spin_lock_irq(&ctx->completion_lock);
+	if (ctx->completed_events) {
+		unsigned head, tail, avail, completed;
+		struct aio_ring *ring;
+
+		ring = kmap_atomic(ctx->ring_pages[0]);
+		head = ACCESS_ONCE(ring->head);
+		tail = ACCESS_ONCE(ring->tail);
+		kunmap_atomic(ring);
+
+		avail = (head <= tail ?  tail : ctx->nr_events) - head;
+		completed = ctx->completed_events;
+		if (avail < completed)
+			completed -= avail;
+		else
+			completed = 0;
+		put_reqs_available(ctx, completed);
+	}
+
+	spin_unlock_irq(&ctx->completion_lock);
+}
+
+
 /* aio_get_req
  *	Allocate a slot for an aio request.
  * Returns NULL if no requests are free.

@@ -865,8 +891,11 @@ static inline struct kiocb *aio_get_req(struct kioctx *ctx)
 {
 	struct kiocb *req;
 
-	if (!get_reqs_available(ctx))
-		return NULL;
+	if (!get_reqs_available(ctx)) {
+		refill_reqs_available(ctx);
+		if (!get_reqs_available(ctx))
+			return NULL;
+	}
 
 	req = kmem_cache_alloc(kiocb_cachep, GFP_KERNEL|__GFP_ZERO);
 	if (unlikely(!req))

@@ -1005,7 +1034,6 @@ void aio_complete(struct kiocb *iocb, long res, long res2)
 
 	/* everything turned out well, dispose of the aiocb. */
 	kiocb_free(iocb);
-	put_reqs_available(ctx, 1);
 
 	/*
 	 * We have to order our ring_info tail store above and test

Dan, does this patch work for you? It seems to pass your test program when I run it in a vm...

-ben

"Thought is the essence of where you are now."

Sorry, I was waiting for a new patch from your direction, I should have replied earlier. What bothered me about the patch you sent is that completed_events is added as a new field but nothing assigns to it, so I wonder how it can be effective.

Dan Aloni

Ah, that was missing a hunk then. Try this version instead.

-ben

"Thought is the essence of where you are now."

index ae63587..fbdcc47 100644
@@ -142,6 +142,7 @@ struct kioctx {
 	struct {
 		unsigned	tail;
 		spinlock_t	completion_lock;
+		unsigned	completed_events;
 	} ____cacheline_aligned_in_smp;
 
 	struct page		*internal_pages[AIO_RING_PAGES];

@@ -857,6 +858,31 @@ out:
 	return ret;
 }
 
+static void refill_reqs_available(struct kioctx *ctx)
+{
+	spin_lock_irq(&ctx->completion_lock);
+	if (ctx->completed_events) {
+		unsigned head, tail, avail, completed;
+		struct aio_ring *ring;
+
+		ring = kmap_atomic(ctx->ring_pages[0]);
+		head = ACCESS_ONCE(ring->head);
+		tail = ACCESS_ONCE(ring->tail);
+		kunmap_atomic(ring);
+
+		avail = (head <= tail ?  tail : ctx->nr_events) - head;
+		completed = ctx->completed_events;
+		if (avail < completed)
+			completed -= avail;
+		else
+			completed = 0;
+		put_reqs_available(ctx, completed);
+	}
+
+	spin_unlock_irq(&ctx->completion_lock);
+}
+
+
 /* aio_get_req
  *	Allocate a slot for an aio request.
  * Returns NULL if no requests are free.

@@ -865,8 +891,11 @@ static inline struct kiocb *aio_get_req(struct kioctx *ctx)
 {
 	struct kiocb *req;
 
-	if (!get_reqs_available(ctx))
-		return NULL;
+	if (!get_reqs_available(ctx)) {
+		refill_reqs_available(ctx);
+		if (!get_reqs_available(ctx))
+			return NULL;
+	}
 
 	req = kmem_cache_alloc(kiocb_cachep, GFP_KERNEL|__GFP_ZERO);
 	if (unlikely(!req))

@@ -958,6 +987,7 @@ void aio_complete(struct kiocb *iocb, long res, long res2)
 	 */
 	spin_lock_irqsave(&ctx->completion_lock, flags);
 
+	ctx->completed_events++;
 	tail = ctx->tail;
 	pos = tail + AIO_EVENTS_OFFSET;
 

@@ -1005,7 +1035,6 @@ void aio_complete(struct kiocb *iocb, long res, long res2)
 
 	/* everything turned out well, dispose of the aiocb. */
 	kiocb_free(iocb);
-	put_reqs_available(ctx, 1);
 
 	/*
 	 * We have to order our ring_info tail store above and test

Ben, seems that the test program needs some twidling to make the bug appear still by setting MAX_IOS to 256 (and it still passes on a kernel with the original patch reverted). Under this condition the ring buffer size remains 128 (here, 32*4 CPUs), and it is overrun on the second attempt.

    $ ./aio_bug 
    Submitting: 256
    Submitted: 126
    Submitting: 130
    Submitted too much, that's okay
    Completed: 126
    Submitting: 130
    Submitted: 130
    <stuck>

I think I have found two problems with your patch: first, the completed_events field is never decremented so it goes up until 2^32 wraparound. So I tested with 'ctx->completed_events -= completed;' there (along with some prints), but testing revealed that this didn't solve the problem, so secondly, I also fixed the 'avail = ' line. The case where the 'head > tail' case didn't look correct to me.

So the good news is that it works now with fix below and MAX_IOS=256 and even with MAX_IOS=512. You can git-amend this it to your patch I guess.

Signed-off: Dan Aloni <[email protected]>

index 6982357d9372..eafc96c60a8c 100644
@@ -893,12 +893,20 @@ static void refill_reqs_available(struct kioctx *ctx)
 		tail = ACCESS_ONCE(ring->tail);
 		kunmap_atomic(ring);
 
-		avail = (head <= tail ?  tail : ctx->nr_events) - head;
+		if (head <= tail)
+			avail = tail - head;
+		else
+			avail = ctx->nr_events - (head - tail);
+
 		completed = ctx->completed_events;
+		pr_debug("%u %u  h%u t%u\n", avail, completed, head, tail);
 		if (avail < completed)
 			completed -= avail;
 		else
 			completed = 0;
+		pr_debug("completed %u\n", completed);
+
+		ctx->completed_events -= completed;
 		put_reqs_available(ctx, completed);
 	}
 

BTW, I am not an expert on this code so I am still not sure that 'ctx->completed_events' wouldn't get wrong if for instance - one SMP core does userspace event reaping and another calls io_submit(). Do you think it would work alright in that case?

Dan Aloni

Ugh.

Ben, at this point my gut feel is that we should just revert the original "fix", and you should take a much deeper look at this all. The original "fix" was more broken then the leak it purported to fix, and now the patch to fix your fix has gone through two iterations and *still* Dan is finding bugs in it. I'm getting the feeling that this code needs more thinking than you are actually putting into it.

Linus

That's why I had't sent it out as an official [PATCH] just yet. I think things worked out okay since the untested patch I sent out pointed Dan in the right direction and he was able to put some effort into it while I didn't have to immediate time to do so. I just put in a few hours to polish off the final details on this fix now, and it should be coming your way as soon as I get an ack back from Dan. Hopefully Kent can review it as well, since I had to modify the approach to try to retain the advantages of his batched reqs_available handling and avoid bouncing the cacheline ctx->completion_lock is on during io_submit(). Cheers,

-ben

"Thought is the essence of where you are now."

Ugh, I should've dug into this a lot sooner... mea culpa, I originally wrote this percpu reqs_available() nonsense (can I unwrite code?)

Coming into the discussion late, here's my understanding, please correct me if I'm wrong and I'll blame the cold medication...

The original patch f8567a3845ac05bb28f3c1b478ef752762bd39ef was broken because it changed the logic so that a slot was considered available once an iocb completion had been delivered to the ring buffer. BUT THE THING THAT IT'S COUNTING IS AVAILABLE SLOTS IN THE RINGBUFFER: get_reqs_available() is reserving a slot in the ringbuffer, we can't do the put() until the event is actually consumed.

took me a minute to figure out the logic in Ben's current patch, but here's what's going on as I understand it

* maintain a count of events we add to the ring
* if there's fewer events in the ring than that count, the difference has been reaped so we can release their slots

and nothing else changes. we're just accounting one more thing, completed_events, and the rest stays the same.

so, it took me a bit to figure out what was being accounted, I think the code could stand some new comments explaining how this works (though I'm certainly not one to throw stones) - but the new version makes sense to me, I do agree with the approach.

I think I owe Ben beer at some point...

Thanks for checking that. I've made some further changes to your test program to be able to support both userspace event reaping and using the io_getevents() syscall. I also added a --max-ios= parameter to allow using different values of MAX_IOS for testing. A copy of the modified source is available at http://www.kvack.org/~bcrl/20140824-aio_bug.c . With this version of the patch, both means of reaping events now work corectly. The aio_bug.c code still needs to be added to libaio's set of tests, but that's a separate email.

Yeah, sorry for not checking that. I didn't have time to sit down for a couple of hours to test and verify it until today. Your changes for handling ring wrap around correctly look good after inspecting the code and how it reacts to the various conditions.

I ended up making a bunch of changes to the patch to ensure the user space tainted value of ring->head is clamped. I also made the code use ctx->tail to avoid having to do a mod operation on the value of tail as well. The code is rearranged to optimistically call refill_reqs_available() in aio_complete() to try to avoid doing this only at io_submit() time. This helps to avoid having to perform refill from the io_submit() path, as io_submit() does not otherwise have to take ctx->completion_lock and potentially bounce that cache line around.

If you can have a quick look and acknowledge with your Signed-off-by, I'll add it to the commit and send a pull request. With these changes and the modified aio_bug.c, the test passes using various random values for max_ios, as well as both with and without --user_getevents validating that both regressions involved have now been fixed.

...

This should be SMP safe: both ctx->completed_events and ctx->tail are protected ctx->completion_lock. Additionally, possible races with grabbing the value of ring->head should also be safe, and I added a comment explaining why. Does that address your concerns? Cheers,

-ben

"Thought is the essence of where you are now."
---- SNIP: from git://git.kvack.org/~bcrl/aio-fixes.git ----

From: Benjamin LaHaise <[email protected]>
Date: Sun, 24 Aug 2014 13:14:05 -0400
Subject: [PATCH] aio: fix reqs_available handling

As reported by Dan Aloni <[email protected]>, the commit "aio: fix aio request leak when events are reaped by userspace" introduces a regression when user code attempts to perform io_submit() with more events than are available in the ring buffer. Reverting that commit would reintroduce a regression when user space event reaping is used.

Fixing this bug is a bit more involved than the previous attempts to fix this regression. Since we do not have a single point at which we can count events as being reaped by user space and io_getevents(), we have to track event completion by looking at the number of events left in the event ring. So long as there are as many events in the ring buffer as there have been completion events generate, we cannot call put_reqs_available(). The code to check for this is now placed in refill_reqs_available().

A test program from Dan and modified by me for verifying this bug is available at http://www.kvack.org/~bcrl/20140824-aio_bug.c .

---
 fs/aio.c | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 73 insertions(+), 4 deletions(-)

index ae63587..24c4fe4 100644
@@ -141,6 +141,7 @@ struct kioctx {
 
 	struct {
 		unsigned	tail;
+		unsigned	completed_events;
 		spinlock_t	completion_lock;
 	} ____cacheline_aligned_in_smp;
 

@@ -857,6 +858,68 @@ out:
 	return ret;
 }
 
+/* refill_reqs_available
+ *	Updates the reqs_available reference counts used for tracking the
+ *	number of free slots in the completion ring.  This can be called
+ *	from aio_complete() (to optimistically update reqs_available) or
+ *	from aio_get_req() (the we're out of events case).  It must be
+ *	called holding ctx->completion_lock.
+ */
+static void refill_reqs_available(struct kioctx *ctx, unsigned head,
+                                  unsigned tail)
+{
+	unsigned events_in_ring, completed;
+
+	/* Clamp head since userland can write to it. */
+	head %= ctx->nr_events;
+	if (head <= tail)
+		events_in_ring = tail - head;
+	else
+		events_in_ring = ctx->nr_events - (head - tail);
+
+	completed = ctx->completed_events;
+	if (events_in_ring < completed)
+		completed -= events_in_ring;
+	else
+		completed = 0;
+
+	if (!completed)
+		return;
+
+	ctx->completed_events -= completed;
+	put_reqs_available(ctx, completed);
+}
+
+/* user_refill_reqs_available
+ *	Called to refill reqs_available when aio_get_req() encounters an
+ *	out of space in the completion ring.
+ */
+static void user_refill_reqs_available(struct kioctx *ctx)
+{
+	spin_lock_irq(&ctx->completion_lock);
+	if (ctx->completed_events) {
+		struct aio_ring *ring;
+		unsigned head;
+
+		/* Access of ring->head may race with aio_read_events_ring() 
+		 * here, but that's okay since whether we read the old version
+		 * or the new version, and either will be valid.  The important
+		 * part is that head cannot pass tail since we prevent
+		 * aio_complete() from updating tail by holding
+		 * ctx->completion_lock.  Even if head is invalid, the check
+		 * against ctx->completed_events below will make sure we do the
+		 * safe/right thing.
+		 */
+		ring = kmap_atomic(ctx->ring_pages[0]);
+		head = ring->head;
+		kunmap_atomic(ring);
+
+		refill_reqs_available(ctx, head, ctx->tail);
+	}
+
+	spin_unlock_irq(&ctx->completion_lock);
+}
+
 /* aio_get_req
  *	Allocate a slot for an aio request.
  * Returns NULL if no requests are free.

@@ -865,8 +928,11 @@ static inline struct kiocb *aio_get_req(struct kioctx *ctx)
 {
 	struct kiocb *req;
 
-	if (!get_reqs_available(ctx))
-		return NULL;
+	if (!get_reqs_available(ctx)) {
+		user_refill_reqs_available(ctx);
+		if (!get_reqs_available(ctx))
+			return NULL;
+	}
 
 	req = kmem_cache_alloc(kiocb_cachep, GFP_KERNEL|__GFP_ZERO);
 	if (unlikely(!req))

@@ -925,8 +991,8 @@ void aio_complete(struct kiocb *iocb, long res, long res2)
 	struct kioctx	*ctx = iocb->ki_ctx;
 	struct aio_ring	*ring;
 	struct io_event	*ev_page, *event;
+	unsigned tail, pos, head;
 	unsigned long	flags;
-	unsigned tail, pos;
 
 	/*
 	 * Special case handling for sync iocbs:

@@ -987,10 +1053,14 @@ void aio_complete(struct kiocb *iocb, long res, long res2)
 	ctx->tail = tail;
 
 	ring = kmap_atomic(ctx->ring_pages[0]);
+	head = ring->head;
 	ring->tail = tail;
 	kunmap_atomic(ring);
 	flush_dcache_page(ctx->ring_pages[0]);
 
+	ctx->completed_events++;
+	if (ctx->completed_events > 1)
+		refill_reqs_available(ctx, head, tail);
 	spin_unlock_irqrestore(&ctx->completion_lock, flags);
 
 	pr_debug("added to ring %p at [%u]\n", iocb, tail);

@@ -1005,7 +1075,6 @@ void aio_complete(struct kiocb *iocb, long res, long res2)
 
 	/* everything turned out well, dispose of the aiocb. */
 	kiocb_free(iocb);
-	put_reqs_available(ctx, 1);
 
 	/*
 	 * We have to order our ring_info tail store above and test

1.8.2.1

I like your extension to the test program. I have a suggestion for the integration inside libaio's harness, since the original issue depended on the size of the ring buffer, let's have max_ios be picked from {ring->nr + 1, ring->nr - 1, ring->nr}*{1,2,3,4} for the various test cases. I guess Jeff who is maintaining it will have some ideas too.

I haven't tested it yet due to lack of more time today (will give it a try tomorrow), but I've reviewed and didn't find any problem.

Yes, thanks.

Dan Aloni

I can't think of anything in addition to what you've mentioned. Attached is a version that I've tested. The only modifications were to make it work with the libaio test harness. If you have time to review it, that would be great. I've tested it on both working and failing kernels.

Cheers,
Jeff

/*
* aio-test-ring-buffer-overflow.c
*
* Copyright (C) 2014, Dan Aloni, Kernelim Ltd.
* Copyright (C) 2014, Benjamin LaHaise <[email protected]>.
* Copyright (C) 2014, Jeff Moyer <[email protected]>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*/
#ifndef _GNU_SOURCE
#define _GNU_SOURCE 1
#endif
#include <assert.h>
#include <errno.h>
#include <libaio.h>
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/param.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <sys/wait.h>

const int max_events = 32;
const int io_size = 0x1000;
struct iocb *io;
struct iocb **iops;
struct iovec *iovecs;
struct io_event *events;
char *data;

long submitted = 0;
long completed = 0;
long pending = 0;

#define SYS_IO_GETEVENTS 0
#define USER_GETEVENTS 1

static volatile sig_atomic_t done = 0;

struct aio_ring {
	unsigned	id;	/* kernel internal index number */
	unsigned	nr;	/* number of io_events */
	volatile unsigned	head;
	volatile unsigned	tail;

	unsigned	magic;
	unsigned	compat_features;
	unsigned	incompat_features;
	unsigned	header_length;	/* size of aio_ring */

	struct io_event io_events[0];
};

int get_ring_size(int nr_events)

{
	io_context_t ctx;
	int ret, ring_size;
	struct aio_ring *ring;

	memset(&ctx, 0, sizeof(ctx));
	ret = io_setup(nr_events, &ctx);
	assert(!ret);

	ring = (void *)ctx;
	ring_size = ring->nr;

	ret = io_destroy(ctx);
	assert(!ret);

	return ring_size;
}

int user_getevents(io_context_t ctx, int nr_events, struct io_event *event)

{
	struct aio_ring *ring = (void *)ctx;
	int completed = 0;
	while ((completed < nr_events) && (ring->head != ring->tail)) {
		unsigned new_head = ring->head;
		*event = ring->io_events[new_head];
		new_head += 1;
		new_head %= ring->nr;
		ring->head = new_head;
		completed++;
	}
	return completed;
}

void prune(io_context_t io_ctx, int max_ios, int getevents_type)

{
	int ret;

	if (getevents_type == USER_GETEVENTS)
        	ret = user_getevents(io_ctx, max_ios, events);
	else
        	ret = io_getevents(io_ctx, pending, max_ios, events, NULL);
	if (ret > 0) {
		printf("Completed: %d\n", ret);
		completed += ret;
		pending -= ret;
	}
}

void run_test(int max_ios, int getevents_type)

{
	int fd, ret;
	long i, to_submit;
	struct iocb **iocb_sub;
	io_context_t io_ctx;
	const char *filename = "testfile";

	memset(&io_ctx, 0, sizeof(io_ctx));
	ret = io_setup(max_events, &io_ctx);
	assert(!ret);

	io = calloc(max_ios, sizeof(*io));
	iops = calloc(max_ios, sizeof(*iops));
	iovecs = calloc(max_ios, sizeof(*iovecs));
	events = calloc(max_ios, sizeof(*events));

	unlink(filename);
	fd = open(filename, O_CREAT | O_RDWR | O_DIRECT, 0644);
	assert(fd >= 0);

	ret = ftruncate(fd, max_ios * io_size);
	assert(!ret);

	data = mmap(NULL, io_size * max_ios, PROT_READ | PROT_WRITE,
		    MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
	assert(data != MAP_FAILED);

	for (i = 0; i < max_ios; i++) {
		iops[i] = &io[i];
		io[i].data = io;
		iovecs[i].iov_base = &data[io_size * i];
		iovecs[i].iov_len = io_size;
		io_prep_preadv(&io[i], fd, &iovecs[i], 1, 0);
	}

	submitted = completed = pending = 0;

	to_submit = max_ios;
	iocb_sub = iops;

	while (submitted < max_ios) {
		printf("Submitting: %ld\n", to_submit);

		ret = io_submit(io_ctx, to_submit, iocb_sub);
		if (ret >= 0) {
			printf("Submitted: %d\n", ret);
			submitted += ret;
			iocb_sub += ret;
			pending += ret;
			to_submit -= ret;
		} else {
			if (ret == -EAGAIN) {
				printf("Submitted too much, that's okay\n");
				prune(io_ctx, max_ios, getevents_type);
			}
		}
	}

	prune(io_ctx, max_ios, getevents_type);
	io_destroy(io_ctx);
	close(fd);
	ret = munmap(data, io_size * max_ios);
	assert(!ret);

	printf("Verifying...\n");

	assert(completed == submitted);

	printf("OK\n");
}

void run_child(void)

{
	int ring_size;

	ring_size = get_ring_size(max_events);

	run_test(ring_size-1, SYS_IO_GETEVENTS);
	run_test(ring_size, SYS_IO_GETEVENTS);
	run_test(ring_size+1, SYS_IO_GETEVENTS);
	run_test(ring_size*2, SYS_IO_GETEVENTS);
	run_test(ring_size*4, SYS_IO_GETEVENTS);

	run_test(ring_size-1, USER_GETEVENTS);
	run_test(ring_size, USER_GETEVENTS);
	run_test(ring_size+1, USER_GETEVENTS);
	run_test(ring_size*2, USER_GETEVENTS);
	run_test(ring_size*4, USER_GETEVENTS);

	exit(0);
}

void sighandler(int signo)

{
	assert(signo == SIGCHLD);
	done = 1;
}

int test_main(void)

{
	unsigned int ret;
	sighandler_t oldhandler;
	pid_t child;

	switch (child = fork()) {
	case 0: /* child */
		run_child();
		break;
	case -1:
		perror("fork");
		exit(1);
	default:
		oldhandler = signal(SIGCHLD, sighandler);
		assert(oldhandler != SIG_ERR);
		break;
	}

	ret = sleep(10);
	if (ret != 0) {
		pid_t pid;
		int status;

		assert(done);

		pid = wait(&status);
		if (pid != child) {
			perror("wait");
			exit(1);
		}

		return WEXITSTATUS(status);
	}

	return 1; /* failed */
}

/*
* Local Variables:
* mode: c
* c-basic-offset: 8
* End:
*/

...

...

[132204.801834] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0
[132204.804503] CPU: 0 PID: 8998 Comm: fio Tainted: G            E  3.17.0-rc1+ #15
[132204.806998] Hardware name: HP ProLiant DL380p Gen8, BIOS P70 09/08/2013
[132204.809292]  0000000000000000 ffff88043f407bb8 ffffffff815af54f 0000000000000001
[132204.812076]  ffffffff81809fe8 ffff88043f407c38 ffffffff815af2ce 0000000000000010
[132204.814858]  ffff88043f407c48 ffff88043f407be8 0000000000000000 0000000000000000
[132204.817557] Call Trace: 
[132204.818442]  <NMI>  [<ffffffff815af54f>] dump_stack+0x49/0x62
[132204.820538]  [<ffffffff815af2ce>] panic+0xbb/0x1f8
[132204.822284]  [<ffffffff810fa651>] watchdog_overflow_callback+0xb1/0xc0
[132204.824512]  [<ffffffff81133cf8>] __perf_event_overflow+0x98/0x230
[132204.826603]  [<ffffffff811345f4>] perf_event_overflow+0x14/0x20
[132204.828618]  [<ffffffff810219dc>] intel_pmu_handle_irq+0x1ec/0x3c0
[132204.830819]  [<ffffffff81018e04>] perf_event_nmi_handler+0x34/0x60
[132204.832933]  [<ffffffff81007d47>] nmi_handle+0x87/0x120
[132204.834748]  [<ffffffff81007ff4>] default_do_nmi+0x54/0x110
[132204.836670]  [<ffffffff81008140>] do_nmi+0x90/0xe0
[132204.838347]  [<ffffffff815b586a>] end_repeat_nmi+0x1e/0x2e
[132204.840248]  [<ffffffff811e4be4>] ? io_submit_one+0x174/0x4b0
[132204.842293]  [<ffffffff811e4be4>] ? io_submit_one+0x174/0x4b0
[132204.844257]  [<ffffffff811e4be4>] ? io_submit_one+0x174/0x4b0
[132204.846161]  <<EOE>>  [<ffffffff811e505c>] do_io_submit+0x13c/0x200
[132204.848438]  [<ffffffff8108cb53>] ? pick_next_task_fair+0x163/0x220
[132204.850642]  [<ffffffff811e5130>] SyS_io_submit+0x10/0x20
[132204.852519]  [<ffffffff815b3b52>] system_call_fastpath+0x16/0x1b
[132204.854608] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff)
[132204.857976] ---[ end Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0

{
        asm volatile(LOCK_PREFIX "addl %1,%0"
    1f8c:       41 8b 44 24 78          mov    0x78(%r12),%eax
    1f91:       f0 01 03                lock add %eax,(%rbx)

        local_irq_save(flags);
        kcpu = this_cpu_ptr(ctx->cpu);
        kcpu->reqs_available += nr;

        while (kcpu->reqs_available >= ctx->req_batch * 2) {
    1f94:       8b 01                   mov    (%rcx),%eax
    1f96:       41 8b 54 24 78          mov    0x78(%r12),%edx
    1f9b:       8d 34 12                lea    (%rdx,%rdx,1),%esi
    1f9e:       39 f0                   cmp    %esi,%eax
    1fa0:       73 e6                   jae    1f88 <io_submit_one+0x168>

...

This crash is not particularly useful, as that version of the patch was buggy and could result in reqs_available getting corrupted (ie, it could overflow), which could lead to the system getting stuck for a significant amount of time in put_reqs_available(). In other words, the crash you reported matches the known problem with that patch.

-ben

"Thought is the essence of where you are now."

A kernel memory disclosure was introduced in aio_read_events_ring() in v3.10 by commit a31ad380bed817aa25f8830ad23e1a0480fef797. The changes made to aio_read_events_ring() failed to correctly limit the index into ctx->ring_pages[], allowing an attacked to cause the subsequent kmap() of an arbitrary page with a copy_to_user() to copy the contents into userspace. This vulnerability has been assigned CVE-2014-0206. Thanks to Mateusz and Petr for disclosing this issue.

This patch applies to v3.12+. A separate backport is needed for 3.10/3.11.

---
 fs/aio.c | 3 +++
 1 file changed, 3 insertions(+)

index 6a9c7e4..955947e 100644
@@ -1063,6 +1063,9 @@ static long aio_read_events_ring(struct kioctx *ctx,
 	if (head == tail)
 		goto out;
 
+	head %= ctx->nr_events;
+	tail %= ctx->nr_events;
+
 	while (ret < nr) {
 		long avail;
 		struct io_event *ev;

1.8.2.1


-- 
"Thought is the essence of where you are now."

Note that a 3.10 backport will need to remove this line from free_ioctx as well:

                atomic_sub(avail, &ctx->reqs_active);

Can you post the backport for 3.10 so the -stable folks have something they can use that was tested? Cheers,

-ben

"Thought is the essence of where you are now."

Sure thing.

Cheers,
Jeff