BUG kmalloc-256: Poison overwritten.
hi,
i'm currently testing mainline kernel to isolate some ugly fault which corrupts memory (see https://bugzilla.kernel.org/show_bug.cgi?id=25782).
during soft raid reogranization (stopping e92.0,e93.0 AoE exports -> dropping md2, md3 -> recreating one bigger md2 -> reexporting md2 as e92.0) i've observed some debug messages (slub_debug=FZPU) in dmesg.
(...)
[ 77.175405] md3: detected capacity change from 0 to 434951421952 [ 77.188128] md3: unknown partition table [ 77.235477] md2: detected capacity change from 0 to 434951421952 [ 77.596970] md2: unknown partition table [ 77.673776] aoe: 002590289362 e92.0 v0100 has 849514496 sectors [ 77.673800] aoe: 002590289363 e93.0 v0100 has 849514496 sectors [ 77.677110] etherd/e93.0: unknown partition table [ 77.677773] etherd/e92.0: unknown partition table [ 1675.621938] md: md3 still in use. [ 1810.215053] md: md2 still in use. [ 1848.694133] md: md2: resync done. [ 1849.116128] md: checkpointing resync of md2. [ 1849.211809] md2: detected capacity change from 434951421952 to 0 [ 1849.211816] md: md2 stopped. [ 1849.211833] md: unbind<sda4> [ 1849.236522] md: export_rdev(sda4) [ 1849.236617] md: unbind<sdb4> [ 1849.266532] md: export_rdev(sdb4) [ 1851.148856] md3: detected capacity change from 434951421952 to 0 [ 1851.148865] md: md3 stopped. [ 1851.148873] md: unbind<sdc4> [ 1851.173201] md: export_rdev(sdc4) [ 1851.173306] md: unbind<sdd4> [ 1851.199873] md: export_rdev(sdd4) [ 1863.448308] ============================================================================= [ 1863.448313] BUG kmalloc-256: Poison overwritten [ 1863.448315] ----------------------------------------------------------------------------- [ 1863.448316] [ 1863.448319] INFO: 0xffff8807ffc7e7c4-0xffff8807ffc7e7c5. First byte 0x6c instead of 0x6b [ 1863.448331] INFO: Allocated in setup_conf+0x12b/0x360 [raid10] age=554800 cpu=5 pid=2766 [ 1863.448336] INFO: Freed in stop+0x66/0x80 [raid10] age=4271 cpu=3 pid=5266 [ 1863.448339] INFO: Slab 0xffffea001bff3b90 objects=24 used=11 fp=0xffff8807ffc7e7b0 flags=0x6000000000040c1 [ 1863.448341] INFO: Object 0xffff8807ffc7e7b0 @offset=1968 fp=0xffff8807ffc7f338 [ 1863.448343] [ 1863.448345] Bytes b4 0xffff8807ffc7e7a0: a9 c6 fe ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ����....ZZZZZZZZ [ 1863.448353] Object 0xffff8807ffc7e7b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448362] Object 0xffff8807ffc7e7c0: 6b 6b 6b 6b 6c 6c 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkllkkkkkkkkkk [ 1863.448369] Object 0xffff8807ffc7e7d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448377] Object 0xffff8807ffc7e7e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448384] Object 0xffff8807ffc7e7f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448391] Object 0xffff8807ffc7e800: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448399] Object 0xffff8807ffc7e810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448406] Object 0xffff8807ffc7e820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448413] Object 0xffff8807ffc7e830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448421] Object 0xffff8807ffc7e840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448428] Object 0xffff8807ffc7e850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448435] Object 0xffff8807ffc7e860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448442] Object 0xffff8807ffc7e870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448450] Object 0xffff8807ffc7e880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448457] Object 0xffff8807ffc7e890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448464] Object 0xffff8807ffc7e8a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk� [ 1863.448472] Redzone 0xffff8807ffc7e8b0: bb bb bb bb bb bb bb bb �������� [ 1863.448478] Padding 0xffff8807ffc7e8f0: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ [ 1863.448487] Pid: 5282, comm: udevd Not tainted 2.6.37-rc8 #1 [ 1863.448489] Call Trace: [ 1863.448499] [<ffffffff8111ea1e>] print_trailer+0xfe/0x160 [ 1863.448503] [<ffffffff8111f074>] check_bytes_and_report+0xf4/0x130 [ 1863.448506] [<ffffffff8111f2da>] check_object+0x22a/0x270 [ 1863.448512] [<ffffffff81137cc9>] ? do_execve+0x59/0x390 [ 1863.448515] [<ffffffff81137cc9>] ? do_execve+0x59/0x390 [ 1863.448519] [<ffffffff81120380>] alloc_debug_processing+0x110/0x1f0 [ 1863.448522] [<ffffffff811211c9>] __slab_alloc+0x3a9/0x410 [ 1863.448528] [<ffffffff8140254c>] ? do_page_fault+0x1cc/0x4b0 [ 1863.448531] [<ffffffff81137cc9>] ? do_execve+0x59/0x390 [ 1863.448534] [<ffffffff81121888>] kmem_cache_alloc_notrace+0xb8/0xc0 [ 1863.448538] [<ffffffff81137cc9>] do_execve+0x59/0x390 [ 1863.448543] [<ffffffff8121f0c1>] ? strncpy_from_user+0x31/0x50 [ 1863.448548] [<ffffffff8100b205>] sys_execve+0x45/0x70 [ 1863.448553] [<ffffffff8100319c>] stub_execve+0x6c/0xc0 [ 1863.448556] FIX kmalloc-256: Restoring 0xffff8807ffc7e7c4-0xffff8807ffc7e7c5=0x6b [ 1863.448557] [ 1863.448559] FIX kmalloc-256: Marking all objects used [ 1884.013716] md: bind<sda4> [ 1884.013908] md: bind<sdb4> [ 1884.015670] md: bind<sdc4> [ 1884.042060] md: bind<sdd4> [ 1884.044120] md/raid0:md2: looking at sdd4 [ 1884.044123] md/raid0:md2: comparing sdd4(849514496) with sdd4(849514496) [ 1884.044126] md/raid0:md2: END [ 1884.044128] md/raid0:md2: ==> UNIQUE [ 1884.044129] md/raid0:md2: 1 zones [ 1884.044130] md/raid0:md2: looking at sdc4 [ 1884.044132] md/raid0:md2: comparing sdc4(849514496) with sdd4(849514496) [ 1884.044135] md/raid0:md2: EQUAL [ 1884.044136] md/raid0:md2: looking at sdb4 [ 1884.044138] md/raid0:md2: comparing sdb4(849514496) with sdd4(849514496) [ 1884.044140] md/raid0:md2: EQUAL [ 1884.044142] md/raid0:md2: looking at sda4 [ 1884.044143] md/raid0:md2: comparing sda4(849514496) with sdd4(849514496) [ 1884.044146] md/raid0:md2: EQUAL [ 1884.044147] md/raid0:md2: FINAL 1 zones [ 1884.044151] md/raid0:md2: done. [ 1884.044153] md/raid0:md2: md_size is 3398057984 sectors. [ 1884.044154] ******* md2 configuration ********* [ 1884.044156] zone0=[sda4/sdb4/sdc4/sdd4/] [ 1884.044160] zone offset=0kb device offset=0kb size=1699028992kb [ 1884.044162] ********************************** [ 1884.044163] [ 1884.044181] md2: detected capacity change from 0 to 1739805687808 [ 1884.044338] md2: detected capacity change from 0 to 1739805687808 [ 1884.044796] md2: unknown partition table [ 2402.479826] aoe: device 92.0 is not up [ 2402.479841] aoe: device 92.0 is not up [ 2402.479852] aoe: device 92.0 is not up [ 2434.879826] aoe: device 93.0 is not up [ 2434.879838] aoe: device 93.0 is not up [ 2434.879849] aoe: device 93.0 is not up [ 2625.045477] FS-Cache: Loaded [ 2625.185242] Registering the id_resolver key type [ 2625.185275] FS-Cache: Netfs 'nfs' registered for caching [ 2662.391950] OCFS2 Node Manager 1.5.0 [ 2662.421336] OCFS2 DLM 1.5.0 [ 2662.445297] OCFS2 DLMFS 1.5.0 [ 2662.445505] OCFS2 User DLM kernel interface loaded [ 2673.692803] OCFS2 1.5.0 [ 2673.727998] ocfs2: Registered cluster interface o2cb [ 2673.728595] o2dlm: Nodes in domain 1901FEFE8A824A46A44C7A6F97655306: 2 [ 2673.757023] JBD: Ignoring recovery information on journal [ 2673.847489] ocfs2: Mounting device (9,2) on (node 2, slot 0) with ordered data mode. [ 2702.479609] ============================================================================= [ 2702.479613] BUG kmalloc-256: Poison overwritten [ 2702.479615] ----------------------------------------------------------------------------- [ 2702.479617] [ 2702.479619] INFO: 0xffff8807fecd690c-0xffff8807fecd690d. First byte 0x6c instead of 0x6b [ 2702.479634] INFO: Allocated in setup_conf+0x12b/0x360 [raid10] age=806494 cpu=4 pid=2766 [ 2702.479638] INFO: Freed in stop+0x66/0x80 [raid10] age=255399 cpu=9 pid=5270 [ 2702.479641] INFO: Slab 0xffffea001bfbced0 objects=24 used=16 fp=0xffff8807fecd68f8 flags=0x6000000000040c1 [ 2702.479644] INFO: Object 0xffff8807fecd68f8 @offset=2296 fp=0xffff8807fecd7480 [ 2702.479645] [ 2702.479647] Bytes b4 0xffff8807fecd68e8: 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ [ 2702.479656] Object 0xffff8807fecd68f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 2702.479664] Object 0xffff8807fecd6908: 6b 6b 6b 6b 6c 6c 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkllkkkkkkkkkk [ 2702.479671] Object 0xffff8807fecd6918: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 2702.479679] Object 0xffff8807fecd6928: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 2702.479686] Object 0xffff8807fecd6938: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 2702.479693] Object 0xffff8807fecd6948: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 2702.479700] Object 0xffff8807fecd6958: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 2702.479708] Object 0xffff8807fecd6968: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 2702.479715] Object 0xffff8807fecd6978: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 2702.479722] Object 0xffff8807fecd6988: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 2702.479729] Object 0xffff8807fecd6998: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 2702.479737] Object 0xffff8807fecd69a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 2702.479744] Object 0xffff8807fecd69b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 2702.479751] Object 0xffff8807fecd69c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 2702.479758] Object 0xffff8807fecd69d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 2702.479766] Object 0xffff8807fecd69e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk� [ 2702.479773] Redzone 0xffff8807fecd69f8: bb bb bb bb bb bb bb bb �������� [ 2702.479780] Padding 0xffff8807fecd6a38: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ [ 2702.479789] Pid: 5970, comm: sshd Not tainted 2.6.37-rc8 #1 [ 2702.479791] Call Trace: [ 2702.479801] [<ffffffff8111ea1e>] print_trailer+0xfe/0x160 [ 2702.479805] [<ffffffff8111f074>] check_bytes_and_report+0xf4/0x130 [ 2702.479808] [<ffffffff8111f2da>] check_object+0x22a/0x270 [ 2702.479815] [<ffffffffa02055a4>] ? autofs4_wait+0x54/0x800 [autofs4] [ 2702.479820] [<ffffffffa02055a4>] ? autofs4_wait+0x54/0x800 [autofs4] [ 2702.479824] [<ffffffff81120380>] alloc_debug_processing+0x110/0x1f0 [ 2702.479827] [<ffffffff811211c9>] __slab_alloc+0x3a9/0x410 [ 2702.479832] [<ffffffffa02055a4>] ? autofs4_wait+0x54/0x800 [autofs4] [ 2702.479836] [<ffffffff81121888>] kmem_cache_alloc_notrace+0xb8/0xc0 [ 2702.479841] [<ffffffffa02055a4>] autofs4_wait+0x54/0x800 [autofs4] [ 2702.479844] [<ffffffff811211c9>] ? __slab_alloc+0x3a9/0x410 [ 2702.479847] [<ffffffff8111f15f>] ? check_object+0xaf/0x270 [ 2702.479851] [<ffffffffa0203cc8>] try_to_fill_dentry+0xd8/0x100 [autofs4] [ 2702.479856] [<ffffffffa0203f71>] autofs4_revalidate+0xf1/0x170 [autofs4] [ 2702.479861] [<ffffffffa020487a>] autofs4_lookup+0x4fa/0x590 [autofs4] [ 2702.479865] [<ffffffff8113b3c7>] d_alloc_and_lookup+0x47/0x90 [ 2702.479870] [<ffffffff811464a0>] ? d_lookup+0x30/0x60 [ 2702.479873] [<ffffffff8113b528>] do_lookup+0x118/0x170 [ 2702.479876] [<ffffffff8113d379>] link_path_walk+0x309/0xac0 [ 2702.479879] [<ffffffff8113d5bb>] link_path_walk+0x54b/0xac0 [ 2702.479883] [<ffffffff8113ee8c>] do_filp_open+0x26c/0x680 [ 2702.479886] [<ffffffff8111f706>] ? init_object+0x46/0x80 [ 2702.479889] [<ffffffff8113cc66>] ? getname+0x36/0x220 [ 2702.479893] [<ffffffff8114a207>] ? alloc_fd+0x47/0x140 [ 2702.479897] [<ffffffff81130534>] do_sys_open+0x64/0x110 [ 2702.479902] [<ffffffff8105e408>] ? sys_setresuid+0x148/0x150 [ 2702.479905] [<ffffffff811305fb>] sys_open+0x1b/0x20 [ 2702.479910] [<ffffffff81002d7b>] system_call_fastpath+0x16/0x1b [ 2702.479914] FIX kmalloc-256: Restoring 0xffff8807fecd690c-0xffff8807fecd690d=0x6b [ 2702.479915] [ 2702.479917] FIX kmalloc-256: Marking all objects used [ 4819.470336] ocfs2: Unmounting device (9,2) on (node 2) [ 6930.496519] aoe: 002590289362 e92.0 v0100 has 3398057984 sectors [ 6930.498397] etherd/e92.0: unknown partition table [ 6930.500655] aoe: ata error cmd=24h stat=41h from e92.0 [ 6930.500664] aoe: ata error cmd=24h stat=41h from e92.0 [ 6930.500671] aoe: ata error cmd=24h stat=41h from e92.0 [ 6930.500677] aoe: ata error cmd=24h stat=41h from e92.0 [ 6930.500682] Buffer I/O error on device etherd/e92.0, logical block 424757232 [ 6930.500864] aoe: ata error cmd=24h stat=41h from e92.0 [ 6930.500872] aoe: ata error cmd=24h stat=41h from e92.0 [ 6930.500878] aoe: ata error cmd=24h stat=41h from e92.0 [ 6930.500883] aoe: ata error cmd=24h stat=41h from e92.0 [ 6930.500887] Buffer I/O error on device etherd/e92.0, logical block 424757232 [ 6930.501082] aoe: ata error cmd=24h stat=41h from e92.0 [ 6930.501091] aoe: ata error cmd=24h stat=41h from e92.0
could you please put some light on this?
BR,
Pawel.
ps).
please CC me on reply.
Quoted 40 lines · Pawel Sikora
[ 1863.448308] ============================================================================= [ 1863.448313] BUG kmalloc-256: Poison overwritten [ 1863.448315] ----------------------------------------------------------------------------- [ 1863.448316] [ 1863.448319] INFO: 0xffff8807ffc7e7c4-0xffff8807ffc7e7c5. First byte 0x6c instead of 0x6b [ 1863.448331] INFO: Allocated in setup_conf+0x12b/0x360 [raid10] age=554800 cpu=5 pid=2766 [ 1863.448336] INFO: Freed in stop+0x66/0x80 [raid10] age=4271 cpu=3 pid=5266 [ 1863.448339] INFO: Slab 0xffffea001bff3b90 objects=24 used=11 fp=0xffff8807ffc7e7b0 flags=0x6000000000040c1 [ 1863.448341] INFO: Object 0xffff8807ffc7e7b0 @offset=1968 fp=0xffff8807ffc7f338 [ 1863.448343] [ 1863.448345] Bytes b4 0xffff8807ffc7e7a0: a9 c6 fe ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ����....ZZZZZZZZ [ 1863.448353] Object 0xffff8807ffc7e7b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448362] Object 0xffff8807ffc7e7c0: 6b 6b 6b 6b 6c 6c 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkllkkkkkkkkkk [ 1863.448369] Object 0xffff8807ffc7e7d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448377] Object 0xffff8807ffc7e7e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448384] Object 0xffff8807ffc7e7f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448391] Object 0xffff8807ffc7e800: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448399] Object 0xffff8807ffc7e810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448406] Object 0xffff8807ffc7e820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448413] Object 0xffff8807ffc7e830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448421] Object 0xffff8807ffc7e840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448428] Object 0xffff8807ffc7e850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448435] Object 0xffff8807ffc7e860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448442] Object 0xffff8807ffc7e870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448450] Object 0xffff8807ffc7e880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448457] Object 0xffff8807ffc7e890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448464] Object 0xffff8807ffc7e8a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk� [ 1863.448472] Redzone 0xffff8807ffc7e8b0: bb bb bb bb bb bb bb bb �������� [ 1863.448478] Padding 0xffff8807ffc7e8f0: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ [ 1863.448487] Pid: 5282, comm: udevd Not tainted 2.6.37-rc8 #1 [ 1863.448489] Call Trace: [ 1863.448499] [<ffffffff8111ea1e>] print_trailer+0xfe/0x160 [ 1863.448503] [<ffffffff8111f074>] check_bytes_and_report+0xf4/0x130 [ 1863.448506] [<ffffffff8111f2da>] check_object+0x22a/0x270 [ 1863.448512] [<ffffffff81137cc9>] ? do_execve+0x59/0x390 [ 1863.448515] [<ffffffff81137cc9>] ? do_execve+0x59/0x390 [ 1863.448519] [<ffffffff81120380>] alloc_debug_processing+0x110/0x1f0 [ 1863.448522] [<ffffffff811211c9>] __slab_alloc+0x3a9/0x410 [ 1863.448528] [<ffffffff8140254c>] ? do_page_fault+0x1cc/0x4b0 [ 1863.448531] [<ffffffff81137cc9>] ? do_execve+0x59/0x390
Quoted 8 lines
[ 1863.448534] [<ffffffff81121888>] kmem_cache_alloc_notrace+0xb8/0xc0 [ 1863.448538] [<ffffffff81137cc9>] do_execve+0x59/0x390 [ 1863.448543] [<ffffffff8121f0c1>] ? strncpy_from_user+0x31/0x50 [ 1863.448548] [<ffffffff8100b205>] sys_execve+0x45/0x70 [ 1863.448553] [<ffffffff8100319c>] stub_execve+0x6c/0xc0 [ 1863.448556] FIX kmalloc-256: Restoring 0xffff8807ffc7e7c4-0xffff8807ffc7e7c5=0x6b [ 1863.448557] [ 1863.448559] FIX kmalloc-256: Marking all objects used
This looks like a use-after-free bug somewhere in drivers/md/raid10.c.
Pekka
Quoted 50 lines · Pekka Enberg
Quoted 40 lines · Pawel Sikora
[ 1863.448308] ============================================================================= [ 1863.448313] BUG kmalloc-256: Poison overwritten [ 1863.448315] ----------------------------------------------------------------------------- [ 1863.448316] [ 1863.448319] INFO: 0xffff8807ffc7e7c4-0xffff8807ffc7e7c5. First byte 0x6c instead of 0x6b [ 1863.448331] INFO: Allocated in setup_conf+0x12b/0x360 [raid10] age=554800 cpu=5 pid=2766 [ 1863.448336] INFO: Freed in stop+0x66/0x80 [raid10] age=4271 cpu=3 pid=5266 [ 1863.448339] INFO: Slab 0xffffea001bff3b90 objects=24 used=11 fp=0xffff8807ffc7e7b0 flags=0x6000000000040c1 [ 1863.448341] INFO: Object 0xffff8807ffc7e7b0 @offset=1968 fp=0xffff8807ffc7f338 [ 1863.448343] [ 1863.448345] Bytes b4 0xffff8807ffc7e7a0: a9 c6 fe ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ����....ZZZZZZZZ [ 1863.448353] Object 0xffff8807ffc7e7b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448362] Object 0xffff8807ffc7e7c0: 6b 6b 6b 6b 6c 6c 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkllkkkkkkkkkk [ 1863.448369] Object 0xffff8807ffc7e7d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448377] Object 0xffff8807ffc7e7e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448384] Object 0xffff8807ffc7e7f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448391] Object 0xffff8807ffc7e800: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448399] Object 0xffff8807ffc7e810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448406] Object 0xffff8807ffc7e820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448413] Object 0xffff8807ffc7e830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448421] Object 0xffff8807ffc7e840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448428] Object 0xffff8807ffc7e850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448435] Object 0xffff8807ffc7e860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448442] Object 0xffff8807ffc7e870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448450] Object 0xffff8807ffc7e880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448457] Object 0xffff8807ffc7e890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448464] Object 0xffff8807ffc7e8a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk� [ 1863.448472] Redzone 0xffff8807ffc7e8b0: bb bb bb bb bb bb bb bb �������� [ 1863.448478] Padding 0xffff8807ffc7e8f0: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ [ 1863.448487] Pid: 5282, comm: udevd Not tainted 2.6.37-rc8 #1 [ 1863.448489] Call Trace: [ 1863.448499] [<ffffffff8111ea1e>] print_trailer+0xfe/0x160 [ 1863.448503] [<ffffffff8111f074>] check_bytes_and_report+0xf4/0x130 [ 1863.448506] [<ffffffff8111f2da>] check_object+0x22a/0x270 [ 1863.448512] [<ffffffff81137cc9>] ? do_execve+0x59/0x390 [ 1863.448515] [<ffffffff81137cc9>] ? do_execve+0x59/0x390 [ 1863.448519] [<ffffffff81120380>] alloc_debug_processing+0x110/0x1f0 [ 1863.448522] [<ffffffff811211c9>] __slab_alloc+0x3a9/0x410 [ 1863.448528] [<ffffffff8140254c>] ? do_page_fault+0x1cc/0x4b0 [ 1863.448531] [<ffffffff81137cc9>] ? do_execve+0x59/0x390Quoted 8 lines
[ 1863.448534] [<ffffffff81121888>] kmem_cache_alloc_notrace+0xb8/0xc0 [ 1863.448538] [<ffffffff81137cc9>] do_execve+0x59/0x390 [ 1863.448543] [<ffffffff8121f0c1>] ? strncpy_from_user+0x31/0x50 [ 1863.448548] [<ffffffff8100b205>] sys_execve+0x45/0x70 [ 1863.448553] [<ffffffff8100319c>] stub_execve+0x6c/0xc0 [ 1863.448556] FIX kmalloc-256: Restoring 0xffff8807ffc7e7c4-0xffff8807ffc7e7c5=0x6b [ 1863.448557] [ 1863.448559] FIX kmalloc-256: Marking all objects used
Quoted 1 line
This looks like a use-after-free bug somewhere in
drivers/md/raid10.c.
Does reverting commit 4e78064f42ad474ce9c31760861f7fb0cfc22532 ("md:
Fix possible deadlock with multiple mempool allocations.") fix the problem?
Quoted 50 lines · On Thursday 30 of December 2010 16:31:38 Pekka Enberg wrote:
Quoted 40 lines · Pawel Sikora
[ 1863.448308] ============================================================================= [ 1863.448313] BUG kmalloc-256: Poison overwritten [ 1863.448315] ----------------------------------------------------------------------------- [ 1863.448316] [ 1863.448319] INFO: 0xffff8807ffc7e7c4-0xffff8807ffc7e7c5. First byte 0x6c instead of 0x6b [ 1863.448331] INFO: Allocated in setup_conf+0x12b/0x360 [raid10] age=554800 cpu=5 pid=2766 [ 1863.448336] INFO: Freed in stop+0x66/0x80 [raid10] age=4271 cpu=3 pid=5266 [ 1863.448339] INFO: Slab 0xffffea001bff3b90 objects=24 used=11 fp=0xffff8807ffc7e7b0 flags=0x6000000000040c1 [ 1863.448341] INFO: Object 0xffff8807ffc7e7b0 @offset=1968 fp=0xffff8807ffc7f338 [ 1863.448343] [ 1863.448345] Bytes b4 0xffff8807ffc7e7a0: a9 c6 fe ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ����....ZZZZZZZZ [ 1863.448353] Object 0xffff8807ffc7e7b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448362] Object 0xffff8807ffc7e7c0: 6b 6b 6b 6b 6c 6c 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkllkkkkkkkkkk [ 1863.448369] Object 0xffff8807ffc7e7d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448377] Object 0xffff8807ffc7e7e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448384] Object 0xffff8807ffc7e7f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448391] Object 0xffff8807ffc7e800: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448399] Object 0xffff8807ffc7e810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448406] Object 0xffff8807ffc7e820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448413] Object 0xffff8807ffc7e830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448421] Object 0xffff8807ffc7e840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448428] Object 0xffff8807ffc7e850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448435] Object 0xffff8807ffc7e860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448442] Object 0xffff8807ffc7e870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448450] Object 0xffff8807ffc7e880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448457] Object 0xffff8807ffc7e890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448464] Object 0xffff8807ffc7e8a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk� [ 1863.448472] Redzone 0xffff8807ffc7e8b0: bb bb bb bb bb bb bb bb �������� [ 1863.448478] Padding 0xffff8807ffc7e8f0: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ [ 1863.448487] Pid: 5282, comm: udevd Not tainted 2.6.37-rc8 #1 [ 1863.448489] Call Trace: [ 1863.448499] [<ffffffff8111ea1e>] print_trailer+0xfe/0x160 [ 1863.448503] [<ffffffff8111f074>] check_bytes_and_report+0xf4/0x130 [ 1863.448506] [<ffffffff8111f2da>] check_object+0x22a/0x270 [ 1863.448512] [<ffffffff81137cc9>] ? do_execve+0x59/0x390 [ 1863.448515] [<ffffffff81137cc9>] ? do_execve+0x59/0x390 [ 1863.448519] [<ffffffff81120380>] alloc_debug_processing+0x110/0x1f0 [ 1863.448522] [<ffffffff811211c9>] __slab_alloc+0x3a9/0x410 [ 1863.448528] [<ffffffff8140254c>] ? do_page_fault+0x1cc/0x4b0 [ 1863.448531] [<ffffffff81137cc9>] ? do_execve+0x59/0x390Quoted 8 lines
[ 1863.448534] [<ffffffff81121888>] kmem_cache_alloc_notrace+0xb8/0xc0 [ 1863.448538] [<ffffffff81137cc9>] do_execve+0x59/0x390 [ 1863.448543] [<ffffffff8121f0c1>] ? strncpy_from_user+0x31/0x50 [ 1863.448548] [<ffffffff8100b205>] sys_execve+0x45/0x70 [ 1863.448553] [<ffffffff8100319c>] stub_execve+0x6c/0xc0 [ 1863.448556] FIX kmalloc-256: Restoring 0xffff8807ffc7e7c4-0xffff8807ffc7e7c5=0x6b [ 1863.448557] [ 1863.448559] FIX kmalloc-256: Marking all objects used
Quoted 3 lines
This looks like a use-after-free bug somewhere in
drivers/md/raid10.c.Pekka
i think it's quite easy to reproduce this problem. here's a mini howto:
- setup two raid10 matrices.
[root@odra ~]# cat /proc/mdstat
Personalities : [raid1] [raid0] [raid10]
md3 : active raid10 sdd4[1] sdc4[0]
424757248 blocks super 1.2 512K chunks 2 far-copies [2/2] [UU]
[>....................] resync = 0.4% (1966592/424757248) finish=82.4min speed=85504K/secmd2 : active raid10 sdb4[1] sda4[0]
424757248 blocks super 1.2 512K chunks 2 far-copies [2/2] [UU]
[>....................] resync = 0.5% (2446080/424757248) finish=97.1min speed=72432K/sec- stop matrices.
[root@odra ~]# mdadm --stop /dev/md2
mdadm: stopped /dev/md2
[root@odra ~]# mdadm --stop /dev/md3
mdadm: stopped /dev/md3
- create raid0 on devices previously used by raid10.
[root@odra ~]# mdadm -C /dev/md2 -l 0 -n 4 /dev/sda4 /dev/sdb4 /dev/sdc4 /dev/sdd4
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md2 started.
[root@odra ~]# cat /proc/mdstat Personalities : [raid1] [raid0] [raid10] md2 : active raid0 sdd4[3] sdc4[2] sdb4[1] sda4[0]
1699028992 blocks super 1.2 512k chunks
- stop it.
[root@odra ~]# mdadm --stop /dev/md2
mdadm: stopped /dev/md2
- create one raid10 matrix once more.
[root@odra ~]# mdadm -C /dev/md2 -l 10 -n 2 --layout f2 /dev/sda4 /dev/sdb4
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md2 started.
- in this moment i can see a bug report.
Dec 30 20:08:46 odra kernel: [12501.627162] ============================================================================= Dec 30 20:08:46 odra kernel: [12501.627166] BUG kmalloc-256: Poison overwritten Dec 30 20:08:46 odra kernel: [12501.627168] ----------------------------------------------------------------------------- Dec 30 20:08:46 odra kernel: [12501.627169] Dec 30 20:08:46 odra kernel: [12501.627172] INFO: 0xffff8803feb5e15c-0xffff8803feb5e15d. First byte 0x6c instead of 0x6b Dec 30 20:08:46 odra kernel: [12501.627178] INFO: Allocated in setup_conf+0x12b/0x360 [raid10] age=58297 cpu=2 pid=12007 Dec 30 20:08:46 odra kernel: [12501.627182] INFO: Freed in stop+0x66/0x80 [raid10] age=47657 cpu=2 pid=12047 Dec 30 20:08:46 odra kernel: [12501.627185] INFO: Slab 0xffffea000dfb7c90 objects=24 used=2 fp=0xffff8803feb5e148 flags=0x2000000000040c1 Dec 30 20:08:46 odra kernel: [12501.627188] INFO: Object 0xffff8803feb5e148 @offset=328 fp=0xffff8803feb5e3d8 Dec 30 20:08:46 odra kernel: [12501.627189] Dec 30 20:08:46 odra kernel: [12501.627191] Bytes b4 0xffff8803feb5e138: df a8 fe ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ߨ��....ZZZZZZZZ Dec 30 20:08:46 odra kernel: [12501.627199] Object 0xffff8803feb5e148: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627208] Object 0xffff8803feb5e158: 6b 6b 6b 6b 6c 6c 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkllkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627215] Object 0xffff8803feb5e168: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627223] Object 0xffff8803feb5e178: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627230] Object 0xffff8803feb5e188: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627237] Object 0xffff8803feb5e198: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627245] Object 0xffff8803feb5e1a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627252] Object 0xffff8803feb5e1b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627259] Object 0xffff8803feb5e1c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627267] Object 0xffff8803feb5e1d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627274] Object 0xffff8803feb5e1e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627281] Object 0xffff8803feb5e1f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627289] Object 0xffff8803feb5e208: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627296] Object 0xffff8803feb5e218: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627303] Object 0xffff8803feb5e228: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627311] Object 0xffff8803feb5e238: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk� Dec 30 20:08:46 odra kernel: [12501.627318] Redzone 0xffff8803feb5e248: bb bb bb bb bb bb bb bb �������� Dec 30 20:08:46 odra kernel: [12501.627325] Padding 0xffff8803feb5e288: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ Dec 30 20:08:46 odra kernel: [12501.627334] Pid: 12168, comm: mdadm Not tainted 2.6.37-rc8 #1 Dec 30 20:08:46 odra kernel: [12501.627336] Call Trace: Dec 30 20:08:46 odra kernel: [12501.627343] [<ffffffff8111ea1e>] print_trailer+0xfe/0x160 Dec 30 20:08:46 odra kernel: [12501.627347] [<ffffffff8111f074>] check_bytes_and_report+0xf4/0x130 Dec 30 20:08:46 odra kernel: [12501.627350] [<ffffffff8111f2da>] check_object+0x22a/0x270 Dec 30 20:08:46 odra kernel: [12501.627354] [<ffffffffa03ff1eb>] ? setup_conf+0x12b/0x360 [raid10] Dec 30 20:08:46 odra kernel: [12501.627358] [<ffffffffa03ff1eb>] ? setup_conf+0x12b/0x360 [raid10] Dec 30 20:08:46 odra kernel: [12501.627361] [<ffffffff81120380>] alloc_debug_processing+0x110/0x1f0 Dec 30 20:08:46 odra kernel: [12501.627365] [<ffffffff811211c9>] __slab_alloc+0x3a9/0x410 Dec 30 20:08:46 odra kernel: [12501.627369] [<ffffffff810de600>] ? mempool_alloc_slab+0x10/0x20 Dec 30 20:08:46 odra kernel: [12501.627372] [<ffffffff8112166f>] ? kmem_cache_alloc_node_notrace+0xbf/0xe0 Dec 30 20:08:46 odra kernel: [12501.627376] [<ffffffff810de7fe>] ? mempool_create_node+0x7e/0x1a0 Dec 30 20:08:46 odra kernel: [12501.627379] [<ffffffffa03ff1eb>] ? setup_conf+0x12b/0x360 [raid10] Dec 30 20:08:46 odra kernel: [12501.627382] [<ffffffff81121888>] kmem_cache_alloc_notrace+0xb8/0xc0 Dec 30 20:08:46 odra kernel: [12501.627386] [<ffffffffa03ff1eb>] setup_conf+0x12b/0x360 [raid10] Dec 30 20:08:46 odra kernel: [12501.627390] [<ffffffffa04026b1>] run+0x21/0x3c0 [raid10] Dec 30 20:08:46 odra kernel: [12501.627413] [<ffffffffa00ca322>] md_run+0x322/0x920 [md_mod] Dec 30 20:08:46 odra kernel: [12501.627417] [<ffffffff813fd7a0>] ? __mutex_lock_interruptible_slowpath+0x1e0/0x2b0 Dec 30 20:08:46 odra kernel: [12501.627425] [<ffffffffa00ca939>] do_md_run+0x19/0xa0 [md_mod] Dec 30 20:08:46 odra kernel: [12501.627432] [<ffffffffa00cbefc>] md_ioctl+0xa1c/0x1350 [md_mod] Dec 30 20:08:46 odra kernel: [12501.627435] [<ffffffff8111f15f>] ? check_object+0xaf/0x270 Dec 30 20:08:46 odra kernel: [12501.627438] [<ffffffff8111f706>] ? init_object+0x46/0x80 Dec 30 20:08:46 odra kernel: [12501.627442] [<ffffffff812039e0>] blkdev_ioctl+0x230/0x720 Dec 30 20:08:46 odra kernel: [12501.627445] [<ffffffff81120846>] ? __slab_free+0x136/0x150 Dec 30 20:08:46 odra kernel: [12501.627449] [<ffffffff811607dc>] block_ioctl+0x3c/0x40 Dec 30 20:08:46 odra kernel: [12501.627453] [<ffffffff811412f8>] do_vfs_ioctl+0x98/0x580 Dec 30 20:08:46 odra kernel: [12501.627456] [<ffffffff81101af9>] ? remove_vma+0x69/0x90 Dec 30 20:08:46 odra kernel: [12501.627460] [<ffffffff81103244>] ? do_munmap+0x2e4/0x360 Dec 30 20:08:46 odra kernel: [12501.627463] [<ffffffff81141861>] sys_ioctl+0x81/0xa0 Dec 30 20:08:46 odra kernel: [12501.627467] [<ffffffff81002d7b>] system_call_fastpath+0x16/0x1b
Please report exactly which kernel you are running (git hash of head) and in particular whether
commit 589a594be1fb8815b3f18e517be696c48664f728
is present?
It looks like something tried to lock conf->device_lock after conf had been freed. It is possible that that could happen due to the bug fixed by the above commit.
Thanks,
NeilBrown
Quoted 160 lines · On Thu, 30 Dec 2010 20:39:33 +0100 Paweł Sikora <[email protected]> wrote:
Quoted 50 lines · On Thursday 30 of December 2010 16:31:38 Pekka Enberg wrote:
Quoted 40 lines · Pawel Sikora
[ 1863.448308] ============================================================================= [ 1863.448313] BUG kmalloc-256: Poison overwritten [ 1863.448315] ----------------------------------------------------------------------------- [ 1863.448316] [ 1863.448319] INFO: 0xffff8807ffc7e7c4-0xffff8807ffc7e7c5. First byte 0x6c instead of 0x6b [ 1863.448331] INFO: Allocated in setup_conf+0x12b/0x360 [raid10] age=554800 cpu=5 pid=2766 [ 1863.448336] INFO: Freed in stop+0x66/0x80 [raid10] age=4271 cpu=3 pid=5266 [ 1863.448339] INFO: Slab 0xffffea001bff3b90 objects=24 used=11 fp=0xffff8807ffc7e7b0 flags=0x6000000000040c1 [ 1863.448341] INFO: Object 0xffff8807ffc7e7b0 @offset=1968 fp=0xffff8807ffc7f338 [ 1863.448343] [ 1863.448345] Bytes b4 0xffff8807ffc7e7a0: a9 c6 fe ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ����....ZZZZZZZZ [ 1863.448353] Object 0xffff8807ffc7e7b0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448362] Object 0xffff8807ffc7e7c0: 6b 6b 6b 6b 6c 6c 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkllkkkkkkkkkk [ 1863.448369] Object 0xffff8807ffc7e7d0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448377] Object 0xffff8807ffc7e7e0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448384] Object 0xffff8807ffc7e7f0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448391] Object 0xffff8807ffc7e800: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448399] Object 0xffff8807ffc7e810: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448406] Object 0xffff8807ffc7e820: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448413] Object 0xffff8807ffc7e830: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448421] Object 0xffff8807ffc7e840: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448428] Object 0xffff8807ffc7e850: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448435] Object 0xffff8807ffc7e860: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448442] Object 0xffff8807ffc7e870: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448450] Object 0xffff8807ffc7e880: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448457] Object 0xffff8807ffc7e890: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 1863.448464] Object 0xffff8807ffc7e8a0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk� [ 1863.448472] Redzone 0xffff8807ffc7e8b0: bb bb bb bb bb bb bb bb �������� [ 1863.448478] Padding 0xffff8807ffc7e8f0: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ [ 1863.448487] Pid: 5282, comm: udevd Not tainted 2.6.37-rc8 #1 [ 1863.448489] Call Trace: [ 1863.448499] [<ffffffff8111ea1e>] print_trailer+0xfe/0x160 [ 1863.448503] [<ffffffff8111f074>] check_bytes_and_report+0xf4/0x130 [ 1863.448506] [<ffffffff8111f2da>] check_object+0x22a/0x270 [ 1863.448512] [<ffffffff81137cc9>] ? do_execve+0x59/0x390 [ 1863.448515] [<ffffffff81137cc9>] ? do_execve+0x59/0x390 [ 1863.448519] [<ffffffff81120380>] alloc_debug_processing+0x110/0x1f0 [ 1863.448522] [<ffffffff811211c9>] __slab_alloc+0x3a9/0x410 [ 1863.448528] [<ffffffff8140254c>] ? do_page_fault+0x1cc/0x4b0 [ 1863.448531] [<ffffffff81137cc9>] ? do_execve+0x59/0x390Quoted 8 lines
[ 1863.448534] [<ffffffff81121888>] kmem_cache_alloc_notrace+0xb8/0xc0 [ 1863.448538] [<ffffffff81137cc9>] do_execve+0x59/0x390 [ 1863.448543] [<ffffffff8121f0c1>] ? strncpy_from_user+0x31/0x50 [ 1863.448548] [<ffffffff8100b205>] sys_execve+0x45/0x70 [ 1863.448553] [<ffffffff8100319c>] stub_execve+0x6c/0xc0 [ 1863.448556] FIX kmalloc-256: Restoring 0xffff8807ffc7e7c4-0xffff8807ffc7e7c5=0x6b [ 1863.448557] [ 1863.448559] FIX kmalloc-256: Marking all objects usedQuoted 3 lines
This looks like a use-after-free bug somewhere in
drivers/md/raid10.c.Pekka
i think it's quite easy to reproduce this problem. here's a mini howto:
- setup two raid10 matrices.
[root@odra ~]# cat
/proc/mdstat
Personalities : [raid1] [raid0] [raid10]
md3 : active raid10 sdd4[1] sdc4[0]424757248 blocks super 1.2 512K chunks 2 far-copies [2/2] [UU] [>....................] resync = 0.4% (1966592/424757248) finish=82.4min speed=85504K/secmd2 : active raid10 sdb4[1] sda4[0]
424757248 blocks super 1.2 512K chunks 2 far-copies [2/2] [UU] [>....................] resync = 0.5% (2446080/424757248) finish=97.1min speed=72432K/sec- stop matrices.
[root@odra ~]# mdadm --stop
/dev/md2
mdadm: stopped/dev/md2
[root@odra ~]# mdadm --stop/dev/md3
mdadm: stopped/dev/md3- create raid0 on devices previously used by raid10.
[root@odra ~]# mdadm -C
/dev/md2-l 0 -n 4/dev/sda4/dev/sdb4/dev/sdc4/dev/sdd4mdadm: Defaulting to version 1.2 metadata
mdadm: array/dev/md2started.[root@odra ~]# cat
/proc/mdstatPersonalities : [raid1] [raid0] [raid10] md2 : active raid0 sdd4[3] sdc4[2] sdb4[1] sda4[0]1699028992 blocks super 1.2 512k chunks- stop it.
[root@odra ~]# mdadm --stop
/dev/md2
mdadm: stopped/dev/md2- create one raid10 matrix once more.
[root@odra ~]# mdadm -C
/dev/md2-l 10 -n 2 --layout f2/dev/sda4/dev/sdb4
mdadm: Defaulting to version 1.2 metadata
mdadm: array/dev/md2started.- in this moment i can see a bug report.
Dec 30 20:08:46 odra kernel: [12501.627162] ============================================================================= Dec 30 20:08:46 odra kernel: [12501.627166] BUG kmalloc-256: Poison overwritten Dec 30 20:08:46 odra kernel: [12501.627168] ----------------------------------------------------------------------------- Dec 30 20:08:46 odra kernel: [12501.627169] Dec 30 20:08:46 odra kernel: [12501.627172] INFO: 0xffff8803feb5e15c-0xffff8803feb5e15d. First byte 0x6c instead of 0x6b Dec 30 20:08:46 odra kernel: [12501.627178] INFO: Allocated in setup_conf+0x12b/0x360 [raid10] age=58297 cpu=2 pid=12007 Dec 30 20:08:46 odra kernel: [12501.627182] INFO: Freed in stop+0x66/0x80 [raid10] age=47657 cpu=2 pid=12047 Dec 30 20:08:46 odra kernel: [12501.627185] INFO: Slab 0xffffea000dfb7c90 objects=24 used=2 fp=0xffff8803feb5e148 flags=0x2000000000040c1 Dec 30 20:08:46 odra kernel: [12501.627188] INFO: Object 0xffff8803feb5e148 @offset=328 fp=0xffff8803feb5e3d8 Dec 30 20:08:46 odra kernel: [12501.627189] Dec 30 20:08:46 odra kernel: [12501.627191] Bytes b4 0xffff8803feb5e138: df a8 fe ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ߨ��....ZZZZZZZZ Dec 30 20:08:46 odra kernel: [12501.627199] Object 0xffff8803feb5e148: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627208] Object 0xffff8803feb5e158: 6b 6b 6b 6b 6c 6c 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkllkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627215] Object 0xffff8803feb5e168: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627223] Object 0xffff8803feb5e178: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627230] Object 0xffff8803feb5e188: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627237] Object 0xffff8803feb5e198: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627245] Object 0xffff8803feb5e1a8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627252] Object 0xffff8803feb5e1b8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627259] Object 0xffff8803feb5e1c8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627267] Object 0xffff8803feb5e1d8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627274] Object 0xffff8803feb5e1e8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627281] Object 0xffff8803feb5e1f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627289] Object 0xffff8803feb5e208: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627296] Object 0xffff8803feb5e218: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627303] Object 0xffff8803feb5e228: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Dec 30 20:08:46 odra kernel: [12501.627311] Object 0xffff8803feb5e238: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk� Dec 30 20:08:46 odra kernel: [12501.627318] Redzone 0xffff8803feb5e248: bb bb bb bb bb bb bb bb �������� Dec 30 20:08:46 odra kernel: [12501.627325] Padding 0xffff8803feb5e288: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ Dec 30 20:08:46 odra kernel: [12501.627334] Pid: 12168, comm: mdadm Not tainted 2.6.37-rc8 #1 Dec 30 20:08:46 odra kernel: [12501.627336] Call Trace: Dec 30 20:08:46 odra kernel: [12501.627343] [<ffffffff8111ea1e>] print_trailer+0xfe/0x160 Dec 30 20:08:46 odra kernel: [12501.627347] [<ffffffff8111f074>] check_bytes_and_report+0xf4/0x130 Dec 30 20:08:46 odra kernel: [12501.627350] [<ffffffff8111f2da>] check_object+0x22a/0x270 Dec 30 20:08:46 odra kernel: [12501.627354] [<ffffffffa03ff1eb>] ? setup_conf+0x12b/0x360 [raid10] Dec 30 20:08:46 odra kernel: [12501.627358] [<ffffffffa03ff1eb>] ? setup_conf+0x12b/0x360 [raid10] Dec 30 20:08:46 odra kernel: [12501.627361] [<ffffffff81120380>] alloc_debug_processing+0x110/0x1f0 Dec 30 20:08:46 odra kernel: [12501.627365] [<ffffffff811211c9>] __slab_alloc+0x3a9/0x410 Dec 30 20:08:46 odra kernel: [12501.627369] [<ffffffff810de600>] ? mempool_alloc_slab+0x10/0x20 Dec 30 20:08:46 odra kernel: [12501.627372] [<ffffffff8112166f>] ? kmem_cache_alloc_node_notrace+0xbf/0xe0 Dec 30 20:08:46 odra kernel: [12501.627376] [<ffffffff810de7fe>] ? mempool_create_node+0x7e/0x1a0 Dec 30 20:08:46 odra kernel: [12501.627379] [<ffffffffa03ff1eb>] ? setup_conf+0x12b/0x360 [raid10] Dec 30 20:08:46 odra kernel: [12501.627382] [<ffffffff81121888>] kmem_cache_alloc_notrace+0xb8/0xc0 Dec 30 20:08:46 odra kernel: [12501.627386] [<ffffffffa03ff1eb>] setup_conf+0x12b/0x360 [raid10] Dec 30 20:08:46 odra kernel: [12501.627390] [<ffffffffa04026b1>] run+0x21/0x3c0 [raid10] Dec 30 20:08:46 odra kernel: [12501.627413] [<ffffffffa00ca322>] md_run+0x322/0x920 [md_mod] Dec 30 20:08:46 odra kernel: [12501.627417] [<ffffffff813fd7a0>] ? __mutex_lock_interruptible_slowpath+0x1e0/0x2b0 Dec 30 20:08:46 odra kernel: [12501.627425] [<ffffffffa00ca939>] do_md_run+0x19/0xa0 [md_mod] Dec 30 20:08:46 odra kernel: [12501.627432] [<ffffffffa00cbefc>] md_ioctl+0xa1c/0x1350 [md_mod] Dec 30 20:08:46 odra kernel: [12501.627435] [<ffffffff8111f15f>] ? check_object+0xaf/0x270 Dec 30 20:08:46 odra kernel: [12501.627438] [<ffffffff8111f706>] ? init_object+0x46/0x80 Dec 30 20:08:46 odra kernel: [12501.627442] [<ffffffff812039e0>] blkdev_ioctl+0x230/0x720 Dec 30 20:08:46 odra kernel: [12501.627445] [<ffffffff81120846>] ? __slab_free+0x136/0x150 Dec 30 20:08:46 odra kernel: [12501.627449] [<ffffffff811607dc>] block_ioctl+0x3c/0x40 Dec 30 20:08:46 odra kernel: [12501.627453] [<ffffffff811412f8>] do_vfs_ioctl+0x98/0x580 Dec 30 20:08:46 odra kernel: [12501.627456] [<ffffffff81101af9>] ? remove_vma+0x69/0x90 Dec 30 20:08:46 odra kernel: [12501.627460] [<ffffffff81103244>] ? do_munmap+0x2e4/0x360 Dec 30 20:08:46 odra kernel: [12501.627463] [<ffffffff81141861>] sys_ioctl+0x81/0xa0 Dec 30 20:08:46 odra kernel: [12501.627467] [<ffffffff81002d7b>] system_call_fastpath+0x16/0x1b
Quoted 5 lines · Neil Brown
Please report exactly which kernel you are running (git hash of head) and in particular whether
commit589a594be1fb8815b3f18e517be696c48664f728is present?
i'm testing the pure 2.6.37-rc8 which afaics contains mentioned merge:
3d0b608 589a594
Author: Linus Torvalds <[email protected]> Date: Tue Dec 14 18:49:40 2010 -0800
Merge branch 'for-linus' of git://neil.brown.name/md