kmemleak reports in kernel 3.9.5+
Subsystems (2)
We had a system go OOM while doing lots of wireless stations. (System had 8GB of RAM, so I suspect a leak).
I enabled kmemleak in a 3.9.5 (plus some local patches) and I see the entries below. Any idea if these are real or not?
unreferenced object 0xffff880212281c80 (size 128):
comm "systemd", pid 1, jiffies 4294682684 (age 1159.517s) hex dump (first 32 bytes):
60 39 27 12 02 88 ff ff 00 02 20 00 00 00 ad de `9'....... .....
10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d605>] __kmalloc+0xf9/0x122
[<ffffffff8154946d>] kzalloc.clone.0+0xe/0x10
[<ffffffff81549494>] fib_default_rule_add+0x25/0x7a
[<ffffffffa014f5a9>] ip6mr_net_init+0x7e/0x118 [ipv6]
[<ffffffff8152c992>] ops_init+0xd6/0xf7
[<ffffffff8152cb51>] register_pernet_operations+0xc2/0x16b
[<ffffffff8152cc87>] register_pernet_subsys+0x2e/0x47
[<ffffffffa016db69>] 0xffffffffa016db69
[<ffffffffa016d109>] 0xffffffffa016d109
[<ffffffff8100207f>] do_one_initcall+0x7f/0x13e
[<ffffffff810f3985>] do_init_module+0x44/0x18f
[<ffffffff810f5da7>] load_module+0x14d1/0x168e
[<ffffffff810f6114>] sys_init_module+0xfd/0x101
[<ffffffff815f6599>] system_call_fastpath+0x16/0x1b
unreferenced object 0xffff880218ed92b0 (size 40):
comm "gnome-session", pid 934, jiffies 4294693971 (age 1148.267s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 00 97 41 8b 01 88 ff ff ..........A.....
02 00 00 00 00 00 00 00 20 44 46 08 00 ea ff ff ........ DF.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff811971f6>] put_object+0x46/0x4a
[<ffffffff811974e3>] delete_object_full+0x2d/0x32
[<ffffffff815de663>] kmemleak_free+0x59/0x7a
[<ffffffff8118bc0a>] slab_free_hook+0x21/0x87
[<ffffffff8118e888>] kmem_cache_free+0xbe/0x15d
[<ffffffff81175d98>] unlink_anon_vmas+0x9f/0x154
unreferenced object 0xffff88021716f870 (size 40):
comm "ip", pid 3146, jiffies 4294741842 (age 1100.453s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 c0 dc 37 ee 01 88 ff ff ..........7.....
01 00 00 00 00 00 00 00 88 35 0c e4 01 88 ff ff .........5......
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810a70df>] init_timer_key+0x2e/0xb3
[<ffffffff81521dcb>] sock_init_data+0x69/0x1ed
[<ffffffff815551a5>] __netlink_create+0x4f/0xb5
[<ffffffff815552fd>] netlink_create+0xf2/0x148
[<ffffffff8151ec54>] __sock_create+0x135/0x1be
[<ffffffff8151ed33>] sock_create+0x30/0x32
[<ffffffff81520517>] sys_socket+0x2f/0x98
[<ffffffff815f6599>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
unreferenced object 0xffff8801ebeb4450 (size 40):
comm "bash", pid 3200, jiffies 4294742742 (age 1099.584s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 c0 e5 91 f1 01 88 ff ff ................
01 00 00 00 00 00 00 00 40 4d 3d e4 01 88 ff ff ........@M=.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff811971f6>] put_object+0x46/0x4a
[<ffffffff811974e3>] delete_object_full+0x2d/0x32
[<ffffffff815de663>] kmemleak_free+0x59/0x7a
[<ffffffff8118bc0a>] slab_free_hook+0x21/0x87
[<ffffffff8118e888>] kmem_cache_free+0xbe/0x15d
[<ffffffff81175d98>] unlink_anon_vmas+0x9f/0x154
unreferenced object 0xffff8801f191e5c0 (size 40):
comm "clock-applet", pid 3100, jiffies 4294742818 (age 1099.508s)
hex dump (first 32 bytes):
50 44 eb eb 01 88 ff ff 70 f8 bb 9a 01 88 ff ff PD......p.......
01 00 00 00 00 00 00 00 58 45 3d e4 01 88 ff ff ........XE=.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff811971f6>] put_object+0x46/0x4a
[<ffffffff811974e3>] delete_object_full+0x2d/0x32
[<ffffffff815de663>] kmemleak_free+0x59/0x7a
[<ffffffff8118bc0a>] slab_free_hook+0x21/0x87
[<ffffffff8118e888>] kmem_cache_free+0xbe/0x15d
[<ffffffff811a51c6>] final_putname+0x38/0x3c
unreferenced object 0xffff8801ed2468a0 (size 40):
comm "wnck-applet", pid 3093, jiffies 4294742922 (age 1099.404s)
hex dump (first 32 bytes):
c0 3c b3 f1 01 88 ff ff 80 19 6a 82 ff ff ff ff .<........j.....
01 00 00 00 00 00 00 00 c8 74 31 e4 01 88 ff ff .........t1.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff811971f6>] put_object+0x46/0x4a
[<ffffffff811974e3>] delete_object_full+0x2d/0x32
[<ffffffff815de663>] kmemleak_free+0x59/0x7a
[<ffffffff8118bc0a>] slab_free_hook+0x21/0x87
[<ffffffff8118daf5>] kfree+0xdd/0x158
[<ffffffff815262b9>] skb_free_head+0x67/0x69
unreferenced object 0xffff8801f193cb80 (size 40):
comm "dbus-daemon", pid 1377, jiffies 4294743052 (age 1099.304s)
hex dump (first 32 bytes):
60 0e 39 ee 01 88 ff ff 70 e1 75 c6 01 88 ff ff `.9.....p.u.....
01 00 00 00 00 00 00 00 a0 2d 01 e5 01 88 ff ff .........-......
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff811971f6>] put_object+0x46/0x4a
[<ffffffff811974e3>] delete_object_full+0x2d/0x32
[<ffffffff815de663>] kmemleak_free+0x59/0x7a
[<ffffffff8118bc0a>] slab_free_hook+0x21/0x87
[<ffffffff8118e888>] kmem_cache_free+0xbe/0x15d
[<ffffffff811a51c6>] final_putname+0x38/0x3c
unreferenced object 0xffff8801f1b33cc0 (size 40):
comm "wpa_cli", pid 3221, jiffies 4294743135 (age 1099.221s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 a0 68 24 ed 01 88 ff ff .........h$.....
01 00 00 00 00 00 00 00 b0 7c 31 e4 01 88 ff ff .........|1.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff8119be2b>] file_free+0x31/0x35
[<ffffffff8119be61>] put_filp+0x32/0x36
[<ffffffff811a6bf1>] path_openat+0x340/0x379
[<ffffffff811a6d39>] do_filp_open+0x3d/0x89
[<ffffffff81199346>] do_sys_open+0x72/0x104
[<ffffffff8119940f>] sys_open+0x21/0x23
unreferenced object 0xffff8801ee390e60 (size 40):
comm "evolution-alarm", pid 3011, jiffies 4294743335 (age 1099.023s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 80 cb 93 f1 01 88 ff ff ................
01 00 00 00 00 00 00 00 b8 25 01 e5 01 88 ff ff .........%......
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff811971f6>] put_object+0x46/0x4a
[<ffffffff811974e3>] delete_object_full+0x2d/0x32
[<ffffffff815de663>] kmemleak_free+0x59/0x7a
[<ffffffff8118bc0a>] slab_free_hook+0x21/0x87
[<ffffffff8118e888>] kmem_cache_free+0xbe/0x15d
[<ffffffff811a51c6>] final_putname+0x38/0x3c
unreferenced object 0xffff880216e7ca10 (size 40):
comm "metacity", pid 2810, jiffies 4294744465 (age 1097.923s)
hex dump (first 32 bytes):
30 fe 97 ee 01 88 ff ff 80 6b 5c 82 ff ff ff ff 0........k\.....
01 00 00 00 00 00 00 00 70 3d 15 e5 01 88 ff ff ........p=......
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff811971f6>] put_object+0x46/0x4a
[<ffffffff811974e3>] delete_object_full+0x2d/0x32
[<ffffffff815de663>] kmemleak_free+0x59/0x7a
[<ffffffff8118bc0a>] slab_free_hook+0x21/0x87
[<ffffffff8118e888>] kmem_cache_free+0xbe/0x15d
[<ffffffff81527493>] __kfree_skb+0x7d/0x82
unreferenced object 0xffff8801ee97fe30 (size 40):
comm "pool", pid 3081, jiffies 4294744513 (age 1097.875s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 10 ca e7 16 02 88 ff ff ................
01 00 00 00 00 00 00 00 88 35 15 e5 01 88 ff ff .........5......
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff811971f6>] put_object+0x46/0x4a
[<ffffffff811974e3>] delete_object_full+0x2d/0x32
[<ffffffff815de663>] kmemleak_free+0x59/0x7a
[<ffffffff8118bc0a>] slab_free_hook+0x21/0x87
[<ffffffff8118e888>] kmem_cache_free+0xbe/0x15d
[<ffffffff811a51c6>] final_putname+0x38/0x3cSignature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 20 lines · Ben Greear
We had a system go OOM while doing lots of wireless stations. (System had 8GB of RAM, so I suspect a leak).
I enabled kmemleak in a 3.9.5 (plus some local patches) and I see the entries below. Any idea if these are real or not?
unreferenced object
0xffff880212281c80(size 128):
comm "systemd", pid 1, jiffies 4294682684 (age 1159.517s) hex dump (first 32 bytes):60 39 27 12 02 88 ff ff 00 02 20 00 00 00 ad de `9'....... ..... 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ backtrace: [<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98 [<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a [<ffffffff8118d605>] __kmalloc+0xf9/0x122 [<ffffffff8154946d>] kzalloc.clone.0+0xe/0x10 [<ffffffff81549494>] fib_default_rule_add+0x25/0x7a [<ffffffffa014f5a9>] ip6mr_net_init+0x7e/0x118 [ipv6] [<ffffffff8152c992>] ops_init+0xd6/0xf7 [<ffffffff8152cb51>] register_pernet_operations+0xc2/0x16b
Quoted 8 lines
[<ffffffff8152cc87>] register_pernet_subsys+0x2e/0x47
[<ffffffffa016db69>] 0xffffffffa016db69
[<ffffffffa016d109>] 0xffffffffa016d109
[<ffffffff8100207f>] do_one_initcall+0x7f/0x13e
[<ffffffff810f3985>] do_init_module+0x44/0x18f
[<ffffffff810f5da7>] load_module+0x14d1/0x168e
[<ffffffff810f6114>] sys_init_module+0xfd/0x101
[<ffffffff815f6599>] system_call_fastpath+0x16/0x1bNo idea yet. You can try:
echo clear > /sys/kernel/debug/kmemleak
and see if there are more appearing after. All seem to have a common allocation path via debug_object_activate -> ... -> rcuhead_fixup_activate -> ... -> __debug_object_init.
--
Catalin
Quoted 30 lines · Catalin Marinas
Quoted 20 lines · Ben Greear
We had a system go OOM while doing lots of wireless stations. (System had 8GB of RAM, so I suspect a leak).
I enabled kmemleak in a 3.9.5 (plus some local patches) and I see the entries below. Any idea if these are real or not?
unreferenced object
0xffff880212281c80(size 128):comm "systemd", pid 1, jiffies 4294682684 (age 1159.517s) hex dump (first 32 bytes): 60 39 27 12 02 88 ff ff 00 02 20 00 00 00 ad de `9'....... ..... 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ backtrace: [<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98 [<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a [<ffffffff8118d605>] __kmalloc+0xf9/0x122 [<ffffffff8154946d>] kzalloc.clone.0+0xe/0x10 [<ffffffff81549494>] fib_default_rule_add+0x25/0x7a [<ffffffffa014f5a9>] ip6mr_net_init+0x7e/0x118 [ipv6] [<ffffffff8152c992>] ops_init+0xd6/0xf7 [<ffffffff8152cb51>] register_pernet_operations+0xc2/0x16bQuoted 8 lines
[<ffffffff8152cc87>] register_pernet_subsys+0x2e/0x47 [<ffffffffa016db69>] 0xffffffffa016db69 [<ffffffffa016d109>] 0xffffffffa016d109 [<ffffffff8100207f>] do_one_initcall+0x7f/0x13e [<ffffffff810f3985>] do_init_module+0x44/0x18f [<ffffffff810f5da7>] load_module+0x14d1/0x168e [<ffffffff810f6114>] sys_init_module+0xfd/0x101 [<ffffffff815f6599>] system_call_fastpath+0x16/0x1b
Quoted 7 lines
No idea yet. You can try:
echo clear >
/sys/kernel/debug/kmemleakand see if there are more appearing after. All seem to have a common allocation path via
debug_object_activate-> ... ->rcuhead_fixup_activate-> ... ->__debug_object_init.
I let the system run overnight, and there are a good many more kmemleak reports this morning. I have not yet tried the clear option.
I have SLUB debugging enabled, so that is probably explains the debug stuff in the stacks.
Here are some more that may be of interest.
unreferenced object 0xffff88010b719700 (size 40):
comm "gua", pid 11301, jiffies 4352877266 (age 29217.682s) hex dump (first 32 bytes):
00 e0 10 08 01 88 ff ff a0 68 1e 13 01 88 ff ff .........h......
01 00 00 00 00 00 00 00 70 3d 56 13 01 88 ff ff ........p=V.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c4c>] debug_object_init_on_stack+0x17/0x19
[<ffffffff810bb5fa>] hrtimer_init_on_stack+0x27/0x75
[<ffffffff815ed663>] schedule_hrtimeout_range_clock+0x67/0x113
[<ffffffff815ed722>] schedule_hrtimeout_range+0x13/0x15
[<ffffffff811aa1fa>] poll_schedule_timeout+0x48/0x64
[<ffffffff811aaff6>] do_select+0x519/0x550
[<ffffffff811df227>] compat_core_sys_select+0x1c7/0x25a
[<ffffffff811df526>] compat_sys_select+0x99/0xc1
[<ffffffff815f7c4c>] sysenter_dispatch+0x7/0x26
[<ffffffffffffffff>] 0xffffffffffffffffunreferenced object 0xffff880111d33420 (size 40):
comm "ps", pid 25114, jiffies 4352875696 (age 29219.150s) hex dump (first 32 bytes):
c0 3c 68 0b 01 88 ff ff 70 41 ec 11 01 88 ff ff .<h.....pA......
01 00 00 00 00 00 00 00 e0 6c 5a 12 01 88 ff ff .........lZ.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff8119be2b>] file_free+0x31/0x35
[<ffffffff8119c05a>] __fput+0x1bb/0x1db
[<ffffffff8119c0ca>] ____fput+0xe/0x10
[<ffffffff810b48d5>] task_work_run+0x85/0xb0
[<ffffffff810106c6>] do_notify_resume+0x6d/0x80
[<ffffffff815f68d2>] int_signal+0x12/0x17unreferenced object 0xffff88010cc92e60 (size 40):
comm "updatedb", pid 24952, jiffies 4352871287 (age 29223.220s) hex dump (first 32 bytes):
00 60 16 0e 01 88 ff ff d0 af 19 10 01 88 ff ff .`..............
01 00 00 00 00 00 00 00 a0 2d 39 0e 01 88 ff ff .........-9.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b296c>] __init_work+0x27/0x29
[<ffffffff81241371>] ext4_alloc_inode+0x173/0x1cb
[<ffffffff811b0db4>] alloc_inode+0x1d/0x78
[<ffffffff811b0e7c>] iget_locked+0x6d/0x12f
[<ffffffff81229817>] ext4_iget+0x2c/0x932
[<ffffffff8122f485>] ext4_lookup+0xd2/0x12a
[<ffffffff811a3556>] lookup_real+0x2c/0x47
[<ffffffff811a3e0c>] __lookup_hash+0x33/0x3c
[<ffffffff811a5897>] walk_component+0x73/0x171
[<ffffffff811a6e2c>] path_lookupat+0xa7/0x304
[<ffffffff811a70b4>] filename_lookup+0x2b/0x82(I sent a similar one to the ext4 mailing list already.)
unreferenced object 0xffff8801c6d2eb80 (size 40):
comm "softirq", pid 0, jiffies 4296641468 (age 85451.733s) hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 70 01 6e 7b 01 88 ff ff ........p.n{....
02 00 00 00 00 00 00 00 58 54 24 ca 01 88 ff ff ........XT$.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810a70df>] init_timer_key+0x2e/0xb3
[<ffffffffa0128e4d>] ipv6_add_addr+0x19f/0x3bc [ipv6]
[<ffffffffa012c885>] addrconf_add_linklocal+0x4c/0x7f [ipv6]
[<ffffffffa012d06a>] addrconf_notify+0x773/0x95e [ipv6]
[<ffffffff815f30d6>] notifier_call_chain+0x37/0x63
[<ffffffff810bc994>] raw_notifier_call_chain+0x14/0x16
[<ffffffff815333ae>] call_netdevice_notifiers+0x4a/0x4f
[<ffffffff8153404d>] netdev_state_change+0x27/0x3a
[<ffffffff815418ef>] linkwatch_do_dev+0x46/0x54
[<ffffffff81541b0e>] __linkwatch_run_queue+0x180/0x1ca
[<ffffffff81541b7d>] linkwatch_event+0x25/0x2cunreferenced object 0xffff8801c8e41e78 (size 192):
comm "kworker/u:2", pid 157, jiffies 4295509873 (age 86582.869s) hex dump (first 32 bytes):
41 0d 00 30 02 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b A..0....kkkkkkkk
6b 6b 6b 6b 6b 6b 6b 6b 69 00 00 00 00 0c 2e 32 kkkkkkkki......2
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d605>] __kmalloc+0xf9/0x122
[<ffffffffa027cb27>] cfg80211_inform_bss_frame+0x114/0x1f8 [cfg80211]
[<ffffffffa03d6865>] ieee80211_bss_info_update+0x66/0x21f [mac80211]
[<ffffffffa040aec6>] ieee80211_rx_bss_info+0x12f/0x1ca [mac80211]
[<ffffffffa040b017>] ieee80211_rx_mgmt_probe_resp+0xb6/0x197 [mac80211]
[<ffffffffa040e8a3>] ieee80211_sta_rx_queued_mgmt+0xdd/0x60e [mac80211]
[<ffffffffa03df0ee>] ieee80211_iface_work+0x238/0x2cc [mac80211]
[<ffffffff810b0cd3>] process_one_work+0x292/0x42e
[<ffffffff810b36af>] worker_thread+0x14f/0x264
[<ffffffff810b7bea>] kthread+0xc7/0xcf
[<ffffffff815f64ec>] ret_from_fork+0x7c/0xb0
[<ffffffffffffffff>] 0xffffffffffffffff(I plan to investigate the one above...I have some understanding of the wifi code, and have some patches of my own applied to that code.)
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 30 lines · Catalin Marinas
Quoted 20 lines · Ben Greear
We had a system go OOM while doing lots of wireless stations. (System had 8GB of RAM, so I suspect a leak).
I enabled kmemleak in a 3.9.5 (plus some local patches) and I see the entries below. Any idea if these are real or not?
unreferenced object
0xffff880212281c80(size 128):comm "systemd", pid 1, jiffies 4294682684 (age 1159.517s) hex dump (first 32 bytes): 60 39 27 12 02 88 ff ff 00 02 20 00 00 00 ad de `9'....... ..... 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ backtrace: [<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98 [<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a [<ffffffff8118d605>] __kmalloc+0xf9/0x122 [<ffffffff8154946d>] kzalloc.clone.0+0xe/0x10 [<ffffffff81549494>] fib_default_rule_add+0x25/0x7a [<ffffffffa014f5a9>] ip6mr_net_init+0x7e/0x118 [ipv6] [<ffffffff8152c992>] ops_init+0xd6/0xf7 [<ffffffff8152cb51>] register_pernet_operations+0xc2/0x16bQuoted 8 lines
[<ffffffff8152cc87>] register_pernet_subsys+0x2e/0x47 [<ffffffffa016db69>] 0xffffffffa016db69 [<ffffffffa016d109>] 0xffffffffa016d109 [<ffffffff8100207f>] do_one_initcall+0x7f/0x13e [<ffffffff810f3985>] do_init_module+0x44/0x18f [<ffffffff810f5da7>] load_module+0x14d1/0x168e [<ffffffff810f6114>] sys_init_module+0xfd/0x101 [<ffffffff815f6599>] system_call_fastpath+0x16/0x1b
Quoted 7 lines
No idea yet. You can try:
echo clear >
/sys/kernel/debug/kmemleakand see if there are more appearing after. All seem to have a common allocation path via
debug_object_activate-> ... ->rcuhead_fixup_activate-> ... ->__debug_object_init.
I tried the command below, and it printed out quite a few things.
I'll try building a kernel without the extra SLUB debugging to see if that helps.
Also, I read the kmemleak.txt documentation, but a question remains:
If I enable kmemleak at compile time, but disable it at boot time using kmemleak=off, is there any significant runtime overhead?
[root@LEC2220-1 ~]# echo clear > /debug/kmemleak;sleep 60;echo scan > /debug/kmemleak; cat /debug/kmemleak unreferenced object 0xffff88021867e450 (size 40):
comm "chrony-helper", pid 1138, jiffies 4294699268 (age 91773.781s)
hex dump (first 32 bytes):
d0 cf e5 18 02 88 ff ff 80 0b be 19 02 88 ff ff ................
01 00 00 00 00 00 00 00 f0 34 a3 12 02 88 ff ff .........4......
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff811971f6>] put_object+0x46/0x4a
[<ffffffff811974e3>] delete_object_full+0x2d/0x32
[<ffffffff815de663>] kmemleak_free+0x59/0x7a
[<ffffffff8118bc0a>] slab_free_hook+0x21/0x87
[<ffffffff8118e888>] kmem_cache_free+0xbe/0x15d
[<ffffffff811acb71>] __d_free+0x56/0x5b
unreferenced object 0xffff880211893b50 (size 40):
comm "nmcli", pid 1178, jiffies 4294699660 (age 91773.390s)
hex dump (first 32 bytes):
00 00 00 00 00 00 00 00 f0 ac bd 19 02 88 ff ff ................
01 00 00 00 00 00 00 00 e0 6c 6e 01 02 88 ff ff .........ln.....
backtrace:
[<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98
[<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a
[<ffffffff8118d9a7>] kmem_cache_alloc+0xb2/0x123
[<ffffffff81316919>] __debug_object_init+0x43/0x35f
[<ffffffff81316c62>] debug_object_init+0x14/0x16
[<ffffffff810b4e0a>] rcuhead_fixup_activate+0x2b/0xba
[<ffffffff81315f12>] debug_object_fixup+0x15/0x1d
[<ffffffff81316557>] debug_object_activate+0x126/0x139
[<ffffffff81118e4a>] __call_rcu.clone.1+0x58/0x22a
[<ffffffff81119065>] call_rcu+0x17/0x19
[<ffffffff8119be2b>] file_free+0x31/0x35
[<ffffffff8119c05a>] __fput+0x1bb/0x1db
[<ffffffff8119c0ca>] ____fput+0xe/0x10
[<ffffffff810b48d5>] task_work_run+0x85/0xb0
[<ffffffff8109ccc1>] do_exit+0x3c9/0x978
[<ffffffff8109d2f3>] do_group_exit+0x83/0xae....
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 7 lines · Ben Greear
Quoted 6 lines · Catalin Marinas
Quoted 5 lines · Ben Greear
We had a system go OOM while doing lots of wireless stations. (System had 8GB of RAM, so I suspect a leak).
I enabled kmemleak in a 3.9.5 (plus some local patches) and I see the entries below. Any idea if these are real or not?
Most of this went away when I disabled SLUB debugging and other kernel hacking options. The wifi cfg80211_inform_bss_frame remains, however. I'll go dig some more on that tomorrow...didn't see anything obvious at first glance.
But, perhaps there could be some improvements to kmemleak to make it deal better with the various kernel debugging features?
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 9 lines · Ben Greear
Quoted 7 lines · Ben Greear
Quoted 6 lines · Catalin Marinas
Quoted 5 lines · Ben Greear
We had a system go OOM while doing lots of wireless stations. (System had 8GB of RAM, so I suspect a leak).
I enabled kmemleak in a 3.9.5 (plus some local patches) and I see the entries below. Any idea if these are real or not?
Quoted 8 lines
Most of this went away when I disabled SLUB debugging and other kernel hacking options. The wifi
cfg80211_inform_bss_frameremains, however. I'll go dig some more on that tomorrow...didn't see anything obvious at first glance.But, perhaps there could be some improvements to kmemleak to make it deal better with the various kernel debugging features?
That's unrelated to the debugging features. Kmemleak cannot find pointers to the allocated objects. They could be real leaks or it simply doesn't scan the right memory where such pointers are stored. The debug objects are stored in a list with the head as static memory, so it should be scanned.
Signature · 1 line
Catalin
Quoted 19 lines · Catalin Marinas
Quoted 9 lines · Ben Greear
Quoted 7 lines · Ben Greear
Quoted 6 lines · Catalin Marinas
Quoted 5 lines · Ben Greear
We had a system go OOM while doing lots of wireless stations. (System had 8GB of RAM, so I suspect a leak).
I enabled kmemleak in a 3.9.5 (plus some local patches) and I see the entries below. Any idea if these are real or not?
Quoted 8 lines
Most of this went away when I disabled SLUB debugging and other kernel hacking options. The wifi
cfg80211_inform_bss_frameremains, however. I'll go dig some more on that tomorrow...didn't see anything obvious at first glance.But, perhaps there could be some improvements to kmemleak to make it deal better with the various kernel debugging features?
Quoted 5 lines
That's unrelated to the debugging features. Kmemleak cannot find pointers to the allocated objects. They could be real leaks or it simply doesn't scan the right memory where such pointers are stored. The debug objects are stored in a list with the head as static memory, so it should be scanned.
I re-ran a similar test just now, but I had added an explicit reference for each of the items that I know this test causes to leak (I add them to a global list.)
This means that memleak would no longer show them as leaked, and that also cleaned up all of the strange leaks reported before.
So, I think kmemleak is at least mostly working properly, even with lots of debugging options.
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 33 lines · Ben Greear
Quoted 32 lines · Catalin Marinas
Quoted 20 lines · Ben Greear
We had a system go OOM while doing lots of wireless stations. (System had 8GB of RAM, so I suspect a leak).
I enabled kmemleak in a 3.9.5 (plus some local patches) and I see the entries below. Any idea if these are real or not?
unreferenced object
0xffff880212281c80(size 128):comm "systemd", pid 1, jiffies 4294682684 (age 1159.517s) hex dump (first 32 bytes): 60 39 27 12 02 88 ff ff 00 02 20 00 00 00 ad de `9'....... ..... 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ backtrace: [<ffffffff815de7bf>] kmemleak_alloc+0x73/0x98 [<ffffffff8118b4d4>] slab_post_alloc_hook+0x28/0x2a [<ffffffff8118d605>] __kmalloc+0xf9/0x122 [<ffffffff8154946d>] kzalloc.clone.0+0xe/0x10 [<ffffffff81549494>] fib_default_rule_add+0x25/0x7a [<ffffffffa014f5a9>] ip6mr_net_init+0x7e/0x118 [ipv6] [<ffffffff8152c992>] ops_init+0xd6/0xf7 [<ffffffff8152cb51>] register_pernet_operations+0xc2/0x16bQuoted 8 lines
[<ffffffff8152cc87>] register_pernet_subsys+0x2e/0x47 [<ffffffffa016db69>] 0xffffffffa016db69 [<ffffffffa016d109>] 0xffffffffa016d109 [<ffffffff8100207f>] do_one_initcall+0x7f/0x13e [<ffffffff810f3985>] do_init_module+0x44/0x18f [<ffffffff810f5da7>] load_module+0x14d1/0x168e [<ffffffff810f6114>] sys_init_module+0xfd/0x101 [<ffffffff815f6599>] system_call_fastpath+0x16/0x1bNo idea yet. You can try:
Quoted 7 lines
Quoted 5 lines
echo clear >
/sys/kernel/debug/kmemleakand see if there are more appearing after. All seem to have a common allocation path via
debug_object_activate-> ... ->rcuhead_fixup_activate-> ... ->__debug_object_init.I tried the command below, and it printed out quite a few things.
Can you send me your .config file? I can't reproduce this.
Quoted 4 lines
Also, I read the kmemleak.txt documentation, but a question remains:
If I enable kmemleak at compile time, but disable it at boot time using kmemleak=off, is there any significant runtime overhead?
You still get the callback into kmemleak for each allocation/freeing but it doesn't walk the stack for the backtrace and it doesn't scan the memory either. I would say the overhead is very small, probably unnoticeable if you already have other debug options enabled.
Signature · 1 line
Catalin