Use-after-free in hacked 2.6.38.8 kernel.

PATCH Ben Greear 1 participant 2 messages Jun 23, 2011 – Jun 24, 2011

Subsystems (3)

Networking NFS client NFS server

lore

Share & tools

lore

Ben Greear

author

2011-06-23 22:29 UTC #0 lore raw

2.6.38.8 kernel, with our NFS bind-source-IP patches and some other stuff, including a tainting module (though that module isn't active in this test).

I'm also running the patch I posted a few days ago that explicitly un-links the xpt_ready list:

net/sunrpc/svc_xprt.c +1 -0

index ab86b79..178716f 100644
@@ -901,6 +901,7 @@ void svc_delete_xprt(struct svc_xprt *xprt)
 	spin_lock_bh(&serv->sv_lock);
 	if (!test_and_set_bit(XPT_DETACHED, &xprt->xpt_flags))
 		list_del_init(&xprt->xpt_list);
+	list_del_init(&xprt->xpt_ready);
 	/*
 	 * We used to delete the transport from whichever list
 	 * it's sk_xprt.xpt_ready node was on, but we don't actually

Test is to create 200 unique mounts (using unique srcaddr) and mount/run-file-io-traffic/unmount them every 15 seconds.

It hit this bug after about 5 hours.

I'm going to try to figure this out, but any help is appreciated!

=============================================================================
BUG kmalloc-64: Poison overwritten
-----------------------------------------------------------------------------

INFO: 0xffff8800c6da9dd0-0xffff8800c6da9e03. First byte 0x48 instead of 0x6b
INFO: Allocated in nfs_get_lock_context+0xa4/0x179 [nfs] age=60 cpu=2 pid=9218
INFO: Freed in nfs_put_lock_context+0x3f/0x44 [nfs] age=70 cpu=0 pid=8543
INFO: Slab 0xffffea0002b7fcf8 objects=30 used=26 fp=0xffff8800c6da9dd0 flags=0x200000000000c1
INFO: Object 0xffff8800c6da9dd0 @offset=3536 fp=0xffff8800c6da9d48

Bytes b4 0xffff8800c6da9dc0: fe b7 0f 01 01 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ??......ZZZZZZZZ
Object 0xffff8800c6da9dd0: 48 90 b9 b3 00 88 ff ff 6b 6b 6b 6b 6b 6b 6b 6b
H.??..??kkkkkkkk
Object 0xffff8800c6da9de0: 06 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
......kkkkkkkkkk
Object 0xffff8800c6da9df0: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b
........kkkkkkkk
Object 0xffff8800c6da9e00: f3 ff ff ff 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5
????kkkkkkkkkkk?
Redzone 0xffff8800c6da9e10: bb bb bb bb bb bb bb bb
????????
Padding 0xffff8800c6da9e50: 5a 5a 5a 5a 5a 5a 5a 5a
ZZZZZZZZ
Pid: 9019, comm: btserver Tainted: P 2.6.38.8+ #9

Call Trace:
 [<ffffffff81100aee>] ? print_trailer+0x12e/0x137
 [<ffffffff81100fb7>] ? check_bytes_and_report+0xb9/0xfd
 [<ffffffffa030ddbb>] ? nfs_get_lock_context+0x94/0x179 [nfs]
 [<ffffffff811010b0>] ? check_object+0xb5/0x192
 [<ffffffffa030ddcb>] ? nfs_get_lock_context+0xa4/0x179 [nfs]
 [<ffffffff811014d1>] ? alloc_debug_processing+0x79/0xf2
 [<ffffffff81102bff>] ? __slab_alloc+0x337/0x375
 [<ffffffffa030ddcb>] ? nfs_get_lock_context+0xa4/0x179 [nfs]
 [<ffffffffa030dd4f>] ? nfs_get_lock_context+0x28/0x179 [nfs]
 [<ffffffffa030ddcb>] ? nfs_get_lock_context+0xa4/0x179 [nfs]
 [<ffffffff81103d87>] ? kmem_cache_alloc_trace+0x76/0xef
 [<ffffffff81465d62>] ? sub_preempt_count+0x92/0xa6
 [<ffffffffa030ddcb>] ? nfs_get_lock_context+0xa4/0x179 [nfs]
 [<ffffffffa0313c32>] ? nfs_file_direct_write+0x1ab/0x752 [nfs]
 [<ffffffff81122b25>] ? pollwake+0x0/0x4f
 [<ffffffff810423db>] ? get_parent_ip+0x11/0x41
 [<ffffffff811026f5>] ? __slab_free+0x86/0xf1
 [<ffffffff811429cf>] ? fsnotify_put_event+0x63/0x67
 [<ffffffff81077d44>] ? trace_hardirqs_on+0xd/0xf
 [<ffffffffa030bd9b>] ? nfs_file_write+0x5d/0x169 [nfs]
 [<ffffffff811134c8>] ? do_sync_write+0xc6/0x103
 [<ffffffff811df2b4>] ? security_file_permission+0x29/0x2e
 [<ffffffff81113e58>] ? vfs_write+0xa9/0x105
 [<ffffffff811145f5>] ? fget_light+0x35/0x94
 [<ffffffff81113f6d>] ? sys_write+0x45/0x6c
 [<ffffffff8100aa92>] ? system_call_fastpath+0x16/0x1b

FIX kmalloc-64: Restoring 0xffff8800c6da9dd0-0xffff8800c6da9e03=0x6b

FIX kmalloc-64: Marking all objects used
=============================================================================
BUG kmalloc-64: Redzone overwritten
-----------------------------------------------------------------------------

INFO: 0xffff8800c6da9e10-0xffff8800c6da9e17. First byte 0xbb instead of 0xcc
INFO: Allocated in nfs_get_lock_context+0xa4/0x179 [nfs] age=173 cpu=2 pid=9218
INFO: Freed in nfs_put_lock_context+0x3f/0x44 [nfs] age=172 cpu=0 pid=8543
INFO: Slab 0xffffea0002b7fcf8 objects=30 used=30 fp=0x (null)
flags=0x20000000000081
INFO: Object 0xffff8800c6da9dd0 @offset=3536 fp=0xffff8800c6da9d48

Bytes b4 0xffff8800c6da9dc0: fe b7 0f 01 01 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ??......ZZZZZZZZ
Object 0xffff8800c6da9dd0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
kkkkkkkkkkkkkkkk
Object 0xffff8800c6da9de0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
kkkkkkkkkkkkkkkk
Object 0xffff8800c6da9df0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
kkkkkkkkkkkkkkkk
Object 0xffff8800c6da9e00: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5
kkkkkkkkkkkkkkk?
Redzone 0xffff8800c6da9e10: bb bb bb bb bb bb bb bb
????????
Padding 0xffff8800c6da9e50: 5a 5a 5a 5a 5a 5a 5a 5a
ZZZZZZZZ
Pid: 13574, comm: mount.nfs Tainted: P 2.6.38.8+ #9

Call Trace:
 [<ffffffff81100aee>] ? print_trailer+0x12e/0x137
 [<ffffffff81100fb7>] ? check_bytes_and_report+0xb9/0xfd
 [<ffffffffa028a9cc>] ? rpcb_create_local+0x6a/0x112 [sunrpc]
 [<ffffffffa028a95d>] ? rpcb_map_release+0x3f/0x44 [sunrpc]
 [<ffffffff81101044>] ? check_object+0x49/0x192
 [<ffffffffa028a95d>] ? rpcb_map_release+0x3f/0x44 [sunrpc]
 [<ffffffff81101efd>] ? free_debug_processing+0x7a/0x18e
 [<ffffffffa028a95d>] ? rpcb_map_release+0x3f/0x44 [sunrpc]
 [<ffffffff8110274b>] ? __slab_free+0xdc/0xf1
 [<ffffffffa028a95d>] ? rpcb_map_release+0x3f/0x44 [sunrpc]
 [<ffffffff811031ad>] ? kfree+0x12e/0x166
 [<ffffffffa028a95d>] ? rpcb_map_release+0x3f/0x44 [sunrpc]
 [<ffffffffa0281eae>] ? rpc_release_calldata+0x12/0x14 [sunrpc]
 [<ffffffffa0282080>] ? rpc_free_task+0x59/0x61 [sunrpc]
 [<ffffffffa028210a>] ? rpc_final_put_task+0x82/0x8a [sunrpc]
 [<ffffffffa028213d>] ? rpc_do_put_task+0x2b/0x32 [sunrpc]
 [<ffffffffa028215e>] ? rpc_put_task+0xb/0xd [sunrpc]
 [<ffffffffa028a8dd>] ? rpcb_getport_async+0x564/0x5a5 [sunrpc]
 [<ffffffff810423db>] ? get_parent_ip+0x11/0x41
 [<ffffffffa027b349>] ? call_bind+0x70/0x75 [sunrpc]
 [<ffffffffa0282911>] ? __rpc_execute+0x78/0x24b [sunrpc]
 [<ffffffff8106750e>] ? wake_up_bit+0x20/0x25
 [<ffffffffa0282b21>] ? rpc_execute+0x3d/0x42 [sunrpc]
 [<ffffffffa027ca9f>] ? rpc_run_task+0xe3/0xef [sunrpc]
 [<ffffffffa027cb89>] ? rpc_call_sync+0x3f/0x60 [sunrpc]
 [<ffffffffa027cbec>] ? rpc_ping+0x42/0x58 [sunrpc]
 [<ffffffff8146275b>] ? _raw_spin_unlock+0x45/0x52
 [<ffffffffa027d4d5>] ? rpc_create+0x493/0x50e [sunrpc]
 [<ffffffffa0307077>] ? nfs_get_client+0x50/0x536 [nfs]
 [<ffffffffa030698e>] ? nfs_create_rpc_client+0xb1/0xf6 [nfs]
 [<ffffffffa0307f92>] ? nfs_create_server+0x170/0x48e [nfs]
 [<ffffffff81077d44>] ? trace_hardirqs_on+0xd/0xf
 [<ffffffffa0312486>] ? nfs_get_sb+0x4e8/0x742 [nfs]
 [<ffffffff81115eb7>] ? vfs_kern_mount+0xea/0x1f6
 [<ffffffff81116021>] ? do_kern_mount+0x48/0xd8
 [<ffffffff8112da55>] ? do_mount+0x708/0x770
 [<ffffffff810f9723>] ? alloc_pages_current+0xaa/0xcd
 [<ffffffff8112db40>] ? sys_mount+0x83/0xbd
 [<ffffffff8100aa92>] ? system_call_fastpath+0x16/0x1b

FIX kmalloc-64: Restoring 0xffff8800c6da9e10-0xffff8800c6da9e17=0xcc

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Ben Greear

author

2011-06-24 15:29 UTC #1 lore raw

Quoted 3 lines · Ben Greear

2.6.38.8 kernel, with our NFS bind-source-IP patches and some other stuff, including a tainting module (though that module isn't active in this test).

And, another one. Different place this time though:

[root@simech2 ~]# general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC last sysfs file: /sys/devices/virtual/bdi/0:51/uevent
CPU 0

Modules linked in:
=============================================================================

BUG kmalloc-64: Poison overwritten
-----------------------------------------------------------------------------

INFO: 0xffff880097422de4-0xffff880097422de5. First byte 0x1 instead of 0x6b
INFO: Allocated in rpcb_getport_async+0x39c/0x5a5 [sunrpc] age=22 cpu=1 pid=23678
INFO: Freed in rpcb_map_release+0x3f/0x44 [sunrpc] age=20 cpu=0 pid=18587
INFO: Slab 0xffffea0002116770 objects=30 used=7 fp=0xffff880097422dd0 flags=0x200000000000c1
INFO: Object 0xffff880097422dd0 @offset=3536 fp=0xffff8800974224c8

Bytes b4 0xffff880097422dc0:  ae db 4e 00 01 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ®ÛN.....ZZZZZZZZ
  Object 0xffff880097422dd0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
  Object 0xffff880097422de0:  6b 6b 6b 6b 01 08 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkk..kkkkkkkkkk
  Object 0xffff880097422df0:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
  Object 0xffff880097422e00:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk¥
 Redzone 0xffff880097422e10:  bb bb bb bb bb bb bb bb                         »»»»»»»»
 Padding 0xffff880097422e50:  5a 5a 5a 5a 5a 5a 5a 5a                         ZZZZZZZZ
Pid: 2259, comm: gnuserver Tainted: P            2.6.38.8+ #9
Call Trace:
 [<ffffffff81100aee>] ? print_trailer+0x12e/0x137
 [<ffffffff81100fb7>] ? check_bytes_and_report+0xb9/0xfd
 [<ffffffff81129cbb>] ? alloc_fdtable+0xb2/0xda
 [<ffffffff811010b0>] ? check_object+0xb5/0x192
 [<ffffffff81129c6b>] ? alloc_fdtable+0x62/0xda
 [<ffffffff811014d1>] ? alloc_debug_processing+0x79/0xf2
 [<ffffffff81102bff>] ? __slab_alloc+0x337/0x375
 [<ffffffff81129c6b>] ? alloc_fdtable+0x62/0xda
 [<ffffffff81129d92>] ? dup_fd+0xaf/0x35b
 [<ffffffff81129c6b>] ? alloc_fdtable+0x62/0xda
 [<ffffffff81103d87>] ? kmem_cache_alloc_trace+0x76/0xef
 [<ffffffff81129c6b>] ? alloc_fdtable+0x62/0xda
 [<ffffffff81129e61>] ? dup_fd+0x17e/0x35b
 [<ffffffff8104957b>] ? copy_process+0x714/0x12a5
 [<ffffffff8105a25f>] ? sigprocmask+0x2f/0xc6
 [<ffffffff8104a217>] ? do_fork+0x10b/0x2ed
 [<ffffffff810e143d>] ? might_fault+0x63/0xb3
 [<ffffffff8111c993>] ? path_put+0x1d/0x22
 [<ffffffff81011fed>] ? sys_clone+0x23/0x25
 [<ffffffff8100ae33>] ? stub_clone+0x13/0x20
 [<ffffffff8100aa92>] ? system_call_fastpath+0x16/0x1b

FIX kmalloc-64: Restoring 0xffff880097422de4-0xffff880097422de5=0x6b

IX kmalloc-64: Marking all objects used
xt_TPROXY nf_tproxy_core xt_socket nf_defrag_ipv6 xt_connlimit 8021q garp macvlan pktgen iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi fuse
ip6table_filter ip6_tables ebtable_nat ebtables stp llc nfs lockd fscache nfs_acl auth_rpcgss sunrpc ipv6 kvm_intel kvm uinput i5k_amb i5000_edac iTCO_wdt ioatdma i2c_i801 shpchp iTCO_vendor_support edac_core pcspkr serio_raw e1000e microcode dca floppy radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core [last unloaded: ipt_addrtype]

Pid: 18587, comm: kworker/0:1 Tainted: P 2.6.38.8+ #9 Supermicro X7DBU/X7DBU

RIP: 0010:[<ffffffffa0294f5b>]  [<ffffffffa0294f5b>] rpcb_getport_done+0x47/0xab [sunrpc]
RSP: 0018:ffff8800c7133d20  EFLAGS: 00010246
RAX: ffffffffa0294f14 RBX: 0000000000000000 RCX: 0000000000000088
RDX: ffff880097422dd0 RSI: 000000006b6b0801 RDI: ffff880124668c80
RBP: ffff8800c7133d40 R08: ffff880097422dd0 R09: 0000000000000000
R10: ffff8800c7133d20 R11: ffff8800c7133c40 R12: ffff880097422dd0
R13: 6b6b6b6b6b6b6b6b R14: ffff880124668c80 R15: ffffffffa028db26
FS:  0000000000000000(0000) GS:ffff8800cfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000002666940 CR3: 00000000a73b6000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/0:1 (pid: 18587, threadinfo ffff8800c7132000, task ffff8800c4fea0a0)
Stack:
 ffff880124668c80 ffff880124668cf0 0000000000000001 0000000000000000
 ffff8800c7133d60 ffffffffa028d520 0000000000000000 ffff880124668c80
 ffff8800c7133db0 ffffffffa028d911 ffff8800ca597080 0000000000000000
Call Trace:
 [<ffffffffa028d520>] rpc_exit_task+0x27/0x55 [sunrpc]
 [<ffffffffa028d911>] __rpc_execute+0x78/0x24b [sunrpc]
 [<ffffffffa028db26>] ? rpc_async_schedule+0x0/0x12 [sunrpc]
 [<ffffffffa028db36>] rpc_async_schedule+0x10/0x12 [sunrpc]
 [<ffffffff81061613>] process_one_work+0x259/0x41b
 [<ffffffff8106153b>] ? process_one_work+0x181/0x41b
 [<ffffffff81063a51>] worker_thread+0x133/0x217
 [<ffffffff8106391e>] ? worker_thread+0x0/0x217
 [<ffffffff81066fc0>] kthread+0x7d/0x85
 [<ffffffff8100b924>] kernel_thread_helper+0x4/0x10
 [<ffffffff81463098>] ? restore_args+0x0/0x30
 [<ffffffff81066f43>] ? kthread+0x0/0x85
 [<ffffffff8100b920>] ? kernel_thread_helper+0x0/0x10
Code: fb 74 05 83 fb a3 75 0e 41 ff 85 f4 05 00 00 bb a3 ff ff ff eb 04 85 db 79 0e 49 8b 45 08 31 f6 4c 89 ef ff 50 20 eb 32 8b 76 14 <49> 8b 45 08 66 85 f6 75 0f 31 f6 4c 89 ef bb f3 ff ff ff ff 50
RIP  [<ffffffffa0294f5b>] rpcb_getport_done+0x47/0xab [sunrpc]
 RSP <ffff8800c7133d20>
---[ end trace 56d19572836bccfa ]---

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com