kernel: signal: NULL ptr deref when killing process
Hi all,
While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:
[ 512.602559] BUG: unable to handle kernel NULL pointer dereference at 000000000000001c [ 512.602566] IP: check_kill_permission (kernel/signal.c:746 kernel/signal.c:781) [ 512.602576] PGD 4c10d0067 PUD 4c1738067 PMD 0 [ 512.602584] Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC [ 512.602612] Dumping ftrace buffer: [ 512.602697] (ftrace buffer empty) [ 512.602704] Modules linked in: [ 512.602708] CPU: 18 PID: 8516 Comm: trinity-watchdo Tainted: G B 3.16.0-next-20140815-sasha-00034-g615561b #1071 [ 512.602711] task: ffff8804c1e88000 ti: ffff8804c1290000 task.ti: ffff8804c1290000 [ 512.602718] RIP: check_kill_permission (kernel/signal.c:746 kernel/signal.c:781) [ 512.602720] RSP: 0018:ffff8804c1293e00 EFLAGS: 00010246 [ 512.602722] RAX: 0000000000000000 RBX: ffff8804c1293ec0 RCX: 0000000000000000 [ 512.602723] RDX: ffff880546803000 RSI: ffff8804c1293ec0 RDI: 0000000000000000 [ 512.602726] RBP: ffff8804c1293e28 R08: 0000000000000000 R09: 0000000000000000 [ 512.602728] R10: 0000000000000000 R11: 0000000000000246 R12: ffff880546803000 [ 512.602729] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8804c3fe4200 [ 512.602732] FS: 00007fb2ed5e8700(0000) GS:ffff88071c400000(0000) knlGS:0000000000000000 [ 512.602735] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 512.602737] CR2: 000000000000001c CR3: 00000004c16ff000 CR4: 00000000000006a0 [ 512.602750] DR0: 00000000006f0000 DR1: 0000000000000000 DR2: 0000000000000000 [ 512.602753] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600 [ 512.602754] Stack: [ 512.602761] 00000000000026ce 0000000000000000 ffff8804c1293ec0 ffff880546803000 [ 512.602769] 0000000000000002 ffff8804c1293e60 ffffffff8c17ccb5 ffffffff8c17cc55 [ 512.602777] 00000000000026ce ffff880565319180 0000000000000000 ffff8804c1293ec0 [ 512.602779] Call Trace: [ 512.602786] group_send_sig_info (kernel/signal.c:1300) [ 512.602791] ? group_send_sig_info (kernel/signal.c:1296) [ 512.602795] kill_pid_info (kernel/signal.c:1338) [ 512.602800] ? kill_pid_info (kernel/signal.c:1330) [ 512.602805] SYSC_kill (kernel/signal.c:1423 kernel/signal.c:2896) [ 512.602809] ? SYSC_kill (include/linux/rcupdate.h:814 kernel/signal.c:1422 kernel/signal.c:2896) [ 512.602814] ? _raw_spin_unlock (./arch/x86/include/asm/preempt.h:98 include/linux/spinlock_api_smp.h:152 kernel/locking/spinlock.c:183) [ 512.602820] ? trace_hardirqs_on (kernel/locking/lockdep.c:2609) [ 512.602824] ? syscall_trace_enter (include/linux/context_tracking.h:27 arch/x86/kernel/ptrace.c:1461) [ 512.602828] SyS_kill (kernel/signal.c:2886) [ 512.602833] tracesys (arch/x86/kernel/entry_64.S:541) [ 512.602921] Code: 06 00 4d 8b b4 24 38 0a 00 00 65 48 8b 04 25 c0 da 00 00 4c 8b b8 40 0a 00 00 e8 ab e3 06 00 85 c0 0f 85 8b 00 00 00 41 8b 47 24 <41> 8b 56 1c 39 d0 74 60 41 8b 4e 14 39 c8 74 58 41 8b 47 14 39 All code ======== 0: 06 (bad) 1: 00 4d 8b add %cl,-0x75(%rbp) 4: b4 24 mov $0x24,%ah 6: 38 0a cmp %cl,(%rdx) 8: 00 00 add %al,(%rax) a: 65 48 8b 04 25 c0 da mov %gs:0xdac0,%rax 11: 00 00 13: 4c 8b b8 40 0a 00 00 mov 0xa40(%rax),%r15 1a: e8 ab e3 06 00 callq 0x6e3ca 1f: 85 c0 test %eax,%eax 21: 0f 85 8b 00 00 00 jne 0xb2 27: 41 8b 47 24 mov 0x24(%r15),%eax 2b:* 41 8b 56 1c mov 0x1c(%r14),%edx <-- trapping instruction 2f: 39 d0 cmp %edx,%eax 31: 74 60 je 0x93 33: 41 8b 4e 14 mov 0x14(%r14),%ecx 37: 39 c8 cmp %ecx,%eax 39: 74 58 je 0x93 3b: 41 8b 47 14 mov 0x14(%r15),%eax 3f: 39 00 cmp %eax,(%rax)
Code starting with the faulting instruction
===========================================
0: 41 8b 56 1c mov 0x1c(%r14),%edx 4: 39 d0 cmp %edx,%eax 6: 74 60 je 0x68 8: 41 8b 4e 14 mov 0x14(%r14),%ecx c: 39 c8 cmp %ecx,%eax e: 74 58 je 0x68 10: 41 8b 47 14 mov 0x14(%r15),%eax 14: 39 00 cmp %eax,(%rax) [ 512.602927] RIP check_kill_permission (kernel/signal.c:746 kernel/signal.c:781) [ 512.602929] RSP <ffff8804c1293e00> [ 512.602931] CR2: 000000000000001c
Thanks,
Sasha
Quoted 5 lines · Sasha Levin
Hi all,
While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:
Thanks...
looks like, kill_ok_by_cred()->__task_cred(t) returns NULL at first glance. perhaps you can show the result of "make kernel/signal.s" to be sure? Or at least the full "objdump -d kernel/signal.o".
This looks strange. ->real_cred can be NULL only after __put_task_struct(), so this task_struct is already freed or soon-to-be-freed. But everything looks correct in this path, this task was found by pid_task() under rcu_read_lock().
Quoted 78 lines
[ 512.602559] BUG: unable to handle kernel NULL pointer dereference at 000000000000001c [ 512.602566] IP: check_kill_permission (kernel/signal.c:746 kernel/signal.c:781) [ 512.602576] PGD 4c10d0067 PUD 4c1738067 PMD 0 [ 512.602584] Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC [ 512.602612] Dumping ftrace buffer: [ 512.602697] (ftrace buffer empty) [ 512.602704] Modules linked in: [ 512.602708] CPU: 18 PID: 8516 Comm: trinity-watchdo Tainted: G B 3.16.0-next-20140815-sasha-00034-g615561b #1071 [ 512.602711] task: ffff8804c1e88000 ti: ffff8804c1290000 task.ti: ffff8804c1290000 [ 512.602718] RIP: check_kill_permission (kernel/signal.c:746 kernel/signal.c:781) [ 512.602720] RSP: 0018:ffff8804c1293e00 EFLAGS: 00010246 [ 512.602722] RAX: 0000000000000000 RBX: ffff8804c1293ec0 RCX: 0000000000000000 [ 512.602723] RDX: ffff880546803000 RSI: ffff8804c1293ec0 RDI: 0000000000000000 [ 512.602726] RBP: ffff8804c1293e28 R08: 0000000000000000 R09: 0000000000000000 [ 512.602728] R10: 0000000000000000 R11: 0000000000000246 R12: ffff880546803000 [ 512.602729] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8804c3fe4200 [ 512.602732] FS: 00007fb2ed5e8700(0000) GS:ffff88071c400000(0000) knlGS:0000000000000000 [ 512.602735] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 512.602737] CR2: 000000000000001c CR3: 00000004c16ff000 CR4: 00000000000006a0 [ 512.602750] DR0: 00000000006f0000 DR1: 0000000000000000 DR2: 0000000000000000 [ 512.602753] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600 [ 512.602754] Stack: [ 512.602761] 00000000000026ce 0000000000000000 ffff8804c1293ec0 ffff880546803000 [ 512.602769] 0000000000000002 ffff8804c1293e60 ffffffff8c17ccb5 ffffffff8c17cc55 [ 512.602777] 00000000000026ce ffff880565319180 0000000000000000 ffff8804c1293ec0 [ 512.602779] Call Trace: [ 512.602786] group_send_sig_info (kernel/signal.c:1300) [ 512.602791] ? group_send_sig_info (kernel/signal.c:1296) [ 512.602795] kill_pid_info (kernel/signal.c:1338) [ 512.602800] ? kill_pid_info (kernel/signal.c:1330) [ 512.602805] SYSC_kill (kernel/signal.c:1423 kernel/signal.c:2896) [ 512.602809] ? SYSC_kill (include/linux/rcupdate.h:814 kernel/signal.c:1422 kernel/signal.c:2896) [ 512.602814] ? _raw_spin_unlock (./arch/x86/include/asm/preempt.h:98 include/linux/spinlock_api_smp.h:152 kernel/locking/spinlock.c:183) [ 512.602820] ? trace_hardirqs_on (kernel/locking/lockdep.c:2609) [ 512.602824] ? syscall_trace_enter (include/linux/context_tracking.h:27 arch/x86/kernel/ptrace.c:1461) [ 512.602828] SyS_kill (kernel/signal.c:2886) [ 512.602833] tracesys (arch/x86/kernel/entry_64.S:541) [ 512.602921] Code: 06 00 4d 8b b4 24 38 0a 00 00 65 48 8b 04 25 c0 da 00 00 4c 8b b8 40 0a 00 00 e8 ab e3 06 00 85 c0 0f 85 8b 00 00 00 41 8b 47 24 <41> 8b 56 1c 39 d0 74 60 41 8b 4e 14 39 c8 74 58 41 8b 47 14 39 All code ======== 0: 06 (bad) 1: 00 4d 8b add %cl,-0x75(%rbp) 4: b4 24 mov $0x24,%ah 6: 38 0a cmp %cl,(%rdx) 8: 00 00 add %al,(%rax) a: 65 48 8b 04 25 c0 da mov %gs:0xdac0,%rax 11: 00 00 13: 4c 8b b8 40 0a 00 00 mov 0xa40(%rax),%r15 1a: e8 ab e3 06 00 callq 0x6e3ca 1f: 85 c0 test %eax,%eax 21: 0f 85 8b 00 00 00 jne 0xb2 27: 41 8b 47 24 mov 0x24(%r15),%eax 2b:* 41 8b 56 1c mov 0x1c(%r14),%edx <-- trapping instruction 2f: 39 d0 cmp %edx,%eax 31: 74 60 je 0x93 33: 41 8b 4e 14 mov 0x14(%r14),%ecx 37: 39 c8 cmp %ecx,%eax 39: 74 58 je 0x93 3b: 41 8b 47 14 mov 0x14(%r15),%eax 3f: 39 00 cmp %eax,(%rax)Code starting with the faulting instruction
===========================================0: 41 8b 56 1c mov 0x1c(%r14),%edx 4: 39 d0 cmp %edx,%eax 6: 74 60 je 0x68 8: 41 8b 4e 14 mov 0x14(%r14),%ecx c: 39 c8 cmp %ecx,%eax e: 74 58 je 0x68 10: 41 8b 47 14 mov 0x14(%r15),%eax 14: 39 00 cmp %eax,(%rax) [ 512.602927] RIP check_kill_permission (kernel/signal.c:746 kernel/signal.c:781) [ 512.602929] RSP <ffff8804c1293e00> [ 512.602931] CR2: 000000000000001cThanks,
Sasha
Quoted 10 lines · Oleg Nesterov
Quoted 5 lines · Sasha Levin
Hi all,
While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:
Thanks...
Quoted 7 lines
looks like, kill_ok_by_cred()->__task_cred(t) returns NULL at first glance. perhaps you can show the result of "make
kernel/signal.s" to be sure? Or at least the full "objdump -dkernel/signal.o".This looks strange. ->real_cred can be NULL only after
__put_task_struct(), so thistask_structis already freed or soon-to-be-freed. But everything looks correct in this path, this task was found bypid_task()underrcu_read_lock().
Hmm. I'll recheck and report tomorrow, but it seems that I found a bug:
bad_fork_free_pid()->free_pid() looks wrong. This _can_ explain this trace afaics, but only in unlikely case...
David, any other reason why ->real_cred can be NULL ? (assuming that I interpret this asm correctly).
Quoted 78 lines
Quoted 78 lines
[ 512.602559] BUG: unable to handle kernel NULL pointer dereference at 000000000000001c [ 512.602566] IP: check_kill_permission (kernel/signal.c:746 kernel/signal.c:781) [ 512.602576] PGD 4c10d0067 PUD 4c1738067 PMD 0 [ 512.602584] Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC [ 512.602612] Dumping ftrace buffer: [ 512.602697] (ftrace buffer empty) [ 512.602704] Modules linked in: [ 512.602708] CPU: 18 PID: 8516 Comm: trinity-watchdo Tainted: G B 3.16.0-next-20140815-sasha-00034-g615561b #1071 [ 512.602711] task: ffff8804c1e88000 ti: ffff8804c1290000 task.ti: ffff8804c1290000 [ 512.602718] RIP: check_kill_permission (kernel/signal.c:746 kernel/signal.c:781) [ 512.602720] RSP: 0018:ffff8804c1293e00 EFLAGS: 00010246 [ 512.602722] RAX: 0000000000000000 RBX: ffff8804c1293ec0 RCX: 0000000000000000 [ 512.602723] RDX: ffff880546803000 RSI: ffff8804c1293ec0 RDI: 0000000000000000 [ 512.602726] RBP: ffff8804c1293e28 R08: 0000000000000000 R09: 0000000000000000 [ 512.602728] R10: 0000000000000000 R11: 0000000000000246 R12: ffff880546803000 [ 512.602729] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8804c3fe4200 [ 512.602732] FS: 00007fb2ed5e8700(0000) GS:ffff88071c400000(0000) knlGS:0000000000000000 [ 512.602735] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 512.602737] CR2: 000000000000001c CR3: 00000004c16ff000 CR4: 00000000000006a0 [ 512.602750] DR0: 00000000006f0000 DR1: 0000000000000000 DR2: 0000000000000000 [ 512.602753] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600 [ 512.602754] Stack: [ 512.602761] 00000000000026ce 0000000000000000 ffff8804c1293ec0 ffff880546803000 [ 512.602769] 0000000000000002 ffff8804c1293e60 ffffffff8c17ccb5 ffffffff8c17cc55 [ 512.602777] 00000000000026ce ffff880565319180 0000000000000000 ffff8804c1293ec0 [ 512.602779] Call Trace: [ 512.602786] group_send_sig_info (kernel/signal.c:1300) [ 512.602791] ? group_send_sig_info (kernel/signal.c:1296) [ 512.602795] kill_pid_info (kernel/signal.c:1338) [ 512.602800] ? kill_pid_info (kernel/signal.c:1330) [ 512.602805] SYSC_kill (kernel/signal.c:1423 kernel/signal.c:2896) [ 512.602809] ? SYSC_kill (include/linux/rcupdate.h:814 kernel/signal.c:1422 kernel/signal.c:2896) [ 512.602814] ? _raw_spin_unlock (./arch/x86/include/asm/preempt.h:98 include/linux/spinlock_api_smp.h:152 kernel/locking/spinlock.c:183) [ 512.602820] ? trace_hardirqs_on (kernel/locking/lockdep.c:2609) [ 512.602824] ? syscall_trace_enter (include/linux/context_tracking.h:27 arch/x86/kernel/ptrace.c:1461) [ 512.602828] SyS_kill (kernel/signal.c:2886) [ 512.602833] tracesys (arch/x86/kernel/entry_64.S:541) [ 512.602921] Code: 06 00 4d 8b b4 24 38 0a 00 00 65 48 8b 04 25 c0 da 00 00 4c 8b b8 40 0a 00 00 e8 ab e3 06 00 85 c0 0f 85 8b 00 00 00 41 8b 47 24 <41> 8b 56 1c 39 d0 74 60 41 8b 4e 14 39 c8 74 58 41 8b 47 14 39 All code ======== 0: 06 (bad) 1: 00 4d 8b add %cl,-0x75(%rbp) 4: b4 24 mov $0x24,%ah 6: 38 0a cmp %cl,(%rdx) 8: 00 00 add %al,(%rax) a: 65 48 8b 04 25 c0 da mov %gs:0xdac0,%rax 11: 00 00 13: 4c 8b b8 40 0a 00 00 mov 0xa40(%rax),%r15 1a: e8 ab e3 06 00 callq 0x6e3ca 1f: 85 c0 test %eax,%eax 21: 0f 85 8b 00 00 00 jne 0xb2 27: 41 8b 47 24 mov 0x24(%r15),%eax 2b:* 41 8b 56 1c mov 0x1c(%r14),%edx <-- trapping instruction 2f: 39 d0 cmp %edx,%eax 31: 74 60 je 0x93 33: 41 8b 4e 14 mov 0x14(%r14),%ecx 37: 39 c8 cmp %ecx,%eax 39: 74 58 je 0x93 3b: 41 8b 47 14 mov 0x14(%r15),%eax 3f: 39 00 cmp %eax,(%rax)Code starting with the faulting instruction
===========================================0: 41 8b 56 1c mov 0x1c(%r14),%edx 4: 39 d0 cmp %edx,%eax 6: 74 60 je 0x68 8: 41 8b 4e 14 mov 0x14(%r14),%ecx c: 39 c8 cmp %ecx,%eax e: 74 58 je 0x68 10: 41 8b 47 14 mov 0x14(%r15),%eax 14: 39 00 cmp %eax,(%rax) [ 512.602927] RIP check_kill_permission (kernel/signal.c:746 kernel/signal.c:781) [ 512.602929] RSP <ffff8804c1293e00> [ 512.602931] CR2: 000000000000001cThanks,
Sasha
Quoted 3 lines · Oleg Nesterov
Hmm. I'll recheck and report tomorrow, but it seems that I found a bug:
bad_fork_free_pid()->free_pid() looks wrong.
No, sorry for noise, I was wrong.
Quoted 2 lines
David, any other reason why ->real_cred can be NULL ? (assuming that I interpret this asm correctly).
Oleg.
Quoted 2 lines · Oleg Nesterov
David, any other reason why ->real_cred can be NULL ? (assuming that I interpret this asm correctly).
It should only be possible to see ->real_cred as being NULL after exit_creds() has been called from __put_task_struct() for a task that has finished construction.
It shouldn't be possible to introduce a NULL pointer through commit_creds() or override_creds() since both of those should crash immediately if given one, but it's possible revert_creds() could be so used.
Is there a race between kill() and exit() brought on by the kill path only using the RCU read lock? This doesn't prevent ->real_cred from being modified, but it looks like this should, in combination with delayed_put_task_struct(), prevent it from being cleared.
David
Quoted 9 lines · David Howells
Quoted 2 lines · Oleg Nesterov
David, any other reason why ->real_cred can be NULL ? (assuming that I interpret this asm correctly).
It should only be possible to see ->real_cred as being NULL after
exit_creds()has been called from__put_task_struct()for a task that has finished construction.
Yes, thanks, this was my understanding...
Quoted 4 lines
Is there a race between
kill()andexit()brought on by the kill path only using the RCU read lock? This doesn't prevent ->real_cred from being modified, but it looks like this should, in combination withdelayed_put_task_struct(), prevent it from being cleared.
Yes, rcu should protect us from both delayed_put_pid() and delayed_put_task(). Everything looks correct... And there are a lot of other similar users of find_vpid/find_task_by_vpid/pid_task/etc under rcu, I can't recall any bug in this area.
I am puzzled. Note also that ->signal == NULL. Will try to think more, but so far I have no any idea.
Oleg.
Quoted 11 lines · Oleg Nesterov
Quoted 4 lines
Is there a race between
kill()andexit()brought on by the kill path onlyQuoted 3 lines
using the RCU read lock? This doesn't prevent ->real_cred from being modified, but it looks like this should, in combination with
delayed_put_task_struct(), prevent it from being cleared.Yes, rcu should protect us from both
delayed_put_pid()anddelayed_put_task(). Everything looks correct... And there are a lot of other similar users of find_vpid/find_task_by_vpid/pid_task/etc under rcu, I can't recall any bug in this area.I am puzzled. Note also that ->signal == NULL. Will try to think more, but so far I have no any idea.
I've hit something similar earlier today, and it might be related:
[ 973.452840] BUG: unable to handle kernel NULL pointer dereference at 00000000000002b0 [ 973.455347] IP: flush_sigqueue_mask (include/linux/signal.h:118 kernel/signal.c:715) [ 973.457526] PGD 4dfdc7067 PUD 5f77d9067 PMD 0 [ 973.459216] Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC [ 973.460086] Dumping ftrace buffer: [ 973.460086] (ftrace buffer empty) [ 973.460086] Modules linked in: [ 973.460086] CPU: 4 PID: 13145 Comm: trinity-c767 Not tainted 3.17.0-rc2-next-20140826-sasha-00031-gc48c9ac-dirty #1079 [ 973.460086] task: ffff880604800000 ti: ffff880586648000 task.ti: ffff880586648000 [ 973.460086] RIP: flush_sigqueue_mask (include/linux/signal.h:118 kernel/signal.c:715) [ 973.460086] RSP: 0018:ffff88058664bec8 EFLAGS: 00010046 [ 973.460086] RAX: 0000000000000000 RBX: fffffffffffff730 RCX: 0000000000010000 [ 973.460086] RDX: 0000000000000000 RSI: 00000000000002a0 RDI: ffff88058664bed8 [ 973.460086] RBP: ffff88058664bf10 R08: 0000000000000001 R09: 0000000000000001 [ 973.460086] R10: 000000000002d201 R11: 0000000000000254 R12: 0000000000000000 [ 973.460086] R13: ffff88058664bf40 R14: ffff880604800000 R15: 0000000000000010 [ 973.460086] FS: 00007fe3a3045700(0000) GS:ffff880277c00000(0000) knlGS:0000000000000000 [ 973.460086] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 973.460086] CR2: 00000000000002b0 CR3: 00000004e23d5000 CR4: 00000000000006a0 [ 973.460086] Stack: [ 973.460086] ffffffffac183690 01017fffb3247180 0000000000010000 0000000000000000 [ 973.460086] 00007fffb3247180 00007fffb3247220 0000000000000011 0000000000000000 [ 973.460086] 0000000000000000 ffff88058664bf78 ffffffffac183ef5 0000000000000000 [ 973.460086] Call Trace: [ 973.460086] ? do_sigaction (kernel/signal.c:3124 (discriminator 17)) [ 973.460086] SyS_rt_sigaction (kernel/signal.c:3360 kernel/signal.c:3341) [ 973.460086] tracesys (arch/x86/kernel/entry_64.S:542) [ 973.460086] Code: b7 49 09 d5 4d 89 6e 10 48 83 c4 08 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 48 8b 0f 31 c0 <48> 8b 56 10 48 85 ca 74 7b 55 48 f7 d1 48 89 e5 41 56 48 21 ca All code ======== 0: b7 49 mov $0x49,%bh 2: 09 d5 or %edx,%ebp 4: 4d 89 6e 10 mov %r13,0x10(%r14) 8: 48 83 c4 08 add $0x8,%rsp c: 5b pop %rbx d: 41 5c pop %r12 f: 41 5d pop %r13 11: 41 5e pop %r14 13: 41 5f pop %r15 15: 5d pop %rbp 16: c3 retq 17: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1) 1e: 00 00 00 21: 66 66 66 66 90 data32 data32 data32 xchg %ax,%ax 26: 48 8b 0f mov (%rdi),%rcx 29: 31 c0 xor %eax,%eax 2b:* 48 8b 56 10 mov 0x10(%rsi),%rdx <-- trapping instruction 2f: 48 85 ca test %rcx,%rdx 32: 74 7b je 0xaf 34: 55 push %rbp 35: 48 f7 d1 not %rcx 38: 48 89 e5 mov %rsp,%rbp 3b: 41 56 push %r14 3d: 48 21 ca and %rcx,%rdx ...
Code starting with the faulting instruction
===========================================
0: 48 8b 56 10 mov 0x10(%rsi),%rdx 4: 48 85 ca test %rcx,%rdx 7: 74 7b je 0x84 9: 55 push %rbp a: 48 f7 d1 not %rcx d: 48 89 e5 mov %rsp,%rbp 10: 41 56 push %r14 12: 48 21 ca and %rcx,%rdx ... [ 973.460086] RIP flush_sigqueue_mask (include/linux/signal.h:118 kernel/signal.c:715) [ 973.460086] RSP <ffff88058664bec8> [ 973.460086] CR2: 00000000000002b0
Thanks,
Sasha
Quoted 11 lines · Sasha Levin
Quoted 9 lines · Oleg Nesterov
Quoted 4 lines
Is there a race between
kill()andexit()brought on by the kill path onlyQuoted 3 lines
using the RCU read lock? This doesn't prevent ->real_cred from being modified, but it looks like this should, in combination with
delayed_put_task_struct(), prevent it from being cleared.Yes, rcu should protect us from both
delayed_put_pid()anddelayed_put_task(). Everything looks correct... And there are a lot of other similar users of find_vpid/find_task_by_vpid/pid_task/etc under rcu, I can't recall any bug in this area.
Quoted 4 lines
Quoted 2 lines
I am puzzled. Note also that ->signal == NULL. Will try to think more, but so far I have no any idea.
I've hit something similar earlier today, and it might be related:
Thanks.
rsi == ¤t->signal->shared_pending == 0x2a0, so current->signal == 544.
Given that this task is current, we can rule out RCU/find_pind/etc problems, this task_struct and its ->signal must be stable.
Looks like, something was broken recently. I do not remember the bug reports like this. Say, an unbalanced put_task_struct()... Hmm.
Quoted 74 lines
[ 973.452840] BUG: unable to handle kernel NULL pointer dereference at 00000000000002b0 [ 973.455347] IP: flush_sigqueue_mask (include/linux/signal.h:118 kernel/signal.c:715) [ 973.457526] PGD 4dfdc7067 PUD 5f77d9067 PMD 0 [ 973.459216] Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC [ 973.460086] Dumping ftrace buffer: [ 973.460086] (ftrace buffer empty) [ 973.460086] Modules linked in: [ 973.460086] CPU: 4 PID: 13145 Comm: trinity-c767 Not tainted 3.17.0-rc2-next-20140826-sasha-00031-gc48c9ac-dirty #1079 [ 973.460086] task: ffff880604800000 ti: ffff880586648000 task.ti: ffff880586648000 [ 973.460086] RIP: flush_sigqueue_mask (include/linux/signal.h:118 kernel/signal.c:715) [ 973.460086] RSP: 0018:ffff88058664bec8 EFLAGS: 00010046 [ 973.460086] RAX: 0000000000000000 RBX: fffffffffffff730 RCX: 0000000000010000 [ 973.460086] RDX: 0000000000000000 RSI: 00000000000002a0 RDI: ffff88058664bed8 [ 973.460086] RBP: ffff88058664bf10 R08: 0000000000000001 R09: 0000000000000001 [ 973.460086] R10: 000000000002d201 R11: 0000000000000254 R12: 0000000000000000 [ 973.460086] R13: ffff88058664bf40 R14: ffff880604800000 R15: 0000000000000010 [ 973.460086] FS: 00007fe3a3045700(0000) GS:ffff880277c00000(0000) knlGS:0000000000000000 [ 973.460086] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 973.460086] CR2: 00000000000002b0 CR3: 00000004e23d5000 CR4: 00000000000006a0 [ 973.460086] Stack: [ 973.460086] ffffffffac183690 01017fffb3247180 0000000000010000 0000000000000000 [ 973.460086] 00007fffb3247180 00007fffb3247220 0000000000000011 0000000000000000 [ 973.460086] 0000000000000000 ffff88058664bf78 ffffffffac183ef5 0000000000000000 [ 973.460086] Call Trace: [ 973.460086] ? do_sigaction (kernel/signal.c:3124 (discriminator 17)) [ 973.460086] SyS_rt_sigaction (kernel/signal.c:3360 kernel/signal.c:3341) [ 973.460086] tracesys (arch/x86/kernel/entry_64.S:542) [ 973.460086] Code: b7 49 09 d5 4d 89 6e 10 48 83 c4 08 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 48 8b 0f 31 c0 <48> 8b 56 10 48 85 ca 74 7b 55 48 f7 d1 48 89 e5 41 56 48 21 ca All code ======== 0: b7 49 mov $0x49,%bh 2: 09 d5 or %edx,%ebp 4: 4d 89 6e 10 mov %r13,0x10(%r14) 8: 48 83 c4 08 add $0x8,%rsp c: 5b pop %rbx d: 41 5c pop %r12 f: 41 5d pop %r13 11: 41 5e pop %r14 13: 41 5f pop %r15 15: 5d pop %rbp 16: c3 retq 17: 66 2e 0f 1f 84 00 00 nopw %cs:0x0(%rax,%rax,1) 1e: 00 00 00 21: 66 66 66 66 90 data32 data32 data32 xchg %ax,%ax 26: 48 8b 0f mov (%rdi),%rcx 29: 31 c0 xor %eax,%eax 2b:* 48 8b 56 10 mov 0x10(%rsi),%rdx <-- trapping instruction 2f: 48 85 ca test %rcx,%rdx 32: 74 7b je 0xaf 34: 55 push %rbp 35: 48 f7 d1 not %rcx 38: 48 89 e5 mov %rsp,%rbp 3b: 41 56 push %r14 3d: 48 21 ca and %rcx,%rdx ...Code starting with the faulting instruction
===========================================0: 48 8b 56 10 mov 0x10(%rsi),%rdx 4: 48 85 ca test %rcx,%rdx 7: 74 7b je 0x84 9: 55 push %rbp a: 48 f7 d1 not %rcx d: 48 89 e5 mov %rsp,%rbp 10: 41 56 push %r14 12: 48 21 ca and %rcx,%rdx ... [ 973.460086] RIP flush_sigqueue_mask (include/linux/signal.h:118 kernel/signal.c:715) [ 973.460086] RSP <ffff88058664bec8> [ 973.460086] CR2: 00000000000002b0Thanks,
Sasha
Quoted 11 lines · Oleg Nesterov
Quoted 5 lines · Sasha Levin
Quoted 5 lines
Hi all,
While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:
Thanks...
looks like, kill_ok_by_cred()->__task_cred(t) returns NULL at first glance. perhaps you can show the result of "make
kernel/signal.s" to be sure? Or at least the full "objdump -dkernel/signal.o".
Attached.
Thanks,
Sasha
Quoted 8 lines · Sasha Levin
Quoted 6 lines · Oleg Nesterov
Quoted 5 lines · Sasha Levin
Quoted 5 lines
Hi all,
While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:
Quoted 7 lines
Quoted 5 lines
Thanks...
looks like, kill_ok_by_cred()->__task_cred(t) returns NULL at first glance. perhaps you can show the result of "make
kernel/signal.s" to be sure? Or at least the full "objdump -dkernel/signal.o".Attached.
Thanks.
Yes, t->real_cred == NULL (r14).
Interestingly, t->signal is NULL too (rcx). And ->signal must be never NULL. So it looks like this task_struct was reallocated/reused. Or corrupted.
t == 0xffff880546803000 (r12), this doesn't look wrong.
Oleg.