Skip to contentSkip to conversation

kernel: signal: NULL ptr deref when killing process

DISCUSSION Sasha Levin 3 participants 10 messages Aug 20, 2014 – Aug 27, 2014

lore
Share & tools
lore
#0 lore raw

Hi all,

While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:

[  512.602559] BUG: unable to handle kernel NULL pointer dereference at 000000000000001c
[  512.602566] IP: check_kill_permission (kernel/signal.c:746 kernel/signal.c:781)
[  512.602576] PGD 4c10d0067 PUD 4c1738067 PMD 0
[  512.602584] Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[  512.602612] Dumping ftrace buffer:
[  512.602697]    (ftrace buffer empty)
[  512.602704] Modules linked in:
[  512.602708] CPU: 18 PID: 8516 Comm: trinity-watchdo Tainted: G    B          3.16.0-next-20140815-sasha-00034-g615561b #1071
[  512.602711] task: ffff8804c1e88000 ti: ffff8804c1290000 task.ti: ffff8804c1290000
[  512.602718] RIP: check_kill_permission (kernel/signal.c:746 kernel/signal.c:781)
[  512.602720] RSP: 0018:ffff8804c1293e00  EFLAGS: 00010246
[  512.602722] RAX: 0000000000000000 RBX: ffff8804c1293ec0 RCX: 0000000000000000
[  512.602723] RDX: ffff880546803000 RSI: ffff8804c1293ec0 RDI: 0000000000000000
[  512.602726] RBP: ffff8804c1293e28 R08: 0000000000000000 R09: 0000000000000000
[  512.602728] R10: 0000000000000000 R11: 0000000000000246 R12: ffff880546803000
[  512.602729] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8804c3fe4200
[  512.602732] FS:  00007fb2ed5e8700(0000) GS:ffff88071c400000(0000) knlGS:0000000000000000
[  512.602735] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  512.602737] CR2: 000000000000001c CR3: 00000004c16ff000 CR4: 00000000000006a0
[  512.602750] DR0: 00000000006f0000 DR1: 0000000000000000 DR2: 0000000000000000
[  512.602753] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
[  512.602754] Stack:
[  512.602761]  00000000000026ce 0000000000000000 ffff8804c1293ec0 ffff880546803000
[  512.602769]  0000000000000002 ffff8804c1293e60 ffffffff8c17ccb5 ffffffff8c17cc55
[  512.602777]  00000000000026ce ffff880565319180 0000000000000000 ffff8804c1293ec0
[  512.602779] Call Trace:
[  512.602786] group_send_sig_info (kernel/signal.c:1300)
[  512.602791] ? group_send_sig_info (kernel/signal.c:1296)
[  512.602795] kill_pid_info (kernel/signal.c:1338)
[  512.602800] ? kill_pid_info (kernel/signal.c:1330)
[  512.602805] SYSC_kill (kernel/signal.c:1423 kernel/signal.c:2896)
[  512.602809] ? SYSC_kill (include/linux/rcupdate.h:814 kernel/signal.c:1422 kernel/signal.c:2896)
[  512.602814] ? _raw_spin_unlock (./arch/x86/include/asm/preempt.h:98 include/linux/spinlock_api_smp.h:152 kernel/locking/spinlock.c:183)
[  512.602820] ? trace_hardirqs_on (kernel/locking/lockdep.c:2609)
[  512.602824] ? syscall_trace_enter (include/linux/context_tracking.h:27 arch/x86/kernel/ptrace.c:1461)
[  512.602828] SyS_kill (kernel/signal.c:2886)
[  512.602833] tracesys (arch/x86/kernel/entry_64.S:541)
[ 512.602921] Code: 06 00 4d 8b b4 24 38 0a 00 00 65 48 8b 04 25 c0 da 00 00 4c 8b b8 40 0a 00 00 e8 ab e3 06 00 85 c0 0f 85 8b 00 00 00 41 8b 47 24 <41> 8b 56 1c 39 d0 74 60 41 8b 4e 14 39 c8 74 58 41 8b 47 14 39
All code
========
   0:	06                   	(bad)
   1:	00 4d 8b             	add    %cl,-0x75(%rbp)
   4:	b4 24                	mov    $0x24,%ah
   6:	38 0a                	cmp    %cl,(%rdx)
   8:	00 00                	add    %al,(%rax)
   a:	65 48 8b 04 25 c0 da 	mov    %gs:0xdac0,%rax
  11:	00 00
  13:	4c 8b b8 40 0a 00 00 	mov    0xa40(%rax),%r15
  1a:	e8 ab e3 06 00       	callq  0x6e3ca
  1f:	85 c0                	test   %eax,%eax
  21:	0f 85 8b 00 00 00    	jne    0xb2
  27:	41 8b 47 24          	mov    0x24(%r15),%eax
  2b:*	41 8b 56 1c          	mov    0x1c(%r14),%edx		<-- trapping instruction
  2f:	39 d0                	cmp    %edx,%eax
  31:	74 60                	je     0x93
  33:	41 8b 4e 14          	mov    0x14(%r14),%ecx
  37:	39 c8                	cmp    %ecx,%eax
  39:	74 58                	je     0x93
  3b:	41 8b 47 14          	mov    0x14(%r15),%eax
  3f:	39 00                	cmp    %eax,(%rax)

Code starting with the faulting instruction
===========================================

   0:	41 8b 56 1c          	mov    0x1c(%r14),%edx
   4:	39 d0                	cmp    %edx,%eax
   6:	74 60                	je     0x68
   8:	41 8b 4e 14          	mov    0x14(%r14),%ecx
   c:	39 c8                	cmp    %ecx,%eax
   e:	74 58                	je     0x68
  10:	41 8b 47 14          	mov    0x14(%r15),%eax
  14:	39 00                	cmp    %eax,(%rax)
[  512.602927] RIP check_kill_permission (kernel/signal.c:746 kernel/signal.c:781)
[  512.602929]  RSP <ffff8804c1293e00>
[  512.602931] CR2: 000000000000001c

Thanks,
Sasha

#1 lore raw
Quoted 5 lines · Sasha Levin

Hi all,

While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:

Thanks...

looks like, kill_ok_by_cred()->__task_cred(t) returns NULL at first glance. perhaps you can show the result of "make kernel/signal.s" to be sure? Or at least the full "objdump -d kernel/signal.o".

This looks strange. ->real_cred can be NULL only after __put_task_struct(), so this task_struct is already freed or soon-to-be-freed. But everything looks correct in this path, this task was found by pid_task() under rcu_read_lock().

Quoted 78 lines
[  512.602559] BUG: unable to handle kernel NULL pointer dereference at 000000000000001c
[  512.602566] IP: check_kill_permission (kernel/signal.c:746 kernel/signal.c:781)
[  512.602576] PGD 4c10d0067 PUD 4c1738067 PMD 0
[  512.602584] Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[  512.602612] Dumping ftrace buffer:
[  512.602697]    (ftrace buffer empty)
[  512.602704] Modules linked in:
[  512.602708] CPU: 18 PID: 8516 Comm: trinity-watchdo Tainted: G    B          3.16.0-next-20140815-sasha-00034-g615561b #1071
[  512.602711] task: ffff8804c1e88000 ti: ffff8804c1290000 task.ti: ffff8804c1290000
[  512.602718] RIP: check_kill_permission (kernel/signal.c:746 kernel/signal.c:781)
[  512.602720] RSP: 0018:ffff8804c1293e00  EFLAGS: 00010246
[  512.602722] RAX: 0000000000000000 RBX: ffff8804c1293ec0 RCX: 0000000000000000
[  512.602723] RDX: ffff880546803000 RSI: ffff8804c1293ec0 RDI: 0000000000000000
[  512.602726] RBP: ffff8804c1293e28 R08: 0000000000000000 R09: 0000000000000000
[  512.602728] R10: 0000000000000000 R11: 0000000000000246 R12: ffff880546803000
[  512.602729] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8804c3fe4200
[  512.602732] FS:  00007fb2ed5e8700(0000) GS:ffff88071c400000(0000) knlGS:0000000000000000
[  512.602735] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  512.602737] CR2: 000000000000001c CR3: 00000004c16ff000 CR4: 00000000000006a0
[  512.602750] DR0: 00000000006f0000 DR1: 0000000000000000 DR2: 0000000000000000
[  512.602753] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
[  512.602754] Stack:
[  512.602761]  00000000000026ce 0000000000000000 ffff8804c1293ec0 ffff880546803000
[  512.602769]  0000000000000002 ffff8804c1293e60 ffffffff8c17ccb5 ffffffff8c17cc55
[  512.602777]  00000000000026ce ffff880565319180 0000000000000000 ffff8804c1293ec0
[  512.602779] Call Trace:
[  512.602786] group_send_sig_info (kernel/signal.c:1300)
[  512.602791] ? group_send_sig_info (kernel/signal.c:1296)
[  512.602795] kill_pid_info (kernel/signal.c:1338)
[  512.602800] ? kill_pid_info (kernel/signal.c:1330)
[  512.602805] SYSC_kill (kernel/signal.c:1423 kernel/signal.c:2896)
[  512.602809] ? SYSC_kill (include/linux/rcupdate.h:814 kernel/signal.c:1422 kernel/signal.c:2896)
[  512.602814] ? _raw_spin_unlock (./arch/x86/include/asm/preempt.h:98 include/linux/spinlock_api_smp.h:152 kernel/locking/spinlock.c:183)
[  512.602820] ? trace_hardirqs_on (kernel/locking/lockdep.c:2609)
[  512.602824] ? syscall_trace_enter (include/linux/context_tracking.h:27 arch/x86/kernel/ptrace.c:1461)
[  512.602828] SyS_kill (kernel/signal.c:2886)
[  512.602833] tracesys (arch/x86/kernel/entry_64.S:541)
[ 512.602921] Code: 06 00 4d 8b b4 24 38 0a 00 00 65 48 8b 04 25 c0 da 00 00 4c 8b b8 40 0a 00 00 e8 ab e3 06 00 85 c0 0f 85 8b 00 00 00 41 8b 47 24 <41> 8b 56 1c 39 d0 74 60 41 8b 4e 14 39 c8 74 58 41 8b 47 14 39
All code
========
   0:	06                   	(bad)
   1:	00 4d 8b             	add    %cl,-0x75(%rbp)
   4:	b4 24                	mov    $0x24,%ah
   6:	38 0a                	cmp    %cl,(%rdx)
   8:	00 00                	add    %al,(%rax)
   a:	65 48 8b 04 25 c0 da 	mov    %gs:0xdac0,%rax
  11:	00 00
  13:	4c 8b b8 40 0a 00 00 	mov    0xa40(%rax),%r15
  1a:	e8 ab e3 06 00       	callq  0x6e3ca
  1f:	85 c0                	test   %eax,%eax
  21:	0f 85 8b 00 00 00    	jne    0xb2
  27:	41 8b 47 24          	mov    0x24(%r15),%eax
  2b:*	41 8b 56 1c          	mov    0x1c(%r14),%edx		<-- trapping instruction
  2f:	39 d0                	cmp    %edx,%eax
  31:	74 60                	je     0x93
  33:	41 8b 4e 14          	mov    0x14(%r14),%ecx
  37:	39 c8                	cmp    %ecx,%eax
  39:	74 58                	je     0x93
  3b:	41 8b 47 14          	mov    0x14(%r15),%eax
  3f:	39 00                	cmp    %eax,(%rax)

Code starting with the faulting instruction
===========================================

   0:	41 8b 56 1c          	mov    0x1c(%r14),%edx
   4:	39 d0                	cmp    %edx,%eax
   6:	74 60                	je     0x68
   8:	41 8b 4e 14          	mov    0x14(%r14),%ecx
   c:	39 c8                	cmp    %ecx,%eax
   e:	74 58                	je     0x68
  10:	41 8b 47 14          	mov    0x14(%r15),%eax
  14:	39 00                	cmp    %eax,(%rax)
[  512.602927] RIP check_kill_permission (kernel/signal.c:746 kernel/signal.c:781)
[  512.602929]  RSP <ffff8804c1293e00>
[  512.602931] CR2: 000000000000001c

Thanks,
Sasha

#2 lore raw
Quoted 10 lines · Oleg Nesterov
Quoted 5 lines · Sasha Levin

Hi all,

While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:

Thanks...

Quoted 7 lines

looks like, kill_ok_by_cred()->__task_cred(t) returns NULL at first glance. perhaps you can show the result of "make kernel/signal.s" to be sure? Or at least the full "objdump -d kernel/signal.o".

This looks strange. ->real_cred can be NULL only after __put_task_struct(), so this task_struct is already freed or soon-to-be-freed. But everything looks correct in this path, this task was found by pid_task() under rcu_read_lock().

Hmm. I'll recheck and report tomorrow, but it seems that I found a bug:
bad_fork_free_pid()->free_pid() looks wrong. This _can_ explain this trace afaics, but only in unlikely case...

David, any other reason why ->real_cred can be NULL ? (assuming that I interpret this asm correctly).

Quoted 78 lines
Quoted 78 lines
[  512.602559] BUG: unable to handle kernel NULL pointer dereference at 000000000000001c
[  512.602566] IP: check_kill_permission (kernel/signal.c:746 kernel/signal.c:781)
[  512.602576] PGD 4c10d0067 PUD 4c1738067 PMD 0
[  512.602584] Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[  512.602612] Dumping ftrace buffer:
[  512.602697]    (ftrace buffer empty)
[  512.602704] Modules linked in:
[  512.602708] CPU: 18 PID: 8516 Comm: trinity-watchdo Tainted: G    B          3.16.0-next-20140815-sasha-00034-g615561b #1071
[  512.602711] task: ffff8804c1e88000 ti: ffff8804c1290000 task.ti: ffff8804c1290000
[  512.602718] RIP: check_kill_permission (kernel/signal.c:746 kernel/signal.c:781)
[  512.602720] RSP: 0018:ffff8804c1293e00  EFLAGS: 00010246
[  512.602722] RAX: 0000000000000000 RBX: ffff8804c1293ec0 RCX: 0000000000000000
[  512.602723] RDX: ffff880546803000 RSI: ffff8804c1293ec0 RDI: 0000000000000000
[  512.602726] RBP: ffff8804c1293e28 R08: 0000000000000000 R09: 0000000000000000
[  512.602728] R10: 0000000000000000 R11: 0000000000000246 R12: ffff880546803000
[  512.602729] R13: 0000000000000000 R14: 0000000000000000 R15: ffff8804c3fe4200
[  512.602732] FS:  00007fb2ed5e8700(0000) GS:ffff88071c400000(0000) knlGS:0000000000000000
[  512.602735] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  512.602737] CR2: 000000000000001c CR3: 00000004c16ff000 CR4: 00000000000006a0
[  512.602750] DR0: 00000000006f0000 DR1: 0000000000000000 DR2: 0000000000000000
[  512.602753] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
[  512.602754] Stack:
[  512.602761]  00000000000026ce 0000000000000000 ffff8804c1293ec0 ffff880546803000
[  512.602769]  0000000000000002 ffff8804c1293e60 ffffffff8c17ccb5 ffffffff8c17cc55
[  512.602777]  00000000000026ce ffff880565319180 0000000000000000 ffff8804c1293ec0
[  512.602779] Call Trace:
[  512.602786] group_send_sig_info (kernel/signal.c:1300)
[  512.602791] ? group_send_sig_info (kernel/signal.c:1296)
[  512.602795] kill_pid_info (kernel/signal.c:1338)
[  512.602800] ? kill_pid_info (kernel/signal.c:1330)
[  512.602805] SYSC_kill (kernel/signal.c:1423 kernel/signal.c:2896)
[  512.602809] ? SYSC_kill (include/linux/rcupdate.h:814 kernel/signal.c:1422 kernel/signal.c:2896)
[  512.602814] ? _raw_spin_unlock (./arch/x86/include/asm/preempt.h:98 include/linux/spinlock_api_smp.h:152 kernel/locking/spinlock.c:183)
[  512.602820] ? trace_hardirqs_on (kernel/locking/lockdep.c:2609)
[  512.602824] ? syscall_trace_enter (include/linux/context_tracking.h:27 arch/x86/kernel/ptrace.c:1461)
[  512.602828] SyS_kill (kernel/signal.c:2886)
[  512.602833] tracesys (arch/x86/kernel/entry_64.S:541)
[ 512.602921] Code: 06 00 4d 8b b4 24 38 0a 00 00 65 48 8b 04 25 c0 da 00 00 4c 8b b8 40 0a 00 00 e8 ab e3 06 00 85 c0 0f 85 8b 00 00 00 41 8b 47 24 <41> 8b 56 1c 39 d0 74 60 41 8b 4e 14 39 c8 74 58 41 8b 47 14 39
All code
========
   0:	06                   	(bad)
   1:	00 4d 8b             	add    %cl,-0x75(%rbp)
   4:	b4 24                	mov    $0x24,%ah
   6:	38 0a                	cmp    %cl,(%rdx)
   8:	00 00                	add    %al,(%rax)
   a:	65 48 8b 04 25 c0 da 	mov    %gs:0xdac0,%rax
  11:	00 00
  13:	4c 8b b8 40 0a 00 00 	mov    0xa40(%rax),%r15
  1a:	e8 ab e3 06 00       	callq  0x6e3ca
  1f:	85 c0                	test   %eax,%eax
  21:	0f 85 8b 00 00 00    	jne    0xb2
  27:	41 8b 47 24          	mov    0x24(%r15),%eax
  2b:*	41 8b 56 1c          	mov    0x1c(%r14),%edx		<-- trapping instruction
  2f:	39 d0                	cmp    %edx,%eax
  31:	74 60                	je     0x93
  33:	41 8b 4e 14          	mov    0x14(%r14),%ecx
  37:	39 c8                	cmp    %ecx,%eax
  39:	74 58                	je     0x93
  3b:	41 8b 47 14          	mov    0x14(%r15),%eax
  3f:	39 00                	cmp    %eax,(%rax)

Code starting with the faulting instruction
===========================================

   0:	41 8b 56 1c          	mov    0x1c(%r14),%edx
   4:	39 d0                	cmp    %edx,%eax
   6:	74 60                	je     0x68
   8:	41 8b 4e 14          	mov    0x14(%r14),%ecx
   c:	39 c8                	cmp    %ecx,%eax
   e:	74 58                	je     0x68
  10:	41 8b 47 14          	mov    0x14(%r15),%eax
  14:	39 00                	cmp    %eax,(%rax)
[  512.602927] RIP check_kill_permission (kernel/signal.c:746 kernel/signal.c:781)
[  512.602929]  RSP <ffff8804c1293e00>
[  512.602931] CR2: 000000000000001c

Thanks,
Sasha

#3 lore raw
Quoted 3 lines · Oleg Nesterov

Hmm. I'll recheck and report tomorrow, but it seems that I found a bug:
bad_fork_free_pid()->free_pid() looks wrong.

No, sorry for noise, I was wrong.

Quoted 2 lines

David, any other reason why ->real_cred can be NULL ? (assuming that I interpret this asm correctly).

Oleg.

#4 lore raw
In reply to Oleg Nesterov
Quoted 2 lines · Oleg Nesterov

David, any other reason why ->real_cred can be NULL ? (assuming that I interpret this asm correctly).

It should only be possible to see ->real_cred as being NULL after exit_creds() has been called from __put_task_struct() for a task that has finished construction.

It shouldn't be possible to introduce a NULL pointer through commit_creds() or override_creds() since both of those should crash immediately if given one, but it's possible revert_creds() could be so used.

Is there a race between kill() and exit() brought on by the kill path only using the RCU read lock? This doesn't prevent ->real_cred from being modified, but it looks like this should, in combination with delayed_put_task_struct(), prevent it from being cleared.

David

#5 lore raw
Quoted 9 lines · David Howells
Quoted 2 lines · Oleg Nesterov

David, any other reason why ->real_cred can be NULL ? (assuming that I interpret this asm correctly).

It should only be possible to see ->real_cred as being NULL after exit_creds() has been called from __put_task_struct() for a task that has finished construction.

Yes, thanks, this was my understanding...

Quoted 4 lines

Is there a race between kill() and exit() brought on by the kill path only using the RCU read lock? This doesn't prevent ->real_cred from being modified, but it looks like this should, in combination with delayed_put_task_struct(), prevent it from being cleared.

Yes, rcu should protect us from both delayed_put_pid() and delayed_put_task(). Everything looks correct... And there are a lot of other similar users of find_vpid/find_task_by_vpid/pid_task/etc under rcu, I can't recall any bug in this area.

I am puzzled. Note also that ->signal == NULL. Will try to think more, but so far I have no any idea.

Oleg.

#6 lore raw
Quoted 11 lines · Oleg Nesterov
Quoted 4 lines

Is there a race between kill() and exit() brought on by the kill path only

Quoted 3 lines

using the RCU read lock? This doesn't prevent ->real_cred from being modified, but it looks like this should, in combination with delayed_put_task_struct(), prevent it from being cleared.

Yes, rcu should protect us from both delayed_put_pid() and delayed_put_task(). Everything looks correct... And there are a lot of other similar users of find_vpid/find_task_by_vpid/pid_task/etc under rcu, I can't recall any bug in this area.

I am puzzled. Note also that ->signal == NULL. Will try to think more, but so far I have no any idea.

I've hit something similar earlier today, and it might be related:

[  973.452840] BUG: unable to handle kernel NULL pointer dereference at 00000000000002b0
[  973.455347] IP: flush_sigqueue_mask (include/linux/signal.h:118 kernel/signal.c:715)
[  973.457526] PGD 4dfdc7067 PUD 5f77d9067 PMD 0
[  973.459216] Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[  973.460086] Dumping ftrace buffer:
[  973.460086]    (ftrace buffer empty)
[  973.460086] Modules linked in:
[  973.460086] CPU: 4 PID: 13145 Comm: trinity-c767 Not tainted 3.17.0-rc2-next-20140826-sasha-00031-gc48c9ac-dirty #1079
[  973.460086] task: ffff880604800000 ti: ffff880586648000 task.ti: ffff880586648000
[  973.460086] RIP: flush_sigqueue_mask (include/linux/signal.h:118 kernel/signal.c:715)
[  973.460086] RSP: 0018:ffff88058664bec8  EFLAGS: 00010046
[  973.460086] RAX: 0000000000000000 RBX: fffffffffffff730 RCX: 0000000000010000
[  973.460086] RDX: 0000000000000000 RSI: 00000000000002a0 RDI: ffff88058664bed8
[  973.460086] RBP: ffff88058664bf10 R08: 0000000000000001 R09: 0000000000000001
[  973.460086] R10: 000000000002d201 R11: 0000000000000254 R12: 0000000000000000
[  973.460086] R13: ffff88058664bf40 R14: ffff880604800000 R15: 0000000000000010
[  973.460086] FS:  00007fe3a3045700(0000) GS:ffff880277c00000(0000) knlGS:0000000000000000
[  973.460086] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  973.460086] CR2: 00000000000002b0 CR3: 00000004e23d5000 CR4: 00000000000006a0
[  973.460086] Stack:
[  973.460086]  ffffffffac183690 01017fffb3247180 0000000000010000 0000000000000000
[  973.460086]  00007fffb3247180 00007fffb3247220 0000000000000011 0000000000000000
[  973.460086]  0000000000000000 ffff88058664bf78 ffffffffac183ef5 0000000000000000
[  973.460086] Call Trace:
[  973.460086] ? do_sigaction (kernel/signal.c:3124 (discriminator 17))
[  973.460086] SyS_rt_sigaction (kernel/signal.c:3360 kernel/signal.c:3341)
[  973.460086] tracesys (arch/x86/kernel/entry_64.S:542)
[ 973.460086] Code: b7 49 09 d5 4d 89 6e 10 48 83 c4 08 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 48 8b 0f 31 c0 <48> 8b 56 10 48 85 ca 74 7b 55 48 f7 d1 48 89 e5 41 56 48 21 ca
All code
========
   0:	b7 49                	mov    $0x49,%bh
   2:	09 d5                	or     %edx,%ebp
   4:	4d 89 6e 10          	mov    %r13,0x10(%r14)
   8:	48 83 c4 08          	add    $0x8,%rsp
   c:	5b                   	pop    %rbx
   d:	41 5c                	pop    %r12
   f:	41 5d                	pop    %r13
  11:	41 5e                	pop    %r14
  13:	41 5f                	pop    %r15
  15:	5d                   	pop    %rbp
  16:	c3                   	retq
  17:	66 2e 0f 1f 84 00 00 	nopw   %cs:0x0(%rax,%rax,1)
  1e:	00 00 00
  21:	66 66 66 66 90       	data32 data32 data32 xchg %ax,%ax
  26:	48 8b 0f             	mov    (%rdi),%rcx
  29:	31 c0                	xor    %eax,%eax
  2b:*	48 8b 56 10          	mov    0x10(%rsi),%rdx		<-- trapping instruction
  2f:	48 85 ca             	test   %rcx,%rdx
  32:	74 7b                	je     0xaf
  34:	55                   	push   %rbp
  35:	48 f7 d1             	not    %rcx
  38:	48 89 e5             	mov    %rsp,%rbp
  3b:	41 56                	push   %r14
  3d:	48 21 ca             	and    %rcx,%rdx
	...

Code starting with the faulting instruction
===========================================

   0:	48 8b 56 10          	mov    0x10(%rsi),%rdx
   4:	48 85 ca             	test   %rcx,%rdx
   7:	74 7b                	je     0x84
   9:	55                   	push   %rbp
   a:	48 f7 d1             	not    %rcx
   d:	48 89 e5             	mov    %rsp,%rbp
  10:	41 56                	push   %r14
  12:	48 21 ca             	and    %rcx,%rdx
	...
[  973.460086] RIP flush_sigqueue_mask (include/linux/signal.h:118 kernel/signal.c:715)
[  973.460086]  RSP <ffff88058664bec8>
[  973.460086] CR2: 00000000000002b0

Thanks,
Sasha

#7 lore raw
Quoted 11 lines · Sasha Levin
Quoted 9 lines · Oleg Nesterov
Quoted 4 lines

Is there a race between kill() and exit() brought on by the kill path only

Quoted 3 lines

using the RCU read lock? This doesn't prevent ->real_cred from being modified, but it looks like this should, in combination with delayed_put_task_struct(), prevent it from being cleared.

Yes, rcu should protect us from both delayed_put_pid() and delayed_put_task(). Everything looks correct... And there are a lot of other similar users of find_vpid/find_task_by_vpid/pid_task/etc under rcu, I can't recall any bug in this area.

Quoted 4 lines
Quoted 2 lines

I am puzzled. Note also that ->signal == NULL. Will try to think more, but so far I have no any idea.

I've hit something similar earlier today, and it might be related:

Thanks.

rsi == &current->signal->shared_pending == 0x2a0, so current->signal == 544.

Given that this task is current, we can rule out RCU/find_pind/etc problems, this task_struct and its ->signal must be stable.

Looks like, something was broken recently. I do not remember the bug reports like this. Say, an unbalanced put_task_struct()... Hmm.

Quoted 74 lines
[  973.452840] BUG: unable to handle kernel NULL pointer dereference at 00000000000002b0
[  973.455347] IP: flush_sigqueue_mask (include/linux/signal.h:118 kernel/signal.c:715)
[  973.457526] PGD 4dfdc7067 PUD 5f77d9067 PMD 0
[  973.459216] Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[  973.460086] Dumping ftrace buffer:
[  973.460086]    (ftrace buffer empty)
[  973.460086] Modules linked in:
[  973.460086] CPU: 4 PID: 13145 Comm: trinity-c767 Not tainted 3.17.0-rc2-next-20140826-sasha-00031-gc48c9ac-dirty #1079
[  973.460086] task: ffff880604800000 ti: ffff880586648000 task.ti: ffff880586648000
[  973.460086] RIP: flush_sigqueue_mask (include/linux/signal.h:118 kernel/signal.c:715)
[  973.460086] RSP: 0018:ffff88058664bec8  EFLAGS: 00010046
[  973.460086] RAX: 0000000000000000 RBX: fffffffffffff730 RCX: 0000000000010000
[  973.460086] RDX: 0000000000000000 RSI: 00000000000002a0 RDI: ffff88058664bed8
[  973.460086] RBP: ffff88058664bf10 R08: 0000000000000001 R09: 0000000000000001
[  973.460086] R10: 000000000002d201 R11: 0000000000000254 R12: 0000000000000000
[  973.460086] R13: ffff88058664bf40 R14: ffff880604800000 R15: 0000000000000010
[  973.460086] FS:  00007fe3a3045700(0000) GS:ffff880277c00000(0000) knlGS:0000000000000000
[  973.460086] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  973.460086] CR2: 00000000000002b0 CR3: 00000004e23d5000 CR4: 00000000000006a0
[  973.460086] Stack:
[  973.460086]  ffffffffac183690 01017fffb3247180 0000000000010000 0000000000000000
[  973.460086]  00007fffb3247180 00007fffb3247220 0000000000000011 0000000000000000
[  973.460086]  0000000000000000 ffff88058664bf78 ffffffffac183ef5 0000000000000000
[  973.460086] Call Trace:
[  973.460086] ? do_sigaction (kernel/signal.c:3124 (discriminator 17))
[  973.460086] SyS_rt_sigaction (kernel/signal.c:3360 kernel/signal.c:3341)
[  973.460086] tracesys (arch/x86/kernel/entry_64.S:542)
[ 973.460086] Code: b7 49 09 d5 4d 89 6e 10 48 83 c4 08 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 48 8b 0f 31 c0 <48> 8b 56 10 48 85 ca 74 7b 55 48 f7 d1 48 89 e5 41 56 48 21 ca
All code
========
   0:	b7 49                	mov    $0x49,%bh
   2:	09 d5                	or     %edx,%ebp
   4:	4d 89 6e 10          	mov    %r13,0x10(%r14)
   8:	48 83 c4 08          	add    $0x8,%rsp
   c:	5b                   	pop    %rbx
   d:	41 5c                	pop    %r12
   f:	41 5d                	pop    %r13
  11:	41 5e                	pop    %r14
  13:	41 5f                	pop    %r15
  15:	5d                   	pop    %rbp
  16:	c3                   	retq
  17:	66 2e 0f 1f 84 00 00 	nopw   %cs:0x0(%rax,%rax,1)
  1e:	00 00 00
  21:	66 66 66 66 90       	data32 data32 data32 xchg %ax,%ax
  26:	48 8b 0f             	mov    (%rdi),%rcx
  29:	31 c0                	xor    %eax,%eax
  2b:*	48 8b 56 10          	mov    0x10(%rsi),%rdx		<-- trapping instruction
  2f:	48 85 ca             	test   %rcx,%rdx
  32:	74 7b                	je     0xaf
  34:	55                   	push   %rbp
  35:	48 f7 d1             	not    %rcx
  38:	48 89 e5             	mov    %rsp,%rbp
  3b:	41 56                	push   %r14
  3d:	48 21 ca             	and    %rcx,%rdx
	...

Code starting with the faulting instruction
===========================================

   0:	48 8b 56 10          	mov    0x10(%rsi),%rdx
   4:	48 85 ca             	test   %rcx,%rdx
   7:	74 7b                	je     0x84
   9:	55                   	push   %rbp
   a:	48 f7 d1             	not    %rcx
   d:	48 89 e5             	mov    %rsp,%rbp
  10:	41 56                	push   %r14
  12:	48 21 ca             	and    %rcx,%rdx
	...
[  973.460086] RIP flush_sigqueue_mask (include/linux/signal.h:118 kernel/signal.c:715)
[  973.460086]  RSP <ffff88058664bec8>
[  973.460086] CR2: 00000000000002b0

Thanks,
Sasha

#8 lore raw
In reply to Oleg Nesterov
Quoted 11 lines · Oleg Nesterov
Quoted 5 lines · Sasha Levin
Quoted 5 lines

Hi all,

While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:

Thanks...

looks like, kill_ok_by_cred()->__task_cred(t) returns NULL at first glance. perhaps you can show the result of "make kernel/signal.s" to be sure? Or at least the full "objdump -d kernel/signal.o".

Attached.

Thanks,
Sasha

#9 lore raw
Quoted 8 lines · Sasha Levin
Quoted 6 lines · Oleg Nesterov
Quoted 5 lines · Sasha Levin
Quoted 5 lines

Hi all,

While fuzzing with trinity inside a KVM tools guest running the latest -next kernel, I've stumbled on the following spew:

Quoted 7 lines
Quoted 5 lines

Thanks...

looks like, kill_ok_by_cred()->__task_cred(t) returns NULL at first glance. perhaps you can show the result of "make kernel/signal.s" to be sure? Or at least the full "objdump -d kernel/signal.o".

Attached.

Thanks.

Yes, t->real_cred == NULL (r14).

Interestingly, t->signal is NULL too (rcx). And ->signal must be never NULL. So it looks like this task_struct was reallocated/reused. Or corrupted.

t == 0xffff880546803000 (r12), this doesn't look wrong.

Oleg.