perf: use-after-free in perf_event_for_each

Hello,

The following program triggers use-after-free in perf_event_for_each:
https://gist.githubusercontent.com/dvyukov/f1c354a8356e42f4d0b3d912e1bec956/raw/31d7ecdf6dc2c7327b80ef8581a39c823bbe405d/gistfile1.txt

BUG: KASAN: use-after-free in perf_event_for_each_child+0x15f/0x180
kernel/events/core.c:4495 at addr ffff8800680ec248
Read of size 8 by task a.out/19370
CPU: 3 PID: 19370 Comm: a.out Not tainted 4.10.0-rc5+ #186
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
 __asan_report_load8_noabort+0x3e/0x40 mm/kasan/report.c:328
 perf_event_for_each_child+0x15f/0x180 kernel/events/core.c:4495
 perf_event_for_each kernel/events/core.c:4514 [inline]
 _perf_ioctl kernel/events/core.c:4671 [inline]
 perf_ioctl+0x9b5/0x1480 kernel/events/core.c:4685
 vfs_ioctl fs/ioctl.c:43 [inline]
 do_vfs_ioctl+0x1bf/0x1790 fs/ioctl.c:683
 SYSC_ioctl fs/ioctl.c:698 [inline]
 SyS_ioctl+0x8f/0xc0 fs/ioctl.c:689
 entry_SYSCALL_64_fastpath+0x1f/0xc2
RIP: 0033:0x44d289
RSP: 002b:00007fb128517cd8 EFLAGS: 00000217 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 000000000044d289
RDX: 0000000000010001 RSI: 0000000000002400 RDI: 0000000000000004
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000217 R12: 0000000000000000
R13: 0000000000000000 R14: 00007fb1285189c0 R15: 00007fb128518700
Object at ffff8800680ec040, in cache kmalloc-2048 size: 2048
Allocated:
PID = 19367
[<ffffffff81a0942b>] kmem_cache_alloc_trace+0x10b/0x670 mm/slab.c:3629
[<ffffffff818328e5>] kzalloc include/linux/slab.h:490 [inline]
[<ffffffff818328e5>] perf_event_alloc+0x1c5/0x1ef0 kernel/events/core.c:9134
[<ffffffff8184432d>] SYSC_perf_event_open+0xb8d/0x31b0 kernel/events/core.c:9692
[<ffffffff81850419>] SyS_perf_event_open+0x39/0x50 kernel/events/core.c:9586
[<ffffffff841c8c81>] entry_SYSCALL_64_fastpath+0x1f/0xc2
Freed:
PID = 19372
[<ffffffff81a0b103>] __cache_free mm/slab.c:3505 [inline]
[<ffffffff81a0b103>] kfree+0xd3/0x250 mm/slab.c:3822
[<ffffffff81819b6c>] free_event_rcu+0x5c/0x70 kernel/events/core.c:3828
[<ffffffff81608600>] __rcu_reclaim kernel/rcu/rcu.h:118 [inline]
[<ffffffff81608600>] rcu_do_batch.isra.70+0x9e0/0xdf0 kernel/rcu/tree.c:2780
[<ffffffff81608e82>] invoke_rcu_callbacks kernel/rcu/tree.c:3043 [inline]
[<ffffffff81608e82>] __rcu_process_callbacks kernel/rcu/tree.c:3010 [inline]
[<ffffffff81608e82>] rcu_process_callbacks+0x472/0xc70 kernel/rcu/tree.c:3027
[<ffffffff841cbfbf>] __do_softirq+0x31f/0xbe7 kernel/softirq.c:284

On commit 095cbe66973771fecd8e8b1e8763181363ef703e (Jan 22).

That's the WARN_ON_ONCE() in there, and since the object that's being UAF'ed is an event, this would be the first dereference in there, this means that this is the event passed in from the next function.

And this is the first caller (ie, !sibling), suggesting that the line above:

  event = event->group_leader;

is the curlprit, because the earlier event->ctx, dereference is fine (and we're holding ctx->mutex).

This also matches with that the program does; which is something along the lines of:

  event = sys_perf_event_open();
  sibling = sys_perf_event_open(.group_fd = event);
  dup3(sibling, event, ..);
  ioctl(sibling);

Where the dup3() will close event (which should then promote the sibling to its own full event) and ioctl() on the sibling, which them comes apart.

So it looks like something fishy on the close the leader and promote the sibling code, but I cannot put my finger on it just yet. I've ran out of time for today, but will continue staring at this tomorrow -- of course hoping that this email will give someone else ideas while I'm off :-)

I've been running 60 concurrent instances of that thing for hours now, and have not been able to reproduce :-/

I did enable CONFIG_KASAN but otherwise booted as normal, and the thing says:

[    0.000000] kasan: KernelAddressSanitizer initialized

Is there anything else I should do?

I've ran out of ideas and it would be very helpful if I could prod at something that fails...

Should be enough.

Try to run more parallel processes at the same time.
This program will run 32 processes in a tight loop:
https://gist.githubusercontent.com/dvyukov/b36aa4398bb016923278fccdd1cc5b45/raw/9607078fd5ac2daf769c13d82561a83e17b06032/gistfile1.txt
It triggered the UAF in several minutes for me. I have 4 CPUs in the VM. If you have more, set number of processes to 8*CPU.

Just in case this is my config:

$ grep "PERF" .config
# CONFIG_CGROUP_PERF is not set
CONFIG_CC_OPTIMIZE_FOR_PERFORMANCE=y
CONFIG_HAVE_PERF_EVENTS=y
CONFIG_PERF_EVENTS=y
# CONFIG_DEBUG_PERF_USE_VMALLOC is not set
CONFIG_HAVE_PERF_EVENTS_NMI=y
CONFIG_HAVE_PERF_REGS=y
CONFIG_HAVE_PERF_USER_STACK_DUMP=y
CONFIG_PERF_EVENTS_INTEL_UNCORE=y
CONFIG_PERF_EVENTS_INTEL_RAPL=y
CONFIG_PERF_EVENTS_INTEL_CSTATE=y
# CONFIG_PERF_EVENTS_AMD_POWER is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
# CONFIG_PCIEASPM_PERFORMANCE is not set
# CONFIG_RCU_PERF_TEST is not set

And here is how I start qemu:

qemu-system-x86_64 -hda wheezy.img -net user,host=10.0.2.10,hostfwd=tcp::10022-:22 -net nic -nographic -kernel arch/x86/boot/bzImage -append "kvm-intel.nested=1 kvm-intel.enable_unrestricted_guest=1 kvm-intel.ept=1 kvm-intel.flexpriority=1 kvm-intel.vpid=1 kvm-intel.emulate_invalid_guest_state=1 kvm-intel.eptad=1 kvm-intel.enable_shadow_vmcs=1 kvm-intel.pml=1 kvm-intel.enable_apicv=1 console=ttyS0 root=/dev/sda earlyprintk=serial slub_debug=UZ vsyscall=native rodata=n oops=panic panic_on_warn=1 panic=86400" -enable-kvm -pidfile vm_pid -m 2G -smp 4 -cpu host -usb -usbdevice mouse -usbdevice tablet -soundhw all

No VM here, I'll up to 320 processes.

The below seems to fix things for me.

---
Subject: perf: Fix use-after-free

Dmitry reported a KASAN use-after-free.

XXX: do a changelog

---
 kernel/events/core.c | 27 +++++++++++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)

index 896aa53..8c1ad98 100644
@@ -1474,7 +1474,6 @@ ctx_group_list(struct perf_event *event, struct perf_event_context *ctx)
 static void
 list_add_event(struct perf_event *event, struct perf_event_context *ctx)
 {
-
 	lockdep_assert_held(&ctx->lock);
 
 	WARN_ON_ONCE(event->attach_state & PERF_ATTACH_CONTEXT);

@@ -1629,6 +1628,8 @@ static void perf_group_attach(struct perf_event *event)
 {
 	struct perf_event *group_leader = event->group_leader, *pos;
 
+	lockdep_assert_held(&event->ctx->lock);
+
 	/*
 	 * We can have double attach due to group movement in perf_event_open.
 	 */

@@ -1702,6 +1703,8 @@ static void perf_group_detach(struct perf_event *event)
 	struct perf_event *sibling, *tmp;
 	struct list_head *list = NULL;
 
+	lockdep_assert_held(&event->ctx->lock);
+
 	/*
 	 * We can have double detach due to exit/hot-unplug + close.
 	 */

@@ -1900,9 +1903,29 @@ __perf_remove_from_context(struct perf_event *event,
  */
 static void perf_remove_from_context(struct perf_event *event, unsigned long flags)
 {
-	lockdep_assert_held(&event->ctx->mutex);
+	struct perf_event_context *ctx = event->ctx;
+
+	lockdep_assert_held(&ctx->mutex);
 
 	event_function_call(event, __perf_remove_from_context, (void *)flags);
+
+	/*
+	 * The above event_function_call() can NO-OP when it hits
+	 * TASK_TOMBSTONE. In that case we must already have been detached
+	 * from the context (by perf_event_exit_event()) but the grouping
+	 * might still be in-tact.
+	 */
+	WARN_ON_ONCE(event->attach_state & PERF_ATTACH_CONTEXT);
+	if ((flags & DETACH_GROUP) &&
+	    (event->attach_state & PERF_ATTACH_GROUP)) {
+		/*
+		 * Since in that case we cannot possibly be scheduled, simply
+		 * detach now.
+		 */
+		raw_spin_lock_irq(&ctx->lock);
+		perf_group_detach(event);
+		raw_spin_unlock_irq(&ctx->lock);
+	}
 }
 
 /*

I've run the program again, it crashed machine in <1 min. Then applied your patch and run the program again, no crashes in 20 minutes. So:

FWIW,