Skip to contentSkip to conversation

perf fuzzer crash [PATCH] perf: Get group events reference before moving the group

PATCH Jiri Olsa 6 participants 13 messages Jan 16, 2015 – Jan 21, 2015 perf events

lore
Share & tools
lore

Jiri Olsa

author
#0 lore raw

hi Vince,
I was able to reproduce the issue you described in:
http://marc.info/?l=linux-kernel&m=141806390822670&w=2

I might have found one way that could lead to screwing up context's refcounts.. could you please try attached patch?

I'm now on 2 days of no crash while it used to happen 3 times a day before.

Or you can use following git branch:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/trinity_fix

thanks,
jirka

---
We need to make sure, that no event in the group lost the last reference and gets removed from the context during the group move in perf syscall.

This could happen if the child exits and calls put_event on the parent event which got already closed, like in following scenario:

- T1 creates software event E1
- T1 creates other software events as group with E1 as group leader
- T1 forks T2
- T2 has cloned E1 event that holds reference on E1
- T1 closes event within E1 group (say E3), the event stays alive due to the T2 reference
- following happens concurently:
A) T1 creates hardware event E2 with groupleader E1
B) T2 exits

ad A) T1 triggers the E1 group move into hardware context:

        mutex_lock(E1->ctx)
          - remove E1 group only from the E1->ctx context, leaving
            the goup links untouched
        mutex_unlock(E1->ctx)
        mutex_lock(E2->ctx)
          - install E1 group into E2->ctx using the E1 group links
        mutex_unlock(E2->ctx)

ad B) put_event(E3) is called and E3 is removed from E1->ctx
completely, including group links

If 'A' and 'B' races, we will get unbalanced refcounts, because of removed group links.

Adding get_group/put_group functions to handle the event ref's increase/decrease for the whole group.

---
 kernel/events/core.c | 61 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 61 insertions(+)
kernel/events/core.c +61 -0
index af0a5ba4e21d..1922bae9f24e 100644
@@ -7250,6 +7250,55 @@ out:
 	return ret;
 }
 
+static void put_group(struct perf_event **group_arr)
+{
+	struct perf_event *event;
+	int i = 0;
+
+	while (event = group_arr[i++]) {
+		put_event(event);
+	}
+
+	kfree(group_arr);
+}
+
+static int get_group(struct perf_event *leader, struct perf_event ***group_arr)
+{
+	struct perf_event_context *ctx = leader->ctx;
+	struct perf_event *sibling, **arr = NULL;
+	int i = 0, err = -ENOMEM;
+	size_t size;
+
+	if (!atomic_long_inc_not_zero(&leader->refcount))
+		return -EINVAL;
+
+	mutex_lock(&ctx->mutex);
+	/* + 1 for leader and +1 for final NULL */
+	size = (leader->nr_siblings + 2) * sizeof(leader);
+
+	arr = *group_arr = kzalloc(size, GFP_KERNEL);
+	if (!arr)
+		goto err;
+
+	err = -EINVAL;
+	arr[i++] = leader;
+
+	list_for_each_entry(sibling, &leader->sibling_list, group_entry) {
+		if (!atomic_long_inc_not_zero(&sibling->refcount))
+			goto err;
+
+		arr[i++] = sibling;
+	}
+
+	mutex_unlock(&ctx->mutex);
+	return 0;
+err:
+	mutex_unlock(&ctx->mutex);
+	if (arr)
+		put_group(arr);
+	return err;
+}
+
 /**
  * sys_perf_event_open - open a performance event, associate it to a task/cpu
  *
@@ -7263,6 +7312,7 @@ SYSCALL_DEFINE5(perf_event_open,
 		pid_t, pid, int, cpu, int, group_fd, unsigned long, flags)
 {
 	struct perf_event *group_leader = NULL, *output_event = NULL;
+	struct perf_event **group_arr = NULL;
 	struct perf_event *event, *sibling;
 	struct perf_event_attr attr;
 	struct perf_event_context *ctx;
@@ -7443,6 +7493,12 @@ SYSCALL_DEFINE5(perf_event_open,
 			goto err_context;
 	}
 
+	if (move_group) {
+		err = get_group(group_leader, &group_arr);
+		if (err)
+			goto err_context;
+	}
+
 	event_file = anon_inode_getfile("[perf_event]", &perf_fops, event,
 					f_flags);
 	if (IS_ERR(event_file)) {
@@ -7490,6 +7546,9 @@ SYSCALL_DEFINE5(perf_event_open,
 	perf_unpin_context(ctx);
 	mutex_unlock(&ctx->mutex);
 
+	if (move_group)
+		put_group(group_arr);
+
 	put_online_cpus();
 
 	event->owner = current;
@@ -7515,6 +7574,8 @@ SYSCALL_DEFINE5(perf_event_open,
 	return event_fd;
 
 err_context:
+	if (group_arr)
+		put_group(group_arr);
 	perf_unpin_context(ctx);
 	put_ctx(ctx);
 err_alloc:
Signature · 1 line
1.9.3
maintainer
#1 lore raw
Quoted 27 lines · Jiri Olsa

We need to make sure, that no event in the group lost the last reference and gets removed from the context during the group move in perf syscall.

This could happen if the child exits and calls put_event on the parent event which got already closed, like in following scenario:

- T1 creates software event E1
- T1 creates other software events as group with E1 as group leader
- T1 forks T2
- T2 has cloned E1 event that holds reference on E1
- T1 closes event within E1 group (say E3), the event stays alive due to the T2 reference
- following happens concurently:
A) T1 creates hardware event E2 with groupleader E1
B) T2 exits

ad A) T1 triggers the E1 group move into hardware context:

        mutex_lock(E1->ctx)
          - remove E1 group only from the E1->ctx context, leaving
            the goup links untouched
        mutex_unlock(E1->ctx)
        mutex_lock(E2->ctx)
          - install E1 group into E2->ctx using the E1 group links
        mutex_unlock(E2->ctx)
Quoted 8 lines

ad B) put_event(E3) is called and E3 is removed from E1->ctx
completely, including group links

If 'A' and 'B' races, we will get unbalanced refcounts, because of removed group links.

Adding get_group/put_group functions to handle the event ref's increase/decrease for the whole group.

Its a bandaid at best :/ The problem is (again) that we changes event->ctx without any kind of serialization.

The issue came up before:

https://lkml.org/lkml/2014/9/5/397

and I've not been able to come up with anything much saner.

maintainer
#2 lore raw
Quoted 8 lines · Peter Zijlstra

Its a bandaid at best :/ The problem is (again) that we changes event->ctx without any kind of serialization.

The issue came up before:

https://lkml.org/lkml/2014/9/5/397

and I've not been able to come up with anything much saner.

A little something like the below is the best I could come up with; I know Linus hated it, but I figure we ought to do something to stop crashing.

---
 init/Kconfig         |    1 +
 kernel/events/core.c |  126 ++++++++++++++++++++++++++++++++++++++++----------
 2 files changed, 103 insertions(+), 24 deletions(-)
init/Kconfig +1 -0
index 9afb971..ebc8522 100644
@@ -1595,6 +1595,7 @@ config PERF_EVENTS
 	depends on HAVE_PERF_EVENTS
 	select ANON_INODES
 	select IRQ_WORK
+	select PERCPU_RWSEM
 	help
 	  Enable kernel support for various performance events provided
 	  by software and hardware.
kernel/events/core.c +102 -24

356 lines not on this page · raw message

#3 lore raw
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Peter Zijlstra

Its a bandaid at best :/ The problem is (again) that we changes event->ctx without any kind of serialization.

The issue came up before:

Quoted 7 lines
Quoted 3 lines

https://lkml.org/lkml/2014/9/5/397

and I've not been able to come up with anything much saner.

A little something like the below is the best I could come up with; I know Linus hated it, but I figure we ought to do something to stop crashing.

I just wanted to report that I've tested both this patch and Jiri's original one and they both keep my easy-to-trigger-testcase from crashing my core2 machine (when applied against 3.18).

This is great! I've been chasing this bug for months.

Vince

#4 lore raw
Quoted 4 lines · Vince Weaver

I just wanted to report that I've tested both this patch and Jiri's original one and they both keep my easy-to-trigger-testcase from crashing my core2 machine (when applied against 3.18).

I've continued fuzzing all weekend, with PeterZ's patch applied to 3.19-rc4. It has managed to stay up without crashing, impressive.

Fuzzing still turns up a few issues but no crashes:

[ 1753.904001] WARNING: CPU: 0 PID: 23800 at arch/x86/kernel/hw_breakpoint.c:119 arch_install_hw_breakpoint+0x6a/0xf6()
[ 1753.904001] Can't find any breakpoint slot

(I thought we had fixed that one).

[80608.209702] Uhhuh. NMI received for unknown reason 3d on CPU 0.
[80608.212041] Do you have a strange power saving mode enabled?
[80608.212041] Dazed and confused, but trying to continue

(these are common but don't seem to affect system stability).

Vince

#5 lore raw
In reply to Peter Zijlstra
Quoted 39 lines · Peter Zijlstra
kernel/events/core.c +26 -0
@@ -42,6 +42,7 @@
 #include <linux/module.h>
 #include <linux/mman.h>
 #include <linux/compat.h>
+#include <linux/percpu-rwsem.h>
 
 #include "internal.h"
 
@@ -122,6 +123,42 @@ static int cpu_function_call(int cpu, int (*func) (void *info), void *info)
 	return data.ret;
 }
 
+/*
+ * Required to migrate events between contexts.
+ *
+ * Migrating events between contexts is rather tricky; there is no real
+ * serialization around the perf_event::ctx pointer.
+ *
+ * So what we do is hold this rwsem over the remove_from_context and
+ * install_in_context. The remove_from_context ensures the event is inactive
+ * and will not be used from IRQ/NMI context anymore, and the remaining
+ * sites can acquire the rwsem read side.
+ */
+static struct percpu_rw_semaphore perf_rwsem;
+
+static inline struct perf_event_context *perf_event_ctx(struct perf_event *event)
+{
+#ifdef CONFIG_LOCKDEP
+	/*
+	 * Assert the locking rules outlined above; in order to dereference
+	 * event->ctx we must either be attached to the context or hold
+	 * perf_rwsem.
+	 *
+	 * XXX not usable from IPIs because the lockdep held lock context
+	 * will be wrong; maybe add trylock variants to the percpu_rw_semaphore
+	 */
+	WARN_ON_ONCE(!(event->attach_state & PERF_ATTACH_CONTEXT) ||

kernel/events/core.c static int cpu_function_call(int cpu, int (*func) (void *info), void *info) · line 160

Quoted 10 lines
kernel/events/core.c +10 -0
+		     (debug_locks && !lockdep_is_held(&perf_rwsem.rw_sem)));
+#endif
+
+	return event->ctx;
+}
+
+static inline struct perf_event_context *__perf_event_ctx(struct perf_event *event)
+{
+	return event->ctx;
+}

So if this approach is acceptable I'd also rename event->ctx to event->__ctx, to make sure it's not used accidentally without serialization in any old (or new) perf related patches.

Thanks,

Ingo

reviewer
#6 lore raw
In reply to Peter Zijlstra
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Peter Zijlstra

Its a bandaid at best :/ The problem is (again) that we changes event->ctx without any kind of serialization.

The issue came up before:

https://lkml.org/lkml/2014/9/5/397

In the end neither the CCI or CCN perf drivers migrate events on hotplug, so ARM is currently safe from the perf_pmu_migrate_context case, but I see that you fix the move_group handling too.

I had a go at testing this by hacking migration back into the CCI PMU driver (atop of v3.19-rc5), but I'm seeing lockups after a few minutes with my original test case (https://lkml.org/lkml/2014/9/1/569 with PMU_TYPE and PMU_EVENT fixed up).

I unfortunately don't have a suitable x86 box spare to run that on. Would someone be able to give it a spin on something with an uncore PMU?

I'll go and dig a bit further. I may just be hitting another latent issue on my board.

Thanks,
Mark.

Quoted 421 lines
Quoted 2 lines

and I've not been able to come up with anything much saner.

A little something like the below is the best I could come up with; I know Linus hated it, but I figure we ought to do something to stop crashing.

---
 init/Kconfig         |    1 +
 kernel/events/core.c |  126 ++++++++++++++++++++++++++++++++++++++++----------
 2 files changed, 103 insertions(+), 24 deletions(-)
init/Kconfig +1 -0
index 9afb971..ebc8522 100644
@@ -1595,6 +1595,7 @@ config PERF_EVENTS
        depends on HAVE_PERF_EVENTS
        select ANON_INODES
        select IRQ_WORK
+       select PERCPU_RWSEM
        help
          Enable kernel support for various performance events provided
          by software and hardware.
kernel/events/core.c +102 -24
index c10124b..fb3971d 100644
@@ -42,6 +42,7 @@
 #include <linux/module.h>
 #include <linux/mman.h>
 #include <linux/compat.h>
+#include <linux/percpu-rwsem.h>
 
 #include "internal.h"
 
@@ -122,6 +123,42 @@ static int cpu_function_call(int cpu, int (*func) (void *info), void *info)
        return data.ret;
 }
 
+/*
+ * Required to migrate events between contexts.
+ *
+ * Migrating events between contexts is rather tricky; there is no real
+ * serialization around the perf_event::ctx pointer.
+ *
+ * So what we do is hold this rwsem over the remove_from_context and
+ * install_in_context. The remove_from_context ensures the event is inactive
+ * and will not be used from IRQ/NMI context anymore, and the remaining
+ * sites can acquire the rwsem read side.
+ */
+static struct percpu_rw_semaphore perf_rwsem;
+
+static inline struct perf_event_context *perf_event_ctx(struct perf_event *event)
+{
+#ifdef CONFIG_LOCKDEP
+       /*
+        * Assert the locking rules outlined above; in order to dereference
+        * event->ctx we must either be attached to the context or hold
+        * perf_rwsem.
+        *
+        * XXX not usable from IPIs because the lockdep held lock context
+        * will be wrong; maybe add trylock variants to the percpu_rw_semaphore
+        */
+       WARN_ON_ONCE(!(event->attach_state & PERF_ATTACH_CONTEXT) ||
+                    (debug_locks && !lockdep_is_held(&perf_rwsem.rw_sem)));
+#endif
+
+       return event->ctx;
+}
+
+static inline struct perf_event_context *__perf_event_ctx(struct perf_event *event)
+{
+       return event->ctx;
+}
+
 #define EVENT_OWNER_KERNEL ((void *) -1)
 
 static bool is_kernel_event(struct perf_event *event)
@@ -380,7 +417,7 @@ perf_cgroup_from_task(struct task_struct *task)
 static inline bool
 perf_cgroup_match(struct perf_event *event)
 {
-       struct perf_event_context *ctx = event->ctx;
+       struct perf_event_context *ctx = __perf_event_ctx(event);
        struct perf_cpu_context *cpuctx = __get_cpu_context(ctx);
 
        /* @event doesn't care about cgroup */
@@ -1054,7 +1091,7 @@ static void update_context_time(struct perf_event_context *ctx)
 
 static u64 perf_event_time(struct perf_event *event)
 {
-       struct perf_event_context *ctx = event->ctx;
+       struct perf_event_context *ctx = __perf_event_ctx(event);
 
        if (is_cgroup_event(event))
                return perf_cgroup_event_time(event);
@@ -1068,7 +1105,7 @@ static u64 perf_event_time(struct perf_event *event)
  */
 static void update_event_times(struct perf_event *event)
 {
-       struct perf_event_context *ctx = event->ctx;
+       struct perf_event_context *ctx = __perf_event_ctx(event);
        u64 run_end;
 
        if (event->state < PERF_EVENT_STATE_INACTIVE ||
@@ -1518,7 +1555,7 @@ static int __perf_remove_from_context(void *info)
 {
        struct remove_event *re = info;
        struct perf_event *event = re->event;
-       struct perf_event_context *ctx = event->ctx;
+       struct perf_event_context *ctx = __perf_event_ctx(event);
        struct perf_cpu_context *cpuctx = __get_cpu_context(ctx);
 
        raw_spin_lock(&ctx->lock);
@@ -1551,7 +1588,7 @@ static int __perf_remove_from_context(void *info)
  */
 static void perf_remove_from_context(struct perf_event *event, bool detach_group)
 {
-       struct perf_event_context *ctx = event->ctx;
+       struct perf_event_context *ctx = perf_event_ctx(event);
        struct task_struct *task = ctx->task;
        struct remove_event re = {
                .event = event,
@@ -1606,7 +1643,7 @@ retry:
 int __perf_event_disable(void *info)
 {
        struct perf_event *event = info;
-       struct perf_event_context *ctx = event->ctx;
+       struct perf_event_context *ctx = __perf_event_ctx(event);
        struct perf_cpu_context *cpuctx = __get_cpu_context(ctx);
 
        /*
@@ -1656,20 +1693,24 @@ int __perf_event_disable(void *info)
  */
 void perf_event_disable(struct perf_event *event)
 {
-       struct perf_event_context *ctx = event->ctx;
-       struct task_struct *task = ctx->task;
+       struct perf_event_context *ctx;
+       struct task_struct *task;
+
+       percpu_down_read(&perf_rwsem);
+       ctx = perf_event_ctx(event);
+       task = ctx->task;
 
        if (!task) {
                /*
                 * Disable the event on the cpu that it's on
                 */
                cpu_function_call(event->cpu, __perf_event_disable, event);
-               return;
+               goto unlock;
        }
 
 retry:
        if (!task_function_call(task, __perf_event_disable, event))
-               return;
+               goto unlock;
 
        raw_spin_lock_irq(&ctx->lock);
        /*
@@ -1694,6 +1735,8 @@ retry:
                event->state = PERF_EVENT_STATE_OFF;
        }
        raw_spin_unlock_irq(&ctx->lock);
+unlock:
+       percpu_up_read(&perf_rwsem);
 }
 EXPORT_SYMBOL_GPL(perf_event_disable);
 
@@ -1937,7 +1980,7 @@ static void perf_event_sched_in(struct perf_cpu_context *cpuctx,
 static int  __perf_install_in_context(void *info)
 {
        struct perf_event *event = info;
-       struct perf_event_context *ctx = event->ctx;
+       struct perf_event_context *ctx = __perf_event_ctx(event);
        struct perf_cpu_context *cpuctx = __get_cpu_context(ctx);
        struct perf_event_context *task_ctx = cpuctx->task_ctx;
        struct task_struct *task = current;
@@ -2076,7 +2119,7 @@ static void __perf_event_mark_enabled(struct perf_event *event)
 static int __perf_event_enable(void *info)
 {
        struct perf_event *event = info;
-       struct perf_event_context *ctx = event->ctx;
+       struct perf_event_context *ctx = __perf_event_ctx(event);
        struct perf_event *leader = event->group_leader;
        struct perf_cpu_context *cpuctx = __get_cpu_context(ctx);
        int err;
@@ -2160,15 +2203,19 @@ unlock:
  */
 void perf_event_enable(struct perf_event *event)
 {
-       struct perf_event_context *ctx = event->ctx;
-       struct task_struct *task = ctx->task;
+       struct perf_event_context *ctx;
+       struct task_struct *task;
+
+       percpu_down_read(&perf_rwsem);
+       ctx = perf_event_ctx(event);
+       task = ctx->task;
 
        if (!task) {
                /*
                 * Enable the event on the cpu that it's on
                 */
                cpu_function_call(event->cpu, __perf_event_enable, event);
-               return;
+               goto unlock;
        }
 
        raw_spin_lock_irq(&ctx->lock);
@@ -2194,7 +2241,7 @@ retry:
        raw_spin_unlock_irq(&ctx->lock);
 
        if (!task_function_call(task, __perf_event_enable, event))
-               return;
+               goto unlock;
 
        raw_spin_lock_irq(&ctx->lock);
 
@@ -2213,6 +2260,8 @@ retry:
 
 out:
        raw_spin_unlock_irq(&ctx->lock);
+unlock:
+       percpu_up_read(&perf_rwsem);
 }
 EXPORT_SYMBOL_GPL(perf_event_enable);
 
@@ -3076,7 +3125,7 @@ void perf_event_exec(void)
 static void __perf_event_read(void *info)
 {
        struct perf_event *event = info;
-       struct perf_event_context *ctx = event->ctx;
+       struct perf_event_context *ctx = __perf_event_ctx(event);
        struct perf_cpu_context *cpuctx = __get_cpu_context(ctx);
 
        /*
@@ -3115,7 +3164,7 @@ static u64 perf_event_read(struct perf_event *event)
                smp_call_function_single(event->oncpu,
                                         __perf_event_read, event, 1);
        } else if (event->state == PERF_EVENT_STATE_INACTIVE) {
-               struct perf_event_context *ctx = event->ctx;
+               struct perf_event_context *ctx = perf_event_ctx(event);
                unsigned long flags;
 
                raw_spin_lock_irqsave(&ctx->lock, flags);
@@ -3440,7 +3489,7 @@ static void perf_remove_from_owner(struct perf_event *event)
  */
 static void put_event(struct perf_event *event)
 {
-       struct perf_event_context *ctx = event->ctx;
+       struct perf_event_context *ctx;
 
        if (!atomic_long_dec_and_test(&event->refcount))
                return;
@@ -3448,6 +3497,8 @@ static void put_event(struct perf_event *event)
        if (!is_kernel_event(event))
                perf_remove_from_owner(event);
 
+       percpu_down_read(&perf_rwsem);
+       ctx = perf_event_ctx(event);
        WARN_ON_ONCE(ctx->parent_ctx);
        /*
         * There are two ways this annotation is useful:
@@ -3464,6 +3515,7 @@ static void put_event(struct perf_event *event)
        mutex_lock_nested(&ctx->mutex, SINGLE_DEPTH_NESTING);
        perf_remove_from_context(event, true);
        mutex_unlock(&ctx->mutex);
+       percpu_up_read(&perf_rwsem);
 
        _free_event(event);
 }
@@ -3647,11 +3699,13 @@ perf_read_hw(struct perf_event *event, char __user *buf, size_t count)
        if (count < event->read_size)
                return -ENOSPC;
 
-       WARN_ON_ONCE(event->ctx->parent_ctx);
+       percpu_down_read(&perf_rwsem);
+       WARN_ON_ONCE(perf_event_ctx(event)->parent_ctx);
        if (read_format & PERF_FORMAT_GROUP)
                ret = perf_event_read_group(event, read_format, buf);
        else
                ret = perf_event_read_one(event, read_format, buf);
+       percpu_up_read(&perf_rwsem);
 
        return ret;
 }
@@ -3689,9 +3743,11 @@ static unsigned int perf_poll(struct file *file, poll_table *wait)
 
 static void perf_event_reset(struct perf_event *event)
 {
+       percpu_down_read(&perf_rwsem);
        (void)perf_event_read(event);
        local64_set(&event->count, 0);
        perf_event_update_userpage(event);
+       percpu_up_read(&perf_rwsem);
 }
 
 /*
@@ -3705,7 +3761,7 @@ static void perf_event_for_each_child(struct perf_event *event,
 {
        struct perf_event *child;
 
-       WARN_ON_ONCE(event->ctx->parent_ctx);
+       WARN_ON_ONCE(__perf_event_ctx(event)->parent_ctx);
        mutex_lock(&event->child_mutex);
        func(event);
        list_for_each_entry(child, &event->child_list, child_list)
@@ -3716,6 +3772,14 @@ static void perf_event_for_each_child(struct perf_event *event,
 static void perf_event_for_each(struct perf_event *event,
                                  void (*func)(struct perf_event *))
 {
+       /*
+        * XXX broken
+        *
+        * lock inversion and recursion issues; ctx->mutex must nest inside
+        * perf_rwsem, but func() will take perf_rwsem again.
+        *
+        * Cure with ugly.
+        */
        struct perf_event_context *ctx = event->ctx;
        struct perf_event *sibling;
 
@@ -3731,7 +3795,7 @@ static void perf_event_for_each(struct perf_event *event,
 
 static int perf_event_period(struct perf_event *event, u64 __user *arg)
 {
-       struct perf_event_context *ctx = event->ctx;
+       struct perf_event_context *ctx;
        int ret = 0, active;
        u64 value;
 
@@ -3744,6 +3808,8 @@ static int perf_event_period(struct perf_event *event, u64 __user *arg)
        if (!value)
                return -EINVAL;
 
+       percpu_down_read(&perf_rwsem);
+       ctx = perf_event_ctx(event);
        raw_spin_lock_irq(&ctx->lock);
        if (event->attr.freq) {
                if (value > sysctl_perf_event_sample_rate) {
@@ -3772,6 +3838,7 @@ static int perf_event_period(struct perf_event *event, u64 __user *arg)
 
 unlock:
        raw_spin_unlock_irq(&ctx->lock);
+       percpu_up_read(&perf_rwsem);
 
        return ret;
 }
@@ -7229,11 +7296,13 @@ perf_event_set_output(struct perf_event *event, struct perf_event *output_event)
        if (output_event->cpu != event->cpu)
                goto out;
 
+       percpu_down_read(&perf_rwsem);
        /*
         * If its not a per-cpu rb, it must be the same task.
         */
-       if (output_event->cpu == -1 && output_event->ctx != event->ctx)
-               goto out;
+       if (output_event->cpu == -1 &&
+           perf_event_ctx(output_event) != perf_event_ctx(event))
+               goto unlock_rwsem;
 
 set:
        mutex_lock(&event->mmap_mutex);
@@ -7253,6 +7322,8 @@ set:
        ret = 0;
 unlock:
        mutex_unlock(&event->mmap_mutex);
+unlock_rwsem:
+       percpu_up_read(&perf_rwsem);
 
 out:
        return ret;
@@ -7461,6 +7532,7 @@ SYSCALL_DEFINE5(perf_event_open,
        if (move_group) {
                struct perf_event_context *gctx = group_leader->ctx;
 
+               percpu_down_write(&perf_rwsem);
                mutex_lock(&gctx->mutex);
                perf_remove_from_context(group_leader, false);
 
@@ -7498,6 +7570,9 @@ SYSCALL_DEFINE5(perf_event_open,
        perf_unpin_context(ctx);
        mutex_unlock(&ctx->mutex);
 
+       if (move_group)
+               percpu_up_write(&perf_rwsem);
+
        put_online_cpus();
 
        event->owner = current;
@@ -7600,6 +7675,7 @@ void perf_pmu_migrate_context(struct pmu *pmu, int src_cpu, int dst_cpu)
        struct perf_event *event, *tmp;
        LIST_HEAD(events);
 
+       percpu_down_write(&perf_rwsem);
        src_ctx = &per_cpu_ptr(pmu->pmu_cpu_context, src_cpu)->ctx;
        dst_ctx = &per_cpu_ptr(pmu->pmu_cpu_context, dst_cpu)->ctx;
 
@@ -7625,6 +7701,7 @@ void perf_pmu_migrate_context(struct pmu *pmu, int src_cpu, int dst_cpu)
                get_ctx(dst_ctx);
        }
        mutex_unlock(&dst_ctx->mutex);
+       percpu_up_write(&perf_rwsem);
 }
 EXPORT_SYMBOL_GPL(perf_pmu_migrate_context);
 
@@ -8261,6 +8338,7 @@ void __init perf_event_init(void)
 
        idr_init(&pmu_idr);
 
+       percpu_init_rwsem(&perf_rwsem);
        perf_event_init_all_cpus();
        init_srcu_struct(&pmus_srcu);
        perf_pmu_register(&perf_swevent, "software", PERF_TYPE_SOFTWARE);
reviewer
#7 lore raw
Quoted 18 lines · Mark Rutland
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Peter Zijlstra

Its a bandaid at best :/ The problem is (again) that we changes event->ctx without any kind of serialization.

The issue came up before:

https://lkml.org/lkml/2014/9/5/397

In the end neither the CCI or CCN perf drivers migrate events on hotplug, so ARM is currently safe from the perf_pmu_migrate_context case, but I see that you fix the move_group handling too.

I had a go at testing this by hacking migration back into the CCI PMU driver (atop of v3.19-rc5), but I'm seeing lockups after a few minutes with my original test case (https://lkml.org/lkml/2014/9/1/569 with PMU_TYPE and PMU_EVENT fixed up).

Quoted 5 lines

I unfortunately don't have a suitable x86 box spare to run that on. Would someone be able to give it a spin on something with an uncore PMU?

I'll go and dig a bit further. I may just be hitting another latent issue on my board.

I'm able to trigger the lockups even without both your patch and the call to perf_pmu_migrate_context, so there is a latent issue.

On vanilla v3.19-rc5 and vanilla v3.18, I'm able to get my hotplug script hung when run concurrently with the test case against the CCI PMU driver (without migration). The v3.18 and v3.19-rc5 lockups are identical:

INFO: task hpall.sh:1506 blocked for more than 120 seconds.
      Not tainted 3.19.0-rc5 #9

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
hpall.sh D 804a6ffc 0 1506 1497 0x00000000
[<804a6ffc>] (__schedule) from [<80022308>] (cpu_hotplug_begin+0xa0/0xac) [<80022308>] (cpu_hotplug_begin) from [<8002236c>] (_cpu_up+0x24/0x180) [<8002236c>] (_cpu_up) from [<8002253c>] (cpu_up+0x74/0x98) [<8002253c>] (cpu_up) from [<802bce60>] (device_online+0x64/0x90) [<802bce60>] (device_online) from [<802bcef4>] (online_store+0x68/0x74) [<802bcef4>] (online_store) from [<8014059c>] (kernfs_fop_write+0xbc/0x1a0) [<8014059c>] (kernfs_fop_write) from [<800e71b0>] (vfs_write+0xa0/0x1ac) [<800e71b0>] (vfs_write) from [<800e7808>] (SyS_write+0x44/0x9c) [<800e7808>] (SyS_write) from [<8000e560>] (ret_fast_syscall+0x0/0x48) 7 locks held by hpall.sh/1506:

 #0:  (sb_writers#6){.+.+.+}, at: [<800e729c>] vfs_write+0x18c/0x1ac
 #1:  (&of->mutex){+.+.+.}, at: [<8014052c>] kernfs_fop_write+0x4c/0x1a0
 #2:  (s_active#15){.+.+.+}, at: [<80140534>] kernfs_fop_write+0x54/0x1a0
 #3:  (device_hotplug_lock){+.+.+.}, at: [<802bbe44>] lock_device_hotplug_sysfs+0xc/0x4c
 #4:  (&dev->mutex){......}, at: [<802bce14>] device_online+0x18/0x90
 #5:  (cpu_add_remove_lock){+.+.+.}, at: [<80022508>] cpu_up+0x40/0x98
 #6:  (cpu_hotplug.lock){++++++}, at: [<80022268>] cpu_hotplug_begin+0x0/0xac

I guess that lockup is my fundamental issue, and with your patch the perf_rwsem manages to spread a transitive dependency on one of those locks all over the perf subsystem. I haven't considered that in great detail, however.

Thanks,
Mark.

reviewer
#8 lore raw
Quoted 55 lines · Mark Rutland
Quoted 18 lines · Mark Rutland
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Peter Zijlstra

Its a bandaid at best :/ The problem is (again) that we changes event->ctx without any kind of serialization.

The issue came up before:

https://lkml.org/lkml/2014/9/5/397

In the end neither the CCI or CCN perf drivers migrate events on hotplug, so ARM is currently safe from the perf_pmu_migrate_context case, but I see that you fix the move_group handling too.

I had a go at testing this by hacking migration back into the CCI PMU driver (atop of v3.19-rc5), but I'm seeing lockups after a few minutes with my original test case (https://lkml.org/lkml/2014/9/1/569 with PMU_TYPE and PMU_EVENT fixed up).

Quoted 5 lines

I unfortunately don't have a suitable x86 box spare to run that on. Would someone be able to give it a spin on something with an uncore PMU?

I'll go and dig a bit further. I may just be hitting another latent issue on my board.

I'm able to trigger the lockups even without both your patch and the call to perf_pmu_migrate_context, so there is a latent issue.

On vanilla v3.19-rc5 and vanilla v3.18, I'm able to get my hotplug script hung when run concurrently with the test case against the CCI PMU driver (without migration). The v3.18 and v3.19-rc5 lockups are identical:

INFO: task hpall.sh:1506 blocked for more than 120 seconds.
      Not tainted 3.19.0-rc5 #9

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
hpall.sh D 804a6ffc 0 1506 1497 0x00000000
[<804a6ffc>] (__schedule) from [<80022308>] (cpu_hotplug_begin+0xa0/0xac) [<80022308>] (cpu_hotplug_begin) from [<8002236c>] (_cpu_up+0x24/0x180) [<8002236c>] (_cpu_up) from [<8002253c>] (cpu_up+0x74/0x98) [<8002253c>] (cpu_up) from [<802bce60>] (device_online+0x64/0x90) [<802bce60>] (device_online) from [<802bcef4>] (online_store+0x68/0x74) [<802bcef4>] (online_store) from [<8014059c>] (kernfs_fop_write+0xbc/0x1a0) [<8014059c>] (kernfs_fop_write) from [<800e71b0>] (vfs_write+0xa0/0x1ac) [<800e71b0>] (vfs_write) from [<800e7808>] (SyS_write+0x44/0x9c) [<800e7808>] (SyS_write) from [<8000e560>] (ret_fast_syscall+0x0/0x48) 7 locks held by hpall.sh/1506:

 #0:  (sb_writers#6){.+.+.+}, at: [<800e729c>] vfs_write+0x18c/0x1ac
 #1:  (&of->mutex){+.+.+.}, at: [<8014052c>] kernfs_fop_write+0x4c/0x1a0
 #2:  (s_active#15){.+.+.+}, at: [<80140534>] kernfs_fop_write+0x54/0x1a0
 #3:  (device_hotplug_lock){+.+.+.}, at: [<802bbe44>] lock_device_hotplug_sysfs+0xc/0x4c
 #4:  (&dev->mutex){......}, at: [<802bce14>] device_online+0x18/0x90
 #5:  (cpu_add_remove_lock){+.+.+.}, at: [<80022508>] cpu_up+0x40/0x98
 #6:  (cpu_hotplug.lock){++++++}, at: [<80022268>] cpu_hotplug_begin+0x0/0xac
Quoted 4 lines

I guess that lockup is my fundamental issue, and with your patch the perf_rwsem manages to spread a transitive dependency on one of those locks all over the perf subsystem. I haven't considered that in great detail, however.

I found that I couldn't trigger the issue with v3.17, and I was able to bisect down to commit b2c4623dcd07af4b ("rcu: More on deadlock between CPU hotplug and expedited grace periods").

I'm currently stressing b2c4623dcd07af4b~1 to make sure my bisect hasn't mislead me.

Thanks,
Mark.

reviewer
#9 lore raw
Quoted 61 lines · Mark Rutland
Quoted 55 lines · Mark Rutland
Quoted 18 lines · Mark Rutland
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Peter Zijlstra

Its a bandaid at best :/ The problem is (again) that we changes event->ctx without any kind of serialization.

The issue came up before:

https://lkml.org/lkml/2014/9/5/397

In the end neither the CCI or CCN perf drivers migrate events on hotplug, so ARM is currently safe from the perf_pmu_migrate_context case, but I see that you fix the move_group handling too.

I had a go at testing this by hacking migration back into the CCI PMU driver (atop of v3.19-rc5), but I'm seeing lockups after a few minutes with my original test case (https://lkml.org/lkml/2014/9/1/569 with PMU_TYPE and PMU_EVENT fixed up).

Quoted 5 lines

I unfortunately don't have a suitable x86 box spare to run that on. Would someone be able to give it a spin on something with an uncore PMU?

I'll go and dig a bit further. I may just be hitting another latent issue on my board.

I'm able to trigger the lockups even without both your patch and the call to perf_pmu_migrate_context, so there is a latent issue.

On vanilla v3.19-rc5 and vanilla v3.18, I'm able to get my hotplug script hung when run concurrently with the test case against the CCI PMU driver (without migration). The v3.18 and v3.19-rc5 lockups are identical:

INFO: task hpall.sh:1506 blocked for more than 120 seconds.
      Not tainted 3.19.0-rc5 #9

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
hpall.sh D 804a6ffc 0 1506 1497 0x00000000
[<804a6ffc>] (__schedule) from [<80022308>] (cpu_hotplug_begin+0xa0/0xac) [<80022308>] (cpu_hotplug_begin) from [<8002236c>] (_cpu_up+0x24/0x180) [<8002236c>] (_cpu_up) from [<8002253c>] (cpu_up+0x74/0x98) [<8002253c>] (cpu_up) from [<802bce60>] (device_online+0x64/0x90) [<802bce60>] (device_online) from [<802bcef4>] (online_store+0x68/0x74) [<802bcef4>] (online_store) from [<8014059c>] (kernfs_fop_write+0xbc/0x1a0) [<8014059c>] (kernfs_fop_write) from [<800e71b0>] (vfs_write+0xa0/0x1ac) [<800e71b0>] (vfs_write) from [<800e7808>] (SyS_write+0x44/0x9c) [<800e7808>] (SyS_write) from [<8000e560>] (ret_fast_syscall+0x0/0x48) 7 locks held by hpall.sh/1506:

 #0:  (sb_writers#6){.+.+.+}, at: [<800e729c>] vfs_write+0x18c/0x1ac
 #1:  (&of->mutex){+.+.+.}, at: [<8014052c>] kernfs_fop_write+0x4c/0x1a0
 #2:  (s_active#15){.+.+.+}, at: [<80140534>] kernfs_fop_write+0x54/0x1a0
 #3:  (device_hotplug_lock){+.+.+.}, at: [<802bbe44>] lock_device_hotplug_sysfs+0xc/0x4c
 #4:  (&dev->mutex){......}, at: [<802bce14>] device_online+0x18/0x90
 #5:  (cpu_add_remove_lock){+.+.+.}, at: [<80022508>] cpu_up+0x40/0x98
 #6:  (cpu_hotplug.lock){++++++}, at: [<80022268>] cpu_hotplug_begin+0x0/0xac
Quoted 4 lines

I guess that lockup is my fundamental issue, and with your patch the perf_rwsem manages to spread a transitive dependency on one of those locks all over the perf subsystem. I haven't considered that in great detail, however.

Quoted 6 lines

I found that I couldn't trigger the issue with v3.17, and I was able to bisect down to commit b2c4623dcd07af4b ("rcu: More on deadlock between CPU hotplug and expedited grace periods").

I'm currently stressing b2c4623dcd07af4b~1 to make sure my bisect hasn't mislead me.

That seems to be solid, and I think I see what's going on.

The task doing hotplug (hpall.sh:1506) gets to cpu_hotplug_begin(), and sets cpu_hotplug.active_writer to current (I assume writes to this are protected by cpu_add_remove_lock from cpu_up()?). Then it loops, acquiring cpu_hotplug.lock and testing the refcount, and if non-zero dropping the lock and going into uninterruptible sleep, expecting to be woken by put_online_cpus().

Concurrently a task holding the refcount non-zero calls put_online_cpus(), and finds there to be contention on cpu_hotplug.lock. Thus it increments cpu_hotplug.puts_pending and goes of on its merry way, without trying to wake the writer.

So the writer is never woken and never gets to handle the non-zero cpu_hotplug.puts_pending.

I'm not sure what the right fix for that is. It looks like the writer could observe the change to puts_pending and so cpu_hotplug.active_writer could change under our feet unless we hold cpu_hotplug.lock. But holding that would reintroduce the deadlock b2c4623dcd07af4b was trying to avoid.

Any ideas?

Mark.

#10 lore raw
Quoted 86 lines · Mark Rutland
Quoted 61 lines · Mark Rutland
Quoted 55 lines · Mark Rutland
Quoted 18 lines · Mark Rutland
Quoted 7 lines · Peter Zijlstra
Quoted 6 lines · Peter Zijlstra

Its a bandaid at best :/ The problem is (again) that we changes event->ctx without any kind of serialization.

The issue came up before:

https://lkml.org/lkml/2014/9/5/397

In the end neither the CCI or CCN perf drivers migrate events on hotplug, so ARM is currently safe from the perf_pmu_migrate_context case, but I see that you fix the move_group handling too.

I had a go at testing this by hacking migration back into the CCI PMU driver (atop of v3.19-rc5), but I'm seeing lockups after a few minutes with my original test case (https://lkml.org/lkml/2014/9/1/569 with PMU_TYPE and PMU_EVENT fixed up).

Quoted 5 lines

I unfortunately don't have a suitable x86 box spare to run that on. Would someone be able to give it a spin on something with an uncore PMU?

I'll go and dig a bit further. I may just be hitting another latent issue on my board.

I'm able to trigger the lockups even without both your patch and the call to perf_pmu_migrate_context, so there is a latent issue.

On vanilla v3.19-rc5 and vanilla v3.18, I'm able to get my hotplug script hung when run concurrently with the test case against the CCI PMU driver (without migration). The v3.18 and v3.19-rc5 lockups are identical:

INFO: task hpall.sh:1506 blocked for more than 120 seconds.
      Not tainted 3.19.0-rc5 #9

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
hpall.sh D 804a6ffc 0 1506 1497 0x00000000
[<804a6ffc>] (__schedule) from [<80022308>] (cpu_hotplug_begin+0xa0/0xac) [<80022308>] (cpu_hotplug_begin) from [<8002236c>] (_cpu_up+0x24/0x180) [<8002236c>] (_cpu_up) from [<8002253c>] (cpu_up+0x74/0x98) [<8002253c>] (cpu_up) from [<802bce60>] (device_online+0x64/0x90) [<802bce60>] (device_online) from [<802bcef4>] (online_store+0x68/0x74) [<802bcef4>] (online_store) from [<8014059c>] (kernfs_fop_write+0xbc/0x1a0) [<8014059c>] (kernfs_fop_write) from [<800e71b0>] (vfs_write+0xa0/0x1ac) [<800e71b0>] (vfs_write) from [<800e7808>] (SyS_write+0x44/0x9c) [<800e7808>] (SyS_write) from [<8000e560>] (ret_fast_syscall+0x0/0x48) 7 locks held by hpall.sh/1506:

 #0:  (sb_writers#6){.+.+.+}, at: [<800e729c>] vfs_write+0x18c/0x1ac
 #1:  (&of->mutex){+.+.+.}, at: [<8014052c>] kernfs_fop_write+0x4c/0x1a0
 #2:  (s_active#15){.+.+.+}, at: [<80140534>] kernfs_fop_write+0x54/0x1a0
 #3:  (device_hotplug_lock){+.+.+.}, at: [<802bbe44>] lock_device_hotplug_sysfs+0xc/0x4c
 #4:  (&dev->mutex){......}, at: [<802bce14>] device_online+0x18/0x90
 #5:  (cpu_add_remove_lock){+.+.+.}, at: [<80022508>] cpu_up+0x40/0x98
 #6:  (cpu_hotplug.lock){++++++}, at: [<80022268>] cpu_hotplug_begin+0x0/0xac
Quoted 4 lines

I guess that lockup is my fundamental issue, and with your patch the perf_rwsem manages to spread a transitive dependency on one of those locks all over the perf subsystem. I haven't considered that in great detail, however.

Quoted 6 lines

I found that I couldn't trigger the issue with v3.17, and I was able to bisect down to commit b2c4623dcd07af4b ("rcu: More on deadlock between CPU hotplug and expedited grace periods").

I'm currently stressing b2c4623dcd07af4b~1 to make sure my bisect hasn't mislead me.

That seems to be solid, and I think I see what's going on.

The task doing hotplug (hpall.sh:1506) gets to cpu_hotplug_begin(), and sets cpu_hotplug.active_writer to current (I assume writes to this are protected by cpu_add_remove_lock from cpu_up()?). Then it loops, acquiring cpu_hotplug.lock and testing the refcount, and if non-zero dropping the lock and going into uninterruptible sleep, expecting to be woken by put_online_cpus().

Concurrently a task holding the refcount non-zero calls put_online_cpus(), and finds there to be contention on cpu_hotplug.lock. Thus it increments cpu_hotplug.puts_pending and goes of on its merry way, without trying to wake the writer.

So the writer is never woken and never gets to handle the non-zero cpu_hotplug.puts_pending.

Quoted 7 lines

I'm not sure what the right fix for that is. It looks like the writer could observe the change to puts_pending and so cpu_hotplug.active_writer could change under our feet unless we hold cpu_hotplug.lock. But holding that would reintroduce the deadlock b2c4623dcd07af4b was trying to avoid.

Any ideas?

You need 87af9e7ff9d90 (hotplugcpu: Avoid deadlocks by waking active_writer), which is in -rcu at:

git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git

With some luck, this will be in -tip soon, and hit mainline during the next merge window.

Thanx, Paul

reviewer
#11 lore raw

[...]

Quoted 64 lines
Quoted 56 lines
Quoted 32 lines
Quoted 27 lines

On vanilla v3.19-rc5 and vanilla v3.18, I'm able to get my hotplug script hung when run concurrently with the test case against the CCI PMU driver (without migration). The v3.18 and v3.19-rc5 lockups are identical:

INFO: task hpall.sh:1506 blocked for more than 120 seconds.
      Not tainted 3.19.0-rc5 #9

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
hpall.sh D 804a6ffc 0 1506 1497 0x00000000
[<804a6ffc>] (__schedule) from [<80022308>] (cpu_hotplug_begin+0xa0/0xac) [<80022308>] (cpu_hotplug_begin) from [<8002236c>] (_cpu_up+0x24/0x180) [<8002236c>] (_cpu_up) from [<8002253c>] (cpu_up+0x74/0x98) [<8002253c>] (cpu_up) from [<802bce60>] (device_online+0x64/0x90) [<802bce60>] (device_online) from [<802bcef4>] (online_store+0x68/0x74) [<802bcef4>] (online_store) from [<8014059c>] (kernfs_fop_write+0xbc/0x1a0) [<8014059c>] (kernfs_fop_write) from [<800e71b0>] (vfs_write+0xa0/0x1ac) [<800e71b0>] (vfs_write) from [<800e7808>] (SyS_write+0x44/0x9c) [<800e7808>] (SyS_write) from [<8000e560>] (ret_fast_syscall+0x0/0x48) 7 locks held by hpall.sh/1506:

 #0:  (sb_writers#6){.+.+.+}, at: [<800e729c>] vfs_write+0x18c/0x1ac
 #1:  (&of->mutex){+.+.+.}, at: [<8014052c>] kernfs_fop_write+0x4c/0x1a0
 #2:  (s_active#15){.+.+.+}, at: [<80140534>] kernfs_fop_write+0x54/0x1a0
 #3:  (device_hotplug_lock){+.+.+.}, at: [<802bbe44>] lock_device_hotplug_sysfs+0xc/0x4c
 #4:  (&dev->mutex){......}, at: [<802bce14>] device_online+0x18/0x90
 #5:  (cpu_add_remove_lock){+.+.+.}, at: [<80022508>] cpu_up+0x40/0x98
 #6:  (cpu_hotplug.lock){++++++}, at: [<80022268>] cpu_hotplug_begin+0x0/0xac
Quoted 4 lines

I guess that lockup is my fundamental issue, and with your patch the perf_rwsem manages to spread a transitive dependency on one of those locks all over the perf subsystem. I haven't considered that in great detail, however.

Quoted 6 lines

I found that I couldn't trigger the issue with v3.17, and I was able to bisect down to commit b2c4623dcd07af4b ("rcu: More on deadlock between CPU hotplug and expedited grace periods").

I'm currently stressing b2c4623dcd07af4b~1 to make sure my bisect hasn't mislead me.

That seems to be solid, and I think I see what's going on.

The task doing hotplug (hpall.sh:1506) gets to cpu_hotplug_begin(), and sets cpu_hotplug.active_writer to current (I assume writes to this are protected by cpu_add_remove_lock from cpu_up()?). Then it loops, acquiring cpu_hotplug.lock and testing the refcount, and if non-zero dropping the lock and going into uninterruptible sleep, expecting to be woken by put_online_cpus().

Concurrently a task holding the refcount non-zero calls put_online_cpus(), and finds there to be contention on cpu_hotplug.lock. Thus it increments cpu_hotplug.puts_pending and goes of on its merry way, without trying to wake the writer.

So the writer is never woken and never gets to handle the non-zero cpu_hotplug.puts_pending.

Quoted 7 lines

I'm not sure what the right fix for that is. It looks like the writer could observe the change to puts_pending and so cpu_hotplug.active_writer could change under our feet unless we hold cpu_hotplug.lock. But holding that would reintroduce the deadlock b2c4623dcd07af4b was trying to avoid.

Any ideas?

Quoted 7 lines

You need 87af9e7ff9d90 (hotplugcpu: Avoid deadlocks by waking active_writer), which is in -rcu at:

git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git

With some luck, this will be in -tip soon, and hit mainline during the next merge window.

Thanks Paul, that fixes the issue for me.

Peter, with that fix applied in addition to your patch, I don't see the CCI PMU code exploding around hotplug, even with event migration hacked into the driver.

Thanks,
Mark.

#12 lore raw
Quoted 69 lines · Mark Rutland

[...]

Quoted 67 lines
Quoted 56 lines
Quoted 32 lines
Quoted 27 lines

On vanilla v3.19-rc5 and vanilla v3.18, I'm able to get my hotplug script hung when run concurrently with the test case against the CCI PMU driver (without migration). The v3.18 and v3.19-rc5 lockups are identical:

INFO: task hpall.sh:1506 blocked for more than 120 seconds.
      Not tainted 3.19.0-rc5 #9

"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
hpall.sh D 804a6ffc 0 1506 1497 0x00000000
[<804a6ffc>] (__schedule) from [<80022308>] (cpu_hotplug_begin+0xa0/0xac) [<80022308>] (cpu_hotplug_begin) from [<8002236c>] (_cpu_up+0x24/0x180) [<8002236c>] (_cpu_up) from [<8002253c>] (cpu_up+0x74/0x98) [<8002253c>] (cpu_up) from [<802bce60>] (device_online+0x64/0x90) [<802bce60>] (device_online) from [<802bcef4>] (online_store+0x68/0x74) [<802bcef4>] (online_store) from [<8014059c>] (kernfs_fop_write+0xbc/0x1a0) [<8014059c>] (kernfs_fop_write) from [<800e71b0>] (vfs_write+0xa0/0x1ac) [<800e71b0>] (vfs_write) from [<800e7808>] (SyS_write+0x44/0x9c) [<800e7808>] (SyS_write) from [<8000e560>] (ret_fast_syscall+0x0/0x48) 7 locks held by hpall.sh/1506:

 #0:  (sb_writers#6){.+.+.+}, at: [<800e729c>] vfs_write+0x18c/0x1ac
 #1:  (&of->mutex){+.+.+.}, at: [<8014052c>] kernfs_fop_write+0x4c/0x1a0
 #2:  (s_active#15){.+.+.+}, at: [<80140534>] kernfs_fop_write+0x54/0x1a0
 #3:  (device_hotplug_lock){+.+.+.}, at: [<802bbe44>] lock_device_hotplug_sysfs+0xc/0x4c
 #4:  (&dev->mutex){......}, at: [<802bce14>] device_online+0x18/0x90
 #5:  (cpu_add_remove_lock){+.+.+.}, at: [<80022508>] cpu_up+0x40/0x98
 #6:  (cpu_hotplug.lock){++++++}, at: [<80022268>] cpu_hotplug_begin+0x0/0xac
Quoted 4 lines

I guess that lockup is my fundamental issue, and with your patch the perf_rwsem manages to spread a transitive dependency on one of those locks all over the perf subsystem. I haven't considered that in great detail, however.

Quoted 6 lines

I found that I couldn't trigger the issue with v3.17, and I was able to bisect down to commit b2c4623dcd07af4b ("rcu: More on deadlock between CPU hotplug and expedited grace periods").

I'm currently stressing b2c4623dcd07af4b~1 to make sure my bisect hasn't mislead me.

That seems to be solid, and I think I see what's going on.

The task doing hotplug (hpall.sh:1506) gets to cpu_hotplug_begin(), and sets cpu_hotplug.active_writer to current (I assume writes to this are protected by cpu_add_remove_lock from cpu_up()?). Then it loops, acquiring cpu_hotplug.lock and testing the refcount, and if non-zero dropping the lock and going into uninterruptible sleep, expecting to be woken by put_online_cpus().

Concurrently a task holding the refcount non-zero calls put_online_cpus(), and finds there to be contention on cpu_hotplug.lock. Thus it increments cpu_hotplug.puts_pending and goes of on its merry way, without trying to wake the writer.

So the writer is never woken and never gets to handle the non-zero cpu_hotplug.puts_pending.

Quoted 7 lines

I'm not sure what the right fix for that is. It looks like the writer could observe the change to puts_pending and so cpu_hotplug.active_writer could change under our feet unless we hold cpu_hotplug.lock. But holding that would reintroduce the deadlock b2c4623dcd07af4b was trying to avoid.

Any ideas?

You need 87af9e7ff9d90 (hotplugcpu: Avoid deadlocks by waking active_writer), which is in -rcu at:

Quoted 6 lines
Quoted 4 lines

git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git

With some luck, this will be in -tip soon, and hit mainline during the next merge window.

Thanks Paul, that fixes the issue for me.

Good to hear! As luck would have it, it is already in -tip, so I cannot apply your Tested-by. :-(

Thanx, Paul

Quoted 7 lines

Peter, with that fix applied in addition to your patch, I don't see the CCI PMU code exploding around hotplug, even with event migration hacked into the driver.

Thanks,
Mark.