Skip to contentSkip to conversation

ftrace/perf_event leak

DISCUSSION Avi Kivity 6 participants 14 messages Sep 1, 2010 – Sep 9, 2010

Subsystems (2)
lore
Share & tools
lore

Avi Kivity

author
#0 lore raw

I recently added perf_event support to kvm_stat, to display kvm
tracepoints as statistics (I'd like to fold this to tools/perf eventually, but that's another story). However I'm seeing a resource leak - after I quit the tool, there are quite a few references into the kvm module:

kvm_intel 43655 0
kvm 272984 269 kvm_intel

The tool is just a python script that reads /sys/kernel/debug/tracing/events/kvm to find out which events are available, uses perf_event_open() to create one group per cpu to which a lot of events are attached. The only special thing I can think of is that we use an ioctl to attach a filter to many perf_event descriptors.

You can find the source at http://git.kernel.org/?p=virt/kvm/qemu-kvm.git;a=blob_plain;f=kvm/kvm_stat;hb=5bd5f131b50cb373ff4e2a3632c6dad00a1f0b55. All it needs are the kvm modules loaded; no need to actually run a guest. Run as root.

Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
#1 lore raw
Quoted 15 lines · Avi Kivity

I recently added perf_event support to kvm_stat, to display kvm tracepoints as statistics (I'd like to fold this to tools/perf eventually, but that's another story). However I'm seeing a resource leak - after I quit the tool, there are quite a few references into the kvm module:

kvm_intel 43655 0
kvm 272984 269 kvm_intel

The tool is just a python script that reads /sys/kernel/debug/tracing/events/kvm to find out which events are available, uses perf_event_open() to create one group per cpu to which a lot of events are attached. The only special thing I can think of is that we use an ioctl to attach a filter to many perf_event descriptors.

Quoted 5 lines

You can find the source at
http://git.kernel.org/?p=virt/kvm/qemu-kvm.git;a=blob_plain;f=kvm/kvm_stat;hb=5bd5f131b50cb373ff4e2a3632c6dad00a1f0b55.
All it needs are the kvm modules loaded; no need to actually run a guest. Run as root.

Does something like the below cure that?

I seem to remember C doesn't make any promises about the order of logic statements, hence we need to explicitly pull out that try_module_get() so that it evaluates after the rest of the conditions.

---
 kernel/trace/trace_event_perf.c |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)
kernel/trace/trace_event_perf.c +6 -4
index 000e6e8..35051f2 100644
@@ -88,10 +88,11 @@ int perf_trace_init(struct perf_event *p_event)
 	mutex_lock(&event_mutex);
 	list_for_each_entry(tp_event, &ftrace_events, list) {
 		if (tp_event->event.type == event_id &&
-		    tp_event->class && tp_event->class->reg &&
-		    try_module_get(tp_event->mod)) {
-			ret = perf_trace_event_init(tp_event, p_event);
-			break;
+		    tp_event->class && tp_event->class->reg) {
+		    	if (try_module_get(tp_event->mod)) {
+				ret = perf_trace_event_init(tp_event, p_event);
+				break;
+			}
 		}
 	}
 	mutex_unlock(&event_mutex);
@@ -138,6 +139,7 @@ void perf_trace_destroy(struct perf_event *p_event)
 
 	free_percpu(tp_event->perf_events);
 	tp_event->perf_events = NULL;
+	module_put(tp_event->mod);
 
 	if (!--total_ref_count) {
 		for (i = 0; i < 4; i++) {
author
#2 lore raw
Quoted 3 lines · Peter Zijlstra

Does something like the below cure that?

Unfortunately not.

Quoted 3 lines

I seem to remember C doesn't make any promises about the order of logic statements, hence we need to explicitly pull out that try_module_get() so that it evaluates after the rest of the conditions.

&& and || are executed in order (so you can write

   if (p && p->x)

).

Signature · 2 lines
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
#3 lore raw
Quoted 7 lines · Avi Kivity
Quoted 3 lines · Peter Zijlstra

Does something like the below cure that?

Unfortunately not.

Then try this:

The bug should be caused by commit 1c024eca51fdc965290acf342ae16a476c2189d0.

---

kernel/trace/trace_event_perf.c +1 -0
index 8a2b73f..9d1d1f2 100644
@@ -161,6 +161,7 @@ void perf_trace_destroy(struct perf_event *p_event)
 		}
 	}
 out:
+	module_put(tp_event->mod);
 	mutex_unlock(&event_mutex);
 }
 
#4 lore raw

kernel/trace/trace_event_perf.c void perf_trace_destroy(struct perf_event *p_event) · line 166

Quoted 12 lines · Li Zefan

---

kernel/trace/trace_event_perf.c +1 -0
index 8a2b73f..9d1d1f2 100644
@@ -161,6 +161,7 @@ void perf_trace_destroy(struct perf_event *p_event)
 		}
 	}
 out:
+	module_put(tp_event->mod);
 	mutex_unlock(&event_mutex);
 }

Ah, indeed, we so that try_module_get() for each reference to the tracepoint, I guess we also should do the below, in case the registration fails.

kernel/trace/trace_event_perf.c +2 -0
index 000e6e8..31cc4cb 100644
@@ -91,6 +91,8 @@ int perf_trace_init(struct perf_event *p_event)
 		    tp_event->class && tp_event->class->reg &&
 		    try_module_get(tp_event->mod)) {
 			ret = perf_trace_event_init(tp_event, p_event);
+			if (ret)
+				module_put(tp_event->mod);
 			break;
 		}
 	}
Tested-by Avi Kivity
#6 lore raw
Quoted 6 lines · Avi Kivity
Quoted 2 lines · Li Zefan

Then try this:

Thanks, queued as:

---
Subject: perf, trace: Fix module leak
From: Li Zefan <[email protected]>
Date: Wed Sep 01 12:58:43 CEST 2010

Commit 1c024eca (perf, trace: Optimize tracepoints by using per-tracepoint-per-cpu hlist to track events) caused a module refcount leak.

LKML-Reference: <[email protected]>

---
 kernel/trace/trace_event_perf.c |    3 +++
 1 file changed, 3 insertions(+)
kernel/trace/trace_event_perf.c +3 -0
Index: linux-2.6/kernel/trace/trace_event_perf.c
===================================================================
@@ -91,6 +91,8 @@ int perf_trace_init(struct perf_event *p
 		    tp_event->class && tp_event->class->reg &&
 		    try_module_get(tp_event->mod)) {
 			ret = perf_trace_event_init(tp_event, p_event);
+			if (ret)
+				module_put(tp_event->mod);
 			break;
 		}
 	}
@@ -147,6 +149,7 @@ void perf_trace_destroy(struct perf_even
 		}
 	}
 out:
+	module_put(tp_event->mod);
 	mutex_unlock(&event_mutex);
 }
 
#7 lore raw
Quoted 15 lines · Peter Zijlstra
Quoted 6 lines · Avi Kivity
Quoted 2 lines · Li Zefan

Then try this:

Thanks, queued as:

---
Subject: perf, trace: Fix module leak
From: Li Zefan <[email protected]>
Date: Wed Sep 01 12:58:43 CEST 2010

Quoted 7 lines

Commit 1c024eca (perf, trace: Optimize tracepoints by using per-tracepoint-per-cpu hlist to track events) caused a module refcount leak.

LKML-Reference: <[email protected]>

Do we need this for -stable too?

Ingo

#8 lore raw
In reply to Peter Zijlstra
Quoted 41 lines · Peter Zijlstra
Quoted 6 lines · Avi Kivity
Quoted 2 lines · Li Zefan

Then try this:

Thanks, queued as:

---
Subject: perf, trace: Fix module leak
From: Li Zefan <[email protected]>
Date: Wed Sep 01 12:58:43 CEST 2010

Commit 1c024eca (perf, trace: Optimize tracepoints by using per-tracepoint-per-cpu hlist to track events) caused a module refcount leak.

LKML-Reference: <[email protected]>

---
 kernel/trace/trace_event_perf.c |    3 +++
 1 file changed, 3 insertions(+)

Index: linux-2.6/kernel/trace/trace_event_perf.c
===================================================================

kernel/trace/trace_event_perf.c +2 -0
@@ -91,6 +91,8 @@ int perf_trace_init(struct perf_event *p
 		    tp_event->class && tp_event->class->reg &&
 		    try_module_get(tp_event->mod)) {
 			ret = perf_trace_event_init(tp_event, p_event);
+			if (ret)
+				module_put(tp_event->mod);
 			break;
 		}
 	}

kernel/trace/trace_event_perf.c void perf_trace_destroy(struct perf_even · line 155

Quoted 7 lines
kernel/trace/trace_event_perf.c +1 -0
@@ -147,6 +149,7 @@ void perf_trace_destroy(struct perf_even
 		}
 	}
 out:
+	module_put(tp_event->mod);
 	mutex_unlock(&event_mutex);
 }
 

Thanks for fixing this.

However, can we split this in two patches to ease the backport?

The lack of a module_put() after perf_trace_init() failure is there for a while (the backport needs to start in 2.6.32).

But the lack of a module_put in the destroy path needs a .35 backport only.

maintainer
#9 lore raw
Quoted 8 lines · Frederic Weisbecker

Thanks for fixing this.

However, can we split this in two patches to ease the backport?

The lack of a module_put() after perf_trace_init() failure is there for a while (the backport needs to start in 2.6.32).

But the lack of a module_put in the destroy path needs a .35 backport only.

I don't think it really needs two patches. Just notify stable (and Greg KH in particular) about the backport requirements. Greg can handle it ;)

-- Steve

#10 lore raw
Quoted 6 lines · Steven Rostedt
Quoted 4 lines · Frederic Weisbecker

Thanks for fixing this.

However, can we split this in two patches to ease the backport?

Quoted 8 lines
Quoted 4 lines

The lack of a module_put() after perf_trace_init() failure is there for a while (the backport needs to start in 2.6.32).

But the lack of a module_put in the destroy path needs a .35 backport only.

I don't think it really needs two patches. Just notify stable (and Greg KH in particular) about the backport requirements. Greg can handle it ;)

Well, Greg certainly has more than enough to handle, so if there's different chunks with different -stable vectors then it would be most helpful to him to split things up!

Manually trying to split up patches is both error-prone and stress-inducing.

Thanks,

Ingo

#11 lore raw
In reply to Frederic Weisbecker
Quoted 40 lines
Quoted 30 lines

Subject: perf, trace: Fix module leak
From: Li Zefan <[email protected]>
Date: Wed Sep 01 12:58:43 CEST 2010

Commit 1c024eca (perf, trace: Optimize tracepoints by using per-tracepoint-per-cpu hlist to track events) caused a module refcount leak.

LKML-Reference: <[email protected]>

---
 kernel/trace/trace_event_perf.c |    3 +++
 1 file changed, 3 insertions(+)

Index: linux-2.6/kernel/trace/trace_event_perf.c
===================================================================

kernel/trace/trace_event_perf.c +2 -0
@@ -91,6 +91,8 @@ int perf_trace_init(struct perf_event *p
 		    tp_event->class && tp_event->class->reg &&
 		    try_module_get(tp_event->mod)) {
 			ret = perf_trace_event_init(tp_event, p_event);
+			if (ret)
+				module_put(tp_event->mod);
 			break;
 		}
 	}

kernel/trace/trace_event_perf.c void perf_trace_destroy(struct perf_even · line 155

Quoted 7 lines
kernel/trace/trace_event_perf.c +1 -0
@@ -147,6 +149,7 @@ void perf_trace_destroy(struct perf_even
 		}
 	}
 out:
+	module_put(tp_event->mod);
 	mutex_unlock(&event_mutex);
 }
 

Thanks for fixing this.

Quoted 4 lines

However, can we split this in two patches to ease the backport?

The lack of a module_put() after perf_trace_init() failure is there for a while (the backport needs to start in 2.6.32).

The failure should be a rare case, I don't think this has to be backported?

Quoted 3 lines

But the lack of a module_put in the destroy path needs a .35 backport only.

#12 lore raw

[tip:perf/core] perf, trace: Fix module leak

In reply to Li Zefan
Commit-ID:  9cb627d5f38830ca19aa0dca52d1d3a633018bf7
Gitweb:     http://git.kernel.org/tip/9cb627d5f38830ca19aa0dca52d1d3a633018bf7
Author:     Li Zefan <[email protected]>
AuthorDate: Wed, 1 Sep 2010 12:58:43 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Thu, 9 Sep 2010 20:38:51 +0200

perf, trace: Fix module leak

Commit 1c024eca (perf, trace: Optimize tracepoints by using per-tracepoint-per-cpu hlist to track events) caused a module refcount leak.

LKML-Reference: <[email protected]>

---
 kernel/trace/trace_event_perf.c |    3 +++
 1 files changed, 3 insertions(+), 0 deletions(-)
kernel/trace/trace_event_perf.c +3 -0
index 000e6e8..31cc4cb 100644
@@ -91,6 +91,8 @@ int perf_trace_init(struct perf_event *p_event)
 		    tp_event->class && tp_event->class->reg &&
 		    try_module_get(tp_event->mod)) {
 			ret = perf_trace_event_init(tp_event, p_event);
+			if (ret)
+				module_put(tp_event->mod);
 			break;
 		}
 	}
@@ -146,6 +148,7 @@ void perf_trace_destroy(struct perf_event *p_event)
 		}
 	}
 out:
+	module_put(tp_event->mod);
 	mutex_unlock(&event_mutex);
 }
 
#13 lore raw
In reply to Avi Kivity
Quoted 6 lines · Avi Kivity
Quoted 3 lines · Peter Zijlstra

Does something like the below cure that?

Unfortunately not.

Hrmm,.. bugger, don't use modules is my motto.. still I'll have another look.