Skip to contentSkip to conversation

perf: optimize perf_output_lock

PATCH In review Peter Zijlstra 2 participants 2 messages Nov 16, 2009 perf events

lore
Share & tools
lore

Peter Zijlstra

author maintainer
#0 lore raw

PATCH 1/1 perf: optimize perf_output_lock

Subject: perf: optimize perf_output_lock
From: Peter Zijlstra <[email protected]>
Date: Mon Nov 16 12:45:14 CET 2009

The purpuse of perf_output_{un,}lock() is to:

1) avoid publishing incomplete data
[ possible when publishing a head that is ahead of an entry
that is still being written ]

2) guarantee fwd progress
[ a simple refcount on pending writers doesn't need to drop to
0, making it so would end up implementing something like forced quiecent states of RCU ]

To satisfy the above without undue complexity it serializes between CPUs, this means that a pending writer can only be the same cpu in a nested context, and since (under normal operation) a cpu always makes progress we're good -- if the head is only published when the bottom most writer completes.

Now we don't need to disable IRQs in order to serialize between CPUs, disabling preemption ought to be sufficient, esp since we already deal with nesting due to NMIs.

This avoids expensive IRQ ops.

---
 include/linux/perf_event.h |    1 -
 kernel/perf_event.c        |   21 +++++++++++----------
 2 files changed, 11 insertions(+), 11 deletions(-)
include/linux/perf_event.h +0 -1
Index: linux-2.6/include/linux/perf_event.h
===================================================================
@@ -739,7 +739,6 @@ struct perf_output_handle {
 	int				nmi;
 	int				sample;
 	int				locked;
-	unsigned long			flags;
 };
 
 #ifdef CONFIG_PERF_EVENTS
kernel/perf_event.c +11 -10
Index: linux-2.6/kernel/perf_event.c
===================================================================
@@ -2696,20 +2696,21 @@ static void perf_output_wakeup(struct pe
 static void perf_output_lock(struct perf_output_handle *handle)
 {
 	struct perf_mmap_data *data = handle->data;
-	int cpu;
+	int cur, cpu = get_cpu();
 
 	handle->locked = 0;
 
-	local_irq_save(handle->flags);
-	cpu = smp_processor_id();
-
-	if (in_nmi() && atomic_read(&data->lock) == cpu)
-		return;
+	for (;;) {
+		cur = atomic_cmpxchg(&data->lock, -1, cpu);
+		if (cur == -1) {
+			handle->locked = 1;
+			break;
+		}
+		if (cur == cpu)
+			break;
 
-	while (atomic_cmpxchg(&data->lock, -1, cpu) != -1)
 		cpu_relax();
-
-	handle->locked = 1;
+	}
 }
 
 static void perf_output_unlock(struct perf_output_handle *handle)
@@ -2755,7 +2756,7 @@ again:
 	if (atomic_xchg(&data->wakeup, 0))
 		perf_output_wakeup(handle);
 out:
-	local_irq_restore(handle->flags);
+	put_cpu();
 }
 
 void perf_output_copy(struct perf_output_handle *handle,
#1 lore raw

[tip:perf/core] perf_event: Optimize perf_output_lock()

Commit-ID:  559fdc3c1b624edb1933a875022fe7e27934d11c
Gitweb:     http://git.kernel.org/tip/559fdc3c1b624edb1933a875022fe7e27934d11c
Author:     Peter Zijlstra <[email protected]>
AuthorDate: Mon, 16 Nov 2009 12:45:14 +0100
Committer:  Ingo Molnar <[email protected]>
CommitDate: Mon, 16 Nov 2009 13:27:45 +0100

perf_event: Optimize perf_output_lock()

The purpose of perf_output_{un,}lock() is to:

1) avoid publishing incomplete data
[ possible when publishing a head that is ahead of an entry
that is still being written ]

2) guarantee fwd progress
[ a simple refcount on pending writers doesn't need to drop to
0, making it so would end up implementing something like forced quiecent states of RCU ]

To satisfy the above without undue complexity it serializes between CPUs, this means that a pending writer can only be the same cpu in a nested context, and since (under normal operation) a cpu always makes progress we're good -- if the head is only published when the bottom most writer completes.

Now we don't need to disable IRQs in order to serialize between CPUs, disabling preemption ought to be sufficient, esp since we already deal with nesting due to NMIs.

This avoids potentially expensive (and needless) local IRQ disable/enable ops.

Cc Paul Mackerras <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Frederic Weisbecker <[email protected]>

LKML-Reference: <1258373161.26714.254.camel@laptop>

---
 include/linux/perf_event.h |    1 -
 kernel/perf_event.c        |   21 +++++++++++----------
 2 files changed, 11 insertions(+), 11 deletions(-)
include/linux/perf_event.h +0 -1
index df4e73e..7f87563 100644
@@ -714,7 +714,6 @@ struct perf_output_handle {
 	int				nmi;
 	int				sample;
 	int				locked;
-	unsigned long			flags;
 };
 
 #ifdef CONFIG_PERF_EVENTS
kernel/perf_event.c +11 -10
index 6f4ed3b..3256e36 100644
@@ -2674,20 +2674,21 @@ static void perf_output_wakeup(struct perf_output_handle *handle)
 static void perf_output_lock(struct perf_output_handle *handle)
 {
 	struct perf_mmap_data *data = handle->data;
-	int cpu;
+	int cur, cpu = get_cpu();
 
 	handle->locked = 0;
 
-	local_irq_save(handle->flags);
-	cpu = smp_processor_id();
-
-	if (in_nmi() && atomic_read(&data->lock) == cpu)
-		return;
+	for (;;) {
+		cur = atomic_cmpxchg(&data->lock, -1, cpu);
+		if (cur == -1) {
+			handle->locked = 1;
+			break;
+		}
+		if (cur == cpu)
+			break;
 
-	while (atomic_cmpxchg(&data->lock, -1, cpu) != -1)
 		cpu_relax();
-
-	handle->locked = 1;
+	}
 }
 
 static void perf_output_unlock(struct perf_output_handle *handle)
@@ -2733,7 +2734,7 @@ again:
 	if (atomic_xchg(&data->wakeup, 0))
 		perf_output_wakeup(handle);
 out:
-	local_irq_restore(handle->flags);
+	put_cpu();
 }
 
 void perf_output_copy(struct perf_output_handle *handle,