Skip to contentSkip to conversation

oprofile: fix race condition in event_buffer free

PATCH In review Robert Richter 2 participants 4 messages Oct 9, 2009 – Oct 12, 2009 1 message not in the archive

lore
Share & tools
lore
#1 lore raw

PATCH 1/2 oprofile: fix race condition in event_buffer free

In reply to a message not in the archive

From: David Rientjes <[email protected]>

Looking at the 2.6.31-rc9 code, it appears there is a race condition in the event_buffer cleanup code path (shutdown). This could lead to kernel panic as some CPUs may be operating on the event buffer AFTER it has been freed. The attached patch solves the problem and makes sure CPUs check if the buffer is not NULL before they access it as some may have been spinning on the mutex while the buffer was being freed.

The race may happen if the buffer is freed during pending reads. But it is not clear why there are races in add_event_entry() since all workqueues or handlers are canceled or flushed before the event buffer is freed.

---
 drivers/oprofile/event_buffer.c |   14 +++++++++++++-
 1 files changed, 13 insertions(+), 1 deletions(-)
drivers/oprofile/event_buffer.c +13 -1
index 2b7ae36..c38adb3 100644
@@ -41,6 +41,12 @@ static atomic_t buffer_ready = ATOMIC_INIT(0);
  */
 void add_event_entry(unsigned long value)
 {
+	/*
+	 * catch potential error
+	 */
+	if (!event_buffer)
+		return;
+
 	if (buffer_pos == buffer_size) {
 		atomic_inc(&oprofile_stats.event_lost_overflow);
 		return;
@@ -92,9 +98,10 @@ out:
 
 void free_event_buffer(void)
 {
+	mutex_lock(&buffer_mutex);
 	vfree(event_buffer);
-
 	event_buffer = NULL;
+	mutex_unlock(&buffer_mutex);
 }
 
 
@@ -167,6 +174,11 @@ static ssize_t event_buffer_read(struct file *file, char __user *buf,
 
 	mutex_lock(&buffer_mutex);
 
+	if (!event_buffer) {
+		retval = -EINTR;
+		goto out;
+	}
+
 	atomic_set(&buffer_ready, 0);
 
 	retval = -EFAULT;
Signature · 1 line
1.6.5.rc2
#4 lore raw

PATCH 2/2 oprofile: warn on freeing event buffer too early

In reply to a message not in the archive

A race shouldn't happen since all workqueues or handlers are canceled or flushed before the event buffer is freed. A warning is triggered now if the buffer is freed too early.

Also, this patch adds some comments about event buffer protection, reworks some code and adds code to clear buffer_pos during alloc and free of the event buffer.

Cc David Rientjes <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 drivers/oprofile/event_buffer.c |   25 +++++++++++++++----------
 1 files changed, 15 insertions(+), 10 deletions(-)
drivers/oprofile/event_buffer.c +15 -10
index c38adb3..5df60a6 100644
@@ -35,17 +35,22 @@ static size_t buffer_pos;
 /* atomic_t because wait_event checks it outside of buffer_mutex */
 static atomic_t buffer_ready = ATOMIC_INIT(0);
 
-/* Add an entry to the event buffer. When we
- * get near to the end we wake up the process
- * sleeping on the read() of the file.
+/*
+ * Add an entry to the event buffer. When we get near to the end we
+ * wake up the process sleeping on the read() of the file. To protect
+ * the event_buffer this function may only be called when buffer_mutex
+ * is set.
  */
 void add_event_entry(unsigned long value)
 {
 	/*
-	 * catch potential error
+	 * This shouldn't happen since all workqueues or handlers are
+	 * canceled or flushed before the event buffer is freed.
 	 */
-	if (!event_buffer)
+	if (!event_buffer) {
+		WARN_ON_ONCE(1);
 		return;
+	}
 
 	if (buffer_pos == buffer_size) {
 		atomic_inc(&oprofile_stats.event_lost_overflow);
@@ -75,7 +80,6 @@ void wake_up_buffer_waiter(void)
 
 int alloc_event_buffer(void)
 {
-	int err = -ENOMEM;
 	unsigned long flags;
 
 	spin_lock_irqsave(&oprofilefs_lock, flags);
@@ -86,13 +90,12 @@ int alloc_event_buffer(void)
 	if (buffer_watershed >= buffer_size)
 		return -EINVAL;
 
+	buffer_pos = 0;
 	event_buffer = vmalloc(sizeof(unsigned long) * buffer_size);
 	if (!event_buffer)
-		goto out;
+		return -ENOMEM;
 
-	err = 0;
-out:
-	return err;
+	return 0;
 }
 
 
@@ -100,6 +103,7 @@ void free_event_buffer(void)
 {
 	mutex_lock(&buffer_mutex);
 	vfree(event_buffer);
+	buffer_pos = 0;
 	event_buffer = NULL;
 	mutex_unlock(&buffer_mutex);
 }
@@ -174,6 +178,7 @@ static ssize_t event_buffer_read(struct file *file, char __user *buf,
 
 	mutex_lock(&buffer_mutex);
 
+	/* May happen if the buffer is freed during pending reads. */
 	if (!event_buffer) {
 		retval = -EINTR;
 		goto out;
Signature · 1 line
1.6.5.rc2
#2 lore raw
In reply to Robert Richter
Quoted 10 lines · Robert Richter

From: David Rientjes <[email protected]>

Looking at the 2.6.31-rc9 code, it appears there is a race condition in the event_buffer cleanup code path (shutdown). This could lead to kernel panic as some CPUs may be operating on the event buffer AFTER it has been freed. The attached patch solves the problem and makes sure CPUs check if the buffer is not NULL before they access it as some may have been spinning on the mutex while the buffer was being freed.

Quoted 8 lines

The race may happen if the buffer is freed during pending reads. But it is not clear why there are races in add_event_entry() since all workqueues or handlers are canceled or flushed before the event buffer is freed.

Ingo,

you can also pull the patches from here:

git://git.kernel.org/pub/scm/linux/kernel/git/rric/oprofile.git urgent

Thanks.

-Robert

Signature · 3 lines
Advanced Micro Devices, Inc.
Operating System Research Center
email: [email protected]
#3 lore raw
Quoted 20 lines · Robert Richter
Quoted 10 lines · Robert Richter

From: David Rientjes <[email protected]>

Looking at the 2.6.31-rc9 code, it appears there is a race condition in the event_buffer cleanup code path (shutdown). This could lead to kernel panic as some CPUs may be operating on the event buffer AFTER it has been freed. The attached patch solves the problem and makes sure CPUs check if the buffer is not NULL before they access it as some may have been spinning on the mutex while the buffer was being freed.

Quoted 8 lines

The race may happen if the buffer is freed during pending reads. But it is not clear why there are races in add_event_entry() since all workqueues or handlers are canceled or flushed before the event buffer is freed.

Quoted 7 lines

Ingo,

you can also pull the patches from here:

git://git.kernel.org/pub/scm/linux/kernel/git/rric/oprofile.git urgent

Thanks.

Pulled, thanks Robert!

Ingo