Skip to contentSkip to conversation

trace: Fix race in trace_open and buffer resize call

PATCH In review Gaurav Kohli 2 participants 7 messages Sep 24, 2020 – Oct 5, 2020

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 trace: Fix race in trace_open and buffer resize call

Below race can come, if trace_open and resize of
cpu buffer is running parallely on different cpus
CPUX CPUY

				    ring_buffer_resize
				    atomic_read(&buffer->resize_disabled)

tracing_open
tracing_reset_online_cpus
ring_buffer_reset_cpu
rb_reset_cpu
rb_update_pages remove/insert pages
resetting pointer
This race can cause data abort or some times infinte loop in rb_remove_pages and rb_insert_pages while checking pages for sanity.

---
Changes since v0:
-Addressed Steven's review comments.

kernel/trace/ring_buffer.c +10 -0
index 93ef0ab..15bf28b 100644
@@ -4866,6 +4866,9 @@ void ring_buffer_reset_cpu(struct trace_buffer *buffer, int cpu)
 	if (!cpumask_test_cpu(cpu, buffer->cpumask))
 		return;
 
+	/* prevent another thread from changing buffer sizes */
+	mutex_lock(&buffer->mutex);
+
 	atomic_inc(&cpu_buffer->resize_disabled);
 	atomic_inc(&cpu_buffer->record_disabled);
 
@@ -4876,6 +4879,8 @@ void ring_buffer_reset_cpu(struct trace_buffer *buffer, int cpu)
 
 	atomic_dec(&cpu_buffer->record_disabled);
 	atomic_dec(&cpu_buffer->resize_disabled);
+
+	mutex_unlock(&buffer->mutex);
 }
 EXPORT_SYMBOL_GPL(ring_buffer_reset_cpu);
 
@@ -4889,6 +4894,9 @@ void ring_buffer_reset_online_cpus(struct trace_buffer *buffer)
 	struct ring_buffer_per_cpu *cpu_buffer;
 	int cpu;
 
+	/* prevent another thread from changing buffer sizes */
+	mutex_lock(&buffer->mutex);
+
 	for_each_online_buffer_cpu(buffer, cpu) {
 		cpu_buffer = buffer->buffers[cpu];
 
@@ -4907,6 +4915,8 @@ void ring_buffer_reset_online_cpus(struct trace_buffer *buffer)
 		atomic_dec(&cpu_buffer->record_disabled);
 		atomic_dec(&cpu_buffer->resize_disabled);
 	}
+
+	mutex_unlock(&buffer->mutex);
 }
 
 /**
Signature · 2 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
#1 lore raw

Hi Steven,

please let us know, if below looks good to you or need modifications.

Thanks
Gaurav

Quoted 65 lines · Gaurav Kohli

Below race can come, if trace_open and resize of
cpu buffer is running parallely on different cpus
CPUX CPUY

				    ring_buffer_resize
				    atomic_read(&buffer->resize_disabled)

tracing_open
tracing_reset_online_cpus
ring_buffer_reset_cpu
rb_reset_cpu
rb_update_pages remove/insert pages
resetting pointer
This race can cause data abort or some times infinte loop in rb_remove_pages and rb_insert_pages while checking pages for sanity.

---
Changes since v0:
-Addressed Steven's review comments.

kernel/trace/ring_buffer.c +10 -0
index 93ef0ab..15bf28b 100644
@@ -4866,6 +4866,9 @@ void ring_buffer_reset_cpu(struct trace_buffer *buffer, int cpu)
  	if (!cpumask_test_cpu(cpu, buffer->cpumask))
  		return;
 
+	/* prevent another thread from changing buffer sizes */
+	mutex_lock(&buffer->mutex);
+
  	atomic_inc(&cpu_buffer->resize_disabled);
  	atomic_inc(&cpu_buffer->record_disabled);
 
@@ -4876,6 +4879,8 @@ void ring_buffer_reset_cpu(struct trace_buffer *buffer, int cpu)
 
  	atomic_dec(&cpu_buffer->record_disabled);
  	atomic_dec(&cpu_buffer->resize_disabled);
+
+	mutex_unlock(&buffer->mutex);
  }
  EXPORT_SYMBOL_GPL(ring_buffer_reset_cpu);
 
@@ -4889,6 +4894,9 @@ void ring_buffer_reset_online_cpus(struct trace_buffer *buffer)
  	struct ring_buffer_per_cpu *cpu_buffer;
  	int cpu;
 
+	/* prevent another thread from changing buffer sizes */
+	mutex_lock(&buffer->mutex);
+
  	for_each_online_buffer_cpu(buffer, cpu) {
  		cpu_buffer = buffer->buffers[cpu];
 
@@ -4907,6 +4915,8 @@ void ring_buffer_reset_online_cpus(struct trace_buffer *buffer)
  		atomic_dec(&cpu_buffer->record_disabled);
  		atomic_dec(&cpu_buffer->resize_disabled);
  	}
+
+	mutex_unlock(&buffer->mutex);
  }
 
  /**
Signature · 3 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project.
maintainer
#2 lore raw
Quoted 3 lines · On Mon, 5 Oct 2020 10:09:34 +0530 Gaurav Kohli <[email protected]> wrote:

Hi Steven,

please let us know, if below looks good to you or need modifications.

Strange, I don't have your original email in my inbox. I do have it in my LKML folder, but that's way too big for me to read. I checked my server logs. I found where I received this from LKML, but there's no log that I received this directly.

How are you sending out your patches? If it doesn't land in my inbox, I'll never see it.

-- Steve

maintainer
#3 lore raw
Quoted 7 lines · On Mon, 5 Oct 2020 10:25:15 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 3 lines · On Mon, 5 Oct 2020 10:09:34 +0530 Gaurav Kohli <[email protected]> wrote:

Hi Steven,

please let us know, if below looks good to you or need modifications.

Quoted 8 lines

Strange, I don't have your original email in my inbox. I do have it in my LKML folder, but that's way too big for me to read. I checked my server logs. I found where I received this from LKML, but there's no log that I received this directly.

How are you sending out your patches? If it doesn't land in my inbox, I'll never see it.

BTW, this is the second time this has happened to me. The first time I assumed it may have been me accidentally deleting it. Now I'm thinking it's on your end.

I have yet to received a patch from you directly. Last time I had to pull it from my LKML folder after you replied to me (letting me know you sent it).

-- Steve

#4 lore raw
Quoted 23 lines · Steven Rostedt
Quoted 7 lines · On Mon, 5 Oct 2020 10:25:15 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 3 lines · On Mon, 5 Oct 2020 10:09:34 +0530 Gaurav Kohli <[email protected]> wrote:

Hi Steven,

please let us know, if below looks good to you or need modifications.

Quoted 8 lines

Strange, I don't have your original email in my inbox. I do have it in my LKML folder, but that's way too big for me to read. I checked my server logs. I found where I received this from LKML, but there's no log that I received this directly.

How are you sending out your patches? If it doesn't land in my inbox, I'll never see it.

BTW, this is the second time this has happened to me. The first time I assumed it may have been me accidentally deleting it. Now I'm thinking it's on your end.

Quoted 7 lines

I have yet to received a patch from you directly. Last time I had to pull it from my LKML folder after you replied to me (letting me know you sent it).

-- Steve

Hi Steven,

I am using normal git send-email(never saw problem with this), Not sure what is wrong. In my older mail i have kept you in to and rest in cc.

Let me try to resent it.

Signature · 3 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project.
maintainer
#5 lore raw
Quoted 6 lines · On Mon, 5 Oct 2020 21:59:02 +0530 Gaurav Kohli <[email protected]> wrote:

Hi Steven,

I am using normal git send-email(never saw problem with this), Not sure what is wrong. In my older mail i have kept you in to and rest in cc.

Let me try to resent it.

The Cc is working (I got it in my LKML box), but I don't see any connection in my server. Note, the [email protected] is a server at my home.

-- Steve

#6 lore raw
Quoted 8 lines · Steven Rostedt
Quoted 5 lines · On Mon, 5 Oct 2020 21:59:02 +0530 Gaurav Kohli <[email protected]> wrote:

Hi Steven,

I am using normal git send-email(never saw problem with this), Not sure what is wrong. In my older mail i have kept you in to and rest in cc.

Quoted 7 lines
Quoted 1 line

Let me try to resent it.

The Cc is working (I got it in my LKML box), but I don't see any connection in my server. Note, the [email protected] is a server at my home.

-- Steve

Thanks for update, i will verify my account settings and will send my patch tomorrow.

Signature · 3 lines
Qualcomm India Private Limited, on behalf of Qualcomm Innovation Center,
Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project.