Skip to contentSkip to conversation

ring-buffer: Allow for rescheduling when removing pages

PATCH In review Vaibhav Nagarnaik 2 participants 9 messages Sep 7, 2018 – Sep 17, 2018

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ring-buffer: Allow for rescheduling when removing pages

When reducing ring buffer size, pages are removed by scheduling a work item on each CPU for the corresponding CPU ring buffer. After the pages are removed from ring buffer linked list, the pages are free()d in a tight loop. The loop does not give up CPU until all pages are removed. In a worst case behavior, when lot of pages are to be freed, it can cause system stall.

After the pages are removed from the list, the free() can happen while the work is rescheduled. Add a check for need_sched() within the loop to prevent the system hangup.

---
 kernel/trace/ring_buffer.c | 3 +++
 1 file changed, 3 insertions(+)
kernel/trace/ring_buffer.c +3 -0
index 1d92d4a982fd..bc1789df7c53 100644
@@ -1546,6 +1546,9 @@ rb_remove_pages(struct ring_buffer_per_cpu *cpu_buffer, unsigned long nr_pages)
 	tmp_iter_page = first_page;
 
 	do {
+		if (need_resched())
+			schedule();
+
 		to_remove_page = tmp_iter_page;
 		rb_inc_page(cpu_buffer, &tmp_iter_page);
 
Signature · 1 line
2.19.0.rc2.392.g5ba43deb5a-goog
maintainer
#1 lore raw
Quoted 22 lines · On Fri, 7 Sep 2018 11:21:31 -0700 Vaibhav Nagarnaik <[email protected]>…

When reducing ring buffer size, pages are removed by scheduling a work item on each CPU for the corresponding CPU ring buffer. After the pages are removed from ring buffer linked list, the pages are free()d in a tight loop. The loop does not give up CPU until all pages are removed. In a worst case behavior, when lot of pages are to be freed, it can cause system stall.

After the pages are removed from the list, the free() can happen while the work is rescheduled. Add a check for need_sched() within the loop to prevent the system hangup.

---
 kernel/trace/ring_buffer.c | 3 +++
 1 file changed, 3 insertions(+)
kernel/trace/ring_buffer.c +0 -0
index 1d92d4a982fd..bc1789df7c53 100644

kernel/trace/ring_buffer.c rb_remove_pages(struct ring_buffer_per_cpu *cpu_buffer, unsigned long nr_pages) · line 1551

Quoted 6 lines
kernel/trace/ring_buffer.c +3 -0
@@ -1546,6 +1546,9 @@ rb_remove_pages(struct ring_buffer_per_cpu *cpu_buffer, unsigned long nr_pages)
 	tmp_iter_page = first_page;
 
 	do {
+		if (need_resched())
+			schedule();
+

Hi, thanks for the patch, but the proper way to do this is to stick in:

		cond_resched();

And that should solve it for you. Want to send in another patch?

-- Steve

Quoted 3 lines
kernel/trace/ring_buffer.c +0 -0
 		to_remove_page = tmp_iter_page;
 		rb_inc_page(cpu_buffer, &tmp_iter_page);
 
#2 lore raw
Quoted 35 lines · Steven Rostedt
Quoted 22 lines · On Fri, 7 Sep 2018 11:21:31 -0700 Vaibhav Nagarnaik <[email protected]>…

When reducing ring buffer size, pages are removed by scheduling a work item on each CPU for the corresponding CPU ring buffer. After the pages are removed from ring buffer linked list, the pages are free()d in a tight loop. The loop does not give up CPU until all pages are removed. In a worst case behavior, when lot of pages are to be freed, it can cause system stall.

After the pages are removed from the list, the free() can happen while the work is rescheduled. Add a check for need_sched() within the loop to prevent the system hangup.

---
 kernel/trace/ring_buffer.c | 3 +++
 1 file changed, 3 insertions(+)
kernel/trace/ring_buffer.c +0 -0
index 1d92d4a982fd..bc1789df7c53 100644

kernel/trace/ring_buffer.c rb_remove_pages(struct ring_buffer_per_cpu *cpu_buffer, unsigned long nr_pages) · line 1551

Quoted 6 lines
kernel/trace/ring_buffer.c +3 -0
@@ -1546,6 +1546,9 @@ rb_remove_pages(struct ring_buffer_per_cpu *cpu_buffer, unsigned long nr_pages)
      tmp_iter_page = first_page;
 
      do {
+             if (need_resched())
+                     schedule();
+

Hi, thanks for the patch, but the proper way to do this is to stick in:

Quoted 3 lines
                cond_resched();

And that should solve it for you. Want to send in another patch?

Sounds good. Let me update the patch. Testing it first though.

Vaibhav

Quoted 6 lines

-- Steve

Quoted 3 lines
kernel/trace/ring_buffer.c +0 -0
              to_remove_page = tmp_iter_page;
              rb_inc_page(cpu_buffer, &tmp_iter_page);
 
#3 lore raw

PATCH 1/1 ring-buffer: Allow for rescheduling when removing pages

In reply to Vaibhav Nagarnaik

When reducing ring buffer size, pages are removed by scheduling a work item on each CPU for the corresponding CPU ring buffer. After the pages are removed from ring buffer linked list, the pages are free()d in a tight loop. The loop does not give up CPU until all pages are removed. In a worst case behavior, when lot of pages are to be freed, it can cause system stall.

After the pages are removed from the list, the free() can happen while the work is rescheduled. Call cond_resched() in the loop to prevent the system hangup.

---
 kernel/trace/ring_buffer.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/trace/ring_buffer.c +2 -0
index 1d92d4a982fd..65bd4616220d 100644
@@ -1546,6 +1546,8 @@ rb_remove_pages(struct ring_buffer_per_cpu *cpu_buffer, unsigned long nr_pages)
 	tmp_iter_page = first_page;
 
 	do {
+		cond_resched();
+
 		to_remove_page = tmp_iter_page;
 		rb_inc_page(cpu_buffer, &tmp_iter_page);
 
Signature · 1 line
2.19.0.rc2.392.g5ba43deb5a-goog
#4 lore raw

Hi Steven,

Does the patch look good? Can this be picked up in the next rc?

Vaibhav

Quoted 34 lines · Vaibhav Nagarnaik

When reducing ring buffer size, pages are removed by scheduling a work item on each CPU for the corresponding CPU ring buffer. After the pages are removed from ring buffer linked list, the pages are free()d in a tight loop. The loop does not give up CPU until all pages are removed. In a worst case behavior, when lot of pages are to be freed, it can cause system stall.

After the pages are removed from the list, the free() can happen while the work is rescheduled. Call cond_resched() in the loop to prevent the system hangup.

---
 kernel/trace/ring_buffer.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/trace/ring_buffer.c +2 -1
index 1d92d4a982fd..65bd4616220d 100644
@@ -1546,6 +1546,8 @@ rb_remove_pages(struct ring_buffer_per_cpu *cpu_buffer, unsigned long nr_pages)
        tmp_iter_page = first_page;
 
        do {
+               cond_resched();
+
                to_remove_page = tmp_iter_page;
                rb_inc_page(cpu_buffer, &tmp_iter_page);
 
--

2.19.0.rc2.392.g5ba43deb5a-goog

maintainer
#5 lore raw
Quoted 4 lines · On Mon, 17 Sep 2018 13:53:05 -0700 Vaibhav Nagarnaik <[email protected]>…

Hi Steven,

Does the patch look good? Can this be picked up in the next rc?

Yes it's fine. I can pick it up. Does it need to be marked for stable?

Thanks!

-- Steve

#6 lore raw
Quoted 10 lines · Steven Rostedt
Quoted 4 lines · On Mon, 17 Sep 2018 13:53:05 -0700 Vaibhav Nagarnaik <[email protected]>…

Hi Steven,

Does the patch look good? Can this be picked up in the next rc?

Yes it's fine. I can pick it up. Does it need to be marked for stable?

Thanks.

It'd be great to be marked for stable, since older kernels are also affected. But I can't judge if it crosses the threshold as a bug fix.

Vaibhav

Quoted 3 lines

Thanks!

-- Steve

maintainer
#7 lore raw
Quoted 10 lines · On Mon, 17 Sep 2018 14:02:59 -0700 Vaibhav Nagarnaik <[email protected]>…
Quoted 9 lines · Steven Rostedt
Quoted 4 lines · On Mon, 17 Sep 2018 13:53:05 -0700 Vaibhav Nagarnaik <[email protected]>…

Hi Steven,

Does the patch look good? Can this be picked up in the next rc?

Quoted 7 lines
Quoted 1 line

Yes it's fine. I can pick it up. Does it need to be marked for stable?

Thanks.

It'd be great to be marked for stable, since older kernels are also affected. But I can't judge if it crosses the threshold as a bug fix.

When backporting to stable I use two (admittedly subjective) metrics. One is the severity of the bug (this is low), the other is the risk of adding it (this is even lower). If the severity divided by the risk is high enough, I tag it for stable. Because the risk is so low, I have no problems with tagging it.

Note, it will run through lots of tests before it gets out the door.

-- Steve

#8 lore raw
Quoted 22 lines · Steven Rostedt
Quoted 10 lines · On Mon, 17 Sep 2018 14:02:59 -0700 Vaibhav Nagarnaik <[email protected]>…
Quoted 9 lines · Steven Rostedt
Quoted 4 lines · On Mon, 17 Sep 2018 13:53:05 -0700 Vaibhav Nagarnaik <[email protected]>…

Hi Steven,

Does the patch look good? Can this be picked up in the next rc?

Quoted 7 lines
Quoted 1 line

Yes it's fine. I can pick it up. Does it need to be marked for stable?

Thanks.

It'd be great to be marked for stable, since older kernels are also affected. But I can't judge if it crosses the threshold as a bug fix.

Quoted 7 lines

When backporting to stable I use two (admittedly subjective) metrics. One is the severity of the bug (this is low), the other is the risk of adding it (this is even lower). If the severity divided by the risk is high enough, I tag it for stable. Because the risk is so low, I have no problems with tagging it.

Note, it will run through lots of tests before it gets out the door.

I think it's low risk as well. Let's mark it for stable.

Thanks

Vaibhav

Quoted 1 line

-- Steve