ring-buffer: Allow for rescheduling when removing pages
Subsystems (2)
PATCH 1/1 ring-buffer: Allow for rescheduling when removing pages
When reducing ring buffer size, pages are removed by scheduling a work item on each CPU for the corresponding CPU ring buffer. After the pages are removed from ring buffer linked list, the pages are free()d in a tight loop. The loop does not give up CPU until all pages are removed. In a worst case behavior, when lot of pages are to be freed, it can cause system stall.
After the pages are removed from the list, the free() can happen while the work is rescheduled. Add a check for need_sched() within the loop to prevent the system hangup.
--- kernel/trace/ring_buffer.c | 3 +++ 1 file changed, 3 insertions(+)
kernel/trace/ring_buffer.c +3 -0
@@ -1546,6 +1546,9 @@ rb_remove_pages(struct ring_buffer_per_cpu *cpu_buffer, unsigned long nr_pages) tmp_iter_page = first_page; do { + if (need_resched()) + schedule(); + to_remove_page = tmp_iter_page; rb_inc_page(cpu_buffer, &tmp_iter_page);
Signature · 1 line
2.19.0.rc2.392.g5ba43deb5a-goog
Steven Rostedt
maintainerQuoted 22 lines · On Fri, 7 Sep 2018 11:21:31 -0700 Vaibhav Nagarnaik <[email protected]>…
When reducing ring buffer size, pages are removed by scheduling a work item on each CPU for the corresponding CPU ring buffer. After the pages are removed from ring buffer linked list, the pages are free()d in a tight loop. The loop does not give up CPU until all pages are removed. In a worst case behavior, when lot of pages are to be freed, it can cause system stall.
After the pages are removed from the list, the
free()can happen while the work is rescheduled. Add a check forneed_sched()within the loop to prevent the system hangup.Reported-by Jason Behmer <[email protected]>Signed-off-by Vaibhav Nagarnaik <[email protected]>--- kernel/trace/ring_buffer.c | 3 +++ 1 file changed, 3 insertions(+)kernel/trace/ring_buffer.c +0
-0
kernel/trace/ring_buffer.c rb_remove_pages(struct ring_buffer_per_cpu *cpu_buffer, unsigned long nr_pages) · line 1551
Quoted 6 lines
kernel/trace/ring_buffer.c +3
-0@@ -1546,6 +1546,9 @@ rb_remove_pages(struct ring_buffer_per_cpu *cpu_buffer, unsigned long nr_pages) tmp_iter_page = first_page; do { + if (need_resched()) + schedule(); +
Hi, thanks for the patch, but the proper way to do this is to stick in:
cond_resched();
And that should solve it for you. Want to send in another patch?
-- Steve
Quoted 3 lines
kernel/trace/ring_buffer.c +0
-0to_remove_page = tmp_iter_page;rb_inc_page (cpu_buffer, &tmp_iter_page);
Quoted 35 lines · Steven Rostedt
Quoted 22 lines · On Fri, 7 Sep 2018 11:21:31 -0700 Vaibhav Nagarnaik <[email protected]>…
When reducing ring buffer size, pages are removed by scheduling a work item on each CPU for the corresponding CPU ring buffer. After the pages are removed from ring buffer linked list, the pages are free()d in a tight loop. The loop does not give up CPU until all pages are removed. In a worst case behavior, when lot of pages are to be freed, it can cause system stall.
After the pages are removed from the list, the
free()can happen while the work is rescheduled. Add a check forneed_sched()within the loop to prevent the system hangup.Reported-by Jason Behmer <[email protected]>Signed-off-by Vaibhav Nagarnaik <[email protected]>--- kernel/trace/ring_buffer.c | 3 +++ 1 file changed, 3 insertions(+)kernel/trace/ring_buffer.c +0
-0kernel/trace/ring_buffer.c rb_remove_pages(struct ring_buffer_per_cpu *cpu_buffer, unsigned long nr_pages) · line 1551
Quoted 6 lines
kernel/trace/ring_buffer.c +3
-0@@ -1546,6 +1546,9 @@ rb_remove_pages(struct ring_buffer_per_cpu *cpu_buffer, unsigned long nr_pages) tmp_iter_page = first_page; do { + if (need_resched()) + schedule(); +Hi, thanks for the patch, but the proper way to do this is to stick in:
Quoted 3 lines
cond_resched();And that should solve it for you. Want to send in another patch?
Sounds good. Let me update the patch. Testing it first though.
Vaibhav
Quoted 6 lines
-- Steve
Quoted 3 lines
kernel/trace/ring_buffer.c +0
-0to_remove_page = tmp_iter_page;rb_inc_page (cpu_buffer, &tmp_iter_page);
PATCH 1/1 ring-buffer: Allow for rescheduling when removing pages
In reply to Vaibhav NagarnaikWhen reducing ring buffer size, pages are removed by scheduling a work item on each CPU for the corresponding CPU ring buffer. After the pages are removed from ring buffer linked list, the pages are free()d in a tight loop. The loop does not give up CPU until all pages are removed. In a worst case behavior, when lot of pages are to be freed, it can cause system stall.
After the pages are removed from the list, the free() can happen while the work is rescheduled. Call cond_resched() in the loop to prevent the system hangup.
--- kernel/trace/ring_buffer.c | 2 ++ 1 file changed, 2 insertions(+)
kernel/trace/ring_buffer.c +2 -0
@@ -1546,6 +1546,8 @@ rb_remove_pages(struct ring_buffer_per_cpu *cpu_buffer, unsigned long nr_pages) tmp_iter_page = first_page; do { + cond_resched(); + to_remove_page = tmp_iter_page; rb_inc_page(cpu_buffer, &tmp_iter_page);
Signature · 1 line
2.19.0.rc2.392.g5ba43deb5a-goog
Hi Steven,
Does the patch look good? Can this be picked up in the next rc?
Vaibhav
Quoted 34 lines · Vaibhav Nagarnaik
When reducing ring buffer size, pages are removed by scheduling a work item on each CPU for the corresponding CPU ring buffer. After the pages are removed from ring buffer linked list, the pages are free()d in a tight loop. The loop does not give up CPU until all pages are removed. In a worst case behavior, when lot of pages are to be freed, it can cause system stall.
After the pages are removed from the list, the
free()can happen while the work is rescheduled. Callcond_resched()in the loop to prevent the system hangup.Reported-by Jason Behmer <[email protected]>Signed-off-by Vaibhav Nagarnaik <[email protected]>--- kernel/trace/ring_buffer.c | 2 ++ 1 file changed, 2 insertions(+)kernel/trace/ring_buffer.c +2
-1@@ -1546,6 +1546,8 @@ rb_remove_pages(struct ring_buffer_per_cpu *cpu_buffer, unsigned long nr_pages) tmp_iter_page = first_page; do { + cond_resched(); + to_remove_page = tmp_iter_page; rb_inc_page(cpu_buffer, &tmp_iter_page); --2.19.0.rc2.392.g5ba43deb5a-goog
Steven Rostedt
maintainerQuoted 4 lines · On Mon, 17 Sep 2018 13:53:05 -0700 Vaibhav Nagarnaik <[email protected]>…
Hi Steven,
Does the patch look good? Can this be picked up in the next rc?
Yes it's fine. I can pick it up. Does it need to be marked for stable?
Thanks!
-- Steve
Quoted 10 lines · Steven Rostedt
Quoted 4 lines · On Mon, 17 Sep 2018 13:53:05 -0700 Vaibhav Nagarnaik <[email protected]>…
Hi Steven,
Does the patch look good? Can this be picked up in the next rc?
Yes it's fine. I can pick it up. Does it need to be marked for stable?
Thanks.
It'd be great to be marked for stable, since older kernels are also affected. But I can't judge if it crosses the threshold as a bug fix.
Vaibhav
Quoted 3 lines
Thanks!
-- Steve
Steven Rostedt
maintainerQuoted 10 lines · On Mon, 17 Sep 2018 14:02:59 -0700 Vaibhav Nagarnaik <[email protected]>…
Quoted 9 lines · Steven Rostedt
Quoted 4 lines · On Mon, 17 Sep 2018 13:53:05 -0700 Vaibhav Nagarnaik <[email protected]>…
Hi Steven,
Does the patch look good? Can this be picked up in the next rc?
Quoted 7 lines
Quoted 1 line
Yes it's fine. I can pick it up. Does it need to be marked for stable?
Thanks.
It'd be great to be marked for stable, since older kernels are also affected. But I can't judge if it crosses the threshold as a bug fix.
When backporting to stable I use two (admittedly subjective) metrics. One is the severity of the bug (this is low), the other is the risk of adding it (this is even lower). If the severity divided by the risk is high enough, I tag it for stable. Because the risk is so low, I have no problems with tagging it.
Note, it will run through lots of tests before it gets out the door.
-- Steve
Quoted 22 lines · Steven Rostedt
Quoted 10 lines · On Mon, 17 Sep 2018 14:02:59 -0700 Vaibhav Nagarnaik <[email protected]>…
Quoted 9 lines · Steven Rostedt
Quoted 4 lines · On Mon, 17 Sep 2018 13:53:05 -0700 Vaibhav Nagarnaik <[email protected]>…
Hi Steven,
Does the patch look good? Can this be picked up in the next rc?
Quoted 7 lines
Quoted 1 line
Yes it's fine. I can pick it up. Does it need to be marked for stable?
Thanks.
It'd be great to be marked for stable, since older kernels are also affected. But I can't judge if it crosses the threshold as a bug fix.
Quoted 7 lines
When backporting to stable I use two (admittedly subjective) metrics. One is the severity of the bug (this is low), the other is the risk of adding it (this is even lower). If the severity divided by the risk is high enough, I tag it for stable. Because the risk is so low, I have no problems with tagging it.
Note, it will run through lots of tests before it gets out the door.
I think it's low risk as well. Let's mark it for stable.
Thanks
Vaibhav
Quoted 1 line
-- Steve