tracing/ring-buffer: Some fixes to ring buffer readers
Subsystems (3)
Steven Rostedt
maintainercover tracing/ring-buffer: Some fixes to ring buffer readers
Three fixes to the ring buffer wake ups:
1) ring_buffer_wait() should not loop.
The ring_buffer_wait() does not have the full context (yet) on if it should loop or not. Just exit the loop as soon as its woken up and let the callers decide to loop or not (they already do, so it's a bit redundant).
2) Fix shortest_full field to be the smallest amount in the ring buffer that a waiter is waiting for. The "shortest_full" field is updated when a new waiter comes in and wants to wait for a smaller amount of data in the ring buffer than other waiters. But after all waiters are woken up, it's not reset, so if another waiter comes in wanting to wait for more data, it will be woken up when the ring buffer has a smaller amount from what the previous waiters were waiting for.
3) The wake up all waiters on close is incorrectly called frome .release() and not from .flush() so it will never wake up any waiters as the .release() will not get called until all .read() calls are finished. And the wakeup is for the waiters in those .read() calls.
Steven Rostedt (Google) (3):
ring-buffer: Fix waking up ring buffer readers
ring-buffer: Fix resetting of shortest_full
tracing: Use .flush() call to wake up readers
----
kernel/trace/ring_buffer.c | 169 ++++++++++++++++++++++++--------------------- kernel/trace/trace.c | 21 ++++-- 2 files changed, 106 insertions(+), 84 deletions(-)
Steven Rostedt
maintainerPATCH 1/3 ring-buffer: Fix waking up ring buffer readers
From: "Steven Rostedt (Google)" <[email protected]>
A task can wait on a ring buffer for when it fills up to a specific watermark. The writer will check the minimum watermark that waiters are waiting for and if the ring buffer is past that, it will wake up all the waiters.
The waiters are in a wait loop, and will first check if a signal is pending and then check if the ring buffer is at the desired level where it should break out of the loop.
If a file that uses a ring buffer closes, and there's threads waiting on the ring buffer, it needs to wake up those threads. To do this, a "wait_index" was used.
Before entering the wait loop, the waiter will read the wait_index. On wakeup, it will check if the wait_index is different than when it entered the loop, and will exit the loop if it is. The waker will only need to update the wait_index before waking up the waiters.
This had a couple of bugs. One trivial one and one broken by design.
The trivial bug was that the waiter checked the wait_index after the schedule() call. It had to be checked between the prepare_to_wait() and the schedule() which it was not.
The main bug is that the first check to set the default wait_index will always be outside the prepare_to_wait() and the schedule(). That's because the ring_buffer_wait() doesn't have enough context to know if it should break out of the loop.
The loop itself is not needed, because all the callers to the ring_buffer_wait() also has their own loop, as the callers have a better sense of what the context is to decide whether to break out of the loop or not.
Just have the ring_buffer_wait() block once, and if it gets woken up, exit the function and let the callers decide what to do next.
--- kernel/trace/ring_buffer.c | 139 ++++++++++++++++++------------------- 1 file changed, 68 insertions(+), 71 deletions(-)
kernel/trace/ring_buffer.c +68 -71
@@ -384,7 +384,6 @@ struct rb_irq_work {struct irq_work work;wait_queue_head_t waiters;wait_queue_head_t full_waiters; -long wait_index;bool waiters_pending;bool full_waiters_pending;bool wakeup_full;
@@ -798,14 +797,40 @@ void ring_buffer_wake_waiters(struct trace_buffer *buffer, int cpu) rbwork = &cpu_buffer->irq_work; } - rbwork->wait_index++; -/* make sure the waiters see the new index */ -smp_wmb (); -/* This can be called in any context */ irq_work_queue (&rbwork->work); } +static bool rb_watermark_hit (struct trace_buffer *buffer,int cpu,int full) +{ +struct ring_buffer_per_cpu *cpu_buffer; +bool ret = false; + +/* Reads of all CPUs always waits for any data */ +if (cpu ==RING_BUFFER_ALL_CPUS ) +return !ring_buffer_empty (buffer); + + cpu_buffer = buffer->buffers[cpu]; + +if (!ring_buffer_empty_cpu (buffer, cpu)) { +unsigned long flags; +bool pagebusy; + +if (!full) +return true; + +raw_spin_lock_irqsave (&cpu_buffer->reader_lock, flags); + pagebusy = cpu_buffer->reader_page == cpu_buffer->commit_page; + ret = !pagebusy &&full_hit (buffer, cpu, full); + +if (!cpu_buffer->shortest_full || + cpu_buffer->shortest_full > full) + cpu_buffer->shortest_full = full; +raw_spin_unlock_irqrestore (&cpu_buffer->reader_lock, flags); + } +return ret; +} +/** * ring_buffer_wait - wait for input to the ring buffer * @buffer: buffer to wait on
@@ -821,7 +846,6 @@ int ring_buffer_wait(struct trace_buffer *buffer, int cpu, int full)struct ring_buffer_per_cpu *cpu_buffer;DEFINE_WAIT (wait);struct rb_irq_work *work; -long wait_index;int ret =0 ;/*
@@ -840,81 +864,54 @@ int ring_buffer_wait(struct trace_buffer *buffer, int cpu, int full) work = &cpu_buffer->irq_work; } - wait_index =READ_ONCE (work->wait_index); - -while (true) { -if (full) -prepare_to_wait (&work->full_waiters, &wait,TASK_INTERRUPTIBLE ); -else -prepare_to_wait (&work->waiters, &wait,TASK_INTERRUPTIBLE ); - -/* -* The events can happen in critical sections where -* checking a work queue can cause deadlocks. -* After adding a task to the queue, this flag is set -* only to notify events to try to wake up the queue -* using irq_work. -* -* We don't clear it even if the buffer is no longer -* empty. The flag only causes the next event to run -* irq_work to do the work queue wake up. The worse -* that can happen if we race with !trace_empty() is that -* an event will cause an irq_work to try to wake up -* an empty queue. -* -* There's no reason to protect this flag either, as -* the work queue and irq_work logic will do the necessary -* synchronization for the wake ups. The only thing -* that is necessary is that the wake up happens after -* a task has been queued. It's OK for spurious wake ups. -*/ -if (full) - work->full_waiters_pending = true; -else - work->waiters_pending = true; - -if (signal_pending (current)) { - ret = -EINTR ; -break ; - } - -if (cpu ==RING_BUFFER_ALL_CPUS && !ring_buffer_empty (buffer)) -break ; - -if (cpu !=RING_BUFFER_ALL_CPUS && - !ring_buffer_empty_cpu (buffer, cpu)) { -unsigned long flags; -bool pagebusy; -bool done; - -if (!full) -break ; - -raw_spin_lock_irqsave (&cpu_buffer->reader_lock, flags); - pagebusy = cpu_buffer->reader_page == cpu_buffer->commit_page; - done = !pagebusy &&full_hit (buffer, cpu, full); +if (full) +prepare_to_wait (&work->full_waiters, &wait,TASK_INTERRUPTIBLE ); +else +prepare_to_wait (&work->waiters, &wait,TASK_INTERRUPTIBLE ); -if (!cpu_buffer->shortest_full || - cpu_buffer->shortest_full > full) - cpu_buffer->shortest_full = full; -raw_spin_unlock_irqrestore (&cpu_buffer->reader_lock, flags); -if (done) -break ; - } +/* +* The events can happen in critical sections where +* checking a work queue can cause deadlocks. +* After adding a task to the queue, this flag is set +* only to notify events to try to wake up the queue +* using irq_work. +* +* We don't clear it even if the buffer is no longer +* empty. The flag only causes the next event to run +* irq_work to do the work queue wake up. The worse +* that can happen if we race with !trace_empty() is that +* an event will cause an irq_work to try to wake up +* an empty queue. +* +* There's no reason to protect this flag either, as +* the work queue and irq_work logic will do the necessary +* synchronization for the wake ups. The only thing +* that is necessary is that the wake up happens after +* a task has been queued. It's OK for spurious wake ups. +*/ +if (full) + work->full_waiters_pending = true; +else + work->waiters_pending = true; -schedule (); +if (rb_watermark_hit (buffer, cpu, full)) + goto out; -/* Make sure to see the new wait index */ -smp_rmb (); -if (wait_index != work->wait_index) -break ; +if (signal_pending (current)) { + ret = -EINTR ; + goto out; } +schedule (); + out:if (full)finish_wait (&work->full_waiters, &wait);else finish_wait (&work->waiters, &wait); +if (!ret && !rb_watermark_hit (buffer, cpu, full) &&signal_pending (current)) + ret = -EINTR ; +return ret; }
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH 2/3 ring-buffer: Fix resetting of shortest_full
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
The "shortest_full" variable is used to keep track of the waiter that is waiting for the smallest amount on the ring buffer before being woken up. When a tasks waits on the ring buffer, it passes in a "full" value that is a percentage. 0 means wake up on any data. 1-100 means wake up from 1% to 100% full buffer.
As all waiters are on the same wait queue, the wake up happens for the waiter with the smallest percentage.
The problem is that the smallest_full on the cpu_buffer that stores the smallest amount doesn't get reset when all the waiters are woken up. It does get reset when the ring buffer is reset (echo > /sys/kernel/tracing/trace).
This means that tasks may be woken up more often then when they want to be. Instead, have the shortest_full field get reset just before waking up all the tasks. If the tasks wait again, they will update the shortest_full before sleeping.
Also add locking around setting of shortest_full in the poll logic, and change "work" to "rbwork" to match the variable name for rb_irq_work structures that are used in other places.
--- kernel/trace/ring_buffer.c | 30 +++++++++++++++++++++++------- 1 file changed, 23 insertions(+), 7 deletions(-)
kernel/trace/ring_buffer.c +23 -7
@@ -755,8 +755,19 @@ static void rb_wake_up_waiters(struct irq_work *work)wake_up_all (&rbwork->waiters);if (rbwork->full_waiters_pending || rbwork->wakeup_full) { +/* Only cpu_buffer sets the above flags */ +struct ring_buffer_per_cpu *cpu_buffer = + container_of(rbwork, struct ring_buffer_per_cpu, irq_work); + +/* Called from interrupt context */ +raw_spin_lock (&cpu_buffer->reader_lock); rbwork->wakeup_full = false; rbwork->full_waiters_pending = false; + +/* Waking up all waiters, they will reset the shortest full */ + cpu_buffer->shortest_full =0 ; +raw_spin_unlock (&cpu_buffer->reader_lock); +wake_up_all (&rbwork->full_waiters); } }
@@ -934,28 +945,33 @@ __poll_t ring_buffer_poll_wait(struct trace_buffer *buffer, int cpu, struct file *filp, poll_table *poll_table, int full) { struct ring_buffer_per_cpu *cpu_buffer; -struct rb_irq_work *work; +struct rb_irq_work *rbwork;if (cpu ==RING_BUFFER_ALL_CPUS ) { - work = &buffer->irq_work; + rbwork = &buffer->irq_work; full =0 ; }else {if (!cpumask_test_cpu (cpu, buffer->cpumask))return EPOLLERR ; cpu_buffer = buffer->buffers[cpu]; - work = &cpu_buffer->irq_work; + rbwork = &cpu_buffer->irq_work; }if (full) { -poll_wait (filp, &work->full_waiters, poll_table); - work->full_waiters_pending = true; +unsigned long flags; + +poll_wait (filp, &rbwork->full_waiters, poll_table); + +raw_spin_lock_irqsave (&cpu_buffer->reader_lock, flags); + rbwork->full_waiters_pending = true;if (!cpu_buffer->shortest_full || cpu_buffer->shortest_full > full) cpu_buffer->shortest_full = full; +raw_spin_unlock_irqrestore (&cpu_buffer->reader_lock, flags); }else { -poll_wait (filp, &work->waiters, poll_table); - work->waiters_pending = true; +poll_wait (filp, &rbwork->waiters, poll_table); + rbwork->waiters_pending = true; }/*
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH 3/3 tracing: Use .flush() call to wake up readers
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
The .release() function does not get called until all readers of a file descriptor are finished.
If a thread is blocked on reading a file descriptor in ring_buffer_wait(), and another thread closes the file descriptor, it will not wake up the other thread as ring_buffer_wake_waiters() is called by .release(), and that will not get called until the .read() is finished.
The issue originally showed up in trace-cmd, but the readers are actually other processes with their own file descriptors. So calling close() would wake up the other tasks because they are blocked on another descriptor then the one that was closed(). But there's other wake ups that solve that issue.
When a thread is blocked on a read, it can still hang even when another thread closed its descriptor.
This is what the .flush() callback is for. Have the .flush() wake up the readers.
--- kernel/trace/trace.c | 21 +++++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-)
kernel/trace/trace.c +15 -6
@@ -8393,6 +8393,20 @@ tracing_buffers_read(struct file *filp, char __user *ubuf,return size; } +static int tracing_buffers_flush (struct file *file,fl_owner_t id) +{ +struct ftrace_buffer_info *info = file->private_data; +struct trace_iterator *iter = &info->iter; + + iter->wait_index++; +/* Make sure the waiters see the new wait_index */ +smp_wmb (); + +ring_buffer_wake_waiters (iter->array_buffer->buffer, iter->cpu_file); + +return 0 ; +} +static int tracing_buffers_release (struct inode *inode,struct file *file) {struct ftrace_buffer_info *info = file->private_data;
@@ -8404,12 +8418,6 @@ static int tracing_buffers_release(struct inode *inode, struct file *file)__trace_array_put (iter->tr); - iter->wait_index++; -/* Make sure the waiters see the new wait_index */ -smp_wmb (); - -ring_buffer_wake_waiters (iter->array_buffer->buffer, iter->cpu_file); -if (info->spare)ring_buffer_free_read_page (iter->array_buffer->buffer, info->spare_cpu, info->spare);
@@ -8625,6 +8633,7 @@ static const struct file_operations tracing_buffers_fops = { .read = tracing_buffers_read, .poll = tracing_buffers_poll, .release = tracing_buffers_release, + .flush = tracing_buffers_flush, .splice_read = tracing_buffers_splice_read, .unlocked_ioctl = tracing_buffers_ioctl, .llseek = no_llseek,
Signature · 1 line
2.43.0