Skip to contentSkip to conversation

ring-buffer: Fix poll wakeup logic

PATCH Superseded Steven Rostedt 2 participants 4 messages Mar 12, 2024 – Mar 13, 2024

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

cover ring-buffer: Fix poll wakeup logic

After making a slight change to wakeups in ring_buffer_wait() the system would hang. Spending several hours going on a wild goose chase I found that the change only triggered the bug because it changed the timings. The bug was there before the update but never was triggered.

The poll code has:

  rbwork->full_waiters_pending = true;
  if (!cpu_buffer->shortest_full ||
      cpu_buffer->shortest_full > full)
         cpu_buffer->shortest_full = full;

The writer will see full_waiters_pending and check if the ring buffer is filled over the percentage of the shortest_full value. If it is, it calls an irq_work to wake up all the waiters.

But the code could get into a circular loop:

CPU 0 CPU 1
----- -----
[ Poll ]
[ shortest_full = 0 ]

   rbwork->full_waiters_pending = true;
                                          if (rbwork->full_waiters_pending &&
                                              [ buffer percent ] > shortest_full) {
                                                 rbwork->wakeup_full = true;
                                                 [ queue_irqwork ]
   cpu_buffer->shortest_full = full;
                                          [ IRQ work ]
                                          if (rbwork->wakeup_full) {
                                                cpu_buffer->shortest_full = 0;
                                                wakeup poll waiters;
  [woken]
   if ([ buffer percent ] > full)
      break;
   rbwork->full_waiters_pending = true;
                                          if (rbwork->full_waiters_pending &&
                                              [ buffer percent ] > shortest_full) {
                                                 rbwork->wakeup_full = true;
                                                 [ queue_irqwork ]
   cpu_buffer->shortest_full = full;
                                          [ IRQ work ]
                                          if (rbwork->wakeup_full) {
                                                cpu_buffer->shortest_full = 0;
                                                wakeup poll waiters;

[woken]

[ Wash, rinse, repeat! ]

The race was triggered when running:

trace-cmd record -p function -m 5000

Which enables function tracing and then creates two files it is writing into where each is 2500K in size. The -m is a "max file size". When trace-cmd writes 2500K to one file it then switches to the other, erasing the old data. To do this, trace-cmd switches between both poll and the reader using both methods of wake up. The change to the reader wakeup was able to change the way the poll was woken to trigger this bug.

The second patch is a clean up and also a way to consolidate the logic of the shortest_full. The read wakeup uses rb_watermark_hit for both full wakeups and !full wakeups. But since poll uses the same logic for full wakeups it can just call that function with full set.

Steven Rostedt (Google) (2):
ring-buffer: Fix full_waiters_pending in poll
ring-buffer: Reuse rb_watermark_hit() for the poll logic

----

 kernel/trace/ring_buffer.c | 30 +++++++++++++++++++-----------
 1 file changed, 19 insertions(+), 11 deletions(-)
author maintainer
#1 lore raw

PATCH 1/2 ring-buffer: Fix full_waiters_pending in poll

From: "Steven Rostedt (Google)" <[email protected]>

If a reader of the ring buffer is doing a poll, and waiting for the ring buffer to hit a specific watermark, there could be a case where it gets into an infinite ping-pong loop.

The poll code has:

  rbwork->full_waiters_pending = true;
  if (!cpu_buffer->shortest_full ||
      cpu_buffer->shortest_full > full)
         cpu_buffer->shortest_full = full;

The writer will see full_waiters_pending and check if the ring buffer is filled over the percentage of the shortest_full value. If it is, it calls an irq_work to wake up all the waiters.

But the code could get into a circular loop:

	CPU 0					CPU 1
	-----					-----
 [ Poll ]
   [ shortest_full = 0 ]
   rbwork->full_waiters_pending = true;
					  if (rbwork->full_waiters_pending &&
					      [ buffer percent ] > shortest_full) {
					         rbwork->wakeup_full = true;
					         [ queue_irqwork ]
   cpu_buffer->shortest_full = full;
					  [ IRQ work ]
					  if (rbwork->wakeup_full) {
					        cpu_buffer->shortest_full = 0;
					        wakeup poll waiters;
  [woken]
   if ([ buffer percent ] > full)
      break;
   rbwork->full_waiters_pending = true;
					  if (rbwork->full_waiters_pending &&
					      [ buffer percent ] > shortest_full) {
					         rbwork->wakeup_full = true;
					         [ queue_irqwork ]
   cpu_buffer->shortest_full = full;
					  [ IRQ work ]
					  if (rbwork->wakeup_full) {
					        cpu_buffer->shortest_full = 0;
					        wakeup poll waiters;

[woken]

[ Wash, rinse, repeat! ]

In the poll, the shortest_full needs to be set before the full_pending_waiters, as once that is set, the writer will compare the current shortest_full (which is incorrect) to decide to call the irq_work, which will reset the shortest_full (expecting the readers to update it).

Also move the setting of full_waiters_pending after the check if the ring buffer has the required percentage filled. There's no reason to tell the writer to wake up waiters if there are no waiters.

Fixes 42fb0a1e84ff5 ("tracing/ring-buffer: Have polling block on watermark")
---
 kernel/trace/ring_buffer.c | 27 ++++++++++++++++++++-------
 1 file changed, 20 insertions(+), 7 deletions(-)
kernel/trace/ring_buffer.c +20 -7
index aa332ace108b..adfe603a769b 100644
@@ -964,16 +964,32 @@ __poll_t ring_buffer_poll_wait(struct trace_buffer *buffer, int cpu,
 		poll_wait(filp, &rbwork->full_waiters, poll_table);
 
 		raw_spin_lock_irqsave(&cpu_buffer->reader_lock, flags);
-		rbwork->full_waiters_pending = true;
 		if (!cpu_buffer->shortest_full ||
 		    cpu_buffer->shortest_full > full)
 			cpu_buffer->shortest_full = full;
 		raw_spin_unlock_irqrestore(&cpu_buffer->reader_lock, flags);
-	} else {
-		poll_wait(filp, &rbwork->waiters, poll_table);
-		rbwork->waiters_pending = true;
+		if (full_hit(buffer, cpu, full))
+			return EPOLLIN | EPOLLRDNORM;
+		/*
+		 * Only allow full_waiters_pending update to be seen after
+		 * the shortest_full is set. If the writer sees the
+		 * full_waiters_pending flag set, it will compare the
+		 * amount in the ring buffer to shortest_full. If the amount
+		 * in the ring buffer is greater than the shortest_full
+		 * percent, it will call the irq_work handler to wake up
+		 * this list. The irq_handler will reset shortest_full
+		 * back to zero. That's done under the reader_lock, but
+		 * the below smp_mb() makes sure that the update to
+		 * full_waiters_pending doesn't leak up into the above.
+		 */
+		smp_mb();
+		rbwork->full_waiters_pending = true;
+		return 0;
 	}
 
+	poll_wait(filp, &rbwork->waiters, poll_table);
+	rbwork->waiters_pending = true;
+
 	/*
 	 * There's a tight race between setting the waiters_pending and
 	 * checking if the ring buffer is empty.  Once the waiters_pending bit
@@ -989,9 +1005,6 @@ __poll_t ring_buffer_poll_wait(struct trace_buffer *buffer, int cpu,
 	 */
 	smp_mb();
 
-	if (full)
-		return full_hit(buffer, cpu, full) ? EPOLLIN | EPOLLRDNORM : 0;
-
 	if ((cpu == RING_BUFFER_ALL_CPUS && !ring_buffer_empty(buffer)) ||
 	    (cpu != RING_BUFFER_ALL_CPUS && !ring_buffer_empty_cpu(buffer, cpu)))
 		return EPOLLIN | EPOLLRDNORM;
Signature · 1 line
2.43.0
author maintainer
#2 lore raw

PATCH 2/2 ring-buffer: Reuse rb_watermark_hit() for the poll logic

In reply to Steven Rostedt

From: "Steven Rostedt (Google)" <[email protected]>

The check for knowing if the poll should wait or not is basically the exact same logic as rb_watermark_hit(). The only difference is that rb_watermark_hit() also handles the !full case. But for the full case, the logic is the same. Just call that instead of duplicating the code in ring_buffer_poll_wait().

---
 kernel/trace/ring_buffer.c | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)
kernel/trace/ring_buffer.c +7 -12
index adfe603a769b..6ef763f57c66 100644
@@ -963,21 +963,16 @@ __poll_t ring_buffer_poll_wait(struct trace_buffer *buffer, int cpu,
 
 		poll_wait(filp, &rbwork->full_waiters, poll_table);
 
-		raw_spin_lock_irqsave(&cpu_buffer->reader_lock, flags);
-		if (!cpu_buffer->shortest_full ||
-		    cpu_buffer->shortest_full > full)
-			cpu_buffer->shortest_full = full;
-		raw_spin_unlock_irqrestore(&cpu_buffer->reader_lock, flags);
-		if (full_hit(buffer, cpu, full))
+		if (rb_watermark_hit(buffer, cpu, full))
 			return EPOLLIN | EPOLLRDNORM;
 		/*
 		 * Only allow full_waiters_pending update to be seen after
-		 * the shortest_full is set. If the writer sees the
-		 * full_waiters_pending flag set, it will compare the
-		 * amount in the ring buffer to shortest_full. If the amount
-		 * in the ring buffer is greater than the shortest_full
-		 * percent, it will call the irq_work handler to wake up
-		 * this list. The irq_handler will reset shortest_full
+		 * the shortest_full is set (in rb_watermark_hit). If the
+		 * writer sees the full_waiters_pending flag set, it will
+		 * compare the amount in the ring buffer to shortest_full.
+		 * If the amount in the ring buffer is greater than the
+		 * shortest_full percent, it will call the irq_work handler
+		 * to wake up this list. The irq_handler will reset shortest_full
 		 * back to zero. That's done under the reader_lock, but
 		 * the below smp_mb() makes sure that the update to
 		 * full_waiters_pending doesn't leak up into the above.
Signature · 1 line
2.43.0
#3 lore raw

Re: [PATCH 2/2] ring-buffer: Reuse rb_watermark_hit() for the poll logic

kernel test robot noticed the following build warnings:

Hi Steven,

kernel test robot noticed the following build warnings:

[auto build test WARNING on linus/master]
[also build test WARNING on v6.8 next-20240312]
[cannot apply to akpm-mm/mm-everything]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Steven-Rostedt/ring-buffer-Fix-full_waiters_pending-in-poll/20240312-195518
base:   linus/master
patch link:    https://lore.kernel.org/r/20240312115543.500445373%40goodmis.org
patch subject: [PATCH 2/2] ring-buffer: Reuse rb_watermark_hit() for the poll logic
config: arc-defconfig (https://download.01.org/0day-ci/archive/20240313/[email protected]/config)
compiler: arc-elf-gcc (GCC) 13.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240313/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

kernel/trace/ring_buffer.c: In function 'ring_buffer_poll_wait':

>> kernel/trace/ring_buffer.c:962:31: warning: unused variable 'flags' [-Wunused-variable]
     962 |                 unsigned long flags;
         |                               ^~~~~

vim +/flags +962 kernel/trace/ring_buffer.c

15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   928) 
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   929) /**
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   930)  * ring_buffer_poll_wait - poll on buffer input
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   931)  * @buffer: buffer to wait on
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   932)  * @cpu: the cpu buffer to wait on
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   933)  * @filp: the file descriptor
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   934)  * @poll_table: The poll descriptor
42fb0a1e84ff52 Steven Rostedt (Google   2022-10-20   935)  * @full: wait until the percentage of pages are available, if @cpu != RING_BUFFER_ALL_CPUS
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   936)  *
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   937)  * If @cpu == RING_BUFFER_ALL_CPUS then the task will wake up as soon
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   938)  * as data is added to any of the @buffer's cpu buffers. Otherwise
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   939)  * it will wait for data to be added to a specific cpu buffer.
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   940)  *
a9a08845e9acbd Linus Torvalds           2018-02-11   941   * Returns EPOLLIN | EPOLLRDNORM if data exists in the buffers,
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   942)  * zero otherwise.
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   943)  */
13292494379f92 Steven Rostedt (VMware   2019-12-13   944) __poll_t ring_buffer_poll_wait(struct trace_buffer *buffer, int cpu,
42fb0a1e84ff52 Steven Rostedt (Google   2022-10-20   945) 			  struct file *filp, poll_table *poll_table, int full)
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   946) {
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   947) 	struct ring_buffer_per_cpu *cpu_buffer;
68282dd930ea38 Steven Rostedt (Google   2024-03-08   948) 	struct rb_irq_work *rbwork;
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   949) 
42fb0a1e84ff52 Steven Rostedt (Google   2022-10-20   950) 	if (cpu == RING_BUFFER_ALL_CPUS) {
68282dd930ea38 Steven Rostedt (Google   2024-03-08   951) 		rbwork = &buffer->irq_work;
42fb0a1e84ff52 Steven Rostedt (Google   2022-10-20   952) 		full = 0;
42fb0a1e84ff52 Steven Rostedt (Google   2022-10-20   953) 	} else {
6721cb60022629 Steven Rostedt (Red Hat  2013-05-23   954) 		if (!cpumask_test_cpu(cpu, buffer->cpumask))
66bbea9ed6446b Vincent Donnefort        2024-01-31   955  			return EPOLLERR;
6721cb60022629 Steven Rostedt (Red Hat  2013-05-23   956) 
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   957) 		cpu_buffer = buffer->buffers[cpu];
68282dd930ea38 Steven Rostedt (Google   2024-03-08   958) 		rbwork = &cpu_buffer->irq_work;
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   959) 	}
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28   960) 
42fb0a1e84ff52 Steven Rostedt (Google   2022-10-20   961) 	if (full) {
68282dd930ea38 Steven Rostedt (Google   2024-03-08  @962) 		unsigned long flags;
68282dd930ea38 Steven Rostedt (Google   2024-03-08   963) 
68282dd930ea38 Steven Rostedt (Google   2024-03-08   964) 		poll_wait(filp, &rbwork->full_waiters, poll_table);
68282dd930ea38 Steven Rostedt (Google   2024-03-08   965) 
4e3268779574c4 Steven Rostedt (Google   2024-03-12   966) 		if (rb_watermark_hit(buffer, cpu, full))
a505bba2acc5f4 Steven Rostedt (Google   2024-03-12   967) 			return EPOLLIN | EPOLLRDNORM;
a505bba2acc5f4 Steven Rostedt (Google   2024-03-12   968) 		/*
a505bba2acc5f4 Steven Rostedt (Google   2024-03-12   969) 		 * Only allow full_waiters_pending update to be seen after
4e3268779574c4 Steven Rostedt (Google   2024-03-12   970) 		 * the shortest_full is set (in rb_watermark_hit). If the
4e3268779574c4 Steven Rostedt (Google   2024-03-12   971) 		 * writer sees the full_waiters_pending flag set, it will
4e3268779574c4 Steven Rostedt (Google   2024-03-12   972) 		 * compare the amount in the ring buffer to shortest_full.
4e3268779574c4 Steven Rostedt (Google   2024-03-12   973) 		 * If the amount in the ring buffer is greater than the
4e3268779574c4 Steven Rostedt (Google   2024-03-12   974) 		 * shortest_full percent, it will call the irq_work handler
4e3268779574c4 Steven Rostedt (Google   2024-03-12   975) 		 * to wake up this list. The irq_handler will reset shortest_full
a505bba2acc5f4 Steven Rostedt (Google   2024-03-12   976) 		 * back to zero. That's done under the reader_lock, but
a505bba2acc5f4 Steven Rostedt (Google   2024-03-12   977) 		 * the below smp_mb() makes sure that the update to
a505bba2acc5f4 Steven Rostedt (Google   2024-03-12   978) 		 * full_waiters_pending doesn't leak up into the above.
a505bba2acc5f4 Steven Rostedt (Google   2024-03-12   979) 		 */
a505bba2acc5f4 Steven Rostedt (Google   2024-03-12   980) 		smp_mb();
a505bba2acc5f4 Steven Rostedt (Google   2024-03-12   981) 		rbwork->full_waiters_pending = true;
a505bba2acc5f4 Steven Rostedt (Google   2024-03-12   982) 		return 0;
a505bba2acc5f4 Steven Rostedt (Google   2024-03-12   983) 	}
a505bba2acc5f4 Steven Rostedt (Google   2024-03-12   984) 
68282dd930ea38 Steven Rostedt (Google   2024-03-08   985) 	poll_wait(filp, &rbwork->waiters, poll_table);
68282dd930ea38 Steven Rostedt (Google   2024-03-08   986) 	rbwork->waiters_pending = true;
42fb0a1e84ff52 Steven Rostedt (Google   2022-10-20   987) 
4ce97dbf502452 Josef Bacik              2014-08-25   988  	/*
4ce97dbf502452 Josef Bacik              2014-08-25   989  	 * There's a tight race between setting the waiters_pending and
4ce97dbf502452 Josef Bacik              2014-08-25   990  	 * checking if the ring buffer is empty.  Once the waiters_pending bit
4ce97dbf502452 Josef Bacik              2014-08-25   991  	 * is set, the next event will wake the task up, but we can get stuck
4ce97dbf502452 Josef Bacik              2014-08-25   992  	 * if there's only a single event in.
4ce97dbf502452 Josef Bacik              2014-08-25   993  	 *
4ce97dbf502452 Josef Bacik              2014-08-25   994  	 * FIXME: Ideally, we need a memory barrier on the writer side as well,
4ce97dbf502452 Josef Bacik              2014-08-25   995  	 * but adding a memory barrier to all events will cause too much of a
4ce97dbf502452 Josef Bacik              2014-08-25   996  	 * performance hit in the fast path.  We only need a memory barrier when
4ce97dbf502452 Josef Bacik              2014-08-25   997  	 * the buffer goes from empty to having content.  But as this race is
4ce97dbf502452 Josef Bacik              2014-08-25   998  	 * extremely small, and it's not a problem if another event comes in, we
4ce97dbf502452 Josef Bacik              2014-08-25   999  	 * will fix it later.
4ce97dbf502452 Josef Bacik              2014-08-25  1000  	 */
4ce97dbf502452 Josef Bacik              2014-08-25  1001  	smp_mb();
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28  1002) 
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28  1003) 	if ((cpu == RING_BUFFER_ALL_CPUS && !ring_buffer_empty(buffer)) ||
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28  1004) 	    (cpu != RING_BUFFER_ALL_CPUS && !ring_buffer_empty_cpu(buffer, cpu)))
a9a08845e9acbd Linus Torvalds           2018-02-11  1005  		return EPOLLIN | EPOLLRDNORM;
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28  1006) 	return 0;
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28  1007) }
15693458c4bc06 Steven Rostedt (Red Hat  2013-02-28  1008) 
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki