Skip to contentSkip to conversation

tracing: Have osnoise_main() add a quiescent state for task rcu

PATCH In review Steven Rostedt 3 participants 3 messages Jun 28, 2021

Subsystems (2)
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

PATCH 1/1 tracing: Have osnoise_main() add a quiescent state for task rcu

From: "Steven Rostedt (VMware)" <[email protected]>

ftracetest triggered:

 INFO: rcu_tasks detected stalls on tasks:
 00000000b92b832d: .. nvcsw: 1/1 holdout: 1 idle_cpu: -1/7
 task:osnoise/7       state:R  running task     stack:    0 pid: 2133 ppid:     2 flags:0x00004000
 Call Trace:
  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
  ? trace_hardirqs_on+0x2b/0xe0
  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
  ? trace_clock_local+0xc/0x20
  ? osnoise_main+0x10e/0x450
  ? trace_softirq_entry_callback+0x50/0x50
  ? kthread+0x153/0x170
  ? __kthread_bind_mask+0x60/0x60
  ? ret_from_fork+0x22/0x30

While running osnoise tracer with other tracers that rely on synchronize_rcu_tasks(), where that just hung.

The reason is that osnoise_main() never schedules out if the interval is less than 1, and this will cause synchronize_rcu_tasks() to never return.

Fixes bce29ac9ce0bb ("trace: Add osnoise tracer")
---
 kernel/trace/trace_osnoise.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/trace/trace_osnoise.c +4 -1
index 38aa5e208ffd..556d530af805 100644
@@ -1216,8 +1216,11 @@ static int osnoise_main(void *data)
 		 * differently from hwlat_detector, the osnoise tracer can run
 		 * without a pause because preemption is on.
 		 */
-		if (interval < 1)
+		if (interval < 1) {
+			/* Let synchronize_rcu_tasks() make progress */
+			cond_resched_tasks_rcu_qs();
 			continue;
+		}
 
 		if (msleep_interruptible(interval))
 			break;
Signature · 1 line
2.29.2
Acked-by on v1 1/1
#1 lore raw
Quoted 22 lines · Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

ftracetest triggered:

 INFO: rcu_tasks detected stalls on tasks:
 00000000b92b832d: .. nvcsw: 1/1 holdout: 1 idle_cpu: -1/7
 task:osnoise/7       state:R  running task     stack:    0 pid: 2133 ppid:     2 flags:0x00004000
 Call Trace:
  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
  ? trace_hardirqs_on+0x2b/0xe0
  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
  ? trace_clock_local+0xc/0x20
  ? osnoise_main+0x10e/0x450
  ? trace_softirq_entry_callback+0x50/0x50
  ? kthread+0x153/0x170
  ? __kthread_bind_mask+0x60/0x60
  ? ret_from_fork+0x22/0x30

While running osnoise tracer with other tracers that rely on synchronize_rcu_tasks(), where that just hung.

Quoted 6 lines

The reason is that osnoise_main() never schedules out if the interval is less than 1, and this will cause synchronize_rcu_tasks() to never return.

Fixes bce29ac9ce0bb ("trace: Add osnoise tracer")

That looks like something I would do!

Quoted 24 lines
---
 kernel/trace/trace_osnoise.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/trace/trace_osnoise.c +4 -1
index 38aa5e208ffd..556d530af805 100644
@@ -1216,8 +1216,11 @@ static int osnoise_main(void *data)
 		 * differently from hwlat_detector, the osnoise tracer can run
 		 * without a pause because preemption is on.
 		 */
-		if (interval < 1)
+		if (interval < 1) {
+			/* Let synchronize_rcu_tasks() make progress */
+			cond_resched_tasks_rcu_qs();
 			continue;
+		}
 
 		if (msleep_interruptible(interval))
 			break;
Signature · 1 line
2.29.2
Reviewed-by on v1 1/1
#2 lore raw
In reply to Steven Rostedt
Quoted 19 lines · Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

ftracetest triggered:

 INFO: rcu_tasks detected stalls on tasks:
 00000000b92b832d: .. nvcsw: 1/1 holdout: 1 idle_cpu: -1/7
 task:osnoise/7       state:R  running task     stack:    0 pid: 2133 ppid:     2 flags:0x00004000
 Call Trace:
  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
  ? trace_hardirqs_on+0x2b/0xe0
  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
  ? trace_clock_local+0xc/0x20
  ? osnoise_main+0x10e/0x450
  ? trace_softirq_entry_callback+0x50/0x50
  ? kthread+0x153/0x170
  ? __kthread_bind_mask+0x60/0x60
  ? ret_from_fork+0x22/0x30
Quoted 5 lines

While running osnoise tracer with other tracers that rely on synchronize_rcu_tasks(), where that just hung.

The reason is that osnoise_main() never schedules out if the interval is less than 1,

and there is no other task setting need resched (it has a regular cond_resched() on the run_osnoise() loop).

and this will cause synchronize_rcu_tasks() to never

Quoted 4 lines

return.

Fixes bce29ac9ce0bb ("trace: Add osnoise tracer")

Thanks!
-- Daniel

Quoted 22 lines
---
 kernel/trace/trace_osnoise.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/trace/trace_osnoise.c +4 -1
index 38aa5e208ffd..556d530af805 100644
@@ -1216,8 +1216,11 @@ static int osnoise_main(void *data)
 		 * differently from hwlat_detector, the osnoise tracer can run
 		 * without a pause because preemption is on.
 		 */
-		if (interval < 1)
+		if (interval < 1) {
+			/* Let synchronize_rcu_tasks() make progress */
+			cond_resched_tasks_rcu_qs();
 			continue;
+		}
 
 		if (msleep_interruptible(interval))
 			break;