Skip to contentSkip to conversation

sched: Dump stacktrace when some task failed to stop

PATCH In review Schspa Shi 2 participants 3 messages May 19, 2022 Scheduler

lore
Share & tools
lore

Schspa Shi

author
#0 lore raw

PATCH 1/1 sched: Dump stacktrace when some task failed to stop

Some processes may not be stopped, such as workqueue workers.

Add this dump_stack to easily figure out which process caused the problems.

Before this change:

<4>[  517.545889] ------------[ cut here ]------------
<4>[  517.545901] Dying CPU not properly vacated!
<4>[  517.546366] ---[ end trace 0000000000000002 ]---
<4>[  518.546831] CPU6 enqueued tasks (2 total):
<4>[  518.546855] 	pid: 49, name: migration/6
<4>[  518.547010] 	pid: 6231, name: kworker/6:0
---
 kernel/sched/core.c | 7 +++++++
 1 file changed, 7 insertions(+)
kernel/sched/core.c +7 -0
index 53596842f0d8..de6b5320e4d7 100644
@@ -9416,7 +9416,14 @@ static void dump_rq_tasks(struct rq *rq, const char *loglvl)
 			continue;
 
 		printk("%s\tpid: %d, name: %s\n", loglvl, p->pid, p->comm);
+		if (p != current)
+			sched_show_task(p);
 	}
+	/*
+	 * We need the stack of each cpu to analyze who is blocking the
+	 * current cpu scheduling
+	 */
+	trigger_all_cpu_backtrace();
 }
 
 int sched_cpu_dying(unsigned int cpu)
Signature · 1 line
2.24.3 (Apple Git-128)
#1 lore raw
Quoted 26 lines · Schspa Shi

Some processes may not be stopped, such as workqueue workers.

Add this dump_stack to easily figure out which process caused the problems.

Before this change:

<4>[  517.545889] ------------[ cut here ]------------
<4>[  517.545901] Dying CPU not properly vacated!
<4>[  517.546366] ---[ end trace 0000000000000002 ]---
<4>[  518.546831] CPU6 enqueued tasks (2 total):
<4>[  518.546855] 	pid: 49, name: migration/6
<4>[  518.547010] 	pid: 6231, name: kworker/6:0
---
 kernel/sched/core.c | 7 +++++++
 1 file changed, 7 insertions(+)
kernel/sched/core.c +0 -0
index 53596842f0d8..de6b5320e4d7 100644
@@ -9416,7 +9416,14 @@ static void dump_rq_tasks(struct rq *rq, const char *loglvl)
 			continue;
 
 		printk("%s\tpid: %d, name: %s\n", loglvl, p->pid, p->comm);

kernel/sched/core.c static void dump_rq_tasks(struct rq *rq, const char *loglvl) · line 9428

Quoted 10 lines
kernel/sched/core.c +7 -0
+		if (p != current)
+			sched_show_task(p);
 	}
+	/*
+	 * We need the stack of each cpu to analyze who is blocking the
+	 * current cpu scheduling
+	 */
+	trigger_all_cpu_backtrace();
 }
 

Given you get a WARN() when this happens, can't you get the same thing with

panic_on_warn=1
panic_print=PANIC_PRINT_ALL_CPU_BT

?

Quoted 3 lines
kernel/sched/core.c +0 -0
 int sched_cpu_dying(unsigned int cpu)
Signature · 1 line
2.24.3 (Apple Git-128)
author
#2 lore raw
Quoted 40 lines · Valentin Schneider
Quoted 26 lines · Schspa Shi

Some processes may not be stopped, such as workqueue workers.

Add this dump_stack to easily figure out which process caused the problems.

Before this change:

<4>[  517.545889] ------------[ cut here ]------------
<4>[  517.545901] Dying CPU not properly vacated!
<4>[  517.546366] ---[ end trace 0000000000000002 ]---
<4>[  518.546831] CPU6 enqueued tasks (2 total):
<4>[  518.546855]    pid: 49, name: migration/6
<4>[  518.547010]    pid: 6231, name: kworker/6:0
---
 kernel/sched/core.c | 7 +++++++
 1 file changed, 7 insertions(+)
kernel/sched/core.c +0 -0
index 53596842f0d8..de6b5320e4d7 100644
@@ -9416,7 +9416,14 @@ static void dump_rq_tasks(struct rq *rq, const char *loglvl)
                     continue;
 
             printk("%s\tpid: %d, name: %s\n", loglvl, p->pid, p->comm);

kernel/sched/core.c static void dump_rq_tasks(struct rq *rq, const char *loglvl) · line 9428

Quoted 10 lines
kernel/sched/core.c +7 -0
+            if (p != current)
+                    sched_show_task(p);
     }
+    /*
+     * We need the stack of each cpu to analyze who is blocking the
+     * current cpu scheduling
+     */
+    trigger_all_cpu_backtrace();
 }
 

Given you get a WARN() when this happens, can't you get the same thing with

Quoted 4 lines

panic_on_warn=1
panic_print=PANIC_PRINT_ALL_CPU_BT

?

Yes, I can get the information with these two parameters set.

But this is generally a difficult problem to reproduce, so wanted to print it here so that useful information can be obtained without turning on these switches.

Quoted 4 lines
Quoted 4 lines
kernel/sched/core.c +0 -1
 int sched_cpu_dying(unsigned int cpu)
--

2.24.3 (Apple Git-128)

--
BRs

Schspa