hung_task: add warning counter to blocked task report
PATCH 1/1 hung_task: add warning counter to blocked task report
From: Ye Liu <[email protected]>
Add a warning counter to each hung task message to make it easier to analyze and locate issues in the logs.
--- kernel/hung_task.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/hung_task.c +4 -2
@@ -58,6 +58,7 @@ EXPORT_SYMBOL_GPL(sysctl_hung_task_timeout_secs);static unsigned long __read_mostly sysctl_hung_task_check_interval_secs;static int __read_mostly sysctl_hung_task_warnings =10 ; +static int hung_task_warning_count;static int __read_mostly did_panic;static bool hung_task_show_lock;
@@ -232,8 +233,9 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout)if (sysctl_hung_task_warnings || hung_task_call_panic) {if (sysctl_hung_task_warnings >0 ) sysctl_hung_task_warnings--; -pr_err ("INFO: task %s:%d blocked for more than %ld seconds.\n" , - t->comm, t->pid, (jiffies - t->last_switch_time) /HZ ); +pr_err ("INFO: task %s:%d blocked for more than %ld seconds. [Warning #%d]\n" , + t->comm, t->pid, (jiffies - t->last_switch_time) /HZ , + ++hung_task_warning_count); pr_err(" %s %s %.*s\n", print_tainted(), init_utsname()->release, (int)strcspn(init_utsname()->version, " "),
Signature · 1 line
2.43.0
Lance Yang
maintainerHi Ye,
Thanks for your patch!
Quoted 24 lines · Ye Liu
From: Ye Liu <[email protected]>
Add a warning counter to each hung task message to make it easier to analyze and locate issues in the logs.
Signed-off-by Ye Liu <[email protected]>--- kernel/hung_task.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)kernel/hung_task.c +1
-0@@ -58,6 +58,7 @@ EXPORT_SYMBOL_GPL(sysctl_hung_task_timeout_secs);static unsigned long __read_mostly sysctl_hung_task_check_interval_secs;static int __read_mostly sysctl_hung_task_warnings =10 ; +static int hung_task_warning_count;static int __read_mostly did_panic;static bool hung_task_show_lock;@@ -232,8 +233,9 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout)if (sysctl_hung_task_warnings || hung_task_call_panic) {
kernel/hung_task.c static void check_hung_task(struct task_struct *t, unsigned long timeout) · line 241
Quoted 10 lines
kernel/hung_task.c +3
-2if (sysctl_hung_task_warnings >0 ) sysctl_hung_task_warnings--; -pr_err ("INFO: task %s:%d blocked for more than %ld seconds.\n" , - t->comm, t->pid, (jiffies - t->last_switch_time) /HZ ); +pr_err ("INFO: task %s:%d blocked for more than %ld seconds. [Warning #%d]\n" , + t->comm, t->pid, (jiffies - t->last_switch_time) /HZ , + ++hung_task_warning_count); pr_err(" %s %s %.*s\n", print_tainted(), init_utsname()->release, (int)strcspn(init_utsname()->version, " "),
A quick thought on this: we already have the hung_task_detect_count counter, which tracks the total number of hung tasks detected since boot ;)
While this patch adds a counter inline with the warning message, the existing counter already provides a way to know how many hung task events have occurred.
Could you elaborate on the specific benefit of printing this count directly in the log, compared to checking the global hung_task_detect_count?
Also, if the goal is to give each warning a unique sequence number, I think the dmesg timestamp prefix serves the same purpose ;)
Thanks,
Lance
Quoted 46 lines · Lance Yang
Hi Ye,
Thanks for your patch!
Quoted 22 lines · Ye Liu
From: Ye Liu <[email protected]>
Add a warning counter to each hung task message to make it easier to analyze and locate issues in the logs.
Signed-off-by Ye Liu <[email protected]>---
kernel/hung_task.c| 6 ++++--1 file changed, 4 insertions(+), 2 deletions(-)kernel/hung_task.c +1
-0@@ -58,6 +58,7 @@ EXPORT_SYMBOL_GPL(sysctl_hung_task_timeout_secs); static unsigned long __read_mostly sysctl_hung_task_check_interval_secs; static int __read_mostly sysctl_hung_task_warnings = 10; +static int hung_task_warning_count; static int __read_mostly did_panic; static bool hung_task_show_lock;@@ -232,8 +233,9 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout) if (sysctl_hung_task_warnings || hung_task_call_panic) {kernel/hung_task.c static void check_hung_task(struct task_struct *t, unsigned long timeout) · line 241
Quoted 10 lines
kernel/hung_task.c +3
-2if (sysctl_hung_task_warnings > 0) sysctl_hung_task_warnings--; - pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n", - t->comm, t->pid, (jiffies - t->last_switch_time) / HZ); + pr_err("INFO: task %s:%d blocked for more than %ld seconds. [Warning #%d]\n", + t->comm, t->pid, (jiffies - t->last_switch_time) / HZ, + ++hung_task_warning_count); pr_err(" %s %s %.*s\n", print_tainted(), init_utsname()->release, (int)strcspn(init_utsname()->version, " "),A quick thought on this: we already have the
hung_task_detect_countcounter, which tracks the total number of hung tasks detected since boot ;)While this patch adds a counter inline with the warning message, the existing counter already provides a way to know how many hung task events have occurred.
Quoted 8 lines
Could you elaborate on the specific benefit of printing this count directly in the log, compared to checking the global
hung_task_detect_count?Also, if the goal is to give each warning a unique sequence number, I think the dmesg timestamp prefix serves the same purpose ;)
Thanks,
Lance
Sorry for not noticing sysctl_hung_task_detect_count.
I just thought adding it directly to the warning message would make the log easier to read and more intuitive than relying on timestamps.
If accepted, I will send V2, like this:
kernel/hung_task.c +3 -2
@@ -232,8 +232,9 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout)if (sysctl_hung_task_warnings || hung_task_call_panic) {if (sysctl_hung_task_warnings >0 ) sysctl_hung_task_warnings--; -pr_err ("INFO: task %s:%d blocked for more than %ld seconds.\n" , - t->comm, t->pid, (jiffies - t->last_switch_time) /HZ ); +pr_err ("INFO: task %s:%d blocked for more than %ld seconds. [#%ld]\n" , + t->comm, t->pid, (jiffies - t->last_switch_time) /HZ , + sysctl_hung_task_detect_count); pr_err(" %s %s %.*s\n", print_tainted(), init_utsname()->release, (int)strcspn(init_utsname()->version, " "),
Lance Yang
maintainerQuoted 55 lines · Ye Liu
Quoted 52 lines · Lance Yang
Hi Ye,
Thanks for your patch!
Quoted 22 lines · Ye Liu
From: Ye Liu <[email protected]>
Add a warning counter to each hung task message to make it easier to analyze and locate issues in the logs.
Signed-off-by Ye Liu <[email protected]>--- kernel/hung_task.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)kernel/hung_task.c +1
-0@@ -58,6 +58,7 @@ EXPORT_SYMBOL_GPL(sysctl_hung_task_timeout_secs); static unsigned long __read_mostly sysctl_hung_task_check_interval_secs; static int __read_mostly sysctl_hung_task_warnings = 10; +static int hung_task_warning_count; static int __read_mostly did_panic; static bool hung_task_show_lock;@@ -232,8 +233,9 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout) if (sysctl_hung_task_warnings || hung_task_call_panic) {kernel/hung_task.c static void check_hung_task(struct task_struct *t, unsigned long timeout) · line 241
Quoted 10 lines
kernel/hung_task.c +3
-2if (sysctl_hung_task_warnings > 0) sysctl_hung_task_warnings--; - pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n", - t->comm, t->pid, (jiffies - t->last_switch_time) / HZ); + pr_err("INFO: task %s:%d blocked for more than %ld seconds. [Warning #%d]\n", + t->comm, t->pid, (jiffies - t->last_switch_time) / HZ, + ++hung_task_warning_count); pr_err(" %s %s %.*s\n", print_tainted(), init_utsname()->release, (int)strcspn(init_utsname()->version, " "),A quick thought on this: we already have the
hung_task_detect_countcounter, which tracks the total number of hung tasks detected since boot ;)While this patch adds a counter inline with the warning message, the existing counter already provides a way to know how many hung task events have occurred.
Could you elaborate on the specific benefit of printing this count directly in the log, compared to checking the global
hung_task_detect_count?Also, if the goal is to give each warning a unique sequence number, I think the dmesg timestamp prefix serves the same purpose ;)
Quoted 8 lines
Quoted 2 lines
Thanks,
LanceSorry for not noticing
sysctl_hung_task_detect_count.
I just thought adding it directly to the warning message would make the log easier to read and more intuitive than relying on timestamps.If accepted, I will send V2, like this:
Let's step back and considet the practical use case. when we are troubleshooting hung task issues in a production log, what information do we actually use?
Typically, we look for:
1) The timestamp, to correlate with other system events
2) The task name and PID (%s:%d)
3) The kernel stack trace that follows, to see where it's stuck
So, my question is: in what specific troubleshooting scenario would knowing the sequence number, like [#N], provide actionable information that the above data points do not?
Unless there's a compelling use case I'm missing, I'd prefer to keep the code as it is ;)
Thanks,
Lance
Quoted 21 lines
kernel/hung_task.c +3
-2@@ -232,8 +232,9 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout)if (sysctl_hung_task_warnings || hung_task_call_panic) {if (sysctl_hung_task_warnings >0 ) sysctl_hung_task_warnings--; -pr_err ("INFO: task %s:%d blocked for more than %ld seconds.\n" , - t->comm, t->pid, (jiffies - t->last_switch_time) /HZ ); +pr_err ("INFO: task %s:%d blocked for more than %ld seconds. [#%ld]\n" , + t->comm, t->pid, (jiffies - t->last_switch_time) /HZ , + sysctl_hung_task_detect_count); pr_err(" %s %s %.*s\n", print_tainted(), init_utsname()->release, (int)strcspn(init_utsname()->version, " "),
Quoted 76 lines · Lance Yang
Quoted 55 lines · Ye Liu
Quoted 52 lines · Lance Yang
Hi Ye,
Thanks for your patch!
Quoted 22 lines · Ye Liu
From: Ye Liu <[email protected]>
Add a warning counter to each hung task message to make it easier to analyze and locate issues in the logs.
Signed-off-by Ye Liu <[email protected]>---
kernel/hung_task.c| 6 ++++--1 file changed, 4 insertions(+), 2 deletions(-)kernel/hung_task.c +1
-0@@ -58,6 +58,7 @@ EXPORT_SYMBOL_GPL(sysctl_hung_task_timeout_secs); static unsigned long __read_mostly sysctl_hung_task_check_interval_secs; static int __read_mostly sysctl_hung_task_warnings = 10; +static int hung_task_warning_count; static int __read_mostly did_panic; static bool hung_task_show_lock;@@ -232,8 +233,9 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout) if (sysctl_hung_task_warnings || hung_task_call_panic) {kernel/hung_task.c static void check_hung_task(struct task_struct *t, unsigned long timeout) · line 241
Quoted 10 lines
kernel/hung_task.c +3
-2if (sysctl_hung_task_warnings > 0) sysctl_hung_task_warnings--; - pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n", - t->comm, t->pid, (jiffies - t->last_switch_time) / HZ); + pr_err("INFO: task %s:%d blocked for more than %ld seconds. [Warning #%d]\n", + t->comm, t->pid, (jiffies - t->last_switch_time) / HZ, + ++hung_task_warning_count); pr_err(" %s %s %.*s\n", print_tainted(), init_utsname()->release, (int)strcspn(init_utsname()->version, " "),A quick thought on this: we already have the
hung_task_detect_countcounter, which tracks the total number of hung tasks detected since boot ;)While this patch adds a counter inline with the warning message, the existing counter already provides a way to know how many hung task events have occurred.
Could you elaborate on the specific benefit of printing this count directly in the log, compared to checking the global
hung_task_detect_count?Also, if the goal is to give each warning a unique sequence number, I think the dmesg timestamp prefix serves the same purpose ;)
Quoted 8 lines
Quoted 2 lines
Thanks,
LanceSorry for not noticing
sysctl_hung_task_detect_count.
I just thought adding it directly to the warning message would make the log easier to read and more intuitive than relying on timestamps.If accepted, I will send V2, like this:
Let's step back and considet the practical use case. when we are troubleshooting hung task issues in a production log, what information do we actually use?
Typically, we look for:
1) The timestamp, to correlate with other system events
2) The task name and PID (%s:%d)
3) The kernel stack trace that follows, to see where it's stuck
Quoted 6 lines
So, my question is: in what specific troubleshooting scenario would knowing the sequence number, like [#N], provide actionable information that the above data points do not?
Unless there's a compelling use case I'm missing, I'd prefer to keep the code as it is ;)
There is no specific use case—the system currently defaults to 10 printouts. When multiple logs are present in the same window, I simply want to quickly identify which one is the Nth instance (e.g., the 3rd out of 10). That’s all.
Quoted 25 lines
Thanks,
LanceQuoted 21 lines
kernel/hung_task.c +3
-2@@ -232,8 +232,9 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout) if (sysctl_hung_task_warnings || hung_task_call_panic) { if (sysctl_hung_task_warnings > 0) sysctl_hung_task_warnings--; - pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n", - t->comm, t->pid, (jiffies - t->last_switch_time) / HZ); + pr_err("INFO: task %s:%d blocked for more than %ld seconds. [#%ld]\n", + t->comm, t->pid, (jiffies - t->last_switch_time) / HZ, + sysctl_hung_task_detect_count); pr_err(" %s %s %.*s\n", print_tainted(), init_utsname()->release, (int)strcspn(init_utsname()->version, " "),
Signature · 2 lines
Thanks, Ye Liu
Lance Yang
maintainerQuoted 83 lines · Ye Liu
Quoted 80 lines · Lance Yang
Quoted 55 lines · Ye Liu
Quoted 52 lines · Lance Yang
Hi Ye,
Thanks for your patch!
Quoted 22 lines · Ye Liu
From: Ye Liu <[email protected]>
Add a warning counter to each hung task message to make it easier to analyze and locate issues in the logs.
Signed-off-by Ye Liu <[email protected]>--- kernel/hung_task.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)kernel/hung_task.c +1
-0@@ -58,6 +58,7 @@ EXPORT_SYMBOL_GPL(sysctl_hung_task_timeout_secs); static unsigned long __read_mostly sysctl_hung_task_check_interval_secs; static int __read_mostly sysctl_hung_task_warnings = 10; +static int hung_task_warning_count; static int __read_mostly did_panic; static bool hung_task_show_lock;@@ -232,8 +233,9 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout) if (sysctl_hung_task_warnings || hung_task_call_panic) {kernel/hung_task.c static void check_hung_task(struct task_struct *t, unsigned long timeout) · line 241
Quoted 10 lines
kernel/hung_task.c +3
-2if (sysctl_hung_task_warnings > 0) sysctl_hung_task_warnings--; - pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n", - t->comm, t->pid, (jiffies - t->last_switch_time) / HZ); + pr_err("INFO: task %s:%d blocked for more than %ld seconds. [Warning #%d]\n", + t->comm, t->pid, (jiffies - t->last_switch_time) / HZ, + ++hung_task_warning_count); pr_err(" %s %s %.*s\n", print_tainted(), init_utsname()->release, (int)strcspn(init_utsname()->version, " "),A quick thought on this: we already have the
hung_task_detect_countcounter, which tracks the total number of hung tasks detected since boot ;)While this patch adds a counter inline with the warning message, the existing counter already provides a way to know how many hung task events have occurred.
Could you elaborate on the specific benefit of printing this count directly in the log, compared to checking the global
hung_task_detect_count?Also, if the goal is to give each warning a unique sequence number, I think the dmesg timestamp prefix serves the same purpose ;)
Quoted 8 lines
Quoted 2 lines
Thanks,
LanceSorry for not noticing
sysctl_hung_task_detect_count.
I just thought adding it directly to the warning message would make the log easier to read and more intuitive than relying on timestamps.If accepted, I will send V2, like this:
Let's step back and considet the practical use case. when we are troubleshooting hung task issues in a production log, what information do we actually use?
Typically, we look for:
1) The timestamp, to correlate with other system events
2) The task name and PID (%s:%d)
3) The kernel stack trace that follows, to see where it's stuckSo, my question is: in what specific troubleshooting scenario would knowing the sequence number, like [#N], provide actionable information that the above data points do not?
Quoted 6 lines
Quoted 2 lines
Unless there's a compelling use case I'm missing, I'd prefer to keep the code as it is ;)
There is no specific use case—the system currently defaults to 10 printouts. When multiple logs are present in the same window, I simply want to quickly identify which one is the Nth instance (e.g., the 3rd out of 10). That’s all.
Ah, I see your point. The main thing for me is that the sysctl_hung_task_detect_count counter already exists for tracking this.
Since this is a minor user-facing change, let's defer to Andrew for the final decision and see what he thinks ;)
Thanks,
Lance
Quoted 27 lines
Quoted 26 lines
Thanks,
LanceQuoted 21 lines
kernel/hung_task.c +3
-2@@ -232,8 +232,9 @@ static void check_hung_task(struct task_struct *t, unsigned long timeout) if (sysctl_hung_task_warnings || hung_task_call_panic) { if (sysctl_hung_task_warnings > 0) sysctl_hung_task_warnings--; - pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n", - t->comm, t->pid, (jiffies - t->last_switch_time) / HZ); + pr_err("INFO: task %s:%d blocked for more than %ld seconds. [#%ld]\n", + t->comm, t->pid, (jiffies - t->last_switch_time) / HZ, + sysctl_hung_task_detect_count); pr_err(" %s %s %.*s\n", print_tainted(), init_utsname()->release, (int)strcspn(init_utsname()->version, " "),