Skip to contentSkip to conversation

hung_task: Differentiate between I/O and Lock/Resource waits

PATCH In review Aaron Tomlin 3 participants 6 messages Jan 25 – Jan 28 Hung task detector

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 hung_task: Differentiate between I/O and Lock/Resource waits

Currently, the hung task reporting mechanism does not differentiate between the underlying causes of a D state, labelling all such tasks merely as "blocked". Consequently, administrators must perform manual stack trace inspection to ascertain if the delay stems from an I/O wait (indicative of hardware or filesystem issues) or a lock wait (indicative of software contention).

This change utilises the in_iowait field from struct task_struct to distinguish between two distinct failure modes in the log output:

1. D state "Disk I/O": The task is waiting in io_schedule(). This typically implies a storage device, filesystem, or network filesystem (e.g., NFS) is unresponsive.

2. D state "Lock/Resource": The task is waiting on a kernel primitive (e.g., mutex). This typically implies a software bug, deadlock, or resource starvation.

It is safe to read in_iowait in this manner because check_hung_uninterruptible_tasks() holds the RCU read lock, preserving the task structure. Moreover, the task is effectively quiescent (in a persistent TASK_UNINTERRUPTIBLE state) and thus cannot update its own in_iowait status, guaranteeing a stable, race-free value.

---
 kernel/hung_task.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/hung_task.c +3 -2
index 350093de0535..608731c7ccba 100644
@@ -250,8 +250,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 	if (sysctl_hung_task_warnings || hung_task_call_panic) {
 		if (sysctl_hung_task_warnings > 0)
 			sysctl_hung_task_warnings--;
-		pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n",
-		       t->comm, t->pid, (jiffies - t->last_switch_time) / HZ);
+		pr_err("INFO: task %s:%d blocked in %s state for more than %ld seconds.\n",
+		       t->comm, t->pid, t->in_iowait ? "D (Disk I/O)" : "D (Lock/Resource)",
+		       (jiffies - t->last_switch_time) / HZ);
 		pr_err("      %s %s %.*s\n",
 			print_tainted(), init_utsname()->release,
 			(int)strcspn(init_utsname()->version, " "),
Signature · 1 line
2.51.0
maintainer
#1 lore raw
Quoted 35 lines · Aaron Tomlin

Currently, the hung task reporting mechanism does not differentiate between the underlying causes of a D state, labelling all such tasks merely as "blocked". Consequently, administrators must perform manual stack trace inspection to ascertain if the delay stems from an I/O wait (indicative of hardware or filesystem issues) or a lock wait (indicative of software contention).

This change utilises the in_iowait field from struct task_struct to distinguish between two distinct failure modes in the log output:

1. D state "Disk I/O": The task is waiting in io_schedule(). This typically implies a storage device, filesystem, or network filesystem (e.g., NFS) is unresponsive.

2. D state "Lock/Resource": The task is waiting on a kernel primitive (e.g., mutex). This typically implies a software bug, deadlock, or resource starvation.

It is safe to read in_iowait in this manner because check_hung_uninterruptible_tasks() holds the RCU read lock, preserving the task structure. Moreover, the task is effectively quiescent (in a persistent TASK_UNINTERRUPTIBLE state) and thus cannot update its own in_iowait status, guaranteeing a stable, race-free value.

---
  kernel/hung_task.c | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)
kernel/hung_task.c +0 -0
index 350093de0535..608731c7ccba 100644
@@ -250,8 +250,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
  	if (sysctl_hung_task_warnings || hung_task_call_panic) {

kernel/hung_task.c static void hung_task_info(struct task_struct *t, unsigned long timeout, · line 258

Quoted 10 lines
kernel/hung_task.c +3 -2
  		if (sysctl_hung_task_warnings > 0)
  			sysctl_hung_task_warnings--;
-		pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n",
-		       t->comm, t->pid, (jiffies - t->last_switch_time) / HZ);
+		pr_err("INFO: task %s:%d blocked in %s state for more than %ld seconds.\n",
+		       t->comm, t->pid, t->in_iowait ? "D (Disk I/O)" : "D (Lock/Resource)",
+		       (jiffies - t->last_switch_time) / HZ);
  		pr_err("      %s %s %.*s\n",
  			print_tainted(), init_utsname()->release,
  			(int)strcspn(init_utsname()->version, " "),

Why do we need this?

It's rather obvious that the stack trace already shows whether it is in "D (Disk I/O)" or "D (Lock/Resource)" or "D (...)".

#2 lore raw
Quoted 4 lines · Lance Yang

Why do we need this?

It's rather obvious that the stack trace already shows whether it is in "D (Disk I/O)" or "D (Lock/Resource)" or "D (...)".

Hi Lance,

Thank you for your review.

While I agree that a seasoned kernel developer can often deduce the root cause by inspecting the stack trace, this level of analysis is not always immediately accessible to some system administrators or first-line support engineers.

The primary benefit of this patch is to provide high-level clarity at a glance.

By explicitly distinguishing between "Disk I/O" and "Lock/Resource" contention in the initial log message, we allow some administrators or support engineers to rapidly route the incident to the appropriate team (e.g., storage/network vs. kernel etc.) without needing to parse or understand the nuances of kernel stack traces.

Kind regards,

Signature · 1 line
Aaron Tomlin
#3 lore raw
In reply to Aaron Tomlin
Quoted 34 lines · On Sun, 25 Jan 2026 15:39:05 -0500 Aaron Tomlin <[email protected]> wrote:

Currently, the hung task reporting mechanism does not differentiate between the underlying causes of a D state, labelling all such tasks merely as "blocked". Consequently, administrators must perform manual stack trace inspection to ascertain if the delay stems from an I/O wait (indicative of hardware or filesystem issues) or a lock wait (indicative of software contention).

This change utilises the in_iowait field from struct task_struct to distinguish between two distinct failure modes in the log output:

1. D state "Disk I/O": The task is waiting in io_schedule(). This typically implies a storage device, filesystem, or network filesystem (e.g., NFS) is unresponsive.

2. D state "Lock/Resource": The task is waiting on a kernel primitive (e.g., mutex). This typically implies a software bug, deadlock, or resource starvation.

It is safe to read in_iowait in this manner because check_hung_uninterruptible_tasks() holds the RCU read lock, preserving the task structure. Moreover, the task is effectively quiescent (in a persistent TASK_UNINTERRUPTIBLE state) and thus cannot update its own in_iowait status, guaranteeing a stable, race-free value.

---
 kernel/hung_task.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/hung_task.c +0 -0
index 350093de0535..608731c7ccba 100644

kernel/hung_task.c static void hung_task_info(struct task_struct *t, unsigned long timeout, · line 254

Quoted 7 lines
kernel/hung_task.c +2 -2
@@ -250,8 +250,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
 	if (sysctl_hung_task_warnings || hung_task_call_panic) {
 		if (sysctl_hung_task_warnings > 0)
 			sysctl_hung_task_warnings--;
-		pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n",
-		       t->comm, t->pid, (jiffies - t->last_switch_time) / HZ);
+		pr_err("INFO: task %s:%d blocked in %s state for more than %ld seconds.\n",
+		       t->comm, t->pid, t->in_iowait ? "D (Disk I/O)" : "D (Lock/Resource)",

If this is only for human readability, I rather like just adding "in iowait" at the end. "D" state seems redundant, and "Lock/Resource" can mislead. What about something like below?

	pr_err("INFO: task %s:%d blocked for more than %ld seconds%s.\n",
		..., t->in_iowait ? " in I/O wait" : "");

Thank you,

Quoted 7 lines
kernel/hung_task.c +1 -0
+		       (jiffies - t->last_switch_time) / HZ);
 		pr_err("      %s %s %.*s\n",
 			print_tainted(), init_utsname()->release,
 			(int)strcspn(init_utsname()->version, " "),
Signature · 1 line
2.51.0
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
maintainer
#4 lore raw
Quoted 49 lines · Masami Hiramatsu (Google)
Quoted 34 lines · On Sun, 25 Jan 2026 15:39:05 -0500 Aaron Tomlin <[email protected]> wrote:

Currently, the hung task reporting mechanism does not differentiate between the underlying causes of a D state, labelling all such tasks merely as "blocked". Consequently, administrators must perform manual stack trace inspection to ascertain if the delay stems from an I/O wait (indicative of hardware or filesystem issues) or a lock wait (indicative of software contention).

This change utilises the in_iowait field from struct task_struct to distinguish between two distinct failure modes in the log output:

1. D state "Disk I/O": The task is waiting in io_schedule(). This typically implies a storage device, filesystem, or network filesystem (e.g., NFS) is unresponsive.

2. D state "Lock/Resource": The task is waiting on a kernel primitive (e.g., mutex). This typically implies a software bug, deadlock, or resource starvation.

It is safe to read in_iowait in this manner because check_hung_uninterruptible_tasks() holds the RCU read lock, preserving the task structure. Moreover, the task is effectively quiescent (in a persistent TASK_UNINTERRUPTIBLE state) and thus cannot update its own in_iowait status, guaranteeing a stable, race-free value.

---
  kernel/hung_task.c | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)
kernel/hung_task.c +0 -0
index 350093de0535..608731c7ccba 100644

kernel/hung_task.c static void hung_task_info(struct task_struct *t, unsigned long timeout, · line 254

Quoted 7 lines
kernel/hung_task.c +2 -2
@@ -250,8 +250,9 @@ static void hung_task_info(struct task_struct *t, unsigned long timeout,
  	if (sysctl_hung_task_warnings || hung_task_call_panic) {
  		if (sysctl_hung_task_warnings > 0)
  			sysctl_hung_task_warnings--;
-		pr_err("INFO: task %s:%d blocked for more than %ld seconds.\n",
-		       t->comm, t->pid, (jiffies - t->last_switch_time) / HZ);
+		pr_err("INFO: task %s:%d blocked in %s state for more than %ld seconds.\n",
+		       t->comm, t->pid, t->in_iowait ? "D (Disk I/O)" : "D (Lock/Resource)",

If this is only for human readability, I rather like just adding "in iowait" at the end. "D" state seems redundant, and "Lock/Resource" can mislead. What about something like below?

Quoted 2 lines
	pr_err("INFO: task %s:%d blocked for more than %ld seconds%s.\n",
		..., t->in_iowait ? " in I/O wait" : "");

That would be better, looks good to me ;)

#5 lore raw
In reply to Masami Hiramatsu (Google)
Quoted 7 lines · Masami Hiramatsu

If this is only for human readability, I rather like just adding "in iowait" at the end. "D" state seems redundant, and "Lock/Resource" can mislead. What about something like below?

	pr_err("INFO: task %s:%d blocked for more than %ld seconds%s.\n",
		..., t->in_iowait ? " in I/O wait" : "");

Hi Masami,

Thank you for your feedback.

I concur. I'll incorporate the suggestion.

Kind regards,

Signature · 1 line
Aaron Tomlin