hung_task: fix missing hung task detection for kthread in TASK_WAKEKILL state
Subsystems (2)
PATCH 1/1 hung_task: fix missing hung task detection for kthread in TASK_WAKEKILL state
We recently encountered an XFS deadlock issue, which is a known problem resolved in the upstream kernel [0]. During the analysis of this issue, I observed that a kernel thread in the TASK_WAKEKILL state could not be detected as a hung task by the hung_task detector. The details are as follows:
Using the following command, I identified nine tasks stuck in the D state:
$ ps -eLo state,comm,tid,wchan | grep ^D D java 4177339 xfs_buf_lock D kworker/93:3+xf 3025535 xfs_buf_lock D kworker/87:0+xf 3426612 xfs_extent_busy_flush D kworker/85:0+xf 3479378 xfs_buf_lock D kworker/91:1+xf 3584478 xfs_buf_lock D kworker/80:3+xf 3655680 xfs_buf_lock D kworker/89:0+xf 3671691 xfs_buf_lock D kworker/84:1+xf 3708397 xfs_buf_lock D kworker/81:1+xf 4005763 xfs_buf_lock
However, the hung_task detector only reported eight of these tasks:
[3108840.650652] INFO: task java:4177339 blocked for more than 247779 seconds. [3108840.654197] INFO: task kworker/93:3:3025535 blocked for more than 248427 seconds. [3108840.657711] INFO: task kworker/85:0:3479378 blocked for more than 247836 seconds. [3108840.661483] INFO: task kworker/91:1:3584478 blocked for more than 249638 seconds. [3108840.664871] INFO: task kworker/80:3:3655680 blocked for more than 249638 seconds. [3108840.668495] INFO: task kworker/89:0:3671691 blocked for more than 249047 seconds. [3108840.672418] INFO: task kworker/84:1:3708397 blocked for more than 247836 seconds. [3108840.676175] INFO: task kworker/81:1:4005763 blocked for more than 247836 seconds.
Task 3426612, although in the D state, was not reported as a hung task.
I confirmed that task 3426612 remained in the D (disk sleep) state and experienced no context switches over a long period:
$ cat /proc/3426612/status | grep -E "State:|ctxt_switches:"; \ sleep 60; echo "----"; \ cat /proc/3426612/status | grep -E "State:|ctxt_switches:" State: D (disk sleep) voluntary_ctxt_switches: 7516 nonvoluntary_ctxt_switches: 0 ---- State: D (disk sleep) voluntary_ctxt_switches: 7516 nonvoluntary_ctxt_switches: 0
The system's hung_task detector settings were configured as follows:
kernel.hung_task_timeout_secs = 28
kernel.hung_task_warnings = -1
The issue lies in the handling of task state in the XFS code. Specifically, the thread in question (3426612) was set to the TASK_KILLABLE state in xfs_extent_busy_flush():
xfs_extent_busy_flush
prepare_to_wait(&pag->pagb_wait, &wait, TASK_KILLABLE);When a task is in the TASK_WAKEKILL state (a subset of TASK_KILLABLE), the hung_task detector ignores it, as it assumes such tasks can be terminated. However, in this case, the kernel thread cannot be killed, meaning it effectively becomes a hung task.
To address this issue, the hung_task detector should report the kthreads in the TASK_WAKEKILL state.
--- kernel/hung_task.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/hung_task.c +2 -1
@@ -220,8 +220,9 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)*/ state =READ_ONCE (t->__state);if ((state &TASK_UNINTERRUPTIBLE ) && + (t->flags &PF_KTHREAD || !(state &TASK_WAKEKILL ) && - !(state &TASK_NOLOAD )) + !(state &TASK_NOLOAD )))check_hung_task (t, timeout); } unlock:
Signature · 1 line
2.43.5
Quoted 90 lines · Yafang Shao
We recently encountered an XFS deadlock issue, which is a known problem resolved in the upstream kernel [0]. During the analysis of this issue, I observed that a kernel thread in the
TASK_WAKEKILLstate could not be detected as a hung task by thehung_taskdetector. The details are as follows:Using the following command, I identified nine tasks stuck in the D state:
$ ps -eLo state,comm,tid,wchan | grep ^D D java 4177339 xfs_buf_lock D kworker/93:3+xf 3025535 xfs_buf_lock D kworker/87:0+xf 3426612 xfs_extent_busy_flush D kworker/85:0+xf 3479378 xfs_buf_lock D kworker/91:1+xf 3584478 xfs_buf_lock D kworker/80:3+xf 3655680 xfs_buf_lock D kworker/89:0+xf 3671691 xfs_buf_lock D kworker/84:1+xf 3708397 xfs_buf_lock D kworker/81:1+xf 4005763 xfs_buf_lockHowever, the
hung_taskdetector only reported eight of these tasks:[3108840.650652] INFO: task java:4177339 blocked for more than 247779 seconds. [3108840.654197] INFO: task kworker/93:3:3025535 blocked for more than 248427 seconds. [3108840.657711] INFO: task kworker/85:0:3479378 blocked for more than 247836 seconds. [3108840.661483] INFO: task kworker/91:1:3584478 blocked for more than 249638 seconds. [3108840.664871] INFO: task kworker/80:3:3655680 blocked for more than 249638 seconds. [3108840.668495] INFO: task kworker/89:0:3671691 blocked for more than 249047 seconds. [3108840.672418] INFO: task kworker/84:1:3708397 blocked for more than 247836 seconds. [3108840.676175] INFO: task kworker/81:1:4005763 blocked for more than 247836 seconds.Task 3426612, although in the D state, was not reported as a hung task.
I confirmed that task 3426612 remained in the D (disk sleep) state and experienced no context switches over a long period:
$ cat /proc/3426612/status | grep -E "State:|ctxt_switches:"; \ sleep 60; echo "----"; \ cat /proc/3426612/status | grep -E "State:|ctxt_switches:" State: D (disk sleep) voluntary_ctxt_switches: 7516 nonvoluntary_ctxt_switches: 0 ---- State: D (disk sleep) voluntary_ctxt_switches: 7516 nonvoluntary_ctxt_switches: 0The system's
hung_taskdetector settings were configured as follows:kernel.hung_task_timeout_secs = 28
kernel.hung_task_warnings = -1The issue lies in the handling of task state in the XFS code. Specifically, the thread in question (3426612) was set to the
TASK_KILLABLEstate inxfs_extent_busy_flush():xfs_extent_busy_flush prepare_to_wait(&pag->pagb_wait, &wait, TASK_KILLABLE);When a task is in the
TASK_WAKEKILLstate (a subset ofTASK_KILLABLE), thehung_taskdetector ignores it, as it assumes such tasks can be terminated. However, in this case, the kernel thread cannot be killed, meaning it effectively becomes a hung task.To address this issue, the
hung_taskdetector should report the kthreads in theTASK_WAKEKILLstate.Signed-off-by Yafang Shao <[email protected]>Cc Dave Chinner <[email protected]>--- kernel/hung_task.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)kernel/hung_task.c +2
-2@@ -220,8 +220,9 @@ static void check_hung_uninterruptible_tasks(unsigned long timeout)*/ state =READ_ONCE (t->__state);if ((state &TASK_UNINTERRUPTIBLE ) && + (t->flags &PF_KTHREAD || !(state &TASK_WAKEKILL ) && - !(state &TASK_NOLOAD )) + !(state &TASK_NOLOAD )))check_hung_task (t, timeout); } unlock: --
Quoted 2 lines
2.43.5
Please disregard this. There may be multiple hung tasks in the TASK_IDLE state. I will send a new one.
--
Regards
Yafang