Write commands abort hang
cover Write commands abort hang
One of our customers reported that sometimes the LIO driver hangs when performing a LUN reset while I/O is underway.
I was able to reproduce the hang by using the following libiscsi patch:
https://arkamax.eu/misc/dataout_abort_libiscsi.patch
INFO: task kworker/5:0:514619 blocked for more than 46 seconds. __wait_for_common+0x94/0x1b0 target_put_cmd_and_wait+0x6d/0xb0 [target_core_mod] core_tmr_drain_state_list+0x2a3/0x330 [target_core_mod] core_tmr_lun_reset+0x93/0x1a0 [target_core_mod] target_tmr_work+0xce/0x100 [target_core_mod] process_one_work+0x188/0x3b0 worker_thread+0x2ef/0x410
Maurizio Lombardi (1):
scsi: target: iscsi: Fix hang for aborted WRITE_PENDING commands
drivers/target/iscsi/iscsi_target.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
Signature · 1 line
2.55.0
PATCH 1/1 target: iscsi: Fix hang for aborted WRITE_PENDING commands
When a LUN_RESET aborts a WRITE command that is in the TRANSPORT_WRITE_PENDING state, the target core sets CMD_T_ABORTED and waits for the frontend to finish processing.
If the initiator subsequently sends the remaining dataout PDUs, __iscsit_check_dataout_hdr() catches the payload, stops the dataout timer if the sequence is final and finally dumps the data.
However, the iSCSI target doesn't trigger the completion process for these aborted commands. Because of this, the abort path hangs indefinitely in target_put_cmd_and_wait(), leading to a deadlocked target worker thread.
Fix this by explicitly calling target_complete_cmd() when the final dataout PDU is received for an aborted WRITE command. target_complete_cmd() detects the CMD_T_ABORTED flag and cleanly routes the command into target_abort_work, allowing the abort completion to successfully unblock.
--- drivers/target/iscsi/iscsi_target.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/target/iscsi/iscsi_target.c +3 -1
@@ -1533,8 +1533,10 @@ __iscsit_check_dataout_hdr(struct iscsit_conn *conn, void *buf,*/ if (se_cmd->transport_state &CMD_T_ABORTED ) {if (hdr->flags &ISCSI_FLAG_CMD_FINAL && - --cmd->outstanding_r2ts <1 ) + --cmd->outstanding_r2ts <1 ) {iscsit_stop_dataout_timer (cmd); +target_complete_cmd (se_cmd,SAM_STAT_TASK_ABORTED ); + }return iscsit_dump_data_payload (conn, payload_length,1 ); }
Signature · 1 line
2.55.0
Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands
Quoted 45 lines · Maurizio Lombardi
When a
LUN_RESETaborts a WRITE command that is in theTRANSPORT_WRITE_PENDINGstate, the target core setsCMD_T_ABORTEDand waits
for the frontend to finish processing.If the initiator subsequently sends the remaining dataout PDUs,
__iscsit_check_dataout_hdr()catches the payload, stops the dataout timer
if the sequence is final and finally dumps the data. However, the iSCSI target doesn't trigger the completion process for these
aborted commands. Because of this, the abort path hangs indefinitely intarget_put_cmd_and_wait(), leading to a deadlocked target worker thread.Fix this by explicitly calling
target_complete_cmd()when the final dataout
PDU is received for an aborted WRITE command.target_complete_cmd()detects
theCMD_T_ABORTEDflag and cleanly routes the command intotarget_abort_work,
allowing the abort completion to successfully unblock.Signed-off-by Maurizio Lombardi <[email protected]>---
drivers/target/iscsi/iscsi_target.c | 4 +++-1 file changed, 3 insertions(+), 1 deletion(-)drivers/target/iscsi/iscsi_target.c +0
-0@@ -1533,8 +1533,10 @@ __iscsit_check_dataout_hdr(struct iscsit_conn*conn, void *buf, */ if (se_cmd->transport_state & CMD_T_ABORTED) {if (
hdr->flags&ISCSI_FLAG_CMD_FINAL&&drivers/target/iscsi/iscsi_target.c +1
-1- --cmd->outstanding_r2ts < 1) + --cmd->outstanding_r2ts < 1) {
iscsit_stop_dataout_timer(cmd);drivers/target/iscsi/iscsi_target.c +1
-0+ target_complete_cmd(se_cmd,
SAM_STAT_TASK_ABORTED);drivers/target/iscsi/iscsi_target.c +1
-0+ }
Quoted 4 lines
returniscsit_dump_data_payload(conn, payload_length, 1);
}
This took a long time to solve with many attempts that did not help until Maurizio's solution. This patch was tested fully at a customer to fully resolve the issue for over a week with no negative impacts.
Looks good.
Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands
In reply to Maurizio LombardiQuoted 11 lines · Maurizio Lombardi
When a
LUN_RESETaborts a WRITE command that is in theTRANSPORT_WRITE_PENDINGstate, the target core setsCMD_T_ABORTEDand waits for the frontend to finish processing.If the initiator subsequently sends the remaining dataout PDUs,
__iscsit_check_dataout_hdr()catches the payload, stops the dataout timer if the sequence is final and finally dumps the data.
However, the iSCSI target doesn't trigger the completion process for these aborted commands. Because of this, the abort path hangs indefinitely intarget_put_cmd_and_wait(), leading to a deadlocked target worker thread.
Quoted 4 lines
Fix this by explicitly calling
target_complete_cmd()when the final dataout PDU is received for an aborted WRITE command.target_complete_cmd()detects theCMD_T_ABORTEDflag and cleanly routes the command intotarget_abort_work, allowing the abort completion to successfully unblock.
Shouldn't a Fixes: tag be added to this patch?
Thanks,
Bart.
Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands
Quoted 12 lines · Bart Van Assche
Quoted 11 lines · Maurizio Lombardi
When a
LUN_RESETaborts a WRITE command that is in theTRANSPORT_WRITE_PENDINGstate, the target core setsCMD_T_ABORTEDand waits for the frontend to finish processing.If the initiator subsequently sends the remaining dataout PDUs,
__iscsit_check_dataout_hdr()catches the payload, stops the dataout timer if the sequence is final and finally dumps the data.
However, the iSCSI target doesn't trigger the completion process for these aborted commands. Because of this, the abort path hangs indefinitely intarget_put_cmd_and_wait(), leading to a deadlocked target worker thread.
Quoted 6 lines
Quoted 4 lines
Fix this by explicitly calling
target_complete_cmd()when the final dataout PDU is received for an aborted WRITE command.target_complete_cmd()detects theCMD_T_ABORTEDflag and cleanly routes the command intotarget_abort_work, allowing the abort completion to successfully unblock.Shouldn't a Fixes: tag be added to this patch?
It's a bit difficult to pinpoint an exact commit.
Probably the roots of this bug could be found in aaa00cc93c1d0fd2693a ("scsi:
target/core: Fix TAS handling for aborted commands"), you are the author of that commit btw.
But at the time (we are talking about 4.x kernels) the abort mechanism was working in a different way, target_abort_work didn't even exists and target_complete_cmd() was quite different...
Maurizio
Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands
In reply to Maurizio LombardiQuoted 39 lines · Maurizio Lombardi
When a
LUN_RESETaborts a WRITE command that is in theTRANSPORT_WRITE_PENDINGstate, the target core setsCMD_T_ABORTEDand waits for the frontend to finish processing.If the initiator subsequently sends the remaining dataout PDUs,
__iscsit_check_dataout_hdr()catches the payload, stops the dataout timer if the sequence is final and finally dumps the data.
However, the iSCSI target doesn't trigger the completion process for these aborted commands. Because of this, the abort path hangs indefinitely intarget_put_cmd_and_wait(), leading to a deadlocked target worker thread.Fix this by explicitly calling
target_complete_cmd()when the final dataout PDU is received for an aborted WRITE command.target_complete_cmd()detects theCMD_T_ABORTEDflag and cleanly routes the command intotarget_abort_work, allowing the abort completion to successfully unblock.Signed-off-by Maurizio Lombardi <[email protected]>--- drivers/target/iscsi/iscsi_target.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/target/iscsi/iscsi_target.c +3
-2@@ -1533,8 +1533,10 @@ __iscsit_check_dataout_hdr(struct iscsit_conn *conn, void *buf,*/ if (se_cmd->transport_state &CMD_T_ABORTED ) {if (hdr->flags &ISCSI_FLAG_CMD_FINAL && - --cmd->outstanding_r2ts <1 ) + --cmd->outstanding_r2ts <1 ) {iscsit_stop_dataout_timer (cmd); +target_complete_cmd (se_cmd,SAM_STAT_TASK_ABORTED ); + }return iscsit_dump_data_payload (conn, payload_length,1 ); } --
Quoted 2 lines
2.55.0
Hi Maurizio,
We also faced that bug several years ago. We solved it by removing all leftovers of hanling of aborted commands in iscsi, because the handling of aborted commands moved to target core an age ago. iSCSI will collect data as usual and pass the command to target core, it in its turns will abort the command.
BR,
Dmitry
From d46bae4973c3467a999818b7a7154109de8facc1 Mon Sep 17 00:00:00 2001
From: Dmitry Bogdanov <[email protected]>
Date: Wed, 26 Jun 2024 14:30:34 +0300
Subject: [PATCH] scsi: target: iscsi: fix hanging WRITE_PENDING commands
When Data comes for an aborted WRITE command in WRITE_PENDING state it is silently received and the command does not pass to the target. That leads to hanging of other session with LUN_RESET aborting that command.
Pass the aborted WRITE commands when all Data has been gathered.
--- drivers/target/iscsi/iscsi_target.c | 17 ----------------- drivers/target/iscsi/iscsi_target_erl1.c | 8 -------- 2 files changed, 25 deletions(-)
drivers/target/iscsi/iscsi_target.c +0 -17
@@ -1536,23 +1536,6 @@ __iscsit_check_dataout_hdr(struct iscsit_conn *conn, void *buf,if (hdr->flags &ISCSI_FLAG_CMD_FINAL )iscsit_stop_dataout_timer (cmd); -return iscsit_dump_data_payload (conn, payload_length,1 ); - } - }else { -/* -* For the normal solicited data path: -* -* Check for a delayed TASK_ABORTED status and dump any -* incoming data out payload if one exists. Also, when the -* ISCSI_FLAG_CMD_FINAL is set to denote the end of the current -* data out sequence, we decrement outstanding_r2ts. Once -* outstanding_r2ts reaches zero, go ahead and send the delayed -* TASK_ABORTED status. -*/ -if (se_cmd->transport_state &CMD_T_ABORTED ) { -if (hdr->flags &ISCSI_FLAG_CMD_FINAL && - --cmd->outstanding_r2ts <1 ) -iscsit_stop_dataout_timer (cmd);return iscsit_dump_data_payload (conn, payload_length,1 ); }
drivers/target/iscsi/iscsi_target_erl1.c +0 -8
@@ -892,8 +892,6 @@ int iscsit_execute_cmd(struct iscsit_cmd *cmd, int ooo)return 0 ; }spin_unlock_bh (&cmd->istate_lock); -if (cmd->se_cmd.transport_state &CMD_T_ABORTED ) -return 0 ;return transport_send_check_condition_and_sense (se_cmd, cmd->sense_reason,0 ); }
@@ -911,9 +909,6 @@ int iscsit_execute_cmd(struct iscsit_cmd *cmd, int ooo)if (!(cmd->cmd_flags &ICF_NON_IMMEDIATE_UNSOLICITED_DATA )) { -if (cmd->se_cmd.transport_state &CMD_T_ABORTED ) -return 0 ; -iscsit_set_dataout_sequence_values (cmd); conn->conn_transport->iscsit_get_dataout (conn, cmd, false); }
@@ -926,9 +921,6 @@ int iscsit_execute_cmd(struct iscsit_cmd *cmd, int ooo)if ((cmd->data_direction ==DMA_TO_DEVICE ) && !(cmd->cmd_flags &ICF_NON_IMMEDIATE_UNSOLICITED_DATA )) { -if (cmd->se_cmd.transport_state &CMD_T_ABORTED ) -return 0 ; -iscsit_set_unsolicited_dataout (cmd); }return target_submit (&cmd->se_cmd);
Signature · 1 line
2.25.1
Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands
Quoted 46 lines · Dmitry Bogdanov
Quoted 39 lines · Maurizio Lombardi
When a
LUN_RESETaborts a WRITE command that is in theTRANSPORT_WRITE_PENDINGstate, the target core setsCMD_T_ABORTEDand waits for the frontend to finish processing.If the initiator subsequently sends the remaining dataout PDUs,
__iscsit_check_dataout_hdr()catches the payload, stops the dataout timer if the sequence is final and finally dumps the data.
However, the iSCSI target doesn't trigger the completion process for these aborted commands. Because of this, the abort path hangs indefinitely intarget_put_cmd_and_wait(), leading to a deadlocked target worker thread.Fix this by explicitly calling
target_complete_cmd()when the final dataout PDU is received for an aborted WRITE command.target_complete_cmd()detects theCMD_T_ABORTEDflag and cleanly routes the command intotarget_abort_work, allowing the abort completion to successfully unblock.Signed-off-by Maurizio Lombardi <[email protected]>--- drivers/target/iscsi/iscsi_target.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)drivers/target/iscsi/iscsi_target.c +3
-2@@ -1533,8 +1533,10 @@ __iscsit_check_dataout_hdr(struct iscsit_conn *conn, void *buf,*/ if (se_cmd->transport_state &CMD_T_ABORTED ) {if (hdr->flags &ISCSI_FLAG_CMD_FINAL && - --cmd->outstanding_r2ts <1 ) + --cmd->outstanding_r2ts <1 ) {iscsit_stop_dataout_timer (cmd); +target_complete_cmd (se_cmd,SAM_STAT_TASK_ABORTED ); + }return iscsit_dump_data_payload (conn, payload_length,1 ); } --Quoted 2 lines
2.55.0
Hi Maurizio,
Quoted 5 lines
We also faced that bug several years ago. We solved it by removing all leftovers of hanling of aborted commands in iscsi, because the handling of aborted commands moved to target core an age ago. iSCSI will collect data as usual and pass the command to target core, it in its turns will abort the command.
Hi Dmitry,
isn't Yadro considering upstreaming those changes?
Maurizio
Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands
In reply to Maurizio LombardiMaurizio,
Quoted 3 lines
When a
LUN_RESETaborts a WRITE command that is in theTRANSPORT_WRITE_PENDINGstate, the target core setsCMD_T_ABORTEDand waits for the frontend to finish processing.
Applied to 7.3/scsi-staging, thanks!
Signature · 1 line
Martin K. Petersen
Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands
In reply to Maurizio LombardiQuoted 4 lines · Maurizio Lombardi
When a
LUN_RESETaborts a WRITE command that is in theTRANSPORT_WRITE_PENDINGstate, the target core setsCMD_T_ABORTEDand waits for the frontend to finish processing.
Quoted 8 lines
If the initiator subsequently sends the remaining dataout PDUs,
__iscsit_check_dataout_hdr()catches the payload, stops the dataout timer if the sequence is final and finally dumps the data.
However, the iSCSI target doesn't trigger the completion process for these aborted commands. Because of this, the abort path hangs indefinitely intarget_put_cmd_and_wait(), leading to a deadlocked target worker thread.[...]
Applied to 7.3/scsi-fixes, thanks!
[1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands
https://git.kernel.org/mkp/scsi/c/d5869dae5080
Signature · 1 line
Martin K. Petersen
Quoted 16 lines · Maurizio Lombardi
One of our customers reported that sometimes the LIO driver hangs when performing a LUN reset while I/O is underway.
I was able to reproduce the hang by using the following libiscsi patch:
https://arkamax.eu/misc/dataout_abort_libiscsi.patch
INFO: task kworker/5:0:514619 blocked for more than 46 seconds. __wait_for_common+0x94/0x1b0 target_put_cmd_and_wait+0x6d/0xb0 [target_core_mod] core_tmr_drain_state_list+0x2a3/0x330 [target_core_mod] core_tmr_lun_reset+0x93/0x1a0 [target_core_mod] target_tmr_work+0xce/0x100 [target_core_mod] process_one_work+0x188/0x3b0 worker_thread+0x2ef/0x410
Quoted 5 lines
Maurizio Lombardi (1):
scsi: target: iscsi: Fix hang for abortedWRITE_PENDINGcommandsdrivers/target/iscsi/iscsi_target.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
Gentle ping on this one.
It fixes a hang that is impacting production workloads.
Thanks,
Maurizio