Skip to contentSkip to conversation

Write commands abort hang

PATCH Applied Maurizio Lombardi 6 participants 10 messages Jul 17 – Sep 3 SCSI target

lore
Share & tools
lore
#0 lore raw

cover Write commands abort hang

One of our customers reported that sometimes the LIO driver hangs when performing a LUN reset while I/O is underway.

I was able to reproduce the hang by using the following libiscsi patch:

https://arkamax.eu/misc/dataout_abort_libiscsi.patch

INFO: task kworker/5:0:514619 blocked for more than 46 seconds.
__wait_for_common+0x94/0x1b0
target_put_cmd_and_wait+0x6d/0xb0 [target_core_mod]
core_tmr_drain_state_list+0x2a3/0x330 [target_core_mod]
core_tmr_lun_reset+0x93/0x1a0 [target_core_mod]
target_tmr_work+0xce/0x100 [target_core_mod]
process_one_work+0x188/0x3b0
worker_thread+0x2ef/0x410

Maurizio Lombardi (1):
scsi: target: iscsi: Fix hang for aborted WRITE_PENDING commands

 drivers/target/iscsi/iscsi_target.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
Signature · 1 line
2.55.0
#1 lore raw

PATCH 1/1 target: iscsi: Fix hang for aborted WRITE_PENDING commands

When a LUN_RESET aborts a WRITE command that is in the TRANSPORT_WRITE_PENDING state, the target core sets CMD_T_ABORTED and waits for the frontend to finish processing.

If the initiator subsequently sends the remaining dataout PDUs, __iscsit_check_dataout_hdr() catches the payload, stops the dataout timer if the sequence is final and finally dumps the data.
However, the iSCSI target doesn't trigger the completion process for these aborted commands. Because of this, the abort path hangs indefinitely in target_put_cmd_and_wait(), leading to a deadlocked target worker thread.

Fix this by explicitly calling target_complete_cmd() when the final dataout PDU is received for an aborted WRITE command. target_complete_cmd() detects the CMD_T_ABORTED flag and cleanly routes the command into target_abort_work, allowing the abort completion to successfully unblock.

---
 drivers/target/iscsi/iscsi_target.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/target/iscsi/iscsi_target.c +3 -1
index 62ada3a52210..124ff269b8e7 100644
@@ -1533,8 +1533,10 @@ __iscsit_check_dataout_hdr(struct iscsit_conn *conn, void *buf,
 		 */
 		if (se_cmd->transport_state & CMD_T_ABORTED) {
 			if (hdr->flags & ISCSI_FLAG_CMD_FINAL &&
-			    --cmd->outstanding_r2ts < 1)
+			    --cmd->outstanding_r2ts < 1) {
 				iscsit_stop_dataout_timer(cmd);
+				target_complete_cmd(se_cmd, SAM_STAT_TASK_ABORTED);
+			}
 
 			return iscsit_dump_data_payload(conn, payload_length, 1);
 		}
Signature · 1 line
2.55.0
Reviewed-by on v1 1/1
#2 lore raw

Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands

Quoted 45 lines · Maurizio Lombardi

When a LUN_RESET aborts a WRITE command that is in the TRANSPORT_WRITE_PENDING state, the target core sets CMD_T_ABORTED and waits
for the frontend to finish processing.

If the initiator subsequently sends the remaining dataout PDUs, __iscsit_check_dataout_hdr() catches the payload, stops the dataout timer
if the sequence is final and finally dumps the data. However, the iSCSI target doesn't trigger the completion process for these
aborted commands. Because of this, the abort path hangs indefinitely in
target_put_cmd_and_wait(), leading to a deadlocked target worker thread.

Fix this by explicitly calling target_complete_cmd() when the final dataout
PDU is received for an aborted WRITE command. target_complete_cmd() detects
the CMD_T_ABORTED flag and cleanly routes the command into target_abort_work,
allowing the abort completion to successfully unblock.

---
 drivers/target/iscsi/iscsi_target.c | 4 +++-

 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/target/iscsi/iscsi_target.c +0 -0
index 62ada3a52210..124ff269b8e7 100644
@@ -1533,8 +1533,10 @@ __iscsit_check_dataout_hdr(struct iscsit_conn
 *conn, void *buf,
  		 */
  		if (se_cmd->transport_state & CMD_T_ABORTED) {

  if (hdr->flags & ISCSI_FLAG_CMD_FINAL &&

drivers/target/iscsi/iscsi_target.c +1 -1
-			    --cmd->outstanding_r2ts < 1)
+			    --cmd->outstanding_r2ts < 1) {

  iscsit_stop_dataout_timer(cmd);

drivers/target/iscsi/iscsi_target.c +1 -0
+				target_complete_cmd(se_cmd,

SAM_STAT_TASK_ABORTED);

drivers/target/iscsi/iscsi_target.c +1 -0
+			}
Quoted 4 lines


  return iscsit_dump_data_payload(conn, payload_length, 1);
  }

This took a long time to solve with many attempts that did not help until Maurizio's solution. This patch was tested fully at a customer to fully resolve the issue for over a week with no negative impacts.

Looks good.

#3 lore raw

Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands

In reply to Maurizio Lombardi
Quoted 11 lines · Maurizio Lombardi

When a LUN_RESET aborts a WRITE command that is in the TRANSPORT_WRITE_PENDING state, the target core sets CMD_T_ABORTED and waits for the frontend to finish processing.

If the initiator subsequently sends the remaining dataout PDUs, __iscsit_check_dataout_hdr() catches the payload, stops the dataout timer if the sequence is final and finally dumps the data.
However, the iSCSI target doesn't trigger the completion process for these aborted commands. Because of this, the abort path hangs indefinitely in target_put_cmd_and_wait(), leading to a deadlocked target worker thread.

Quoted 4 lines

Fix this by explicitly calling target_complete_cmd() when the final dataout PDU is received for an aborted WRITE command. target_complete_cmd() detects the CMD_T_ABORTED flag and cleanly routes the command into target_abort_work, allowing the abort completion to successfully unblock.

Shouldn't a Fixes: tag be added to this patch?

Thanks,

Bart.

#4 lore raw

Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands

Quoted 12 lines · Bart Van Assche
Quoted 11 lines · Maurizio Lombardi

When a LUN_RESET aborts a WRITE command that is in the TRANSPORT_WRITE_PENDING state, the target core sets CMD_T_ABORTED and waits for the frontend to finish processing.

If the initiator subsequently sends the remaining dataout PDUs, __iscsit_check_dataout_hdr() catches the payload, stops the dataout timer if the sequence is final and finally dumps the data.
However, the iSCSI target doesn't trigger the completion process for these aborted commands. Because of this, the abort path hangs indefinitely in target_put_cmd_and_wait(), leading to a deadlocked target worker thread.

Quoted 6 lines
Quoted 4 lines

Fix this by explicitly calling target_complete_cmd() when the final dataout PDU is received for an aborted WRITE command. target_complete_cmd() detects the CMD_T_ABORTED flag and cleanly routes the command into target_abort_work, allowing the abort completion to successfully unblock.

Shouldn't a Fixes: tag be added to this patch?

It's a bit difficult to pinpoint an exact commit.

Probably the roots of this bug could be found in aaa00cc93c1d0fd2693a ("scsi:
target/core: Fix TAS handling for aborted commands"), you are the author of that commit btw.

But at the time (we are talking about 4.x kernels) the abort mechanism was working in a different way, target_abort_work didn't even exists and target_complete_cmd() was quite different...

Maurizio

#5 lore raw

Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands

In reply to Maurizio Lombardi
Quoted 39 lines · Maurizio Lombardi

When a LUN_RESET aborts a WRITE command that is in the TRANSPORT_WRITE_PENDING state, the target core sets CMD_T_ABORTED and waits for the frontend to finish processing.

If the initiator subsequently sends the remaining dataout PDUs, __iscsit_check_dataout_hdr() catches the payload, stops the dataout timer if the sequence is final and finally dumps the data.
However, the iSCSI target doesn't trigger the completion process for these aborted commands. Because of this, the abort path hangs indefinitely in target_put_cmd_and_wait(), leading to a deadlocked target worker thread.

Fix this by explicitly calling target_complete_cmd() when the final dataout PDU is received for an aborted WRITE command. target_complete_cmd() detects the CMD_T_ABORTED flag and cleanly routes the command into target_abort_work, allowing the abort completion to successfully unblock.

---
 drivers/target/iscsi/iscsi_target.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/target/iscsi/iscsi_target.c +3 -2
index 62ada3a52210..124ff269b8e7 100644
@@ -1533,8 +1533,10 @@ __iscsit_check_dataout_hdr(struct iscsit_conn *conn, void *buf,
                 */
                if (se_cmd->transport_state & CMD_T_ABORTED) {
                        if (hdr->flags & ISCSI_FLAG_CMD_FINAL &&
-                           --cmd->outstanding_r2ts < 1)
+                           --cmd->outstanding_r2ts < 1) {
                                iscsit_stop_dataout_timer(cmd);
+                               target_complete_cmd(se_cmd, SAM_STAT_TASK_ABORTED);
+                       }
 
                        return iscsit_dump_data_payload(conn, payload_length, 1);
                }
--
Quoted 2 lines

2.55.0

Hi Maurizio,

We also faced that bug several years ago. We solved it by removing all leftovers of hanling of aborted commands in iscsi, because the handling of aborted commands moved to target core an age ago. iSCSI will collect data as usual and pass the command to target core, it in its turns will abort the command.

BR,
Dmitry

From d46bae4973c3467a999818b7a7154109de8facc1 Mon Sep 17 00:00:00 2001
From: Dmitry Bogdanov <[email protected]>
Date: Wed, 26 Jun 2024 14:30:34 +0300
Subject: [PATCH] scsi: target: iscsi: fix hanging WRITE_PENDING commands

When Data comes for an aborted WRITE command in WRITE_PENDING state it is silently received and the command does not pass to the target. That leads to hanging of other session with LUN_RESET aborting that command.

Pass the aborted WRITE commands when all Data has been gathered.

---
 drivers/target/iscsi/iscsi_target.c      | 17 -----------------
 drivers/target/iscsi/iscsi_target_erl1.c |  8 --------
 2 files changed, 25 deletions(-)
drivers/target/iscsi/iscsi_target.c +0 -17
index d1887337e..a18fe1c74 100644
@@ -1536,23 +1536,6 @@ __iscsit_check_dataout_hdr(struct iscsit_conn *conn, void *buf,
 			if (hdr->flags & ISCSI_FLAG_CMD_FINAL)
 				iscsit_stop_dataout_timer(cmd);
 
-			return iscsit_dump_data_payload(conn, payload_length, 1);
-		}
-	} else {
-		/*
-		 * For the normal solicited data path:
-		 *
-		 * Check for a delayed TASK_ABORTED status and dump any
-		 * incoming data out payload if one exists.  Also, when the
-		 * ISCSI_FLAG_CMD_FINAL is set to denote the end of the current
-		 * data out sequence, we decrement outstanding_r2ts.  Once
-		 * outstanding_r2ts reaches zero, go ahead and send the delayed
-		 * TASK_ABORTED status.
-		 */
-		if (se_cmd->transport_state & CMD_T_ABORTED) {
-			if (hdr->flags & ISCSI_FLAG_CMD_FINAL &&
-			    --cmd->outstanding_r2ts < 1)
-				iscsit_stop_dataout_timer(cmd);
 
 			return iscsit_dump_data_payload(conn, payload_length, 1);
 		}
drivers/target/iscsi/iscsi_target_erl1.c +0 -8
index 2e69632b3..6e8a0b41f 100644
@@ -892,8 +892,6 @@ int iscsit_execute_cmd(struct iscsit_cmd *cmd, int ooo)
 				return 0;
 			}
 			spin_unlock_bh(&cmd->istate_lock);
-			if (cmd->se_cmd.transport_state & CMD_T_ABORTED)
-				return 0;
 			return transport_send_check_condition_and_sense(se_cmd,
 					cmd->sense_reason, 0);
 		}
@@ -911,9 +909,6 @@ int iscsit_execute_cmd(struct iscsit_cmd *cmd, int ooo)
 
 			if (!(cmd->cmd_flags &
 					ICF_NON_IMMEDIATE_UNSOLICITED_DATA)) {
-				if (cmd->se_cmd.transport_state & CMD_T_ABORTED)
-					return 0;
-
 				iscsit_set_dataout_sequence_values(cmd);
 				conn->conn_transport->iscsit_get_dataout(conn, cmd, false);
 			}
@@ -926,9 +921,6 @@ int iscsit_execute_cmd(struct iscsit_cmd *cmd, int ooo)
 
 		if ((cmd->data_direction == DMA_TO_DEVICE) &&
 		    !(cmd->cmd_flags & ICF_NON_IMMEDIATE_UNSOLICITED_DATA)) {
-			if (cmd->se_cmd.transport_state & CMD_T_ABORTED)
-				return 0;
-
 			iscsit_set_unsolicited_dataout(cmd);
 		}
 		return target_submit(&cmd->se_cmd);
Signature · 1 line
2.25.1
#6 lore raw

Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands

Quoted 46 lines · Dmitry Bogdanov
Quoted 39 lines · Maurizio Lombardi

When a LUN_RESET aborts a WRITE command that is in the TRANSPORT_WRITE_PENDING state, the target core sets CMD_T_ABORTED and waits for the frontend to finish processing.

If the initiator subsequently sends the remaining dataout PDUs, __iscsit_check_dataout_hdr() catches the payload, stops the dataout timer if the sequence is final and finally dumps the data.
However, the iSCSI target doesn't trigger the completion process for these aborted commands. Because of this, the abort path hangs indefinitely in target_put_cmd_and_wait(), leading to a deadlocked target worker thread.

Fix this by explicitly calling target_complete_cmd() when the final dataout PDU is received for an aborted WRITE command. target_complete_cmd() detects the CMD_T_ABORTED flag and cleanly routes the command into target_abort_work, allowing the abort completion to successfully unblock.

---
 drivers/target/iscsi/iscsi_target.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/target/iscsi/iscsi_target.c +3 -2
index 62ada3a52210..124ff269b8e7 100644
@@ -1533,8 +1533,10 @@ __iscsit_check_dataout_hdr(struct iscsit_conn *conn, void *buf,
                 */
                if (se_cmd->transport_state & CMD_T_ABORTED) {
                        if (hdr->flags & ISCSI_FLAG_CMD_FINAL &&
-                           --cmd->outstanding_r2ts < 1)
+                           --cmd->outstanding_r2ts < 1) {
                                iscsit_stop_dataout_timer(cmd);
+                               target_complete_cmd(se_cmd, SAM_STAT_TASK_ABORTED);
+                       }
 
                        return iscsit_dump_data_payload(conn, payload_length, 1);
                }
--
Quoted 2 lines

2.55.0

Hi Maurizio,

Quoted 5 lines

We also faced that bug several years ago. We solved it by removing all leftovers of hanling of aborted commands in iscsi, because the handling of aborted commands moved to target core an age ago. iSCSI will collect data as usual and pass the command to target core, it in its turns will abort the command.

Hi Dmitry,

isn't Yadro considering upstreaming those changes?

Maurizio

maintainer Applied
#7 lore raw

Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands

In reply to Maurizio Lombardi

Maurizio,

Quoted 3 lines

When a LUN_RESET aborts a WRITE command that is in the TRANSPORT_WRITE_PENDING state, the target core sets CMD_T_ABORTED and waits for the frontend to finish processing.

Applied to 7.3/scsi-staging, thanks!

Signature · 1 line
Martin K. Petersen
maintainer Applied
#8 lore raw

Re: [PATCH 1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands

In reply to Maurizio Lombardi
Quoted 4 lines · Maurizio Lombardi

When a LUN_RESET aborts a WRITE command that is in the TRANSPORT_WRITE_PENDING state, the target core sets CMD_T_ABORTED and waits for the frontend to finish processing.

Quoted 8 lines

If the initiator subsequently sends the remaining dataout PDUs, __iscsit_check_dataout_hdr() catches the payload, stops the dataout timer if the sequence is final and finally dumps the data.
However, the iSCSI target doesn't trigger the completion process for these aborted commands. Because of this, the abort path hangs indefinitely in target_put_cmd_and_wait(), leading to a deadlocked target worker thread.

[...]

Applied to 7.3/scsi-fixes, thanks!

[1/1] target: iscsi: Fix hang for aborted WRITE_PENDING commands

      https://git.kernel.org/mkp/scsi/c/d5869dae5080
Signature · 1 line
Martin K. Petersen
#9 lore raw
In reply to Maurizio Lombardi
Quoted 16 lines · Maurizio Lombardi

One of our customers reported that sometimes the LIO driver hangs when performing a LUN reset while I/O is underway.

I was able to reproduce the hang by using the following libiscsi patch:

https://arkamax.eu/misc/dataout_abort_libiscsi.patch

INFO: task kworker/5:0:514619 blocked for more than 46 seconds.
__wait_for_common+0x94/0x1b0
target_put_cmd_and_wait+0x6d/0xb0 [target_core_mod]
core_tmr_drain_state_list+0x2a3/0x330 [target_core_mod]
core_tmr_lun_reset+0x93/0x1a0 [target_core_mod]
target_tmr_work+0xce/0x100 [target_core_mod]
process_one_work+0x188/0x3b0
worker_thread+0x2ef/0x410
Quoted 5 lines

Maurizio Lombardi (1):
scsi: target: iscsi: Fix hang for aborted WRITE_PENDING commands

 drivers/target/iscsi/iscsi_target.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

Gentle ping on this one.
It fixes a hang that is impacting production workloads.

Thanks,
Maurizio