Skip to contentSkip to conversation

scsi: target: iscsi: fix stalled aborted WRITE commands

PATCH Superseded Laurence Oberman 3 participants 8 messages Oct 16 – Oct 26

Subsystems (2)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v2 cover scsi: target: iscsi: fix stalled aborted WRITE commands

These two patches apply on top of
commit d5869dae5080 ("scsi: target: iscsi: Fix hang for aborted WRITE_PENDING commands").

A WRITE command that is still in TRANSPORT_WRITE_PENDING has never been passed to the backend, so when a TMR aborts it the iSCSI fabric is the one that has to complete it. That commit fixed the case where the final DataOut PDU for an already-issued R2T arrives after the abort. Two more paths can leave such a command with nothing to complete it:

- Patch 1 is the hang this work started from. A WRITE that arrives with immediate data, aborted after that data has been received but before the first R2T is built, is never completed by anything: no R2T has been sent so the initiator cannot send more data, and no DataOut timer is armed so the ERL=0 timeout cannot fire either. core_tmr_lun_reset() then blocks for as long as the connection stays up.

- Patch 2 closes the equivalent window where a new R2T would have been sent for an already-aborted command.

Patch 1 also adds a one-shot completion claim. Completing the command in iscsit_execute_cmd() introduces a second site which iscsit_release_commands_from_conn() would also complete during connection teardown, because t_state is still TRANSPORT_WRITE_PENDING there and target_complete_cmd() on an aborted command records nothing that would say otherwise. The claim makes sure only one of them does it.

Changes since v1:
- Patch 2: route the DATAOUT_SEND_R2T completion through iscsit_complete_aborted_cmd() rather than calling target_complete_cmd() directly, so the completion is claimed and cannot be repeated by iscsit_release_commands_from_conn() during connection teardown.
- Patch 1 unchanged.

v1: https://lore.kernel.org/linux-scsi/[email protected]/

How this was found and tested:

The hang was reported against an Oracle RAC cluster on two-session iSCSI LIO targets, where one node's LUN_RESET aborts WRITEs belonging to the other node's session. It was not reproducible in a lab, so the diagnosis came from instrumented kernels in the reporter's environment, logging every site that completes or waits on an aborted command.

Across two hosts every stall had the identical signature given in patch 1, and the stall only ever ended when the initiator gave up and dropped the connection. 217 of 222 LUN_RESET drains completed in under a second; the rest took 46 to 54 seconds, which is where the hung-task panics came from. One 53.7 second stall recovered on its own and the host ran for a further 19 hours.

Both patches, backported to a 5.14-based distro kernel, have since run 300 reset cycles with no hang, in two independent runs: one instrumented and one with the tracing removed. The same reproducer previously failed at 95 and 196 cycles. The code here is equivalent to what was tested; only comments differ.

One thing to flag for review: patch 2's window was never observed. The instrumented kernels recorded no completions at that site in either run, and the reproducer never reached it. It is included because the same reasoning applies there, but it has no field evidence behind it, and I am happy to drop it or send it separately if you would prefer.

An earlier attempt at this fix completed the command from core_tmr_drain_state_list() instead. That is not safe: at that point the initiator may still be sending DataOut for the command, and the result was a double completion -- re-queued se_cmd->work, list_del corruption and a refcount underflow. Mentioning it in case the same idea comes up in review.

Laurence Oberman (2):
scsi: target: iscsi: Complete an aborted WRITE stuck before the first

    R2T

scsi: target: iscsi: Complete an aborted WRITE when a new R2T would be
sent

 drivers/target/iscsi/iscsi_target.c      | 28 +++++++++++++--
 drivers/target/iscsi/iscsi_target_erl1.c | 10 +++++-
 drivers/target/iscsi/iscsi_target_util.c | 46 ++++++++++++++++++++++++
 drivers/target/iscsi/iscsi_target_util.h |  2 ++
 include/target/iscsi/iscsi_target_core.h |  5 +++
 5 files changed, 88 insertions(+), 3 deletions(-)
Signature · 1 line
2.52.0
#1 lore raw

PATCH v2 1/2 scsi: target: iscsi: Complete an aborted WRITE stuck before the first R2T

A WRITE that arrives with immediate data, and whose data length exceeds FirstBurstLength, is left in TRANSPORT_WRITE_PENDING by iscsit_execute_cmd() until the first R2T is built. If a LUN_RESET or ABORT_TASK from another session sets CMD_T_ABORTED in that window, iscsit_execute_cmd() returns without sending the R2T.

Nothing then completes the command. No R2T has been sent, so the initiator cannot legally send more data; no DataOut timer was armed, so the ERL=0 timeout cannot fire either. The command stays in TRANSPORT_WRITE_PENDING with CMD_T_ABORTED set, and core_tmr_drain_state_list() blocks in target_put_cmd_and_wait() on it for as long as the connection stays up:

  INFO: task kworker/5:2 blocked for more than 46 seconds.
   target_put_cmd_and_wait+0x6d/0xb0 [target_core_mod]
   core_tmr_drain_state_list+0x2dc/0x380 [target_core_mod]
   core_tmr_lun_reset+0x93/0x1a0 [target_core_mod]

Complete it in iscsit_execute_cmd() instead. No DataOut path can race with this: without an R2T the initiator cannot send further data for the command, and the branch is only reached when no unsolicited DataOut is expected either.

That introduces a second site which can complete a command that iscsit_release_commands_from_conn() would also complete during connection teardown, because t_state is still TRANSPORT_WRITE_PENDING there. target_complete_cmd() on a command that already has CMD_T_ABORTED set goes straight into target_cmd_interrupted(), which queues target_abort_work and returns without setting CMD_T_COMPLETE and without changing t_state, so nothing in target core records that a completion has happened. A second target_complete_cmd() re-initialises and re-queues se_cmd->work while it is still queued and drops a command reference that was never taken. Add a one-shot claim, taken under se_cmd->t_state_lock, so that only one of the two sites completes the command.

This was reported against an Oracle RAC cluster on two-session iSCSI LIO targets, where one node's LUN_RESET aborts WRITEs belonging to the other node's session. It was not reproducible in a lab, so the diagnosis came from instrumented kernels in the reporter's environment. Across two hosts, every stalled command had the same signature: DMA_TO_DEVICE, TRANSPORT_WRITE_PENDING, CMD_T_ABORTED | CMD_T_TAS, outstanding_r2ts == 0, write_data_done == first_burst_len, and an empty cmd_r2t_list. 217 of 222 LUN_RESET drains completed in under a second; the rest took 46 to 54 seconds, which is where the hung-task panics came from. One 53.7 second stall recovered on its own and the host ran for a further 19 hours. The stall only ever ended when the initiator gave up and dropped the connection, which let iscsit_release_commands_from_conn() complete the command.

---
 drivers/target/iscsi/iscsi_target.c      | 12 +++++--
 drivers/target/iscsi/iscsi_target_erl1.c | 10 +++++-
 drivers/target/iscsi/iscsi_target_util.c | 46 ++++++++++++++++++++++++
 drivers/target/iscsi/iscsi_target_util.h |  2 ++
 include/target/iscsi/iscsi_target_core.h |  5 +++
 5 files changed, 72 insertions(+), 3 deletions(-)
drivers/target/iscsi/iscsi_target.c +10 -2
index 124ff269b8e7..aab9237a6776 100644
@@ -4198,10 +4198,18 @@ static void iscsit_release_commands_from_conn(struct iscsit_conn *conn)
 		if (cmd->se_cmd.t_state == TRANSPORT_WRITE_PENDING) {
 			/*
 			 * We never submitted the cmd to LIO core, so we have
-			 * to tell LIO to perform the completion process.
+			 * to tell LIO to perform the completion process --
+			 * unless iscsit_execute_cmd() already did.  t_state
+			 * stays TRANSPORT_WRITE_PENDING across that
+			 * completion, so the claim is the only thing that can
+			 * tell us.
 			 */
+			bool claimed = iscsit_claim_completion_locked(cmd);
+
 			spin_unlock_irq(&se_cmd->t_state_lock);
-			target_complete_cmd(&cmd->se_cmd, SAM_STAT_TASK_ABORTED);
+			if (claimed)
+				target_complete_cmd(&cmd->se_cmd,
+						    SAM_STAT_TASK_ABORTED);
 			continue;
 		}
 		spin_unlock_irq(&se_cmd->t_state_lock);
drivers/target/iscsi/iscsi_target_erl1.c +9 -1
index d8ca06e697d6..cf72c79be355 100644
@@ -927,8 +927,16 @@ int iscsit_execute_cmd(struct iscsit_cmd *cmd, int ooo)
 
 			if (!(cmd->cmd_flags &
 					ICF_NON_IMMEDIATE_UNSOLICITED_DATA)) {
-				if (cmd->se_cmd.transport_state & CMD_T_ABORTED)
+				/*
+				 * No R2T has been sent for this command yet,
+				 * so returning here leaves the initiator with
+				 * nothing to send and no DataOut timer armed.
+				 * Complete it, or nothing ever will.
+				 */
+				if (cmd->se_cmd.transport_state & CMD_T_ABORTED) {
+					iscsit_complete_aborted_cmd(cmd);
 					return 0;
+				}
 
 				iscsit_set_dataout_sequence_values(cmd);
 				conn->conn_transport->iscsit_get_dataout(conn, cmd, false);
drivers/target/iscsi/iscsi_target_util.c +46 -0

58 lines not on this page · raw message

drivers/target/iscsi/iscsi_target_util.h +2 -0
index 7ae48a8a5cbf..8f3fe7428428 100644
@@ -18,6 +18,8 @@ extern struct iscsi_r2t *iscsit_get_r2t_from_list(struct iscsit_cmd *);
 extern void iscsit_free_r2t(struct iscsi_r2t *, struct iscsit_cmd *);
 extern void iscsit_free_r2ts_from_list(struct iscsit_cmd *);
 extern struct iscsit_cmd *iscsit_allocate_cmd(struct iscsit_conn *, int);
+extern bool iscsit_claim_completion_locked(struct iscsit_cmd *cmd);
+extern void iscsit_complete_aborted_cmd(struct iscsit_cmd *cmd);
 extern struct iscsi_seq *iscsit_get_seq_holder_for_datain(struct iscsit_cmd *, u32);
 extern struct iscsi_seq *iscsit_get_seq_holder_for_r2t(struct iscsit_cmd *);
 extern struct iscsi_r2t *iscsit_get_holder_for_r2tsn(struct iscsit_cmd *, u32);
include/target/iscsi/iscsi_target_core.h +5 -0
index 51ca80abacf7..0707a9520810 100644
@@ -496,6 +496,11 @@ struct iscsit_cmd {
 	u32			first_data_sg_off;
 	u32			kmapped_nents;
 	sense_reason_t		sense_reason;
+	/*
+	 * Set once the fabric has called target_complete_cmd() for this
+	 * command.  Protected by se_cmd.t_state_lock.
+	 */
+	bool			completion_claimed;
 }  ____cacheline_aligned;
 
 struct iscsi_tmr_req {
Signature · 1 line
2.52.0
#2 lore raw

PATCH v2 2/2 scsi: target: iscsi: Complete an aborted WRITE when a new R2T would be sent

In reply to Laurence Oberman

iscsit_check_dataout_payload() hands a command to iscsit_get_dataout() to request the next burst whenever iscsit_check_post_dataout() returns DATAOUT_SEND_R2T, without checking CMD_T_ABORTED first.

If a TMR aborted the command while the burst that just finished was in flight, no further DataOut will arrive: the initiator is about to be told the command was aborted. The CMD_T_ABORTED handling in __iscsit_check_dataout_hdr() does not cover this, because it only runs for DataOut PDUs belonging to R2Ts that were already issued, and outstanding_r2ts has already been decremented to zero for the burst that completed. The command is left in TRANSPORT_WRITE_PENDING with nothing to complete it.

Complete it here instead, through iscsit_complete_aborted_cmd() so that the completion is claimed. target_complete_cmd() on a command that already has CMD_T_ABORTED set goes to target_cmd_interrupted(), which queues target_abort_work and returns without setting CMD_T_COMPLETE or changing t_state. The command therefore stays in TRANSPORT_WRITE_PENDING and on conn_cmd_list until target_abort_work reaches iscsit_aborted_task(), and a connection drop in that window would let iscsit_release_commands_from_conn() complete it a second time. Guard on outstanding_r2ts so that with MaxOutstandingR2T > 1 the completion is still left to the last R2T to drain in __iscsit_check_dataout_hdr().

Unlike patch 1 of this series, this window was never observed in the
field: instrumented kernels on two hosts recorded no completions at this site across 300 reset cycles, and the reproducer that triggered the other one never reached it. It is closed by code inspection, for symmetry with the other paths that can leave an aborted WRITE_PENDING command stranded.

---
 drivers/target/iscsi/iscsi_target.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
drivers/target/iscsi/iscsi_target.c +16 -0
index aab9237a6776..3dad7f69f3ba 100644
@@ -1669,6 +1669,22 @@ iscsit_check_dataout_payload(struct iscsit_cmd *cmd, struct iscsi_data *hdr,
 	if ((rc == DATAOUT_NORMAL) || (rc == DATAOUT_WITHIN_COMMAND_RECOVERY))
 		return 0;
 	else if (rc == DATAOUT_SEND_R2T) {
+		/*
+		 * A new R2T would never be answered for an aborted command.
+		 * The CMD_T_ABORTED handling in __iscsit_check_dataout_hdr()
+		 * does not cover this: it only runs for DataOut PDUs
+		 * belonging to R2Ts that were already issued, and
+		 * iscsit_dataout_update_r2t() has already decremented
+		 * outstanding_r2ts for the burst that just finished.  Guard
+		 * on outstanding_r2ts so that with MaxOutstandingR2T > 1 the
+		 * completion is still left to the last R2T to drain.
+		 */
+		if (cmd->se_cmd.transport_state & CMD_T_ABORTED) {
+			if (cmd->outstanding_r2ts < 1)
+				iscsit_complete_aborted_cmd(cmd);
+			return 0;
+		}
+
 		iscsit_set_dataout_sequence_values(cmd);
 		conn->conn_transport->iscsit_get_dataout(conn, cmd, false);
 	} else if (rc == DATAOUT_SEND_TO_TRANSPORT) {
Signature · 1 line
2.52.0
#3 lore raw

Re: [PATCH v2 2/2] scsi: target: iscsi: Complete an aborted WRITE when a new R2T would be sent

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [High] iscsi: double completion due to incomplete CMD_T_ABORTED fix - [Medium] iscsi: data race on…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] iscsi: double completion due to incomplete CMD_T_ABORTED fix
- [Medium] iscsi: data race on cmd->outstanding_r2ts during abort
--

commit 538cfb5a3bbcee47d2aa78e30d5cf64fe5bed2fe
Author: Laurence Oberman <[email protected]>

scsi: target: iscsi: Complete an aborted WRITE when a new R2T would be sent

This commit adds a check for aborted commands in the DataOut processing path when a new R2T is requested. It ensures the command is safely completed using iscsit_complete_aborted_cmd() to claim the completion. This prevents a race condition where a connection drop could cause a double-completion of the aborted command.

Quoted 10 lines
drivers/target/iscsi/iscsi_target.c +2 -0
index aab9237a67768..3dad7f69f3bae 100644
@@ -1669,6 +1669,22 @@ iscsit_check_dataout_payload(struct iscsit_cmd *cmd, struct iscsi_data *hdr,
 	if ((rc == DATAOUT_NORMAL) || (rc == DATAOUT_WITHIN_COMMAND_RECOVERY))
 		return 0;
 	else if (rc == DATAOUT_SEND_R2T) {
+		/*
+		 * A new R2T would never be answered for an aborted command.

drivers/target/iscsi/iscsi_target.c iscsit_check_dataout_payload(struct iscsit_cmd *cmd, struct iscsi_data *hdr, · line 1683

Quoted 10 lines
drivers/target/iscsi/iscsi_target.c +10 -0
+		 * The CMD_T_ABORTED handling in __iscsit_check_dataout_hdr()
+		 * does not cover this: it only runs for DataOut PDUs
+		 * belonging to R2Ts that were already issued, and
+		 * iscsit_dataout_update_r2t() has already decremented
+		 * outstanding_r2ts for the burst that just finished.  Guard
+		 * on outstanding_r2ts so that with MaxOutstandingR2T > 1 the
+		 * completion is still left to the last R2T to drain.
+		 */
+		if (cmd->se_cmd.transport_state & CMD_T_ABORTED) {
+			if (cmd->outstanding_r2ts < 1)

[Severity: High]
This patch introduces iscsit_complete_aborted_cmd() to fix a double-completion race by safely claiming the completion. However, does this still leave two other paths vulnerable to the same issue?

If cmd->outstanding_r2ts >= 1, the patch explicitly leaves completion to __iscsit_check_dataout_hdr(). But __iscsit_check_dataout_hdr() uses target_complete_cmd() directly without claiming the completion:

__iscsit_check_dataout_hdr() {
    ...
		if (se_cmd->transport_state & CMD_T_ABORTED) {
			if (hdr->flags & ISCSI_FLAG_CMD_FINAL &&
			    --cmd->outstanding_r2ts < 1) {
				iscsit_stop_dataout_timer(cmd);
				target_complete_cmd(se_cmd, SAM_STAT_TASK_ABORTED);
			}
    ...
}

Similarly, if the final payload for an aborted command is received in this function, it reaches DATAOUT_SEND_TO_TRANSPORT and calls target_execute_cmd(), which also queues the abort work without claiming the completion:

iscsit_check_dataout_payload() {
    ...
	} else if (rc == DATAOUT_SEND_TO_TRANSPORT) {
        ...
		target_execute_cmd(&cmd->se_cmd);
		return 0;
    ...
}

Could a connection drop in either of these windows allow iscsit_release_commands_from_conn() to complete the command a second time, resulting in corruption of the target_completion_wq?

[Severity: Medium]
Is it safe to read cmd->outstanding_r2ts without holding cmd->r2t_lock here?

If iscsit_build_r2ts_for_cmd() encounters an allocation failure, the target core queues the command to qf_work_queue. The worker thread will retry building the R2T asynchronously via transport_write_pending_qf() -> iscsit_build_r2ts_for_cmd(), incrementing the counter under cmd->r2t_lock:

iscsit_build_r2ts_for_cmd() {
    ...
		cmd->outstanding_r2ts++;
		first_r2t = 0;

		if (iscsit_add_r2t_to_list(cmd, offset, xfer_len, 0, 0) < 0) {
    ...
}

If the command is aborted concurrently, the RX thread processing incoming DataOut PDUs will locklessly decrement the counter in __iscsit_check_dataout_hdr() and locklessly read it here. Could this concurrent non-atomic access cause lost updates, leading to a corrupted counter and a leaked command?

Quoted 7 lines
drivers/target/iscsi/iscsi_target.c +4 -0
+				iscsit_complete_aborted_cmd(cmd);
+			return 0;
+		}
+
 		iscsit_set_dataout_sequence_values(cmd);
 		conn->conn_transport->iscsit_get_dataout(conn, cmd, false);
 	} else if (rc == DATAOUT_SEND_TO_TRANSPORT) {
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#4 lore raw

Re: [PATCH v2 2/2] scsi: target: iscsi: Complete an aborted WRITE when a new R2T would be sent

Quoted 126 lines · On Fri, 2026-10-16 at 18:08 +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] iscsi: double completion due to incomplete CMD_T_ABORTED fix
- [Medium] iscsi: data race on cmd->outstanding_r2ts during abort --

commit 538cfb5a3bbcee47d2aa78e30d5cf64fe5bed2fe
Author: Laurence Oberman <[email protected]>

scsi: target: iscsi: Complete an aborted WRITE when a new R2T would be sent

This commit adds a check for aborted commands in the DataOut processing
path when a new R2T is requested. It ensures the command is safely completed using iscsit_complete_aborted_cmd() to claim the completion.
This prevents a race condition where a connection drop could cause a double-completion of the aborted command.

Quoted 27 lines
drivers/target/iscsi/iscsi_target.c +0 -0
index aab9237a67768..3dad7f69f3bae 100644
@@ -1669,6 +1669,22 @@ iscsit_check_dataout_payload(struct
 iscsit_cmd *cmd, struct iscsi_data *hdr,

  if ((rc == DATAOUT_NORMAL) || (rc ==
DATAOUT_WITHIN_COMMAND_RECOVERY))
  return 0;
  else if (rc == DATAOUT_SEND_R2T) {

drivers/target/iscsi/iscsi_target.c +2 -0
+		/*
+		 * A new R2T would never be answered for an

aborted command.

drivers/target/iscsi/iscsi_target.c +1 -0
+		 * The CMD_T_ABORTED handling in

__iscsit_check_dataout_hdr()

drivers/target/iscsi/iscsi_target.c +1 -0
+		 * does not cover this: it only runs for DataOut

PDUs

drivers/target/iscsi/iscsi_target.c +2 -0
+		 * belonging to R2Ts that were already issued, and
+		 * iscsit_dataout_update_r2t() has already

decremented

drivers/target/iscsi/iscsi_target.c +1 -0
+		 * outstanding_r2ts for the burst that just

finished.  Guard

drivers/target/iscsi/iscsi_target.c +1 -0
+		 * on outstanding_r2ts so that with

MaxOutstandingR2T > 1 the

drivers/target/iscsi/iscsi_target.c +1 -0
+		 * completion is still left to the last R2T to

drain.

drivers/target/iscsi/iscsi_target.c iscsit_check_dataout_payload(struct iscsit_cmd *cmd, struct iscsi_data *hdr, · line 1683

Quoted 3 lines
drivers/target/iscsi/iscsi_target.c +3 -0
+		 */
+		if (cmd->se_cmd.transport_state & CMD_T_ABORTED) {
+			if (cmd->outstanding_r2ts < 1)

[Severity: High]
This patch introduces iscsit_complete_aborted_cmd() to fix a double- completion
race by safely claiming the completion. However, does this still leave two
other paths vulnerable to the same issue?

If cmd->outstanding_r2ts >= 1, the patch explicitly leaves completion to
__iscsit_check_dataout_hdr(). But __iscsit_check_dataout_hdr() uses target_complete_cmd() directly without claiming the completion:

__iscsit_check_dataout_hdr() {
    ...

		if (se_cmd->transport_state & CMD_T_ABORTED) {
			if (hdr->flags & ISCSI_FLAG_CMD_FINAL &&
			    --cmd->outstanding_r2ts < 1) {
				iscsit_stop_dataout_timer(cmd);
				target_complete_cmd(se_cmd,
SAM_STAT_TASK_ABORTED);
			}

    ...
}

Similarly, if the final payload for an aborted command is received in this
function, it reaches DATAOUT_SEND_TO_TRANSPORT and calls target_execute_cmd(),
which also queues the abort work without claiming the completion:

iscsit_check_dataout_payload() {
    ...

	} else if (rc == DATAOUT_SEND_TO_TRANSPORT) {
        ...
		target_execute_cmd(&cmd->se_cmd);
		return 0;

    ...
}

Could a connection drop in either of these windows allow iscsit_release_commands_from_conn() to complete the command a second time,
resulting in corruption of the target_completion_wq?

[Severity: Medium]
Is it safe to read cmd->outstanding_r2ts without holding cmd-

Quoted 1 line

r2t_lock here?

If iscsit_build_r2ts_for_cmd() encounters an allocation failure, the target
core queues the command to qf_work_queue. The worker thread will retry
building the R2T asynchronously via transport_write_pending_qf() -> iscsit_build_r2ts_for_cmd(), incrementing the counter under cmd-

Quoted 1 line

r2t_lock:

iscsit_build_r2ts_for_cmd() {
    ...

		cmd->outstanding_r2ts++;
		first_r2t = 0;

		if (iscsit_add_r2t_to_list(cmd, offset, xfer_len, 0,

0) < 0) {
    ...
}

If the command is aborted concurrently, the RX thread processing incoming
DataOut PDUs will locklessly decrement the counter in __iscsit_check_dataout_hdr() and locklessly read it here. Could this concurrent non-atomic access cause lost updates, leading to a corrupted
counter and a leaked command?

Quoted 8 lines
Quoted 8 lines
drivers/target/iscsi/iscsi_target.c +4 -0
+				iscsit_complete_aborted_cmd(cmd);
+			return 0;
+		}
+

  iscsit_set_dataout_sequence_values(cmd); conn->conn_transport->iscsit_get_dataout(conn, cmd, false);
  } else if (rc == DATAOUT_SEND_TO_TRANSPORT) {

Both of these look like repeats or non-issues to me.

On the first: this is the same finding the review raised against patch 1
of v1, which I answered on that thread. The exposure in __iscsit_check_dataout_hdr() and in the DATAOUT_SEND_TO_TRANSPORT path into target_execute_cmd() is pre-existing - mainline calls target_complete_cmd() unconditionally in iscsit_release_commands_from_conn() today, so both are already reachable
without this series. v2 does not widen them. I have offered to post a separate patch routing those two sites through the claim, and that offer
stands.

On the second: r2t_lock is indeed the lock for outstanding_r2ts, and six
of the seven accesses in the subsystem hold it - the increment in iscsit_build_r2ts_for_cmd(), the decrement in iscsit_dataout_update_r2t(), and the three TMR sites. The one that does not is the --cmd->outstanding_r2ts in __iscsit_check_dataout_hdr(), which
is pre-existing and is the only unlocked read-modify-write of the field.

The line this patch adds is a read, not an RMW, so it cannot contribute to the lost update described. A stale read is also harmless here: if it reads low, iscsit_complete_aborted_cmd() takes the claim and the later completion in __iscsit_check_dataout_hdr() becomes a no-op; if it reads high, the completion is left to that path or to connection teardown. Exactly one completion happens either way, which is what the claim is for.

Taking r2t_lock for the read would not help and cannot be done as
written: iscsit_complete_aborted_cmd() calls iscsit_stop_dataout_timer(),
which sleeps in timer_delete_sync(), so the lock would have to be dropped
before the completion and the value could change in that gap regardless.

The inconsistent locking on that pre-existing decrement is a fair observation and I will fold it into the follow-up patch for the two unclaimed sites.

Given how much testing went into this set of two patches, 300 attempts and was stable I am inclined to just make the v2 change and that's it.

Thanks
Laurence

#5 lore raw
In reply to Laurence Oberman
Quoted 116 lines · Laurence Oberman
Quoted 13 lines

These two patches apply on top of
commit d5869dae5080 ("scsi: target: iscsi: Fix hang for aborted WRITE_PENDING commands").

A WRITE command that is still in TRANSPORT_WRITE_PENDING has never been
passed to the backend, so when a TMR aborts it the iSCSI fabric is the one
that has to complete it. That commit fixed the case where the final DataOut PDU for an already-issued R2T arrives after the abort. Two more
paths can leave such a command with nothing to complete it:

Quoted 5 lines

 - Patch 1 is the hang this work started from. A WRITE that arrives with
   immediate data, aborted after that data has been received but before
   the first R2T is built, is never completed by anything: no R2T

has

Quoted 2 lines

been
   sent so the initiator cannot send more data, and no DataOut

timer

Quoted 10 lines

is
   armed so the ERL=0 timeout cannot fire either. core_tmr_lun_reset()
   then blocks for as long as the connection stays up.

 - Patch 2 closes the equivalent window where a new R2T would have been
   sent for an already-aborted command.

Patch 1 also adds a one-shot completion claim. Completing the

command

Quoted 5 lines

in
iscsit_execute_cmd() introduces a second site which iscsit_release_commands_from_conn() would also complete during connection
teardown, because t_state is still TRANSPORT_WRITE_PENDING there

and

Quoted 12 lines

target_complete_cmd() on an aborted command records nothing that would say
otherwise. The claim makes sure only one of them does it.

Changes since v1:
 - Patch 2: route the DATAOUT_SEND_R2T completion through iscsit_complete_aborted_cmd() rather than calling target_complete_cmd()
   directly, so the completion is claimed and cannot be repeated by iscsit_release_commands_from_conn() during connection teardown.
 - Patch 1 unchanged.

Quoted 2 lines

v1:

https://lore.kernel.org/linux-scsi/[email protected]/

Quoted 10 lines

How this was found and tested:

The hang was reported against an Oracle RAC cluster on two-session iSCSI
LIO targets, where one node's LUN_RESET aborts WRITEs belonging to the
other node's session. It was not reproducible in a lab, so the diagnosis
came from instrumented kernels in the reporter's environment,

logging

Quoted 6 lines

every
site that completes or waits on an aborted command.

Across two hosts every stall had the identical signature given in patch 1,
and the stall only ever ended when the initiator gave up and

dropped

Quoted 2 lines

the
connection. 217 of 222 LUN_RESET drains completed in under a

second;

Quoted 2 lines

the
rest took 46 to 54 seconds, which is where the hung-task panics

came

Quoted 8 lines

from.
One 53.7 second stall recovered on its own and the host ran for a further
19 hours.

Both patches, backported to a 5.14-based distro kernel, have since run 300
reset cycles with no hang, in two independent runs: one

instrumented

Quoted 4 lines

and
one with the tracing removed. The same reproducer previously failed at 95
and 196 cycles. The code here is equivalent to what was tested;

only

Quoted 7 lines

comments differ.

One thing to flag for review: patch 2's window was never observed. The
instrumented kernels recorded no completions at that site in either run,
and the reproducer never reached it. It is included because the

same

Quoted 1 line

reasoning applies there, but it has no field evidence behind it,

and

Quoted 5 lines

I am
happy to drop it or send it separately if you would prefer.

An earlier attempt at this fix completed the command from core_tmr_drain_state_list() instead. That is not safe: at that

point

Quoted 8 lines

the
initiator may still be sending DataOut for the command, and the result was
a double completion -- re-queued se_cmd->work, list_del corruption and a
refcount underflow. Mentioning it in case the same idea comes up in review.

Quoted 16 lines
Quoted 5 lines

Laurence Oberman (2):
  scsi: target: iscsi: Complete an aborted WRITE stuck before the first
    R2T
  scsi: target: iscsi: Complete an aborted WRITE when a new R2T

would

Quoted 10 lines

be
    sent

 drivers/target/iscsi/iscsi_target.c      | 28 +++++++++++++-- drivers/target/iscsi/iscsi_target_erl1.c | 10 +++++- drivers/target/iscsi/iscsi_target_util.c | 46
++++++++++++++++++++++++
 drivers/target/iscsi/iscsi_target_util.h |  2 ++ include/target/iscsi/iscsi_target_core.h |  5 +++

 5 files changed, 88 insertions(+), 3 deletions(-)

Gentle ping:
If anybody has bandwidth to review the v2 in this series which also handled the sashiko concerns.
We have customers waiting on this series.

 As always much appreciated
 Laurence Oberman

#6 lore raw
Quoted 1 line · Laurence Oberman

Gentle ping:

Please take a look at the attached review report. The report has been generated by an LLM. As known LLMs may make mistakes.

Bart.

Review https://lore.kernel.org/all/[email protected]/

I've reviewed v2 of Laurence Oberman's two-patch series that fixes stalled aborted iSCSI WRITE commands, plus the Sashiko bot's review and the author's reply. I checked it against the local tree, which already contains the prerequisite commit d5869dae5080.

**Overall:** the diagnosis for patch 1 looks right, and the "claim the completion once" idea is reasonable. But the claim is only applied in some of the places that complete an aborted command. So the double-completion problem the series describes is only partly fixed, and part of that problem was introduced by d5869dae5080 itself.

## Main issues

**1. Not every completion site takes the claim, and one of the gaps is a recent regression.**
- After d5869dae5080, __iscsit_check_dataout_hdr() calls target_complete_cmd() on an aborted TRANSPORT_WRITE_PENDING command without taking the claim.
- When the abort is the asynchronous kind (target_cmd_interrupted() → target_abort_work), t_state is never changed. So if the connection drops before the command is freed, iscsit_release_commands_from_conn() completes it a second time.
- That second completion is the same INIT_WORK()/queue_work() on a still-queued se_cmd->work that the cover letter describes as the failure of the earlier attempt.
- d5869dae5080 is only in 7.3/scsi-staging so far. The fix belongs in this series, with Fixes: d5869dae5080, not in a later follow-up.
- iscsit_check_dataout_payload() → DATAOUT_SEND_TO_TRANSPORT → target_execute_cmd() has the same problem. It is older (it dates from ea87981a0ee8), but once the claim helper exists it is a two-line change. Please include it too.

**2. The author's answer to Sashiko's second point relies on the gap above.**
- He says that if outstanding_r2ts is read low, "the later completion in __iscsit_check_dataout_hdr() becomes a no-op". In v2 that path does not check the claim, so the statement is false as posted.
- The race itself is unlikely: both accesses happen on the same connection's RX thread. The only writers on other threads are the queue-full retry (qf_work) and the ERL2 task-reassign code.
- Once issue 1 is fixed, the reasoning holds. Until then, the reply should not be accepted as a justification.

**3. Consider a fix in target core instead.**
- The real cause is that target_cmd_interrupted() queues target_abort_work but records nothing under t_state_lock.
- A test-and-set flag in transport_state there (with a WARN on a second call) would cover every site in every fabric, including iSER and cxgbit, which also reach iscsit_check_dataout_payload().
- That would be better than a private iSCSI flag that each call site has to remember to check. At the very least, the author should say why he chose not to do this.

## Smaller points

- **Missing Fixes: and Cc: stable tags** on patch 1. Candidates are ea87981a0ee8 ("scsi: target: iscsi: Handle abort for WRITE_PENDING cmds") and aaa00cc93c1d, which added the bare return 0 in iscsit_execute_cmd(). Note that aaa00cc93c1d is your own 2018 TAS change.
- **Commit message for patch 2:** "Unlike patch 1 of this series…" and the paragraph about what was seen in the field should not end up in git history. Move them below the --- line or into the cover letter.
- **Comment in iscsi_target_core.h:** it says completion_claimed is set "once the fabric has called target_complete_cmd()". It is actually set before the call. Reword it to "has claimed the completion".
- **Patch 2 is untested by the author's own account.** It looks correct to me, including the iscsit_stop_dataout_timer() call, which matters on this path. Ask for at least a fault-injection test, or a test with MaxOutstandingR2T > 1, before taking it.
- **Testing was on a 5.14-based distro backport.** Ask for a run on upstream (7.3-rc plus scsi-staging) as well.
- **Nit:** the unlocked transport_state & CMD_T_ABORTED reads in iscsit_execute_cmd() should use READ_ONCE(). This predates the series.

## Suggested verdict

Ask for a v3 that:
- sends every completion of an aborted WRITE_PENDING command through iscsit_complete_aborted_cmd(), including the d5869dae5080 site and DATAOUT_SEND_TO_TRANSPORT, or else moves the one-shot guard into target core;
- adds the Fixes:/stable tags;
- cleans up the commit message for patch 2.

Patch 1's core change (completing the command instead of returning when no R2T has been sent yet) is fine on its own merits.

#7 lore raw
Quoted 6 lines · Bart Van Assche
Quoted 1 line · Laurence Oberman

Gentle ping:

Please take a look at the attached review report. The report has been generated by an LLM. As known LLMs may make mistakes.

Bart.

Hello Bart,
Thank you very much for using your LLM review this. I am following up with a v3.

The report was right about the one thing: the claim in v2 did not cover every site that completes an aborted WRITE_PENDING command, and my answer to the Sashiko review wrongly assumed iscsit_check_dataout_hdr() checked it. The upcoming v3 adds patch 3 for that site and patch 4 for the DATAOUT_SEND_TO_TRANSPORT path, and the cover letter corrects the Sashiko answer. The comment reword and the patch 2 commit-message cleanup are in there too.

Two things in the report are not correct, in case the same tool is used again:

d5869dae5080 is not confined to 7.3/scsi-staging:

  $ git tag --contains d5869dae5080
  mkp-scsi-fixes
  mkp-scsi-staging
  v7.3
  v7.3-rc2
  ... v7.3-rc7

It went in via mkp-scsi-fixes during the 7.3 cycle, it is in released v7.3, it carries CVE-2026-97951, and it is in 5.10.y, 5.15.y, 6.1.y, 6.6.y, 6.12.y, 6.18.y and 7.2.y. That does not change the conclusion that the gap belongs in this series, but it is why patch 3 has Cc:
stable, and why it may be better taken standalone for those branches, none of which carry patch 1's helper.

It seems aaa00cc93c1d is yours from git history.

  $ git show --no-patch --format='%an <%ae> %ad' aaa00cc93c1d
  Bart Van Assche <[email protected]> Tue Nov 27 15:52:02 2018 -0800

It is the right Fixes: for patch 1, though, and patch 1 now carries it. git log -L on that hunk shows your commit replaced a transport_check_aborted_status() call -- which sent the delayed TASK_ABORTED status -- with a bare CMD_T_ABORTED test and a return. That works where the core can finish the command itself, but a WRITE still in TRANSPORT_WRITE_PENDING has never reached the backend, so the core's abort path is itself waiting in target_put_cmd_and_wait() for the fabric. The other three commits touching that hunk since 2011 are cosmetic.

Thank you
Laurence