Skip to contentSkip to conversation

Revert "scsi: target/iscsi: Detect conn_cmd_list corruption early"

PATCH In review Maurizio Lombardi 3 participants 7 messages Oct 2, 2020 – Oct 3, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Revert "scsi: target/iscsi: Detect conn_cmd_list corruption early"

This reverts commit b0055acaedf56a2717a6e2a4b700f1959a1b60df.

This warning is duplicated because the very same condition is already checked in __iscsit_free_cmd().

---
 drivers/target/iscsi/iscsi_target_util.c | 2 --
 1 file changed, 2 deletions(-)
drivers/target/iscsi/iscsi_target_util.c +0 -2
index 45ba07c6ec27..ff7830ddbd7b 100644
@@ -764,8 +764,6 @@ void iscsit_free_cmd(struct iscsi_cmd *cmd, bool shutdown)
 	struct se_cmd *se_cmd = cmd->se_cmd.se_tfo ? &cmd->se_cmd : NULL;
 	int rc;
 
-	WARN_ON(!list_empty(&cmd->i_conn_node));
-
 	__iscsit_free_cmd(cmd, shutdown);
 	if (se_cmd) {
 		rc = transport_generic_free_cmd(se_cmd, shutdown);
Signature · 1 line
2.26.2
Reviewed-by on v1 1/1
#1 lore raw
Quoted 16 lines · Maurizio Lombardi

This reverts commit b0055acaedf56a2717a6e2a4b700f1959a1b60df.

This warning is duplicated because the very same condition is already checked in __iscsit_free_cmd().

---
 drivers/target/iscsi/iscsi_target_util.c | 2 --
 1 file changed, 2 deletions(-)
drivers/target/iscsi/iscsi_target_util.c +0 -0
index 45ba07c6ec27..ff7830ddbd7b 100644

drivers/target/iscsi/iscsi_target_util.c void iscsit_free_cmd(struct iscsi_cmd *cmd, bool shutdown) · line 769

Quoted 8 lines
drivers/target/iscsi/iscsi_target_util.c +0 -2
@@ -764,8 +764,6 @@ void iscsit_free_cmd(struct iscsi_cmd *cmd, bool shutdown)
 	struct se_cmd *se_cmd = cmd->se_cmd.se_tfo ? &cmd->se_cmd : NULL;
 	int rc;
 
-	WARN_ON(!list_empty(&cmd->i_conn_node));
-
 	__iscsit_free_cmd(cmd, shutdown);
 	if (se_cmd) {
 		rc = transport_generic_free_cmd(se_cmd, shutdown);

Maurizio, are you hitting these WARN()s?

Patch looks ok.

#2 lore raw

Dne 02. 10. 20 v 20:06 Mike Christie napsal(a):

Quoted 19 lines · Maurizio Lombardi
Quoted 18 lines

This reverts commit b0055acaedf56a2717a6e2a4b700f1959a1b60df.

This warning is duplicated because the very same condition is already checked in __iscsit_free_cmd().

---
 drivers/target/iscsi/iscsi_target_util.c | 2 --
 1 file changed, 2 deletions(-)
drivers/target/iscsi/iscsi_target_util.c +0 -0
index 45ba07c6ec27..ff7830ddbd7b 100644
@@ -764,8 +764,6 @@ void iscsit_free_cmd(struct iscsi_cmd *cmd, bool shutdown)
 	struct se_cmd *se_cmd = cmd->se_cmd.se_tfo ? &cmd->se_cmd : NULL;
 	int rc;
 
Quoted 8 lines

drivers/target/iscsi/iscsi_target_util.c void iscsit_free_cmd(struct iscsi_cmd *cmd, bool shutdown) · line 769

Quoted 6 lines
drivers/target/iscsi/iscsi_target_util.c +0 -2
-	WARN_ON(!list_empty(&cmd->i_conn_node));
-
 	__iscsit_free_cmd(cmd, shutdown);
 	if (se_cmd) {
 		rc = transport_generic_free_cmd(se_cmd, shutdown);

Maurizio, are you hitting these WARN()s?

We received a number of bug reports against RHEL7 about warnings like the following:

Feb 18 12:41:01 hostname kernel: ------------[ cut here ]------------
Feb 18 12:41:01 hostname kernel: WARNING: CPU: 1 PID: 21173 at drivers/target/iscsi/iscsi_target_util.c:720 __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
...
Feb 18 12:41:01 hostname kernel: CPU: 1 PID: 21173 Comm: kworker/u8:3 Kdump: loaded Not tainted 3.10.0-1062.4.1.el7.x86_64 #1
Feb 18 12:41:01 hostname kernel: Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
Feb 18 12:41:01 hostname kernel: Workqueue: tmr-user target_tmr_work [target_core_mod]
Feb 18 12:41:01 hostname kernel: Call Trace:
Feb 18 12:41:01 hostname kernel: [<ffffffff91d78ba4>] dump_stack+0x19/0x1b
Feb 18 12:41:01 hostname kernel: [<ffffffff9169a958>] __warn+0xd8/0x100
Feb 18 12:41:01 hostname kernel: [<ffffffff9169aa9d>] warn_slowpath_null+0x1d/0x20
Feb 18 12:41:01 hostname kernel: [<ffffffffc0a6f69e>] __iscsit_free_cmd+0x26e/0x290 [iscsi_target_mod]
Feb 18 12:41:01 hostname kernel: [<ffffffffc0a70bc4>] iscsit_aborted_task+0x64/0x70 [iscsi_target_mod]
Feb 18 12:41:01 hostname kernel: [<ffffffffc0a7830a>] lio_aborted_task+0x2a/0x30 [iscsi_target_mod]
Feb 18 12:41:01 hostname kernel: [<ffffffffc09fa516>] transport_cmd_finish_abort+0x66/0xb0 [target_core_mod]
Feb 18 12:41:01 hostname kernel: [<ffffffffc09f4d92>] core_tmr_abort_task+0x102/0x180 [target_core_mod]
Feb 18 12:41:01 hostname kernel: [<ffffffffc09f7bb2>] target_tmr_work+0x152/0x170 [target_core_mod]
Feb 18 12:41:01 hostname kernel: [<ffffffff916bd1df>] process_one_work+0x17f/0x440
Feb 18 12:41:01 hostname kernel: [<ffffffff916be2f6>] worker_thread+0x126/0x3c0
Feb 18 12:41:01 hostname kernel: [<ffffffff916be1d0>] ? manage_workers.isra.26+0x2a0/0x2a0
Feb 18 12:41:01 hostname kernel: [<ffffffff916c51b1>] kthread+0xd1/0xe0
Feb 18 12:41:01 hostname kernel: [<ffffffff916c50e0>] ? insert_kthread_work+0x40/0x40
Feb 18 12:41:01 hostname kernel: [<ffffffff91d8bd37>] ret_from_fork_nospec_begin+0x21/0x21
Feb 18 12:41:01 hostname kernel: [<ffffffff916c50e0>] ? insert_kthread_work+0x40/0x40
Feb 18 12:41:01 hostname kernel: ---[ end trace ed2119501826ec7a ]---

I was discussing the matter with Bart Van Assche in private and maybe I found the bug, this is the content of the last email I sent him:

The iscsit_release_commands_from_conn() function does the following operations:

1) locks the cmd_lock spinlock
2) Scans the list of commands and sets the CMD_T_FABRIC_STOP flag
3) Releases the cmd_lock spinlock
4) Rescans the list again and clears the i_conn_node link of each command

But what happens if an abort timer is fired between points 3 and 4?

void iscsit_aborted_task(struct iscsi_conn *conn, struct iscsi_cmd *cmd)

{
        spin_lock_bh(&conn->cmd_lock);
        if (!list_empty(&cmd->i_conn_node) &&
            !(cmd->se_cmd.transport_state & CMD_T_FABRIC_STOP))
                list_del_init(&cmd->i_conn_node);
        spin_unlock_bh(&conn->cmd_lock);

        __iscsit_free_cmd(cmd, true);
}

iscsit_aborted_task() will find the cmd_lock spinlock unlocked.
will also find a non-empty i_conn_node link but with the CMD_T_FABRIC_STOP flag set. Therefore it will not call list_del_init(i_conn_node) and will trigger the warning in __iscsit_free_cmd().

Sounds it possible to you?
If I am right, this has been introduced by commit 064cdd2d91c2805d788876082f31cc63506f22c3

Maurizio

#3 lore raw
In reply to Maurizio Lombardi
Quoted 15 lines · Maurizio Lombardi

This reverts commit b0055acaedf56a2717a6e2a4b700f1959a1b60df.

This warning is duplicated because the very same condition is already checked in __iscsit_free_cmd().

---
  drivers/target/iscsi/iscsi_target_util.c | 2 --
  1 file changed, 2 deletions(-)
drivers/target/iscsi/iscsi_target_util.c +0 -0
index 45ba07c6ec27..ff7830ddbd7b 100644

drivers/target/iscsi/iscsi_target_util.c void iscsit_free_cmd(struct iscsi_cmd *cmd, bool shutdown) · line 769

Quoted 8 lines
drivers/target/iscsi/iscsi_target_util.c +0 -2
@@ -764,8 +764,6 @@ void iscsit_free_cmd(struct iscsi_cmd *cmd, bool shutdown)
  	struct se_cmd *se_cmd = cmd->se_cmd.se_tfo ? &cmd->se_cmd : NULL;
  	int rc;
 
-	WARN_ON(!list_empty(&cmd->i_conn_node));
-
  	__iscsit_free_cmd(cmd, shutdown);
  	if (se_cmd) {
  		rc = transport_generic_free_cmd(se_cmd, shutdown);

Hi Maurizio,

I agree that the same WARN_ON() occurs inside __iscsit_free_cmd(). What is not clear to me is how removing the WARN_ON() statement from iscsit_free_cmd() can help since an identical statement occurs inside __iscsit_free_cmd()?

Thanks,

Bart.

#4 lore raw

Dne 03. 10. 20 v 2:23 Bart Van Assche napsal(a):

Quoted 5 lines

Hi Maurizio,

I agree that the same WARN_ON() occurs inside __iscsit_free_cmd(). What is not clear to me is how removing the WARN_ON() statement from iscsit_free_cmd() can help since an identical statement occurs inside __iscsit_free_cmd()?

It doesn't help indeed, this patch just removes one duplicate warning but doesn't really change anything.

The bug I am trying to fix will need a different patch to prevent the race condition.

Maurizio

#5 lore raw
Quoted 9 lines · Maurizio Lombardi

Dne 03. 10. 20 v 2:23 Bart Van Assche napsal(a):

Quoted 3 lines

I agree that the same WARN_ON() occurs inside __iscsit_free_cmd(). What is not clear to me is how removing the WARN_ON() statement from iscsit_free_cmd() can help since an identical statement occurs inside __iscsit_free_cmd()?

It doesn't help indeed, this patch just removes one duplicate warning but doesn't really change anything.

The bug I am trying to fix will need a different patch to prevent the race condition.

How about addressing both issues with a single patch?

Thanks,

Bart.

#6 lore raw

Dne 03. 10. 20 v 17:17 Bart Van Assche napsal(a):

Quoted 6 lines · Maurizio Lombardi
Quoted 5 lines

Dne 03. 10. 20 v 2:23 Bart Van Assche napsal(a):

Quoted 3 lines

I agree that the same WARN_ON() occurs inside __iscsit_free_cmd(). What is not clear to me is how removing the WARN_ON() statement from iscsit_free_cmd() can help since an identical statement occurs inside __iscsit_free_cmd()?

Quoted 6 lines
Quoted 4 lines

It doesn't help indeed, this patch just removes one duplicate warning but doesn't really change anything.

The bug I am trying to fix will need a different patch to prevent the race condition.

How about addressing both issues with a single patch?

I've nothing against it, I will just need a bit more time to come up with a patch.

Maurizio