scsi: libiscsi: fix NOP race condition
[email protected]
maintainerv2 cover scsi: libiscsi: fix NOP race condition
From: Lee Duncan <[email protected]>
A customer that uses iSCSI NOPs extensively found a race condition caused in part by the two-lock system used in iscsi (a forward and a back lock), since sending an iSCSI NOP uses the forward lock, and receiving one uses the back lock. Because of this, processing of the "send" can still be in progress when the "receive" occurs, on a sufficiently fast multicore system.
To handle this case, we add a new state to the "ping_task" pointer besides unassigned and assigned, called "invalid", which means the "not yet completed sending". Tests show this closes this race condition hole.
Changes since V1:
- Removed two redundant lines in iscsi_send_nopout()
- Updated commit text to be more clear
- Added this cover letter with even more info
Lee Duncan (1):
scsi: libiscsi: fix NOP race condition
drivers/scsi/libiscsi.c | 13 ++++++++++--- include/scsi/libiscsi.h | 3 +++ 2 files changed, 13 insertions(+), 3 deletions(-)
Signature · 1 line
2.26.2
[email protected]
maintainerPATCH v2 1/1 scsi: libiscsi: fix NOP race condition
From: Lee Duncan <[email protected]>
iSCSI NOPs are sometimes "lost", mistakenly sent to the user-land iscsid daemon instead of handled in the kernel, as they should be, resulting in a message from the daemon like:
Quoted 1 line
iscsid: Got nop in, but kernel supports nop handling.
This can occur because of the forward- and back-locks in the kernel iSCSI code, and the fact that an iSCSI NOP response can be processed before processing of the NOP send is complete. This can result in "conn->ping_task" being NULL in iscsi_nop_out_rsp(), when the pointer is actually in the process of being set.
To work around this, we add a new state to the "ping_task" pointer. In addition to NULL (not assigned) and a pointer (assigned), we add the state "being set", which is signaled with an INVALID pointer (using "-1").
--- drivers/scsi/libiscsi.c | 13 ++++++++++--- include/scsi/libiscsi.h | 3 +++ 2 files changed, 13 insertions(+), 3 deletions(-)
drivers/scsi/libiscsi.c +10 -3
@@ -738,6 +738,9 @@ __iscsi_conn_send_pdu(struct iscsi_conn *conn, struct iscsi_hdr *hdr, task->conn->session->age); } +if (unlikely (READ_ONCE (conn->ping_task) ==INVALID_SCSI_TASK )) +WRITE_ONCE (conn->ping_task, task); +if (!ihost->workq) {if (iscsi_prep_mgmt_task (conn, task)) goto free_task;
@@ -941,8 +944,11 @@ static int iscsi_send_nopout(struct iscsi_conn *conn, struct iscsi_nopin *rhdr)struct iscsi_nopout hdr;struct iscsi_task *task; -if (!rhdr && conn->ping_task) -return -EINVAL ; +if (!rhdr) { +if (READ_ONCE (conn->ping_task)) +return -EINVAL ; +WRITE_ONCE (conn->ping_task,INVALID_SCSI_TASK ); + }memset (&hdr,0 ,sizeof (struct iscsi_nopout )); hdr.opcode =ISCSI_OP_NOOP_OUT |ISCSI_OP_IMMEDIATE ;
@@ -957,11 +963,12 @@ static int iscsi_send_nopout(struct iscsi_conn *conn, struct iscsi_nopin *rhdr) task =__iscsi_conn_send_pdu (conn, (struct iscsi_hdr *)&hdr,NULL ,0 );if (!task) { +if (!rhdr) +WRITE_ONCE (conn->ping_task,NULL );iscsi_conn_printk (KERN_ERR , conn,"Could not send nopout\n" );return -EIO ; }else if (!rhdr) {/* only track our nops */ - conn->ping_task = task; conn->last_ping = jiffies; }
include/scsi/libiscsi.h +3 -0
@@ -132,6 +132,9 @@ struct iscsi_task {void *dd_data;/* driver/transport data */ }; +/* invalid scsi_task pointer */ +#define INVALID_SCSI_TASK (struct iscsi_task *)-1l +static inline int iscsi_task_has_unsol_data (struct iscsi_task *task) {return task->unsol_r2t.data_length > task->unsol_r2t.sent;
Signature · 1 line
2.26.2
Lee Duncan
maintainerQuoted 74 lines · On 9/25/20 11:41 AM, [email protected] wrote:
From: Lee Duncan <[email protected]>
iSCSI NOPs are sometimes "lost", mistakenly sent to the user-land iscsid daemon instead of handled in the kernel, as they should be, resulting in a message from the daemon like:
Quoted 1 line
iscsid: Got nop in, but kernel supports nop handling.
This can occur because of the forward- and back-locks in the kernel iSCSI code, and the fact that an iSCSI NOP response can be processed before processing of the NOP send is complete. This can result in "
conn->ping_task" being NULL iniscsi_nop_out_rsp(), when the pointer is actually in the process of being set.To work around this, we add a new state to the "
ping_task" pointer. In addition to NULL (not assigned) and a pointer (assigned), we add the state "being set", which is signaled with an INVALID pointer (using "-1").Signed-off-by Lee Duncan <[email protected]>--- drivers/scsi/libiscsi.c | 13 ++++++++++--- include/scsi/libiscsi.h | 3 +++ 2 files changed, 13 insertions(+), 3 deletions(-)drivers/scsi/libiscsi.c +10
-3@@ -738,6 +738,9 @@ __iscsi_conn_send_pdu(struct iscsi_conn *conn, struct iscsi_hdr *hdr, task->conn->session->age); } +if (unlikely (READ_ONCE (conn->ping_task) ==INVALID_SCSI_TASK )) +WRITE_ONCE (conn->ping_task, task); +if (!ihost->workq) {if (iscsi_prep_mgmt_task (conn, task)) goto free_task;@@ -941,8 +944,11 @@ static int iscsi_send_nopout(struct iscsi_conn *conn, struct iscsi_nopin *rhdr)struct iscsi_nopout hdr;struct iscsi_task *task; -if (!rhdr && conn->ping_task) -return -EINVAL ; +if (!rhdr) { +if (READ_ONCE (conn->ping_task)) +return -EINVAL ; +WRITE_ONCE (conn->ping_task,INVALID_SCSI_TASK ); + }memset (&hdr,0 ,sizeof (struct iscsi_nopout )); hdr.opcode =ISCSI_OP_NOOP_OUT |ISCSI_OP_IMMEDIATE ;@@ -957,11 +963,12 @@ static int iscsi_send_nopout(struct iscsi_conn *conn, struct iscsi_nopin *rhdr) task =__iscsi_conn_send_pdu (conn, (struct iscsi_hdr *)&hdr,NULL ,0 );if (!task) { +if (!rhdr) +WRITE_ONCE (conn->ping_task,NULL );iscsi_conn_printk (KERN_ERR , conn,"Could not send nopout\n" );return -EIO ; }else if (!rhdr) {/* only track our nops */ - conn->ping_task = task; conn->last_ping = jiffies; }include/scsi/libiscsi.h +0
-0
include/scsi/libiscsi.h struct iscsi_task { · line 140
Quoted 9 lines
include/scsi/libiscsi.h +3
-0@@ -132,6 +132,9 @@ struct iscsi_task {void *dd_data;/* driver/transport data */ }; +/* invalid scsi_task pointer */ +#define INVALID_SCSI_TASK (struct iscsi_task *)-1l +static inline int iscsi_task_has_unsol_data (struct iscsi_task *task) {return task->unsol_r2t.data_length > task->unsol_r2t.sent;
Ping?
Signature · 1 line
Lee Duncan
Mike Christie
maintainerQuoted 31 lines · On 9/25/20 1:41 PM, [email protected] wrote:
From: Lee Duncan <[email protected]>
iSCSI NOPs are sometimes "lost", mistakenly sent to the user-land iscsid daemon instead of handled in the kernel, as they should be, resulting in a message from the daemon like:
Quoted 1 line
iscsid: Got nop in, but kernel supports nop handling.
This can occur because of the forward- and back-locks in the kernel iSCSI code, and the fact that an iSCSI NOP response can be processed before processing of the NOP send is complete. This can result in "
conn->ping_task" being NULL iniscsi_nop_out_rsp(), when the pointer is actually in the process of being set.To work around this, we add a new state to the "
ping_task" pointer. In addition to NULL (not assigned) and a pointer (assigned), we add the state "being set", which is signaled with an INVALID pointer (using "-1").Signed-off-by Lee Duncan <[email protected]>--- drivers/scsi/libiscsi.c | 13 ++++++++++--- include/scsi/libiscsi.h | 3 +++ 2 files changed, 13 insertions(+), 3 deletions(-)drivers/scsi/libiscsi.c +0
-0
drivers/scsi/libiscsi.c __iscsi_conn_send_pdu(struct iscsi_conn *conn, struct iscsi_hdr *hdr, · line 746
Quoted 9 lines
drivers/scsi/libiscsi.c +3
-0@@ -738,6 +738,9 @@ __iscsi_conn_send_pdu(struct iscsi_conn *conn, struct iscsi_hdr *hdr, task->conn->session->age); } +if (unlikely (READ_ONCE (conn->ping_task) ==INVALID_SCSI_TASK )) +WRITE_ONCE (conn->ping_task, task); +if (!ihost->workq) {if (iscsi_prep_mgmt_task (conn, task)) goto free_task;
I think the API gets a little weird now where in some cases __iscsi_conn_send_pdu checks the opcode to see what type of request it is but above we the caller sets the ping_task.
For login, tmfs and passthrough, we assume the __iscsi_conn_send_pdu has sent or cleaned up everything. I think it might be nicer to just have __iscsi_conn_send_pdu set the ping_task field before doing the xmit/queue call. It would then work similar to the conn->login_task case where that function knows about that special task too.
So in __iscsi_conn_send_pdu add a "if (opcode == ISCSI_OP_NOOP_OUT)", and check if it's a nop we need to track. If so set conn->ping_task.
Mike Christie
maintainerQuoted 46 lines · Mike Christie
Quoted 31 lines · On 9/25/20 1:41 PM, [email protected] wrote:
From: Lee Duncan <[email protected]>
iSCSI NOPs are sometimes "lost", mistakenly sent to the user-land iscsid daemon instead of handled in the kernel, as they should be, resulting in a message from the daemon like:
Quoted 1 line
iscsid: Got nop in, but kernel supports nop handling.
This can occur because of the forward- and back-locks in the kernel iSCSI code, and the fact that an iSCSI NOP response can be processed before processing of the NOP send is complete. This can result in "
conn->ping_task" being NULL iniscsi_nop_out_rsp(), when the pointer is actually in the process of being set.To work around this, we add a new state to the "
ping_task" pointer. In addition to NULL (not assigned) and a pointer (assigned), we add the state "being set", which is signaled with an INVALID pointer (using "-1").Signed-off-by Lee Duncan <[email protected]>--- drivers/scsi/libiscsi.c | 13 ++++++++++--- include/scsi/libiscsi.h | 3 +++ 2 files changed, 13 insertions(+), 3 deletions(-)drivers/scsi/libiscsi.c +0
-0drivers/scsi/libiscsi.c __iscsi_conn_send_pdu(struct iscsi_conn *conn, struct iscsi_hdr *hdr, · line 746
Quoted 9 lines
drivers/scsi/libiscsi.c +3
-0@@ -738,6 +738,9 @@ __iscsi_conn_send_pdu(struct iscsi_conn *conn, struct iscsi_hdr *hdr, task->conn->session->age); } +if (unlikely (READ_ONCE (conn->ping_task) ==INVALID_SCSI_TASK )) +WRITE_ONCE (conn->ping_task, task); +if (!ihost->workq) {if (iscsi_prep_mgmt_task (conn, task)) goto free_task;I think the API gets a little weird now where in some cases
__iscsi_conn_send_pduchecks the opcode to see what type of request it is but above we the caller sets theping_task.
Quoted 9 lines
For login, tmfs and passthrough, we assume the
__iscsi_conn_send_pduhas sent or cleaned up everything. I think it might be nicer to just have__iscsi_conn_send_pduset theping_taskfield before doing the xmit/queue call. It would then work similar to theconn->login_taskcase where that function knows about that special task too.So in
__iscsi_conn_send_pduadd a "if (opcode ==ISCSI_OP_NOOP_OUT)", and check if it's a nop we need to track. If so setconn->ping_task.
Ignore this. It won't work nicely either. To figure out if the nop is our internal transport test ping vs a userspace ping that also needs a reply, we would need to do something like you did above so there is no point.
Lee Duncan
maintainerQuoted 53 lines · Mike Christie
Quoted 52 lines · Mike Christie
Quoted 31 lines · On 9/25/20 1:41 PM, [email protected] wrote:
From: Lee Duncan <[email protected]>
iSCSI NOPs are sometimes "lost", mistakenly sent to the user-land iscsid daemon instead of handled in the kernel, as they should be, resulting in a message from the daemon like:
Quoted 1 line
iscsid: Got nop in, but kernel supports nop handling.
This can occur because of the forward- and back-locks in the kernel iSCSI code, and the fact that an iSCSI NOP response can be processed before processing of the NOP send is complete. This can result in "
conn->ping_task" being NULL iniscsi_nop_out_rsp(), when the pointer is actually in the process of being set.To work around this, we add a new state to the "
ping_task" pointer. In addition to NULL (not assigned) and a pointer (assigned), we add the state "being set", which is signaled with an INVALID pointer (using "-1").Signed-off-by Lee Duncan <[email protected]>--- drivers/scsi/libiscsi.c | 13 ++++++++++--- include/scsi/libiscsi.h | 3 +++ 2 files changed, 13 insertions(+), 3 deletions(-)drivers/scsi/libiscsi.c +0
-0drivers/scsi/libiscsi.c __iscsi_conn_send_pdu(struct iscsi_conn *conn, struct iscsi_hdr *hdr, · line 746
Quoted 9 lines
drivers/scsi/libiscsi.c +3
-0@@ -738,6 +738,9 @@ __iscsi_conn_send_pdu(struct iscsi_conn *conn, struct iscsi_hdr *hdr, task->conn->session->age); } +if (unlikely (READ_ONCE (conn->ping_task) ==INVALID_SCSI_TASK )) +WRITE_ONCE (conn->ping_task, task); +if (!ihost->workq) {if (iscsi_prep_mgmt_task (conn, task)) goto free_task;I think the API gets a little weird now where in some cases
__iscsi_conn_send_pduchecks the opcode to see what type of request it is but above we the caller sets theping_task.For login, tmfs and passthrough, we assume the
__iscsi_conn_send_pduhas sent or cleaned up everything. I think it might be nicer to just have__iscsi_conn_send_pduset theping_taskfield before doing the xmit/queue call. It would then work similar to theconn->login_taskcase where that function knows about that special task too.
Quoted 8 lines
Quoted 3 lines
So in
__iscsi_conn_send_pduadd a "if (opcode ==ISCSI_OP_NOOP_OUT)", and check if it's a nop we need to track. If so setconn->ping_task.Ignore this. It won't work nicely either. To figure out if the nop is our internal transport test ping vs a userspace ping that also needs a reply, we would need to do something like you did above so there is no point.
Hi Mike:
I've read this a few times, and I'm still no sure I'm parsing it correctly.
Are you saying that my original patch submission is ok, or are you saying there's nothing we can do and we're up the proverbial creek?
Signature · 1 line
Lee Duncan