Fix a deadlock in the ib_srp driver
Subsystems (2)
Bart Van Assche
maintainercover Fix a deadlock in the ib_srp driver
Hi Jason,
This patch series fixes a deadlock in the ib_srp driver that was discovered by syzbot. Please consider these patches for kernel v5.18.
Thanks,
Bart.
Bart Van Assche (3):
ib_srp: Add more documentation
ib_srp: Protect the target list with a mutex instead of a spinlock
ib_srp: Fix a deadlock
drivers/infiniband/ulp/srp/ib_srp.c | 24 +++++++++++------------- drivers/infiniband/ulp/srp/ib_srp.h | 13 +++++++++++-- 2 files changed, 22 insertions(+), 15 deletions(-)
Bart Van Assche
maintainerPATCH 1/3 ib_srp: Add more documentation
Make it more clear what the different ib_srp data structures represent.
--- drivers/infiniband/ulp/srp/ib_srp.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
drivers/infiniband/ulp/srp/ib_srp.h +10 -1
@@ -92,6 +92,9 @@ enum srp_iu_type { };/* +* RDMA adapter in the initiator system. +* +* @dev_list: List of RDMA ports associated with this RDMA adapter (srp_host). * @mr_page_mask: HCA memory registration page mask. * @mr_page_size: HCA memory registration page size. * @mr_max_size: Maximum size in bytes of a single FR registration request.
@@ -109,6 +112,12 @@ struct srp_device {bool use_fast_reg; }; +/* +* One port of an RDMA adapter in the initiator system. +* +* @target_list: List of connected target ports (struct srp_target_port). +* @target_lock: Protects @target_list. +*/ struct srp_host {struct srp_device *srp_dev;u8 port;
@@ -183,7 +192,7 @@ struct srp_rdma_ch { };/** -* struct srp_target_port +* struct srp_target_port - RDMA port in the SRP target system * @comp_vector: Completion vector used by the first RDMA channel created for * this target port. */
Re: [PATCH 1/3] ib_srp: Add more documentation
Quoted 13 lines · Bart Van Assche
Make it more clear what the different
ib_srpdata structures represent.Cc Tetsuo Handa <[email protected]>Signed-off-by Bart Van Assche <[email protected]>--- drivers/infiniband/ulp/srp/ib_srp.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)drivers/infiniband/ulp/srp/ib_srp.h +0
-0
drivers/infiniband/ulp/srp/ib_srp.h enum srp_iu_type { · line 97
Quoted 6 lines
drivers/infiniband/ulp/srp/ib_srp.h +3
-0@@ -92,6 +92,9 @@ enum srp_iu_type { };/* +* RDMA adapter in the initiator system. +* +* @dev_list: List of RDMA ports associated with this RDMA adapter (srp_host).
Isn't this list of RDMA devices and not ports?
Quoted 25 lines
drivers/infiniband/ulp/srp/ib_srp.h +7
-1* @mr_page_mask: HCA memory registration page mask. * @mr_page_size: HCA memory registration page size. * @mr_max_size: Maximum size in bytes of a single FR registration request. @@ -109,6 +112,12 @@ struct srp_device {bool use_fast_reg; }; +/* +* One port of an RDMA adapter in the initiator system. +* +* @target_list: List of connected target ports (struct srp_target_port). +* @target_lock: Protects @target_list. +*/ struct srp_host {struct srp_device *srp_dev;u8 port;@@ -183,7 +192,7 @@ struct srp_rdma_ch { };/** -* struct srp_target_port +* struct srp_target_port - RDMA port in the SRP target system * @comp_vector: Completion vector used by the first RDMA channel created for * this target port. */
Bart Van Assche
maintainerRe: [PATCH 1/3] ib_srp: Add more documentation
Quoted 7 lines · Leon Romanovsky
Quoted 4 lines · Bart Van Assche
/*
+ * RDMA adapter in the initiator system.
+ *
+ * @dev_list: List of RDMA ports associated with this RDMA adapter (srp_host).Isn't this list of RDMA devices and not ports?
Please take a look at srp_add_port(). I think that function establishes a 1:1 relationship between an RDMA port and struct srp_host.
Thanks,
Bart.
Leon Romanovsky
maintainerRe: [PATCH 1/3] ib_srp: Add more documentation
Quoted 11 lines · Bart Van Assche
Quoted 7 lines · Leon Romanovsky
Quoted 4 lines · Bart Van Assche
/*
+ * RDMA adapter in the initiator system.
+ *
+ * @dev_list: List of RDMA ports associated with this RDMA adapter (srp_host).Isn't this list of RDMA devices and not ports?
Please take a look at
srp_add_port(). I think that function establishes a 1:1 relationship between an RDMA port and structsrp_host.
Right, but this list contains devices and not ports.
Anyway, it is not important.
Thanks
Quoted 5 lines
Thanks,
Bart.
Bart Van Assche
maintainerPATCH 2/3 ib_srp: Protect the target list with a mutex instead of a spinlock
In reply to Bart Van AsscheThis patch makes the next patch in this series easier to read.
--- drivers/infiniband/ulp/srp/ib_srp.c | 18 +++++++++--------- drivers/infiniband/ulp/srp/ib_srp.h | 4 ++-- 2 files changed, 11 insertions(+), 11 deletions(-)
drivers/infiniband/ulp/srp/ib_srp.c +9 -9
@@ -1062,9 +1062,9 @@ static void srp_remove_target(struct srp_target_port *target)kfree (target->ch); target->ch =NULL ; -spin_lock (&target->srp_host->target_lock); +mutex_lock (&target->srp_host->target_mutex);list_del (&target->list); -spin_unlock (&target->srp_host->target_lock); +mutex_unlock (&target->srp_host->target_mutex);scsi_host_put (target->scsi_host); }
@@ -3146,9 +3146,9 @@ static int srp_add_target(struct srp_host *host, struct srp_target_port *target) rport->lld_data = target; target->rport = rport; -spin_lock (&host->target_lock); +mutex_lock (&host->target_mutex);list_add_tail (&target->list, &host->target_list); -spin_unlock (&host->target_lock); +mutex_unlock (&host->target_mutex);scsi_scan_target (&target->scsi_host->shost_gendev,0 , target->scsi_id,SCAN_WILD_CARD ,SCSI_SCAN_INITIAL );
@@ -3203,7 +3203,7 @@ static bool srp_conn_unique(struct srp_host *host, ret = true; -spin_lock (&host->target_lock); +mutex_lock (&host->target_mutex);list_for_each_entry (t, &host->target_list, list) { if (t != target && target->id_ext == t->id_ext &&
@@ -3213,7 +3213,7 @@ static bool srp_conn_unique(struct srp_host *host,break ; } } -spin_unlock (&host->target_lock); +mutex_unlock (&host->target_mutex); out:return ret;
@@ -3898,7 +3898,7 @@ static struct srp_host *srp_add_port(struct srp_device *device, u8 port)return NULL ;INIT_LIST_HEAD (&host->target_list); -spin_lock_init (&host->target_lock); +mutex_init (&host->target_mutex);init_completion (&host->released);mutex_init (&host->add_target_mutex); host->srp_dev = device;
@@ -4041,10 +4041,10 @@ static void srp_remove_one(struct ib_device *device, void *client_data)/* * Remove all target ports. */ -spin_lock (&host->target_lock); +mutex_lock (&host->target_mutex);list_for_each_entry (target, &host->target_list, list)srp_queue_remove_work (target); -spin_unlock (&host->target_lock); +mutex_unlock (&host->target_mutex);/* * Wait for tl_err and target port removal tasks.
drivers/infiniband/ulp/srp/ib_srp.h +2 -2
@@ -116,14 +116,14 @@ struct srp_device {* One port of an RDMA adapter in the initiator system. * * @target_list: List of connected target ports (struct srp_target_port). -* @target_lock: Protects @target_list. +* @target_mutex: Protects @target_list. */ struct srp_host {struct srp_device *srp_dev;u8 port;struct device dev;struct list_head target_list; -spinlock_t target_lock; +struct mutex target_mutex;struct completion released;struct list_head list;struct mutex add_target_mutex;
Re: [PATCH 2/3] ib_srp: Protect the target list with a mutex instead of a spinlock
Quoted 1 line · Bart Van Assche
This patch makes the next patch in this series easier to read.
It is not readability change, but move to lock that can sleep (mutex), which is always good thing.
Quoted 97 lines
Cc Tetsuo Handa <[email protected]>Signed-off-by Bart Van Assche <[email protected]>--- drivers/infiniband/ulp/srp/ib_srp.c | 18 +++++++++--------- drivers/infiniband/ulp/srp/ib_srp.h | 4 ++-- 2 files changed, 11 insertions(+), 11 deletions(-)drivers/infiniband/ulp/srp/ib_srp.c +9
-9@@ -1062,9 +1062,9 @@ static void srp_remove_target(struct srp_target_port *target)kfree (target->ch); target->ch =NULL ; -spin_lock (&target->srp_host->target_lock); +mutex_lock (&target->srp_host->target_mutex);list_del (&target->list); -spin_unlock (&target->srp_host->target_lock); +mutex_unlock (&target->srp_host->target_mutex);scsi_host_put (target->scsi_host); }@@ -3146,9 +3146,9 @@ static int srp_add_target(struct srp_host *host, struct srp_target_port *target) rport->lld_data = target; target->rport = rport; -spin_lock (&host->target_lock); +mutex_lock (&host->target_mutex);list_add_tail (&target->list, &host->target_list); -spin_unlock (&host->target_lock); +mutex_unlock (&host->target_mutex);scsi_scan_target (&target->scsi_host->shost_gendev,0 , target->scsi_id,SCAN_WILD_CARD ,SCSI_SCAN_INITIAL );@@ -3203,7 +3203,7 @@ static bool srp_conn_unique(struct srp_host *host, ret = true; -spin_lock (&host->target_lock); +mutex_lock (&host->target_mutex);list_for_each_entry (t, &host->target_list, list) { if (t != target && target->id_ext == t->id_ext &&@@ -3213,7 +3213,7 @@ static bool srp_conn_unique(struct srp_host *host,break ; } } -spin_unlock (&host->target_lock); +mutex_unlock (&host->target_mutex); out:return ret;@@ -3898,7 +3898,7 @@ static struct srp_host *srp_add_port(struct srp_device *device, u8 port)return NULL ;INIT_LIST_HEAD (&host->target_list); -spin_lock_init (&host->target_lock); +mutex_init (&host->target_mutex);init_completion (&host->released);mutex_init (&host->add_target_mutex); host->srp_dev = device;@@ -4041,10 +4041,10 @@ static void srp_remove_one(struct ib_device *device, void *client_data)/* * Remove all target ports. */ -spin_lock (&host->target_lock); +mutex_lock (&host->target_mutex);list_for_each_entry (target, &host->target_list, list)srp_queue_remove_work (target); -spin_unlock (&host->target_lock); +mutex_unlock (&host->target_mutex);/* * Wait for tl_err and target port removal tasks. drivers/infiniband/ulp/srp/ib_srp.h +2
-2@@ -116,14 +116,14 @@ struct srp_device {* One port of an RDMA adapter in the initiator system. * * @target_list: List of connected target ports (struct srp_target_port). -* @target_lock: Protects @target_list. +* @target_mutex: Protects @target_list. */ struct srp_host {struct srp_device *srp_dev;u8 port;struct device dev;struct list_head target_list; -spinlock_t target_lock; +struct mutex target_mutex;struct completion released;struct list_head list;struct mutex add_target_mutex;
Bart Van Assche
maintainerRe: [PATCH 2/3] ib_srp: Protect the target list with a mutex instead of a spinlock
Quoted 7 lines · Leon Romanovsky
Quoted 1 line · Bart Van Assche
This patch makes the next patch in this series easier to read.
It is not readability change, but move to lock that can sleep (mutex), which is always good thing.
Reviewed-by Leon Romanovsky <[email protected]>
Thanks for the review, but please note that I wrote that this patch improves readability of the next patch in this series.
Bart.
Bart Van Assche
maintainerPATCH 3/3 ib_srp: Fix a deadlock
In reply to Bart Van AsscheWait on tl_err_work instead of flushing system_long_wq since flushing system_long_wq is deadlock-prone.
--- drivers/infiniband/ulp/srp/ib_srp.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
drivers/infiniband/ulp/srp/ib_srp.c +2 -4
@@ -4044,12 +4044,10 @@ static void srp_remove_one(struct ib_device *device, void *client_data)mutex_lock (&host->target_mutex);list_for_each_entry (target, &host->target_list, list)srp_queue_remove_work (target); +list_for_each_entry (target, &host->target_list, list) +flush_work (&target->tl_err_work);mutex_unlock (&host->target_mutex); -/* -* Wait for tl_err and target port removal tasks. -*/ -flush_workqueue (system_long_wq);flush_workqueue (srp_remove_wq);kfree (host);
Re: [PATCH 3/3] ib_srp: Fix a deadlock
Quoted 16 lines · Bart Van Assche
Wait on
tl_err_workinstead of flushingsystem_long_wqsince flushingsystem_long_wqis deadlock-prone.Cc Tetsuo Handa <[email protected]>Fixes ef6c49d87c34 ("IB/srp: Eliminate state SRP_TARGET_DEAD")Reported-by [email protected]Signed-off-by Bart Van Assche <[email protected]>--- drivers/infiniband/ulp/srp/ib_srp.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)drivers/infiniband/ulp/srp/ib_srp.c +0
-0
drivers/infiniband/ulp/srp/ib_srp.c static void srp_remove_one(struct ib_device *device, void *client_data) · line 4048
Quoted 5 lines
drivers/infiniband/ulp/srp/ib_srp.c +2
-0@@ -4044,12 +4044,10 @@ static void srp_remove_one(struct ib_device *device, void *client_data)mutex_lock (&host->target_mutex);list_for_each_entry (target, &host->target_list, list)srp_queue_remove_work (target); +list_for_each_entry (target, &host->target_list, list) +flush_work (&target->tl_err_work);
Sorry for my silly question, but why do you do flush and not cancel here? You anyway remove SRP device, so the result of flush is not really important, am I right?
Thanks
Quoted 9 lines
drivers/infiniband/ulp/srp/ib_srp.c +0
-4mutex_unlock (&host->target_mutex); -/* -* Wait for tl_err and target port removal tasks. -*/ -flush_workqueue (system_long_wq);flush_workqueue (srp_remove_wq);kfree (host);
Bart Van Assche
maintainerRe: [PATCH 3/3] ib_srp: Fix a deadlock
Quoted 15 lines · Leon Romanovsky
drivers/infiniband/ulp/srp/ib_srp.c static void srp_remove_one(struct ib_device *device, void *client_data) · line 4048
Quoted 10 lines · Bart Van Assche
drivers/infiniband/ulp/srp/ib_srp.c +2
-0@@ -4044,12 +4044,10 @@ static void srp_remove_one(struct ib_device *device, void *client_data)mutex_lock (&host->target_mutex);list_for_each_entry (target, &host->target_list, list)srp_queue_remove_work (target); +list_for_each_entry (target, &host->target_list, list) +flush_work (&target->tl_err_work);Sorry for my silly question, but why do you do flush and not cancel here? You anyway remove SRP device, so the result of flush is not really important, am I right?
That's a great question. It probably doesn't matter much whether flush_work() or cancel_work_sync() is called in this context since srp_queue_remove_work() indirectly cancels tl_err_work. See also the following code in srp_remove_target():
cancel_work_sync(&target->tl_err_work);
Thanks,
Bart.
Re: [PATCH 3/3] ib_srp: Fix a deadlock
Quoted 16 lines · Bart Van Assche
Quoted 14 lines · Leon Romanovsky
drivers/infiniband/ulp/srp/ib_srp.c static void srp_remove_one(struct ib_device *device, void *client_data) · line 4048
Quoted 9 lines · Bart Van Assche
drivers/infiniband/ulp/srp/ib_srp.c +3
-0+++ b/drivers/infiniband/ulp/srp/ib_srp.c@@ -4044,12 +4044,10 @@ static void srp_remove_one(struct ib_device *device, void *client_data)mutex_lock (&host->target_mutex);list_for_each_entry (target, &host->target_list, list)srp_queue_remove_work (target); +list_for_each_entry (target, &host->target_list, list) +flush_work (&target->tl_err_work);Sorry for my silly question, but why do you do flush and not cancel here? You anyway remove SRP device, so the result of flush is not really important, am I right?
Quoted 5 lines
That's a great question. It probably doesn't matter much whether
flush_work()orcancel_work_sync()is called in this context sincesrp_queue_remove_work()indirectly cancelstl_err_work. See also the following code insrp_remove_target():
cancel_work_sync(&target->tl_err_work);
If it is already canceled then why call flush?
Jason
Bart Van Assche
maintainerRe: [PATCH 3/3] ib_srp: Fix a deadlock
Quoted 17 lines · Jason Gunthorpe
Quoted 16 lines · Bart Van Assche
Quoted 14 lines · Leon Romanovsky
drivers/infiniband/ulp/srp/ib_srp.c static void srp_remove_one(struct ib_device *device, void *client_data) · line 4048
Quoted 9 lines · Bart Van Assche
drivers/infiniband/ulp/srp/ib_srp.c +3
-0+++ b/drivers/infiniband/ulp/srp/ib_srp.c@@ -4044,12 +4044,10 @@ static void srp_remove_one(struct ib_device *device, void *client_data)mutex_lock (&host->target_mutex);list_for_each_entry (target, &host->target_list, list)srp_queue_remove_work (target); +list_for_each_entry (target, &host->target_list, list) +flush_work (&target->tl_err_work);Sorry for my silly question, but why do you do flush and not cancel here? You anyway remove SRP device, so the result of flush is not really important, am I right?
Quoted 7 lines
Quoted 5 lines
That's a great question. It probably doesn't matter much whether
flush_work()orcancel_work_sync()is called in this context sincesrp_queue_remove_work()indirectly cancelstl_err_work. See also the following code insrp_remove_target():
cancel_work_sync(&target->tl_err_work);If it is already canceled then why call flush?
I'll see whether I can leave the flush_work(&target->tl_err_work) calls out.
Thanks,
Bart.