Skip to contentSkip to conversation

scsi_dh_alua: avoid crash during alua_bus_detach()

PATCH Applied Hannes Reinecke 5 participants 5 messages Sep 24, 2020 – Nov 3, 2020 1 message not in the archive SCSI

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 scsi_dh_alua: avoid crash during alua_bus_detach()

alua_bus_detach() might be running concurrently with alua_rtpg_work(), so we might trip over h->sdev == NULL and call BUG_ON(). The correct way of handling it would be to not set h->sdev to NULL in alua_bus_detach(), and call rcu_synchronize() before the final delete to ensure that all concurrent threads have left the critical section.
Then we can get rid of the BUG_ON(), and replace it with a simple if condition.

---
 drivers/scsi/device_handler/scsi_dh_alua.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +5 -4
index f32da0ca529e..308bda2e9c00 100644
@@ -658,8 +658,8 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 					rcu_read_lock();
 					list_for_each_entry_rcu(h,
 						&tmp_pg->dh_list, node) {
-						/* h->sdev should always be valid */
-						BUG_ON(!h->sdev);
+						if (!h->sdev)
+							continue;
 						h->sdev->access_state = desc[0];
 					}
 					rcu_read_unlock();
@@ -705,7 +705,8 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 			pg->expiry = 0;
 			rcu_read_lock();
 			list_for_each_entry_rcu(h, &pg->dh_list, node) {
-				BUG_ON(!h->sdev);
+				if (!h->sdev)
+					continue;
 				h->sdev->access_state =
 					(pg->state & SCSI_ACCESS_STATE_MASK);
 				if (pg->pref)
@@ -1147,7 +1148,6 @@ static void alua_bus_detach(struct scsi_device *sdev)
 	spin_lock(&h->pg_lock);
 	pg = rcu_dereference_protected(h->pg, lockdep_is_held(&h->pg_lock));
 	rcu_assign_pointer(h->pg, NULL);
-	h->sdev = NULL;
 	spin_unlock(&h->pg_lock);
 	if (pg) {
 		spin_lock_irq(&pg->lock);
@@ -1156,6 +1156,7 @@ static void alua_bus_detach(struct scsi_device *sdev)
 		kref_put(&pg->kref, release_port_group);
 	}
 	sdev->handler_data = NULL;
+	synchronize_rcu();
 	kfree(h);
 }
 
Signature · 1 line
2.16.4

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
In reply to Hannes Reinecke
Quoted 49 lines · Hannes Reinecke

alua_bus_detach() might be running concurrently with alua_rtpg_work(), so we might trip over h->sdev == NULL and call BUG_ON(). The correct way of handling it would be to not set h->sdev to NULL in alua_bus_detach(), and call rcu_synchronize() before the final delete to ensure that all concurrent threads have left the critical section.
Then we can get rid of the BUG_ON(), and replace it with a simple if condition.

---
 drivers/scsi/device_handler/scsi_dh_alua.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +4 -4
index f32da0ca529e..308bda2e9c00 100644
@@ -658,8 +658,8 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 					rcu_read_lock();
 					list_for_each_entry_rcu(h,
 						&tmp_pg->dh_list, node) {
-						/* h->sdev should always be valid */
-						BUG_ON(!h->sdev);
+						if (!h->sdev)
+							continue;
 						h->sdev->access_state = desc[0];
 					}
 					rcu_read_unlock();
@@ -705,7 +705,8 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 			pg->expiry = 0;
 			rcu_read_lock();
 			list_for_each_entry_rcu(h, &pg->dh_list, node) {
-				BUG_ON(!h->sdev);
+				if (!h->sdev)
+					continue;
 				h->sdev->access_state =
 					(pg->state & SCSI_ACCESS_STATE_MASK);
 				if (pg->pref)
@@ -1147,7 +1148,6 @@ static void alua_bus_detach(struct scsi_device *sdev)
 	spin_lock(&h->pg_lock);
 	pg = rcu_dereference_protected(h->pg, lockdep_is_held(&h->pg_lock));
 	rcu_assign_pointer(h->pg, NULL);
-	h->sdev = NULL;
 	spin_unlock(&h->pg_lock);
 	if (pg) {
 		spin_lock_irq(&pg->lock);

drivers/scsi/device_handler/scsi_dh_alua.c static void alua_bus_detach(struct scsi_device *sdev) · line 1161

Quoted 6 lines
drivers/scsi/device_handler/scsi_dh_alua.c +1 -0
@@ -1156,6 +1156,7 @@ static void alua_bus_detach(struct scsi_device *sdev)
 		kref_put(&pg->kref, release_port_group);
 	}
 	sdev->handler_data = NULL;
+	synchronize_rcu();
 	kfree(h);
 }

Hi Hannes,

Do you agree that the changes in alua_bus_detach() make the changes in alua_rtpg() superfluous?

How about freezing command processing for 'sdev' while detaching a device handler instead of inserting a synchronize_rcu() call in alua_bus_detach()? I'm concerned that the alua_bus_detach() changes are not sufficient to fix all possible races between detaching a device handler and the following code from the SCSI error handler:

	if (sdev->handler && sdev->handler->check_sense) {
		int rc;

		rc = sdev->handler->check_sense(sdev, &sshdr);
		if (rc != SCSI_RETURN_NOT_HANDLED)
			return rc;
		/* handler does not care. Drop down to default handling */
	}

Thanks,

Bart.

#4 lore raw
Quoted 71 lines · Bart Van Assche
Quoted 60 lines · Hannes Reinecke

alua_bus_detach() might be running concurrently with alua_rtpg_work(),
so we might trip over h->sdev == NULL and call BUG_ON(). The correct way of handling it would be to not set h->sdev to NULL in alua_bus_detach(), and call rcu_synchronize() before the final delete to ensure that all concurrent threads have left the critical section.
Then we can get rid of the BUG_ON(), and replace it with a simple if condition.

---
 drivers/scsi/device_handler/scsi_dh_alua.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
drivers/scsi/device_handler/scsi_dh_alua.c +0 -1
index f32da0ca529e..308bda2e9c00 100644
@@ -658,8 +658,8 @@ static int alua_rtpg(struct scsi_device *sdev,
 struct alua_port_group *pg)
 					rcu_read_lock();
 					list_for_each_entry_rcu(h,
 						&tmp_pg->dh_list, node)
 {
-						/* h->sdev should

always be valid */

drivers/scsi/device_handler/scsi_dh_alua.c +2 -1
-						BUG_ON(!h->sdev);
+						if (!h->sdev)
+							continue;
 						h->sdev->access_state =

desc[0];

drivers/scsi/device_handler/scsi_dh_alua.c +2 -1
 					}
 					rcu_read_unlock();
@@ -705,7 +705,8 @@ static int alua_rtpg(struct scsi_device *sdev,
 struct alua_port_group *pg)
 			pg->expiry = 0;
 			rcu_read_lock();
 			list_for_each_entry_rcu(h, &pg->dh_list, node)
 {
-				BUG_ON(!h->sdev);
+				if (!h->sdev)
+					continue;
 				h->sdev->access_state =

(pg->state &
SCSI_ACCESS_STATE_MASK);

drivers/scsi/device_handler/scsi_dh_alua.c +0 -1
 				if (pg->pref)
@@ -1147,7 +1148,6 @@ static void alua_bus_detach(struct
 scsi_device *sdev)
 	spin_lock(&h->pg_lock);
 	pg = rcu_dereference_protected(h->pg, lockdep_is_held(&h-
 >pg_lock));
 	rcu_assign_pointer(h->pg, NULL);
-	h->sdev = NULL;
 	spin_unlock(&h->pg_lock);
drivers/scsi/device_handler/scsi_dh_alua.c +0 -0
 	if (pg) {
 		spin_lock_irq(&pg->lock);

drivers/scsi/device_handler/scsi_dh_alua.c static void alua_bus_detach(struct · line 1162

Quoted 7 lines
drivers/scsi/device_handler/scsi_dh_alua.c +1 -0
@@ -1156,6 +1156,7 @@ static void alua_bus_detach(struct
 scsi_device *sdev)
 		kref_put(&pg->kref, release_port_group);
 	}
 	sdev->handler_data = NULL;
+	synchronize_rcu();
 	kfree(h);
 }

Hi Hannes,

Quoted 3 lines

Do you agree that the changes in alua_bus_detach() make the changes in
alua_rtpg() superfluous?

I agree that the "if (!h->sdev) continue;" should not be needed in
alua_rtpg() if the h->sdev remains valid while in the list.

I'm a little concerned about adding the synchronize_rcu() as this is called in the scsi_device_dev_release_usercontext() path, with a lot of LUNs it could take a while to remove all the devices, see e.g.:

f983622ae605 scsi: core: Avoid calling synchronize_rcu() for each
device in scsi_host_block()

It doesn't look like we ever NULL sdev->handler on detach even though we do a module_put() on the DH. But we have already called the release() function so perhaps this doesn't cause a problem in practice.

-Ewan

Quoted 22 lines

How about freezing command processing for 'sdev' while detaching a device handler instead of inserting a synchronize_rcu() call in alua_bus_detach()? I'm concerned that the alua_bus_detach() changes are
not sufficient to fix all possible races between detaching a device handler and the following code from the SCSI error handler:

	if (sdev->handler && sdev->handler->check_sense) {
		int rc;

		rc = sdev->handler->check_sense(sdev, &sshdr);
		if (rc != SCSI_RETURN_NOT_HANDLED)
			return rc;
		/* handler does not care. Drop down to default handling
*/
	}

Thanks,

Bart.

#5 lore raw
In reply to Hannes Reinecke
Quoted 8 lines · Hannes Reinecke

alua_bus_detach() might be running concurrently with alua_rtpg_work(), so we might trip over h->sdev == NULL and call BUG_ON(). The correct way of handling it would be to not set h->sdev to NULL in alua_bus_detach(), and call rcu_synchronize() before the final delete to ensure that all concurrent threads have left the critical section.
Then we can get rid of the BUG_ON(), and replace it with a simple if condition.

Applied to 5.10/scsi-fixes, thanks!

[1/1] scsi: scsi_dh_alua: Avoid crash during alua_bus_detach()

      https://git.kernel.org/mkp/scsi/c/5faf50e9e9fd
Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering