Skip to contentSkip to conversation

restrict legal sdev_state transitions via sysfs

PATCH Applied Uday Shankar 5 participants 6 messages Sep 24, 2022 – Oct 18, 2022 SCSI

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 restrict legal sdev_state transitions via sysfs

Userspace can currently write to sysfs to transition sdev_state to RUNNING or OFFLINE from any source state. This causes issues because proper transitioning out of some states involves steps besides just changing sdev_state, so allowing userspace to change sdev_state regardless of the source state can result in inconsistencies; e.g. with iscsi we can end up with sdev_state == SDEV_RUNNING while the device queue is quiesced. Any task attempting IO on the device will then hang, and in more recent kernels, iscsid will hang as well. More detail about this bug is provided in my first attempt:
https://groups.google.com/g/open-iscsi/c/PNKca4HgPDs/m/CXaDkntOAQAJ

---
Looking for feedback on the "allowed source states" list. The bug I hit is solved by prohibiting transitions out of SDEV_BLOCKED, but I think most others shouldn't be allowed either.

 drivers/scsi/scsi_sysfs.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/scsi/scsi_sysfs.c +8 -0
index 9dad2fd5297f..b38c30fe681d 100644
@@ -822,6 +822,14 @@ store_state_field(struct device *dev, struct device_attribute *attr,
 	}
 
 	mutex_lock(&sdev->state_mutex);
+	switch (sdev->sdev_state) {
+	case SDEV_RUNNING:
+	case SDEV_OFFLINE:
+		break;
+	default:
+		mutex_unlock(&sdev->state_mutex);
+		return -EINVAL;
+	}
 	if (sdev->sdev_state == SDEV_RUNNING && state == SDEV_RUNNING) {
 		ret = 0;
 	} else {

base-commit: 7f615c1b5986ff08a725ee489e838c90f8197bcd

Signature · 1 line
2.25.1
#1 lore raw
Quoted 4 lines · Uday Shankar

---
Looking for feedback on the "allowed source states" list. The bug I hit is solved by prohibiting transitions out of SDEV_BLOCKED, but I think most others shouldn't be allowed either.

I think it's ok to be restrictive:

1. BLOCKED is just broken. When the transport classes and scsi_lib transition out of that state they do a lot more than just set the set. We are leaving the kernel in mismatched state where the device state is running but the block layerand transport classes are not ready for IO.

2. CREATED does not happen. We go into RUNNING then do scsi_sysfs_add_sdev so userspace should not see the CREATED state.

3. I'm not 100% sure about SDEV_QUIESCE though. It looks like it has similar issues as BLOCKED where scsi_device_resume will undo things it did in scsi_device_quiesce, so we can't just set the state to RUNNING and expect things to work. I'm not sure about the scsi_transport_spi cases.

It would be best for James or Hannes to comment because they know that code well.

4. The transport classes are the ones that put devs in SDEV_TRANSPORT_OFFLINE and then transition them when they are ready. The block layer is at least in the correct state, but the transport classes may not be ready for IO since they are not expecting IO to be queued to them at that time.

Quoted 25 lines
 drivers/scsi/scsi_sysfs.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/scsi/scsi_sysfs.c +8 -0
index 9dad2fd5297f..b38c30fe681d 100644
@@ -822,6 +822,14 @@ store_state_field(struct device *dev, struct device_attribute *attr,
 	}
 
 	mutex_lock(&sdev->state_mutex);
+	switch (sdev->sdev_state) {
+	case SDEV_RUNNING:
+	case SDEV_OFFLINE:
+		break;
+	default:
+		mutex_unlock(&sdev->state_mutex);
+		return -EINVAL;
+	}
 	if (sdev->sdev_state == SDEV_RUNNING && state == SDEV_RUNNING) {
 		ret = 0;
 	} else {

base-commit: 7f615c1b5986ff08a725ee489e838c90f8197bcd

#2 lore raw
Quoted 16 lines · Mike Christie
Quoted 4 lines · Uday Shankar

---
Looking for feedback on the "allowed source states" list. The bug I hit is solved by prohibiting transitions out of SDEV_BLOCKED, but I think most others shouldn't be allowed either.

I think it's ok to be restrictive:

1. BLOCKED is just broken. When the transport classes and scsi_lib transition out of that state they do a lot more than just set the set. We are leaving the kernel in mismatched state where the device state is running but the block layerand transport classes are not ready for IO.

2. CREATED does not happen. We go into RUNNING then do scsi_sysfs_add_sdev so userspace should not see the CREATED state.

Quoted 6 lines

3. I'm not 100% sure about SDEV_QUIESCE though. It looks like it has similar issues as BLOCKED where scsi_device_resume will undo things it did in scsi_device_quiesce, so we can't just set the state to RUNNING and expect things to work. I'm not sure about the scsi_transport_spi cases.

It would be best for James or Hannes to comment because they know that code well.

Adding Hannes to CC.

Quoted 31 lines

4. The transport classes are the ones that put devs in SDEV_TRANSPORT_OFFLINE and then transition them when they are ready. The block layer is at least in the correct state, but the transport classes may not be ready for IO since they are not expecting IO to be queued to them at that time.

Quoted 25 lines
 drivers/scsi/scsi_sysfs.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/scsi/scsi_sysfs.c +8 -0
index 9dad2fd5297f..b38c30fe681d 100644
@@ -822,6 +822,14 @@ store_state_field(struct device *dev, struct device_attribute *attr,
 	}
 
 	mutex_lock(&sdev->state_mutex);
+	switch (sdev->sdev_state) {
+	case SDEV_RUNNING:
+	case SDEV_OFFLINE:
+		break;
+	default:
+		mutex_unlock(&sdev->state_mutex);
+		return -EINVAL;
+	}
 	if (sdev->sdev_state == SDEV_RUNNING && state == SDEV_RUNNING) {
 		ret = 0;
 	} else {

base-commit: 7f615c1b5986ff08a725ee489e838c90f8197bcd

Reviewed-by
#3 lore raw
In reply to Mike Christie
Quoted 16 lines · Mike Christie
Quoted 4 lines · Uday Shankar

---
Looking for feedback on the "allowed source states" list. The bug I hit is solved by prohibiting transitions out of SDEV_BLOCKED, but I think most others shouldn't be allowed either.

I think it's ok to be restrictive:

1. BLOCKED is just broken. When the transport classes and scsi_lib transition out of that state they do a lot more than just set the set. We are leaving the kernel in mismatched state where the device state is running but the block layerand transport classes are not ready for IO.

2. CREATED does not happen. We go into RUNNING then do scsi_sysfs_add_sdev so userspace should not see the CREATED state.

Quoted 7 lines

3. I'm not 100% sure about SDEV_QUIESCE though. It looks like it has similar issues as BLOCKED where scsi_device_resume will undo things it did in scsi_device_quiesce, so we can't just set the state to RUNNING and expect things to work. I'm not sure about the scsi_transport_spi cases.

It would be best for James or Hannes to comment because they know that code well.

Indeed, you are correct.
The only sensible state transitions to be modified from userland is indeed between SDEV_RUNNING and SDEV_OFFLINE.
All other states are in fact part of the SCSI midlayer operations, and there a quite strict rules on which state transitions are allowed and who should be initiating them.

So yes, I'm fine with this patch.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew
Myers, Andrew McDonald, Martje Boudien Moerman
#4 lore raw
In reply to Uday Shankar
Quoted 30 lines · Uday Shankar

Userspace can currently write to sysfs to transition sdev_state to RUNNING or OFFLINE from any source state. This causes issues because proper transitioning out of some states involves steps besides just changing sdev_state, so allowing userspace to change sdev_state regardless of the source state can result in inconsistencies; e.g. with iscsi we can end up with sdev_state == SDEV_RUNNING while the device queue is quiesced. Any task attempting IO on the device will then hang, and in more recent kernels, iscsid will hang as well. More detail about this bug is provided in my first attempt:
https://groups.google.com/g/open-iscsi/c/PNKca4HgPDs/m/CXaDkntOAQAJ

---
Looking for feedback on the "allowed source states" list. The bug I hit is solved by prohibiting transitions out of SDEV_BLOCKED, but I think most others shouldn't be allowed either.

  drivers/scsi/scsi_sysfs.c | 8 ++++++++
  1 file changed, 8 insertions(+)
drivers/scsi/scsi_sysfs.c +1 -0
index 9dad2fd5297f..b38c30fe681d 100644
@@ -822,6 +822,14 @@ store_state_field(struct device *dev, struct device_attribute *attr,
  	}
 
  	mutex_lock(&sdev->state_mutex);
+	switch (sdev->sdev_state) {

drivers/scsi/scsi_sysfs.c store_state_field(struct device *dev, struct device_attribute *attr, · line 835

Quoted 10 lines
drivers/scsi/scsi_sysfs.c +7 -0
+	case SDEV_RUNNING:
+	case SDEV_OFFLINE:
+		break;
+	default:
+		mutex_unlock(&sdev->state_mutex);
+		return -EINVAL;
+	}
  	if (sdev->sdev_state == SDEV_RUNNING && state == SDEV_RUNNING) {
  		ret = 0;
  	} else {

The return value -EAGAIN might be more appropriate since it is not the value written into sysfs that is invalid but the current state that is inappropriate for the requested transition.

Thanks,

Bart.

#5 lore raw
In reply to Uday Shankar
Quoted 11 lines · Uday Shankar

Userspace can currently write to sysfs to transition sdev_state to RUNNING or OFFLINE from any source state. This causes issues because proper transitioning out of some states involves steps besides just changing sdev_state, so allowing userspace to change sdev_state regardless of the source state can result in inconsistencies; e.g. with iscsi we can end up with sdev_state == SDEV_RUNNING while the device queue is quiesced. Any task attempting IO on the device will then hang, and in more recent kernels, iscsid will hang as well. More detail about this bug is provided in my first attempt:
https://groups.google.com/g/open-iscsi/c/PNKca4HgPDs/m/CXaDkntOAQAJ

Quoted 1 line

[...]

Applied to 6.1/scsi-fixes, thanks!

[1/1] restrict legal sdev_state transitions via sysfs

      https://git.kernel.org/mkp/scsi/c/2331ce6126be
Signature · 1 line
Martin K. Petersen	Oracle Linux Engineering