scsi_dh_alua: handle RTPG sense code correctly during state transitions

PATCH Applied Hannes Reinecke 5 participants 7 messages Oct 7, 2019 – Oct 10, 2019 SCSI

lore

Share & tools

lore

Hannes Reinecke

author

2019-10-07 13:57 UTC #0 lore raw

PATCH 1/1 scsi_dh_alua: handle RTPG sense code correctly during state transitions

From: Hannes Reinecke <[email protected]>

Some arrays are not capable of returning RTPG data during state transitioning, but rather return an 'LUN not accessible, asymmetric access state transition' sense code. In these cases we can set the state to 'transitioning' directly and don't need to evaluate the RTPG data (which we won't have anyway).

Signed-off-by Hannes Reinecke <[email protected]>

---
 drivers/scsi/device_handler/scsi_dh_alua.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

drivers/scsi/device_handler/scsi_dh_alua.c +16 -5

index 4971104b1817..f32da0ca529e 100644
@@ -512,6 +512,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 	unsigned int tpg_desc_tbl_off;
 	unsigned char orig_transition_tmo;
 	unsigned long flags;
+	bool transitioning_sense = false;
 
 	if (!pg->expiry) {
 		unsigned long transition_tmo = ALUA_FAILOVER_TIMEOUT * HZ;

@@ -572,13 +573,19 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 			goto retry;
 		}
 		/*
-		 * Retry on ALUA state transition or if any
-		 * UNIT ATTENTION occurred.
+		 * If the array returns with 'ALUA state transition'
+		 * sense code here it cannot return RTPG data during
+		 * transition. So set the state to 'transitioning' directly.
 		 */
 		if (sense_hdr.sense_key == NOT_READY &&
-		    sense_hdr.asc == 0x04 && sense_hdr.ascq == 0x0a)
-			err = SCSI_DH_RETRY;
-		else if (sense_hdr.sense_key == UNIT_ATTENTION)
+		    sense_hdr.asc == 0x04 && sense_hdr.ascq == 0x0a) {
+			transitioning_sense = true;
+			goto skip_rtpg;
+		}
+		/*
+		 * Retry on any other UNIT ATTENTION occurred.
+		 */
+		if (sense_hdr.sense_key == UNIT_ATTENTION)
 			err = SCSI_DH_RETRY;
 		if (err == SCSI_DH_RETRY &&
 		    pg->expiry != 0 && time_before(jiffies, pg->expiry)) {

@@ -666,7 +673,11 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 		off = 8 + (desc[7] * 4);
 	}
 
+ skip_rtpg:
 	spin_lock_irqsave(&pg->lock, flags);
+	if (transitioning_sense)
+		pg->state = SCSI_ACCESS_STATE_TRANSITIONING;
+
 	sdev_printk(KERN_INFO, sdev,
 		    "%s: port group %02x state %c %s supports %c%c%c%c%c%c%c\n",
 		    ALUA_DH_NAME, pg->group_id, print_alua_state(pg->state),

Signature · 1 line

2.16.4

Laurence Oberman

Reviewed-by on v1 1/1

2019-10-07 14:15 UTC #1 lore raw

Quoted 69 lines · Hannes Reinecke

From: Hannes Reinecke <[email protected]>

Signed-off-by Hannes Reinecke <[email protected]>

---
 drivers/scsi/device_handler/scsi_dh_alua.c | 21 ++++++++++++++++--
---
 1 file changed, 16 insertions(+), 5 deletions(-)

drivers/scsi/device_handler/scsi_dh_alua.c +1 -0

index 4971104b1817..f32da0ca529e 100644
@@ -512,6 +512,7 @@ static int alua_rtpg(struct scsi_device *sdev,
 struct alua_port_group *pg)
 	unsigned int tpg_desc_tbl_off;
 	unsigned char orig_transition_tmo;
 	unsigned long flags;
+	bool transitioning_sense = false;
 
 	if (!pg->expiry) {

unsigned long transition_tmo = ALUA_FAILOVER_TIMEOUT *
HZ;

drivers/scsi/device_handler/scsi_dh_alua.c +3 -2

@@ -572,13 +573,19 @@ static int alua_rtpg(struct scsi_device *sdev,
 struct alua_port_group *pg)
 			goto retry;
 		}
 		/*
-		 * Retry on ALUA state transition or if any
-		 * UNIT ATTENTION occurred.
+		 * If the array returns with 'ALUA state transition'
+		 * sense code here it cannot return RTPG data during
+		 * transition. So set the state to 'transitioning'

directly.

drivers/scsi/device_handler/scsi_dh_alua.c +8 -3

 		 */
 		if (sense_hdr.sense_key == NOT_READY &&
-		    sense_hdr.asc == 0x04 && sense_hdr.ascq == 0x0a)
-			err = SCSI_DH_RETRY;
-		else if (sense_hdr.sense_key == UNIT_ATTENTION)
+		    sense_hdr.asc == 0x04 && sense_hdr.ascq == 0x0a) {
+			transitioning_sense = true;
+			goto skip_rtpg;
+		}
+		/*
+		 * Retry on any other UNIT ATTENTION occurred.
+		 */
+		if (sense_hdr.sense_key == UNIT_ATTENTION)
 			err = SCSI_DH_RETRY;
 		if (err == SCSI_DH_RETRY &&
 		    pg->expiry != 0 && time_before(jiffies, pg-

Quoted 1 line

expiry)) {

drivers/scsi/device_handler/scsi_dh_alua.c +4 -0

@@ -666,7 +673,11 @@ static int alua_rtpg(struct scsi_device *sdev,
 struct alua_port_group *pg)
 		off = 8 + (desc[7] * 4);
 	}
 
+ skip_rtpg:
 	spin_lock_irqsave(&pg->lock, flags);
+	if (transitioning_sense)
+		pg->state = SCSI_ACCESS_STATE_TRANSITIONING;
+
 	sdev_printk(KERN_INFO, sdev,
 		    "%s: port group %02x state %c %s supports

Quoted 3 lines

%c%c%c%c%c%c%c\n",
ALUA_DH_NAME, pg->group_id, print_alua_state(pg-
Quoted 1 line
state),

This makes sense to me and has affected recovery timeouts in the past. Code looks correct to me.

Reviewed-by Laurence Oberman <[email protected]>

Ewan D. Milne

2019-10-07 20:45 UTC #2 lore raw

In reply to Hannes Reinecke

See below.

Quoted 52 lines · Hannes Reinecke

From: Hannes Reinecke <[email protected]>

Signed-off-by Hannes Reinecke <[email protected]>

---
 drivers/scsi/device_handler/scsi_dh_alua.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

drivers/scsi/device_handler/scsi_dh_alua.c +12 -5

index 4971104b1817..f32da0ca529e 100644
@@ -512,6 +512,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 	unsigned int tpg_desc_tbl_off;
 	unsigned char orig_transition_tmo;
 	unsigned long flags;
+	bool transitioning_sense = false;
 
 	if (!pg->expiry) {
 		unsigned long transition_tmo = ALUA_FAILOVER_TIMEOUT * HZ;

@@ -572,13 +573,19 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 			goto retry;
 		}
 		/*
-		 * Retry on ALUA state transition or if any
-		 * UNIT ATTENTION occurred.
+		 * If the array returns with 'ALUA state transition'
+		 * sense code here it cannot return RTPG data during
+		 * transition. So set the state to 'transitioning' directly.
 		 */
 		if (sense_hdr.sense_key == NOT_READY &&
-		    sense_hdr.asc == 0x04 && sense_hdr.ascq == 0x0a)
-			err = SCSI_DH_RETRY;
-		else if (sense_hdr.sense_key == UNIT_ATTENTION)
+		    sense_hdr.asc == 0x04 && sense_hdr.ascq == 0x0a) {
+			transitioning_sense = true;
+			goto skip_rtpg;
+		}
+		/*
+		 * Retry on any other UNIT ATTENTION occurred.
+		 */
+		if (sense_hdr.sense_key == UNIT_ATTENTION)
 			err = SCSI_DH_RETRY;
 		if (err == SCSI_DH_RETRY &&
 		    pg->expiry != 0 && time_before(jiffies, pg->expiry)) {

@@ -666,7 +673,11 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 		off = 8 + (desc[7] * 4);

drivers/scsi/device_handler/scsi_dh_alua.c static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg) · line 683

Quoted 10 lines

drivers/scsi/device_handler/scsi_dh_alua.c +4 -0

 	}
 
+ skip_rtpg:
 	spin_lock_irqsave(&pg->lock, flags);
+	if (transitioning_sense)
+		pg->state = SCSI_ACCESS_STATE_TRANSITIONING;
+
 	sdev_printk(KERN_INFO, sdev,
 		    "%s: port group %02x state %c %s supports %c%c%c%c%c%c%c\n",
 		    ALUA_DH_NAME, pg->group_id, print_alua_state(pg->state),

The patch itself looks OK, but I was wondering about a couple of things:

- There are other places in scsi_dh_alua where the ASC/ASCQ 04 0A is checked and we retry, I understand that this is a particular case you are solving but is the changing of the state to -> transitioning (because that's what the device said the state was) applicable in those other cases?
- The code originally seems to have been under the assumption that the transitioning state was a transient event, so the retry would pick up the eventual state. Now, some storage arrays spend a long time in the transitioning state, but if we don't send another command are we going to get the sense (or the UA) that triggers entry to the eventual ALUA state?

-Ewan

Hannes Reinecke

author

2019-10-08 06:21 UTC #3 lore raw

Quoted 68 lines · Ewan D. Milne

See below.

Quoted 52 lines · Hannes Reinecke

From: Hannes Reinecke <[email protected]>

Signed-off-by Hannes Reinecke <[email protected]>

---
 drivers/scsi/device_handler/scsi_dh_alua.c | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

drivers/scsi/device_handler/scsi_dh_alua.c +12 -5

index 4971104b1817..f32da0ca529e 100644
@@ -512,6 +512,7 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 	unsigned int tpg_desc_tbl_off;
 	unsigned char orig_transition_tmo;
 	unsigned long flags;
+	bool transitioning_sense = false;
 
 	if (!pg->expiry) {
 		unsigned long transition_tmo = ALUA_FAILOVER_TIMEOUT * HZ;

@@ -572,13 +573,19 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 			goto retry;
 		}
 		/*
-		 * Retry on ALUA state transition or if any
-		 * UNIT ATTENTION occurred.
+		 * If the array returns with 'ALUA state transition'
+		 * sense code here it cannot return RTPG data during
+		 * transition. So set the state to 'transitioning' directly.
 		 */
 		if (sense_hdr.sense_key == NOT_READY &&
-		    sense_hdr.asc == 0x04 && sense_hdr.ascq == 0x0a)
-			err = SCSI_DH_RETRY;
-		else if (sense_hdr.sense_key == UNIT_ATTENTION)
+		    sense_hdr.asc == 0x04 && sense_hdr.ascq == 0x0a) {
+			transitioning_sense = true;
+			goto skip_rtpg;
+		}
+		/*
+		 * Retry on any other UNIT ATTENTION occurred.
+		 */
+		if (sense_hdr.sense_key == UNIT_ATTENTION)
 			err = SCSI_DH_RETRY;
 		if (err == SCSI_DH_RETRY &&
 		    pg->expiry != 0 && time_before(jiffies, pg->expiry)) {

@@ -666,7 +673,11 @@ static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg)
 		off = 8 + (desc[7] * 4);

drivers/scsi/device_handler/scsi_dh_alua.c static int alua_rtpg(struct scsi_device *sdev, struct alua_port_group *pg) · line 683

Quoted 10 lines

drivers/scsi/device_handler/scsi_dh_alua.c +4 -0

 	}
 
+ skip_rtpg:
 	spin_lock_irqsave(&pg->lock, flags);
+	if (transitioning_sense)
+		pg->state = SCSI_ACCESS_STATE_TRANSITIONING;
+
 	sdev_printk(KERN_INFO, sdev,
 		    "%s: port group %02x state %c %s supports %c%c%c%c%c%c%c\n",
 		    ALUA_DH_NAME, pg->group_id, print_alua_state(pg->state),

The patch itself looks OK, but I was wondering about a couple of things:

Quoted 4 lines

- There are other places in scsi_dh_alua where the ASC/ASCQ 04 0A is checked and we retry, I understand that this is a particular case you are solving but is the changing of the state to -> transitioning (because that's what the device said the state was) applicable in those other cases?

No. The original code was built around the assumption that RTPG would return the status of the device; consequently we would have to retry RTPG until we get a final status. But as mentioned, there are arrays which cannot return RTPG data during transitioning, so the code would never be able to detect a transitioning state.
With this patch we set the state directly once the said sense code is received.
But this applies _only_ to the RTPG command, as this is required to move the state machine along.
None of the other commands are affected.

Quoted 6 lines

- The code originally seems to have been under the assumption that the transitioning state was a transient event, so the retry would pick up the eventual state. Now, some storage arrays spend a long time in the transitioning state, but if we don't send another command are we going to get the sense (or the UA) that triggers entry to the eventual ALUA state?

Note, there are two types of retries.
The one is the 'normal' command retry, where we resend a command a given number of times to retrieve the final status.
This is precisely the error which caused this patch.

And then there is a scheduled retry; here we essentially poll the array with sending RTPG in regular intervals until the 'transitioning' state is gone. (Check for 'alua_rtpg()' and the handling of the SCSI_DH_RETRY return value). With the patch we continue to trigger that second type of retries, which will eventually clear the transitioning state.

Cheers,

Hannes

Signature · 4 lines

Dr. Hannes Reinecke		      Teamlead Storage & Networking
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 247165 (AG München), GF: Felix Imendörffer

Ewan D. Milne

Reviewed-by

2019-10-08 15:58 UTC #4 lore raw

Quoted 38 lines · Hannes Reinecke

Quoted 7 lines · Ewan D. Milne
The patch itself looks OK, but I was wondering about a couple of things:
- There are other places in scsi_dh_alua where the ASC/ASCQ 04 0A is checked and we retry, I understand that this is a particular case you are solving but is the changing of the state to -> transitioning (because that's what the device said the state was) applicable in those other cases?
No. The original code was built around the assumption that RTPG would return the status of the device; consequently we would have to retry RTPG until we get a final status. But as mentioned, there are arrays which cannot return RTPG data during transitioning, so the code would never be able to detect a transitioning state.
With this patch we set the state directly once the said sense code is received.
But this applies _only_ to the RTPG command, as this is required to move the state machine along.
None of the other commands are affected.
Quoted 6 lines
- The code originally seems to have been under the assumption that the transitioning state was a transient event, so the retry would pick up the eventual state. Now, some storage arrays spend a long time in the transitioning state, but if we don't send another command are we going to get the sense (or the UA) that triggers entry to the eventual ALUA state?
Note, there are two types of retries.
The one is the 'normal' command retry, where we resend a command a given number of times to retrieve the final status.
This is precisely the error which caused this patch.
And then there is a scheduled retry; here we essentially poll the array with sending RTPG in regular intervals until the 'transitioning' state is gone. (Check for 'alua_rtpg()' and the handling of the SCSI_DH_RETRY return value). With the patch we continue to trigger that second type of retries, which will eventually clear the transitioning state.

Quoted 3 lines

Cheers,
Hannes

Thanks for the explanation. The patch looks good.

Reviewed-by Ewan D. Milne <[email protected]>

Bart Van Assche

Reviewed-by on v1 1/1

2019-10-09 16:31 UTC #5 lore raw

In reply to Hannes Reinecke

Quoted 5 lines · Hannes Reinecke

Some arrays are not capable of returning RTPG data during state transitioning, but rather return an 'LUN not accessible, asymmetric access state transition' sense code. In these cases we can set the state to 'transitioning' directly and don't need to evaluate the RTPG data (which we won't have anyway).

Reviewed-by Bart Van Assche <[email protected]>

Martin K. Petersen

Applied

2019-10-10 02:43 UTC #6 lore raw

In reply to Hannes Reinecke

Hannes,

Quoted 5 lines

Some arrays are not capable of returning RTPG data during state transitioning, but rather return an 'LUN not accessible, asymmetric access state transition' sense code. In these cases we can set the state to 'transitioning' directly and don't need to evaluate the RTPG data (which we won't have anyway).

Applied to 5.4/scsi-fixes, thanks you!

Signature · 1 line

Martin K. Petersen	Oracle Linux Engineering