Skip to contentSkip to conversation

multipathd: the sysfs prioritizer can return stale data

PATCH In review Brian Bunker 3 participants 9 messages Jan 24, 2024 – Jan 31, 2024 Device mapper

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 multipathd: the sysfs prioritizer can return stale data

When a path is lost and then reinstated later, the ALUA device handler will not pick up this change and continue to possibly provide incorrect (stale) information about its ALUA state to the 'sysfs' prioritizer if the path's priorities were changed prior to the loss of those paths.

On the loss of an I_T nexus, the path state should not continue to use its last known state. Many things and a lot of time could have passed between the path loss and its eventual reinstatemnt.

The ALUA device handler didn't have this issue since it always got the ALUA state from the target by sending an RTPG request on the path and updated the priority. However with the detect priority set to true by default, the 'sysfs' prioritzier will be used on targets that support ALUA rather than the 'alua' prioritizer.

Without re-evaluating the ALUA state when a path returns multipath is left with path states which do not reflect the actual ALUA states the target is providing.

3624a9370f0f545fc7c3e46a100011010 dm-2 PURE,FlashArray size=3.0T features='0' hwhandler='1 alua' wp=rw

|-+- policy='service-time 0' prio=50 status=active
| |- 13:0:0:1 sdg 8:96  active ready running
| |- 12:0:0:1 sdf 8:80  active ready running
| |- 9:0:0:1  sdc 8:32  active ready running
| `- 1:0:0:1  sdb 8:16  active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  |- 14:0:0:1 sdh 8:112 active ready running
  |- 15:0:0:1 sdi 8:128 active ready running
  |- 10:0:0:1 sdd 8:48  active ready running

`- 11:0:0:1 sde 8:64 active ready running

 # sg_rtpg /dev/sdh (Active/Optimized)
target port group asymmetric access state : 0x00
---
 multipathd/main.c | 5 +++++
 1 file changed, 5 insertions(+)
multipathd/main.c +5 -0
index 230c9d10..dd48be74 100644
@@ -1937,6 +1937,11 @@ reinstate_path (struct path * pp)
 	else {
 		condlog(2, "%s: reinstated", pp->dev_t);
 		update_queue_mode_add_path(pp->mpp);
+		if (strcmp(pp->prio.name, PRIO_SYSFS) == 0) {
+			condlog(2, "%s: rescan target to update priorities",
+				pp->dev_t);
+			rescan_path(pp->udev);
+		}
 	}
 }
 
Signature · 1 line
2.43.0
#1 lore raw
Quoted 49 lines · Brian Bunker

When a path is lost and then reinstated later, the ALUA device handler will not pick up this change and continue to possibly provide incorrect (stale) information about its ALUA state to the 'sysfs' prioritizer if the path's priorities were changed prior to the loss of those paths.

On the loss of an I_T nexus, the path state should not continue to use its last known state. Many things and a lot of time could have passed between the path loss and its eventual reinstatemnt.

The ALUA device handler didn't have this issue since it always got the ALUA state from the target by sending an RTPG request on the path and updated the priority. However with the detect priority set to true by default, the 'sysfs' prioritzier will be used on targets that support ALUA rather than the 'alua' prioritizer.

Without re-evaluating the ALUA state when a path returns multipath is left with path states which do not reflect the actual ALUA states the target is providing.

3624a9370f0f545fc7c3e46a100011010 dm-2 PURE,FlashArray size=3.0T features='0' hwhandler='1 alua' wp=rw

|-+- policy='service-time 0' prio=50 status=active
| |- 13:0:0:1 sdg 8:96  active ready running
| |- 12:0:0:1 sdf 8:80  active ready running
| |- 9:0:0:1  sdc 8:32  active ready running
| `- 1:0:0:1  sdb 8:16  active ready running
`-+- policy='service-time 0' prio=10 status=enabled
  |- 14:0:0:1 sdh 8:112 active ready running
  |- 15:0:0:1 sdi 8:128 active ready running
  |- 10:0:0:1 sdd 8:48  active ready running

`- 11:0:0:1 sde 8:64 active ready running

 # sg_rtpg /dev/sdh (Active/Optimized)
target port group asymmetric access state : 0x00
---
 multipathd/main.c | 5 +++++
 1 file changed, 5 insertions(+)
multipathd/main.c +0 -0
index 230c9d10..dd48be74 100644

multipathd/main.c reinstate_path (struct path * pp) · line 1943

Quoted 7 lines
multipathd/main.c +4 -0
@@ -1937,6 +1937,11 @@ reinstate_path (struct path * pp)
 	else {
 		condlog(2, "%s: reinstated", pp->dev_t);
 		update_queue_mode_add_path(pp->mpp);
+		if (strcmp(pp->prio.name, PRIO_SYSFS) == 0) {
+			condlog(2, "%s: rescan target to update priorities",
+				pp->dev_t);
+			rescan_path(pp->udev);

I can see why this is necessary with the sysfs target, but AFAICT rescanning the scsi device will end up calling scsi_execute_cmd to update the vpd pages, and this can block. Obviously, we are doing this after the checker has verified that the path is up, but even still, we're trying to cut down on the amount of code that can block multipathd on an inaccessible device, instead of increase it.

Perhaps it would be better to make the prioritizer itself smarter, so that it could know when it needs to run the rescan(), and it could do that in a separate thread.

Thoughts?

-Ben

Quoted 6 lines
multipathd/main.c +1 -0
+		}
 	}
 }
 
Signature · 1 line
2.43.0
#2 lore raw
Quoted 68 lines · Benjamin Marzinski
Quoted 49 lines · Brian Bunker

When a path is lost and then reinstated later, the ALUA device handler will not pick up this change and continue to possibly provide incorrect (stale) information about its ALUA state to the 'sysfs' prioritizer if the path's priorities were changed prior to the loss of those paths.

On the loss of an I_T nexus, the path state should not continue to use its last known state. Many things and a lot of time could have passed between the path loss and its eventual reinstatemnt.

The ALUA device handler didn't have this issue since it always got the ALUA state from the target by sending an RTPG request on the path and updated the priority. However with the detect priority set to true by default, the 'sysfs' prioritzier will be used on targets that support ALUA rather than the 'alua' prioritizer.

Without re-evaluating the ALUA state when a path returns multipath is left with path states which do not reflect the actual ALUA states the target is providing.

3624a9370f0f545fc7c3e46a100011010 dm-2 PURE,FlashArray size=3.0T features='0' hwhandler='1 alua' wp=rw

Quoted 5 lines

-+- policy='service-time 0' prio=50 status=active

Quoted 3 lines

- 13:0:0:1 sdg 8:96  active ready running
- 12:0:0:1 sdf 8:80  active ready running
- 9:0:0:1  sdc 8:32  active ready running

`- 1:0:0:1  sdb 8:16  active ready running

`-+- policy='service-time 0' prio=10 status=enabled

  |- 14:0:0:1 sdh 8:112 active ready running
  |- 15:0:0:1 sdi 8:128 active ready running
  |- 10:0:0:1 sdd 8:48  active ready running

  `- 11:0:0:1 sde 8:64  active ready running

 # sg_rtpg /dev/sdh (Active/Optimized)
target port group asymmetric access state : 0x00

---
 multipathd/main.c | 5 +++++

 1 file changed, 5 insertions(+)
multipathd/main.c +0 -0
index 230c9d10..dd48be74 100644

multipathd/main.c reinstate_path (struct path * pp) · line 1944

Quoted 8 lines
multipathd/main.c +4 -0
@@ -1937,6 +1937,11 @@ reinstate_path (struct path * pp)
  	else {
  		condlog(2, "%s: reinstated", pp->dev_t);
  		update_queue_mode_add_path(pp->mpp);
+		if (strcmp(pp->prio.name, PRIO_SYSFS) == 0) {
+			condlog(2, "%s: rescan target to update
 priorities",
+				pp->dev_t);
+			rescan_path(pp->udev);

I can see why this is necessary with the sysfs target, but AFAICT rescanning the scsi device will end up calling scsi_execute_cmd to update the vpd pages, and this can block.  Obviously, we are doing this
after the checker has verified that the path is up, but even still, we're trying to cut down on the amount of code that can block multipathd
on an inaccessible device, instead of increase it.

Quoted 5 lines

Perhaps it would be better to make the prioritizer itself smarter, so that it could know when it needs to run the rescan(), and it could do that in a separate thread.

Thoughts?

A full rescan shouldn't be necessary. All that's needed is that the kernel issue another RTPG. AFAICS that should happen as soon as the target responds to any command with a sense key of UNIT ATTENTION with ASC=0x2a and ascq=6 or 7 (ALUA state change, ALUA state transition failed).

@Brian, does your storage unit not do this? If so, I suggest we disable the sysfs prioritizer for pure storage.

Otherwise, as far as multipathd is concerned, when a path is reinstated, it should be sufficient to send any IO command to trigger an RTPG. Or am I missing something here?

Martin

#3 lore raw
Quoted 84 lines · Martin Wilck
Quoted 68 lines · Benjamin Marzinski
Quoted 49 lines · Brian Bunker

When a path is lost and then reinstated later, the ALUA device handler will not pick up this change and continue to possibly provide incorrect (stale) information about its ALUA state to the 'sysfs' prioritizer if the path's priorities were changed prior to the loss of those paths.

On the loss of an I_T nexus, the path state should not continue to use its last known state. Many things and a lot of time could have passed between the path loss and its eventual reinstatemnt.

The ALUA device handler didn't have this issue since it always got the ALUA state from the target by sending an RTPG request on the path and updated the priority. However with the detect priority set to true by default, the 'sysfs' prioritzier will be used on targets that support ALUA rather than the 'alua' prioritizer.

Without re-evaluating the ALUA state when a path returns multipath is left with path states which do not reflect the actual ALUA states the target is providing.

3624a9370f0f545fc7c3e46a100011010 dm-2 PURE,FlashArray size=3.0T features='0' hwhandler='1 alua' wp=rw

Quoted 5 lines

-+- policy='service-time 0' prio=50 status=active

Quoted 3 lines
- 13:0:0:1 sdg 8:96  active ready running
- 12:0:0:1 sdf 8:80  active ready running
- 9:0:0:1  sdc 8:32  active ready running

`- 1:0:0:1 sdb 8:16 active ready running

`-+- policy='service-time 0' prio=10 status=enabled

  |- 14:0:0:1 sdh 8:112 active ready running
  |- 15:0:0:1 sdi 8:128 active ready running
  |- 10:0:0:1 sdd 8:48  active ready running

`- 11:0:0:1 sde 8:64 active ready running

 # sg_rtpg /dev/sdh (Active/Optimized)
target port group asymmetric access state : 0x00
---
 multipathd/main.c | 5 +++++
 1 file changed, 5 insertions(+)
multipathd/main.c +0 -0
index 230c9d10..dd48be74 100644

multipathd/main.c reinstate_path (struct path * pp) · line 1944

Quoted 8 lines
multipathd/main.c +4 -0
@@ -1937,6 +1937,11 @@ reinstate_path (struct path * pp)
  else {
  condlog(2, "%s: reinstated", pp->dev_t);
  update_queue_mode_add_path(pp->mpp);
+ if (strcmp(pp->prio.name, PRIO_SYSFS) == 0) {
+ condlog(2, "%s: rescan target to update
 priorities",
+ pp->dev_t);
+ rescan_path(pp->udev);

I can see why this is necessary with the sysfs target, but AFAICT rescanning the scsi device will end up calling scsi_execute_cmd to update the vpd pages, and this can block. Obviously, we are doing this
after the checker has verified that the path is up, but even still, we're trying to cut down on the amount of code that can block multipathd
on an inaccessible device, instead of increase it.

Quoted 5 lines

Perhaps it would be better to make the prioritizer itself smarter, so that it could know when it needs to run the rescan(), and it could do that in a separate thread.

Thoughts?

A full rescan shouldn't be necessary. All that's needed is that the kernel issue another RTPG. AFAICS that should happen as soon as the target responds to any command with a sense key of UNIT ATTENTION with ASC=0x2a and ascq=6 or 7 (ALUA state change, ALUA state transition failed).

Quoted 8 lines

@Brian, does your storage unit not do this? If so, I suggest we disable the sysfs prioritizer for pure storage.

Otherwise, as far as multipathd is concerned, when a path is reinstated, it should be sufficient to send any IO command to trigger an RTPG. Or am I missing something here?

Martin

Martin,

What I am gaining with the rescan is exactly that. You are correct the ALUA device handler the kernel has to send an RTPG to the target.

We do set a unit attention to have the initiator paths go into the ANO state before we reboot leading to the path loss, but we do not set a unit attention when the paths come back up. We have relied on the initiator’s polling to pick up the ALUA state change which they always have in the past and the ‘alua’ prioritizer still will. For us to add a unit attention would work, but there are couple of issues with that.

1. Unit attentions may not get back to the initiator. It is not guaranteed.
2. Paths could take a very long time to come back. We might not get these paths back for a very long time. Sometimes it is just a reboot. Other times it is a hardware replacement. It is possible for us to keep this state forever and post when when that I_T nexus returns but we haven’t had to.

If we did post the unit attention, everything works as expected. I have verified this, but I would also hope that the polling of the checkers would also unstick my stale ALUA state and we won’t have to.

I put this rescan_path inline to show the problem and the fix. I wasn’t sure the ‘right’ place to put it. I get that it would be better not to block on this. It should be possible to put this in a thread so that it does not. The other caller of rescan_path I guess is also doing the same thing when it is handling the wwid change.

Thanks,
Brian

#4 lore raw

Hi Brian,

Quoted 28 lines · Brian Bunker
Quoted 12 lines

A full rescan shouldn't be necessary. All that's needed is that the kernel issue another RTPG. AFAICS that should happen as soon as the target responds to any command with a sense key of UNIT ATTENTION with
ASC=0x2a and ascq=6 or 7 (ALUA state change, ALUA state transition failed).

@Brian, does your storage unit not do this? If so, I suggest we disable
the sysfs prioritizer for pure storage.

Quoted 6 lines

Otherwise, as far as multipathd is concerned, when a path is reinstated, it should be sufficient to send any IO command to trigger
an RTPG. Or am I missing something here?

Martin

Martin,

What I am gaining with the rescan is exactly that. You are correct the ALUA device handler the kernel has to send an RTPG to the target.

We do set a unit attention to have the initiator paths go into the ANO state before we reboot leading to the path loss, but we do not set a unit attention when the paths come back up.

Quoted 7 lines

We have relied on the initiator’s polling to pick up the ALUA state
change which they always have in the past and the ‘alua’ prioritizer still will. For us to add a unit attention would work, but there are couple of issues with that.

1. Unit attentions may not get back to the initiator. It is not guaranteed.

That's news for me. If that happens, wouldn't it mean that the initiator sees a timeout (no response) to some command, IOW that there's still something very wrong with this I_T nexus?

Quoted 5 lines

2. Paths could take a very long time to come back. We might not get these paths back for a very long time. Sometimes it is just a reboot. Other times it is a hardware replacement. It is possible for us to keep this state forever and post when when that I_T nexus returns but we haven’t had to.

No offense, that sounds somewhat lazy ;-) Note that it's also kind of dangerous. You are hiding the state change from the initiator. If the Linux kernel decided to use the access_state device attribute for anything else but feeding the sysfs attribute, things might go badly wrong.

Quoted 9 lines

If we did post the unit attention, everything works as expected. I have verified this, but I would also hope that the polling of the checkers would also unstick my stale ALUA state and we won’t have to.

I put this rescan_path inline to show the problem and the fix. I wasn’t sure the ‘right’ place to put it. I get that it would be better not to block on this. It should be possible to put this in a thread so that it does not. The other caller of rescan_path I guess is also doing the same thing when it is handling the wwid change.

That's true and not optimal, but wwid changes are rare events and an error condition in its own right. Patches converting moving rescan_path() into a thread would receive sympathetic reviews :-) The big benefit of the sysfs prioritizer is that it never blocks, without needing pthreads complexity.

Btw, I think you'd need to wait for the RTPG to finish after triggering the rescan, if you want to obtain the right priority (alua_rescan() -> alua_initialize() -> alua_check_vpd() will only queue an RTPG and not wait for the result).

Unfortunately, the kernel has no API for manually triggering an update of the access_state. I believe that would be useful elsewhere, too. We can consider adding it, but it won't help with current kernels.

IMO the best option for your storage arrays would is to force using the alua prioritizer rather than the sysfs one. You are not alone, we're doing this for RDAC already (see check_rdac() call in detect_prio()). This can be configured in multipath.conf right now by setting "detect_prio no" and "prio alua", and we can make it the default for your storage with a trivial patch for hwtable.c.

Regards
Martin

#5 lore raw
Quoted 40 lines · Martin Wilck

Hi Brian,

Quoted 33 lines · Brian Bunker
Quoted 12 lines

A full rescan shouldn't be necessary. All that's needed is that the kernel issue another RTPG. AFAICS that should happen as soon as the target responds to any command with a sense key of UNIT ATTENTION with
ASC=0x2a and ascq=6 or 7 (ALUA state change, ALUA state transition failed).

@Brian, does your storage unit not do this? If so, I suggest we disable
the sysfs prioritizer for pure storage.

Quoted 6 lines

Otherwise, as far as multipathd is concerned, when a path is reinstated, it should be sufficient to send any IO command to trigger
an RTPG. Or am I missing something here?

Martin

Martin,

What I am gaining with the rescan is exactly that. You are correct the ALUA device handler the kernel has to send an RTPG to the target.

We do set a unit attention to have the initiator paths go into the ANO state before we reboot leading to the path loss, but we do not set a unit attention when the paths come back up.

We have relied on the initiator’s polling to pick up the ALUA state change which they always have in the past and the ‘alua’ prioritizer still will. For us to add a unit attention would work, but there are couple of issues with that.

Quoted 6 lines
Quoted 2 lines

1. Unit attentions may not get back to the initiator. It is not guaranteed.

That's news for me. If that happens, wouldn't it mean that the initiator sees a timeout (no response) to some command, IOW that there's still something very wrong with this I_T nexus?

Probably. My point is just that any individual response could be lost wherever and there is no burden on the target to ensure the initiator got the unit attention.

Quoted 7 lines
Quoted 6 lines

2. Paths could take a very long time to come back. We might not get these paths back for a very long time. Sometimes it is just a reboot. Other times it is a hardware replacement. It is possible for us to keep this state forever and post when when that I_T nexus returns but we haven’t had to.

Quoted 5 lines

No offense, that sounds somewhat lazy ;-) Note that it's also kind of dangerous. You are hiding the state change from the initiator. If the Linux kernel decided to use the access_state device attribute for anything else but feeding the sysfs attribute, things might go badly wrong.

That is fair. We definitely could do better here. In general, that unit attention coming out of the preferred state didn’t buy us any speed. Those non-preferred paths weren’t serving I/O so the first I/O that would pick up the unit attention on those paths would be the path checker. The same run of the path checker picked up the new ALUA state. When going into the preferred state, there is read and write I/O which means those unit attentions are picked up very quickly and the ALUA state change is picked up in the kernel before the checker runs again.

Have you ever considered a checker of RTPG as opposed to TUR? That would seeming solve a lot of this too since you would be getting path state and priority in the same trip.

Quoted 17 lines
Quoted 10 lines

If we did post the unit attention, everything works as expected. I have verified this, but I would also hope that the polling of the checkers would also unstick my stale ALUA state and we won’t have to.

I put this rescan_path inline to show the problem and the fix. I wasn’t sure the ‘right’ place to put it. I get that it would be better not to block on this. It should be possible to put this in a thread so that it does not. The other caller of rescan_path I guess is also doing the same thing when it is handling the wwid change.

That's true and not optimal, but wwid changes are rare events and an error condition in its own right. Patches converting moving rescan_path() into a thread would receive sympathetic reviews :-) The big benefit of the sysfs prioritizer is that it never blocks, without needing pthreads complexity.

Quoted 4 lines

Btw, I think you'd need to wait for the RTPG to finish after triggering the rescan, if you want to obtain the right priority (alua_rescan() -> alua_initialize() -> alua_check_vpd() will only queue an RTPG and not wait for the result).

For our purposes we didn’t really need to wait for the rescan. As long as it happened. The next time the checker ran it would pick it up. These paths returning for us are redundant paths. We want them back as soon as possible but we have other paths that can serve I/O while waiting for the HA paths.

I can create a patch in the sysfs prioritizer to do the rescan_path in a thread that the checker and priority run doesn’t wait on. Would that be well received or I am better served by either posting a unit attention or just using detect_prio set to no and leaving the ’sysfs’ prioritizer alone?

Quoted 11 lines

Unfortunately, the kernel has no API for manually triggering an update of the access_state. I believe that would be useful elsewhere, too. We can consider adding it, but it won't help with current kernels.

IMO the best option for your storage arrays would is to force using the alua prioritizer rather than the sysfs one. You are not alone, we're doing this for RDAC already (see check_rdac() call in detect_prio()). This can be configured in multipath.conf right now by setting "detect_prio no" and "prio alua", and we can make it the default for your storage with a trivial patch for hwtable.c.

This is what we are doing now in our recommended configuration. I will probably add a patch for our hw table entry soon. It is a bit strange to me still that detect_prio would mean replace the one that I am explicitly stating in the device section. To me detect_prio would be if I didn’t provide one and wanted multipath to choose for me.

Quoted 4 lines

Regards
Martin

Thanks,
Brian

#6 lore raw

Hi Brian,

Quoted 54 lines · Brian Bunker
Quoted 54 lines · Martin Wilck

Hi Brian,

Quoted 46 lines · Brian Bunker
Quoted 16 lines

A full rescan shouldn't be necessary. All that's needed is that the
kernel issue another RTPG. AFAICS that should happen as soon as the
target responds to any command with a sense key of UNIT ATTENTION
with
ASC=0x2a and ascq=6 or 7 (ALUA state change, ALUA state transition
failed).

@Brian, does your storage unit not do this? If so, I suggest we disable
the sysfs prioritizer for pure storage.

Quoted 6 lines

Otherwise, as far as multipathd is concerned, when a path is reinstated, it should be sufficient to send any IO command to trigger
an RTPG. Or am I missing something here?

Martin

Martin,

What I am gaining with the rescan is exactly that. You are correct
the ALUA device handler the kernel has to send an RTPG to the target.

We do set a unit attention to have the initiator paths go into the
ANO state before we reboot leading to the path loss, but we do not
set a unit attention when the paths come back up.

We have relied on the initiator’s polling to pick up the ALUA state
change which they always have in the past and the ‘alua’ prioritizer
still will. For us to add a unit attention would work, but there are
couple of issues with that.

1. Unit attentions may not get back to the initiator. It is not guaranteed.

Quoted 6 lines
Quoted 3 lines

That's news for me. If that happens, wouldn't it mean that the initiator sees a timeout (no response) to some command, IOW that there's still something very wrong with this I_T nexus?

Probably. My point is just that any individual response could be lost wherever and there is no burden on the target to ensure the initiator got the unit attention.

Right. In the specific case of the ALUA-related UAs that we've been discussing, the storage *might* implement a logic to repeat responding with UA until an RTPG is received. But that may have unwanted side effects; I haven't thought it through.

Quoted 30 lines
Quoted 11 lines
Quoted 10 lines

2. Paths could take a very long time to come back. We might not get
these paths back for a very long time. Sometimes it is just a reboot.
Other times it is a hardware replacement. It is possible for us to
keep this state forever and post when when that I_T nexus returns but
we haven’t had to.

Quoted 8 lines

No offense, that sounds somewhat lazy ;-) Note that it's also kind of
dangerous. You are hiding the state change from the initiator. If the
Linux kernel decided to use the access_state device attribute for anything else but feeding the sysfs attribute, things might go badly
wrong.

That is fair. We definitely could do better here. In general, that unit
attention coming out of the preferred state didn’t buy us any speed. Those non-preferred paths weren’t serving I/O so the first I/O that would pick up the unit attention on those paths would be the path checker. The same run of the path checker picked up the new ALUA state. When going into the preferred state, there is read and write I/O which means those unit attentions are picked up very quickly and the ALUA state change is picked up in the kernel before the checker runs again.

Quoted 3 lines

Have you ever considered a checker of RTPG as opposed to TUR? That would seeming solve a lot of this too since you would be getting path state and priority in the same trip.

Interesting idea. AFAICT, noone has thought about it so far. In the past I've invested some thought in tieing checker and prioritizer more closely together. Unfortunately, the current multipathd architecture treats them as entirely separate, which makes it complicated to achieve this.

Quoted 38 lines
Quoted 22 lines
Quoted 7 lines

If we did post the unit attention, everything works as expected. I
have verified this, but I would also hope that the polling of the checkers would also unstick my stale ALUA state and we won’t have to.

Quoted 7 lines

I put this rescan_path inline to show the problem and the fix. I wasn’t sure the ‘right’ place to put it. I get that it would be better not to block on this. It should be possible to put this in a
thread so that it does not. The other caller of rescan_path I guess
is also doing the same thing when it is handling the wwid change.

That's true and not optimal, but wwid changes are rare events and an
error condition in its own right. Patches converting moving rescan_path() into a thread would receive sympathetic reviews :-) The big benefit of the sysfs prioritizer is that it never blocks, without needing pthreads complexity.

Quoted 7 lines

Btw, I think you'd need to wait for the RTPG to finish after triggering
the rescan, if you want to obtain the right priority (alua_rescan() ->
alua_initialize() -> alua_check_vpd() will only queue an RTPG and not
wait for the result).

For our purposes we didn’t really need to wait for the rescan. As long as
it happened. The next time the checker ran it would pick it up. These paths
returning for us are redundant paths. We want them back as soon as possible
but we have other paths that can serve I/O while waiting for the HA paths.

Quoted 7 lines

I can create a patch in the sysfs prioritizer to do the rescan_path in a thread
that the checker and priority run doesn’t wait on. Would that be well received
or I am better served by either posting a unit attention or just using detect_prio
set to no and leaving the ’sysfs’ prioritizer alone?

I am not sure. Like I said, the beauty of the sysfs prioritizer is that it doesn't do any IO. rescanning a device from this prioritizer basically voids this benefit. You might as well just use alua. As I said, we do it for RDAC as well.

But if you want to invest more effort, feel free to submit patches ;-) You can do this any time on top of the hwtable change.

Quoted 16 lines
Quoted 6 lines

Unfortunately, the kernel has no API for manually triggering an update
of the access_state. I believe that would be useful elsewhere, too. We
can consider adding it, but it won't help with current kernels.

Quoted 10 lines

IMO the best option for your storage arrays would is to force using the
alua prioritizer rather than the sysfs one. You are not alone, we're
doing this for RDAC already (see check_rdac() call in detect_prio()).
This can be configured in multipath.conf right now by setting "detect_prio no" and "prio alua", and we can make it the default for
your storage with a trivial patch for hwtable.c.

Quoted 8 lines

This is what we are doing now in our recommended configuration. I will
probably add a patch for our hw table entry soon. It is a bit strange to
me still that detect_prio would mean replace the one that I am explicitly
stating in the device section. To me detect_prio would be if I didn’t provide one and wanted multipath to choose for me.

True. The design of the "detect_prio" option is awkward and confusing. It happens all the time that people set "prio" and wonder why the setting isn't applied. Or worse, they think it is applied but it's not. I think the original idea was to use ALUA whenever supported, even if historically the hwtable contained something else for a given storage. It has been this way for a long time, I don't think we can change the semantics easily.

Regards
Martin

Quoted 7 lines
Quoted 4 lines

Regards
Martin

Thanks,
Brian

#7 lore raw
Quoted 10 lines · Martin Wilck

Hi Brian,

Quoted 8 lines

This is what we are doing now in our recommended configuration. I will
probably add a patch for our hw table entry soon. It is a bit strange to
me still that detect_prio would mean replace the one that I am explicitly
stating in the device section. To me detect_prio would be if I didn’t provide one and wanted multipath to choose for me.

Quoted 8 lines

True. The design of the "detect_prio" option is awkward and confusing. It happens all the time that people set "prio" and wonder why the setting isn't applied. Or worse, they think it is applied but it's not. I think the original idea was to use ALUA whenever supported, even if historically the hwtable contained something else for a given storage. It has been this way for a long time, I don't think we can change the semantics easily.

It's so that you can select the failback prioritizer if multipath can't autodetect one. This made more sense when detect_prio defaulted to "no", so autodetection was only used by devices that specifically enabled it. Now it just is what it is.

-Ben

#8 lore raw
In reply to Martin Wilck
Quoted 106 lines · Martin Wilck

Hi Brian,

Quoted 54 lines · Brian Bunker
Quoted 54 lines · Martin Wilck

Hi Brian,

Quoted 46 lines · Brian Bunker
Quoted 16 lines

A full rescan shouldn't be necessary. All that's needed is that the
kernel issue another RTPG. AFAICS that should happen as soon as the
target responds to any command with a sense key of UNIT ATTENTION
with
ASC=0x2a and ascq=6 or 7 (ALUA state change, ALUA state transition
failed).

@Brian, does your storage unit not do this? If so, I suggest we disable
the sysfs prioritizer for pure storage.

Quoted 6 lines

Otherwise, as far as multipathd is concerned, when a path is reinstated, it should be sufficient to send any IO command to trigger
an RTPG. Or am I missing something here?

Martin

Martin,

What I am gaining with the rescan is exactly that. You are correct
the ALUA device handler the kernel has to send an RTPG to the target.

We do set a unit attention to have the initiator paths go into the
ANO state before we reboot leading to the path loss, but we do not
set a unit attention when the paths come back up.

We have relied on the initiator’s polling to pick up the ALUA state
change which they always have in the past and the ‘alua’ prioritizer
still will. For us to add a unit attention would work, but there are
couple of issues with that.

1. Unit attentions may not get back to the initiator. It is not guaranteed.

Quoted 6 lines
Quoted 3 lines

That's news for me. If that happens, wouldn't it mean that the initiator sees a timeout (no response) to some command, IOW that there's still something very wrong with this I_T nexus?

Probably. My point is just that any individual response could be lost wherever and there is no burden on the target to ensure the initiator got the unit attention.

Right. In the specific case of the ALUA-related UAs that we've been discussing, the storage *might* implement a logic to repeat responding with UA until an RTPG is received. But that may have unwanted side effects; I haven't thought it through.

Quoted 30 lines
Quoted 11 lines
Quoted 10 lines

2. Paths could take a very long time to come back. We might not get
these paths back for a very long time. Sometimes it is just a reboot.
Other times it is a hardware replacement. It is possible for us to
keep this state forever and post when when that I_T nexus returns but
we haven’t had to.

Quoted 8 lines

No offense, that sounds somewhat lazy ;-) Note that it's also kind of
dangerous. You are hiding the state change from the initiator. If the
Linux kernel decided to use the access_state device attribute for anything else but feeding the sysfs attribute, things might go badly
wrong.

That is fair. We definitely could do better here. In general, that unit
attention coming out of the preferred state didn’t buy us any speed. Those non-preferred paths weren’t serving I/O so the first I/O that would pick up the unit attention on those paths would be the path checker. The same run of the path checker picked up the new ALUA state. When going into the preferred state, there is read and write I/O which means those unit attentions are picked up very quickly and the ALUA state change is picked up in the kernel before the checker runs again.

Quoted 3 lines

Have you ever considered a checker of RTPG as opposed to TUR? That would seeming solve a lot of this too since you would be getting path state and priority in the same trip.

Quoted 5 lines

Interesting idea. AFAICT, noone has thought about it so far. In the past I've invested some thought in tieing checker and prioritizer more closely together. Unfortunately, the current multipathd architecture treats them as entirely separate, which makes it complicated to achieve This.

Yeah. It would be a re-do of the separation between checkers and prioritizers and might leave you with a mess if you had to keep everything functioning the same way wasn’t ALUA. I think it might make check_path much cleaner though.

Quoted 46 lines
Quoted 38 lines
Quoted 22 lines
Quoted 7 lines

If we did post the unit attention, everything works as expected. I
have verified this, but I would also hope that the polling of the checkers would also unstick my stale ALUA state and we won’t have to.

Quoted 7 lines

I put this rescan_path inline to show the problem and the fix. I wasn’t sure the ‘right’ place to put it. I get that it would be better not to block on this. It should be possible to put this in a
thread so that it does not. The other caller of rescan_path I guess
is also doing the same thing when it is handling the wwid change.

That's true and not optimal, but wwid changes are rare events and an
error condition in its own right. Patches converting moving rescan_path() into a thread would receive sympathetic reviews :-) The big benefit of the sysfs prioritizer is that it never blocks, without needing pthreads complexity.

Quoted 7 lines

Btw, I think you'd need to wait for the RTPG to finish after triggering
the rescan, if you want to obtain the right priority (alua_rescan() ->
alua_initialize() -> alua_check_vpd() will only queue an RTPG and not
wait for the result).

For our purposes we didn’t really need to wait for the rescan. As long as
it happened. The next time the checker ran it would pick it up. These paths
returning for us are redundant paths. We want them back as soon as possible
but we have other paths that can serve I/O while waiting for the HA paths.

Quoted 7 lines

I can create a patch in the sysfs prioritizer to do the rescan_path in a thread
that the checker and priority run doesn’t wait on. Would that be well received
or I am better served by either posting a unit attention or just using detect_prio
set to no and leaving the ’sysfs’ prioritizer alone?

Quoted 7 lines

I am not sure. Like I said, the beauty of the sysfs prioritizer is that it doesn't do any IO. rescanning a device from this prioritizer basically voids this benefit. You might as well just use alua. As I said, we do it for RDAC as well.

But if you want to invest more effort, feel free to submit patches ;-) You can do this any time on top of the hwtable change.

OK I will submit the hwtable change first since there is no controversy there at all.

My change only affects the reinstate path so it is isn’t like ‘alua’ where priority is evaluated with RTPG at each checker instance. The rescan should be rare since I_T nexus loss shouldn’t be too common.

Quoted 26 lines
Quoted 17 lines
Quoted 7 lines

Unfortunately, the kernel has no API for manually triggering an update
of the access_state. I believe that would be useful elsewhere, too. We
can consider adding it, but it won't help with current kernels.

Quoted 10 lines

IMO the best option for your storage arrays would is to force using the
alua prioritizer rather than the sysfs one. You are not alone, we're
doing this for RDAC already (see check_rdac() call in detect_prio()).
This can be configured in multipath.conf right now by setting "detect_prio no" and "prio alua", and we can make it the default for
your storage with a trivial patch for hwtable.c.

Quoted 8 lines

This is what we are doing now in our recommended configuration. I will
probably add a patch for our hw table entry soon. It is a bit strange to
me still that detect_prio would mean replace the one that I am explicitly
stating in the device section. To me detect_prio would be if I didn’t provide one and wanted multipath to choose for me.

Quoted 7 lines

True. The design of the "detect_prio" option is awkward and confusing. It happens all the time that people set "prio" and wonder why the setting isn't applied. Or worse, they think it is applied but it's not. I think the original idea was to use ALUA whenever supported, even if historically the hwtable contained something else for a given storage. It has been this way for a long time, I don't think we can change the semantics easily.

I saw Ben’s later comment explaining the history of how it became the way it became.

Quoted 3 lines

Regards
Martin

Thanks,
Brian

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

Regards
Martin

Thanks,
Brian