multipath-tools: document why dev_loss_tmo is set to infinity for HPE 3PAR
PATCH 1/1 multipath-tools: document why dev_loss_tmo is set to infinity for HPE 3PAR
It's needed by Peer Persistence, documented in SLES and RHEL guides:
https://support.hpe.com/hpsc/doc/public/display?docId=a00053835
https://support.hpe.com/hpsc/doc/public/display?docId=c04448818
--- libmultipath/hwtable.c | 1 + 1 file changed, 1 insertion(+)
libmultipath/hwtable.c +1 -0
@@ -116,6 +116,7 @@ static struct hwentry default_hw[] = { .prio_name = PRIO_ALUA, .no_path_retry = 18, .fast_io_fail = 10, + /* infinity is needed by Peer Persistence */ .dev_loss = MAX_DEV_LOSS_TMO, }, {
Signature · 1 line
2.19.2
On thing that seems to be a mess with the tmo value that is being inherited from the underlying driver, is that the setting for the scsi layer is significantly different from what multipath calls TMO.
In the case I have seen with the lpfc driver this is often set fairly low (HPE's doc references 14 seconds, and this is similar to what my employer is using).
parm: lpfc_devloss_tmo:Seconds driver will hold I/O waiting
for a device to come back (int)
But setting this on the scsi layer causes it to quickly return an error to the multipath layer. It does not mean that the scsi layer removes the device from the system, just that it returns an error so
that the layer above it can deal with it. The multipath layer
interprets its value of TMO as when to clean up/remove the underlying
path that when dev_loss_tmo is hit. TMO is used in both names, but
they are not the same usage and meaning and the scsi layer's TMO should not be inherited by the multipath layer, as they don't appear
to actually be the same thing. In multipath it should probably be
called remove_fault_paths or something similar.
This incorrect inheritance has caused issues, as prior to multipath inheriting TMO from the scsi layer, multipath did not remove the paths
when IO failed for TMO time. The paths prior to the inheritance
stayed around and errored until the underlying issue was fixed, or a reboot happened, or until someone manually removed the failing paths. When I first saw this I had processes to deal with this, and we did noticed when it stated automatically cleaning up paths and it was good since it eliminated manual work, that is until it caused issues during firmware update. HPE's update to infinity will be a response to the inherited TMO change causing issues.
Quoted 31 lines · Xose Vazquez Perez
It's needed by Peer Persistence, documented in SLES and RHEL guides:
https://support.hpe.com/hpsc/doc/public/display?docId=a00053835
https://support.hpe.com/hpsc/doc/public/display?docId=c04448818Cc Christophe Varoqui <[email protected]>Cc DM-DEVEL ML <[email protected]>Signed-off-by Xose Vazquez Perez <[email protected]>--- libmultipath/hwtable.c | 1 + 1 file changed, 1 insertion(+)libmultipath/hwtable.c +1
-1@@ -116,6 +116,7 @@ static struct hwentry default_hw[] = { .prio_name = PRIO_ALUA, .no_path_retry = 18, .fast_io_fail = 10, + /* infinity is needed by Peer Persistence */ .dev_loss = MAX_DEV_LOSS_TMO, }, { --2.19.2
--
dm-devel mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/dm-devel
Quoted 11 lines · Roger Heflin
On thing that seems to be a mess with the tmo value that is being inherited from the underlying driver, is that the setting for the scsi
layer is significantly different from what multipath calls TMO.In the case I have seen with the lpfc driver this is often set fairly low (HPE's doc references 14 seconds, and this is similar to what my employer is using).
parm: lpfc_devloss_tmo:Seconds driver will hold I/O waiting
for a device to come back (int)
Quoted 4 lines
But setting this on the scsi layer causes it to quickly return an error to the multipath layer. It does not mean that the scsi layer removes the device from the system, just that it returns an error so that the layer above it can deal with it.
You are confusing fast_io_fail_tmo and dev_loss_tmo. What you just described is fast_io_fail_tmo. If dev_loss_tmo expires, the SCSI layer does indeed remove the SCSI target. See comments on the fc_remote_port_delete() function.
(https://elixir.bootlin.com/linux/latest/source/drivers/scsi/scsi_transport_fc.c#L2906)
For multipath, what really matters is fast_io_fail_tmo. dev_loss_tmo only matters if fast_io_fail_tmo is unset. fast_io_fail is preferred, because path failure/reinstantiation is much easier to handle than path removal/re-addition, on both kernel and user space level. The reason dev_loss_tmo is not infinity by default is twofold: 1) if fast_io_fail is not used and dev_loss_tmo is infinity, IOs might block on a removed device forever; 2) even with fast_io_fail, if a lost device doesn't come back after a long time, it might be good not to carry it around forever - chances are that the storage admin really removed the device or changed the zoning.
Quoted 7 lines
The multipath layer
interprets its value of TMO as when to clean up/remove the underlying
path that whendev_loss_tmois hit. TMO is used in both names, but
they are not the same usage and meaning and the scsi layer's TMO should not be inherited by the multipath layer, as they don't appear
to actually be the same thing. In multipath it should probably be
calledremove_fault_pathsor something similar.
I'm not sure what you mean with "multipath layer". The kernel dm- multipath layer has nothing to do with dev_loss_tmo at all. multipath- tools don't "inherit" this value, either. They *set* it to match the settings from multipath.conf and the internal hwtable, taking other related settings into account (in particular, no_path_retry).
Quoted 4 lines
This incorrect inheritance has caused issues, as prior to multipath inheriting TMO from the scsi layer, multipath did not remove the paths
when IO failed for TMO time.
Sorry, no. multipathd *never* removes SCSI paths. If it receives an event about removal of a path, it updates its own data structures, and the maps in the dm-multipath layer. That's it.
The only thing that multipath-tools do that may cause SCSI devices to get removed is to set dev_loss_tmo to a low value. But that would be a matter of (unusual) configuration.
Quoted 10 lines
The paths prior to the inheritance
stayed around and errored until the underlying issue was fixed, or a reboot happened, or until someone manually removed the failing paths. When I first saw this I had processes to deal with this, and we did noticed when it stated automatically cleaning up paths and it was good
since it eliminated manual work, that is until it caused issues during
firmware update. HPE's update to infinity will be a response to the inherited TMO change causing issues.
I'm wondering what you're talking about. dev_loss_tmo has been in the SCSI layer for ages.
Regards
Martin
Quoted 5 lines
You are confusing
fast_io_fail_tmoanddev_loss_tmo. What you just described isfast_io_fail_tmo. Ifdev_loss_tmoexpires, the SCSI layer does indeed remove the SCSI target. See comments on thefc_remote_port_delete()function.
(https://elixir.bootlin.com/linux/latest/source/drivers/scsi/scsi_transport_fc.c#L2906)
the lpfc driver lets one set dev_loss_tmo and the description on the parameter seems like it should be fast_io_fail_tmo rather that dev_loss_tmo, from how it is working it appears to be used to setdev_loss_tmo in the scsi layer. And the lpfc driver does not have a
setting for the fast_io_fail_tmo and that would seem to be what is actually needed/wanted. The reason for setting it was we have had fc fabric failures that did not result in an error being return to multipath, such that multipath could not failover to the other working paths.
Quoted 11 lines
For multipath, what really matters is
fast_io_fail_tmo.dev_loss_tmoonly matters iffast_io_fail_tmois unset.fast_io_failis preferred, because path failure/reinstantiation is much easier to handle than path removal/re-addition, on both kernel and user space level. The reasondev_loss_tmois not infinity by default is twofold: 1) iffast_io_failis not used anddev_loss_tmois infinity, IOs might block on a removed device forever; 2) even withfast_io_fail, if a lost device doesn't come back after a long time, it might be good not to carry it around forever - chances are that the storage admin really removed the device or changed the zoning.
We are thinking of setting dev_loss_tmo to 86400 (24 hours) as that is a happy medium, and leaves the paths around during reasonable events, but results in a clean-up at 24 hours.
Quoted 9 lines
Quoted 8 lines
The multipath layer
interprets its value of TMO as when to clean up/remove the underlying
path that whendev_loss_tmois hit. TMO is used in both names, but
they are not the same usage and meaning and the scsi layer's TMO should not be inherited by the multipath layer, as they don't appear
to actually be the same thing. In multipath it should probably be
calledremove_fault_pathsor something similar.
Quoted 5 lines
I'm not sure what you mean with "multipath layer". The kernel dm- multipath layer has nothing to do with
dev_loss_tmoat all. multipath- tools don't "inherit" this value, either. They *set* it to match the settings from multipath.conf and the internal hwtable, taking other related settings into account (in particular,no_path_retry).
ok.
Quoted 22 lines
Quoted 5 lines
This incorrect inheritance has caused issues, as prior to multipath inheriting TMO from the scsi layer, multipath did not remove the paths
when IO failed for TMO time.Sorry, no. multipathd *never* removes SCSI paths. If it receives an event about removal of a path, it updates its own data structures, and the maps in the dm-multipath layer. That's it.
Quoted 10 lines
The paths prior to the inheritance
stayed around and errored until the underlying issue was fixed, or a reboot happened, or until someone manually removed the failing paths. When I first saw this I had processes to deal with this, and we did noticed when it stated automatically cleaning up paths and it was good
since it eliminated manual work, that is until it caused issues during
firmware update. HPE's update to infinity will be a response to the inherited TMO change causing issues.
Quoted 2 lines
I'm wondering what you're talking about.
dev_loss_tmohas been in the SCSI layer for ages.
Do you have an idea how many years ago the dev_loss_tmo started
actually removing the device? I am guessing when that was backported
into rhel was what I saw it start, but iI don't know exactly when it was backported.
Prior to that we had processes to evaluate why a given path was erroring and either fix it or clean it up, so the change was fairly easy for us to see, and maybe when that change went in the lpfc driver should have started setting the fast_io_fail_tmo rather than the tmo_dev_loss in the scsi layer as fast_io_fail_tmo is closer to what the described option does on the lpfc driver.
thanks, I think this helps my understand how to tune things.
Quoted 10 lines · Roger Heflin
Quoted 6 lines
You are confusing
fast_io_fail_tmoanddev_loss_tmo. What you just described isfast_io_fail_tmo. Ifdev_loss_tmoexpires, the SCSI layer
does indeed remove the SCSI target. See comments on thefc_remote_port_delete()function.
(the lpfc driver lets one set
dev_loss_tmoand the description on the parameter seems like it should befast_io_fail_tmorather thatdev_loss_tmo, from how it is working it appears to be used to setdev_loss_tmoin the scsi layer.
On my system, the docstring of lpfc.devloss_tmo says "Seconds driver will hold I/O waiting for a device to come back". Which is basically true, although it does not say that when the waiting is over, the device node is removed.
Quoted 3 lines
And the lpfc driver does not have a
setting for thefast_io_fail_tmoand that would seem to be what is actually needed/wanted.
That is set via the generic scsi_transport_fc layer. Normally you do it with multipath-tools, as the parameter is only useful in multipath scenarios.
Quoted 5 lines
The reason for setting it was we have had fc
fabric failures that did not result in an error being return to multipath, such that multipath could not failover to the other working
paths.
You should have been setting fast_io_fail_tmo in multipath.conf.
Quoted 19 lines
Quoted 10 lines
For multipath, what really matters is
fast_io_fail_tmo.dev_loss_tmo
only matters iffast_io_fail_tmois unset.fast_io_failis preferred,
because path failure/reinstantiation is much easier to handle than path
removal/re-addition, on both kernel and user space level. The reasondev_loss_tmois not infinity by default is twofold: 1) iffast_io_failQuoted 8 lines
is not used and
dev_loss_tmois infinity, IOs might block on a removed
device forever; 2) even withfast_io_fail, if a lost device doesn't come back after a long time, it might be good not to carry it around
forever - chances are that the storage admin really removed the device
or changed the zoning.
Quoted 4 lines
We are thinking of setting
dev_loss_tmoto 86400 (24 hours) as that is
a happy medium, and leaves the paths around during reasonable events, but results in a clean-up at 24 hours.
That sounds reasonable. But that's a matter of policy, which differs vastly between different installations and administrator preferences. The point I'm trying to make is: it doesn't make a lot of sense to tie this setting to the storage hardware properties, as multipath currently does. It's really much more a matter of data center administration. That's different for fast_io_fail_tmo - it makes sense to relate this timeout to hardware properties, e.g. the time it takes to do failover or failback.
IMO, in a way, the different dev_loss_tmo settings in multipath's hardware table reflect the different vendor's ideas of how the storage should be administrated rather than the actual properties of the hardware.
Quoted 10 lines
Quoted 4 lines
I'm wondering what you're talking about.
dev_loss_tmohas been in the
SCSI layer for ages.Do you have an idea how many years ago the
dev_loss_tmostarted
actually removing the device? I am guessing when that was
backported
into rhel was what I saw it start, but iI don't know exactly when it was backported.
I can see it in 2.6.12 (2005):
https://elixir.bootlin.com/linux/v2.6.12/source/drivers/scsi/scsi_transport_fc.c#L1549
You need to understand that, when time starts ticking towards the dev_loss_tmo, the FC remote port port is *already gone*. On the transport layer, there's nothing to "remove" any more. The kernel just keeps the SCSI layer structures and waits if the device comes back, as it would for temporary failures such as network outage or an operator having pulled the wrong cable.
Regards
Martin
Quoted 13 lines · Martin Wilck
Quoted 10 lines · Roger Heflin
Quoted 6 lines
You are confusing
fast_io_fail_tmoanddev_loss_tmo. What you just described isfast_io_fail_tmo. Ifdev_loss_tmoexpires, the SCSI layer
does indeed remove the SCSI target. See comments on thefc_remote_port_delete()function.
(the lpfc driver lets one set
dev_loss_tmoand the description on the parameter seems like it should befast_io_fail_tmorather thatdev_loss_tmo, from how it is working it appears to be used to setdev_loss_tmoin the scsi layer.
Quoted 4 lines
On my system, the docstring of lpfc.devloss_tmo says "Seconds driver will hold I/O waiting for a device to come back". Which is basically true, although it does not say that when the waiting is over, the device node is removed.
In older version the device was not removed, so there was a behavior change.
Quoted 9 lines
Quoted 4 lines
And the lpfc driver does not have a
setting for thefast_io_fail_tmoand that would seem to be what is actually needed/wanted.That is set via the generic
scsi_transport_fclayer. Normally you do it with multipath-tools, as the parameter is only useful in multipath scenarios.
Quoted 7 lines
Quoted 5 lines
The reason for setting it was we have had fc
fabric failures that did not result in an error being return to multipath, such that multipath could not failover to the other working
paths.You should have been setting
fast_io_fail_tmoin multipath.conf.
When we started setting the lpfc parameter, multipath did not yet manage fast_io_fail_tmo nor dev_loss_tmo so it was not an option.
Quoted 14 lines
Quoted 4 lines
We are thinking of setting
dev_loss_tmoto 86400 (24 hours) as that is
a happy medium, and leaves the paths around during reasonable events, but results in a clean-up at 24 hours.That sounds reasonable. But that's a matter of policy, which differs vastly between different installations and administrator preferences. The point I'm trying to make is: it doesn't make a lot of sense to tie this setting to the storage hardware properties, as multipath currently does. It's really much more a matter of data center administration. That's different for
fast_io_fail_tmo- it makes sense to relate this timeout to hardware properties, e.g. the time it takes to do failover or failback.
Quoted 4 lines
IMO, in a way, the different
dev_loss_tmosettings in multipath's hardware table reflect the different vendor's ideas of how the storage should be administrated rather than the actual properties of the hardware.
And that is kind of the conclusion we were coming to, it is a preference based on datacenter admin.
Quoted 8 lines
Quoted 7 lines
Do you have an idea how many years ago the
dev_loss_tmostarted
actually removing the device? I am guessing when that was
backported
into rhel was what I saw it start, but iI don't know exactly when it was backported.
Quoted 4 lines
I can see it in 2.6.12 (2005):
https://elixir.bootlin.com/linux/v2.6.12/source/drivers/scsi/scsi_transport_fc.c#L1549
In rhel5 (2.6.18+) it did not actually delete the device I believe until around 5.8, so all of the magic may not have quite been working yet.
Quoted 7 lines
You need to understand that, when time starts ticking towards the
dev_loss_tmo, the FC remote port port is *already gone*. On the transport layer, there's nothing to "remove" any more. The kernel just keeps the SCSI layer structures and waits if the device comes back, as it would for temporary failures such as network outage or an operator having pulled the wrong cable.
We understand that. The issues seem to be that once the device is deleted the process that brings the devices back as a path when the rport/cable is fixed is not reliable (it fails say 1 in 100 events and causes issues) when the cable is far enough removed from the host (ie
the array port connect to a fc switch). And when routed fc storage
is involved everything gets much less reliable all around. It always works if the cable issue being fixed is the cable to the host, but
once it gets far enough away there seem to be an issue sometimes. So
the hope is if the device is still there and still being probed my multipath that there will be less reliance on the imperfect fc magic, it may or may not help anything, and we may have to upgrade something and actively scan and resolve path issues manually before upgrading the next component that will take out another path.