scsi_transport_sas: make minimum and maximum linkrate settable quantities
I meant to cc this to the list ...
Quoted 20 lines · James Bottomley
Quoted 7 lines · Douglas Gilbert
Quoted 1 line · James Bottomley
According to SPEC, the
minimum_linkrateandmaximum_linkrateshould be^ ^
programmable programmableQuoted 2 lines
settable by the user. This patch introduces a callback that allows the sas class to pass these settings on to the driver.
There are also "hardware" variants of min/max_linkrate.
hardware rates represent the max and min capabilities of the hw ... they're not settable.
Quoted 7 lines
Hmmm. I get a bit confused reading the above. Are you talking about a SAS initiator that this machine controls (i.e. a "host"), an expander phy in a visible SAS domain, a visible SSP target (e.g. a SAS disk) and/or a visible SATA device? All of them? Will all levels (i.e. initiator, expander and targets) have these fields (if they exist) represented by an attribute in sysfs?
Quoted 3 lines
I mean for all of the phys that are visible in the sysfs representation of the domain. Non expander end device phys aren't visible in our representation.
I'm still confused. An attached phy either identifies itself as an expander phy or an end device phy. So what is the difference between a "non expander end device phy" and a "end device phy"? [There can be SAS (virtual) phys within an expander (e.g. associated with an enclosure service target) but each SAS virtual phy is paired with an expander virtual phy.]
Dropping the double negative I read your last sentence as "Only expander phys are visible in our representation."
Quoted 3 lines
However, this is irrelevant really, since it takes two phys to make a link, so if you force one side to a specific rate, the other must follow or fail the link negotiation.
True, as long as a phy at one end of the link allows its programmable linkrates to be changed and a link or hard reset to be instigated.
Quoted 17 lines
Quoted 4 lines
The SATA device is easy to rule out (as is a SATA port selector). There may be some way of doing linkrate changes via the SATA command set.
Quoted 12 lines
SSP targets (e.g. SAS disks) have the "phy control and discover" mode subpage. The SAS disks that I have examined (all from one manufacturer) do not allow the programmable min/max_linkrate fields to be changed by the user (i.e. the changeable mode page bits are all zero). Worse still, their product manuals say that both ports (phys) will negotiate the same linkrate which will be the linkrate first attained. Looking at the spec (sas2r05b section 10.2.7.2.3), it just says all _other_ fields in that mode subpage shall not be changeable. So your "should be settable" statement above is a bit strong. How about "might be settable for SSP targets".
Quoted 3 lines
The same one end follows the other should pertain even to SSP targets. I can't validate this, though, since I don't have any SATA-2 disks (SATA-1 can't negotiate speed, being fixed at 1.5GB).
I can connect a SATA disk that claims to do 3 Gbps (Seagate 7200.9 series) directly to a LSIL HBA phy and I can see:
# cat /sys/class/sas_phy/phy-4:0/negotiated_linkrate 3.0 Gbit
My 48300 (aic94xx based) HBA says in its BIOS that it negotiates with all SATA disks at 1.5 Gbps. I believe the SASx12 expander I have has the same restriction.
<snip>
Quoted 14 lines
Quoted 5 lines
Also folks playing with these fields should realize that link reset is the appropriate type of reset to issue to cause a linkrate change. A hard reset will do it as well but may cause additional collateral damage. For example, if any phy on a SAS target device does a hard reset
Quoted 8 lines
sequence, then all LUs within that target device receive a LU reset: clearing out all SCSI commands in progress and those pending. The idea with link reset is that only the link layers and below are impacted, as longer of the renegotiation succeeds in a timely fashion. Setting the
prog_max_linkrateto 1.5 Gbps at one end of physical link while setting theprog_min_linkrateto 3 Gbps at the other end will cause the renegotiation following a link reset to fail.
Quoted 2 lines
Yes, the setting algorithms do a link reset, if you look at the code ... and there's logic to prevent silly things like min going above max.
However the user may not want a link reset sent immediately. For example, in SAS-2 with 6 Gbps available, to force 3 Gbps the min_linkrate needs to be increased to 3 Gbps and the max_linkrate needs to be decreased to 3 Gbps, then a reset sent. It is also possible that the user may want a (expander) phy's next negotiated link_rate changed but the phy left disabled.
Building SAS discovery into the kernel SAS transport layer might be a necessity now but it could cause problems down the track. Consideration should be given to having a switch to turn it off. SAS discovery in SAS-1 and SAS-1.1 does not scale well with every SAS management client (i.e. the SAS kernel transport layer) thinking that it has to do a full or partial discovery of a potentially large SAS domain every time a BROADCAST(CHANGE) event is received. [BTW a link reset causes a BROADCAST(CHANGE) event so sending more link resets than necessary will impact performance.]
In SAS-2 there are several improvements:
- higher level SMP commands that move more data in one function (e.g. DISCOVER LIST)
- much smarter expanders that can
- self config and configure other (older) expanders
- indicate to management clients when the are finished
- more efficient routing tables (expander based rather than phy based)
- support expander based zoning which will control the accessibility of end devices to one another.
Together these changes could relieve the burden of the SAS discovery process on the SAS kernel transport layer. However there seems nothing to stop such a kernel layer ploughing ahead and doing discovery in the old inefficient way making the worst case assumptions.
As ever finding the boot disk and root file system could be a problem, but hopefully folks will soon be using flash cards for that!
Doug Gilbert
Quoted 2 lines · Douglas Gilbert
Dropping the double negative I read your last sentence as "Only expander phys are visible in our representation."
Only expander and host phys are visible in the sysfs representation.
Quoted 7 lines
However the user may not want a link reset sent immediately. For example, in SAS-2 with 6 Gbps available, to force 3 Gbps the
min_linkrateneeds to be increased to 3 Gbps and themax_linkrateneeds to be decreased to 3 Gbps, then a reset sent. It is also possible that the user may want a (expander) phy's next negotiatedlink_ratechanged but the phy left disabled.
A link reset is merely supposed to renegotiate ... it's not supposed to be disruptive
Quoted 7 lines
Building SAS discovery into the kernel SAS transport layer might be a necessity now but it could cause problems down the track. Consideration should be given to having a switch to turn it off. SAS discovery in SAS-1 and SAS-1.1 does not scale well with every SAS management client (i.e. the SAS kernel transport layer) thinking that it has to do a full or partial discovery of a potentially large SAS domain
We already have this, but it's optional: aic94xx uses it, mptsas doesn't.
Quoted 3 lines
every time a BROADCAST(CHANGE) event is received. [BTW a link reset causes a BROADCAST(CHANGE) event so sending more link resets than necessary will impact performance.]
Yes, these are actually used to update the phy parameters (they trigger SMP probes to find out what changed).
Quoted 11 lines
In SAS-2 there are several improvements:
- higher level SMP commands that move more data in one function (e.g. DISCOVER LIST)
- much smarter expanders that can
- self config and configure other (older) expanders
- indicate to management clients when the are finished
- more efficient routing tables (expander based rather than phy based)
- support expander based zoning which will control the accessibility of end devices to one another.
Quoted 5 lines
Together these changes could relieve the burden of the SAS discovery process on the SAS kernel transport layer. However there seems nothing to stop such a kernel layer ploughing ahead and doing discovery in the old inefficient way making the worst case assumptions.
I'm not really planning to do anything about the new SAS-2 features until I see expanders which support them. The current discovery doesn't reprogram an expander, it queries the route table to see if it's usable as is and only modifies it if it isn't. It can also support table to table routing, so it should support a large chunk of the new features anyway.
James