Skip to contentSkip to conversation

scsi_transport_sas: make minimum and maximum linkrate settable quantities

PATCH Douglas Gilbert 2 participants 2 messages Sep 7, 2006 SCSI

lore
Share & tools
lore
#0 lore raw

I meant to cc this to the list ...

Quoted 20 lines · James Bottomley
Quoted 7 lines · Douglas Gilbert
Quoted 1 line · James Bottomley

According to SPEC, the minimum_linkrate and maximum_linkrate should be

^ ^
programmable programmable

Quoted 2 lines

settable by the user. This patch introduces a callback that allows the sas class to pass these settings on to the driver.

There are also "hardware" variants of min/max_linkrate.

hardware rates represent the max and min capabilities of the hw ... they're not settable.

Quoted 7 lines

Hmmm. I get a bit confused reading the above. Are you talking about a SAS initiator that this machine controls (i.e. a "host"), an expander phy in a visible SAS domain, a visible SSP target (e.g. a SAS disk) and/or a visible SATA device? All of them? Will all levels (i.e. initiator, expander and targets) have these fields (if they exist) represented by an attribute in sysfs?

Quoted 3 lines

I mean for all of the phys that are visible in the sysfs representation of the domain. Non expander end device phys aren't visible in our representation.

I'm still confused. An attached phy either identifies itself as an expander phy or an end device phy. So what is the difference between a "non expander end device phy" and a "end device phy"? [There can be SAS (virtual) phys within an expander (e.g. associated with an enclosure service target) but each SAS virtual phy is paired with an expander virtual phy.]

Dropping the double negative I read your last sentence as "Only expander phys are visible in our representation."

Quoted 3 lines

However, this is irrelevant really, since it takes two phys to make a link, so if you force one side to a specific rate, the other must follow or fail the link negotiation.

True, as long as a phy at one end of the link allows its programmable linkrates to be changed and a link or hard reset to be instigated.

Quoted 17 lines
Quoted 4 lines

The SATA device is easy to rule out (as is a SATA port selector). There may be some way of doing linkrate changes via the SATA command set.

Quoted 12 lines

SSP targets (e.g. SAS disks) have the "phy control and discover" mode subpage. The SAS disks that I have examined (all from one manufacturer) do not allow the programmable min/max_linkrate fields to be changed by the user (i.e. the changeable mode page bits are all zero). Worse still, their product manuals say that both ports (phys) will negotiate the same linkrate which will be the linkrate first attained. Looking at the spec (sas2r05b section 10.2.7.2.3), it just says all _other_ fields in that mode subpage shall not be changeable. So your "should be settable" statement above is a bit strong. How about "might be settable for SSP targets".

Quoted 3 lines

The same one end follows the other should pertain even to SSP targets. I can't validate this, though, since I don't have any SATA-2 disks (SATA-1 can't negotiate speed, being fixed at 1.5GB).

I can connect a SATA disk that claims to do 3 Gbps (Seagate 7200.9 series) directly to a LSIL HBA phy and I can see:

# cat /sys/class/sas_phy/phy-4:0/negotiated_linkrate
3.0 Gbit

My 48300 (aic94xx based) HBA says in its BIOS that it negotiates with all SATA disks at 1.5 Gbps. I believe the SASx12 expander I have has the same restriction.

<snip>

Quoted 14 lines
Quoted 5 lines

Also folks playing with these fields should realize that link reset is the appropriate type of reset to issue to cause a linkrate change. A hard reset will do it as well but may cause additional collateral damage. For example, if any phy on a SAS target device does a hard reset

Quoted 8 lines

sequence, then all LUs within that target device receive a LU reset: clearing out all SCSI commands in progress and those pending. The idea with link reset is that only the link layers and below are impacted, as longer of the renegotiation succeeds in a timely fashion. Setting the prog_max_linkrate to 1.5 Gbps at one end of physical link while setting the prog_min_linkrate to 3 Gbps at the other end will cause the renegotiation following a link reset to fail.

Quoted 2 lines

Yes, the setting algorithms do a link reset, if you look at the code ... and there's logic to prevent silly things like min going above max.

However the user may not want a link reset sent immediately. For example, in SAS-2 with 6 Gbps available, to force 3 Gbps the min_linkrate needs to be increased to 3 Gbps and the max_linkrate needs to be decreased to 3 Gbps, then a reset sent. It is also possible that the user may want a (expander) phy's next negotiated link_rate changed but the phy left disabled.

Building SAS discovery into the kernel SAS transport layer might be a necessity now but it could cause problems down the track. Consideration should be given to having a switch to turn it off. SAS discovery in SAS-1 and SAS-1.1 does not scale well with every SAS management client (i.e. the SAS kernel transport layer) thinking that it has to do a full or partial discovery of a potentially large SAS domain every time a BROADCAST(CHANGE) event is received. [BTW a link reset causes a BROADCAST(CHANGE) event so sending more link resets than necessary will impact performance.]

In SAS-2 there are several improvements:
- higher level SMP commands that move more data in one function (e.g. DISCOVER LIST)
- much smarter expanders that can
- self config and configure other (older) expanders
- indicate to management clients when the are finished
- more efficient routing tables (expander based rather than phy based)
- support expander based zoning which will control the accessibility of end devices to one another.

Together these changes could relieve the burden of the SAS discovery process on the SAS kernel transport layer. However there seems nothing to stop such a kernel layer ploughing ahead and doing discovery in the old inefficient way making the worst case assumptions.

As ever finding the boot disk and root file system could be a problem, but hopefully folks will soon be using flash cards for that!

Doug Gilbert

#1 lore raw
Quoted 2 lines · Douglas Gilbert

Dropping the double negative I read your last sentence as "Only expander phys are visible in our representation."

Only expander and host phys are visible in the sysfs representation.

Quoted 7 lines

However the user may not want a link reset sent immediately. For example, in SAS-2 with 6 Gbps available, to force 3 Gbps the min_linkrate needs to be increased to 3 Gbps and the max_linkrate needs to be decreased to 3 Gbps, then a reset sent. It is also possible that the user may want a (expander) phy's next negotiated link_rate changed but the phy left disabled.

A link reset is merely supposed to renegotiate ... it's not supposed to be disruptive

Quoted 7 lines

Building SAS discovery into the kernel SAS transport layer might be a necessity now but it could cause problems down the track. Consideration should be given to having a switch to turn it off. SAS discovery in SAS-1 and SAS-1.1 does not scale well with every SAS management client (i.e. the SAS kernel transport layer) thinking that it has to do a full or partial discovery of a potentially large SAS domain

We already have this, but it's optional: aic94xx uses it, mptsas doesn't.

Quoted 3 lines

every time a BROADCAST(CHANGE) event is received. [BTW a link reset causes a BROADCAST(CHANGE) event so sending more link resets than necessary will impact performance.]

Yes, these are actually used to update the phy parameters (they trigger SMP probes to find out what changed).

Quoted 11 lines

In SAS-2 there are several improvements:
- higher level SMP commands that move more data in one function (e.g. DISCOVER LIST)
- much smarter expanders that can
- self config and configure other (older) expanders
- indicate to management clients when the are finished
- more efficient routing tables (expander based rather than phy based)
- support expander based zoning which will control the accessibility of end devices to one another.

Quoted 5 lines

Together these changes could relieve the burden of the SAS discovery process on the SAS kernel transport layer. However there seems nothing to stop such a kernel layer ploughing ahead and doing discovery in the old inefficient way making the worst case assumptions.

I'm not really planning to do anything about the new SAS-2 features until I see expanders which support them. The current discovery doesn't reprogram an expander, it queries the route table to see if it's usable as is and only modifies it if it isn't. It can also support table to table routing, so it should support a large chunk of the new features anyway.

James