Default force_link_speed value in opensm configuration

Hi,

Is there a reason for setting default force_link_speed to 15 (set enabled as supported) and not to 0 (don't modify PortInfo:LinkSpeedEnabled )?

Thanks,
Eli
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://*vger.kernel.org/majordomo-info.html

Quoted 4 lines · On Tue, 06 Oct 2009 17:56:01 +0200 "Eli Dorfman (Voltaire)"…

Hi,

Is there a reason for setting default force_link_speed to 15 (set enabled as supported) and not to 0 (don't modify PortInfo:LinkSpeedEnabled )?

The thought was to enable all ports to the maximum supported by default. This was the behavior before this option was added so it was though to leave it alone by default.

Is there a reason this should not happen? I guess since the Volume 2 spec says that the power on default for this is LinkSpeedSupported there is no reason to have the SM send these Set's. Perhaps there is broken hardware out there?

Ira

Quoted 3 lines

Thanks,
Eli
--

Quoted 12 lines · Ira Weiny

Quoted 4 lines · On Tue, 06 Oct 2009 17:56:01 +0200 "Eli Dorfman (Voltaire)"…

Hi,

Is there a reason for setting default force_link_speed to 15 (set enabled as supported) and not to 0 (don't modify PortInfo:LinkSpeedEnabled )?

The thought was to enable all ports to the maximum supported by default. This was the behavior before this option was added so it was though to leave it alone by default.

Quoted 4 lines

Is there a reason this should not happen? I guess since the Volume 2 spec says that the power on default for this is LinkSpeedSupported there is no reason to have the SM send these Set's. Perhaps there is broken hardware out there?

well actually there is a hardware that supports only 5.0Gbps (but reports 2.5 or 5.0 Gbps since there is no option for 5.0 only) and powers on with enabled speed of 5.0Gbps only.
Setting the enabled speed to the supported speed will fail further negotiation (after link reset) since the device does not support 2.5Gbps.

So if the default for all other devices is enabled=supported then we can change the opensm configuration force_link_speed to 0.

Quoted 8 lines · On Mon, 12 Oct 2009 11:47:37 +0200 "Eli Dorfman (Voltaire)"…

Quoted 7 lines · Ira Weiny

Quoted 4 lines · On Tue, 06 Oct 2009 17:56:01 +0200 "Eli Dorfman (Voltaire)"…

Hi,

Is there a reason for setting default force_link_speed to 15 (set enabled as supported) and not to 0 (don't modify PortInfo:LinkSpeedEnabled )?

Quoted 10 lines

Quoted 9 lines

The thought was to enable all ports to the maximum supported by default. This was the behavior before this option was added so it was though to leave it alone by default.

Is there a reason this should not happen? I guess since the Volume 2 spec says that the power on default for this is LinkSpeedSupported there is no reason to have the SM send these Set's. Perhaps there is broken hardware out there?

Quoted 3 lines

well actually there is a hardware that supports only 5.0Gbps (but reports 2.5 or 5.0 Gbps since there is no option for 5.0 only) and powers on with enabled speed of 5.0Gbps only.
Setting the enabled speed to the supported speed will fail further negotiation (after link reset) since the device does not support 2.5Gbps.

That seems like broken hardware...

Quoted 3 lines

So if the default for all other devices is enabled=supported then we can change the opensm configuration force_link_speed to 0.

I have no problem changing it. As I said if hardware is compliant I think it might speed things up. I will test out a patch with the hardware I can test.

However, I would like to know if anyone else knows of hardware which needs it set to "15". If so, this is going to be an interesting problem... ;-)

Ira

--
Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 11 lines · Ira Weiny

Quoted 8 lines · On Mon, 12 Oct 2009 11:47:37 +0200 "Eli Dorfman (Voltaire)"…

Quoted 7 lines · Ira Weiny

Quoted 4 lines · On Tue, 06 Oct 2009 17:56:01 +0200 "Eli Dorfman (Voltaire)"…

Hi,

Is there a reason for setting default force_link_speed to 15 (set enabled as supported) and not to 0 (don't modify PortInfo:LinkSpeedEnabled )?

This is going back some 2 years to 7/15/07 commit ID c7cbf11992b111d2b9c7347e398e67f2b5806edb

commit c7cbf11992b111d2b9c7347e398e67f2b5806edb

Author: Hal Rosenstock <halr-smomgflXvOZWk0Htik3J/[email protected]>
Date:   Sun Jul 15 06:43:20 2007 -0400

OpenSM: Change force_link_speed to allow for local policy and more flexibili

OpenSM: Change force_link_speed to allow for local policy and more flexibility

Extend (and change) the use of force_link_speed as follows:
0 - no change
1 - set to SDR
15 - set as supported (default)
(Non zero values are used to set LinkSpeedEnabled component in PortInfo)

Note that force_link_speed 0 which used to force SDR is now force_link_speed 1

"Ideally", there were be a per port configuration of this.

[Note this is largely untested.]

Signed-off-by Hal Rosenstock <[email protected]>

I think that the default was chosen based on IBA 1.2.1 vol 2 p. 108 stating:
"The default value following power on shall be set to LinkSpeedSupported." which should not be an issue with compliant hardware.

Quoted 3 lines

However, I would like to know if anyone else knows of hardware which needs it set to "15". If so, this is going to be an interesting problem... ;-)

I'm not sure about need but I think there is a tradeoff here in changing this. The pro is it will work with non compliant hardware. The con is that it may result in less than the max LSS being used in some scenarios.

Should the default be changed for non compliant hardware or should this need to be configured in subnets including such hardware as it has since this was introduced ?

Quoted 34 lines · On Tue, 13 Oct 2009 09:39:08 -0400 Hal Rosenstock…

Quoted 11 lines · Ira Weiny

Quoted 8 lines · On Mon, 12 Oct 2009 11:47:37 +0200 "Eli Dorfman (Voltaire)"…

Quoted 7 lines · Ira Weiny

Quoted 4 lines · On Tue, 06 Oct 2009 17:56:01 +0200 "Eli Dorfman (Voltaire)"…

Hi,

Is there a reason for setting default force_link_speed to 15 (set enabled as supported) and not to 0 (don't modify PortInfo:LinkSpeedEnabled )?

This is going back some 2 years to 7/15/07 commit ID c7cbf11992b111d2b9c7347e398e67f2b5806edb

commit c7cbf11992b111d2b9c7347e398e67f2b5806edb
Author: Hal Rosenstock <halr-smomgflXvOZWk0Htik3J/[email protected]>
Date:   Sun Jul 15 06:43:20 2007 -0400

    OpenSM: Change force_link_speed to allow for local policy and more flexibili

    OpenSM: Change force_link_speed to allow for local policy and more flexibility

    Extend (and change) the use of force_link_speed as follows:
    0 - no change
    1 - set to SDR
    15 - set as supported (default)
    (Non zero values are used to set LinkSpeedEnabled component in PortInfo)

    Note that force_link_speed 0 which used to force SDR is now force_link_speed 1

Quoted 1 line

    "Ideally", there were be a per port configuration of this.

BTW I have a patch which allows a user to except some nodes from this restriction.  I don't know that anyone but us is using this "feature" so I did not submit the patch.  If someone else would like some finer granularity I can clean up the patch and make it available.

Quoted 3 lines

    [Note this is largely untested.]

BTW, now, this is highly tested, to force SDR, on some of our clusters! Thanks!

Quoted 7 lines

Signed-off-by Hal Rosenstock <[email protected]>

I think that the default was chosen based on IBA 1.2.1 vol 2 p. 108 stating:
"The default value following power on shall be set to LinkSpeedSupported." which should not be an issue with compliant hardware.

[snip]

Quoted 9 lines

Quoted 3 lines

However, I would like to know if anyone else knows of hardware which needs it set to "15".  If so, this is going to be an interesting problem...  ;-)

I'm not sure about need but I think there is a tradeoff here in changing this. The pro is it will work with non compliant hardware. The con is that it may result in less than the max LSS being used in some scenarios.

Quoted 4 lines

Should the default be changed for non compliant hardware or should this need to be configured in subnets including such hardware as it has since this was introduced ?

I think that is a good way of thinking about things.

Ira