opensm configuration question

DISCUSSION Tom Ammon 2 participants 4 messages Dec 14, 2010 – Dec 15, 2010 3 messages not in the archive InfiniBand

lore

Share & tools

lore

Tom Ammon

author

2010-12-14 22:42 UTC #0 lore raw

Hi,

I've got an opensm question.

With opensm running, I run opensm --help, and it tells me what I am expecting from the configuration I have given it. I configured it for updn routing with /etc/opensm/rootguids.conf as the root guids file:

-------------------------------------------------
OpenSM 3.3.7
Reading Cached Option File: /etc/opensm/opensm.conf Loading Cached Option:guid = 0x0002c9030009714d Loading Cached Option:routing_engine = updn
Loading Cached Option:root_guid_file = /etc/opensm/rootguids.conf Loading Cached Option:sm_priority = 15
Loading Cached Option:log_flags = 4
Loading Cached Option:console = local
Command Line Arguments:

However, when I do an ibtracert between two nodes on different leaf switches, it appears to be using minhop routing (the root guids I specified are not in the path):

[root@sm2 ~]# ibtracert 24 69
From ca {0x0002c903000bb5d8} portnum 1 lid 24-24 "em001 HCA-1"
[1] -> switch port {0x0008f10500201d8c}[1] lid 15-15 "Voltaire 4036 # komas-is-a1-1"
[36] -> switch port {0x0008f105006515ca}[19] lid 9-9 "Voltaire sLB-4018 Line 8 Chip 1 4200 #komas-is-a6-1"
[20] -> switch port {0x0008f10500201b32}[36] lid 18-18 "Voltaire 4036 # komas-is-a2-1"
[28] -> ca port {0x0002c903000bb5f9}[1] lid 69-69 "em056 HCA-1"
To ca {0x0002c903000bb5f8} portnum 1 lid 69-69 "em056 HCA-1"

Looking in the opensm log file, I see this over and over:

OpenSM $ OpenSM $ OpenSM $ Dec : Command not found

Supported commands and syntax:
help [<command>]
quit (not valid in local mode; use ctl-c)
loglevel [<log-level>]
priority [<sm-priority>]
resweep [heavy|light]
reroute
sweep [on|off]
status [loop]
logflush -- flush the opensm.log file
querylid lid -- print internal information about the lid specified
portstatus [ca|switch|router]
switchbalance [verbose] [guid]
lidbalance [switchguid]
dump_conf
update_desc
version -- print the OSM version
dump_portguid [file filename] regexp1 [regexp2 [regexp3 ...]] -- Dump port GUID matching a regexp OpenSM $

Any ideas?

Also, how can I use the console? I noticed a couple of config options for it, but I don't understand how to actually attach to the console to try things out.

Thanks,

Tom

Signature · 12 lines

Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

Ira Weiny

2010-12-15 00:12 UTC #2 lore raw

Quoted 21 lines · On Tue, 14 Dec 2010 14:42:32 -0800 Tom Ammon…

Hi,
I've got an opensm question.
With opensm running, I run opensm --help, and it tells me what I am expecting from the configuration I have given it. I configured it for updn routing with /etc/opensm/rootguids.conf as the root guids file:
-------------------------------------------------
OpenSM 3.3.7
Reading Cached Option File: /etc/opensm/opensm.conf Loading Cached Option:guid = 0x0002c9030009714d Loading Cached Option:routing_engine = updn
Loading Cached Option:root_guid_file = /etc/opensm/rootguids.conf Loading Cached Option:sm_priority = 15
Loading Cached Option:log_flags = 4
Loading Cached Option:console = local
Command Line Arguments:

Quoted 3 lines

However, when I do an ibtracert between two nodes on different leaf switches, it appears to be using minhop routing (the root guids I specified are not in the path):

Up/Down does not mean that everything will be routed through your spine switches. Up/Down only requires that your path can't go "up" after starting "down". From the trace below it appears that the path from em001 to em056 only required 3 hops through the line board. A node farther away might require routing through the spine board.

Here's an example:

     S
    / \
  L1   L2
 / \  / \
E1  E2  E3
|    |   |
N1  N2   N3

Min Hop == Up/Down when routing from N1 to N2 (3 hops).

But when routing N1 to N3 Min Hop can choose either E1,L1,S,L2,E3 OR E1,L1,E2,L2,E3 (both 5 hops) but Up/Down can only chose the former through the Spine.

Quoted 38 lines

[root@sm2 ~]# ibtracert 24 69
From ca {0x0002c903000bb5d8} portnum 1 lid 24-24 "em001 HCA-1"
[1] -> switch port {0x0008f10500201d8c}[1] lid 15-15 "Voltaire 4036 # komas-is-a1-1"
[36] -> switch port {0x0008f105006515ca}[19] lid 9-9 "Voltaire sLB-4018
    Line 8  Chip 1 4200 #komas-is-a6-1"
[20] -> switch port {0x0008f10500201b32}[36] lid 18-18 "Voltaire 4036 # komas-is-a2-1"
[28] -> ca port {0x0002c903000bb5f9}[1] lid 69-69 "em056 HCA-1" To ca {0x0002c903000bb5f8} portnum 1 lid 69-69 "em056 HCA-1"
Looking in the opensm log file, I see this over and over:
OpenSM $ OpenSM $ OpenSM $ Dec : Command not found
Supported commands and syntax:
help [<command>]
quit (not valid in local mode; use ctl-c)
loglevel [<log-level>]
priority [<sm-priority>]
resweep [heavy|light]
reroute
sweep [on|off]
status [loop]
logflush -- flush the opensm.log file
querylid lid -- print internal information about the lid specified portstatus [ca|switch|router]
switchbalance [verbose] [guid]
lidbalance [switchguid]
dump_conf
update_desc
version -- print the OSM version
dump_portguid [file filename] regexp1 [regexp2 [regexp3 ...]] -- Dump port GUID matching a regexp
OpenSM $

Quoted 1 line

Any ideas?

In the log file??? Not sure.

Quoted 4 lines

Also, how can I use the console? I noticed a couple of config options for it, but I don't understand how to actually attach to the console to try things out.

What options do you have set for the console?

16:08:54 > opensm -h | grep console
--console, -q [off|local|socket|loopback]
This option activates the OpenSM console (default off).
--console-port, -C <port>
Specify an alternate telnet port for the console (default 10000).

Ira

Quoted 20 lines

Thanks,

Tom

Signature · 12 lines

Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 9 lines

Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

Tom Ammon

author

2010-12-15 16:12 UTC #4 lore raw

Ira,

Thanks! I misunderstood up/down, thanks for clearing that up. That helps a lot.

I set the console to local, and then tried telnetting to localhost port 10000 but got a connection refused. Should I set it to socket instead?

Tom

Quoted 131 lines · Ira Weiny

Quoted 21 lines · On Tue, 14 Dec 2010 14:42:32 -0800 Tom Ammon<[email protected]…
Hi,
I've got an opensm question.
With opensm running, I run opensm --help, and it tells me what I am expecting from the configuration I have given it. I configured it for updn routing with /etc/opensm/rootguids.conf as the root guids file:
-------------------------------------------------
OpenSM 3.3.7
Reading Cached Option File: /etc/opensm/opensm.conf Loading Cached Option:guid = 0x0002c9030009714d Loading Cached Option:routing_engine = updn
Loading Cached Option:root_guid_file = /etc/opensm/rootguids.conf Loading Cached Option:sm_priority = 15
Loading Cached Option:log_flags = 4
Loading Cached Option:console = local
Command Line Arguments:
Quoted 4 lines
However, when I do an ibtracert between two nodes on different leaf switches, it appears to be using minhop routing (the root guids I specified are not in the path):
Up/Down does not mean that everything will be routed through your spine switches. Up/Down only requires that your path can't go "up" after starting "down". From the trace below it appears that the path from em001 to em056 only required 3 hops through the line board. A node farther away might require routing through the spine board.
Here's an example:
      S
     / \
   L1   L2
  / \  / \
E1  E2  E3
|    |   |
N1  N2   N3
Min Hop == Up/Down when routing from N1 to N2 (3 hops).
But when routing N1 to N3 Min Hop can choose either E1,L1,S,L2,E3 OR E1,L1,E2,L2,E3 (both 5 hops) but Up/Down can only chose the former through the Spine.
Quoted 37 lines
[root@sm2 ~]# ibtracert 24 69
From ca {0x0002c903000bb5d8} portnum 1 lid 24-24 "em001 HCA-1"
[1] -> switch port {0x0008f10500201d8c}[1] lid 15-15 "Voltaire 4036 # komas-is-a1-1"
[36] -> switch port {0x0008f105006515ca}[19] lid 9-9 "Voltaire sLB-4018
     Line 8  Chip 1 4200 #komas-is-a6-1"
[20] -> switch port {0x0008f10500201b32}[36] lid 18-18 "Voltaire 4036 # komas-is-a2-1"
[28] -> ca port {0x0002c903000bb5f9}[1] lid 69-69 "em056 HCA-1" To ca {0x0002c903000bb5f8} portnum 1 lid 69-69 "em056 HCA-1"
Looking in the opensm log file, I see this over and over:
OpenSM $ OpenSM $ OpenSM $ Dec : Command not found
Supported commands and syntax:
help [<command>]
quit (not valid in local mode; use ctl-c)
loglevel [<log-level>]
priority [<sm-priority>]
resweep [heavy|light]
reroute
sweep [on|off]
status [loop]
logflush -- flush the opensm.log file
querylid lid -- print internal information about the lid specified portstatus [ca|switch|router]
switchbalance [verbose] [guid]
lidbalance [switchguid]
dump_conf
update_desc
version -- print the OSM version
dump_portguid [file filename] regexp1 [regexp2 [regexp3 ...]] -- Dump port GUID matching a regexp
OpenSM $
Quoted 2 lines
Any ideas?
In the log file??? Not sure.
Quoted 4 lines
Also, how can I use the console? I noticed a couple of config options for it, but I don't understand how to actually attach to the console to try things out.
What options do you have set for the console?
16:08:54> opensm -h | grep console
--console, -q [off|local|socket|loopback]
This option activates the OpenSM console (default off).
--console-port, -C<port>
Specify an alternate telnet port for the console (default 10000).
Ira
Quoted 20 lines
Thanks,
Tom
Signature · 12 lines
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 14 lines

--------------------------------------------------------------------
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

Ira Weiny

2010-12-15 16:53 UTC #6 lore raw

Quoted 7 lines · On Wed, 15 Dec 2010 08:12:46 -0800 Tom Ammon…

Ira,
Thanks! I misunderstood up/down, thanks for clearing that up. That helps a lot.
I set the console to local, and then tried telnetting to localhost port 10000 but got a connection refused. Should I set it to socket instead?

loopback is best as it should limit access to the local node. You can also set your hosts.[allow|deny] to limit access.

Ira

Quoted 151 lines

Tom
Quoted 131 lines · Ira Weiny
Quoted 21 lines · On Tue, 14 Dec 2010 14:42:32 -0800 Tom Ammon<[email protected]…
Hi,
I've got an opensm question.
With opensm running, I run opensm --help, and it tells me what I am expecting from the configuration I have given it. I configured it for updn routing with /etc/opensm/rootguids.conf as the root guids file:
-------------------------------------------------
OpenSM 3.3.7
Reading Cached Option File: /etc/opensm/opensm.conf Loading Cached Option:guid = 0x0002c9030009714d Loading Cached Option:routing_engine = updn
Loading Cached Option:root_guid_file = /etc/opensm/rootguids.conf Loading Cached Option:sm_priority = 15
Loading Cached Option:log_flags = 4
Loading Cached Option:console = local
Command Line Arguments:
Quoted 4 lines
However, when I do an ibtracert between two nodes on different leaf switches, it appears to be using minhop routing (the root guids I specified are not in the path):
Up/Down does not mean that everything will be routed through your spine switches. Up/Down only requires that your path can't go "up" after starting "down". From the trace below it appears that the path from em001 to em056 only required 3 hops through the line board. A node farther away might require routing through the spine board.
Here's an example:
      S
     / \
   L1   L2
  / \  / \
E1  E2  E3
|    |   |
N1  N2   N3
Min Hop == Up/Down when routing from N1 to N2 (3 hops).
But when routing N1 to N3 Min Hop can choose either E1,L1,S,L2,E3 OR E1,L1,E2,L2,E3 (both 5 hops) but Up/Down can only chose the former through the Spine.
Quoted 37 lines
[root@sm2 ~]# ibtracert 24 69
From ca {0x0002c903000bb5d8} portnum 1 lid 24-24 "em001 HCA-1"
[1] -> switch port {0x0008f10500201d8c}[1] lid 15-15 "Voltaire 4036 # komas-is-a1-1"
[36] -> switch port {0x0008f105006515ca}[19] lid 9-9 "Voltaire sLB-4018
     Line 8  Chip 1 4200 #komas-is-a6-1"
[20] -> switch port {0x0008f10500201b32}[36] lid 18-18 "Voltaire 4036 # komas-is-a2-1"
[28] -> ca port {0x0002c903000bb5f9}[1] lid 69-69 "em056 HCA-1" To ca {0x0002c903000bb5f8} portnum 1 lid 69-69 "em056 HCA-1"
Looking in the opensm log file, I see this over and over:
OpenSM $ OpenSM $ OpenSM $ Dec : Command not found
Supported commands and syntax:
help [<command>]
quit (not valid in local mode; use ctl-c)
loglevel [<log-level>]
priority [<sm-priority>]
resweep [heavy|light]
reroute
sweep [on|off]
status [loop]
logflush -- flush the opensm.log file
querylid lid -- print internal information about the lid specified portstatus [ca|switch|router]
switchbalance [verbose] [guid]
lidbalance [switchguid]
dump_conf
update_desc
version -- print the OSM version
dump_portguid [file filename] regexp1 [regexp2 [regexp3 ...]] -- Dump port GUID matching a regexp
OpenSM $
Quoted 2 lines
Any ideas?
In the log file??? Not sure.
Quoted 4 lines
Also, how can I use the console? I noticed a couple of config options for it, but I don't understand how to actually attach to the console to try things out.
What options do you have set for the console?
16:08:54> opensm -h | grep console
--console, -q [off|local|socket|loopback]
This option activates the OpenSM console (default off).
--console-port, -C<port>
Specify an alternate telnet port for the console (default 10000).
Ira
Quoted 20 lines
Thanks,
Tom
Signature · 12 lines
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 14 lines
--------------------------------------------------------------------
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 9 lines

Ira Weiny
Math Programmer/Computer Scientist
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html