basic questions about partitions

Hi,

I have some basic questions about IB partitions.

Can an HCA port belong to more than 1 partition at a time?

How do you configure partitions with opensm? From reading the opensmd man page, it looks like you just create a file called /etc/osm-partitions.conf, with port GUIDs and such, but is this current?

I ask because according to the man page the opensm configuration file is in /etc/opensm/ . Can you tell opensm where to look for the partitions file?

Thanks,

Tom

--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi Tom,

Quoted 5 lines · Tom Ammon

  Hi,

I have some basic questions about IB partitions.

Can an HCA port belong to more than 1 partition at a time?

Yes.

Quoted 3 lines

How do you configure partitions with opensm? From reading the opensmd man page, it looks like you just create a file called /etc/osm-partitions.conf, with port GUIDs and such, but is this current?

The default location depends on how OpenSM is configured/built.

Quoted 3 lines

I ask because according to the man page the opensm configuration file is in /etc/opensm/ . Can you tell opensm where to look for the partitions file?

Yes, with either the -P option on the command line or partition_config_file line in the options file.

-- Hal

Quoted 19 lines

Thanks,

Tom

--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at  http://vger.kernel.org/majordomo-info.html

Quoted 7 lines · Hal Rosenstock

Hi Tom,

Quoted 4 lines · Tom Ammon

   Hi,

I have some basic questions about IB partitions.

Quoted 7 lines

Quoted 1 line

Can an HCA port belong to more than 1 partition at a time?

Yes.

Quoted 3 lines

How do you configure partitions with opensm? From reading the opensmd man page, it looks like you just create a file called /etc/osm-partitions.conf, with port GUIDs and such, but is this current?

The default location depends on how OpenSM is configured/built.

I missed this the first time around:

and yes, the syntax indicated is current.

-- Hal

Quoted 26 lines

Quoted 2 lines

I ask because according to the man page the opensm configuration file is in /etc/opensm/ . Can you tell opensm where to look for the partitions file?

Yes, with either the -P option on the command line or partition_config_file line in the options file.

-- Hal

Quoted 19 lines

Thanks,

Tom

--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

 Okay, how about this.

With ethernet and a linux server, I can set up up aliases (virtual IP addresses) and assign multiple IPs to a single NIC. Can this be done with an HCA and IPoIB?

For example, could I configure ib0:1 to be 192.168.0.1/24 and ib0:2 to be 10.0.0.1/24?

I have tried this on a RHEL5 box and it doesn't seem to want to accept it, but I don't know if there is another piece somewhere that I need to configure.

Tom

Quoted 20 lines · Hal Rosenstock

Quoted 17 lines · Hal Rosenstock

Hi Tom,

Quoted 6 lines · Tom Ammon

  Hi,

I have some basic questions about IB partitions.

Can an HCA port belong to more than 1 partition at a time?

Yes.

Quoted 3 lines

How do you configure partitions with opensm? From reading the opensmd man page, it looks like you just create a file called /etc/osm-partitions.conf, with port GUIDs and such, but is this current?

Quoted 36 lines

Quoted 1 line

The default location depends on how OpenSM is configured/built.

I missed this the first time around:

and yes, the syntax indicated is current.

-- Hal

Quoted 28 lines

Quoted 3 lines

I ask because according to the man page the opensm configuration file is in /etc/opensm/ . Can you tell opensm where to look for the partitions file?

Yes, with either the -P option on the command line or partition_config_file line in the options file.

-- Hal

Quoted 19 lines

Thanks,

Tom

--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

The procedure is a little different. You'll need to create a child interface (on a partition) first and then you will be able to configure it as follows:

echo 0x8001> /sys/class/net/ib0/create_child
ifconfig ib0.8001 ...

Note that you'll want the 0x8000 bit on for full membership.

-- Hal

Quoted 63 lines

Tom

Quoted 15 lines · Hal Rosenstock

Quoted 13 lines · Hal Rosenstock

Hi Tom,

Quoted 5 lines · Tom Ammon

   Hi,

I have some basic questions about IB partitions.

Can an HCA port belong to more than 1 partition at a time?

Yes.

Quoted 3 lines

How do you configure partitions with opensm? From reading the opensmd man page, it looks like you just create a file called /etc/osm-partitions.conf, with port GUIDs and such, but is this current?

Quoted 34 lines

Quoted 1 line

The default location depends on how OpenSM is configured/built.

I missed this the first time around:

and yes, the syntax indicated is current.

-- Hal

Quoted 27 lines

Quoted 3 lines

I ask because according to the man page the opensm configuration file is in /etc/opensm/ . Can you tell opensm where to look for the partitions file?

Yes, with either the -P option on the command line or partition_config_file line in the options file.

-- Hal

Quoted 19 lines

Thanks,

Tom

--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

 Hal,

Quoted 4 lines

The procedure is a little different. You'll need to create a child interface (on a partition) first and then you will be able to configure it as follows:

Quoted 4 lines

echo 0x8001>  /sys/class/net/ib0/create_child
ifconfig ib0.8001 ...

Note that you'll want the 0x8000 bit on for full membership.

What does "full membership" mean?

And are you saying that for full
membership, any value beginning with 0x8 would work?

Can an HCA have full membership on multiple partitions simultaneously?

Thanks,

Tom

Quoted 75 lines

-- Hal

Quoted 73 lines

Tom

Quoted 21 lines · Hal Rosenstock

Quoted 18 lines · Hal Rosenstock

Hi Tom,

Quoted 6 lines · Tom Ammon

  Hi,

I have some basic questions about IB partitions.

Can an HCA port belong to more than 1 partition at a time?

Yes.

Quoted 4 lines

How do you configure partitions with opensm? From reading the opensmd man page, it looks like you just create a file called /etc/osm-partitions.conf, with port GUIDs and such, but is this current?

Quoted 38 lines

Quoted 1 line

The default location depends on how OpenSM is configured/built.

I missed this the first time around:

and yes, the syntax indicated is current.

-- Hal

Quoted 30 lines

Quoted 4 lines

I ask because according to the man page the opensm configuration file is
in /etc/opensm/ . Can you tell opensm where to look for the partitions file?

Yes, with either the -P option on the command line or partition_config_file line in the options file.

-- Hal

Quoted 20 lines

Thanks,

Tom

--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected] More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

Tom,

Quoted 11 lines · Tom Ammon

Hal,

Quoted 8 lines

The procedure is a little different. You'll need to create a child interface (on a partition) first and then you will be able to configure it as follows:

echo 0x8001> /sys/class/net/ib0/create_child
ifconfig ib0.8001 ...

Note that you'll want the 0x8000 bit on for full membership.

What does "full membership" mean?

For a partition, full members can talk with both full and limited members whereas limited members can only talk with full members.

Quoted 2 lines

And are you saying that for full
membership, any value beginning with 0x8 would work?

The 0x8000 bit is the (full) membership bit so 0x8abc is the pkey for full membership in the 0xabc partition. 15 bits of pkey can be used for partition although my example above only used 12 bits.

Quoted 1 line

Can an HCA have full membership on multiple partitions simultaneously?

Yes.

-- Hal

Quoted 84 lines

Thanks,

Tom

Quoted 68 lines

-- Hal

Quoted 66 lines

Tom

Quoted 16 lines · Hal Rosenstock

Quoted 14 lines · Hal Rosenstock

Hi Tom,

Quoted 5 lines · Tom Ammon

   Hi,

I have some basic questions about IB partitions.

Can an HCA port belong to more than 1 partition at a time?

Yes.

Quoted 4 lines

How do you configure partitions with opensm? From reading the opensmd man page, it looks like you just create a file called /etc/osm-partitions.conf, with port GUIDs and such, but is this current?

Quoted 36 lines

Quoted 1 line

The default location depends on how OpenSM is configured/built.

I missed this the first time around:

and yes, the syntax indicated is current.

-- Hal

Quoted 29 lines

Quoted 4 lines

I ask because according to the man page the opensm configuration file is
in /etc/opensm/ . Can you tell opensm where to look for the partitions file?

Yes, with either the -P option on the command line or partition_config_file line in the options file.

-- Hal

Quoted 20 lines

Thanks,

Tom

--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

 Hal,

I have some more partition questions. Do I have to configure the nodes as members of partition(s) both in opensm and on the individual node (if the node is a member of two partitions)?

And what about the switches - do the
switches need be configured as part of any partition?

Tom

Quoted 28 lines · Hal Rosenstock

Tom,

Quoted 7 lines · Tom Ammon

 Hal,

Quoted 4 lines

The procedure is a little different. You'll need to create a child interface (on a partition) first and then you will be able to configure it as follows:

Quoted 6 lines

Quoted 4 lines

echo 0x8001>    /sys/class/net/ib0/create_child
ifconfig ib0.8001 ...

Note that you'll want the 0x8000 bit on for full membership.

What does "full membership" mean?

For a partition, full members can talk with both full and limited members whereas limited members can only talk with full members.

Quoted 2 lines

And are you saying that for full
membership, any value beginning with 0x8 would work?

The 0x8000 bit is the (full) membership bit so 0x8abc is the pkey for full membership in the 0xabc partition. 15 bits of pkey can be used for partition although my example above only used 12 bits.

Quoted 101 lines

Quoted 1 line

Can an HCA have full membership on multiple partitions simultaneously?

Yes.

-- Hal

Quoted 95 lines

Thanks,

Tom

Quoted 79 lines

-- Hal

Quoted 77 lines

Tom

Quoted 22 lines · Hal Rosenstock

Quoted 19 lines · Hal Rosenstock

Hi Tom,

Quoted 6 lines · Tom Ammon

  Hi,

I have some basic questions about IB partitions.

Can an HCA port belong to more than 1 partition at a time?

Yes.

Quoted 5 lines

How do you configure partitions with opensm? From reading the opensmd
man page, it looks like you just create a file called /etc/osm-partitions.conf, with port GUIDs and such, but is this current?

Quoted 41 lines

Quoted 1 line

The default location depends on how OpenSM is configured/built.

I missed this the first time around:

and yes, the syntax indicated is current.

-- Hal

Quoted 33 lines

Quoted 6 lines

I ask because according to the man page the opensm configuration file
is
in /etc/opensm/ . Can you tell opensm where to look for the partitions
file?

Yes, with either the -P option on the command line or partition_config_file line in the options file.

-- Hal

Quoted 21 lines

Thanks,

Tom

--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma"
in
the body of a message to [email protected] More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu