Allow for easily configuring multiple fabrics on one opensm server

PATCH In review Doug Ledford 4 participants 14 messages Feb 29, 2012 – Mar 5, 2012 10 messages not in the archive InfiniBand

lore

Share & tools

lore

Doug Ledford

author

2012-02-29 05:01 UTC #0 lore raw

PATCH 1/1 Allow for easily configuring multiple fabrics on one opensm server

There are two things that stand in the way of opensm being run on redundant fabrics easily:

1) The opensm init script only starts one instance of opensm and opensm will only work on one fabric per instance
2) Even if you start multiple instances, you have to hand modify config files for each instance and then when you upgrade the opensm rpm you either loose your modifications or loose getting new default settings

I worked around both of these issues, I've attached the files I used to do so.

First, I have an opensm init script that allows starting multiple opensm instances. It supports configuring this in one of two ways:

1) Create multiple opensm.conf files, each with a numbered suffix (so opensm.conf.1, opensm.conf.2, etc.) and it will start one opensm instance per config file. This allows an admin to copy the default config over and edit the things they need, and on rpm upgrade there will be a new default opensm.conf file so they can diff between their edited version and the new default and see if there are changes they need to bring back in. This also allows for complete flexibility in setting up the different fabrics, for instance you could use one type of routing on one and a totally different type on the others.

2) Edit the file /etc/sysconfig/opensm and define more than one GUID in the GUIDs variable. This will cause the opensm init script to automatically start one instance per GUID, passing the GUID in on the command line.

For the most part, this works well. However, openmpi in particular doesn't like you to have physically separate fabrics that have the same subnet_prefix, and you can't specify a subnet_prefix on the command line to go along with the GUIDs. So I wrote a patch for that and made the init script unilaterally increment the subnet prefix for each different GUID it's attaching to.

All in all, we use the attached opensm file in /etc/sysconfig as the standard place you put options belonging to an init script, we have the opensm init script, the subnet_prefix patch I wrote, and with those combined things work quite well.

However, I will note that our init script does not (and will not ever) play the passwordless root ssh stuff that upstream does. This is considered a serious security risk on side. The idea that a customer (let's say a wall street bank) should set up passwordless root ssh on their cluster that's a backend to their web farm? Oh hell no...

I might recommend that it is long since past time for that particular misfeature of the upstream opensm init script to be done away with. Personally, I would simply recommend that on failover from a primary to a backup that it simply scan the fabric and build a "current guid2lid" map from what it finds, then start updating from there. Or something like that. But passwordless ssh...bleh.

Oh, and while I've got your ear...is there a good reason the opensm libs have been soname bumping so frequently? Is it not possible to extend the APIs without soname bumps quite so often?

Signature · 3 lines

Doug Ledford <[email protected]>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

#!/bin/bash
#
# Bring up/down opensm
#
# chkconfig: - 15 85
# description: Activates/Deactivates InfiniBand Subnet Manager # config: /etc/ofed/opensm.conf
#
### BEGIN INIT INFO
# Provides: opensm
# Default-Stop: 0 1 2 3 4 5 6
# Required-Start: rdma
# Required-Stop: rdma
# Short-Description: Starts/Stops the InfiniBand Subnet Manager # Description: Starts/Stops the InfiniBand Subnet Manager ### END INIT INFO

. /etc/rc.d/init.d/functions

prog=/usr/sbin/opensm PID_FILE=/var/run/opensm.pid
[ -f /etc/sysconfig/opensm ] && . /etc/sysconfig/opensm

[ -n "$PRIORITY" ] && prio="-p $PRIORITY"

ACTION=$1

start()

{
    local OSM_PID=
    if [ -z "$GUIDS" ]; then
        CONFIGS=""
        CONFIG_CNT=0
        for conf in /etc/rdma/opensm.conf.[0-9]*; do
            CONFIGS="$CONFIGS $conf"
            let CONFIG_CNT++
        done
    else
        GUID_CNT=0
        for guid in $GUIDS; do
            let GUID_CNT++
        done
    fi
    [ -f /var/lock/subsys/opensm ] && return 0
    # Start opensm
    echo -n "Starting IB Subnet Manager: "
    [ -n "$PRIORITY" ] && echo -n "Priority=$PRIORITY "
    [ -n "$GUIDS" ] && echo -n "$GUID_CNT guids "
    [ -n "$CONFIGS" ] && echo -n "$CONFIG_CNT instances "
    if [ -n "$GUIDS" ]; then
	SUBNET_COUNT=0
        for guid in $GUIDS; do
	    SUBNET_PREFIX=`printf "0xfe800000000000%02d" $SUBNET_COUNT`
            $prog -B $prio -g $guid --subnet_prefix $SUBNET_PREFIX >/dev/null 2>&1
	    let SUBNET_COUNT++
        done
    elif [ -n "$CONFIGS" ]; then
        for config in $CONFIGS; do
            $prog -B $prio -F $config >/dev/null 2>&1
        done
    else
        $prog -B $prio >/dev/null 2>&1
    fi
    sleep 1
    OSM_PID=`pidof $prog`
    checkpid $OSM_PID
    RC=$?
    [ $RC -eq 0 ] && echo_success || echo_failure
    [ $RC -eq 0 ] && touch /var/lock/subsys/opensm
    [ $RC -eq 0 ] && echo $OSM_PID > $PID_FILE
    echo
    return $RC    
}

stop()

{
    [ -f /var/lock/subsys/opensm ] || return 0

    echo -n "Stopping IB Subnet Manager(s)."

    OSM_PID=`cat $PID_FILE`

    checkpid $OSM_PID
    RC=$?
    if [ $RC -ne 0 ]; then
	rm -f $PID_FILE
	rm -f /var/lock/subsys/opensm
	echo_success
	return 0
    fi
    # Kill opensm
    kill -15 $OSM_PID >/dev/null 2>&1
    cnt=0
    while [ $cnt -lt 6 ]; do
        checkpid $OSM_PID
	if [ $? -ne 0 ]; then
	    break
	fi
        echo -n "."
	sleep 1
	let cnt++
    done

    checkpid $OSM_PID
    if [ $? -eq 0 ]; then
	kill -KILL $OSM_PID > /dev/null 2>&1
	echo -n "+"
        sleep 1
    fi
    checkpid $OSM_PID
    DEAD=$?
    if [ $DEAD -eq 0 ]; then
	echo_failure
	echo
	return 1
    fi
    echo_success 
    echo
 
    # Remove pid file if any.
    rm -f $PID_FILE
    rm -f /var/lock/subsys/opensm
    return 0    
}

restart ()
{
stop
start
}

condrestart ()

{
	[ -f /var/lock/subsys/opensm ] && restart || return 0
}

reload ()

{
	[ -f $PID_FILE ] || return 0
	OSM_PID=`cat $PID_FILE`
	action $"Rescanning IB Subnet:" kill -HUP $OSM_PID
	return $?
}

usage ()
{
echo
echo "Usage: basename $0 {start|stop|restart|condrestart|try-restart|force-reload|status}" echo
return 2
}

case $ACTION in

	start|stop|restart|reload|condrestart|try-restart|force-reload)
	    [ `id -u` != "0" ] && exit 4 ;;

esac

case $ACTION in

	start) start; RC=$? ;;
	stop) stop; RC=$? ;;
	restart) restart; RC=$? ;;
	reload) reload; RC=$? ;;
	condrestart) condrestart; RC=$? ;;
	try-restart) condrestart; RC=$? ;;
	force-reload) condrestart; RC=$? ;;
	status) status $prog; RC=$? ;;
	*) usage; RC=$? ;;

esac

exit $RC

# Problem #1: Multiple IB fabrics needing a subnet manager #
# In the event that a machine has more than one IB subnet attached,

# and that machine is an opensm server, by default, opensm will
# only attach to one port and will not manage the fabric on the
# other port.  There are two ways to solve this problem:
#
# 1) Start opensm on multiple machines and configure it to manage
#    different fabrics on each machine
# 2) Configure opensm to start multiple instances on a single
#    machine
#
# Both solutions to this problem require non-standard configurations.
# In other words, you would normally have to modify /etc/rdma/opensm.conf
# and once you do that, the file will no longer be updated for new
# options when opensm is upgraded.  In an effort to allow people to
# have more than one subnet managed by opensm without having to modify
# the system default opensm.conf file, we have enabled two methods
# for modifying the default opensm config items needed to enable
# multiple fabric management.
#
# Method #1: Create multiple opensm.conf files in non-standard locations
#   Copy /etc/rdma/opensm.conf to /etc/rdma/opensm.conf.<number>
#     (do this once for each instance you want started)
#   Edit each copy of the opensm.conf file to reflect the necessary changes
#     for a multiple instance startup.  If you need to manage more than
#     one fabric, you will have to change the guid option in each file
#     to specify the guid of the specific port you want opensm attached
#     to.
#
# The advantage to method #1 is that, on the off chance you want to do
# really special custom things on different ports, like have different
# QoS settings depending on which port you are attached to, you have the
# freedom to edit any and all settings for each instance without those
# changes affecting other instances or being lost when opensm upgrades.
#
# Method #2: Specify multiple GUIDS variable entries in this file
#   Uncomment the below GUIDS variable and enter each guid you need to attach
#     to into the list.  If using this method you need to enter each
#     guid into the list as we won't attach to any default ports, only
#     those specified in the list.
#
#GUIDS="0x0002c90300048ca1 0x0002c90300048ca2"
#
# The obvious advantage to method #2 is that it's simple and doesn't
# clutter up your file system, but it is far more limited in what you
# can do.  If you enable method #2, then even if you create the files
# referenced in method #1, they will be ignored.
#
# Problem #2: Activating a backup subnet manager
#
# The default priority of opensm is set so that it wants to be the
# primary subnet manager.  This is great when you are only running
# opensm on one server, but if you want to have a non-primary opensm
# instance for failover, then you have to manually edit the opensm.conf
# file like for problem #1.  This carries with it all the problems
# listed above.  If you wish to enable opensm as a non-primary manager,
# then you can uncomment the PRIORITY variable below and set it to
# some number between 0 and 15, where 15 is the highest priority and
# the primary manager, with 0 being the lowest backup server.  This method
# will work with the GUIDS option above, and also with the multiple
# config files in method #1 above.  However, only a single priority is
# supported here.  If you wanted more than one priority (say this machine
# is the primary on the first fabric, and second on the second fabric,
# while the other opensm server is primary on the second fabric and
# second on the primary), then the only way to do that is to use method #1
# above and individually edit the config files.  If you edit the config
# files to set the priority and then also set the priority here, then
# this setting will override the config files and render that particular
# edit useless.
#
#PRIORITY=15

man/opensm.8.in renamed from man/opensm.8.in.prefix +8 -0

diff -up opensm-3.3.13/man/opensm.8.in.prefix opensm-3.3.13/man/opensm.8.in
@@ -11,6 +11,7 @@ opensm \- InfiniBand subnet manager and 
 [\-g(uid) <GUID in hex>]
 [\-l(mc) <LMC>]
 [\-p(riority) <PRIORITY>]
+[\-\-subnet_prefix <PREFIX in hex>]
 [\-smkey <SM_Key>]
 [\-\-sm_sl <SL number>]
 [\-r(eassign_lids)]

@@ -130,6 +131,13 @@ This will effect the handover cases, whe
 is chosen by priority and GUID.  Range goes from 0
 (default and lowest priority) to 15 (highest).
 .TP
+\fB\-\-subnet_prefix\fR <PREFIX in hex>
+This option specifies the subnet prefix to use in
+on the fabric.  The default prefix is
+0xfe80000000000000.  OpenMPI in particular requires
+separate fabrics plugged into different ports to
+have different prefixes or else it won't run.
+.TP
 \fB\-smkey\fR <SM_Key value>
 This option specifies the SM\'s SM_Key (64 bits).
 This will effect SM authentication.

opensm/main.c renamed from opensm/main.c.prefix +9 -0

diff -up opensm-3.3.13/opensm/main.c.prefix opensm-3.3.13/opensm/main.c
@@ -156,6 +156,9 @@ static void show_usage(void)
 	       "          This will effect the handover cases, where master\n"
 	       "          is chosen by priority and GUID.  Range goes\n"
 	       "          from 0 (lowest priority) to 15 (highest).\n\n");
+	printf("--subnet_prefix <prefix>\n"
+	       "          Set the subnet prefix to something other than the\n"
+	       "          default value of 0xfe80000000000000\n\n");
 	printf("--smkey, -k <SM_Key>\n"
 	       "          This option specifies the SM's SM_Key (64 bits).\n"
 	       "          This will effect SM authentication.\n"

@@ -607,6 +610,7 @@ int main(int argc, char *argv[])
 		{"once", 0, NULL, 'o'},
 		{"reassign_lids", 0, NULL, 'r'},
 		{"priority", 1, NULL, 'p'},
+		{"subnet_prefix", 1, NULL, 13},
 		{"smkey", 1, NULL, 'k'},
 		{"routing_engine", 1, NULL, 'R'},
 		{"ucast_cache", 0, NULL, 'A'},

@@ -911,6 +915,11 @@ int main(int argc, char *argv[])
 			printf(" Priority = %d\n", temp);
 			break;
 
+		case 13:
+			opt.subnet_prefix = cl_hton64(strtoull(optarg, NULL, 16));
+			printf(" Subnet_Prefix = <0x%" PRIx64 ">\n", cl_hton64(opt.subnet_prefix));
+			break;
+
 		case 'k':
 			sm_key = cl_hton64(strtoull(optarg, NULL, 16));
 			printf(" SM Key <0x%" PRIx64 ">\n", cl_hton64(sm_key));

A message that is not in the archive (sent off-list or not yet received): [email protected]

Ira Weiny

2012-02-29 19:22 UTC #2 lore raw

Doug,

First thanks for this. Some comments below.

Quoted 25 lines · On Wed, 29 Feb 2012 00:01:16 -0500 Doug Ledford…

There are two things that stand in the way of opensm being run on redundant fabrics easily:
1) The opensm init script only starts one instance of opensm and opensm will only work on one fabric per instance
2) Even if you start multiple instances, you have to hand modify config files for each instance and then when you upgrade the opensm rpm you either loose your modifications or loose getting new default settings
I worked around both of these issues, I've attached the files I used to do so.
First, I have an opensm init script that allows starting multiple opensm instances. It supports configuring this in one of two ways:
1) Create multiple opensm.conf files, each with a numbered suffix (so opensm.conf.1, opensm.conf.2, etc.) and it will start one opensm instance per config file. This allows an admin to copy the default config over and edit the things they need, and on rpm upgrade there will be a new default opensm.conf file so they can diff between their edited version and the new default and see if there are changes they need to bring back in. This also allows for complete flexibility in setting up the different fabrics, for instance you could use one type of routing on one and a totally different type on the others.

Quoted 4 lines

2) Edit the file /etc/sysconfig/opensm and define more than one GUID in the GUIDs variable. This will cause the opensm init script to automatically start one instance per GUID, passing the GUID in on the command line.

I know you are going for ease of use here, which is good, however, I worry about this file becoming a redefinition of opensm.conf.

Quoted 7 lines

For the most part, this works well. However, openmpi in particular doesn't like you to have physically separate fabrics that have the same subnet_prefix, and you can't specify a subnet_prefix on the command line to go along with the GUIDs. So I wrote a patch for that and made the init script unilaterally increment the subnet prefix for each different GUID it's attaching to.

If you only allow option 1 above this takes care of itself by making the admin configure his subnet prefixes in each config file as appropriate. The only down side is the loss of new configuration options as you upgrade. However, that is probably better taken care of by a default config file with each package. I mentioned this to Sasha years back and was denied since "you can always generate a new one with '-c'". :-(

Alex would a default config file be acceptable? It would mean more work on your part.

Ira

Quoted 28 lines

All in all, we use the attached opensm file in /etc/sysconfig as the standard place you put options belonging to an init script, we have the opensm init script, the subnet_prefix patch I wrote, and with those combined things work quite well.
However, I will note that our init script does not (and will not ever) play the passwordless root ssh stuff that upstream does. This is considered a serious security risk on side. The idea that a customer (let's say a wall street bank) should set up passwordless root ssh on their cluster that's a backend to their web farm? Oh hell no...
I might recommend that it is long since past time for that particular misfeature of the upstream opensm init script to be done away with. Personally, I would simply recommend that on failover from a primary to a backup that it simply scan the fabric and build a "current guid2lid" map from what it finds, then start updating from there. Or something like that. But passwordless ssh...bleh.
Oh, and while I've got your ear...is there a good reason the opensm libs have been soname bumping so frequently? Is it not possible to extend the APIs without soname bumps quite so often?
Signature · 3 lines
Doug Ledford <[email protected]>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

Signature · 9 lines

Ira Weiny
Member of Technical Staff
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

Doug Ledford

author

2012-02-29 19:47 UTC #4 lore raw

Quoted 32 lines · Ira Weiny

Doug,
First thanks for this. Some comments below.
Quoted 25 lines · On Wed, 29 Feb 2012 00:01:16 -0500 Doug Ledford…
There are two things that stand in the way of opensm being run on redundant fabrics easily:
1) The opensm init script only starts one instance of opensm and opensm will only work on one fabric per instance
2) Even if you start multiple instances, you have to hand modify config files for each instance and then when you upgrade the opensm rpm you either loose your modifications or loose getting new default settings
I worked around both of these issues, I've attached the files I used to do so.
First, I have an opensm init script that allows starting multiple opensm instances. It supports configuring this in one of two ways:
1) Create multiple opensm.conf files, each with a numbered suffix (so opensm.conf.1, opensm.conf.2, etc.) and it will start one opensm instance per config file. This allows an admin to copy the default config over and edit the things they need, and on rpm upgrade there will be a new default opensm.conf file so they can diff between their edited version and the new default and see if there are changes they need to bring back in. This also allows for complete flexibility in setting up the different fabrics, for instance you could use one type of routing on one and a totally different type on the others.

Quoted 6 lines

Quoted 4 lines
2) Edit the file /etc/sysconfig/opensm and define more than one GUID in the GUIDs variable. This will cause the opensm init script to automatically start one instance per GUID, passing the GUID in on the command line.
I know you are going for ease of use here, which is good, however, I worry about this file becoming a redefinition of opensm.conf.

Hehehe, I don't think you'll ever have to worry about that. You have looked at opensm.conf in recent times I take it? Replacing that with command line options in a shell startup script isn't reasonable.

However, if you are going to run a redundant fabric setup, then the two things you *know* you will have to set are the guid and subnet_prefix (assuming you want to use openmpi). If you are going to run master/slave setup, then the one thing you *know* you will have to set is the priority. Supporting setting those items in an init script is reasonable. Beyond that, I would agree, you should just edit the config files.

Signature · 3 lines

Doug Ledford <[email protected]>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): 20120301021501.GB961-7GFyYy+Av7rWWZS0+0nfmVaTQe2KTcn/@public.gmane.org

Doug Ledford

author

2012-03-01 13:31 UTC #7 lore raw

Quoted 45 lines · Brian Ginsbach

Quoted 44 lines · Doug Ledford
Quoted 32 lines · Ira Weiny
Doug,
First thanks for this. Some comments below.
Quoted 25 lines · On Wed, 29 Feb 2012 00:01:16 -0500 Doug Ledford…
There are two things that stand in the way of opensm being run on redundant fabrics easily:
1) The opensm init script only starts one instance of opensm and opensm will only work on one fabric per instance
2) Even if you start multiple instances, you have to hand modify config files for each instance and then when you upgrade the opensm rpm you either loose your modifications or loose getting new default settings
I worked around both of these issues, I've attached the files I used to do so.
First, I have an opensm init script that allows starting multiple opensm instances. It supports configuring this in one of two ways:
1) Create multiple opensm.conf files, each with a numbered suffix (so opensm.conf.1, opensm.conf.2, etc.) and it will start one opensm instance per config file. This allows an admin to copy the default config over and edit the things they need, and on rpm upgrade there will be a new default opensm.conf file so they can diff between their edited version and the new default and see if there are changes they need to bring back in. This also allows for complete flexibility in setting up the different fabrics, for instance you could use one type of routing on one and a totally different type on the others.
Quoted 6 lines
Quoted 4 lines
2) Edit the file /etc/sysconfig/opensm and define more than one GUID in the GUIDs variable. This will cause the opensm init script to automatically start one instance per GUID, passing the GUID in on the command line.
I know you are going for ease of use here, which is good, however, I worry about this file becoming a redefinition of opensm.conf.
Hehehe, I don't think you'll ever have to worry about that. You have looked at opensm.conf in recent times I take it? Replacing that with command line options in a shell startup script isn't reasonable.

Quoted 7 lines

Quoted 3 lines
However, if you are going to run a redundant fabric setup, then the two things you *know* you will have to set are the guid and subnet_prefix (assuming you want to use openmpi). If you are going to run
Assuming you are doing this for openmpi. The subnet_prefix should not be needed if the separate subnets are for disjoint networks (mpi and storage) or multiple storage networks.

True enough, but that's why I said openmpi. It is, after all, a primary IB fabric user.

Quoted 11 lines

Quoted 5 lines
master/slave setup, then the one thing you *know* you will have to set is the priority. Supporting setting those items in an init script is reasonable. Beyond that, I would agree, you should just edit the config files.
Not everything can be done in the config files. I'm not sure that it is a good idea to have every opensm instance using the same temporary and cache directories (OSM_TMP_DIR and OSM_CACHE_DIR environment variables). Seems like these fall into the *know* you will have to set category.

Unless opensm is smart enough to allow more than one instance to open the same log file and interleave their log messages successfully. Temporary files or cache files could do something like use a pid suffix if need be. But yes, I see your point. Opensm has lots of junk it likes to put on the drive :-/

Quoted 4 lines

You'd also want to make sure that other potentially very useful things are configured in the config files (e.g. log_file and log_prefix). Aren't these also things you *know* you will have to set.

I would say we are simply getting to the point where we *know* we need opensm to handle more than one fabric from a single instance ;-)

Signature · 3 lines

Doug Ledford <[email protected]>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

A message that is not in the archive (sent off-list or not yet received): [email protected]

Hal Rosenstock

2012-03-05 12:52 UTC #9 lore raw

Quoted 2 lines · Doug Ledford

I would say we are simply getting to the point where we *know* we need opensm to handle more than one fabric from a single instance ;-)

Why does a single OpenSM need to handle multiple subnets/fabrics ? What's the issue with running multiple OpenSMs with each handling it's own subnet/fabric ?

-- Hal
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4F54B707.1070606-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org

Doug Ledford

author

2012-03-05 15:28 UTC #11 lore raw

----- Original Message -----

Quoted 9 lines · Doug Ledford

Quoted 3 lines
I would say we are simply getting to the point where we *know* we need
opensm to handle more than one fabric from a single instance ;-)
Why does a single OpenSM need to handle multiple subnets/fabrics ? What's the issue with running multiple OpenSMs with each handling it's
own subnet/fabric ?

Because it wasn't built with the idea in mind of sharing logs or cache directories, etc. Not to mention that the proliferation of files when you start multiple instances of opensm is pretty insane (have you counted how many different files opensm can be configured to require now a days...).

Signature · 8 lines

Doug Ledford <[email protected]>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 2962b1d0-a679-45d0-a82b-5d624e2081f9-HOthUlaS0a9+R5eDjrG6zsCp5Q1pQRjfhaY/[email protected]

Hal Rosenstock

2012-03-05 15:53 UTC #13 lore raw

Quoted 6 lines · Doug Ledford

----- Original Message -----
Quoted 5 lines · Doug Ledford
Quoted 3 lines
I would say we are simply getting to the point where we *know* we need
opensm to handle more than one fabric from a single instance ;-)

Quoted 7 lines

Quoted 4 lines
Why does a single OpenSM need to handle multiple subnets/fabrics ? What's the issue with running multiple OpenSMs with each handling it's
own subnet/fabric ?
Because it wasn't built with the idea in mind of sharing logs or cache directories, etc.

Yes, each instance has it's own configuration and output files.

Quoted 4 lines

Not to mention that the proliferation of files when
you start multiple instances of opensm is pretty insane (have you counted how many different files opensm can be configured to require now a days...).

I'm not disagreeing with the fact that there are numerous config files (the main config file is already large enough without the myriad of features with their own separate config files) but different subnets have totally different configuration requirements. It's not just the subnet prefix, SM priority, and the few other config items that you identified.

So what would make this better in your mind ?

-- Hal
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4F54E177.9030302-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org

Doug Ledford

author

2012-03-05 17:25 UTC #15 lore raw

----- Original Message -----

Quoted 24 lines · Doug Ledford

Quoted 6 lines
----- Original Message -----
Quoted 5 lines · Doug Ledford
Quoted 3 lines
I would say we are simply getting to the point where we *know* we need
opensm to handle more than one fabric from a single instance ;-)
Quoted 8 lines
Quoted 4 lines
Why does a single OpenSM need to handle multiple subnets/fabrics ? What's the issue with running multiple OpenSMs with each handling it's
own subnet/fabric ?
Because it wasn't built with the idea in mind of sharing logs or cache
directories, etc.
Yes, each instance has it's own configuration and output files.
Quoted 5 lines
Not to mention that the proliferation of files when
you start multiple instances of opensm is pretty insane (have you counted how many different files opensm can be configured to require
now a days...).

Quoted 7 lines

I'm not disagreeing with the fact that there are numerous config files
(the main config file is already large enough without the myriad of features with their own separate config files) but different subnets have totally different configuration requirements. It's not just the subnet prefix, SM priority, and the few other config items that you identified.

In some cases, and in other cases the different subnets are perfect mirrors of each other (redundant, identical fabrics).

Quoted 1 line

So what would make this better in your mind ?

If I got to wave a magic wand, I would say it's time that opensm management started getting simpler, not more complex. With all of the various options for routing engines and QoS and partitions, it's gotten to where you need the equivalent of the old Cisco CNA in order to configure opensm ;-)

Signature · 8 lines

Doug Ledford <[email protected]>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Ira Weiny

2012-03-01 22:46 UTC #16 lore raw

In reply to a message not in the archive

Quoted 59 lines · On Wed, 29 Feb 2012 20:15:02 -0600 Brian Ginsbach…

Quoted 40 lines · Doug Ledford
Quoted 32 lines · Ira Weiny
Doug,
First thanks for this. Some comments below.
Quoted 25 lines · On Wed, 29 Feb 2012 00:01:16 -0500 Doug Ledford…
There are two things that stand in the way of opensm being run on redundant fabrics easily:
1) The opensm init script only starts one instance of opensm and opensm will only work on one fabric per instance
2) Even if you start multiple instances, you have to hand modify config files for each instance and then when you upgrade the opensm rpm you either loose your modifications or loose getting new default settings
I worked around both of these issues, I've attached the files I used to do so.
First, I have an opensm init script that allows starting multiple opensm instances. It supports configuring this in one of two ways:
1) Create multiple opensm.conf files, each with a numbered suffix (so opensm.conf.1, opensm.conf.2, etc.) and it will start one opensm instance per config file. This allows an admin to copy the default config over and edit the things they need, and on rpm upgrade there will be a new default opensm.conf file so they can diff between their edited version and the new default and see if there are changes they need to bring back in. This also allows for complete flexibility in setting up the different fabrics, for instance you could use one type of routing on one and a totally different type on the others.
Quoted 6 lines
Quoted 4 lines
2) Edit the file /etc/sysconfig/opensm and define more than one GUID in the GUIDs variable. This will cause the opensm init script to automatically start one instance per GUID, passing the GUID in on the command line.
I know you are going for ease of use here, which is good, however, I worry about this file becoming a redefinition of opensm.conf.
Quoted 7 lines
Hehehe, I don't think you'll ever have to worry about that. You have looked at opensm.conf in recent times I take it? Replacing that with command line options in a shell startup script isn't reasonable.
However, if you are going to run a redundant fabric setup, then the two things you *know* you will have to set are the guid and subnet_prefix (assuming you want to use openmpi). If you are going to run
Assuming you are doing this for openmpi. The subnet_prefix should not be needed if the separate subnets are for disjoint networks (mpi and storage) or multiple storage networks.
Quoted 5 lines
master/slave setup, then the one thing you *know* you will have to set is the priority. Supporting setting those items in an init script is reasonable. Beyond that, I would agree, you should just edit the config files.

Quoted 5 lines

Not everything can be done in the config files. I'm not sure that it is a good idea to have every opensm instance using the same temporary and cache directories (OSM_TMP_DIR and OSM_CACHE_DIR environment variables). Seems like these fall into the *know* you will have to set category.

Brian brings up a really good point. Even though some things can't be configured now, opensm.conf is the better way to configure log file placement etc. So in my mind this re-emphasises the need to simply allow for multiple opensm.conf's and not introduce another config file. But as I said before it is Alex's call.

Ira

Quoted 8 lines

You'd also want to make sure that other potentially very useful things are configured in the config files (e.g. log_file and log_prefix). Aren't these also things you *know* you will have to set.
Signature · 1 line
Brian Ginsbach                          Cray Inc.

Signature · 9 lines

Ira Weiny
Member of Technical Staff
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

Alex Netes

2012-03-02 10:13 UTC #18 lore raw

Quoted 68 lines · Ira Weiny

Quoted 59 lines · On Wed, 29 Feb 2012 20:15:02 -0600 Brian Ginsbach…
Quoted 40 lines · Doug Ledford
Quoted 32 lines · Ira Weiny
Doug,
First thanks for this. Some comments below.
Quoted 25 lines · On Wed, 29 Feb 2012 00:01:16 -0500 Doug Ledford…
There are two things that stand in the way of opensm being run on redundant fabrics easily:
1) The opensm init script only starts one instance of opensm and opensm will only work on one fabric per instance
2) Even if you start multiple instances, you have to hand modify config files for each instance and then when you upgrade the opensm rpm you either loose your modifications or loose getting new default settings
I worked around both of these issues, I've attached the files I used to do so.
First, I have an opensm init script that allows starting multiple opensm instances. It supports configuring this in one of two ways:
1) Create multiple opensm.conf files, each with a numbered suffix (so opensm.conf.1, opensm.conf.2, etc.) and it will start one opensm instance per config file. This allows an admin to copy the default config over and edit the things they need, and on rpm upgrade there will be a new default opensm.conf file so they can diff between their edited version and the new default and see if there are changes they need to bring back in. This also allows for complete flexibility in setting up the different fabrics, for instance you could use one type of routing on one and a totally different type on the others.
Quoted 6 lines
Quoted 4 lines
2) Edit the file /etc/sysconfig/opensm and define more than one GUID in the GUIDs variable. This will cause the opensm init script to automatically start one instance per GUID, passing the GUID in on the command line.
I know you are going for ease of use here, which is good, however, I worry about this file becoming a redefinition of opensm.conf.
Quoted 7 lines
Hehehe, I don't think you'll ever have to worry about that. You have looked at opensm.conf in recent times I take it? Replacing that with command line options in a shell startup script isn't reasonable.
However, if you are going to run a redundant fabric setup, then the two things you *know* you will have to set are the guid and subnet_prefix (assuming you want to use openmpi). If you are going to run
Assuming you are doing this for openmpi. The subnet_prefix should not be needed if the separate subnets are for disjoint networks (mpi and storage) or multiple storage networks.
Quoted 5 lines
master/slave setup, then the one thing you *know* you will have to set is the priority. Supporting setting those items in an init script is reasonable. Beyond that, I would agree, you should just edit the config files.
Quoted 5 lines
Not everything can be done in the config files. I'm not sure that it is a good idea to have every opensm instance using the same temporary and cache directories (OSM_TMP_DIR and OSM_CACHE_DIR environment variables). Seems like these fall into the *know* you will have to set category.

Quoted 4 lines

Brian brings up a really good point. Even though some things can't be configured now, opensm.conf is the better way to configure log file placement etc. So in my mind this re-emphasises the need to simply allow for multiple opensm.conf's and not introduce another config file. But as I said before it is Alex's call.
Ira

I agree with Ira on that point. sysconfig/opensm is being used for syncing guid2lid cache file between remote SMs, which is out of scope of opensm configuration and it's completely external mechanism.

Regarding the first approach (using multiple opensm.conf files), I think that beyond starting all opensm instances, the init script should be able to manipulate (restart/stop/signal HUP,USR1) to any of the instances of opensm separately.
Moreover, OSM_TMP_DIR and OSM_CACHE_DIR can be added as options to opensm.conf so if configured will overwrite the global setting or alternatively add pid or opensm id as a suffix to the log/state files.

Quoted 20 lines

Quoted 8 lines
You'd also want to make sure that other potentially very useful things are configured in the config files (e.g. log_file and log_prefix). Aren't these also things you *know* you will have to set.
Signature · 1 line
Brian Ginsbach                          Cray Inc.
Signature · 9 lines
Ira Weiny
Member of Technical Staff
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Signature · 5 lines

-- Alex
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Alex Netes

2012-03-02 10:30 UTC #19 lore raw

In reply to a message not in the archive

Quoted 47 lines · Ira Weiny

Doug,
First thanks for this. Some comments below.
Quoted 25 lines · On Wed, 29 Feb 2012 00:01:16 -0500 Doug Ledford…
There are two things that stand in the way of opensm being run on redundant fabrics easily:
1) The opensm init script only starts one instance of opensm and opensm will only work on one fabric per instance
2) Even if you start multiple instances, you have to hand modify config files for each instance and then when you upgrade the opensm rpm you either loose your modifications or loose getting new default settings
I worked around both of these issues, I've attached the files I used to do so.
First, I have an opensm init script that allows starting multiple opensm instances. It supports configuring this in one of two ways:
1) Create multiple opensm.conf files, each with a numbered suffix (so opensm.conf.1, opensm.conf.2, etc.) and it will start one opensm instance per config file. This allows an admin to copy the default config over and edit the things they need, and on rpm upgrade there will be a new default opensm.conf file so they can diff between their edited version and the new default and see if there are changes they need to bring back in. This also allows for complete flexibility in setting up the different fabrics, for instance you could use one type of routing on one and a totally different type on the others.
Quoted 4 lines
2) Edit the file /etc/sysconfig/opensm and define more than one GUID in the GUIDs variable. This will cause the opensm init script to automatically start one instance per GUID, passing the GUID in on the command line.
I know you are going for ease of use here, which is good, however, I worry about this file becoming a redefinition of opensm.conf.
Quoted 7 lines
For the most part, this works well. However, openmpi in particular doesn't like you to have physically separate fabrics that have the same subnet_prefix, and you can't specify a subnet_prefix on the command line to go along with the GUIDs. So I wrote a patch for that and made the init script unilaterally increment the subnet prefix for each different GUID it's attaching to.

Quoted 4 lines

If you only allow option 1 above this takes care of itself by making the admin configure his subnet prefixes in each config file as appropriate. The only down side is the loss of new configuration options as you upgrade. However, that is probably better taken care of by a default config file with each package. I mentioned this to Sasha years back and was denied since "you can always generate a new one with '-c'". :-(
Alex would a default config file be acceptable? It would mean more work on your part.

What the default opensm.conf would be used for? Just as a reference to the default values?

-- Alex
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

Doug Ledford

author

2012-03-02 15:31 UTC #20 lore raw

Quoted 48 lines · Alex Netes

Quoted 47 lines · Ira Weiny
Doug,
First thanks for this. Some comments below.
Quoted 25 lines · On Wed, 29 Feb 2012 00:01:16 -0500 Doug Ledford…
There are two things that stand in the way of opensm being run on redundant fabrics easily:
1) The opensm init script only starts one instance of opensm and opensm will only work on one fabric per instance
2) Even if you start multiple instances, you have to hand modify config files for each instance and then when you upgrade the opensm rpm you either loose your modifications or loose getting new default settings
I worked around both of these issues, I've attached the files I used to do so.
First, I have an opensm init script that allows starting multiple opensm instances. It supports configuring this in one of two ways:
1) Create multiple opensm.conf files, each with a numbered suffix (so opensm.conf.1, opensm.conf.2, etc.) and it will start one opensm instance per config file. This allows an admin to copy the default config over and edit the things they need, and on rpm upgrade there will be a new default opensm.conf file so they can diff between their edited version and the new default and see if there are changes they need to bring back in. This also allows for complete flexibility in setting up the different fabrics, for instance you could use one type of routing on one and a totally different type on the others.
Quoted 4 lines
2) Edit the file /etc/sysconfig/opensm and define more than one GUID in the GUIDs variable. This will cause the opensm init script to automatically start one instance per GUID, passing the GUID in on the command line.
I know you are going for ease of use here, which is good, however, I worry about this file becoming a redefinition of opensm.conf.
Quoted 7 lines
For the most part, this works well. However, openmpi in particular doesn't like you to have physically separate fabrics that have the same subnet_prefix, and you can't specify a subnet_prefix on the command line to go along with the GUIDs. So I wrote a patch for that and made the init script unilaterally increment the subnet prefix for each different GUID it's attaching to.

Quoted 7 lines

Quoted 4 lines
If you only allow option 1 above this takes care of itself by making the admin configure his subnet prefixes in each config file as appropriate. The only down side is the loss of new configuration options as you upgrade. However, that is probably better taken care of by a default config file with each package. I mentioned this to Sasha years back and was denied since "you can always generate a new one with '-c'". :-(
Alex would a default config file be acceptable? It would mean more work on your part.
What the default opensm.conf would be used for? Just as a reference to the default values?

No, he's referring to having a default config file that is parsed, then an override config file that is parsed where you only put options you want to update in the override config file. That way you could have, for instance, a default opensm.conf in the normal location and totally unedited so that it gets updated with each update of the opensm rpm, then you could create an opensm.conf.1 that is empty except for just a guid setting, a subnet_prefix setting, maybe a cache dir setting, etc. In that way, if say the default routing engine gets a new option in the future, your override config file won't already be populated with the old stuff. It's a means of inheritance that is functionally identical to specifying all this stuff on the command line, but doesn't require a huge command line or a complex init script.

Signature · 6 lines

Doug Ledford <[email protected]>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

Infiniband specific RPMs available at
	      http://people.redhat.com/dledford/Infiniband

A message that is not in the archive (sent off-list or not yet received): [email protected]

Doug Ledford

author

2012-03-02 15:47 UTC #22 lore raw

Quoted 4 lines · Doug Ledford

Quoted 2 lines · Alex Netes
What the default opensm.conf would be used for? Just as a reference to the default values?

Quoted 12 lines

No, he's referring to having a default config file that is parsed, then an override config file that is parsed where you only put options you want to update in the override config file. That way you could have, for instance, a default opensm.conf in the normal location and totally unedited so that it gets updated with each update of the opensm rpm, then you could create an opensm.conf.1 that is empty except for just a guid setting, a subnet_prefix setting, maybe a cache dir setting, etc. In that way, if say the default routing engine gets a new option in the future, your override config file won't already be populated with the old stuff. It's a means of inheritance that is functionally identical to specifying all this stuff on the command line, but doesn't require a huge command line or a complex init script.

And for what it's worth, it could be as simply done as the attached (untested, but compiled) patch.

Signature · 6 lines

Doug Ledford <[email protected]>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford

Infiniband specific RPMs available at
	      http://people.redhat.com/dledford/Infiniband

opensm/main.c renamed from opensm/main.c.config +21 -2

diff -up opensm-3.3.13/opensm/main.c.config opensm-3.3.13/opensm/main.c
@@ -131,6 +131,13 @@ static void show_usage(void)
 	       "          The name of the OpenSM config file. When not specified\n"
 	       "          " OSM_DEFAULT_CONFIG_FILE
 	       " will be used (if exists).\n\n");
+	printf("--extra-config, -E <file-name>\n"
+	       "          The name of an OpenSM config file used to over ride\n"
+	       "          the entries in the primary config file.  This is\n"
+	       "          useful when you have more than one opensm instance\n"
+	       "          to manage and you want them all to have a central,\n"
+	       "          shared set of options and you want a second, smaller\n"
+	       "          config file to hold their fabric specific options.\n\n");
 	printf("--create-config, -c <file-name>\n"
 	       "          OpenSM will dump its configuration to the specified file and exit.\n"
 	       "          This is a way to generate OpenSM configuration file template.\n\n");

@@ -569,10 +576,10 @@ int main(int argc, char *argv[])
 	boolean_t run_once_flag = FALSE;
 	int32_t vendor_debug = 0;
 	int next_option;
-	char *conf_template = NULL, *config_file = NULL;
+	char *conf_template, *config_file, *extra_config_file;
 	uint32_t val;
 	const char *const short_option =
-	    "F:c:i:w:O:f:ed:D:g:l:L:s:t:a:u:m:X:R:zM:U:S:P:Y:ANBIQvVhoryxp:n:q:k:C:G:H:";
+	    "F:E:c:i:w:O:f:ed:D:g:l:L:s:t:a:u:m:X:R:zM:U:S:P:Y:ANBIQvVhoryxp:n:q:k:C:G:H:";
 
 	/*
 	   In the array below, the 2nd parameter specifies the number

@@ -584,6 +591,7 @@ int main(int argc, char *argv[])
 	const struct option long_option[] = {
 		{"version", 0, NULL, 12},
 		{"config", 1, NULL, 'F'},
+		{"extra-config", 1, NULL, 'E'},
 		{"create-config", 1, NULL, 'c'},
 		{"debug", 1, NULL, 'd'},
 		{"guid", 1, NULL, 'g'},

@@ -647,6 +655,7 @@ int main(int argc, char *argv[])
 		{"torus_config", 1, NULL, 10},
 		{NULL, 0, NULL, 0}	/* Required at the end of the array */
 	};
+	conf_template = config_file = extra_config_file = NULL;
 
 	/* force stdout to be line-buffered */
 	setvbuf(stdout, NULL, _IOLBF, BUFSIZ);

@@ -672,6 +681,11 @@ int main(int argc, char *argv[])
 			config_file = optarg;
 			printf("Config file is `%s`:\n", config_file);
 			break;
+		case 'E':
+			extra_config_file = optarg;
+			printf("Extra Config file is `%s`:\n",
+			       extra_config_file);
+			break;
 		default:
 			break;
 		}

@@ -687,6 +701,11 @@ int main(int argc, char *argv[])
 	if (osm_subn_parse_conf_file(config_file, &opt) < 0)
 		printf("\nFail to parse config file \'%s\'\n", config_file);
 
+	if (extra_config_file)
+		if (osm_subn_parse_conf_file(extra_config_file, &opt) < 0)
+			printf("\nFailed to parse extra config file `%s`\n",
+			       extra_config_file);
+
 	printf("Command Line Arguments:\n");
 	do {
 		next_option = getopt_long_only(argc, argv, short_option,

Ira Weiny

2012-03-05 20:51 UTC #23 lore raw

In reply to Alex Netes

Quoted 48 lines · On Fri, 2 Mar 2012 12:30:15 +0200 Alex Netes…

Quoted 47 lines · Ira Weiny
Doug,
First thanks for this. Some comments below.
Quoted 25 lines · On Wed, 29 Feb 2012 00:01:16 -0500 Doug Ledford…
There are two things that stand in the way of opensm being run on redundant fabrics easily:
1) The opensm init script only starts one instance of opensm and opensm will only work on one fabric per instance
2) Even if you start multiple instances, you have to hand modify config files for each instance and then when you upgrade the opensm rpm you either loose your modifications or loose getting new default settings
I worked around both of these issues, I've attached the files I used to do so.
First, I have an opensm init script that allows starting multiple opensm instances. It supports configuring this in one of two ways:
1) Create multiple opensm.conf files, each with a numbered suffix (so opensm.conf.1, opensm.conf.2, etc.) and it will start one opensm instance per config file. This allows an admin to copy the default config over and edit the things they need, and on rpm upgrade there will be a new default opensm.conf file so they can diff between their edited version and the new default and see if there are changes they need to bring back in. This also allows for complete flexibility in setting up the different fabrics, for instance you could use one type of routing on one and a totally different type on the others.
Quoted 4 lines
2) Edit the file /etc/sysconfig/opensm and define more than one GUID in the GUIDs variable. This will cause the opensm init script to automatically start one instance per GUID, passing the GUID in on the command line.
I know you are going for ease of use here, which is good, however, I worry about this file becoming a redefinition of opensm.conf.
Quoted 7 lines
For the most part, this works well. However, openmpi in particular doesn't like you to have physically separate fabrics that have the same subnet_prefix, and you can't specify a subnet_prefix on the command line to go along with the GUIDs. So I wrote a patch for that and made the init script unilaterally increment the subnet prefix for each different GUID it's attaching to.

Quoted 7 lines

Quoted 4 lines
If you only allow option 1 above this takes care of itself by making the admin configure his subnet prefixes in each config file as appropriate. The only down side is the loss of new configuration options as you upgrade. However, that is probably better taken care of by a default config file with each package. I mentioned this to Sasha years back and was denied since "you can always generate a new one with '-c'". :-(
Alex would a default config file be acceptable? It would mean more work on your part.
What the default opensm.conf would be used for? Just as a reference to the default values?

Mainly as a reference. Right now the "-c" option generates the config with options enabled and set to the defaults. However, most examples I have seen supply the options commented out for the user to comment in what they want set.

This is the method I chose to use in /etc/infiniband-diags/ibdiag.conf.

Another example is httpd.

bash-4.1# rpm -qil httpd | egrep "httpd\.conf"
/etc/httpd/conf/httpd.conf

bash-4.1# head /etc/httpd/conf/httpd.conf
#
# This is the main Apache server configuration file. It contains the

# configuration directives that give the server its instructions.
# See <URL:http://httpd.apache.org/docs/2.2/> for detailed information.
# In particular, see
# <URL:http://httpd.apache.org/docs/2.2/mod/directives.html>
# for a discussion of each configuration directive.
#
#
# Do NOT simply read the instructions in here without understanding

Ira

Quoted 2 lines

-- Alex

Signature · 9 lines

Ira Weiny
Member of Technical Staff
Lawrence Livermore National Lab
925-423-8008
[email protected]
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html