Skip to contentSkip to conversation

PATCH: opensm enhancements

PATCH Jeff Becker 2 participants 4 messages Jun 26, 2013 – Jul 3, 2013 3 messages not in the archive InfiniBand

lore
Share & tools
lore
#0 lore raw

Hi Hal. At the OFA workshop, I mentioned that I've been working on some modifications to opensm that we use at NASA. Following extensive testing of these applied to opensm 3.3.13 (the version we run here), I have ported these to top of tree opensm, and have tested them on a small cluster.

The first patch modifies the console logflush command to take "on" or "off" as an argument for toggling. The second (more extensive) patch adds a command line option to specify a file in which each line contains a switch GUID/port pair to be ignored by opensm. The idea is to specify this file when you start opensm (it can be empty), and add ports to ignore (one per line for each end of a connection) to the file. At the next heavy sweep (or HUP) the sm will reprogram the forwarding tables without including the ignored links. We use this for replacing cables, as well as for system expansion (adding new racks).

Please let me know if you have any questions/issues with these. Thanks.

-jeff

Quoted 1 line

From cfb1c75a2b3fe7862f376bba44ebe3671b976ccd Mon Sep 17 00:00:00 2001

From: Jeffrey C. Becker <[email protected]>
Date: Tue, 25 Jun 2013 10:29:45 -0700
Subject: [PATCH 1/2] opensm: permit toggling log flush from console

---
 opensm/osm_console.c |   18 ++++++++++++++++--
 1 files changed, 16 insertions(+), 2 deletions(-)
opensm/osm_console.c +16 -2
index 0f80bdb..c065453 100644
@@ -178,7 +178,7 @@ static void help_status(FILE * out, int detail)
 
 static void help_logflush(FILE * out, int detail)
 {
-	fprintf(out, "logflush -- flush the opensm.log file\n");
+	fprintf(out, "logflush [on|off] -- toggle opensm.log file flushing\n");
 }
 
 static void help_querylid(FILE * out, int detail)
@@ -599,7 +599,21 @@ static void sweep_parse(char **p_last, osm_opensm_t * p_osm, FILE * out)
 
 static void logflush_parse(char **p_last, osm_opensm_t * p_osm, FILE * out)
 {
-	fflush(p_osm->log.out_port);
+	char *p_cmd;
+
+	p_cmd = next_token(p_last);
+	if (!p_cmd ||
+	    (strcmp(p_cmd, "on") != 0 && strcmp(p_cmd, "off") != 0)) {
+		fprintf(out, "Invalid logflush command\n");
+		help_sweep(out, 1);
+	} else {
+		if (strcmp(p_cmd, "on") == 0) {
+			p_osm->log.flush = TRUE;
+	                fflush(p_osm->log.out_port);
+                }
+		else
+			p_osm->log.flush = FALSE;
+	}
 }
 
 static void querylid_parse(char **p_last, osm_opensm_t * p_osm, FILE * out)
Signature · 1 line
1.7.1
Quoted 1 line

From 12a80ef0a81134edc0be779ae14ceafd41c2c124 Mon Sep 17 00:00:00 2001

From: Jeffrey C. Becker <[email protected]>
Date: Tue, 25 Jun 2013 10:36:17 -0700
Subject: [PATCH 2/2] opensm: add option to ignore guid/port pairs specified in a file

---
 include/opensm/osm_subnet.h    |    4 ++
 include/opensm/osm_ucast_mgr.h |   18 +++++++
 opensm/main.c                  |   15 ++++++-
 opensm/osm_drop_mgr.c          |   40 +++++++++++++++++
 opensm/osm_node_info_rcv.c     |    8 +++-
 opensm/osm_state_mgr.c         |    9 ++++
 opensm/osm_subnet.c            |    8 +++
 opensm/osm_ucast_mgr.c         |   96 ++++++++++++++++++++++++++++++++++++++++
 8 files changed, 196 insertions(+), 2 deletions(-)
include/opensm/osm_subnet.h +4 -0
index 2f98ae0..6dac079 100644
@@ -298,6 +298,7 @@ typedef struct osm_subn_opt {
 	char *console;
 	uint16_t console_port;
 	char *port_prof_ignore_file;
+	char *ignore_ports_file;
 	char *hop_weights_file;
 	char *port_search_ordering_file;
 	boolean_t port_profile_switch_nodes;
@@ -512,6 +513,9 @@ typedef struct osm_subn_opt {
 *	port_prof_ignore_file
 *		Name of file with port guids to be ignored by port profiling.
 *
+*	ignore_ports_file
+*		Name of file with port guids to be ignored in discovery
+*
 *	port_profile_switch_nodes
 *		If TRUE will count the number of switch nodes routed through
 *		the link. If FALSE - only CA/RT nodes are counted.
include/opensm/osm_ucast_mgr.h +18 -0
index c534b7e..5f79ad0 100644
@@ -101,6 +101,7 @@ typedef struct osm_ucast_mgr {
 	boolean_t some_hop_count_set;
 	cl_qmap_t cache_sw_tbl;
 	boolean_t cache_valid;
+	cl_qlist_t ports_to_ignore;
 } osm_ucast_mgr_t;
 /*
 * FIELDS
@@ -137,6 +138,23 @@ typedef struct osm_ucast_mgr {
 *	Unicast Manager object
 *********/
 
+/* struct to keep track of ports to be ignored in discovery */
+
+typedef struct port_to_ignore {
+	uint64_t guid;
+	unsigned port;
+} port_to_ignore_t;
+
+/* delete list of ports to ignore */
+boolean_t delete_ports_to_ignore(IN osm_ucast_mgr_t * p_mgr);
+
+/* initialize list of ports to ignore */
+boolean_t setup_ports_to_ignore(IN osm_ucast_mgr_t * p_mgr);
+
+/* lookup guid,port in list of ports to ignore */
+boolean_t lookup_ignore_port(IN osm_ucast_mgr_t * p_mgr, osm_node_t * p_node,
+                                    const uint8_t port_num);
+
 /****f* OpenSM: Unicast Manager/osm_ucast_mgr_construct
 * NAME
 *	osm_ucast_mgr_construct
opensm/main.c +14 -1
index 9349d79..5152134 100644
@@ -290,6 +290,9 @@ static void show_usage(void)
 	       "          This option provides the means to define a set of ports\n"
 	       "          (by guid) that will be ignored by the link load\n"
 	       "          equalization algorithm.\n\n");
+	printf("--ignore_ports, -j <ignore-ports-file>\n"
+	       "          This option provides the means to define a set of ports\n"
+	       "          (by guid) that will be ignored by opensm.\n\n");
 	printf("--hop_weights_file, -w <path to file>\n"
 	       "          This option provides the means to define a weighting\n"
 	       "          factor per port for customizing the least weight\n"
@@ -607,7 +610,7 @@ int main(int argc, char *argv[])
 	const char *config_file = NULL;
 	uint32_t val;
 	const char *const short_option =
-	    "F:c:i:w:O:f:ed:D:g:l:L:s:t:a:u:m:X:R:zM:U:S:P:Y:ANZ:WBIQvVhoryxp:n:q:k:C:G:H:";
+	    "F:c:i:j:w:O:f:ed:D:g:l:L:s:t:a:u:m:X:R:zM:U:S:P:Y:ANZ:WBIQvVhoryxp:n:q:k:C:G:H:";
 
 	/*
 	   In the array below, the 2nd parameter specifies the number
@@ -623,6 +626,7 @@ int main(int argc, char *argv[])
 		{"debug", 1, NULL, 'd'},
 		{"guid", 1, NULL, 'g'},
 		{"ignore_guids", 1, NULL, 'i'},
+		{"ignore_ports", 1, NULL, 'j'},
 		{"hop_weights_file", 1, NULL, 'w'},
 		{"dimn_ports_file", 1, NULL, 'O'},
 		{"port_search_ordering_file", 1, NULL, 'O'},
@@ -766,6 +770,15 @@ int main(int argc, char *argv[])
 			       opt.port_prof_ignore_file);
 			break;
 
+		case 'j':
+			/*
+			   Specifies ignore ports file.
+			 */
+			SET_STR_OPT(opt.ignore_ports_file, optarg);
+			printf(" Ignore Ports File = %s\n",
+			       opt.ignore_ports_file);
+			break;
+
 		case 'w':
 			SET_STR_OPT(opt.hop_weights_file, optarg);
 			printf(" Hop Weights File = %s\n",
opensm/osm_drop_mgr.c +40 -0
index b309273..6886827 100644
@@ -480,6 +480,46 @@ static void drop_mgr_check_node(osm_sm_t * sm, IN osm_node_t * p_node)
 			}
 		}
 	}
+       /* Unlink ignored links */
+       if (sm->p_subn->opt.ignore_ports_file) {
+                for (port_num = 1; port_num < p_node->physp_tbl_size; port_num++) {
+                        if (lookup_ignore_port(&sm->ucast_mgr, p_node, port_num)) {
+                                p_physp = osm_node_get_physp_ptr(p_node, port_num);
+                                if (!p_physp)
+                                        continue;
+                                p_remote_physp = osm_physp_get_remote(p_physp);
+                                if (!p_remote_physp)
+                                        continue;
+
+                                p_remote_node =
+                                    osm_physp_get_node_ptr(p_remote_physp);
+                                remote_port_num =
+                                    osm_physp_get_port_num(p_remote_physp);
+
+                                OSM_LOG(sm->p_log, OSM_LOG_INFO,
+                                        "Ignored link: "
+                                        "Unlinking local node 0x%" PRIx64
+                                        ", port %u"
+                                        "\n\t\t\t\tand remote node 0x%" PRIx64
+                                        ", port %u\n.",
+                                        cl_ntoh64(osm_node_get_node_guid
+                                                  (p_node)), port_num,
+                                        cl_ntoh64(osm_node_get_node_guid
+                                                  (p_remote_node)),
+                                        remote_port_num);
+
+                                if (sm->ucast_mgr.cache_valid)
+                                        osm_ucast_cache_add_link(&sm->ucast_mgr,
+                                                                 p_physp,
+                                                                 p_remote_physp);
+
+                                osm_node_unlink(p_node, (uint8_t) port_num,
+                                                p_remote_node,
+                                                (uint8_t) remote_port_num);
+                        }
+                }
+        }
+
 Exit:
 	OSM_LOG_EXIT(sm->p_log);
 	return;
opensm/osm_node_info_rcv.c +7 -1
index 592f2de..eab14a2 100644
@@ -166,6 +166,7 @@ static void ni_rcv_set_links(IN osm_sm_t * sm, osm_node_t * p_node,
 						p_neighbor_node,
 						p_ni_context->port_num));
 
+
 	if (osm_node_link_exists(p_node, port_num,
 				 p_neighbor_node, p_ni_context->port_num)) {
 		OSM_LOG(sm->p_log, OSM_LOG_DEBUG, "Link already exists\n");
@@ -256,8 +257,9 @@ static void ni_rcv_set_links(IN osm_sm_t * sm, osm_node_t * p_node,
 					       p_neighbor_node,
 					       p_ni_context->port_num);
 
+
 	osm_node_link(p_node, port_num, p_neighbor_node,
-		      p_ni_context->port_num);
+			      p_ni_context->port_num);
 
 	p_physp = osm_node_get_physp_ptr(p_node, port_num);
 	p_remote_physp = osm_node_get_physp_ptr(p_neighbor_node,
@@ -309,6 +311,10 @@ static void ni_rcv_get_port_info(IN osm_sm_t * sm, IN osm_node_t * node,
 	context.pi_context.active_transition = FALSE;
 
 	for (; port < num_ports; port++) {
+		if (sm->p_subn->opt.ignore_ports_file &&
+		    lookup_ignore_port(&sm->ucast_mgr, node, port)) {
+                	continue;
+		}
 		status = osm_req_get(sm, osm_physp_get_dr_path_ptr(physp),
 				     IB_MAD_ATTR_PORT_INFO, cl_hton32(port),
 				     CL_DISP_MSGID_NONE, &context);
opensm/osm_state_mgr.c +9 -0
index 1b73834..8ed7557 100644
@@ -1087,6 +1087,15 @@ static void do_sweep(osm_sm_t * sm)
 				"osm_subn_rescan_conf_file failed\n");
 		else
 			config_parsed = 1;
+		if (sm->p_subn->opt.ignore_ports_file) {
+                        boolean_t ports_removed, ports_added;
+			ports_removed = delete_ports_to_ignore(&sm->ucast_mgr);
+			ports_added = setup_ports_to_ignore(&sm->ucast_mgr);
+                        if (ports_removed || ports_added) {
+                                sm->p_subn->force_reroute = TRUE;
+		                sm->p_subn->ignore_existing_lfts = TRUE;
+                        }
+		}
 	}
 
 	if (sm->p_subn->sm_state != IB_SMINFO_STATE_MASTER &&
opensm/osm_subnet.c +8 -0
index 7ab1671..cfbf85e 100644
@@ -735,6 +735,7 @@ static const opt_rec_t opt_tbl[] = {
 	{ "polling_retry_number", OPT_OFFSET(polling_retry_number), opts_parse_uint32, NULL, 1 },
 	{ "force_heavy_sweep", OPT_OFFSET(force_heavy_sweep), opts_parse_boolean, NULL, 1 },
 	{ "port_prof_ignore_file", OPT_OFFSET(port_prof_ignore_file), opts_parse_charp, NULL, 0 },
+	{ "ignore_ports_file", OPT_OFFSET(ignore_ports_file), opts_parse_charp, NULL, 0 },
 	{ "hop_weights_file", OPT_OFFSET(hop_weights_file), opts_parse_charp, NULL, 0 },
 	{ "dimn_ports_file", OPT_OFFSET(port_search_ordering_file), opts_parse_charp, NULL, 0 },
 	{ "port_search_ordering_file", OPT_OFFSET(port_search_ordering_file), opts_parse_charp, NULL, 0 },
@@ -1009,6 +1010,7 @@ static void subn_opt_destroy(IN osm_subn_opt_t * p_opt)
 {
 	free(p_opt->console);
 	free(p_opt->port_prof_ignore_file);
+	free(p_opt->ignore_ports_file);
 	free(p_opt->hop_weights_file);
 	free(p_opt->port_search_ordering_file);
 	free(p_opt->routing_engine_names);
@@ -1511,6 +1513,7 @@ void osm_subn_set_default_opt(IN osm_subn_opt_t * p_opt)
 	p_opt->qos_policy_file = strdup(OSM_DEFAULT_QOS_POLICY_FILE);
 	p_opt->accum_log_file = TRUE;
 	p_opt->port_prof_ignore_file = NULL;
+	p_opt->ignore_ports_file = NULL;
 	p_opt->hop_weights_file = NULL;
 	p_opt->port_search_ordering_file = NULL;
 	p_opt->port_profile_switch_nodes = FALSE;
@@ -2363,6 +2366,11 @@ int osm_subn_output_conf(FILE *out, IN osm_subn_opt_t * p_opts)
 		p_opts->port_prof_ignore_file : null_str);
 
 	fprintf(out,
+		"# Name of file with port guids to be ignored in discovery\n"
+		"ignore_ports_file %s\n\n", p_opts->ignore_ports_file ?
+		p_opts->ignore_ports_file : null_str);
+
+	fprintf(out,
 		"# The file holding routing weighting factors per output port\n"
 		"hop_weights_file %s\n\n",
 		p_opts->hop_weights_file ? p_opts->hop_weights_file : null_str);
opensm/osm_ucast_mgr.c +96 -0

108 lines not on this page · raw message

Signature · 1 line
1.7.1

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw

HI Jeff,

Quoted 5 lines · Jeff Becker

Hi Hal. At the OFA workshop, I mentioned that I've been working on some modifications to opensm that we use at NASA. Following extensive testing of these applied to opensm 3.3.13 (the version we run here), I have ported these to top of tree opensm, and have tested them on a small cluster.

Thanks for getting this done! For future reference, patches should be sent as plain text as this makes it easier to comment.

Quoted 2 lines

The first patch modifies the console logflush command to take "on" or "off" as an argument for toggling.

Thanks. Applied.

Quoted 8 lines

The second (more extensive) patch
adds a command line option to specify a file in which each line contains a switch GUID/port pair to be ignored by opensm. The idea is to specify this file when you start opensm (it can be empty), and add ports to ignore (one per line for each end of a connection) to the file. At the next heavy sweep (or HUP) the sm will reprogram the forwarding tables without including the ignored links. We use this for replacing cables, as well as for system expansion (adding new racks).

I'll comment on this one later.

-- Hal

Quoted 3 lines

Please let me know if you have any questions/issues with these. Thanks.

-jeff

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 51D3FBA7.9040604-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org

#4 lore raw

Hi Hal,

I have some testing info about the second patch below.

Quoted 10 lines · Hal Rosenstock

HI Jeff,

Quoted 5 lines · Jeff Becker

Hi Hal. At the OFA workshop, I mentioned that I've been working on some modifications to opensm that we use at NASA. Following extensive testing of these applied to opensm 3.3.13 (the version we run here), I have ported these to top of tree opensm, and have tested them on a small cluster.

Thanks for getting this done! For future reference, patches should be sent as plain text as this makes it easier to comment.

OK. So I just send the output of git-format-patch directly? It appears to be formatted properly.

Quoted 13 lines
Quoted 3 lines

The first patch modifies the console logflush command to take "on" or "off" as an argument for toggling.

Thanks. Applied.

Quoted 8 lines

The second (more extensive) patch
adds a command line option to specify a file in which each line contains a switch GUID/port pair to be ignored by opensm. The idea is to specify this file when you start opensm (it can be empty), and add ports to ignore (one per line for each end of a connection) to the file. At the next heavy sweep (or HUP) the sm will reprogram the forwarding tables without including the ignored links. We use this for replacing cables, as well as for system expansion (adding new racks).

Quoted 1 line

I'll comment on this one later.

Dale (cc'd) did some testing with my patch on Pleiades in preparation for a system augmentation (new racks) happening soon. He found that the SM correctly produces routes that do not use links marked to be ignored, but when you then remove or disable the links, the SM re-routes the fabric anyway and comes up with different routes than before. This rerouting causes problems with existing connections. There also appears to be a bookkeeping problem such that some of these links get added to the SM's "light sampling" list and never get removed. This ties up outstanding MAD packet slots, causing the SM to become unresponsive for several seconds every time it reviews its light sampling list.

I'm working on fixing these. I'll take care of the second problem (incorrectly getting added to the light sampling list) first. Is it possible this problem is related to the re-routing on port disable problem? Anyhow, if you have any specific comments about these issues, that would be great. Thanks, and have a great Fourth of July.

-jeff

Quoted 6 lines

-- Hal

Quoted 3 lines

Please let me know if you have any questions/issues with these. Thanks.

-jeff

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw

Hi again Jeff,

Quoted 33 lines · Jeff Becker

Hi Hal,

I have some testing info about the second patch below.

Quoted 10 lines · Hal Rosenstock

HI Jeff,

Quoted 5 lines · Jeff Becker

Hi Hal. At the OFA workshop, I mentioned that I've been working on some modifications to opensm that we use at NASA. Following extensive testing of these applied to opensm 3.3.13 (the version we run here), I have ported these to top of tree opensm, and have tested them on a small cluster.

Thanks for getting this done! For future reference, patches should be sent as plain text as this makes it easier to comment.

OK. So I just send the output of git-format-patch directly? It appears to be formatted properly.

Quoted 13 lines
Quoted 3 lines

The first patch modifies the console logflush command to take "on" or "off" as an argument for toggling.

Thanks. Applied.

Quoted 8 lines

The second (more extensive) patch
adds a command line option to specify a file in which each line contains a switch GUID/port pair to be ignored by opensm. The idea is to specify this file when you start opensm (it can be empty), and add ports to ignore (one per line for each end of a connection) to the file. At the next heavy sweep (or HUP) the sm will reprogram the forwarding tables without including the ignored links. We use this for replacing cables, as well as for system expansion (adding new racks).

Quoted 1 line

I'll comment on this one later.

Quoted 10 lines

Dale (cc'd) did some testing with my patch on Pleiades in preparation for a system augmentation (new racks) happening soon. He found that the SM correctly produces routes that do not use links marked to be ignored, but when you then remove or disable the links, the SM re-routes the fabric anyway and comes up with different routes than before. This rerouting causes problems with existing connections. There also appears to be a bookkeeping problem such that some of these links get added to the SM's "light sampling" list and never get removed. This ties up outstanding MAD packet slots, causing the SM to become unresponsive for several seconds every time it reviews its light sampling list.

Yes, this is one of several issues with using this approach.

I plan on detailing these later as well as posting a slightly different approach for this but that may take a little longer...

Quoted 7 lines

I'm working on fixing these. I'll take care of the second problem (incorrectly getting added to the light sampling list) first. Is it possible this problem is related to the re-routing on port disable problem? Anyhow, if you have any specific comments about these issues, that would be great.

Thanks, and have a great Fourth of July.

Thanks; you too!

-- Hal

Quoted 9 lines

-jeff

Quoted 6 lines

-- Hal

Quoted 3 lines

Please let me know if you have any questions/issues with these. Thanks.

-jeff

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html